Inference acceleration method, device, medium, product and system for pre-trained models

Abstract

The present application discloses a method, device, medium, product and system for accelerating the reasoning of a pre-trained model, and relates to the field of reasoning technology of pre-trained models. The method extracts semantic features at different levels from the input text, ensuring that the model can capture information in various aspects such as vocabulary, grammar, context and long-term dependencies, which helps to improve the expressiveness and generalization capabilities of the model. The important fusion features and non-important fusion features of multiple fusion features are determined based on the importance of the fusion features, and the non-important fusion features are pruned to obtain pruned fusion features, so as to prune out those non-important fusion features that have little impact on the final result. Therefore, the method solves the problem that the existing solutions are difficult to achieve the expected effect on the acceleration of the reasoning speed of the pre-trained model.

Description

Technical Field

[0001] The present application relates to the field of reasoning technology of pre-trained models, and in particular to a reasoning acceleration method of a pre-trained model, a reasoning acceleration device of a pre-trained model, a computer-readable storage medium, a computer program product, and a reasoning acceleration system of a pre-trained model. Background Art

[0002] Large-scale pre-trained models have become standard for natural language processing (NLP) and other machine learning tasks. However, these large models often have billions or more parameters. While they significantly improve model performance, they also come with high computational costs and long inference times. While existing solutions can reduce model inference time by accelerating inference through pruning, quantization, knowledge distillation (KD), and sparsity, the inherent complexity of pre-trained models makes it difficult for a single technique to achieve satisfactory acceleration. Summary of the Invention

[0003] The present application provides a method for accelerating the reasoning of a pre-trained model, a device for accelerating the reasoning of a pre-trained model, a computer-readable storage medium, a computer program product, and a system for accelerating the reasoning of a pre-trained model, so as to at least solve the problem that the existing solutions are difficult to achieve the expected effect in accelerating the reasoning speed of the pre-trained model.

[0004] The present application provides a method for accelerating reasoning of a pre-trained model, including: obtaining natural language text and extracting semantic features at different levels from the natural language text; performing feature fusion on the semantic features at different levels to obtain multiple fusion features; determining important fusion features and unimportant fusion features of the multiple fusion features based on the importance of the fusion features, wherein the importance represents the degree of influence on the output results of the pre-trained model; performing trimming on the unimportant fusion features to obtain trimmed fusion features, and applying the trimmed fusion features and the important fusion features to the reasoning of the pre-trained model.

[0005] The present application also provides an inference acceleration device for a pre-trained model, including: a first processing unit, used to obtain natural language text and extract semantic features of different levels from the natural language text; a second processing unit, used to perform feature fusion on semantic features of different levels to obtain multiple fusion features; a third processing unit, used to determine important fusion features and unimportant fusion features of multiple fusion features based on the importance of the fusion features, wherein the importance represents the degree of influence on the output results of the pre-trained model; a fourth processing unit, used to trim unimportant fusion features to obtain trimmed fusion features, and apply the trimmed fusion features and important fusion features to the inference of the pre-trained model.

[0006] The present application also provides a computer-readable storage medium, which stores a computer program, wherein when the computer program is executed by a processor, the steps of the inference acceleration method of any of the above-mentioned pre-trained models are implemented.

[0007] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the inference acceleration method of any of the above-mentioned pre-trained models.

[0008] The present application also provides an inference acceleration system for a model, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include an inference acceleration method for executing any pre-trained model.

[0009] Through this application, semantic features at different levels are extracted from the input text, ensuring that the model can capture multiple aspects of information such as vocabulary, grammar, context, and long-term dependencies, which helps to improve the expressiveness and generalization capabilities of the model. Based on the importance of the fusion features, important fusion features and unimportant fusion features of multiple fusion features are determined, and the unimportant fusion features are pruned to obtain pruned fusion features, so as to prune out those unimportant fusion features that have little impact on the final result. Therefore, the problem that the existing solutions are difficult to achieve the expected effect on the acceleration of the inference speed of the pre-trained model is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0011] Figure 1 A flowchart of a method for accelerating reasoning of a pre-trained model provided in an embodiment of the present application;

[0012] Figure 2 A schematic diagram showing the relationship between the data loading layer, model layer, and output layer provided in an embodiment of the present application;

[0013] Figure 3 A schematic diagram of the process of accelerating the reasoning of the pre-trained model provided in the embodiment of the present application;

[0014] Figure 4 A schematic diagram of the structure of an inference acceleration device for a pre-trained model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0016] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0017] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0018] The specific application environment architecture or specific hardware architecture on which the execution of the inference acceleration method of the pre-trained model depends is described here.

[0019] An embodiment of the present application provides a method for accelerating reasoning of a pre-trained model, and the method is described in detail in conjunction with the execution process of the method for accelerating reasoning of a pre-trained model.

[0020] like Figure 1 As shown, a method for accelerating inference of a pre-trained model includes the following steps:

[0021] Step S101: obtaining a natural language text and extracting semantic features at different levels from the natural language text;

[0022] Natural language text refers to textual information composed of the language and characters used in daily human communication, including both written and spoken text. In the field of natural language processing (NLP), natural language text is the target of analysis, understanding, and generation, involving analysis at multiple levels, including grammar, semantics, and discourse structure.

[0023] Multi-level feature extraction technology refers to extracting features at different levels of abstraction by designing a multi-layered neural network structure in deep learning models, particularly convolutional neural networks (CNNs) and recurrent neural networks (RNNs). Lower layers typically extract simple features from the raw signal (such as edges in images or parts of speech in natural language), while higher layers further abstract these features to form more complex concepts or semantic features, helping the model understand the deeper meaning of the input data.

[0024] Semantic features are meaning-related features that a model can capture when processing text or speech data. In natural language processing, semantic features can refer to the meaning of words, the implications of sentences, and even the overall idea of ​​a passage. These features are crucial for understanding context and inferring intent and sentiment.

[0025] Step S102: performing feature fusion on semantic features at different levels to obtain multiple fused features;

[0026] Step S102 requires adaptive feature fusion technology, which dynamically adjusts the fusion method and weighting of features from different sources or levels within the model. Based on the characteristics of the input data and the requirements of the current task, it automatically determines which features are more important and which can be weakened, thereby utilizing information more efficiently and accurately in the model decision-making process.

[0027] Step S103, determining important fusion features and unimportant fusion features of the plurality of fusion features based on the importance of the fusion features, where the importance represents the degree of influence on the output result of the pre-training model;

[0028] The non-important fusion features are fusion features that have a smaller impact on the output results of the pre-training model than the important fusion features.

[0029] In step S104, non-important fusion features are pruned to obtain pruned fusion features, and the pruned fusion features and important fusion features are applied to the reasoning of the pre-trained model.

[0030] Step S104 requires a dynamic pruning strategy. This strategy selectively closes or skips some computational branches or layers based on real-time analysis of inputs and outputs during model execution to reduce unnecessary computational overhead. Unlike static pruning, dynamic pruning allows for flexible adjustments to computational load based on actual conditions, while maintaining model performance, improving operational efficiency.

[0031] Real-time evaluation refers to a technique that continuously analyzes model state, output results, or input data during model inference or execution to rapidly adjust model behavior or parameters. In deep learning, this mechanism can be used for dynamic pruning, adaptive learning rate adjustment, or model performance monitoring, helping models make better decisions in complex and changing environments.

[0032] In the above steps, semantic features at different levels are extracted from the input text, ensuring that the model can capture multiple aspects of information such as vocabulary, grammar, context, and long-term dependencies, which helps to improve the expressiveness and generalization capabilities of the model. Based on the importance of the fusion features, the important fusion features and unimportant fusion features of multiple fusion features are determined, and the unimportant fusion features are pruned to obtain pruned fusion features, so as to prune out those unimportant fusion features that have little impact on the final result. Therefore, the problem that the existing solutions are difficult to achieve the expected effect on the acceleration of the inference speed of the pre-trained model is solved.

[0033] like Figure 2 As shown in Figure 2, the above method includes but is not limited to three parts: a data loading layer, a model layer, and an output layer. The data loading layer is responsible for preparing input samples; the model layer is composed of multiple Transformer encoders and decoders, each of which is further subdivided into functional modules such as the Multi-Head Self-Attention mechanism (MHA), a Feed-Forward Neural Network (FNN), and a Residual Connection; and the output layer performs classification or regression operations on the final feature vector to obtain the prediction result.

[0034] First, a dynamic padding strategy is employed to dynamically pad the sequences in each batch based on the longest sequence length. By grouping sequences of similar length into batches and converting the text sequences into vectors through tokenization as input, the amount of padding can be reduced, thereby accelerating computation. This strategy effectively improves data processing speed and efficiency. Next, multiple rounds of Transformer encoder and decoder processing are performed, effectively capturing long-range dependencies. Finally, the Softmax function is used. (Softmax is a commonly used activation function in machine learning and deep learning, primarily used in the output layer of multi-classification tasks. It converts a vector of arbitrary real numbers into a probability distribution. Specifically, the Softmax function maps each element in the input vector to a value between (0, 1), and the sum of all outputs is 1, thus interpreting it as a probability.) The probability distribution for each class is calculated, and the class with the highest probability is selected as the output. On this basis, a method for accelerating inference of pre-trained models based on hierarchical feature fusion and dynamic clipping strategy is proposed. The former aims to enhance the expressiveness of the model through multi-level feature interaction, while the latter intelligently clips non-important fusion features without compromising accuracy, thereby achieving the purpose of acceleration.

[0035] In a specific embodiment of the present application, clipping of non-important fusion features includes one of the following: clipping all non-important fusion features; clipping part of the non-important fusion features in all non-important fusion features; clipping at least part of the sub-features of the non-important fusion features in all non-important fusion features; dividing the non-important fusion features into a first part and a second part, and clipping at least part of the sub-features of the non-important fusion features of the non-important fusion features in the first part, and clipping the non-important fusion features in the second part.

[0036] Different layers or modules of the model capture semantic information of different granularities:

[0037] Low-level features: part of speech, basic phrase structure, local word meaning (usually from the bottom layer of the model).

[0038] Mid-level features: syntactic relations, sentence-level semantics, and context-dependent word meanings (usually from the mid-level model).

[0039] High-level features: paragraph / chapter-level semantics, theme, sentiment, and intent (usually from the top level of the model or a specific aggregation layer).

[0040] Here we only take the example of clipping some of the non-important fusion features from all the non-important fusion features to obtain the clipped fusion features (the same applies to other clipping, so we will not go into details). For example, if the non-important fusion features include bottom-level features, middle-level features and high-level features, then we can only clip the bottom-level features of the non-important fusion features, or only clip the bottom-level features and middle-level features of the non-important fusion features and retain the high-level features.

[0041] For another example, the unimportant fusion feature includes two levels of sub-features, namely, first-level sub-features and second-level sub-features. Only the first-level sub-features of the unimportant fusion feature are clipped, and the second-level sub-features are retained.

[0042] Specifically, pruning all non-important fused features can significantly reduce the model's computational requirements, as the model no longer needs to perform computations related to non-important features. This directly reduces computation time and is particularly important for large-scale models and real-time applications. However, completely pruning all non-important features may have a slight impact on model performance. Trimming some features is a compromise, reducing computational cost while maintaining the model's prediction accuracy as much as possible. Sub-features may be redundant or low-impact components of non-important features. Trimming these sub-features can further reduce computation while maintaining the integrity of key features, which is particularly beneficial for accelerating model inference. This is a more refined pruning strategy that considers not only the importance of the entire feature but also sub-features within the feature. This approach allows the model to focus more on key information while reducing unnecessary computation and improving operational efficiency. Trimming non-important fused features also reduces the amount of information that needs to be stored, thereby reducing memory usage. This is particularly important for applications deployed on edge devices or with strict memory constraints.

[0043] In a specific embodiment of the present application, some non-important fusion features are clipped from all non-important fusion features to obtain clipped fusion features, including: sorting the non-important fusion features in descending order according to their importance to obtain a non-important fusion feature sequence; and clipping non-important fusion features whose importance is less than a preset importance in the non-important fusion feature sequence.

[0044] Specifically, each computational branch in the model consumes a certain amount of computing resources and time, and fusion features that contribute little to the overall performance of the model may lead to computational redundancy. Especially in large-scale data and complex model structures, by sorting and pruning unimportant fusion features, unnecessary computations can be significantly reduced, improving the model's running speed and efficiency. The storage requirements of the model are directly related to the complexity of the model, including the number and width of the neural network layers and the number of fusion features. Pruning unimportant fusion features reduces the parameters and intermediate results that need to be stored in the model, thereby reducing storage requirements. This is especially important for deployment on devices with limited computing resources, such as mobile devices or edge computing devices.

[0045] In a specific embodiment of the present application, different levels of semantic features are extracted from natural language text, including: introducing a self-adjusting activation function and a residual connection for enhancing information transfer into a pre-trained model to extract different levels of semantic features from the natural language text, and using key-value pairs to represent the semantic features, and the self-adjusting activation function is used to provide a nonlinear transformation for the pre-trained model;

[0046] Feature fusion is performed on semantic features at different levels to obtain multiple fused features, including: based on semantic features, local sensitive hashing is used to approximately search for the most relevant key-value pairs to determine multiple target key-value pairs; when the type of the target key-value pairs is linear, the fusion weights of the semantic features of each level corresponding to the target key-value pairs are determined according to the task complexity; based on the fusion weights, a linear fusion function is used to perform weighted summation on the target key-value pairs to obtain multiple fused features.

[0047] The key-value pair is the vectorized representation of the semantic feature, and the key-value pair corresponds one-to-one to the semantic feature.

[0048] Specifically, this application provides a specific use scenario for obtaining fusion features: for example, developing a real-time translation application, where users can input long natural language texts, and the system needs to return translation results immediately. In such a scenario, the model not only has to process a large amount of text information, but also has to respond quickly, which places extremely high demands on the computational efficiency of the model. When processing long texts, the traditional Transformer model, because its self-attention mechanism involves the calculation of all word pairs, results in a huge amount of computation, making it difficult to achieve real-time response.

[0049] In the Transformer's Feedforward Neural Network (FNN), Swish (a self-regulating activation function, which has the following advantages over the traditional ReLU function: Self-regulation: The slope of the Swish function can be dynamically adjusted according to the input in the negative region, avoiding the "dead neuron" problem of ReLU, that is, the output is 0 when the input is negative, causing the gradient to disappear; Smoothness: The Swish function is continuous and smooth throughout the entire domain, which means that its derivative exists at all points, which is conducive to the propagation of the gradient and helps the model converge during the inference process; Nonlinearity: Swish still maintains the characteristics of nonlinear transformation, which is very necessary for building deep neural networks because it can help the network learn complex patterns in the data) function instead of the traditional ReLU activation function (Rectified Linear Unit is a commonly used activation function with the formula f(x) = max(0, x). In the positive range, the ReLU activation function is linear, with a derivative of 1. This facilitates rapid gradient propagation and accelerates the neural network's inference process. In the negative range, the output is 0, which helps the network automatically ignore unimportant features and achieve a certain degree of sparsity. The Swish function has smooth properties and can self-adjust during inference, providing better nonlinear transformations and helping the model learn more complex feature representations. Furthermore, the introduction of residual connections ensures smooth information transfer even as the network deepens, preventing the vanishing gradient problem and improving model stability and efficiency. The multi-head self-attention mechanism, rather than performing comprehensive computations for all positions, utilizes LSH (Locality Sensitive Hashing) to approximate searches for the most relevant key-value pairs, particularly in terms of semantic features. For large text sequences, LSH efficiently finds the subset of key-value pairs most relevant to the query, narrowing the focus of attention computation to candidate key-value pairs highly relevant to the query. This significantly reduces computational effort, especially when processing long dependencies, and significantly improves inference speed. The fusion weights of the features at each level corresponding to the target key-value pair are dynamically adjusted based on the complexity of the task. For example, in translation tasks, the model may rely more heavily on high-level, abstract semantic features. In this case, the weights of these high-level features should be appropriately increased. This allows the model to focus more closely on features critical to the task, accelerating the inference process without compromising translation quality. After determining the target key-value pairs and their corresponding fusion weights, a linear fusion function (such as weighted average) is used to perform a weighted summation of the target key-value pairs to generate the fused features. This method is simple and computationally efficient, making it suitable as a foundational fusion strategy for maintaining high efficiency across large computational workloads.

[0050] LSH is an algorithm for approximate nearest neighbor search, mainly used to process large-scale high-dimensional data sets. The core idea of ​​LSH is to map similar data points to the same hash bucket through a hash function, so that a set of data points closest to the query point can be quickly found without comparing all data points. This method has brought significant improvements in both computational efficiency and storage efficiency. Especially when processing large-scale data sets, such as text data and image data, LSH can greatly reduce the complexity of distance calculation, making approximate search feasible in practical application scenarios. In natural language processing, LSH is often used to optimize the self-attention mechanism of the Transformer model, accelerating model reasoning by reducing unnecessary key-value pair calculations.

[0051] The technical advantages of a specific usage scenario of fusion features are obtained: the introduction of self-adjusting activation functions, residual connections and local sensitive hashing technology effectively reduces the computational load of the model when processing long texts, greatly improves the inference speed of large-scale language models, and meets the performance requirements of real-time translation applications; it not only reduces the computational intensity, but also reduces the required memory resources, so that the model can run on resource-constrained devices (such as mobile devices), expanding the scope of application; by dynamically adjusting the fusion weights, the model can focus on utilizing the features most relevant to the task, even while optimizing efficiency, it can maintain high translation quality and semantic understanding accuracy; these technologies are not only applicable to translation tasks, but also widely applicable to other natural language processing fields, such as text classification, sentiment analysis, etc., providing a general and effective solution for accelerating pre-trained model inference.

[0052] In a specific embodiment of the present application, residual connections that enhance information transmission are introduced into the pre-trained model, including: determining target jump layers of multiple levels based on multiple different preset inter-layer residuals, wherein there is a preset inter-layer residual between any two adjacent levels; and performing jump-layer transmission on each level based on the target jump layer.

[0053] Specifically, there are certain deviations. The following are corrections and additions to the technical details:

[0054] Relying on residuals for skipping decisions is a non-standard practice. Skip connections are actually fixed by the network structure or dynamically controlled by a gating mechanism. Skip connections are pre-set, not ad hoc. The residual itself does not contain information about the skip target; the skip path is determined by the architecture design (e.g., skipping two layers in ResNet). The skip target (e.g., skipping from layer 3 to layer 5) is pre-defined and not calculated from the residual. There are three types of skip connections: fixed skips (e.g., skipping two layers in ResNet to a pre-set path, independent of the residual value); dynamic gated skips (e.g., skipping weights controlled by learnable parameters); and dense skips (e.g., skipping multiple layers to a pre-set path, with the residual as one of the inputs). Therefore, skipping is determined by the pre-set architecture design or dynamic gating unit calculations, and does not directly depend on the residual size. In deep neural networks, particularly the Transformer architecture, each layer produces specific output features that represent the model's understanding at different levels of abstraction. Inter-layer residuals involve computing the difference between the output of the current layer and the output of the previous layer. This information reflects the importance or unique contribution of the current layer's features relative to the previous layer's. Skip-layer transfer is a key feature of ResNet, allowing the network to pass information directly between certain layers without passing through all intermediate layers. This not only accelerates model execution but also ensures that the model focuses on and utilizes truly influential deep-layer features when handling complex tasks. The skip-layer transfer mechanism has been further refined. When the model determines that the residual information of a layer is relatively small or has little impact on the final task, it can skip several layers and pass the information directly to a more distant layer. The target layer is determined by the size of the residual. By skipping unnecessary layers, the computational effort is significantly reduced, especially in scenarios with a large number of layers. This efficiency improvement is particularly significant. Skip-layer transfer reduces reliance on computing resources, allowing the model to run with lower power consumption or less hardware resources, which is crucial for edge computing devices or resource-constrained scenarios. Through intelligent feature selection and information transfer, the model can better generalize to unseen data and reduce the risk of overfitting caused by over-reliance on features from specific layers.

[0055] Dynamic skipping based on residuals, introducing a gating mechanism:

[0056] Calculate the L2 norm of the residual (Euclidean norm, also known as the Euclidean norm, is a length measurement method of a vector):

[0057] ;

[0058] is the feature result outputted by the lth level, The feature result output by the l-1th level compresses the residual tensor into a scalar r for gating decision.

[0059] Generate a gating signal:

[0060] ;

[0061] Among them, g is the gate signal strength, σ represents the Sigmoid activation function, W is the weight parameter, r is the residual strength, and b is the bias parameter.

[0062] Decision jump layer:

[0063] ;

[0064] Among them, H output is the feature result output by the target skip layer, H l+k is the feature result outputted at the l+kth level, the threshold is a hyperparameter, and k is the preset jump step size. The size of is set using a level-dependent threshold, and the formula is:

[0065] ;

[0066] in, Indicates the skip layer threshold of layer l, l is the depth of the current layer in the model (layer index), usually counting from 0 or 1, and L is the total number of layers in the model. is the slope parameter, between 0.5 and 0.7, It is the basic threshold parameter, between -0.3 and 0.1.

[0067] In a specific embodiment of the present application, feature fusion is performed on semantic features at different levels to obtain multiple fused features, including: when the type of the target key-value pair is nonlinear, the weight of each semantic feature is adjusted by the gradient of the back propagation model to obtain the adjusted weight of each semantic feature; a voting mechanism is used to determine the feature fusion relationship, the feature fusion relationship represents other semantic features that need to be fused with each semantic feature; based on the adjusted weight and feature fusion relationship, weighted summation processing is performed on the semantic features at different levels to obtain multiple fused features.

[0068] Feature fusion combines features extracted from different levels (which may also include different sources, such as word vectors, syntactic features, and entity recognition results). Common fusion methods include: concatenation: directly concatenating different feature vectors end to end; weighted summation: assigning weights to features at different levels and then adding them together; attention mechanism: allowing the model to dynamically learn the importance of features at different levels for the current task and then perform a weighted fusion; gating mechanism: learning to control the flow of information, determining which level of feature information is passed to the next stage. A new goal is to create a more comprehensive and richer unified representation (i.e., fused features) that encompasses everything from detailed to global information.

[0069] The process of backpropagating the gradient of the model to determine the contribution of each feature to the prediction results of the pre-trained model includes the following steps:

[0070] Step 1: Forward propagation:

[0071] First, the model undergoes a normal forward propagation process, inputting specific sample data to obtain the model's prediction results. During this process, the model performs a series of mathematical operations on the input features based on its weights and bias parameters, and finally produces a predicted value at the output layer.

[0072] Step 2: Calculate the loss:

[0073] The difference between the model's predictions and the true labels is calculated, which is called loss. The choice of loss function depends on the specific problem. For example, mean squared error (MSE) is suitable for regression problems, while cross entropy loss is usually used for classification tasks.

[0074] Step 3: Back propagation;

[0075] Then, backpropagation is used to calculate the gradient of the loss with respect to each parameter (including weights and biases) layer by layer, starting from the output layer. In a multi-layer neural network, backpropagation decomposes the loss gradient of the output layer to each layer through the chain rule until the input layer.

[0076] Step 4: Calculate the gradient of the feature:

[0077] When backpropagation reaches the input layer, we can obtain the gradient of the loss function with respect to the input features. This is the direction and magnitude of change in the model's predictions when the input features change slightly. Therefore, a larger gradient means that the feature has a greater impact on the model's predictions.

[0078] Step 5: Feature importance assessment:

[0079] Next, the importance of features can be assessed in the following ways:

[0080] Absolute gradient value: The larger the absolute value of the gradient of a feature, the greater its contribution to the prediction result.

[0081] Integral gradient: Accumulating the gradient from a certain eigenvalue to the benchmark eigenvalue can provide a more comprehensive contribution estimate.

[0082] Gradient input: Multiplying the gradient of a feature by its original input value measures the relationship between the actual value of the input and its gradient, thereby reflecting the actual impact of the feature on the prediction result.

[0083] Specifically, when the pre-trained model processes nonlinear relationships, backpropagation is used to calculate the gradient contribution of each feature to the prediction result. Gradient information reflects the importance of each feature in the model prediction and helps to identify which features are most critical to the model decision. The calculated gradient contribution is used to generate a saliency map, which intuitively shows which feature regions have the greatest impact on the output results of the pre-trained model. Based on the saliency map, the weight distribution in the feature fusion process can be dynamically adjusted to ensure that the most important features receive more attention, while the computational contribution of non-critical features is relatively reduced. A voting mechanism is used between features at different levels to allow lower-level features to negotiate information with upper-level features (i.e., a voting mechanism is used to determine the feature fusion relationship representation and other semantic features required for feature fusion for each fusion requirement). This is essentially a self-organizing feature selection process, in which the contribution of lower-level features is determined by their influence on upper-level features, ensuring that the model's decision-making process fully considers global information and avoids prediction bias caused by neglecting local information. A depth-wise convolution operation is performed on the extracted features, which performs independent convolution on each feature channel. Compared to traditional convolution operations, this method reduces cross-channel interactions but can more finely process information within each channel. This is particularly important for feature extraction in nonlinear relationships because it can preserve the specificity of each channel while promoting effective fusion between features.

[0084] When dealing with nonlinear relationships, computing resources are allocated more intelligently while ensuring that the most important features are fully utilized, thereby improving the quality and efficiency of model decisions. Resource optimization and computational cost reduction: By reducing the computational investment in non-critical features, dynamically adjusting the weights in the feature fusion process, and independently processing feature channels in deep convolution operations, computational costs are significantly reduced. This optimization is particularly critical when dealing with large-scale, high-dimensional data. The generation of saliency maps provides a window into the model's internal decision-making process, allowing users and developers to better understand why the model makes specific predictions, thereby enhancing the model's interpretability and debuggability, especially in application scenarios requiring human-machine collaboration or trust. The voting mechanism and dynamic weight adjustment strategy enable the model to adaptively respond to different types of input data and changes in task requirements. This flexibility is crucial for processing variable real-world data.

[0085] The nonlinear fusion function can use depth-wise convolution to assign different weights to different channels. A cross-attention layer introduces a learnable attention module during the feature fusion stage. The query-key-value mechanism calculates the weights of features at each level. Gradient backpropagation through Gradient Cam adjusts feature importance using the saliency map. A voting mechanism allows lower-level features to negotiate their contribution to higher-level features. This further extracts nonlinear relationships between features and enhances their expressiveness.

[0086] Depth-wise convolution is a convolution operation used in convolutional neural networks (CNNs), particularly in mobile deep neural networks. While traditional convolution operations apply a single kernel to all input channels, depth-wise convolution breaks the convolution process down into two steps: first, a kernel is applied to each input channel (this is the "depth-wise" part), and then a 1x1 convolution (called a "point-wise" convolution) is used to combine the outputs of these channels. This significantly reduces the amount of computation and the number of parameters, thereby improving model efficiency.

[0087] Grad-CAM is a technique for visualizing the decision-making process of convolutional neural networks. It generates heatmaps by computing class-specific gradients relative to the feature maps of a layer in the network. Specifically, it uses the gradient information of the target class to weight the feature maps to highlight the regions of the input image that have the greatest influence on the decision for that class. This visualization method can help understand and explain the behavior of neural networks and is a powerful tool for analyzing model decisions.

[0088] The query-key-value mechanism is a core concept in the Transformer architecture and is used to implement the self-attention mechanism. It involves three main components:

[0089] Query: This is extracted from the input and is used to "ask" what information is relevant.

[0090] Key: It is also extracted from the input and represents the characteristics of the input data.

[0091] Value: It is the value corresponding to the Key and represents the output information.

[0092] The self-attention mechanism calculates the similarity between the query and the key (usually using a dot product) to generate an attention weight, which is then used to weight the value to generate the final output. This mechanism allows the model to focus on other related elements in the input sequence when processing each input element and is the basis of many modern models in natural language processing and computer vision.

[0093] Key: It is a vectorized representation of semantic features (such as text embedding vectors extracted by BERT and Transformer). It serves as an "index identifier" and stores high-level semantic information of the text (such as topic, intent, and entity type).

[0094] Value: Meta information or task-related parameters associated with the key: feature weight (such as the importance of the semantics under the current task), label (such as classification category), structured data (such as entity attributes in the knowledge graph).

[0095] In a specific embodiment of the present application, in the process of determining multiple target key-value pairs by approximately searching for the most relevant key-value pairs based on semantic features using locality-sensitive hashing, the method further includes:

[0096] according to , determine the hash value of the vector o of the current dimension;

[0097] Where h(o) is the hash value of the vector o of the current dimension, a is a standard Gaussian distributed random vector, b is an offset uniformly distributed in [0, w], and w is the shard width parameter.

[0098] Specifically, the core advantage of LSH lies in its ability to quickly find similar items in high-dimensional space. By mapping vectors to buckets (hash buckets), vectors with similar Euclidean distances have a greater probability of being mapped to the same bucket. When processing large-scale datasets, traditional full-scale computation methods become extremely slow and resource-intensive, while LSH can significantly accelerate this process. This is especially true when processing word embeddings or key-value pairs in large-scale language models, effectively reducing the computational complexity of the attention mechanism. LSH reduces computational complexity by reducing the number of direct comparisons of all vectors, which means that the required computing resources and storage space can be significantly reduced. This is especially important for models running on resource-constrained devices, such as mobile devices and edge servers, as it can effectively improve the model's operational efficiency and deployment feasibility. Since the hash function is designed based on the random vector a and the offset b, this method can better adapt to dynamic changes in data. Even when data distribution changes or the model continuously learns new information, LSH maintains its search efficiency and stability. Within the Transformer model's attention mechanism, this approach allows each query to be calculated only with a small number of key-value pairs within the same bucket during inference. Compared to the original mechanism, which requires calculations with all key-value pairs, this significantly reduces the amount of computation and greatly improves the model's inference speed. By using specific hash functions and parameters (such as the shard width w), the model's inference and optimization processes can be simplified. For example, instead of making detailed adjustments for every possible key-value pair, the model's performance can be indirectly optimized by adjusting the hash parameters, reducing the complexity and uncertainty of parameter tuning.

[0099] In a specific embodiment of the present application, based on the fusion weight, a linear fusion function is used to perform weighted summation on the target key-value pairs to obtain fusion features, including:

[0100] according to , determine the fusion features;

[0101] in, To fusion features, is the weight of the i-th feature, is the value of the i-th feature, and n is the total number of features.

[0102] Specifically, through weighted summation, the model can highlight the features that contribute most to the current task based on the weights learned during inference. This means that when making predictions, the model will place greater emphasis on these key features, thereby improving prediction accuracy and reliability. The linear fusion function has a simpler calculation process. Compared to complex nonlinear fusion, it reduces additional parameter adjustments and function operations, significantly improving the computational efficiency of feature fusion. This is especially true for large-scale key-value pair processing, significantly accelerating the inference process.

[0103] In a specific embodiment of the present application, before determining important fusion features and non-important fusion features of multiple fusion features based on the importance of the fusion features, the method further includes: obtaining the importance of the fusion features, wherein obtaining the importance of the fusion features includes: obtaining the initial importance of the fusion features; allocating a dynamic scaling factor to the key attention head based on a preset importance score to adjust the initial importance of the fusion features, thereby obtaining the adjusted importance of the fusion features;

[0104] Determining important fusion features and unimportant fusion features of multiple fusion features based on the importance of the fusion features, including: determining a preset importance based on current business needs, where the current business needs are related to the output results of the pre-trained model; determining the fusion feature as an important fusion feature when the adjusted importance of the fusion feature is greater than or equal to the preset importance; and determining the fusion feature as an unimportant fusion feature when the adjusted importance of the fusion feature is less than the preset importance.

[0105] Among them, a dynamic scaling factor (scaling range 0.5 to 2.0) is applied to the key attention heads, and real-time spectral analysis is performed on the multi-head attention matrix of the Transformer layer (the principal component contribution rate is 85%).

[0106] The Dynamic Scaling Factor directly adjusts the output of the attention head, but does not directly generate feature importance. The specific steps are as follows:

[0107] Step 1: Evaluate the importance of each attention head in the multi-head self-attention and generate an importance score {S_head};

[0108] Step 2: Assign a dynamic scaling factor {α_head} to each attention head based on {S_head};

[0109] Step 3: Use {α_head} to scale the output weights of each attention head to achieve dynamic adjustment;

[0110] The adjusted attention mechanism focuses more on key features, indirectly increasing the weights of important components in the fusion features.

[0111] In the context of dynamic attention adjustment, key-value pairs (KVs)—the key-value matrix in the attention mechanism—store the projected representation of semantic features. The dynamic scaling factor strengthens the focus on specific features, increasing the weight of corresponding semantic features in the fusion. By adjusting the dynamic scaling factor, the contribution of different semantic features to the fusion is controlled.

[0112] Specifically, a dynamic scaling factor adjusts the weights of different attention heads in real time based on the characteristics of the input data. This means the model can more accurately identify and focus on the most relevant and critical pieces of information in the input sequence, thereby improving the accuracy and quality of predictions or generated results. Dynamically adjusting the weights of attention heads effectively reduces computational effort on non-critical information, particularly when processing long sequences of data. This significantly reduces computing resource consumption and accelerates model inference while ensuring the complete processing of important information. The dynamic scaling factor enables the model to adaptively adjust its information processing strategy based on different tasks and scenarios. For example, in some application scenarios, the model may need to focus more on local information, while in others, it may need to prioritize global information. This dynamic adjustment mechanism allows the model to flexibly adapt to these changes, thereby improving its generalization and performance. Certain types of attention heads may be more important than others for certain business needs. By setting an importance mapping relationship, the model can prioritize more important information based on current business needs, enabling personalized feature extraction and processing to meet diverse requirements. Based on the importance of fusion features and the preset importance of the target, the model can intelligently identify non-important fusion features and take appropriate methods (such as trimming non-important fusion features) to reduce redundant calculations, avoid resource waste, and ensure maximum computing efficiency; dynamic adjustment and importance setting increase the transparency of the model decision-making process, helping to understand and explain why the model chooses certain information for processing, which is extremely important for model auditing, debugging and optimization, especially in application scenarios that require high interpretability.

[0113] In a specific embodiment of the present application, the method further includes: during the pruning of non-important fusion features and / or the inference process of the pre-trained model, adjusting the hyperparameters of the pre-trained model in real time according to current business needs.

[0114] Adjust the Batch Normalization (BN) hyperparameters based on the characteristics of the input samples to ensure the optimal learning rate at each iteration. This allows the entire model to automatically adjust and optimize the feature fusion weights and clipping strategies at each level during inference. During inference, checkpoint files are regularly saved for subsequent recovery. Monitor system log output to promptly identify and resolve problems.

[0115] Specifically, hyperparameters directly affect the model's reasoning process and final performance. Through real-time adjustments, the model can better adapt to the characteristics of specific tasks, such as data distribution, model complexity, etc., thereby improving prediction accuracy and stability while maintaining efficiency. Dynamic adjustment of hyperparameters can reasonably allocate computing resources according to the importance of business needs. In resource-constrained environments, this strategy is particularly critical, ensuring that the model runs in the most efficient manner and avoiding resource waste or performance bottlenecks caused by fixed configurations; different business scenarios may have different performance requirements for the model. For example, some scenarios may place more emphasis on prediction speed, while others may focus more on prediction accuracy. Real-time adjustment of hyperparameters allows the model to flexibly respond to these changes and adaptively optimize its behavior to meet the needs of specific scenarios.

[0116] In a specific embodiment of the present application, in the process of extracting semantic features of different levels from natural language text, the method includes: determining the difference in length between different text sequences of the natural language text; batch processing the text sequences with the target text sequence, where the target text sequence is a text sequence whose absolute value of the length difference is less than a preset difference.

[0117] Specifically, traditional batch padding strategies expand all sequences to the length of the longest sequence in the batch, meaning that a large amount of padding bits in shorter sequences will participate in the computation without contributing any meaningful information. By selecting target text sequences for batch processing whose absolute length difference is less than a preset difference, this type of ineffective computation can be significantly reduced, as these sequences have less padding and are closer to the actual length to be processed. This reduction in computation directly leads to an increase in model inference speed. In deep learning model inference, the effective utilization of computing resources is one of the key factors determining inference efficiency. Batching strategies for sequences of similar lengths reduce unnecessary computational load, thereby accelerating the model's inference progress. Reducing invalid padding means that less storage space is used to store meaningless values, which not only reduces memory usage but also reduces the memory bandwidth pressure on the GPU or CPU, as less data needs to be transferred between memory and computing units. This is particularly important for resource-constrained devices, ensuring that the model runs more smoothly under limited hardware conditions.

[0118] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0119] This application also provides a method for accelerating the training of a pre-trained model, specifically including:

[0120] Acquire natural language text and extract semantic features at different levels from the natural language text; fuse the semantic features at different levels to obtain multiple fusion features; determine important fusion features and unimportant fusion features of the multiple fusion features based on the importance of the fusion features; trim the unimportant fusion features to obtain trimmed fusion features, and apply the trimmed fusion features and important fusion features to the training of the pre-trained model.

[0121] Dynamically adjust the floating-point precision of the pre-trained model during the training process of the pre-trained model, including: when the pre-trained model is loaded, adjusting the floating-point precision to a first floating-point precision; when the pre-trained model is predicted, adjusting the floating-point precision to a second floating-point precision, and the second floating-point precision is higher than the first floating-point precision.

[0122] Loading phase (first floating point precision, such as FP16): Using lower precision floating point numbers (such as FP16) can significantly reduce the storage space of model parameters, which is especially beneficial for model loading, because the entire model usually needs to be loaded into RAM during the loading phase. Lower precision means smaller model size, which reduces the demand for memory; in distributed systems, model parameters need to be transmitted over the network. The use of FP16 reduces the amount of data, thereby speeding up the transmission of the model, which is conducive to faster model synchronization and improved efficiency of distributed training; the reduced model size also means increased loading speed, which is crucial for quickly starting model applications or services, especially in environments with limited resources or requiring immediate response.

[0123] Prediction stage (secondary floating-point precision, such as FP32): Using higher precision (such as FP32) during the model prediction stage can avoid numerical overflow or underflow caused by insufficient precision, improving the stability and reliability of the model output. This is crucial for applications that require extremely high prediction accuracy, such as financial analysis and autonomous driving. The prediction stage often requires more accurate calculations to ensure the accuracy of model outputs. Using FP32 can better preserve the details of the model's internal calculations, avoid prediction errors caused by loss of precision, and ensure the high quality and credibility of model predictions. Although FP32 increases storage and computing requirements, this increased resource consumption during the prediction stage is traded for better model performance. Many modern GPUs and accelerators have optimized support for FP32 operations, providing higher throughput and shorter latency, thereby improving overall prediction speed and efficiency.

[0124] In a specific embodiment of the present application, the method further includes: recording various indicators of the pre-training model during the training process.

[0125] Specifically, by recording key metrics during training, the model's training status can be monitored in real time, including changes in the learning rate, whether the loss function has converged, and the accuracy improvement trend. This facilitates timely adjustments to training parameters or strategies to avoid training stuck in local optima or overfitting. Metric recording during training can help developers diagnose potential problems encountered during model training, such as vanishing or exploding gradients and performance bottlenecks. By analyzing these records, model failure points can be located and appropriate debugging strategies implemented to ensure smooth training. Recorded metrics data can serve as a basis for adjusting hyperparameters, such as the learning rate, regularization strength, and batch size. By comparing and analyzing metrics under different settings, the optimal hyperparameter combination best suited for the task and dataset can be found, thereby optimizing model performance. Metric recording provides a quantitative method for evaluating model performance over the training cycle. For example, by observing the changes in accuracy on the validation set, one can determine whether the model is beginning to generalize well; by observing the changing trends in the loss function, one can understand the model's fit. These are important references for evaluating the final performance of the model.

[0126] like Figure 3 As shown in Figure 2, the training acceleration process of the pre-trained model includes:

[0127] Initialize environment settings to ensure sufficient computing resources are available;

[0128] Prepare training and validation sets according to the preset format;

[0129] Write or call the corresponding code to implement the model definition;

[0130] Model end-to-end training;

[0131] Model testing.

[0132] Extracting semantic features at different levels from the input text ensures that the model can capture information from multiple aspects, including vocabulary, grammar, context, and long-term dependencies, which helps improve the model's expressiveness and generalization capabilities. The dynamic pruning strategy dynamically determines the importance of the fused feature vector based on a real-time evaluation mechanism, pruning away computational branches that have little impact on the final result. To accelerate the model training process, end-to-end mixed-precision training is adopted to achieve the optimal balance between speed and accuracy. This solves the problem that existing solutions fail to achieve the desired results in accelerating the inference speed of pre-trained models. This approach can be applied to the optimization of specific deep learning models in fields such as natural language processing, computer vision, and speech recognition, broadening the scope of application of high-performance AI services, enabling them to serve a more diverse range of demand groups and promoting the popularization and development of artificial intelligence technology.

[0133] In addition to the aforementioned approach based on combining hierarchical feature fusion with dynamic cropping, other potential optimization paths can be explored, such as introducing lightweight convolution operations instead of traditional matrix multiplications, or adopting more advanced attention mechanism designs (such as linear attention) instead of the standard self-attention mechanism. We can also explore combining dynamic quantization techniques for secondary optimization, adding dynamic adjustment factors to static quantization to better adapt to the unpredictable distribution of real-world data. All three of these improvements can help further reduce the computational burden and accelerate the inference process.

[0134] For example, using V100 / A100 GPUs or other higher-performance accelerator cards as the computing platform, and employing PyTorch (an open-source deep learning framework) or TensorFlow (an open-source framework suitable for machine learning and deep learning across a wide range of tasks) as the underlying support tools to build the model structure. Experiments have proven that this solution significantly improves inference speed while barely impacting model performance, while also reducing the pressure on computing resources. Future work will continue to explore how to better adapt it to different types of tasks.

[0135] The final fused features are used to train the pre-trained model based on end-to-end mixed-precision training. The floating-point precision of the pre-trained model is dynamically adjusted during training. End-to-end mixed-precision training is a technique for optimizing inference speed and memory usage. Part of the model is calculated using lower-precision floating-point numbers (such as FP16, or half-precision), while key parts use higher precision (such as FP32, or single-precision). This method speeds up the training process while ensuring the stability and accuracy of model training.

[0136] Floating-point numbers are a numerical format used in computer science to represent real numbers. They can represent both very large and very small numbers, including decimals. A floating-point number consists of an integer part, a decimal part, and an exponent. By adjusting the exponent, values ​​of varying sizes can be represented. In deep learning, model parameters and calculation results are often represented using floating-point numbers to support high-precision mathematical operations.

[0137] By introducing hierarchical feature fusion and dynamic pruning strategies, significant inference acceleration is achieved while ensuring the quality of model output, which is particularly suitable for processing large-scale complex models. Compared with traditional hardware upgrade solutions, the technical method of the present invention is more economical and easy to deploy, and is particularly suitable for resource-constrained edge computing devices. The feedforward layer structure is reconstructed to improve the operational efficiency of each step. Overall, this helps to save more time in the entire inference process. The method of improving the self-attention mechanism with the help of local sensitive hashing technology greatly reduces the computational load, especially showing stronger superiority when facing large-scale sequence inputs. The method of dynamically adjusting batch normalization parameters avoids the rigid rule restrictions of a one-size-fits-all approach, making the entire training process more intelligent and controllable. The use of mixed precision training effectively alleviates the probability of overfitting, while also promoting the long-term stable development of the model.

[0138] The embodiment of the present application also provides a pre-trained model reasoning acceleration device, such as Figure 4 As shown, the device includes:

[0139] The first processing unit 41 is used to obtain natural language text and extract semantic features of different levels from the natural language text; the second processing unit 42 is used to fuse semantic features of different levels to obtain multiple fusion features; the third processing unit 43 is used to determine important fusion features and unimportant fusion features of multiple fusion features based on the importance of the fusion features, and the importance represents the degree of influence on the output results of the pre-training model; the fourth processing unit 44 is used to trim unimportant fusion features to obtain trimmed fusion features, and apply the trimmed fusion features and important fusion features to the reasoning of the pre-training model.

[0140] In the above-mentioned device, semantic features at different levels are extracted from the input text, ensuring that the model can capture multiple aspects of information such as vocabulary, grammar, context and long-term dependencies, which helps to improve the expressiveness and generalization capabilities of the model. Based on the importance of the fusion features, the important fusion features and unimportant fusion features of multiple fusion features are determined, and the unimportant fusion features are pruned to obtain pruned fusion features, so as to prune out those unimportant fusion features that have little impact on the final result. Therefore, the problem that the existing solutions are difficult to achieve the expected effect on the acceleration of the inference speed of the pre-trained model is solved.

[0141] In a specific embodiment of the present application, the fourth processing unit includes: a cropping module for performing one of the following steps: cropping all non-important fusion features; cropping part of the non-important fusion features in all non-important fusion features; cropping at least part of the sub-features of the non-important fusion features in all non-important fusion features; dividing the non-important fusion features into a first part and a second part, and cropping at least part of the sub-features of the non-important fusion features of the non-important fusion features of the first part, and cropping the non-important fusion features of the second part.

[0142] In a specific embodiment of the present application, the cropping module includes: a first processing submodule for sorting non-important fusion features in descending order according to their importance, to obtain a non-important fusion feature sequence; a second processing submodule for cropping non-important fusion features in the non-important fusion feature sequence whose importance is less than a preset importance.

[0143] In a specific embodiment of the present application, the first processing unit includes: a first processing module for introducing a self-adjusting activation function and a residual connection for enhancing information transfer into a pre-trained model to extract semantic features at different levels from natural language text and represent the semantic features using key-value pairs, wherein the self-adjusting activation function is used to provide a nonlinear transformation for the pre-trained model;

[0144] The first processing module includes: a second processing module for searching the most relevant key-value pairs based on semantic features using local sensitive hashing to determine multiple target key-value pairs; a third processing module for determining the fusion weights of semantic features of each level corresponding to the target key-value pairs according to the task complexity when the type of the target key-value pairs is linear; and a fourth processing module for performing weighted summation of the target key-value pairs using a linear fusion function based on the fusion weights to obtain multiple fusion features.

[0145] In a specific embodiment of the present application, the first processing module includes: an acquisition submodule for determining target jump layers of multiple levels based on multiple different preset inter-layer residuals, wherein there is a preset inter-layer residual between any two adjacent levels; and a third processing submodule for performing jump layer transfer on each level based on the target jump layer.

[0146] In a specific embodiment of the present application, the second processing unit includes: a fifth processing module for adjusting the weight of each semantic feature through the gradient of the back propagation model when the type of the target key-value pair is nonlinear, so as to obtain the adjusted weight of each semantic feature; a sixth processing module for adopting a voting mechanism to determine the feature fusion relationship, and the feature fusion relationship represents other semantic features that need to be fused with each semantic feature; a seventh processing module for performing weighted summation processing on semantic features at different levels based on the adjusted weights and feature fusion relationships to obtain multiple fusion features.

[0147] In a specific embodiment of the present application, the second processing module includes a fourth processing submodule for, in the process of determining multiple target key-value pairs by using local sensitive hashing to approximately search for the most relevant key-value pairs based on semantic features, , determine the hash value of the vector o of the current dimension;

[0148] Where h(o) is the hash value of the vector o of the current dimension, a is a standard Gaussian distributed random vector, b is an offset uniformly distributed in [0, w], and w is the shard width parameter.

[0149] In a specific embodiment of the present application, the fourth processing module includes:

[0150] The fifth processing submodule is used to , determine the fusion features;

[0151] in, To fusion features, is the weight of the i-th feature, is the value of the i-th feature, and n is the total number of features.

[0152] In a specific embodiment of the present application, the apparatus further includes: an acquisition unit configured to acquire the importance of the fused features before determining the important fused features and the unimportant fused features of the plurality of fused features based on the importance of the fused features, wherein the acquisition unit includes: an eighth processing module configured to acquire the initial importance of the fused features; a ninth processing module configured to assign a dynamic scaling factor to the key attention head based on a preset importance score to adjust the initial importance of the fused features, thereby obtaining the adjusted importance of the fused features;

[0153] The third processing unit includes: a tenth processing module for determining a preset importance based on current business needs, where the current business needs are related to the output results of the pre-trained model; an eleventh processing module for determining that a fusion feature is an important fusion feature when the adjusted importance of the fusion feature is greater than or equal to the preset importance; and a twelfth processing module for determining that a fusion feature is a non-important fusion feature when the adjusted importance of the fusion feature is less than the preset importance.

[0154] In a specific embodiment of the present application, the device also includes: a fifth processing unit for adjusting the hyperparameters of the pre-trained model in real time according to current business needs during the clipping of non-important fusion features and / or the inference process of the pre-trained model.

[0155] In a specific embodiment of the present application, the first processing unit includes: a thirteenth processing module for determining the length difference between different text sequences of natural language text in the process of extracting semantic features of different levels from natural language text; a fifteenth processing module for batch processing with a target text sequence, where the target text sequence is a text sequence whose absolute value of the length difference is less than a preset difference.

[0156] For the description of the features in the embodiment corresponding to the inference acceleration device of the pre-trained model, please refer to the relevant description of the embodiment corresponding to the inference acceleration method of the pre-trained model, and no further details will be given here.

[0157] The present application also provides an inference acceleration system for a model, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include an inference acceleration method for executing any pre-trained model.

[0158] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps of any of the above-mentioned pre-trained model reasoning acceleration method embodiments when running.

[0159] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0160] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned embodiments of the method for accelerating inference of a pre-trained model.

[0161] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned pre-trained model reasoning acceleration method embodiments.

[0162] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0163] The above is a detailed introduction to the reasoning acceleration method of the pre-trained model, the reasoning acceleration device of the pre-trained model, the computer-readable storage medium, the computer program product and the reasoning acceleration system of the pre-trained model provided by this application. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method of this application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

Claims

1. A method for accelerating inference of a pre-trained model, characterized in that: include: Acquiring natural language text, and extracting semantic features at different levels from the natural language text; Performing feature fusion on the semantic features at different levels to obtain multiple fused features; Determining important fusion features and non-important fusion features of the multiple fusion features based on the importance of the fusion features, wherein the importance represents the degree of influence on the output result of the pre-training model; Clipping the non-important fusion features to obtain clipped fusion features, and applying the clipped fusion features and the important fusion features to reasoning of the pre-trained model; Performing feature fusion on semantic features at different levels to obtain multiple fused features, including: when the type of the target key-value pair is nonlinear, adjusting the weight of each semantic feature by using the gradient of the back-propagation model to obtain the adjusted weight of each semantic feature; using a voting mechanism to determine a feature fusion relationship, wherein the feature fusion relationship represents other semantic features that need to be fused with each semantic feature; and performing weighted summation processing on the semantic features at different levels based on the adjusted weights and the feature fusion relationship to obtain the multiple fused features; Different layers or modules of the pre-trained model capture semantic information of different granularities: low-level features: part of speech, basic phrase structure, local word meaning; mid-level features: syntactic relations, sentence-level semantics, context-related word meaning; high-level features: paragraph / chapter-level semantics, theme, sentiment, and intention.

2. The method for accelerating the reasoning of a pre-trained model according to claim 1, wherein: The non-important fusion features are trimmed, including one of the following: Crop out all the non-important fusion features; Cutting out some of the non-important fused features from all the non-important fused features; Crop out at least part of the sub-features of the non-important fused features in all the non-important fused features; The non-important fused features are divided into a first part and a second part, and at least part of the sub-features of the non-important fused features of the first part are clipped, and the non-important fused features of the second part are clipped.

3. The method for accelerating the reasoning of a pre-trained model according to claim 2, wherein: Clipping some of the non-important fused features from all the non-important fused features to obtain the clipped fused features, including: Sorting the non-important fusion features in descending order according to their importance to obtain a non-important fusion feature sequence; The non-important fusion features whose importance is less than a preset importance are trimmed from the non-important fusion feature sequence.

4. The method for accelerating the reasoning of a pre-trained model according to claim 1, wherein: Extracting semantic features at different levels from the natural language text, including: introducing a self-adjusting activation function and a residual connection for enhancing information transfer into the pre-trained model to extract the semantic features at different levels from the natural language text, and representing the semantic features using key-value pairs, wherein the self-adjusting activation function is used to provide a nonlinear transformation for the pre-trained model; Feature fusion is performed on semantic features at different levels to obtain multiple fused features, including: based on the semantic features, using local sensitive hashing to approximately search for the most relevant key-value pairs to determine multiple target key-value pairs; when the type of the target key-value pairs is linear, according to the task complexity, the fusion weights of the semantic features of each level corresponding to the target key-value pairs are determined; based on the fusion weights, a linear fusion function is used to perform weighted summation on the target key-value pairs to obtain the multiple fused features.

5. The method for accelerating the reasoning of a pre-trained model according to claim 4, wherein: Introducing the residual connection for enhancing information transfer into the pre-trained model includes: Determining target skip layers of multiple levels based on multiple different preset inter-layer residuals, wherein there is the preset inter-layer residual between any two adjacent levels; A layer skipping transfer is performed on each of the layers based on the target skipping layer.

6. The method for accelerating the reasoning of a pre-trained model according to claim 4, wherein: In the process of approximately searching for the most relevant key-value pairs based on the semantic features using locality-sensitive hashing to determine multiple target key-value pairs, the method further includes: according to , determine the hash value of the vector o of the current dimension; Where h(o) is the hash value of the vector o of the current dimension, a is a standard Gaussian distributed random vector, is the offset uniformly distributed in [0, w], and w is the shard width parameter.

7. The method for accelerating the reasoning of a pre-trained model according to claim 4, wherein: Based on the fusion weight, a linear fusion function is used to perform weighted summation on the target key-value pairs to obtain the fusion feature, including: according to , determining the fusion feature; in, is the fusion feature, is the weight of the i-th feature, is the value of the i-th feature, and n is the total number of features.

8. The method for accelerating the reasoning of a pre-trained model according to claim 1, wherein: Before determining important fusion features and non-important fusion features of the multiple fusion features based on the importance of the fusion features, the method further includes: obtaining the importance of the fusion features, wherein obtaining the importance of the fusion features includes: obtaining an initial importance of the fusion features; allocating a dynamic scaling factor to a key attention head based on a preset importance score to adjust the initial importance of the fusion features, thereby obtaining an adjusted importance of the fusion features; Determining important fusion features and unimportant fusion features of the multiple fusion features based on the importance of the fusion features includes: determining a preset importance based on current business needs, where the current business needs are related to the output results of the pre-trained model; determining that the fusion feature is the important fusion feature when the adjusted importance of the fusion feature is greater than or equal to the preset importance; and determining that the fusion feature is the unimportant fusion feature when the adjusted importance of the fusion feature is less than the preset importance.

9. The method for accelerating inference of a pre-trained model according to claim 1, wherein: The method further comprises: During the pruning of the non-important fusion features and / or the inference process of the pre-trained model, the hyperparameters of the pre-trained model are adjusted in real time according to current business needs.

10. The method for accelerating the reasoning of a pre-trained model according to claim 1, wherein: In the process of extracting semantic features at different levels from the natural language text, the method includes: determining a difference in length between different text sequences of the natural language text; Batch processing is performed on the target text sequence, where the target text sequence is the text sequence whose absolute value of the length difference is less than a preset difference.

11. A pre-trained model reasoning acceleration device, characterized in that: include: A first processing unit is configured to obtain a natural language text and extract semantic features at different levels from the natural language text; A second processing unit is used to perform feature fusion on the semantic features at different levels to obtain multiple fused features; a third processing unit, configured to determine important fusion features and non-important fusion features of the plurality of fusion features based on the importance of the fusion features, wherein the importance represents a degree of influence on an output result of the pre-trained model; a fourth processing unit, configured to clip the non-important fusion features to obtain clipped fusion features, and apply the clipped fusion features and the important fusion features to reasoning of the pre-trained model; The second processing unit includes: a fifth processing module for adjusting the weight of each semantic feature by back-propagating the gradient of the model when the type of the target key-value pair is nonlinear, thereby obtaining the adjusted weight of each semantic feature; a sixth processing module for determining a feature fusion relationship using a voting mechanism, wherein the feature fusion relationship represents other semantic features that need to be fused with each semantic feature; and a seventh processing module for performing weighted summation processing on semantic features at different levels based on the adjusted weights and the feature fusion relationship, thereby obtaining a plurality of fused features; Different layers or modules of the pre-trained model capture semantic information of different granularities: low-level features: part of speech, basic phrase structure, local word meaning; mid-level features: syntactic relations, sentence-level semantics, context-related word meaning; high-level features: paragraph / chapter-level semantics, theme, sentiment, and intention.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method for accelerating the inference of the pre-trained model as described in any one of claims 1 to 10 are implemented.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for accelerating the inference of the pre-trained model described in any one of claims 1 to 10 are implemented.

14. A pre-trained model reasoning acceleration system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include an inference acceleration method for executing the pre-trained model described in any one of claims 1 to 10.

Images