Model training method, medium, device and program product of transverse mixed attention mechanism

Abstract

The application provides a model training method, medium, equipment and program product of a transverse mixed attention mechanism. The method comprises: acquiring a data set containing multiple sample sequences, each sample sequence in the data set being composed of multiple Tokens arranged in sequence after tokenization processing; constructing a to-be-trained model based on a pre-trained full attention model and adding new parameters for linear attention calculation; in the same transverse mixed attention layer, performing full attention calculation on a Token set located within a preset full attention calculation range, performing linear attention calculation on all Tokens, and fusing the results of the two to obtain transverse mixed attention output for forward reasoning and loss calculation; based on the output and a prediction result, only updating the new parameters to optimize the to-be-trained model until the to-be-trained model converges. The application reduces the computational complexity and memory occupation of long text sequence processing, and improves the reasoning speed and resource utilization rate.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to a model training method, medium, device and program product based on a lateral hybrid attention mechanism. Background Technology

[0002] Attention mechanisms, as a core technology in the field of deep learning, have been widely applied to tasks involving sequence data processing, such as natural language processing and computer vision.

[0003] Existing full attention mechanisms assign attention weights by calculating the similarity between each element in the input sequence and all other elements, thereby increasing the focus on important information. However, when the input sequence length is N, the computational complexity of the full attention mechanism is O(N^2). 2 With a memory footprint of O(N), it leads to significant computational resource consumption and inference latency when processing long sequences of text data (such as long texts, script content, and multi-day continuous dialogue records), especially when generating the first output token (the smallest computational unit after word segmentation), which is difficult to meet the low latency requirements.

[0004] To reduce computational complexity, linear attention was proposed. It linearizes the calculation of attention weights, reducing computational complexity to a level linear with the sequence length. However, linear attention is insufficient in capturing complex dependencies between elements in a sequence. It is prone to losing key information or causing information confusion during computation, thus affecting the model's accuracy in certain tasks.

[0005] Therefore, existing full attention mechanisms and linear attention mechanisms suffer from inefficiency or insufficient accuracy in long sequence tasks, respectively. There is an urgent need for an improved attention mechanism that can significantly reduce computational complexity while maintaining model accuracy. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this application provides a model training method, medium, device, and program product based on a lateral hybrid attention mechanism, which at least solves the technical problem that existing technologies cannot effectively balance the model's prediction accuracy with computational overhead when processing long sequences of text input.

[0007] To achieve the above objectives and other advantages, some embodiments of this application provide the following aspects:

[0008] In a first aspect, some embodiments of this application provide a model training method for a lateral hybrid attention mechanism, including:

[0009] Obtain a dataset containing multiple sample sequences, wherein each sample sequence in the dataset consists of multiple tokens obtained through word segmentation arranged in sequence;

[0010] A model to be trained is constructed based on the pre-trained full attention model. The model to be trained retains the parameters of the full attention model and adds new parameters for linear attention calculation.

[0011] In the same horizontal hybrid attention layer of the model to be trained, full attention calculation is performed on the set of tokens in the input sample sequence that are within the preset full attention calculation range to obtain the full attention calculation result. Linear attention calculation is performed on all tokens in the sample sequence to obtain the linear attention calculation result. The full attention calculation result and the linear attention calculation result are fused to obtain a horizontal hybrid attention output for forward inference and loss calculation based on the forward inference result.

[0012] Based on the lateral mixed attention output and the prediction results of the model to be trained, the model to be trained is trained by updating only the newly added parameters to optimize the model to be trained until the model to be trained converges, so as to obtain the trained model to be trained.

[0013] Secondly, some embodiments of this application also provide an electronic device, the electronic device comprising:

[0014] One or more processors; and a memory storing computer program instructions that, when executed, cause the processors to perform a model training method using a lateral hybrid attention mechanism as described above.

[0015] Thirdly, some embodiments of this application also provide a computer-readable storage medium having a computer program and / or instructions stored thereon, which, when executed by a processor, implement a model training method for the lateral hybrid attention mechanism as described above.

[0016] Fourthly, some embodiments of this application also provide a computer program product, including a computer program and / or instructions, which, when executed by a processor, implement a model training method for a lateral hybrid attention mechanism as described above.

[0017] Compared with related technologies, the solution provided in this application, by performing full attention calculation on some tokens of the input sample sequence and linear attention calculation on all tokens in the same horizontal hybrid attention layer, and fusing the results of the two calculations, can effectively reduce the overall computational complexity in the long text sequence processing process, while ensuring sufficient modeling of key positional information, and reduce memory usage and computational latency. Compared with the traditional method of directly performing full attention calculation on all tokens, this application can significantly improve the inference speed and resource utilization in long text sequence tasks, and support larger-scale data processing while ensuring model accuracy. In addition, by training only by updating the newly added parameters related to linear attention, the parameter structure and representation capabilities of the original full attention model are retained, achieving optimization of hybrid attention computation performance at a lower training cost, thus balancing training convergence efficiency and model deployment flexibility, and is suitable for efficient inference scenarios under various hardware environments and different resource constraints. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other implementation methods can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic flowchart of a model training method using a lateral hybrid attention mechanism provided in an embodiment of this application;

[0020] Figure 2 This is a flowchart illustrating the process of dynamically adjusting the range of the Token set during the inference phase of the model training method of the horizontal hybrid attention mechanism provided in this application embodiment;

[0021] Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] First Embodiment

[0024] The first embodiment of this application relates to a model training method using a lateral hybrid attention mechanism, referring to... Figure 1 As shown, the method may include the following steps:

[0025] Step S1: Obtain a dataset containing multiple sample sequences. Each sample sequence in the dataset consists of multiple tokens obtained through word segmentation arranged in sequence.

[0026] Specifically, in step S1, the sample sequence refers to the text sequence or other type of symbol sequence used to train the natural language processing model. Its data sources can include open-domain corpora (such as encyclopedic text, news, and dialogue data), domain-specific labeled data (such as medical literature and financial contracts), user interaction logs, etc. Each sample sequence undergoes word segmentation, which can be based on dictionary-based segmentation, statistical model-based segmentation, or sub-word unit encoding. This maps the original text into multiple tokens, which are then arranged in the order they appear in the original sequence, thus preserving the word order information and contextual dependencies of the input data.

[0027] Tokens can be mapped to unique index numbers by looking up predefined words, and further mapped to fixed-dimensional vector embeddings in subsequent steps, which are then used as input to the attention mechanism for feature modeling. This step standardizes unstructured or semi-structured raw inputs into a unified format that the model can process, ensuring that the input's positional information, symbol boundary information, and contextual semantics are fully preserved in subsequent full attention and linear attention calculations.

[0028] Step S2: Construct a model to be trained based on the pre-trained full attention model. The model to be trained retains the parameters of the full attention model and adds new parameters for linear attention calculation.

[0029] Specifically, in step S2, the pre-trained full attention model on a large-scale corpus is loaded into the training environment. This full attention model uses a multi-head self-attention mechanism to model the global dependencies of long sequences, and its parameters include multiple sets of query, key, and value mapping matrices, positional encoding parameters, and feedforward network layer weights and biases. To preserve the global context representation capabilities learned by this full attention model during the pre-training phase, the parameters of the full attention model are frozen in this step and no gradient updates are performed in subsequent training.

[0030] Furthermore, the new parameters include: linear projection weight matrix, feature compression bias term, and kernel function mapping parameters; among them, the new parameters are initialized using a random initialization method that is orthogonal to the corresponding weight matrix in the full attention model, and the original parameters of the full attention model are kept frozen during training, with the new parameters only updated based on the gradient of backpropagation, to ensure that the linear attention calculation and the full attention calculation are independent of each other.

[0031] Based on this, this embodiment introduces a linear attention calculation branch into the existing structure of the full attention model. The new parameters of this linear attention calculation branch include: a linear projection weight matrix for mapping the input token representation to a low-dimensional space, a feature compression bias term for the dimensionality reduction process, and a kernel function mapping parameter for calculating attention weights based on kernel function feature mapping.

[0032] When full attention and linear attention computations are executed in parallel within the same layer, if their initial parameter values ​​are highly correlated in the feature space, the representational capabilities of the two branches may overlap or even interfere with each other in the early stages of training. This reduces the diversity and complementarity of the hybrid attention mechanism. Therefore, by ensuring that newly added parameters (such as the linear projection weight matrix) are orthogonal to the weight matrix corresponding to full attention during initialization, their initial feature mapping directions can be guaranteed to be independent in numerical computation, thereby enabling the linear attention branch to learn a different representation mode than full attention. Specifically, firstly, the weight matrix corresponding to the linear attention computation function (e.g., the weight matrix used for query projection) is extracted from the full attention model and used as the reference matrix; a random matrix of the same shape is generated, and the initial values ​​of the random matrix can be generated through uniform or Gaussian distribution; subsequently, orthogonalization is performed on the random matrix and the reference matrix, for example, using the Gram-Schmidt orthogonalization method, or by combining the two into a large matrix and then performing QR decomposition, thereby obtaining an initialization result that is computationally independent of the initial feature mapping direction of the reference matrix.

[0033] In terms of model structure, the linear attention computation branch and the original full attention computation branch share the same batch of token representation vectors at the input end and execute the attention computation process in parallel. The full attention computation branch is used to achieve high-precision global information capture within a preset range, preserving the precise dependencies of key positions; the linear attention computation branch, through low-rank feature mapping and inner product operations, reduces the computational complexity of attention weights from O(N) to O(N). 2 The computational cost is reduced to O(N), thereby achieving near-global modeling across the entire sequence and significantly reducing computational overhead.

[0034] Specifically, for step S3, the horizontal hybrid attention layer refers to an attention computation unit that integrates full attention computation path and linear attention computation path in parallel within the same layer of the multi-layer attention computation structure of the model to be trained. This unit is used to simultaneously acquire accurate local dependency modeling capability and efficient global information capture capability in one forward propagation.

[0035] For example, a sample sequence X consists of multiple tokens arranged in their original order, with a length of N. Based on a predefined full attention calculation range K (K≤N, K is a positive integer), the first K tokens in the sample sequence are determined as the full attention calculation set, and the full attention function F is used to calculate this set. The full attention function F calculates the similarity score between any token in the set and other tokens in the set, and obtains the attention weight through Softmax normalization. This weighted sum is then applied to the value vector to generate the corresponding full attention calculation result. This approach can preserve accurate global dependency modeling effects within local regions and is particularly suitable for capturing high-precision feature interactions between key locations.

[0036] Meanwhile, the linear attention path performs linearized attention computation on all N input tokens based on the linear attention function L. This linear attention function maps the Query and Key vectors to a low-dimensional space through kernel function feature mapping, reducing the computational complexity of attention weights from O(N) to O(N). 2 The computational cost is reduced to O(N), significantly reducing computational overhead and memory usage, and it can capture long-range dependencies spanning the entire sequence range.

[0037] After completing the two-way attention calculations, the lateral hybrid attention layer uses a fusion function H = F(X[:K]) + L(X) to weight and fuse the full attention calculation results with the linear attention calculation results. The fusion weights can be learned through training or preset by the system to achieve a dynamic balance between accuracy and performance. Finally, the fused lateral hybrid attention output is used as the output of this layer and input into the subsequent forward inference process. The loss function is calculated based on the inference results, thereby driving the entire model to significantly improve the processing efficiency and scalability of long sequences while retaining high-precision local modeling capabilities.

[0038] Step S4: Based on the lateral mixed attention output and the prediction results of the model to be trained, train the model to be trained by updating only the newly added parameters to optimize the model to be trained until the model to be trained converges, so as to obtain the trained model to be trained.

[0039] Specifically, for step S4, the loss function (such as cross-entropy, mean squared error, or task-specific loss) is calculated using the laterally mixed attention output and the prediction results of the model to be trained. Backpropagation is then performed on the model to be trained with this loss as the target. During the backpropagation phase, this embodiment employs a parameter freezing strategy, which keeps the parameter weights of the pre-trained full attention part unchanged, and only performs gradient calculations and updates for newly trainable parameters introduced in the linear attention calculation path. This strategy effectively reduces the overall computational cost of training, avoids the forgetting effect on the high-precision feature modeling capabilities already learned by the original full attention part during incremental training, and improves the stability and convergence speed of the training process.

[0040] When the training process meets the preset convergence conditions (e.g., the loss value on the validation set tends to stabilize, the prediction accuracy reaches a threshold, or the number of training iterations reaches the upper limit), the frozen full attention parameters and the newly added parameters of the updated linear attention are solidified together, and the final trained model is output. This model possesses both the high-precision local dependency modeling capability of the original full attention model during the inference phase and significantly improves the efficiency and scalability of long sequence processing, thus achieving a balanced optimization of performance and efficiency.

[0041] Compared with related technologies, the solution provided in this application, by performing full attention calculation on some tokens of the input sample sequence and linear attention calculation on all tokens in the same horizontal hybrid attention layer, and fusing the results of the two calculations, can effectively reduce the overall computational complexity in the long text sequence processing process, while ensuring sufficient modeling of key positional information, and reduce memory usage and computational latency. Compared with the traditional method of directly performing full attention calculation on all tokens, this application can significantly improve the inference speed and resource utilization in long text sequence tasks, and support larger-scale data processing while ensuring model accuracy. In addition, by training only by updating the newly added parameters related to linear attention, the parameter structure and representation capabilities of the original full attention model are retained, achieving optimization of hybrid attention computation performance at a lower training cost, thus balancing training convergence efficiency and model deployment flexibility, and is suitable for efficient inference scenarios under various hardware environments and different resource constraints.

[0042] Second Embodiment

[0043] The second embodiment of this application relates to a model training method using a lateral hybrid attention mechanism. The second embodiment is an improvement upon the first embodiment, specifically in that a multi-objective loss calculation mechanism is introduced during the training process. That is, step S4 can further include the following steps:

[0044] Step S401: Input the sample sequences in the dataset into the model to be trained and the full attention model respectively to obtain the horizontal mixed attention output of the model to be trained in the horizontal mixed attention layer, as well as the prediction results of the model to be trained in the output layer and the output results of the full attention model.

[0045] In this step, the starting model is assumed to be Pi0, which is a pre-trained full attention model with its parameters frozen; the fitted model is Pi1, which adds parameters related to linear attention calculation to model Pi0. The sample sequence X from the dataset is input into models Pi1 and Pi0 respectively. The sample sequence, after preprocessing, includes a token sequence, corresponding labels, and necessary mask information. Model Pi1 outputs the lateral hybrid attention output (containing both full attention and linear attention calculation results) of the lateral hybrid attention layer during forward inference, as well as the prediction results of the output layer; model Pi0 outputs the baseline prediction results of the full attention path and provides TopK Logits data for distillation to reduce noise and focus on the main prediction categories.

[0046] Step S402: Calculate the first loss value based on the mean square error between the full attention calculation result and the linear attention calculation result in the lateral hybrid attention output.

[0047] In this step, the difference between the full attention path and the linear attention path at the feature level is calculated. The full attention calculation result in the lateral hybrid attention output is denoted as F(X), and the linear attention calculation result is denoted as L(X). The difference is calculated based on the mean squared error (MSE) formula to form the attention distillation loss function:

[0048] Loss attention_distill =MSE(F(X)-L(X))

[0049] This loss function is used to constrain the linear attention features in Pi1 to approximate the full attention feature distribution in Pi0, thereby improving the accuracy of the fused output.

[0050] Step S403: Determine the second loss value based on the distribution difference between the prediction results of the model to be trained and the output results of the full attention model.

[0051] Further, in step S403, from the output of the full attention model, for the prediction results of each token position in the sample sequence, the top K candidate tokens with the highest predicted scores and their corresponding predicted scores are selected from the vocabulary to obtain the distilled TopK logits data; based on the Kullback-Leibler divergence between the prediction results of the model to be trained and the distilled TopK logits data, the second loss value is determined.

[0052] Specifically, when the full attention model infers on an input sample sequence, the vocabulary size for the prediction result of the 6th token position in the sample sequence is V = 50,000. The full attention model outputs the predicted score (logits) of this position across the entire vocabulary. In this embodiment, the top K = 5 candidate tokens are selected in descending order of scores, for example, {"apple": 2.1, "pear": 1.9, "orange": 1.5, "banana": 1.4, "grape": 1.3}, and their corresponding predicted scores are combined with the candidate tokens to form the distilled TopK logits data for that position. For the model to be trained, the corresponding predicted probability distribution is output at the same token position. In this embodiment, the predicted distribution and the distilled TopK logits data of the full attention model are compared using Kullback-Leibler (KL) divergence calculation to measure the difference in distribution between the two on high-confidence candidates. If the KL divergence value is small, it means that the prediction distribution of the model to be trained at this position has approximated the full attention model well, thus reducing the gradient update magnitude at this position; otherwise, the update magnitude should be increased to improve the prediction consistency of the model to be trained for high-importance candidate tokens.

[0053] In this step, the TopK Logits of Pi0 are used as the teacher distribution. The distribution difference between the predicted results Pi1(X) of Pi1 and the predicted results Pi0(X) of Pi0 is measured. The distillation loss of the logits layer is calculated using KL divergence, and its loss function is:

[0054] Loss logits =KL(Pi1(X),Pi0(X))

[0055] This loss function encourages Pi1 to align with the prediction distribution of Pi0 in the output space, which avoids the high cost of full distillation of the entire vocabulary while preserving the knowledge transfer effect of the main prediction results.

[0056] Step S404: Determine the third loss value based on the difference between the prediction results of the model to be trained and the target labels of the dataset.

[0057] Further, in step S404, the cross-entropy loss function is used to calculate the difference between the prediction result of the model to be trained in the language model task and the target label of the dataset, and the third loss value is determined to measure the performance of the model to be trained in the language model task.

[0058] For each token position in the input sample sequence, the predicted probability distribution generated by the model to be trained in the output layer is obtained and compared with the target label at the corresponding position; the target label is the correctly labeled token identifier in the dataset. Subsequently, the difference between the above predicted probability distribution and the target label is calculated using the cross-entropy loss function to obtain the loss value corresponding to that position. The loss values ​​of all token positions in the sequence are averaged to form the third loss value of the sample.

[0059] The third loss value is used to measure the predictive performance of the model under training on language modeling tasks. The closer the predicted result is to the target label, the smaller the cross-entropy loss value, indicating that the model can more accurately generate output sequences consistent with the training corpus; conversely, the newly added linear attention parameter is adjusted through backpropagation to improve the model's ability to fit semantics and context. For example, when the target label is "apple"...

[0060] If the model predicts a probability of 0.92 for "apple" and the sum of the probabilities of the other candidate tokens is 0.08, then the cross-entropy loss value at that position is relatively small, indicating that the prediction is relatively accurate. If the predicted probability distribution is more dispersed, the loss value will increase significantly, indicating that the model needs further optimization.

[0061] In this step, the predicted result Pi1(X) of Pi1 is compared with the target label Y of the dataset to evaluate its performance on real-world tasks (such as language modeling). The loss function is:

[0062] LossLM = L(Pi1(X),Y)

[0063] This loss function ensures that Pi1 closely approximates the teacher model distribution while retaining its ability to fit the true labels.

[0064] Step S405: The first loss value, the second loss value, and the third loss value are weighted and summed according to preset weights to obtain the total loss value. The newly added parameters are then iteratively updated based on the total loss value, and the model to be trained is trained until convergence.

[0065] In this step, the three loss values ​​are weighted and summed according to preset weights α, β, and γ to obtain the total loss function:

[0066] Loss total =α×Loss attention_distill +β×Loss logits +γ×Loss LM

[0067] Backpropagation is performed based on the loss value of this total loss function, updating only the parameters related to the newly added linear attention calculation in Pi1, while keeping the original full attention calculation parameters in Pi0 unchanged. The training process is iterated until the total loss value converges or the performance of the validation set meets the stopping condition, ultimately obtaining the target model that achieves the expected results in both accuracy and efficiency.

[0068] Through knowledge distillation and fitting training driven by multi-source loss, this embodiment can maintain the original accurate modeling capability of the full attention path while gradually making the linear attention path approach the full attention path in terms of feature distribution and prediction results. This achieves an optimal balance between computational resource consumption and model performance in long sequence modeling, and significantly improves inference efficiency and scalability. Even with only about 30B of pre-training data, it can adequately fit and catch up with the validation set loss curve in the early training stages, and even slightly surpass the validation set loss in some test scenarios. Compared to the control model without the optimization mechanism of this application, this method significantly reduces the training data size and computational resource consumption while maintaining or even improving the model's convergence and generalization performance, thus effectively improving the feasibility and economy of model deployment under limited computing power and data resources.

[0069] Third Embodiment

[0070] The third embodiment of this application relates to a model training method using a lateral hybrid attention mechanism. The third embodiment is an improvement upon the first embodiment, specifically in that... Figure 2 As shown, after the model to be trained is trained to obtain an optimized model, the following steps may also be included:

[0071] Step A1: During the inference phase, based on the computing resource status information, input data feature information, and preset performance constraints, dynamically determine the value range of the Token set that uses full attention computing in the inference input sequence. The performance constraints include the maximum response latency threshold and the tolerance for accuracy degradation.

[0072] Step A2: During the inference process, adjust the value range of the Token set in real time based on the current computing resource status and changes in predicted performance indicators;

[0073] Step A3: When the response latency exceeds the maximum response latency threshold or the accuracy degradation exceeds the accuracy degradation tolerance, switch the value range of different Token sets.

[0074] In this embodiment, during the inference phase, the current computing resource status information is first obtained, including but not limited to the available computing power of the inference server or terminal device (such as GPU / TPU core utilization, video memory usage, CPU usage), I / O bandwidth usage, and the current task queue length, etc. At the same time, feature information of the input sequence to be inferred is extracted, such as sequence length, token distribution density, token ratio at key positions, and accuracy prediction indicators provided by historical inference records or model performance monitoring modules.

[0075] Based on the above information, the system performs calculations according to preset performance constraints, including a maximum response latency threshold and a tolerance for accuracy degradation. For example, when performing inference on a real-time dialogue task, the maximum response latency threshold can be set to 200 milliseconds, and the tolerance for accuracy degradation can be set to a task performance index decrease of no more than 1.5%. Under the premise of meeting the performance constraints, the system dynamically determines the range K (K≤N, where N is the total length of the sequence) of the token set used for full attention computation in the inference input sequence.

[0076] During actual operation, the K value is adjusted in real time based on changes in computing resource status and prediction performance metrics. For example, when GPU utilization and memory usage remain at low load levels while model prediction accuracy remains stable, the system can appropriately increase the K value to allow more tokens to participate in full attention computation, thereby further improving inference accuracy. Conversely, when the response latency is detected to be close to or exceed the maximum response latency threshold, or when the accuracy degradation calculated by the performance monitoring module exceeds the accuracy degradation tolerance, the system immediately switches to a smaller K value, retaining only key tokens for full attention computation, while the remaining tokens are processed by linear attention paths, thus reducing computational complexity and ensuring response speed.

[0077] Taking a long-sequence question-answering task as an example, assuming the input sequence length is 1024 tokens, under conditions of ample computing resources and stable accuracy, the system can set K to 512, allowing the first half of the tokens to undergo full attention computation. When it detects that the memory utilization is close to the limit or the response latency exceeds the standard, K will be dynamically adjusted to 256 or even 128 to reduce the full attention computation load. This dynamic adjustment mechanism can achieve a real-time balance between inference accuracy and response speed, adapting to the changing needs of different hardware resources and task scenarios.

[0078] Further, step A2 specifically includes:

[0079] Step A201: Based on historical inference task performance monitoring data, construct a performance prediction model covering response latency, accuracy indicators, and computing resource consumption;

[0080] Step A202: Upon receiving a new inference input sequence, the performance prediction model is invoked to estimate the prediction performance under different Token set value ranges, and the initial Token set value range is set before inference begins based on the estimation results.

[0081] Step A203: During the inference process, when the deviation between the real-time performance metric and the prediction result of the performance prediction model exceeds a preset threshold, the adaptive correction of the value range of the Token set is triggered to balance inference latency and prediction accuracy.

[0082] Specifically, firstly, based on performance monitoring data of historical inference tasks under different operating environments, multi-dimensional performance data samples covering inference response latency, prediction accuracy, and computational resource consumption are collected and recorded. Subsequently, feature engineering and data cleaning are performed using these performance data samples to construct a performance prediction model covering all performance dimensions. This performance prediction model can be implemented using structures such as regression analysis, gradient boosting trees, or lightweight neural networks, and is used to output corresponding latency estimates, accuracy estimates, and resource consumption estimates given a range of token values.

[0083] Upon receiving a new inference input sequence, the inference control module invokes the performance prediction model to calculate prediction performance metrics for multiple candidate token sets, including inference latency, prediction accuracy, and computational resource utilization. The inference control module performs a multi-objective trade-off based on the prediction performance metrics and preset performance constraints, automatically setting the initial token set's value range before inference begins to achieve a balance between expected accuracy and computational latency.

[0084] During inference execution, real-time performance metrics are continuously monitored and compared with the prediction results of the performance prediction model. When the deviation between the real-time performance metrics and the prediction results exceeds a preset threshold (e.g., response latency exceeds the predicted value by 10% or the accuracy decrease exceeds the tolerance), an adaptive correction of the Token set value range is triggered. This correction process can be achieved by increasing or decreasing the size of the Token set, adjusting the Token selection strategy, etc., to continuously maintain a balance between inference response latency and prediction accuracy under conditions of dynamic computing resource fluctuations and changes in task characteristics.

[0085] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0086] Furthermore, some embodiments of this application also provide an electronic device. The electronic device can be various forms of digital computer, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device can also be various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0087] The electronic device includes: one or more processors; and a memory storing computer program instructions, which, when executed, cause the processor to perform a model training method for a lateral hybrid attention mechanism as provided in any one or more of the above embodiments. Figure 3 An exemplary structural diagram of the electronic device is disclosed. The electronic device includes one or more processors 1101, a memory 1102, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations. The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0088] The electronic device may further include an input device 1103 and an output device 1104. The processor 1101, memory 1102, input device 1103, and output device 1104 may be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0089] Input device 1103 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 1104 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display, a light-emitting diode display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0090] To provide interaction with the user, the electronic device can be a computer. The computer has: a display device (e.g., a cathode ray tube or LCD monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback); and input from the user can be received in any form (e.g., voice input or tactile input).

[0091] In this embodiment, a computer-readable medium stores a computer program / instruction, which, when executed by a processor, implements a model training method for a lateral hybrid attention mechanism provided in any one or more of the above embodiments. This computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into that device. The aforementioned computer-readable medium carries one or more computer-readable instructions.

[0092] The memory 1102 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 1102, thereby implementing the program instructions / modules corresponding to the methods provided in any one or more of the embodiments described above in this application.

[0093] The memory 1102 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 1102 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1102 may optionally include memory remotely located relative to the processor 1101, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0094] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0095] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technologies, read-only optical discs, digital versatile optical discs or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0096] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including local area networks (LANs) or wide area networks (WANs), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0097] In the above embodiments, all or part of the implementation can be achieved through software, hardware, firmware, or any combination thereof. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the above steps or functions. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. In addition, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0098] The computer program product provided in this application includes one or more computer programs / instructions. When executed by a processor, these computer programs / instructions generate, in whole or in part, the processes or functions described in this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0099] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0100] The scope of this application is defined by the appended claims rather than the foregoing description, and is therefore intended to encompass all variations falling within the meaning and scope of equivalents of the claims. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device in software or hardware. Terms such as "first," "second," etc., are used only for distinguishing descriptions and do not indicate any particular order, nor should they be construed as indicating or implying relative importance.

[0101] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily made by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims, and the above embodiments should be regarded as exemplary and non-limiting.

Claims

1. A model training method using a lateral hybrid attention mechanism, characterized in that, include: Obtain a dataset containing multiple sample sequences, which are text sequences used to train a natural language processing model. Each sample sequence in the dataset consists of multiple tokens obtained through word segmentation arranged in sequence. A model to be trained is constructed based on the pre-trained full attention model. The model to be trained retains the parameters of the full attention model and adds new parameters for linear attention calculation. In the same horizontal hybrid attention layer of the model to be trained, full attention calculation is performed on the set of tokens in the input sample sequence that are within the preset full attention calculation range to obtain the full attention calculation result. Linear attention calculation is performed on all tokens in the sample sequence to obtain the linear attention calculation result. The full attention calculation result and the linear attention calculation result are fused to obtain a horizontal hybrid attention output for forward inference and loss calculation based on the forward inference result. Based on the lateral mixed attention output and the prediction results of the model to be trained, the model to be trained is trained by updating only the newly added parameters to optimize the model to be trained until the model to be trained converges, so as to obtain the trained model to be trained.

2. The model training method of the lateral hybrid attention mechanism according to claim 1, characterized in that, The step of training the model to be trained based on the dataset, optimizing the model to be trained by only updating the newly added parameters, until the model to be trained converges to obtain the trained model to be trained includes: The sample sequences in the dataset are input into the model to be trained and the full attention model respectively to obtain the horizontal mixed attention output of the model to be trained in the horizontal mixed attention layer, the prediction result of the model to be trained in the output layer and the output result of the full attention model. Based on the mean square error between the full attention calculation result and the linear attention calculation result in the lateral hybrid attention output, the first loss value is calculated; The second loss value is determined based on the distribution difference between the prediction results of the model to be trained and the output results of the full attention model. A third loss value is determined based on the difference between the prediction results of the model to be trained and the target labels of the dataset; The first loss value, the second loss value, and the third loss value are weighted and summed according to preset weights to obtain a total loss value. The newly added parameters are then iteratively updated based on the total loss value to train the model until convergence.

3. The model training method of the lateral hybrid attention mechanism according to claim 2, characterized in that, The step of determining the second loss value based on the distribution difference between the prediction result of the model to be trained and the output result of the full attention model includes: From the output of the full attention model, for the prediction result of each token position in the sample sequence, select the top K candidate tokens with the highest prediction scores and their corresponding prediction scores from the vocabulary to obtain the distilled TopK logits data; The second loss value is determined based on the Kullback-Leibler divergence between the prediction results of the model to be trained and the distilled TopK logits data.

4. The model training method of the lateral hybrid attention mechanism according to claim 2, characterized in that, The step of determining the third loss value based on the difference between the prediction result of the model to be trained and the target label of the dataset includes: The third loss value is determined by calculating the difference between the prediction result of the model to be trained in the language model task and the target label of the dataset using the cross-entropy loss function.

5. The model training method of the lateral hybrid attention mechanism according to claim 1, characterized in that, After the model to be trained is trained to obtain an optimized model, during the inference stage, based on the computing resource status information, input data feature information and preset performance constraints, the value range of the token set using full attention computing in the inference input sequence is dynamically determined. The performance constraints include the maximum response latency threshold and the accuracy degradation tolerance. During the reasoning process, the value range of the Token set is adjusted in real time based on changes in the current computing resource status and predicted performance indicators; When the response latency exceeds the maximum response latency threshold or the accuracy degradation exceeds the accuracy degradation tolerance, the value range of the different Token sets is switched.

6. The model training method of the lateral hybrid attention mechanism according to claim 5, characterized in that, The step of adjusting the value range of the Token set in real time based on changes in the current computing resource status and predicted performance indicators during the inference process includes: Based on historical inference task performance monitoring data, a performance prediction model covering response latency, accuracy indicators and computing resource consumption is constructed. Upon receiving a new inference input sequence, the performance prediction model is invoked to estimate the prediction performance under different Token set value ranges, and the initial Token set value range is set before inference begins based on the estimation results. During the inference process, when the deviation between the real-time performance metric and the prediction result of the performance prediction model exceeds a preset threshold, the value range of the Token set is adaptively corrected to balance inference latency and prediction accuracy.

7. The model training method of the lateral hybrid attention mechanism according to claim 1, characterized in that, The newly added parameters include: linear projection weight matrix, feature compression bias term, and kernel function mapping parameters; The newly added parameters are initialized using a random initialization method that is orthogonal to the corresponding weight matrix in the full attention model. During training, the original parameters of the full attention model are kept frozen, and the newly added parameters are updated only based on the gradient of backpropagation, so as to ensure that the linear attention calculation and the full attention calculation are independent of each other.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; and a memory storing computer program instructions, which, when executed, cause the processors to perform a model training method using a lateral hybrid attention mechanism as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program and / or instructions stored thereon, characterized in that, When the computer program and / or instructions are executed by the processor, they implement the model training method of the lateral hybrid attention mechanism as described in any one of claims 1-7.

10. A computer program product comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by the processor, they implement the model training method of the lateral hybrid attention mechanism as described in any one of claims 1-7.

Images