Transverse mixed attention mechanism model training method, medium, device and program product

Abstract

The invention provides a model training method for a transverse mixed attention mechanism, a medium, equipment and a program product, and the method comprises the steps: obtaining a data set containing a plurality of sample sequences, each sample sequence in the data set being formed by arranging a plurality of Token sequences obtained through word segmentation; constructing a to-be-trained model based on the pre-trained full attention model, and adding newly added parameters for linear attention calculation; in the same transverse mixed attention layer, executing total attention calculation on a Token set in a preset total attention calculation range, executing linear attention calculation on all Tokens, and fusing results of the total attention calculation and the linear attention calculation to obtain transverse mixed attention output used for forward reasoning and loss calculation; and based on the output and prediction result, only updating the newly added parameters to optimize the to-be-trained model until the to-be-trained model converges. According to the method, the calculation complexity and video memory occupation of long text sequence processing are reduced, and the reasoning speed and the resource utilization rate are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, and particularly relates to a model training method of a transverse mixed attention mechanism, a medium, an equipment and a program product. BACKGROUND

[0002] As a core technology in the field of deep learning, the attention mechanism has been widely applied in natural language processing, computer vision and other tasks involving sequence data processing.

[0003] The existing full attention mechanism calculates the similarity between each element in the input sequence and all other elements to allocate attention weights, thereby improving the attention degree to important information. However, in the case of an input sequence length of N, the calculation complexity of the full attention mechanism is O(N 2 ), and the memory occupation size is O(N). When processing long sequence text data (such as long texts, script content, multi-day continuous dialogue records, etc.), it will cause significant consumption of computing resources and inference delay, especially when generating the first output Token (the smallest calculation unit after word segmentation processing), it is difficult to meet the low delay requirement.

[0004] To reduce the calculation complexity, the linear attention mechanism is proposed, which reduces the calculation complexity to a linear relationship with the sequence length by linearizing the calculation process of the attention weight. However, the linear attention mechanism has insufficient ability to capture complex dependencies between elements in the sequence, and is prone to lose key information or cause information confusion in the calculation process, thereby affecting the accuracy performance of the model in some tasks.

[0005] Therefore, the existing full attention mechanism and linear attention mechanism have the problems of low efficiency or insufficient accuracy in long sequence tasks, and an improved attention mechanism is needed that can significantly reduce the calculation complexity while maintaining the accuracy of the model. SUMMARY

[0006] In view of the deficiencies of the prior art, the present application provides a model training method of a transverse mixed attention mechanism, a medium, an equipment and a program product, at least to solve the technical problem that the prior art cannot effectively consider the calculation overhead while maintaining the prediction accuracy of the model when processing long sequence text input.

[0007] In order to achieve the above-mentioned purposes and other advantages, some embodiments of the present application provide the following aspects:

[0008] In a first aspect, some embodiments of the present application provide a model training method of a transverse mixed attention mechanism, comprising:

[0009] obtain a dataset containing a plurality of sample sequences, each sample sequence in the dataset being composed of a plurality of tokens arranged in sequence after tokenization processing;

[0010] construct a to-be-trained model based on a pre-trained full-attention model, the to-be-trained model adding new parameters for linear attention calculation on the basis of retaining parameters of the full-attention model;

[0011] In the same horizontal mixed attention layer of the to-be-trained model, perform full-attention calculation on a token set in the input sample sequence located within a preset full-attention calculation range to obtain a full-attention calculation result, perform linear attention calculation on all tokens of the sample sequence to obtain a linear attention calculation result, fuse the full-attention calculation result and the linear attention calculation result to obtain a horizontal mixed attention output used for forward inference and loss calculation based on a forward inference result;

[0012] based on the horizontal mixed attention output and a prediction result of the to-be-trained model, train the to-be-trained model, update only the new parameters to optimize the to-be-trained model, until the to-be-trained model converges, to obtain a trained to-be-trained model.

[0013] In a second aspect, some embodiments of the present application further provide an electronic device, which comprises:

[0014] one or more processors; and a memory storing computer program instructions which, when executed, cause the processor to perform the model training method of the horizontal mixed attention mechanism as described in any of the above.

[0015] In a third aspect, some embodiments of the present application further provide a computer-readable storage medium having stored thereon computer programs and / or instructions which, when executed by a processor, implement the model training method of the horizontal mixed attention mechanism as described in any of the above.

[0016] In a fourth aspect, some embodiments of the present application further provide a computer program product comprising computer programs and / or instructions which, when executed by a processor, implement the model training method of the horizontal mixed attention mechanism as described in any of the above.

[0017] Compared with the related art, in the scheme provided by the embodiment of the application, full attention calculation is performed on part of Token of the input sample sequence in the same transverse mixed attention layer, linear attention calculation is performed on all Token, and the calculation results of the two are fused, so that the overall calculation complexity in the long text sequence processing process is effectively reduced under the premise of ensuring sufficient modeling of key position information, and the memory occupation and calculation delay are reduced. Compared with the traditional method of directly performing full attention calculation on all Token, the application can significantly improve the reasoning speed and resource utilization rate under the long text sequence task, support larger scale data processing while ensuring the model accuracy. In addition, by training only the newly added parameters related to linear attention, the parameter structure and representation ability of the original full attention model are preserved, and the optimization of mixed attention calculation performance is realized under lower training cost, so as to balance the training convergence efficiency and the flexibility of model deployment, and is suitable for efficient reasoning scenes under various hardware environments and different resource constraints. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other embodiments can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0019] Figure 1 is a flowchart of a model training method of a transverse mixed attention mechanism provided by the embodiment of the application;

[0020] Figure 2 is a flowchart of dynamic adjustment of Token set range in the inference stage of the model training method of the transverse mixed attention mechanism provided by the embodiment of the application;

[0021] Figure 3 is a structural schematic diagram of an electronic device provided by the embodiment of the application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical scheme and advantages of the embodiments of the application more clear, the technical scheme in the embodiments of the application will be clearly and completely described below in conjunction with the drawings in the embodiments of the application. Obviously, the described embodiments are some of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0023] First embodiment

[0024] The first embodiment of the present application relates to a model training method of a transverse mixed attention mechanism, referring to Figure 1 The method can include the following steps:

[0025] Step S1: Obtain a dataset containing multiple sample sequences, each sample sequence in the dataset is composed of multiple Tokens arranged in sequence after word segmentation processing.

[0026] For step S1, specifically, the sample sequence refers to a text sequence or other types of symbol sequence used to train a natural language processing model, and its data sources can include open field corpus (such as encyclopedia text, news, dialogue data), specific field annotated data (such as medical literature, financial contract), user interaction log, etc. Each sample sequence is processed by word segmentation, which can be based on dictionary segmentation, statistical model-based segmentation or sub-word unit-based encoding, mapping the original text to multiple Tokens and arranging them in sequence to form the original sequence, thereby preserving the syntax information and context dependency of the input data.

[0027] The Token can be mapped to a unique index number by looking up a predefined word mapping, and further mapped to a fixed-dimensional vector embedding in subsequent steps for input to the attention mechanism for feature modeling. Through this step, the unstructured or semi-structured original input can be standardized into a unified model processable format, ensuring that the position information, symbol boundary information and context semantics of the input are completely preserved in subsequent full attention calculation and linear attention calculation.

[0028] Step S2: Construct a to-be-trained model based on the pre-trained full attention model, and the to-be-trained model increases new parameters for linear attention calculation on the basis of preserving the parameters of the full attention model.

[0029] For step S2, specifically, the full attention model that has completed pre-training on large-scale corpus data is loaded into the training environment, which uses a multi-head self-attention mechanism to model the global dependency of long sequences, and contains multiple sets of Query, Key, Value mapping matrices, position encoding parameters, and feedforward network layer weights and biases, etc. In order to preserve the global context representation ability learned by the full attention model in the pre-training stage, the parameters of the full attention model are kept in a frozen state in this step, and are not updated in the subsequent training.

[0030] Further, the new parameters include a linear projection weight matrix, a feature compression bias term, and a kernel function mapping parameter; wherein the new parameters are initialized in an orthogonal random initialization manner with the corresponding weight matrix in the full attention model, and the original parameters of the full attention model are kept frozen during the training process, and only the new parameters are updated based on the gradient of back propagation, to ensure the independence of linear attention calculation and full attention calculation.

[0031] On this basis, the embodiment introduces a linear attention calculation branch into the existing structure of the full attention model. The new parameters of the linear attention calculation branch include a linear projection weight matrix for mapping input Token representation to a low-dimensional space, a feature compression bias term for the dimension reduction process, and a kernel function mapping parameter for calculating attention weights based on kernel function feature mapping.

[0032] Since the full attention calculation and the linear attention calculation are executed in parallel in the same layer, if their initial values are highly correlated in the feature space, it may cause the representation ability of the two branches to overlap or even interfere with each other at the beginning of training, which will reduce the diversity and complementarity of the hybrid attention mechanism. Therefore, by making the new parameters (such as the linear projection weight matrix) orthogonal to the corresponding weight matrix of the full attention at initialization, their initial feature mapping directions can be ensured to be independent in numerical calculation, thereby prompting the linear attention branch to learn different representation patterns from the full attention. Specifically, first, extract the weight matrix corresponding to the linear attention calculation function (for example, the weight matrix for Query projection) from the full attention model, and use this weight matrix as a reference matrix; generate a random matrix of the same shape, which can generate initial values through uniform distribution or Gaussian distribution; then, orthogonalize the random matrix and the reference matrix, for example, use the Gram-Schmidt orthogonalization method, or combine them to form a large matrix and perform QR decomposition, to obtain an initialization result that is independent in calculation from the initial feature mapping direction of the reference matrix.

[0033] In terms of model structure, the linear attention calculation branch and the original full attention calculation branch share the same batch of Token representation vectors at the input end and perform attention calculation in parallel. Among them, the full attention calculation branch is used to achieve high-precision global information capture within a predetermined range, and the precise dependency relationship of key positions is preserved; the linear attention calculation branch reduces the attention weight calculation complexity from O(N 2 ) to O(N) through low-rank feature mapping and inner product operation, thereby achieving approximate global modeling within the full sequence range and significantly reducing the computational overhead.

[0034] For step S3, specifically, the transverse mixed attention layer refers to an attention calculation unit that is set in a multi-layer attention calculation structure of a to-be-trained model and integrates a full attention calculation path and a linear attention calculation path in parallel within the same layer, to simultaneously obtain accurate local dependency modeling capability and efficient global information capturing capability in one forward propagation.

[0035] For example, a sample sequence X is composed of a plurality of Tokens arranged in an original order, with a length of N. According to a preset full attention calculation range K (K≤N, K is a positive integer), Tokens located at the first K positions in the sample sequence are determined as a full attention calculation set, and a full attention function F is used to calculate the Token set. The full attention function F is used to calculate the similarity score between any Token in the set and other Tokens in the set, and the attention weight is obtained by Softmax normalization, and then the Value vector is weighted and summed to generate the corresponding full attention calculation result. This path can retain accurate global dependency modeling effect in the local area, and is particularly suitable for capturing high-precision feature interaction between key positions.

[0036] At the same time, the linear attention path performs linear attention calculation on all N Tokens based on a linear attention function L. The linear attention function maps the Query and Key vectors to a low-dimensional space through kernel function feature mapping, so that the calculation complexity of the attention weight is reduced from O(N 2 ) to O(N), significantly reducing the calculation overhead and memory occupancy, and enabling the capture of long-range dependencies across the full sequence range.

[0037] After completing the above two-way attention calculation, the transverse mixed attention layer performs weighted fusion of the full attention calculation result and the linear attention calculation result based on a fusion function H=F(X[:K])+L(X). The fusion weight can be obtained by training learning or preset by the system to achieve dynamic balance between accuracy and performance. Finally, the fused transverse mixed attention output is used as the output result of the layer, which is input to the subsequent forward inference process, and the loss function is calculated based on the inference result, so as to drive the entire model to significantly improve the processing efficiency and scalability of long sequences while retaining high-precision local modeling capability.

[0038] Step S4: training the to-be-trained model based on the transverse mixed attention output and the prediction result of the to-be-trained model, updating only the newly added parameters to optimize the to-be-trained model, until the to-be-trained model converges, to obtain the trained to-be-trained model.

[0039] For step S4, specifically, the loss function (such as cross-entropy, mean square error or task-specific loss) is calculated using the transverse mixed attention output and the prediction result of the to-be-trained model, and the back propagation is performed on the to-be-trained model with the loss as the target. In the back propagation stage, the parameter freezing strategy is adopted in this embodiment, that is, the parameter weights of the pre-trained full attention part are kept unchanged, and only the gradient calculation and update are performed on the newly added trainable parameters introduced in the linear attention calculation path. This strategy can effectively reduce the calculation cost of the overall training, avoid the forgetting effect on the high-precision feature modeling ability learned by the original full attention part in the incremental training process, and improve the stability and convergence speed of the training process.

[0040] When the training process meets the preset convergence condition (for example, the loss value on the validation set tends to be stable, the prediction accuracy reaches a threshold, or the number of training iteration rounds reaches an upper limit), the frozen full attention parameters and the updated newly added parameters of the linear attention are solidified together, and the finally trained to-be-trained model is output. The model has high-precision local dependence modeling capability of the original full attention model in the inference stage, and significantly improves the long sequence processing efficiency and scalability, thereby realizing the balanced optimization of performance and efficiency.

[0041] Compared with the related art, in the scheme provided by the embodiments of the present application, the full attention calculation is performed on part of the Token of the input sample sequence in the same transverse mixed attention layer, the linear attention calculation is performed on all Tokens at the same time, and the calculation results of the two are fused, which can effectively reduce the overall calculation complexity in the long text sequence processing process under the premise of ensuring sufficient modeling of key position information, and reduce the memory occupation and calculation delay. Compared with the traditional method of directly performing full attention calculation on all Tokens, the present application can significantly improve the inference speed and resource utilization rate under the long text sequence task, support larger scale data processing while ensuring the model accuracy. In addition, by updating only the newly added parameters related to the linear attention for training, the parameter structure and representation ability of the original full attention model are preserved, the mixed attention calculation performance is optimized at a lower training cost, thereby balancing the training convergence efficiency and the flexibility of model deployment, and being suitable for efficient inference scenarios in various hardware environments and different resource constraints.

[0042] Second embodiment

[0043] The second embodiment of the present application relates to a model training method of a transverse mixed attention mechanism. The second embodiment is an improvement based on the first embodiment, and the specific improvement is that in the second embodiment of the present application, a multi-objective loss calculation mechanism is introduced in the training process, that is, step S4 can further include the following steps:

[0044] Step S401: input the sample sequences in the data set into the to-be-trained model and the full attention model respectively, to obtain the lateral mixed attention output of the to-be-trained model in the lateral mixed attention layer, the prediction result of the to-be-trained model in the output layer, and the output result of the full attention model.

[0045] In this step, the starting point model is Pi0, which is a pre-trained full attention model and keeps the parameters frozen; the fitting model is Pi1, which adds parameters related to linear attention calculation based on the model Pi0. The sample sequences X in the data set are input into the models Pi1 and Pi0, respectively, wherein the sample sequences contain Token sequences, corresponding labels and necessary mask information after preprocessing. The model Pi1 outputs the lateral mixed attention output (including the full attention calculation result and the linear attention calculation result) of the lateral mixed attention layer and the prediction result of the output layer in the forward inference; the model Pi0 outputs the baseline prediction result of the full attention path and provides TopK Logits data for distillation to reduce noise and focus on the main prediction categories.

[0046] Step S402: calculate a first loss value based on the mean square error between the full attention calculation result and the linear attention calculation result in the lateral mixed attention output.

[0047] In this step, the difference between the full attention path and the linear attention path at the feature level is calculated. The full attention calculation result in the lateral mixed attention output is denoted as F(X), and the linear attention calculation result is denoted as L(X). The difference is calculated based on the mean square error (MSE) formula to form an attention distillation loss function:

[0048] Loss attention_distill = MSE(F(X)-L(X)

[0049] The loss function is used to constrain the linear attention features in Pi1 to approximate the full attention feature distribution of Pi0, thereby improving the accuracy of the fusion output.

[0050] Step S403: determine a second loss value based on the distribution difference between the prediction result of the to-be-trained model and the output result of the full attention model.

[0051] Further, in step S403, from the output result of the full attention model, for the prediction result of each Token position in the sample sequence, the top K candidate Tokens with the highest prediction scores and the corresponding prediction scores are selected from the vocabulary to obtain distillation TopK logits data; the second loss value is determined based on the Kullback-Leibler divergence between the prediction result of the to-be-trained model and the distillation TopK logits data.

[0052] Specifically, when the full attention model reasons on a certain input sample sequence, for the prediction result of the 6th Token position in the sample sequence, the vocabulary size is V = 50,000. The full attention model outputs the prediction score (logits) of this position on the entire vocabulary, and the embodiment selects the top K = 5 candidate Tokens in the order of the scores from high to low, for example {“apple”: 2.1, “pear”: 1.9, “orange”: 1.5, “banana”: 1.4, “grape”: 1.3}, and the corresponding prediction scores of the candidate Tokens form the distillation TopK logits data of this position. For the to-be-trained model, the corresponding prediction probability distribution is output at the same Token position, and the embodiment performs Kullback-Leibler (KL) divergence calculation on the prediction distribution and the distillation TopK logits data of the full attention model, which is used to measure the distribution difference of the two on the high-confidence candidate. If the KL divergence value is small, it means that the prediction distribution of the to-be-trained model at this position has well approximated the full attention model, so that the gradient update amplitude at this position can be reduced; otherwise, the update intensity is increased to improve the prediction consistency of the to-be-trained model on the high-importance candidate Token.

[0053] In this step, the TopK Logits of Pi0 is used as the teacher distribution to measure the distribution difference between the prediction result Pi1(X) of Pi1 and the prediction result Pi0(X) of Pi0, and the KL divergence is used to calculate the distillation loss of the logits layer, and the loss function is:

[0054] Loss logits = KL(Pi1(X), Pi0(X))

[0055] The loss function encourages Pi1 to align the prediction distribution of Pi0 in the output space, which not only avoids the high overhead of complete distillation calculation on the full vocabulary, but also preserves the knowledge transfer effect on the main prediction result.

[0056] Step S404: determining a third loss value based on the difference between the prediction result of the to-be-trained model and the target label of the data set.

[0057] Further, in step S404, the cross-entropy loss function is used to calculate the difference between the prediction result of the to-be-trained model in the language model task and the target label of the data set, to determine the third loss value, which is used to measure the performance of the to-be-trained model in the language model task.

[0058] For each Token position in the input sample sequence, the prediction probability distribution generated by the to-be-trained model at the output layer is obtained, and is compared with the target label at the corresponding position; the target label is the correct Token label annotated in the data set. Then, the difference between the prediction probability distribution and the target label is calculated using a cross-entropy loss function, to obtain the loss value corresponding to the position, and the loss values of all Token positions in the sequence are averaged to form the third loss value of the sample.

[0059] The third loss value is used to measure the prediction performance of the to-be-trained model on the language model task. When the prediction result is closer to the target label, the cross-entropy loss value is smaller, indicating that the model can more accurately generate an output sequence consistent with the training corpus; otherwise, the newly added linear attention parameters are adjusted through the back propagation mechanism to improve the model's fitting ability to semantics and context. For example, when the target label is "apple"

[0060] , if the prediction probability of the model for "apple" is 0.92, and the sum of the probabilities of the remaining candidate Tokens is 0.08, then the cross-entropy loss value at this position is smaller, indicating that the prediction is more accurate; if the prediction probability distribution is more dispersed, then the loss value will significantly increase, prompting the model to be further optimized.

[0061] In this step, the prediction result Pi1(X) of Pi1 is compared with the target label Y of the data set to evaluate its performance on the actual task (such as language modeling). The loss function is:

[0062] LossLM=L(Pi1(X),Y)

[0063] This loss function ensures that Pi1 is close to the teacher model distribution while retaining the fitting ability to the true label.

[0064] Step S405: The first loss value, the second loss value, and the third loss value are weighted and summed according to the preset weights to obtain a total loss value, and the newly added parameters are iteratively updated based on the total loss value to train the to-be-trained model until convergence.

[0065] In this step, the three loss values are weighted and summed according to the preset weights a, b, g to obtain a total loss function:

[0066] Loss total =α×Loss attention_distill +β×Loss logits +γ×Loss LM

[0067] The loss value based on the total loss function is used to perform back propagation, only the parameters related to the newly added linear attention calculation in Pi1 are updated, and the parameters of the original full attention calculation in Pi0 remain unchanged. The training process is iteratively performed until the total loss value converges or the performance of the validation set meets the stopping condition, and finally the target model that meets the expected accuracy and efficiency is obtained.

[0068] Through multi-source loss driven knowledge distillation and fitting training, the embodiment can gradually approach the full attention path in terms of feature distribution and prediction results while maintaining the original accurate modeling capability of the full attention path, thereby achieving the best balance between computational resource occupation and model performance in long sequence modeling, and significantly improving the inference efficiency and scalability. Even under the condition of using only about 30B token pre-training data, the model can fully fit and flatten the validation set loss curve in the early stage of training, and achieve a small amount of surpassing of the validation set loss in some test scenarios. Compared with the control model without using the optimization mechanism of the present application, the present method can maintain or even improve the convergence effect and generalization performance of the model while significantly reducing the training data size and computational resource consumption, thereby effectively improving the feasibility and economy of model deployment under the condition of limited computing power and data resources.

[0069] Third embodiment

[0070] The third embodiment of the present application relates to a model training method of a transverse mixed attention mechanism. The third embodiment is an improvement based on the first embodiment, and the specific improvement is that, as shown in Figure 2 After the to-be-trained model is trained to obtain the optimized model, the following steps can be included:

[0071] Step A1: In the inference stage, based on the computing resource state information, the input data feature information, and the preset performance constraint condition, the value range of the Token set using full attention calculation in the inference input sequence is dynamically determined, wherein the performance constraint condition includes a maximum response delay threshold and an accuracy drop tolerance;

[0072] Step A2: In the inference process, the value range of the Token set is adjusted in real time according to the changes of the current computing resource state and the predicted performance index;

[0073] Step A3: When it is detected that the response delay exceeds the maximum response delay threshold or the accuracy drop exceeds the accuracy drop tolerance, the value range of the different Token set is switched.

[0074] In the inference stage of this embodiment, first, the current computing resource state information is obtained, including but not limited to the available computing power of the inference server or terminal device (such as GPU / TPU core utilization rate, video memory occupancy rate, CPU occupancy rate), I / O bandwidth usage, and current task queue length, etc.; at the same time, the feature information of the input sequence to be inferred is extracted, such as sequence length, Token distribution density, key position Token proportion, and accuracy prediction indicators provided by historical inference records or model performance monitoring modules.

[0075] Based on the above information, the system performs calculation according to the preset performance constraint conditions, including the maximum response delay threshold and the accuracy decline tolerance. For example, when inferring a real-time dialogue task, the maximum response delay threshold can be set to 200 milliseconds, and the accuracy decline tolerance can be set to a task performance index decline amplitude of not more than 1.5%. Under the premise of meeting the performance constraint conditions, the value range K (K≤N, N is the total length of the sequence) of the Token set using full attention calculation in the inference input sequence is dynamically determined.

[0076] In actual operation, the K value can be adjusted in real time according to the changes of the computing resource state and the predicted performance indicators. For example, when it is detected that the GPU utilization rate and the video memory occupancy rate are continuously at a low load level, and the model prediction accuracy remains stable, the system can appropriately increase the K value, so that more Tokens participate in full attention calculation, to further improve the inference accuracy. Conversely, when the response delay approaches or exceeds the maximum response delay threshold, or the accuracy decline amplitude calculated by the performance monitoring module exceeds the accuracy decline tolerance, the system immediately switches to a smaller K value, only keeping the key position Tokens participating in full attention calculation, and the rest of the Tokens are processed by the linear attention path, thereby reducing the calculation complexity and ensuring the response speed.

[0077] Taking a long sequence question and answer task as an example, assuming that the input sequence length is 1024 Tokens, under the condition of sufficient computing resources and stable accuracy, the system can set K to 512, so that the first half of the Tokens perform full attention calculation; when it is detected that the video memory utilization rate approaches the upper limit or the response delay exceeds the standard, K will be dynamically adjusted to 256 or even 128, to reduce the full attention calculation load. This dynamic adjustment mechanism can realize real-time balance between inference accuracy and response speed, and adapt to the changing needs of different hardware resources and task scenarios.

[0078] Further, step A2 specifically includes:

[0079] Step A201: based on the performance monitoring data of historical inference tasks, a performance prediction model covering response delay, accuracy indicators, and computing resource consumption is constructed;

[0080] Step A202: Upon receiving a new inference input sequence, the performance prediction model is called to estimate the predicted performance under different value ranges of Token set, and based on the estimation results, the value range of the initial Token set is set before the inference begins;

[0081] Step A203: During the inference process, when the deviation between the real-time performance indicators and the estimation results of the performance prediction model exceeds the preset threshold, the adaptive correction of the value range of the Token set is triggered to balance the inference delay and the prediction accuracy.

[0082] Specifically, first, based on the performance monitoring data of historical inference tasks under different running environments, multi-dimensional performance data samples covering inference response delay, prediction accuracy indicators and computing resource consumption are collected and recorded. Subsequently, feature engineering and data cleaning are performed using the performance data samples to build a performance prediction model covering performance dimensions. The performance prediction model can be implemented using structures such as regression analysis, gradient boosting tree or lightweight neural network, and is used to output the estimated values of delay, accuracy and resource occupation under the given value range of Token set.

[0083] When a new inference input sequence is received, the inference control module calls the performance prediction model to calculate the predicted performance indicator values, including inference delay, prediction accuracy and computing resource occupation rate, for multiple candidate Token set value ranges. The inference control module performs multi-objective trade-off based on the predicted performance indicator values and the preset performance constraint conditions to automatically set the value range of the initial Token set before the inference begins, to balance the expected accuracy and computing delay.

[0084] During the inference execution process, real-time performance indicators are continuously monitored and compared with the estimation results of the performance prediction model. When the deviation between the real-time performance indicators and the estimation results exceeds the preset threshold (e.g., the response delay exceeds the predicted value by 10% or the accuracy decreases by more than the tolerance), the adaptive correction of the value range of the Token set is triggered. This correction process can be achieved by increasing or reducing the size of the Token set, adjusting the Token selection strategy, etc., to continuously maintain the balance between inference response delay and prediction accuracy under dynamic computing resource fluctuations and task characteristic changes.

[0085] The step division of the above methods is only for clarity, and in implementation, one step can be combined or some steps can be split into multiple steps, as long as the same logical relationship is included, and all are within the protection scope of the present application; adding insignificant modifications or introducing insignificant designs in the algorithm or process, but not changing the core design of the algorithm and process, are within the protection scope of the present application.

[0086] In addition, some embodiments of the present application also provide an electronic device. The electronic device can be various forms of digital computers, such as a laptop computer, a desktop computer, a workstation, a personal digital assistant, a server, a blade server, a mainframe computer, and the like. The electronic device can also be various forms of mobile devices, such as a personal digital processing, a cellular phone, a smart phone, a wearable device, and other similar computing devices.

[0087] The electronic device includes one or more processors, and a memory storing computer program instructions which, when executed, cause the processor to perform a model training method of a transverse mixed attention mechanism as provided by any one or more embodiments described above. Figure 3 An exemplary structural diagram of the electronic device is disclosed. The electronic device includes one or more processors 1101, a memory 1102, and an interface for connecting components, including a high-speed interface and a low-speed interface. Various components are connected to each other by different buses, and can be installed on a common motherboard or otherwise as needed. The processor can process instructions executed within the electronic device, including instructions stored in the memory or on the memory to display graphical information of a GUI on an external input / output device, such as a display device coupled to the interface. In some other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memories, if desired. Also, multiple electronic devices can be connected, each device providing part of the necessary operations. Among them, the components shown herein, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or claimed herein.

[0088] The electronic device can also include an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 can be connected by a bus or other means, Figure 3 For example, by bus connection.

[0089] The input device 1103 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device, such as touch screen, keypad, mouse, trackpad, touchpad, pointing stick, one or more mouse buttons, trackball, joystick, and the like input device. The output device 1104 can include a display device, an auxiliary lighting device (e.g., LED), and a tactile feedback device (e.g., a vibration motor), and the like. The display device can include, but is not limited to, a liquid crystal display, a light emitting diode display, and a plasma display. In some embodiments, the display device can be a touch screen.

[0090] To provide for interaction with a user, the electronic device can be a computer. The computer has a display device (e.g., a cathode ray tube or LCD monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, etc.); and input from the user can be received in any form (e.g., acoustic input, speech input, tactile input, etc.).

[0091] In the embodiments of the present application, the computer program / instruction is stored on the computer readable medium, and the computer program / instruction is executed by the processor to implement the model training method of the transverse mixed attention mechanism provided by any one or more of the above embodiments. The computer readable medium can be included in the electronic device described in the above embodiments, or can exist separately and not be assembled into the device. The computer readable medium carries one or more computer readable instructions.

[0092] The memory 1102 can be used as a non-transitory computer readable storage medium to store non-transitory software programs, non-transitory computer executable programs and modules. The processor 1101 executes various functions and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 1102, so as to implement the program instructions / modules corresponding to the method provided by any one or more of the above embodiments in the embodiments of the present application.

[0093] The memory 1102 can include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 1102 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory 1102 can optionally include a memory disposed remotely with respect to the processor 1101, and these remote memories can be connected to the electronic device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0094] Note that the computer-readable medium described herein can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable medium can be, for example but not limited to, a system, a device, or a computer program product embodied in one or more computer readable media embodying computer readable instructions, data structures, program modules, or other data. The computer-readable storage medium can be, for example but not limited to, a non-transitory computer-readable medium, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, the computer-readable medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0095] The computer-readable medium includes permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technology, read-only optical disc, digital versatile disc or other optical storage, magnetic cassette, magnetic tape disc storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0096] Computer program code for carrying out operations of the present application can be written in one or more programming languages or combinations of languages including object-oriented programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the C programming language or similar programming languages. Program code can be executed entirely on a user computer, partially on a user computer, as a separate software package, partially on a user computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network or a wide area network, or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0097] In the above-described embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. For example, application specific integrated circuits, general purpose computers or any other similar hardware devices can be used. In some embodiments, the software programs of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software programs of the present application (including related data structures) can be stored in a computer-readable recording medium, such as a RAM memory, a magnetic or optical drive or a soft disk and the like. In addition, some steps or functions of the present application can be implemented by hardware, such as a circuit cooperating with a processor to perform the respective steps or functions.

[0098] The computer program product provided by the embodiments of the present application includes one or more computer programs / instructions, which, when executed by a processor, generate all or part of the processes or functions described in the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as DVD), or semiconductor media (such as solid state disk) and the like.

[0099] The flowchart or block diagram in the drawings illustrates the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, a segment, or a portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks noted in succession can in fact be executed substantially concurrently, or the blocks can sometimes be executed in reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, or combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by special-purpose hardware-based systems that perform the specified functions or operations, or combinations of special-purpose hardware and computer instructions.

[0100] The scope of the application is defined by the appended claims rather than by the description set forth herein, and therefore the specification is not intended to limit the claims in any way. No admission is made that any of the related art references are prior art. It will be apparent that modifications and / or improvements can be made to the embodiments described herein without departing from the scope of the application. Accordingly, while the application is presented in terms of embodiments, it should be appreciated that the application is not limited to these embodiments. The application is limited only by the claims that follow, the full scope of which is deemed to encompass any changes which come within the meaning and range of equivalency of the claims. None of the features or limitations described herein are intended to be limiting but are intended to be examples of presently preferred embodiments. Thus, it should be apparent that the application can be practiced otherwise than as specifically outlined herein without departing from the scope and spirit of the application. Any discussion of mechanisms underlying the application should be treated as pure speculation, capable of

[0101] The above description is only specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily make changes or replacements within the technical range disclosed by the present application, which should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be limited by the protection scope of the claims, and the above embodiments should be regarded as exemplary and non-limiting.

Claims

1. A model training method using a lateral hybrid attention mechanism, characterized in that, include: Obtain a dataset containing multiple sample sequences, wherein each sample sequence in the dataset consists of multiple tokens obtained through word segmentation arranged in sequence; A model to be trained is constructed based on the pre-trained full attention model. The model to be trained retains the parameters of the full attention model and adds new parameters for linear attention calculation. In the same horizontal hybrid attention layer of the model to be trained, full attention calculation is performed on the set of tokens in the input sample sequence that are within the preset full attention calculation range to obtain the full attention calculation result. Linear attention calculation is performed on all tokens in the sample sequence to obtain the linear attention calculation result. The full attention calculation result and the linear attention calculation result are fused to obtain a horizontal hybrid attention output for forward inference and loss calculation based on the forward inference result. Based on the lateral mixed attention output and the prediction results of the model to be trained, the model to be trained is trained by updating only the newly added parameters to optimize the model to be trained until the model to be trained converges, so as to obtain the trained model to be trained.

2. The model training method of the lateral hybrid attention mechanism according to claim 1, characterized in that, The step of training the model to be trained based on the dataset, optimizing the model to be trained by only updating the newly added parameters, until the model to be trained converges to obtain the trained model to be trained includes: The sample sequences in the dataset are input into the model to be trained and the full attention model respectively to obtain the horizontal mixed attention output of the model to be trained in the horizontal mixed attention layer, the prediction result of the model to be trained in the output layer and the output result of the full attention model. Based on the mean square error between the full attention calculation result and the linear attention calculation result in the lateral hybrid attention output, the first loss value is calculated; The second loss value is determined based on the distribution difference between the prediction results of the model to be trained and the output results of the full attention model. A third loss value is determined based on the difference between the prediction results of the model to be trained and the target labels of the dataset; The first loss value, the second loss value, and the third loss value are weighted and summed according to preset weights to obtain a total loss value. The newly added parameters are then iteratively updated based on the total loss value to train the model until convergence.

3. The model training method of the lateral hybrid attention mechanism according to claim 2, characterized in that, The step of determining the second loss value based on the distribution difference between the prediction result of the model to be trained and the output result of the full attention model includes: From the output of the full attention model, for the prediction result of each token position in the sample sequence, select the top K candidate tokens with the highest prediction scores and their corresponding prediction scores from the vocabulary to obtain the distilled TopKlogits data; The second loss value is determined based on the Kullback-Leibler divergence between the prediction results of the model to be trained and the distilled TopK logits data.

4. The model training method of the lateral hybrid attention mechanism according to claim 2, characterized in that, The step of determining the third loss value based on the difference between the prediction result of the model to be trained and the target label of the dataset includes: The third loss value is determined by calculating the difference between the prediction result of the model to be trained in the language model task and the target label of the dataset using the cross-entropy loss function.

5. The model training method of the lateral hybrid attention mechanism according to claim 1, characterized in that, After the model to be trained is trained to obtain an optimized model, during the inference stage, based on the computing resource status information, input data feature information and preset performance constraints, the value range of the token set using full attention computing in the inference input sequence is dynamically determined. The performance constraints include the maximum response latency threshold and the accuracy degradation tolerance. During the reasoning process, the value range of the Token set is adjusted in real time based on changes in the current computing resource status and predicted performance indicators; When the response latency exceeds the maximum response latency threshold or the accuracy degradation exceeds the accuracy degradation tolerance, the value range of the different Token sets is switched.

6. The model training method of the lateral hybrid attention mechanism according to claim 5, characterized in that, The step of adjusting the value range of the Token set in real time based on changes in the current computing resource status and predicted performance indicators during the inference process includes: Based on historical inference task performance monitoring data, a performance prediction model covering response latency, accuracy indicators and computing resource consumption is constructed. Upon receiving a new inference input sequence, the performance prediction model is invoked to estimate the prediction performance under different Token set value ranges, and the initial Token set value range is set before inference begins based on the estimation results. During the inference process, when the deviation between the real-time performance metric and the prediction result of the performance prediction model exceeds a preset threshold, the value range of the Token set is adaptively corrected to balance inference latency and prediction accuracy.

7. The model training method of the lateral hybrid attention mechanism according to claim 1, characterized in that, The newly added parameters include: linear projection weight matrix, feature compression bias term, and kernel function mapping parameters; The newly added parameters are initialized using a random initialization method that is orthogonal to the corresponding weight matrix in the full attention model. During training, the original parameters of the full attention model are kept frozen, and the newly added parameters are updated only based on the gradient of backpropagation, so as to ensure that the linear attention calculation and the full attention calculation are independent of each other.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; and a memory storing computer program instructions, which, when executed, cause the processors to perform a model training method using a lateral hybrid attention mechanism as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program and / or instructions stored thereon, characterized in that, When the computer program and / or instructions are executed by the processor, they implement the model training method of the lateral hybrid attention mechanism as described in any one of claims 1-7.

10. A computer program product comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by the processor, they implement the model training method of the lateral hybrid attention mechanism as described in any one of claims 1-7.

Images