Attention mechanism improvement method and device for long sequence and medium

Abstract

The invention discloses an attention mechanism improvement method and device for a long sequence and a medium, belongs to the technical field of deep learning, and is used for solving the problems that in an existing Transform architecture, a self-attention mechanism needs complex position coding to process sequence order information, the complexity of a model is increased, and attention distraction is likely to occur. And the long-distance dependency relationship is difficult to model effectively. The method comprises the following steps: carrying out attention score calculation of related front and back positions on an input sequence of an attention mechanism module to obtain an attention score of a current position; performing probability transformation on the attention score to obtain a breakpoint probability; performing attention calculation between the current position and the target position to obtain an attention weight; performing weighted summation processing on the output vector of the current position to obtain a final output weight vector; and deploying the improved attention mechanism module corresponding to the final output weight vector into the application model, and determining a final output result.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to an improved method, device and medium for an attention mechanism for long sequences. Background Art

[0002] Since the Transformer architecture was proposed, it has made breakthrough progress in various sequence processing tasks. However, the traditional Transformer has the following technical problems: first, its attention mechanism requires complex position encoding to process sequence order information, which increases the complexity of the model; second, the softmax-based attention calculation is prone to distraction problems, affecting the model's ability to model long-distance dependencies; in addition, when processing long sequences, the computational complexity of the traditional attention mechanism grows quadratically, limiting the practical application scope of the model.

[0003] At present, the academic community has proposed a variety of improvement schemes, such as relative position encoding, linear attention, etc. However, these methods still require additional position information input or sacrifice computational efficiency while maintaining model performance. Therefore, how to design an attention mechanism that can effectively model sequence order information and efficiently process long sequences has become an important issue in current research. Summary of the invention

[0004] The embodiments of the present application provide an improved method, device and medium for an attention mechanism for long sequences, which are used to solve the following technical problems: In the existing Transformer architecture, the self-attention mechanism requires complex position encoding to process sequence order information, which increases the complexity of the model and is prone to attention distraction problems, making it difficult to effectively model long-distance dependencies.

[0005] The present application embodiment adopts the following technical solutions: On the one hand, an embodiment of the present application provides an improved method for an attention mechanism for long sequences, including: calculating the attention scores of the previous and next positions of the input sequence of the attention mechanism module in the Transformer architecture to obtain the attention score of the current position; performing probability conversion on the attention score through a nonlinear activation function to obtain a breakpoint probability; performing attention calculation between the current position and the target position according to the breakpoint probability to obtain an attention weight; performing numerical conversion processing on the attention weight based on a preset logarithmic space; performing weighted sum processing on the output vector of the current position based on the attention weight after numerical conversion to obtain a final output weight vector; deploying the improved attention mechanism module corresponding to the final output weight vector into the application model, and determining the final output result of the attention mechanism module in the application model.

[0006] The embodiment of the present application naturally encodes the position information in the attention calculation by introducing the stick-breaking segmentation process, without the need for an additional position encoding module, thus simplifying the model structure. The improved method of the attention mechanism for long sequences has a natural local preference, making it easier for the application model to capture local dependencies in the sequence while maintaining the ability to model long-distance dependencies. Finally, the use of logarithmic space calculations and improved softplus functions effectively solves the stability problem in numerical calculations, showing significant advantages in long sequence processing tasks.

[0007] In a feasible implementation, the attention scores of the previous and next positions of the input sequence of the attention mechanism module in the Transformer architecture are calculated to obtain the attention score of the current position, specifically including: , get the attention score of the current position j ;in, represents the query vector at position j, Represents the key vector at position i, where d is the vector dimension.

[0008] In a feasible implementation, the attention score is converted into a probability through a nonlinear activation function to obtain a breakpoint probability, specifically including: through a nonlinear activation function, and according to , and obtain the breakpoint probability ;in, represents the attention score of the current position j, For mathematical symbols.

[0009] In a feasible implementation, according to the breakpoint probability, attention is calculated between the current position and the target position to obtain an attention weight, specifically including: , get the attention weight ;in, is the breakpoint probability, and also represents the initial attention probability assigned to position i by position j; It represents the probability that the current position j does not allocate attention to all the intermediate positions k, where the intermediate position k is the position between the current position j and position i.

[0010] In a feasible implementation, based on a preset logarithmic space, the attention weight is numerically converted, specifically including: , get the attention weight The logarithmic form of ;in, represents the attention score of the current position j, and the middle position k is the position between the current position j and the position i; is a mathematical operation symbol; is the softplus function Numerical stabilization of attention scores.

[0011] In a feasible implementation, based on the attention weight after numerical conversion, the output vector of the current position is weighted and summed to obtain the final output weight vector, specifically including: , and obtain the final output weight vector ;in, is the attention weight; is the value vector at position i.

[0012] In a feasible implementation manner, determining the final output result of the attention mechanism module in the application model specifically includes: , get the final output result of the attention mechanism module in the application model ;in, is the output projection matrix; h is the output vector of multi-head attention; Calculate the concatenation of the output vectors.

[0013] In a feasible implementation, the improved attention mechanism module corresponding to the final output weight vector is deployed into the application model, specifically including: generating the improved attention mechanism module based on the stick-breaking segmentation attention mechanism contained in the final output weight vector; wherein the input sequence in the improved attention mechanism module includes the query vector, key vector and value vector after the linear transformation matrix projection; through the preset cross entropy loss function, optimizer and cosine learning rate decay strategy, the improved attention mechanism module is subjected to the associated deployment processing related to the application model to obtain a new application model; based on the improved attention mechanism module in the new application model, the final output result under the attention mechanism module in the task data is obtained.

[0014] In a second aspect, an embodiment of the present application also provides an improved attention mechanism device for long sequences, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor so that the at least one processor can execute an improved attention mechanism method for long sequences described in any of the above embodiments.

[0015] In the third aspect, an embodiment of the present application also provides a non-volatile computer storage medium, characterized in that the storage medium is a non-volatile computer-readable storage medium, and the non-volatile computer-readable storage medium stores at least one program, each of which includes instructions, and when the instructions are executed by a terminal, the terminal executes an improved method for an attention mechanism for long sequences described in any of the above-mentioned embodiments.

[0016] The present application provides a method, device and medium for improving the attention mechanism for long sequences. Compared with the prior art, the embodiments of the present application have the following beneficial technical effects: The embodiment of the present application naturally encodes the position information in the attention calculation by introducing the stick-breaking segmentation process, without the need for an additional position encoding module, thus simplifying the model structure. The improved method of the attention mechanism for long sequences has a natural local preference, making it easier for the application model to capture local dependencies in the sequence while maintaining the ability to model long-distance dependencies. Finally, the use of logarithmic space calculations and improved softplus functions effectively solves the stability problem in numerical calculations, showing significant advantages in long sequence processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings: Figure 1 A flow chart of an improved method for attention mechanism for long sequences provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of an improved device for attention mechanism for long sequences provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0019] It should be noted that in the existing Transformer architecture, the self-attention mechanism requires complex positional encoding to process sequence order information, which not only increases the complexity of the model, but also limits the model's ability to process long sequences. In addition, the traditional softmax attention mechanism is prone to distraction problems and is difficult to effectively model long-distance dependencies.

[0020] The present application embodiment provides an improved method for the attention mechanism of long sequences, such as Figure 1 As shown, the improved method for the attention mechanism of long sequences specifically includes steps S101-S106: S101. Calculate the attention scores of the previous and next positions of the input sequence of the attention mechanism module in the Transformer architecture to obtain the attention score of the current position.

[0021] Specifically, according to , get the attention score of the current position j .in, represents the query vector at position j, Represents the key vector at position i, where d is the vector dimension.

[0022] In one embodiment, each Transformer layer adopts a multi-head attention design, and the input sequence is first projected into a query vector, a key vector, and a value vector through three independent linear transformation matrices. , where L is the sequence length and d is the hidden dimension, obtained by linear transformation: ,in , , is the learnable parameter matrix.

[0023] In one embodiment, in each attention head, the attention weight is calculated using the stick-breaking segmentation process, which is a key innovation that distinguishes it from the traditional Transformer. For the calculation of the attention weight of position j to position i, the initial attention score is first calculated: ,in is the dimension of the attention head.

[0024] S102. Perform probability conversion on the attention score through a nonlinear activation function to obtain a breakpoint probability.

[0025] It should be noted that it is necessary and reasonable to use nonlinear activation functions for probability conversion: 1. Raw Attention Score It is calculated by the dot product of the query vector and the key vector, and its value range is (-∞, +∞). However, under the semantics of breakpoint probability, it needs to be mapped to the interval (0,1) to indicate the probability of "breaking" at a certain position. This mapping must be achieved through nonlinear transformation.

[0026] 2. Linear transformation cannot meet this requirement because: Linear transformation can only scale and translate values. It cannot compress the unbounded real value domain into a bounded interval. It cannot guarantee that the output value is always a valid probability value.

[0027] 3. Advantages of choosing a nonlinear activation function (such as the sigmoid function) for conversion: It can naturally map any real number to the (0,1) probability interval. It maintains the relative size relationship of the original attention scores. It provides a smooth gradient, which is conducive to model training.

[0028] Specifically, through a nonlinear activation function, and according to , and get the breakpoint probability .in, represents the attention score of the current position j, For mathematical symbols.

[0029] In one embodiment, the score is converted to a breakpoint probability by a sigmoid function The nonlinear activation function used in this application is specifically the Sigmoid function, whose mathematical expression is: σ(x)=1 / (1+e^(-x)). The selection of the Sigmoid function as the nonlinear activation function is based on the following in-depth technical considerations: 1. The Sigmoid function has the property of compressing any real value into the (0,1) interval, which perfectly meets the need to convert attention scores into breakpoint probabilities. In the attention mechanism, the attention scores at different positions need to be normalized into probability distributions, and the Sigmoid function can naturally achieve this.

[0030] 2. The Sigmoid function is continuous and differentiable everywhere, and has good mathematical properties: the function derivative is simple and easy to calculate the gradient: σ'(x) = σ(x)(1-σ(x)). The function value is always in the interval (0,1), ensuring the validity of the probability. The derivative reaches its maximum value at x=0, which is conducive to the stability of model training.

[0031] 3. The Sigmoid function has a natural saturation property: when the input is large or small, the output will approach 1 or 0. This property helps to produce a sparse attention distribution, making it easier for the model to focus on key positions. It has a natural inhibitory effect on outliers, improving the robustness of the model. In one embodiment, the specific implementation steps of probability conversion are as follows: First, obtain the attention score calculated in step S101 , which is obtained by taking the dot product of the query vector and the key vector and scaling it: , where d is the vector dimension. Then, the attention score Transformed by a non-linear activation function: = Finally, the β_{i,j} obtained after the conversion is the breakpoint probability, which ranges from (0,1) and represents the probability of allocating attention from position j to position i. In other words, this conversion can map any range of attention scores to the (0,1) interval, maintaining the relative relationship of the attention scores and providing a reasonable probability basis for subsequent attention calculations.

[0032] As a feasible implementation method, through the above-mentioned specific nonlinear activation function implementation and probability conversion process, the present application can: map the attention scores of any range to the (0,1) interval to make it have probabilistic meaning. Maintain the relative size relationship of the attention scores, and effectively compress the extreme values. Provide a reasonable probabilistic basis for the subsequent stick-breaking process, thereby achieving effective attention allocation.

[0033] S103. According to the breakpoint probability, attention is calculated between the current position and the target position to obtain an attention weight.

[0034] Specifically, according to , get the attention weight .in, is the breakpoint probability, and also represents the initial attention probability assigned to position i by position j. It represents the probability that the current position j does not allocate attention to all the intermediate positions k, where the intermediate position k is the position between the current position j and position i.

[0035] As a feasible implementation method, . This formula is a progressive attention allocation process: represents the initial attention probability assigned to position i by position j, and represents the probability that position j does not allocate attention to all intermediate positions k (i.e., positions between i and j). This calculation method has the following characteristics: when position j allocates higher attention to a certain intermediate position k (i.e., When the Due to the effect of the term, the attention of position j to the earlier position i will be naturally suppressed. This mechanism makes the model naturally tend to focus on recent information when processing sequence information, and can capture the order relationship of the sequence without additional position encoding.

[0036] In one embodiment, if we want to calculate the attention weight of position 5 to position 2 , we need to consider the influence of positions 3 and 4. First, calculate the initial attention probability of position 5 to position 2 , and then multiply it by the probability that position 5 does not pay attention to position 3 and the probability of not paying attention to position 4 If position 5 strongly follows position 4 (i.e. close to 1), then will approach 0, resulting in the final is very small, which reflects the characteristic that the current position tends to focus more on recent information.

[0037] S104: Based on a preset logarithmic space, perform numerical conversion processing on the attention weight.

[0038] Specifically, according to , get the attention weight The logarithmic form of .in, represents the attention score of the current position j, and the middle position k is the position between the current position j and position i. A mathematical operation symbol. is the softplus function Numerical stabilization of attention scores.

[0039] As a feasible implementation method, in order to solve the problem of numerical calculation stability that may occur in practical applications, this application designs a calculation method based on logarithmic space. Convert the above attention weight calculation into logarithmic form: The improved softplus function is specially designed to handle the numerical overflow problem: when the input x is greater than the threshold (such as 15), x is directly returned; otherwise, .

[0040] S105. Based on the attention weight after numerical conversion, the output vector of the current position is weighted summed to obtain the final output weight vector.

[0041] Specifically, according to , and get the final output weight vector .in, is the attention weight. is the value vector at position i.

[0042] In one embodiment, based on the calculated attention weights, the output vector is obtained by weighted summation: .

[0043] S106. Deploy the improved attention mechanism module corresponding to the final output weight vector into the application model, and determine the final output result of the attention mechanism module in the application model.

[0044] Specifically, based on the stick-breaking attention mechanism included in the final output weight vector, an improved attention mechanism module is generated, wherein the input sequence in the improved attention mechanism module includes the query vector, the key vector, and the value vector after the linear transformation matrix projection.

[0045] Through the preset cross entropy loss function, optimizer and cosine learning rate decay strategy, the improved attention mechanism module is used to carry out the associated deployment processing of the relevant application model to obtain a new application model.

[0046] Based on the improved attention mechanism module in the new application model, the final output result of the attention mechanism module in the task data is obtained.

[0047] In one embodiment, the attention mechanism is applied to the pre-trained language model. In the pre-training stage, the model adopts the autoregressive language modeling objective and the cross entropy loss function. The optimizer uses Adam and adopts the cosine learning rate decay strategy. In the model fine-tuning stage, the application adopts the low-rank adaptation (LoRA) technology to improve efficiency. For the original parameter matrix , introduce two low-rank matrices and , where r is much smaller than d (e.g. r=8). During fine-tuning, the original weights remain frozen and only these low-rank matrices are updated: , where α is a scaling factor (e.g., α=16). Applying LoRA to the query, key, and value projection matrices in the attention mechanism allows the model to efficiently adapt to different downstream tasks.

[0048] As a feasible implementation method, according to , and obtain the final output result of the attention mechanism module in the application model .in, is the output projection matrix; h is the output vector of multi-head attention. Calculate the concatenation of the output vectors.

[0049] In one embodiment, for a multi-head attention mechanism, the output vectors of the h attention heads are concatenated and then output through the projection matrix Transform and get the final output representation: This multi-head design enables the model to focus on different sequence patterns at the same time, enhancing its expressive power. In addition, adding the input to the output of the attention layer through residual connections facilitates the effective transfer of information: .

[0050] In addition, the present application embodiment also provides an improved device for attention mechanism of long sequence, such as Figure 2 As shown, the attention mechanism improvement device for long sequences specifically includes: At least one processor 201. And a memory 202 in communication with the at least one processor 201. The memory 202 stores instructions that can be executed by the at least one processor 201, so that the at least one processor 201 can execute: Calculate the attention scores of the previous and next positions of the input sequence of the attention mechanism module in the Transformer architecture to obtain the attention score of the current position; Through the nonlinear activation function, the attention score is converted into probability to obtain the breakpoint probability; According to the breakpoint probability, the attention between the current position and the target position is calculated to obtain the attention weight; Based on the preset logarithmic space, the attention weights are numerically converted; Based on the attention weights after numerical conversion, the output vector of the current position is weighted summed to obtain the final output weight vector; The improved attention mechanism module corresponding to the final output weight vector is deployed to the application model, and the final output result of the attention mechanism module in the application model is determined.

[0051] The embodiment of the present application naturally encodes the position information in the attention calculation by introducing the stick-breaking segmentation process, without the need for an additional position encoding module, thus simplifying the model structure. The improved method of the attention mechanism for long sequences has a natural local preference, making it easier for the application model to capture local dependencies in the sequence while maintaining the ability to model long-distance dependencies. Finally, the use of logarithmic space calculations and improved softplus functions effectively solves the stability problem in numerical calculations, showing significant advantages in long sequence processing tasks.

[0052] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0053] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0054] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0055] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0056] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0057] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0058] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0059] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0060] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0061] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0062] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the specification of the present application.

Claims

1. An improved method for attention mechanism for long sequences, characterized in that: The method comprises: Calculate the attention scores of the previous and next positions of the input sequence of the attention mechanism module in the Transformer architecture to obtain the attention score of the current position; The attention score is converted into a probability through a nonlinear activation function to obtain a breakpoint probability; According to the breakpoint probability, attention is calculated between the current position and the target position to obtain an attention weight; Based on a preset logarithmic space, performing numerical conversion processing on the attention weight; Based on the attention weights after numerical conversion, the output vector of the current position is weighted summed to obtain the final output weight vector; The improved attention mechanism module corresponding to the final output weight vector is deployed in the application model, and the final output result of the attention mechanism module in the application model is determined.

2. The method for improving the attention mechanism for long sequences according to claim 1, characterized in that: The attention scores of the input sequence of the attention mechanism module in the Transformer architecture are calculated for the previous and next positions to obtain the attention score of the current position, including: according to , get the attention score of the current position j ;in, represents the query vector at position j, Represents the key vector at position i, where d is the vector dimension.

3. The method for improving the attention mechanism for long sequences according to claim 1, characterized in that: The attention score is converted into a probability through a nonlinear activation function to obtain the breakpoint probability, which specifically includes: Through nonlinear activation function, and according to , and obtain the breakpoint probability ;in, represents the attention score of the current position j, For mathematical symbols.

4. The method for improving the attention mechanism for long sequences according to claim 1, characterized in that: According to the breakpoint probability, the attention between the current position and the target position is calculated to obtain the attention weight, which specifically includes: according to , get the attention weight ;in, is the breakpoint probability, and also represents the initial attention probability assigned to position i by position j; It represents the probability that the current position j does not allocate attention to all the intermediate positions k, where the intermediate position k is the position between the current position j and position i.

5. The method for improving the attention mechanism for long sequences according to claim 1, characterized in that: Based on a preset logarithmic space, the attention weight is numerically converted, specifically including: according to , get the attention weight The logarithmic form of ;in, represents the attention score of the current position j, and the middle position k is the position between the current position j and the position i; is a mathematical operation symbol; is the softplus function Numerical stabilization of attention scores.

6. The method for improving the attention mechanism for long sequences according to claim 1, characterized in that: Based on the attention weights after numerical conversion, the output vector of the current position is weighted and summed to obtain the final output weight vector, which includes: according to , get the final output weight vector ;in, is the attention weight; is the value vector at position i.

7. The method for improving the attention mechanism for long sequences according to claim 1, characterized in that: Determine the final output result of the attention mechanism module in the application model, specifically including: according to , get the final output result of the attention mechanism module in the application model ;in, is the output projection matrix; h is the output vector of multi-head attention; Calculate the concatenation of the output vectors.

8. The method for improving the attention mechanism for long sequences according to claim 1, characterized in that: Deploying the improved attention mechanism module corresponding to the final output weight vector to the application model specifically includes: Based on the stick-breaking segmentation attention mechanism included in the final output weight vector, the improved attention mechanism module is generated; wherein the input sequence in the improved attention mechanism module includes the query vector, the key vector and the value vector after the linear transformation matrix projection; By using a preset cross entropy loss function, an optimizer, and a cosine learning rate decay strategy, the improved attention mechanism module is subjected to an associated deployment process related to the application model to obtain a new application model; Based on the improved attention mechanism module in the new application model, the final output result of the attention mechanism module in the task data is obtained.

9. An improved device for attention mechanism for long sequences, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor so that the at least one processor can execute an improved method for an attention mechanism for long sequences according to any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that: The storage medium is a non-volatile computer-readable storage medium, which stores at least one program, each of which includes instructions, and when the instructions are executed by the terminal, the terminal executes an improved method for the attention mechanism for long sequences according to any one of claims 1-8.

Images