Attention model training method and device, equipment and medium

Abstract

The embodiment of the invention provides an attention model training method and device, equipment and a medium, and the method comprises the steps: setting the length of a corresponding attention window according to the level depth of an attention layer in an attention model training process, and setting the attention window to be in a mutual crossing state, in the process that the model processes the input sequence to generate the output sequence, the length of the attention window is dynamically adjusted according to the generated output sequence so as to completely cover the output sequence, a trained attention model is obtained, the length of the attention window is set in a stepped mode, the attention model is helped to capture context information step by step, and the training efficiency is improved. According to the method, the calculation efficiency is improved, so that the performance and accuracy of the model are improved, and crossed attention windows are arranged in the multi-head attention layer, so that the understanding of the model on a sequence context is enhanced, the adaptability of the model is improved, and the attention windows are dynamically adjusted to completely cover an output sequence so as to ensure the uniformity of output contents of the model.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a training method for an attention model, a training device for an attention model, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the field of natural language processing (NLP), attention mechanism models have become the mainstream method for processing various tasks. The model uses the attention mechanism to capture long-distance dependencies in the input sequence, thereby achieving significant performance improvements in tasks such as machine translation, text summarization, and question-answering systems.

[0003] However, existing attention models usually perform attention calculations on each sequence position relative to the entire sequence within a fixed attention window, which may lead to unnecessary calculations and affect the computational performance of the model. In addition, due to the limitations of the attention window, the global information of the sequence may not be obtained, and long-distance dependencies cannot be effectively captured, resulting in a decrease in the performance of the model. Summary of the invention

[0004] In view of the above problems, embodiments of the present invention provide a method, apparatus, device and medium for training an attention model to overcome the above problems or at least partially solve the above problems.

[0005] The embodiment of the present invention discloses a method for training an attention model, the method comprising:

[0006] Obtaining an initial attention model, wherein the initial attention model includes multiple multi-head attention layers;

[0007] Obtaining the hierarchical depth corresponding to the multi-head attention layer, and determining the attention window length for the multi-head attention layer according to the hierarchical depth;

[0008] Setting a crossed attention window in the multi-head attention layer according to the attention window length;

[0009] Obtain an input sequence, input the input sequence into the initial attention model for training, obtain an output sequence corresponding to the input sequence, and adjust the parameters of the attention window according to the output sequence until the attention window completely covers the output sequence in length, stop model training, and output a target attention model corresponding to the initial attention model.

[0010] Optionally, obtaining the layer depth corresponding to the multi-head attention layer, and determining the attention window length for the multi-head attention layer according to the layer depth, includes:

[0011] Obtain the total number of layers of the initial attention model and the number of layers where the multi-head attention layer is located;

[0012] Calculate according to the total number of layers of the initial attention model and the number of layers where the multi-head attention layer is located, to obtain the hierarchical depth corresponding to the multi-head attention layer;

[0013] The attention window length for the multi-head attention layer is determined according to the hierarchical depth, and the attention window length increases as the number of layers in which the multi-head attention layer is located increases.

[0014] Optionally, determining the attention window length for the multi-head attention layer according to the hierarchical depth includes:

[0015] According to the following formula (1), the total number of layers of the initial attention model and the number of layers where the multi-head attention layer is located are used to calculate and obtain the attention window length:

[0016]

[0017] Among them, w is the length of the attention window, l is the layer number of the multi-head attention layer, N is the total number of layers of the initial attention model, and k is a preset fixed value.

[0018] Optionally, the multi-head attention layer includes multiple attention heads, and setting a cross attention window in the multi-head attention layer according to the attention window length includes:

[0019] Dividing the multiple attention heads equally to obtain a first group of attention heads and a second group of attention heads;

[0020] Obtaining a difference in starting points of the attention windows for the first group of attention heads and the second group of attention heads;

[0021] The attention windows for the first group of attention heads and the second group of attention heads are set according to the attention window length and the difference between the attention window starting points.

[0022] Optionally, setting the attention windows for the first group of attention heads and the second group of attention heads according to the difference between the attention window length and the attention window starting point includes:

[0023] Get the attention window length w for the multi-head attention layer;

[0024] According to the length of the attention window, the difference between the starting points of the attention window is set to

[0025] According to the difference in the starting point of the attention window, the starting point position of the attention window of the first group of attention heads is set to zero, and the starting point of the attention window of the second group of attention heads is The attention window length is w.

[0026] Optionally, adjusting parameters of the attention window according to the output sequence includes:

[0027] Acquire the position information of the output sequence, and select an initial attention window according to the position information of the output sequence, wherein the initial attention window is an attention window that does not completely cover the output sequence in length;

[0028] The attention window length of the initial attention window is adjusted according to the position information of the output sequence to obtain a target attention window, wherein the target attention window completely covers the output sequence in length.

[0029] Optionally, selecting an initial attention window according to position information of the output sequence includes:

[0030] An attention window whose starting point position in the attention window is smaller than the starting point position of the output sequence and whose ending point position is smaller than the ending point position of the output sequence is used as the initial attention window.

[0031] The embodiment of the present invention further provides an optimization device for an attention model, the device comprising:

[0032] A model acquisition module, used to acquire an initial attention model, wherein the initial attention model includes multiple layers of multi-head attention layers;

[0033] An attention window length determination module, used to obtain the hierarchical depth corresponding to the multi-head attention layer, and determine the attention window length for the multi-head attention layer according to the hierarchical depth;

[0034] An attention window setting module, used for setting a cross attention window in the multi-head attention layer according to the attention window length;

[0035] An attention window adjustment module is used to obtain an input sequence, input the input sequence into the initial attention model for training, obtain an output sequence corresponding to the input sequence, and adjust the parameters of the attention window according to the output sequence until the attention window completely covers the output sequence in length, stop model training, and output a target attention model corresponding to the initial attention model.

[0036] Optionally, the attention window length determination module includes:

[0037] A layer number acquisition submodule is used to obtain the total number of layers of the initial attention model and the number of layers where the multi-head attention layer is located;

[0038] The window length calculation submodule is used to calculate according to the total number of layers of the initial attention model and the number of layers where the multi-head attention layer is located, to obtain the hierarchical depth corresponding to the multi-head attention layer; according to the hierarchical depth, the attention window length for the multi-head attention layer is determined, and the attention window length increases with the increase in the number of layers where the multi-head attention layer is located.

[0039] Optionally, the window length calculation submodule is specifically used to calculate the attention window length according to the following formula (1), using the total number of layers of the initial attention model and the number of layers where the multi-head attention layer is located:

[0040]

[0041] Among them, w is the length of the attention window, l is the layer number of the multi-head attention layer, N is the total number of layers of the initial attention model, and k is a preset fixed value.

[0042] Optionally, the multi-head attention layer includes multiple attention heads, and the attention window setting module includes:

[0043] A grouping submodule, used to equally divide the multiple attention heads to obtain a first group of attention heads and a second group of attention heads;

[0044] A starting point difference determination module, used to obtain the starting point difference of the attention window for the first group of attention heads and the second group of attention heads;

[0045] A window setting submodule is used to set the attention windows for the first group of attention heads and the second group of attention heads according to the difference between the attention window length and the attention window starting point.

[0046] Optionally, the starting point difference determination module is specifically used to obtain the attention window length w for the multi-head attention layer; according to the attention window length, the starting point difference of the attention window is set to

[0047] The window setting submodule is specifically used to set the starting position of the attention window of the first group of attention heads to zero according to the starting point difference of the attention window, and the starting point of the attention window of the second group of attention heads is The attention window length is w.

[0048] Optionally, the attention window adjustment module includes:

[0049] A window selection submodule, used to obtain the position information of the output sequence, and select an initial attention window according to the position information of the output sequence, wherein the initial attention window is an attention window that does not completely cover the output sequence in length;

[0050] The window adjustment submodule is used to adjust the attention window length of the initial attention window according to the position information of the output sequence to obtain a target attention window, wherein the target attention window completely covers the output sequence in length.

[0051] Optionally, the window selection submodule is specifically used to use an attention window whose starting point position in the attention window is smaller than the starting point position of the output sequence and whose ending point position is smaller than the ending point of the output sequence as the initial attention window.

[0052] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0053] The memory is used to store computer programs;

[0054] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0055] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0056] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0057] In an embodiment of the present invention, during the training process of the attention model, the length of the corresponding attention window is set according to the hierarchical depth of the attention layer, and the attention windows are set to a cross-linked state. In addition, during the process of the model processing the input sequence to generate an output sequence, the length of the attention window is dynamically adjusted according to the generated output sequence to completely cover the output sequence, thereby obtaining a trained attention model, thereby achieving a step-by-step setting of the length of the attention window, which helps the attention model to gradually capture contextual information and improve computational efficiency to enhance the performance and accuracy of the model. In addition, setting cross-attention windows in the multi-head attention layer helps to enhance the model's understanding of the sequence context and enhance the model's adaptability, and dynamically adjust the attention window to completely cover the output sequence to ensure the uniformity of the model output content. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 is a flowchart of a method for optimizing an attention model provided by an embodiment of the present invention;

[0060] Figure 2 It is a structural block diagram of an optimization device for an attention model provided by an embodiment of the present invention;

[0061] Figure 3 is a block diagram of an electronic device provided in an embodiment of the present invention;

[0062] Figure 4 is a schematic diagram of a computer-readable storage medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] As an example, existing attention mechanism models usually perform attention calculations on each sequence position relative to the entire sequence within a fixed attention window. This fixed attention window is not suitable for all sequences, especially when the sequence length varies greatly. The fixed attention window may cause waste or insufficiency of computing resources, thereby causing unnecessary calculations and affecting the computational performance of the attention model. In addition, due to the limitation of the window, the global information of the sequence may not be obtained, and the long-distance dependencies cannot be effectively captured, resulting in the performance degradation of the model when processing complex sequences.

[0065] In this regard, the advantages of the present invention over the prior art are that, in the process of training the attention model, the length of the corresponding attention window is set according to the hierarchical depth of the attention layer, and the attention windows are set to a cross-linked state, and in the process of the model processing the input sequence to generate an output sequence, the length of the attention window is dynamically adjusted according to the generated output sequence to completely cover the output sequence, thereby obtaining a trained attention model, thereby realizing a step-by-step setting of the length of the attention window, which helps the attention model to gradually capture contextual information and improve computational efficiency to enhance the performance and accuracy of the model, and setting cross-attention windows in the multi-head attention layer helps to enhance the model's understanding of the sequence context and enhance the model's adaptability, and dynamically adjust the attention window to completely cover the output sequence to ensure the uniformity of the model output content.

[0066] Reference Figure 1 , shows a flowchart of a method for optimizing an attention model provided in an embodiment of the present invention, which may specifically include the following steps:

[0067] Step 101: Obtain an initial attention model, where the initial attention model includes multiple layers of multi-head attention layers;

[0068] In an embodiment of the present invention, first, an initial attention model is selected or constructed to provide a starting point for the training process, so as to facilitate the subsequent further parameter adjustment and optimization of the initial attention model to obtain the target attention model. The initial attention model includes at least multiple layers of multi-head attention layers. Specifically, the initial attention model is a model that adopts an attention mechanism, such as a Seq2Seq model (Sequence-to-Sequence Model) or a Transformer architecture model, but the present invention is not limited to this.

[0069] Step 102: Obtain the level depth corresponding to the multi-head attention layer, and determine the attention window length for the multi-head attention layer according to the level depth;

[0070] In an embodiment of the present invention, the length of the attention window, also referred to as the size of the attention window, is used to characterize the range that the attention window can process in a sequence. Specifically, an embodiment of the present invention determines the hierarchical depth of each layer by analyzing the hierarchical structure of the multi-head attention layer in the initial attention model, and then sets the length of the attention window of each layer according to the different hierarchical depths. By determining the length of the attention window according to the hierarchical structure of the model, the attention model can adopt different attention windows at different levels to capture contextual information of different ranges, thereby improving the model's ability to understand the sequence context and enhancing the model's expressiveness.

[0071] In a preferred embodiment of the present invention, in step 102, obtaining the hierarchical depth corresponding to the multi-head attention layer, and determining the attention window length for the multi-head attention layer according to the hierarchical depth, includes:

[0072] Obtain the total number of layers of the initial attention model and the number of layers where the multi-head attention layer is located;

[0073] Calculate according to the total number of layers of the initial attention model and the number of layers where the multi-head attention layer is located, to obtain the hierarchical depth corresponding to the multi-head attention layer;

[0074] The attention window length for the multi-head attention layer is determined according to the hierarchical depth, and the attention window length increases as the number of layers in which the multi-head attention layer is located increases.

[0075] In an embodiment of the present invention, the relative position of the multi-head attention layer in the initial attention model, that is, the hierarchical depth, can be determined according to the number of layers where the multi-head attention layer is located and the total number of layers of the initial attention model. Then, the attention window length for the multi-head attention layer is determined according to the relative position of the multi-head attention layer in the initial attention model. As the relative position of the multi-head attention layer deepens, the corresponding attention window length also increases. That is, the attention window length increases with the increase in the number of layers where the multi-head attention layer is located. By setting the length of each layer of the attention window in the attention model in a step-by-step manner, the attention model uses a smaller attention window at a low number of layers and a larger attention window at a high number of layers. As the number of layers increases, the attention window is gradually expanded. While ensuring computational efficiency, the model's coverage of sequence information can be gradually increased, which helps the attention model to gradually capture contextual information, improve computational efficiency, and avoid unnecessary calculations.

[0076] In a preferred embodiment of the present invention, the attention window length is obtained by calculating the total number of layers of the initial attention model and the number of layers where the multi-head attention layer is located according to the following formula (1):

[0077]

[0078] Among them, w is the length of the attention window, l is the layer number of the multi-head attention layer, N is the total number of layers of the initial attention model, and k is a preset fixed value.

[0079] In an embodiment of the present invention, the attention window length w corresponding to the multi-head attention layer is determined by performing a square root operation on the ratio of the number of layers l in which the multi-head attention layer is located to the total number of layers N of the initial attention model, and then multiplying it by a preset fixed value k and rounding it up. This allows a smaller attention window to be set at a low level of the attention model to reduce the amount of calculation, and a larger attention window to be set at a high level of the attention model to capture a wider range of contextual information, thereby optimizing the allocation of computing resources and enabling the attention model to gradually expand its scope of attention, thereby improving the model's adaptability, computing resource allocation efficiency, and contextual understanding ability.

[0080] Among them, the preset fixed value k is used as a regulating factor to affect the growth rate of the attention window length. By adjusting the value of k, the change range of the attention window length can be controlled, thereby affecting the context capture ability of the model. Optionally, the preset fixed value k is determined by expert experience and can be fixed to k=1000, or the optimal k value is selected by comprehensively considering the scale of the attention model, computing resources, task requirements, and experimental verification. To ensure that the model maintains the training effect while reducing the computational cost of attention in each layer, the present invention is not limited to this.

[0081] Step 103: setting a cross attention window in the multi-head attention layer according to the attention window length;

[0082] In an embodiment of the present invention, after determining the length of the attention window corresponding to the multi-head attention layer, a cross-attention window is further set in the multi-head attention layer according to the length of the attention window. By setting the cross-attention window in the multi-head attention layer, the attention window cross-covers the entire sequence, and information can flow in the entire sequence, so that the attention model can enhance the understanding of the sequence context, improve information coverage, enhance the adaptability of the attention model, promote information fusion, and improve the expressiveness of the attention model.

[0083] In a preferred embodiment of the present invention, the multi-head attention layer includes multiple attention heads, and step 103, setting a cross attention window in the multi-head attention layer according to the attention window length, includes:

[0084] Dividing the multiple attention heads equally to obtain a first group of attention heads and a second group of attention heads;

[0085] Obtaining a difference in starting points of the attention windows for the first group of attention heads and the second group of attention heads;

[0086] The attention windows for the first group of attention heads and the second group of attention heads are set according to the attention window length and the difference between the attention window starting points.

[0087] In an embodiment of the present invention, multiple attention heads in a multi-head attention layer are divided into two groups to obtain a first group of attention heads and a second group of attention heads. The difference between the starting points of the attention windows of the first group of attention heads and the second group of attention heads is set to characterize the relative distance between the starting positions of the sequences paid attention to by the two groups of attention heads. Then, according to the attention window length and the difference between the starting points of the attention windows, respective attention windows are set for the first group and the second group of attention heads to achieve a certain overlap in the attention windows corresponding to the two groups of attention heads. In other words, the parts of the sequences paid attention to by the two groups of attention heads will overlap. By setting the crossed attention windows, different attention heads can pay attention to different parts of the sequence, which is helpful for information fusion between different levels, so that the model can capture the local details and global structure of the sequence at the same time, thereby improving the performance of the model.

[0088] In a preferred embodiment of the present invention, setting the attention windows for the first group of attention heads and the second group of attention heads according to the difference between the attention window length and the attention window starting point includes:

[0089] Get the attention window length w for the multi-head attention layer;

[0090] According to the length of the attention window, the difference between the starting points of the attention window is set to

[0091] According to the difference in the starting point of the attention window, the starting point position of the attention window of the first group of attention heads is set to zero, and the starting point of the attention window of the second group of attention heads is The attention window length is w.

[0092] In an embodiment of the present invention, the starting point difference of the attention windows corresponding to the two groups of attention heads can be determined to be half the window length according to the attention window length corresponding to the multi-head attention layer, and the starting point position of the attention window of the first group of attention heads is set to zero, that is, the first group of attention heads starts to pay attention from the beginning position of the sequence, and the range of attention is the attention window of length w, then the starting point position of the attention window of the second group of attention heads is set to zero according to the starting point difference That is, the second group of attention heads starts to focus from the middle of the sequence, and the range of attention is also the attention window of length w. Each group of attention heads focuses on different parts of the sequence. Together, they can cover the entire input sequence more comprehensively, enhancing the model's adaptability to different sequence lengths and structures.

[0093] Step 104: obtaining an input sequence, inputting the input sequence into the initial attention model for training, obtaining an output sequence corresponding to the input sequence, and adjusting the parameters of the attention window according to the output sequence until the attention window completely covers the output sequence in length, stopping model training, and outputting a target attention model corresponding to the initial attention model;

[0094] In an embodiment of the present invention, input sequence data for training is obtained. Optionally, the input sequence can be a long text sequence. The input sequence is input into an initial attention model for training. In the process of the model generating a corresponding output sequence based on the input sequence, the attention window is dynamically adjusted to ensure that the output sequence can be completely covered by an attention window. When the attention window completely covers the output sequence, the training process is stopped, and the trained and adjusted target attention model is output. By dynamically adjusting the attention window to completely cover the output sequence, the model can take into account every element in the sequence, thereby ensuring the integrity of the output and the comprehensiveness of the information, which helps to ensure that the content output by the model does not omit important information. The optimized attention model can dynamically adjust the attention window to completely cover the output sequence, and then the optimized attention model can generate unified and high-quality output content in various sequence processing tasks, thereby improving the overall quality and reliability of the model.

[0095] In a preferred embodiment of the present invention, the step of adjusting parameters of the attention window according to the output sequence includes:

[0096] Acquire the position information of the output sequence, and select an initial attention window according to the position information of the output sequence, wherein the initial attention window is an attention window that does not completely cover the output sequence in length;

[0097] The attention window length of the initial attention window is adjusted according to the position information of the output sequence to obtain a target attention window, wherein the target attention window completely covers the output sequence in length.

[0098] In an embodiment of the present invention, during the process of model training of the initial attention model according to the output sequence, an attention window that does not completely cover the output sequence in length is selected and adjusted according to the position information of the output sequence, and is dynamically adjusted to a target attention window that completely covers the output sequence in length according to the position information of the output sequence. By completely covering the output sequence in length, the target attention window model can take into account the dependencies and contextual information in the output sequence, thereby improving the quality of the finally generated output sequence, thereby improving the performance and accuracy of the model.

[0099] In a preferred embodiment of the present invention, selecting an initial attention window according to the position information of the output sequence includes:

[0100] Taking an attention window in which the starting position is less than the starting position of the output sequence and the ending position is less than the ending position of the output sequence in the attention window as the initial attention window.

[0101] In an embodiment of the present invention, during the process of generating an output sequence, an initial attention window that first covers the output sequence but does not completely cover the output sequence can be selected according to the attention window and the starting position and terminal position of the output sequence for dynamic adjustment, so as to obtain a target attention window that completely covers the output sequence, and the unity of the model output content can be ensured through the target attention window.

[0102] As an example, the position of the attention window w p is (i, j), and the position of the already generated output sequence Y is (m, n). If i < m and j < n, it means that the attention window w p is an attention window that covers the output sequence but does not completely cover the output sequence. Taking the attention window w p as the initial attention window for dynamic adjustment, according to the position (m, n) of the output sequence, the length of the attention window w p is adjusted to n - i, and the starting position remains unchanged. Then the position of the attention window w p window is correspondingly adjusted to (i, n), and the attention window w p completely covers the output sequence Y.

[0103] In an embodiment of the present invention, during the training process of the attention model, by setting the length of the corresponding attention window according to the hierarchical depth of the attention layer, and setting the attention windows to be in an intersecting state, and during the process of the model processing the input sequence to generate the output sequence, dynamically adjusting the length of the attention window according to the already generated output sequence to completely cover the output sequence, a trained attention model is obtained, thereby realizing a stepped setting of the length of the attention window, which helps the attention model to gradually capture context information, improve computational efficiency, so as to improve the performance and accuracy of the model, and setting intersecting attention windows in the multi-head attention layer helps to enhance the model's understanding of the sequence context, enhance the adaptability of the model, and dynamically adjust the attention window to completely cover the output sequence to ensure the unity of the model output content.

[0104] As an example, the optimized attention model, namely the target attention model, can be applied to the process of long text summary generation or long text question answering. In long text related tasks, the model needs to process a longer sequence. The long text sequence is input into the target attention model for processing to output the target sequence. By setting attention windows of different sizes at different layers of the target attention model, the model can more flexibly express context information at different levels in the sequence. For example, the target attention model uses a smaller attention window at the low layers of the model to capture local context information in the input sequence. As the number of layers increases, the attention window gradually expands, and the model can capture longer-distance dependencies and global context information. In addition, using a smaller attention window at the low layers can reduce the amount of calculation and improve the computational efficiency of the model. As the number of layers increases, the window is gradually expanded, which can gradually increase the model's coverage of sequence information while ensuring computational efficiency.

[0105] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0106] Reference Figure 2 , shows a structural block diagram of an optimization device for an attention model provided in an embodiment of the present invention, which may specifically include the following modules:

[0107] A model acquisition module 201 is used to acquire an initial attention model, wherein the initial attention model includes multiple layers of multi-head attention layers;

[0108] An attention window length determination module 202 is used to obtain the layer depth corresponding to the multi-head attention layer, and determine the attention window length for the multi-head attention layer according to the layer depth;

[0109] An attention window setting module 203, used for setting a cross attention window in the multi-head attention layer according to the attention window length;

[0110] The attention window adjustment module 204 is used to obtain an input sequence, input the input sequence into the initial attention model for training, obtain an output sequence corresponding to the input sequence, and adjust the parameters of the attention window according to the output sequence until the attention window completely covers the output sequence in length, stop model training, and output a target attention model corresponding to the initial attention model.

[0111] In one embodiment of the present invention, the attention window length determination module 202 includes:

[0112] A layer number acquisition submodule is used to obtain the total number of layers of the initial attention model and the number of layers where the multi-head attention layer is located;

[0113] The window length calculation submodule is used to calculate according to the total number of layers of the initial attention model and the number of layers where the multi-head attention layer is located, to obtain the hierarchical depth corresponding to the multi-head attention layer; according to the hierarchical depth, the attention window length for the multi-head attention layer is determined, and the attention window length increases with the increase in the number of layers where the multi-head attention layer is located.

[0114] In one embodiment of the present invention, the window length calculation submodule is specifically used to calculate the attention window length according to the following formula (1), using the total number of layers of the initial attention model and the number of layers where the multi-head attention layer is located:

[0115]

[0116] Among them, w is the length of the attention window, l is the layer number of the multi-head attention layer, N is the total number of layers of the initial attention model, and k is a preset fixed value.

[0117] In one embodiment of the present invention, the multi-head attention layer includes multiple attention heads, and the attention window setting module 203 includes:

[0118] A grouping submodule, used to equally divide the multiple attention heads to obtain a first group of attention heads and a second group of attention heads;

[0119] A starting point difference determination module, used to obtain the starting point difference of the attention window for the first group of attention heads and the second group of attention heads;

[0120] A window setting submodule is used to set the attention windows for the first group of attention heads and the second group of attention heads according to the difference between the attention window length and the attention window starting point.

[0121] In one embodiment of the present invention, the starting point difference determination module is specifically used to obtain the attention window length w for the multi-head attention layer; according to the attention window length, the starting point difference of the attention window is set to

[0122] The window setting submodule is specifically used to set the starting position of the attention window of the first group of attention heads to zero according to the starting point difference of the attention window, and the starting point of the attention window of the second group of attention heads is The attention window length is w.

[0123] In one embodiment of the present invention, the attention window adjustment module 204 includes:

[0124] A window selection submodule, used to obtain the position information of the output sequence, and select an initial attention window according to the position information of the output sequence, wherein the initial attention window is an attention window that does not completely cover the output sequence in length;

[0125] The window adjustment submodule is used to adjust the attention window length of the initial attention window according to the position information of the output sequence to obtain a target attention window, wherein the target attention window completely covers the output sequence in length.

[0126] In one embodiment of the present invention, the window selection submodule is specifically used to use an attention window whose starting point position in the attention window is smaller than the starting point position of the output sequence and whose ending point position is smaller than the ending point of the output sequence as the initial attention window.

[0127] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0128] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 3 As shown, it includes a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.

[0129] Memory 303, used for storing computer programs;

[0130] The processor 301 is used to implement the method described in the above embodiment when executing the program stored in the memory 303.

[0131] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0132] The communication interface is used for communication between the above terminal and other devices.

[0133] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0134] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0135] like Figure 4 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 401 is further provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the method described in the above embodiment.

[0136] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0137] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0138] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0139] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A training method for an attention model, characterized in that: The method comprises: Obtaining an initial attention model, wherein the initial attention model includes multiple multi-head attention layers; Obtaining the hierarchical depth corresponding to the multi-head attention layer, and determining the attention window length for the multi-head attention layer according to the hierarchical depth; Setting a crossed attention window in the multi-head attention layer according to the attention window length; Obtain an input sequence, input the input sequence into the initial attention model for training, obtain an output sequence corresponding to the input sequence, and adjust the parameters of the attention window according to the output sequence until the attention window completely covers the output sequence in length, stop model training, and output a target attention model corresponding to the initial attention model.

2. The method according to claim 1, characterized in that The obtaining the level depth corresponding to the multi-head attention layer, and determining the attention window length for the multi-head attention layer according to the level depth, comprises: Obtain the total number of layers of the initial attention model and the number of layers where the multi-head attention layer is located; Calculate according to the total number of layers of the initial attention model and the number of layers where the multi-head attention layer is located, to obtain the hierarchical depth corresponding to the multi-head attention layer; The attention window length for the multi-head attention layer is determined according to the hierarchical depth, and the attention window length increases as the number of layers in which the multi-head attention layer is located increases.

3. The method according to claim 2, characterized in that The determining the attention window length for the multi-head attention layer according to the hierarchical depth includes: According to the following formula (1), the total number of layers of the initial attention model and the number of layers where the multi-head attention layer is located are used to calculate and obtain the attention window length: Among them, w is the length of the attention window, l is the layer number of the multi-head attention layer, N is the total number of layers of the initial attention model, and k is a preset fixed value.

4. The method according to claim 1, characterized in that The multi-head attention layer includes a plurality of attention heads, and the setting of a cross attention window in the multi-head attention layer according to the attention window length includes: Dividing the multiple attention heads equally to obtain a first group of attention heads and a second group of attention heads; Obtaining a difference in starting points of the attention windows for the first group of attention heads and the second group of attention heads; The attention windows for the first group of attention heads and the second group of attention heads are set according to the attention window length and the difference between the attention window starting points.

5. The method according to claim 4, characterized in that The step of setting the attention windows for the first group of attention heads and the second group of attention heads according to the attention window length and the difference between the attention window starting points includes: Get the attention window length w for the multi-head attention layer; According to the length of the attention window, the difference between the starting points of the attention window is set to According to the difference in the starting point of the attention window, the starting point position of the attention window of the first group of attention heads is set to zero, and the starting point of the attention window of the second group of attention heads is The attention window length is w.

6. The method according to claim 1, characterized in that The step of adjusting parameters of the attention window according to the output sequence includes: Acquire the position information of the output sequence, and select an initial attention window according to the position information of the output sequence, wherein the initial attention window is an attention window that does not completely cover the output sequence in length; The attention window length of the initial attention window is adjusted according to the position information of the output sequence to obtain a target attention window, wherein the target attention window completely covers the output sequence in length.

7. The method according to claim 6, characterized in that The selecting an initial attention window according to the position information of the output sequence comprises: An attention window whose starting point position in the attention window is smaller than the starting point position of the output sequence and whose ending point position is smaller than the ending point position of the output sequence is used as the initial attention window.

8. An optimization device for an attention model, characterized in that: The device comprises: A model acquisition module, used to acquire an initial attention model, wherein the initial attention model includes multiple layers of multi-head attention layers; An attention window length determination module, used to obtain the hierarchical depth corresponding to the multi-head attention layer, and determine the attention window length for the multi-head attention layer according to the hierarchical depth; An attention window setting module, used for setting a cross attention window in the multi-head attention layer according to the attention window length; An attention window adjustment module is used to obtain an input sequence, input the input sequence into the initial attention model for training, obtain an output sequence corresponding to the input sequence, and adjust the parameters of the attention window according to the output sequence until the attention window completely covers the output sequence in length, stop model training, and output a target attention model corresponding to the initial attention model.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

Images