Dynamic memory enhancement method, system and device of self-attention network and medium

Abstract

The invention relates to the field of deep learning, and provides a dynamic memory enhancement method, system and device for a self-attention network and a medium, and the method comprises the steps: obtaining a to-be-processed input sequence, and calculating an average value of the input sequence; combining the input sequence average value on a preset memory matrix to obtain an updated memory matrix; converting the updated memory matrix, and fusing the converted memory matrix into a value of an input sequence to obtain an output sequence; and performing output projection on the output sequence. According to the method, the cross-sequence long-term dependency relationship can be better captured. By integrating the memory matrix into the causal self-attention mechanism, the model can utilize a wider range of context information, including historical data, thereby improving the performance in language understanding and generation tasks. According to the dynamic updating strategy of the memory matrix, limited computing resources can be effectively utilized, the common gradient disappearance or explosion problem in long sequence processing is avoided, and the stability and efficiency of the model are improved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a method, system, device and medium for dynamic memory enhancement of a self-attention network. Background Art

[0002] In recent years, the self-attention mechanism has attracted widespread attention in the field of natural language processing, especially the introduction of the Transformer architecture, which has completely changed the traditional way of sequence modeling. Although traditional recurrent neural networks (RNNs) and their variants, such as long short-term memory networks (LSTMs) and gated recurrent units (GRUs), perform well in processing sequence data, they are limited by their inherent serial processing characteristics and are difficult to parallelize, resulting in low training efficiency. In contrast, the self-attention mechanism allows the model to focus on different positions in the sequence at the same time, greatly accelerating the training process and improving the model's ability to capture long-distance dependencies.

[0003] When processing sequence data with a clear order, such as language generation tasks, the traditional self-attention mechanism has an obvious flaw: it allows the word at each position to access the information of its context at the same time. However, in some scenarios, such as generative tasks, the leakage of future information will destroy the learning process of the model. To solve this problem, the causal self-attention mechanism was proposed, which introduces a mask to limit the model to only see the current position and the previous position, thereby avoiding information leakage and ensuring the rationality of the generation process.

[0004] Although causal self-attention has achieved remarkable results in many tasks, the model still faces the challenge of capturing long-distance dependencies when processing very long sequences. This is because as the sequence length increases, the computational complexity and memory requirements of the self-attention mechanism will also grow significantly, making model training difficult. In addition, traditional causal self-attention mechanisms tend to rely too much on the recent context and ignore potential connections further away.

[0005] Current technical solutions often need to compromise between computational efficiency, memory consumption, and model performance when processing long sequences. This compromise is particularly evident when the model needs to capture dependencies spanning hundreds or even thousands of time steps. Summary of the invention

[0006] In order to solve the above problems, the present invention aims to enhance the causal self-attention mechanism of the GPT model based on the Transformer architecture, by introducing a trainable memory matrix and a set of mechanisms specifically used for dynamic updating of the memory matrix, thereby significantly improving the model's contextual understanding and generation capabilities when processing long sequence data.

[0007] Based on the above objectives, the present invention proposes a dynamic memory enhancement method for a self-attention network, comprising: Get the input sequence to be processed and calculate the average value of the input sequence; Combining the input sequence average value with a preset memory matrix to obtain an updated memory matrix; The updated memory matrix is ​​transformed and merged into the value of the input sequence to obtain an output sequence; The output sequence is output-projected.

[0008] In some embodiments, the step of obtaining an input sequence to be processed and calculating the average value of the input sequence includes: Get the input sequence to be processed; Linearly transform the input sequence into input keys and input values; The input sequence mean is obtained by averaging along the sequence dimension of the input sequence.

[0009] In some embodiments, the step of combining the input sequence average value with a preset memory matrix to obtain an updated memory matrix includes: Converting all the input sequence averages into update signals; Averaging the update signal to obtain an average signal; The average signal is accumulated and updated into the memory matrix to obtain an updated memory matrix.

[0010] In some embodiments, the step of converting the updated memory matrix and fusing it into the value of the input sequence to obtain the output sequence includes: Expand the updated memory matrix to the current batch size and transform it into a shape similar to the input key and input value; Performing attention calculation on the transformed memory matrix to obtain a memory attention matrix; The memory attention matrix and the input value are concatenated to obtain an output sequence.

[0011] In some embodiments, the step of concatenating the memory attention matrix and the input value to obtain an output sequence comprises: Converting the memory attention matrix into a memory key and a memory value; A memory attention weight is obtained by performing a dot product calculation on the memory key and the average signal; Optimizing the memory attention weight and applying it to the memory value to obtain a weighted memory value; The weighted memory value is matched to the length of the input sequence and concatenated with the input value to obtain an output sequence.

[0012] In some embodiments, the step of outputting the output sequence includes: Mapping the output sequence to the dimension of the output space through output projection to obtain an output result; The output result is subjected to residual connection and regularization and then outputted.

[0013] In some embodiments, the steps of constructing the preset memory matrix are as follows: Define an expandable memory matrix; The size of the memory matrix is ​​determined by the number of memory slots required and the embedding dimension.

[0014] The present invention proposes a dynamic memory enhancement system of a self-attention network, comprising: An acquisition unit, configured to acquire an input sequence to be processed and calculate an average value of the input sequence; an updating unit, configured to combine the input sequence average value with a preset memory matrix to obtain an updated memory matrix; a fusion unit configured to transform the updated memory matrix and fuse it into the value of the input sequence to obtain an output sequence; An output unit is configured to output the output sequence.

[0015] The present invention provides a computer device, comprising: At least one processor; and a memory, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps of the dynamic memory enhancement method of a self-attention network are performed.

[0016] The present invention proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the dynamic memory enhancement method of a self-attention network are executed.

[0017] The present invention has at least the following beneficial technical effects: The present invention proposes a dynamic memory enhancement method, system, device and medium for a self-attention network, the method comprising: obtaining an input sequence to be processed and calculating an average value of the input sequence; combining the average value of the input sequence with a preset memory matrix to obtain an updated memory matrix; converting the updated memory matrix and fusing it into the value of the input sequence to obtain an output sequence; and outputting the output sequence through output projection.

[0018] The present invention is used to enhance the GPT model under the Transformer architecture, especially in the performance when processing long sequence data. By introducing a trainable memory matrix and a set of mechanisms dedicated to the dynamic update of the memory matrix into the model, this scheme enables the model to better capture and utilize historical information and improve the ability to understand long-distance dependencies. The memory matrix can be dynamically updated according to the input sequence. This feature allows the model to be adaptable and flexible when processing different sequences and can better capture long-term dependencies across sequences. By integrating the memory matrix into the causal self-attention mechanism, the model can utilize a wider range of contextual information, including historical data, thereby improving performance in language understanding and generation tasks. The dynamic update strategy of the memory matrix can effectively utilize limited computing resources, avoid the common gradient vanishing or exploding problems in long sequence processing, and improve the stability and efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A flow chart of a dynamic memory enhancement method for a self-attention network provided by the present invention; Figure 2 A module diagram of a dynamic memory enhancement system of a self-attention network provided by the present invention; Figure 3 A structural diagram of the original causal self-attention mechanism of an embodiment of a dynamic memory enhancement method of a self-attention network provided by the present invention; Figure 4 A structural diagram of a causal self-attention mechanism for enhancing memory in one embodiment of a dynamic memory enhancement method for a self-attention network provided by the present invention; Figure 5 A detailed flow chart of a causal self-attention mechanism for enhancing memory in one embodiment of a dynamic memory enhancement method for a self-attention network provided by the present invention; Figure 6 A schematic diagram of the structure of an embodiment of a computer device provided by the present invention; Figure 7 A schematic diagram of the structure of an embodiment of a computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0022] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are for distinguishing two non-identical entities with the same name or non-identical parameters. It can be seen that "first" and "second" are only for the convenience of expression and should not be understood as limitations on the embodiments of the present invention. The subsequent embodiments will not explain this one by one.

[0023] This paper proposes a dynamic memory enhancement method for self-attention network, please refer to Figure 1 ,include: S1: Get the input sequence to be processed and calculate the average value of the input sequence; S2: combining the input sequence average value with a preset memory matrix to obtain an updated memory matrix; S3: transform the updated memory matrix and merge it into the value of the input sequence to obtain an output sequence; S4: Perform output projection on the output sequence.

[0024] The present invention provides an enhanced causal self-attention mechanism, which is specially designed to improve the ability to capture long-term and short-term dependencies in sequence modeling tasks, such as text generation, translation and understanding in natural language processing (NLP). By introducing a trainable memory matrix and a dedicated attention mechanism for dynamically updating this memory matrix, the present invention significantly improves the performance of the model when processing complex sequence data.

[0025] The key is the introduction of a trainable memory matrix that acts as a "long-term memory" for the model, capable of storing and updating important features learned from previous sequences. This allows the model to rely not only on immediate contextual information when processing new inputs, but also on patterns extracted from historical data, thereby enhancing the handling of long-range dependencies.

[0026] In order to make the update of the memory matrix more efficient and targeted, a set of attention-based update mechanisms was further developed. This mechanism determines which memory entries need to be updated and the extent of the update by calculating the correlation between the input sequence and the memory matrix. In this way, the model can selectively update the memory matrix during each forward propagation process, avoiding unnecessary computational overhead and ensuring that the memory matrix can continue to adapt and reflect changes in the input data.

[0027] In some embodiments, see Figure 1 , the step of obtaining the input sequence to be processed and calculating the average value of the input sequence includes: Get the input sequence to be processed; Linearly transform the input sequence into input keys and input values; The input sequence mean is obtained by averaging along the sequence dimension of the input sequence.

[0028] The sequence data of the input sequence is converted into a matrix format. If it is represented as a two-dimensional tensor, the first dimension is the batch size and the second dimension is the sequence length. dim=1 means that when performing operations on a two-dimensional tensor, such as sum, average, maximum value, etc., the operation is performed along the second dimension (i.e., the sequence length dimension). Calculating the average along the second dimension can help extract the key feature information of each row (or each sample). These statistics can be used as feature representations of samples. Compared to processing the entire tensor, operating along the sequence dimension can significantly reduce the number of elements that need to be processed. This not only saves computing resources, but also speeds up calculations and improves overall processing efficiency. Since some columns may contain noise or outliers, such operations can smooth the effects of these noise and outliers to a certain extent, making the results more stable and reliable.

[0029] In some embodiments, see Figure 1 The step of combining the input sequence average value with a preset memory matrix to obtain an updated memory matrix comprises: Converting all the input sequence averages into update signals; Averaging the update signal to obtain an average signal; The average signal is accumulated and updated into the memory matrix to obtain an updated memory matrix.

[0030] An additional linear layer (memory_update_layer) and an attention mechanism are used to determine how to update the memory matrix based on the current input sequence.

[0031] By calculating the average representation of the input sequence as the query (memory_queries), that is, the vector of average signals, interacting with the keys (memory_keys) in the memory matrix to calculate the attention weights (attn_weights_memory), thereby guiding the update direction and magnitude of the memory matrix.

[0032] By accumulating and averaging the signals, the random noise and outliers in the original signals can be smoothed out, making the data in the memory matrix more stable and reliable. This helps reduce incorrect decisions or analysis biases caused by noise interference. By accumulating and averaging, the excessive dependence of the memory matrix on specific data can be reduced to a certain extent, thereby reducing the risk of overfitting. This allows the memory matrix to maintain good capabilities when applied to new data or unknown scenarios.

[0033] In some embodiments, see Figure 1 , the step of converting the updated memory matrix and fusing it into the value of the input sequence to obtain the output sequence comprises: Expand the updated memory matrix to the current batch size and transform it into a shape similar to the input key and input value; Performing attention calculation on the transformed memory matrix to obtain a memory attention matrix; The memory attention matrix and the input value are concatenated to obtain an output sequence.

[0034] The memory matrix is ​​converted into keys and values ​​(k_memory, v_memory) and integrated into the standard causal attention calculation process, allowing the model to not only pay attention to the information in the current sequence, but also extract relevant information from the memory matrix.

[0035] In sequence processing tasks, the current value of the input sequence is often closely related to past historical information. The memory matrix can capture and store this historical information. By fusing it into the current input, the model can better learn the long-term dependencies in the sequence, thereby improving the generalization ability on unseen sequences. Since the conversion and fusion process of the memory matrix does not directly depend on the specific length of the input sequence, this method has good flexibility in processing sequences of different lengths, which helps to improve the model's adaptability to sequences of different lengths. The information stored in the memory matrix can be used multiple times without having to be recalculated every time a new input is processed. This helps to reduce the waste of computing resources and improve the computational efficiency of the model. By using the memory matrix as an intermediate layer to fuse historical information and current input, the overall structure of the model can be simplified, making the model more compact and efficient.

[0036] In some embodiments, see Figure 1 , the step of concatenating the memory attention matrix and the input value to obtain an output sequence comprises: Converting the memory attention matrix into a memory key and a memory value; A memory attention weight is obtained by performing a dot product calculation on the memory key and the average signal; Optimizing the memory attention weight and applying it to the memory value to obtain a weighted memory value; The weighted memory value is matched to the length of the input sequence and concatenated with the input value to obtain an output sequence.

[0037] By calculating the average representation of the input sequence as the query (memory_queries), the query is converted into a vector to obtain the average query vector (q_memory) of the input sequence, that is, the average signal, which is used to calculate the interaction with the memory matrix and obtain the weighted memory value (weighted_memory), which is then expanded and integrated into the value of the current sequence, enriching the representation ability of the model.

[0038] The memory attention matrix captures the relationship between elements in the input sequence, while the input value directly reflects the information of the element itself. The model can consider both the original input information and the information processed by the attention mechanism. The attention matrix determines which parts of the information should be paid more attention to by dynamically calculating the relationship between elements in the input sequence. Splicing this part of information with the original input value helps the model to focus on important information more accurately and ignore irrelevant information in subsequent processing. When processing long sequence data, traditional models often have difficulty capturing long-range dependencies. The attention mechanism helps the model capture such long-range dependencies by dynamically calculating the interactions between elements in the sequence. Splicing the attention matrix with the input value can further enhance the model's ability to process long sequence data. In multimodal learning tasks (such as image-to-text generation, video-to-speech conversion, etc.), the splicing of the memory attention matrix and the input value helps the model better integrate information from different modalities. This helps the model respond more accurately and coherently in complex multimodal environments. In a multi-head attention mechanism, each head generates an attention matrix. Concatenating these attention matrices with the corresponding input values ​​and further fusing them through linear transformation can further improve the representation ability and performance of the model.

[0039] In some embodiments, see Figure 1 , the step of performing output projection on the output sequence comprises: Mapping the output sequence to the dimension of the output space through output projection to obtain an output result; The output result is subjected to residual connection and regularization and then outputted.

[0040] Output projection is often used to map high-dimensional output sequences to low-dimensional space, which helps reduce the amount of computation and improve efficiency. Residual connections allow the network to selectively ignore certain input information during training while retaining other important information. This mechanism helps improve information utilization. Regularization also helps improve stability. It makes the parameter distribution more uniform, avoids extreme values ​​or singular points, and makes the output more stable and reliable.

[0041] In some embodiments, see Figure 1 , the steps of constructing the preset memory matrix are as follows: Define an expandable memory matrix; The size of the memory matrix is ​​determined by the number of memory slots required and the embedding dimension.

[0042] A trainable memory matrix is ​​embedded in the model, which acts as an external memory storage that can be dynamically adjusted as the input sequence is updated. This allows the model to not only rely on the current input sequence, but also utilize previously learned information, enhancing the capture of long-distance dependencies.

[0043] This paper proposes a dynamic memory enhancement system of self-attention network, please refer to Figure 2 ,include: The acquisition unit 100 is configured to acquire an input sequence to be processed and calculate an average value of the input sequence; An updating unit 200 is configured to combine the input sequence average value with a preset memory matrix to obtain an updated memory matrix; A fusion unit 300 is configured to transform and fuse the updated memory matrix into the value of the input sequence to obtain an output sequence; The output unit 400 is configured to output the output sequence for projection.

[0044] The present invention is used to enhance the GPT model under the Transformer architecture, especially in the performance when processing long sequence data. By introducing a trainable memory matrix and a set of mechanisms dedicated to the dynamic update of the memory matrix into the model, this scheme enables the model to better capture and utilize historical information and improve the ability to understand long-distance dependencies. The memory matrix can be dynamically updated according to the input sequence. This feature allows the model to be adaptable and flexible when processing different sequences and can better capture long-term dependencies across sequences. By integrating the memory matrix into the causal self-attention mechanism, the model can utilize a wider range of contextual information, including historical data, thereby improving performance in language understanding and generation tasks. The dynamic update strategy of the memory matrix can effectively utilize limited computing resources, avoid the common gradient vanishing or exploding problems in long sequence processing, and improve the stability and efficiency of the model.

[0045] In some embodiments, see Figure 3 and Figure 4 In the original causal self-attention mechanism, the matrix is ​​input to perform matrix attention calculation and then updated and output. This has the problems mentioned in the background technology. In the present invention, a memory matrix is ​​introduced, matrix self-attention calculation is performed on both the memory matrix and the input matrix, and then the matrix is ​​updated. Finally, the updated memory matrix and output matrix are obtained respectively, which can better capture the long-term dependencies across sequences.

[0046] In some embodiments, see Figure 5 , the present invention enhances the model's ability to capture long-term dependencies when processing sequence data, and details how to perform the memory matrix update process in each forward propagation step: Initialize the memory matrix: During the model initialization phase, a trainable memory matrix memory_matrix is ​​defined, whose size is memory_size x n_embd. Among them, memory_size represents the number of memory slots, and n_embd is the embedding dimension, that is, the vector representation dimension of each element in the model.

[0047] Sequence average calculation: In each forward propagation step, we first calculate the average seq_avg of the input sequence x. This is done by averaging the sequence dimension dim=1, and the resulting shape is (B, C), where B is the batch size and C is the embedding dimension.

[0048] Update the memory matrix: Use the sequence average seq_avg to generate the update signal memory_update through self.memory_update_layer, and the signal shape is also (B, C). Then, average this update signal and accumulate it to memory_matrix to realize the dynamic update of the memory matrix.

[0049] Convert memory matrix to keys and values: The updated memory_matrix is ​​expanded to the same size as the current batch and reshaped to a shape similar to the keys and values ​​in the input sequence. This allows the memory matrix to participate in attention calculations by converting it to the form of keys k_memory and values ​​v_memory.

[0050] Attention weight calculation: The memory attention weight is calculated by performing a dot product operation using the average query vector q_memory of the input sequence and the memory key k_memory. These weights are optimized for numerical stability and processed by the softmax function to ensure the rationality of the weight distribution.

[0051] Weighted memory value application: The attention weights are applied to the memory value v_memory to obtain the weighted memory value weighted_memory. These weighted memory values ​​are then expanded or cropped to match the length of the input sequence and concatenated with the value v of the original sequence, increasing the amount of information the model has when processing the sequence.

[0052] Output projection: Finally, the output y of the attention mechanism is passed through the output projection layer self.c_proj, after which residual connection and dropout operations are applied to increase the generalization ability of the model and prevent overfitting.

[0053] This dynamic update mechanism of the memory matrix significantly improves the model's ability to handle long-term dependencies in sequence data, especially when dealing with long sequence tasks with complex structures.

[0054] Based on the same inventive concept, according to another aspect of the present invention, Figure 6 As shown, an embodiment of the present invention further provides a computer device 30, which includes a processor 310 and a memory 320. The memory 320 stores a computer program 321 that can be run on the processor. When the processor 310 executes the program, the steps of the above method are performed.

[0055] Based on the same inventive concept, according to another aspect of the present invention, Figure 7 As shown, an embodiment of the present invention further provides a computer-readable storage medium 40, which stores a computer program 410 for executing the above method when executed by a processor.

[0056] The embodiment of the present invention may also include a corresponding computer device. The computer device includes a memory, at least one processor, and a computer program stored in the memory and executable on the processor, and the processor executes any one of the above methods when executing the program.

[0057] The memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules in the embodiments of the present application. The processor executes various functional applications and data processing of the device by running the non-volatile software programs, instructions and modules stored in the memory, that is, implementing the above method.

[0058] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In an embodiment, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the local module via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0059] Finally, it should be noted that a person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium of the program can be a disk, an optical disk, a read-only storage memory (ROM) or a random access memory (RAM), etc. The above-mentioned computer program embodiments can achieve the same or similar effects as the corresponding above-mentioned arbitrary method embodiments.

[0060] It will also be appreciated by those skilled in the art that various exemplary logic blocks, modules, circuits and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, a general description has been given to the functions of various schematic components, blocks, modules, circuits and steps. Whether this function is implemented as software or hardware depends on specific applications and the design constraints imposed on the entire system. Those skilled in the art can implement the function in various ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the disclosed scope of the embodiments of the present invention.

[0061] The above are exemplary embodiments disclosed in the present invention, but it should be noted that various changes and modifications may be made without departing from the scope of the embodiments disclosed in the present invention as defined in the claims. The functions, steps and / or actions of the method claims according to the disclosed embodiments described herein do not need to be performed in any particular order. The serial numbers of the embodiments disclosed in the above embodiments of the present invention are for description only and do not represent the advantages and disadvantages of the embodiments. In addition, although the elements disclosed in the embodiments of the present invention may be described or required in individual form, they may also be understood as multiple unless explicitly limited to the singular.

[0062] It should be understood that, as used herein, the singular forms "a", "an" are intended to include the plural forms as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" refers to any and all possible combinations including one or more of the associated listed items.

[0063] A person skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the disclosure of the embodiments of the present invention (including the claims) is limited to these examples; under the concept of the embodiments of the present invention, the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes in different aspects of the above embodiments of the present invention, which are not provided in detail for the sake of simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.

Claims

1. A dynamic memory enhancement method for a self-attention network, characterized in that: include: Get the input sequence to be processed and calculate the average value of the input sequence; Combining the input sequence average value with a preset memory matrix to obtain an updated memory matrix; The updated memory matrix is ​​transformed and merged into the value of the input sequence to obtain an output sequence; The output sequence is output-projected.

2. A method for dynamic memory enhancement of a self-attention network according to claim 1, characterized in that: The step of obtaining an input sequence to be processed and calculating the average value of the input sequence comprises: Get the input sequence to be processed; Linearly transform the input sequence into input keys and input values; The input sequence mean is obtained by averaging along the sequence dimension of the input sequence.

3. A method for dynamic memory enhancement of a self-attention network according to claim 2, characterized in that: The step of combining the input sequence average value with a preset memory matrix to obtain an updated memory matrix comprises: Converting all the input sequence averages into update signals; Averaging the update signal to obtain an average signal; The average signal is accumulated and updated into the memory matrix to obtain an updated memory matrix.

4. The method for dynamic memory enhancement of a self-attention network according to claim 3, characterized in that: The step of converting the updated memory matrix and fusing it into the value of the input sequence to obtain the output sequence comprises: Expand the updated memory matrix to the current batch size and transform it into a shape similar to the input key and input value; Performing attention calculation on the transformed memory matrix to obtain a memory attention matrix; The memory attention matrix and the input value are concatenated to obtain an output sequence.

5. A method for dynamic memory enhancement of a self-attention network according to claim 4, characterized in that: The step of concatenating the memory attention matrix and the input value to obtain an output sequence comprises: Converting the memory attention matrix into a memory key and a memory value; A memory attention weight is obtained by performing a dot product calculation on the memory key and the average signal; Optimizing the memory attention weight and applying it to the memory value to obtain a weighted memory value; The weighted memory value is matched to the length of the input sequence and concatenated with the input value to obtain an output sequence.

6. The method for dynamic memory enhancement of a self-attention network according to claim 1, characterized in that: The step of performing output projection on the output sequence comprises: Mapping the output sequence to the dimension of the output space through output projection to obtain an output result; The output result is subjected to residual connection and regularization and then outputted.

7. The method for dynamic memory enhancement of a self-attention network according to claim 1, characterized in that: The steps of constructing the preset memory matrix are as follows: Define an expandable memory matrix; The size of the memory matrix is ​​determined by the number of memory slots required and the embedding dimension.

8. A dynamic memory enhancement system for a self-attention network, characterized in that: include: An acquisition unit, configured to acquire an input sequence to be processed and calculate an average value of the input sequence; an updating unit, configured to combine the input sequence average value with a preset memory matrix to obtain an updated memory matrix; a fusion unit configured to transform the updated memory matrix and fuse it into the value of the input sequence to obtain an output sequence; An output unit is configured to output the output sequence.

9. A computer device comprising: at least one processor; and a memory storing a computer program executable on the processor, wherein the processor executes the steps of a dynamic memory enhancement method for a self-attention network as described in any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a dynamic memory enhancement method for a self-attention network as described in any one of claims 1 to 7 are performed.

Images