Dual-channel attention memory enhancement system v1.0

Abstract

The application discloses a double-channel attention memory enhancement system and method, and belongs to the technical field of artificial intelligence. The application captures the context dependency relationship inside the current input sequence and the association with the historical information in the external memory bank through the parallel short-time attention channel and long-time attention channel; the dynamic fusion module is used for adaptively weighting and fusing the outputs of the two channels, wherein the adaptive fusion adopts neural network dynamic calculation to calculate the position-by-position fusion weight; the learnable gating unit is arranged at the output end of the long-time attention channel, and is initialized with a negative value to control the inflow intensity of the historical information; the memory bank management module performs average pooling compression on the early memory exceeding the threshold value, so that the memory bank is prevented from unlimited growth. The application realizes the real-time parallel interaction of the current context and the historical memory, solves the problems of information forgetting, unstable training and memory bank expansion in long sequence processing, and can be widely applied to intelligent dialogue, text generation and other scenes requiring long-term memory.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a language model memory enhancement system and method based on a dual-channel attention mechanism, applicable to natural language processing tasks that require processing long sequence data, such as intelligent dialogue, text generation, machine translation, etc. Background Technology

[0002] In recent years, language models based on the Transformer architecture have achieved great success in the field of natural language processing. However, the standard Transformer model uses a fixed-length context window, which makes it easy to forget early information when processing long dialogues or documents, resulting in the model's inability to effectively utilize historical information. To address this issue, researchers have proposed various memory enhancement methods, such as Transformer-XL recursively concatenating historical hidden states at the fragment level, and MemoryNetworks storing historical information through external memory modules. However, these methods have the following shortcomings: 1. Transformer-XL uses a splicing method to process historical information. As the sequence grows, the computational load increases dramatically, and the method of integrating historical and current information is relatively simple. 2. Memory Networks typically employ a serial query mechanism, processing the current input before querying the memory, which cannot achieve real-time parallel fusion; 3. Existing methods lack a dynamic compression mechanism for memory management, which can easily lead to the memory growing indefinitely and exceeding hardware resource limitations; 4. Historical information in the early stages of training may introduce noise, leading to unstable model training.

[0003] Therefore, a more efficient, flexible and stable memory enhancement scheme is needed to improve the model's ability to model long sequences. Summary of the Invention

[0004] This invention aims to solve the above problems by providing a dual-channel attention memory enhancement system and method. By setting up short-term and long-term attention channels in parallel and introducing a dynamic fusion module and a learnable gating mechanism, the long-term memory capability of the model is effectively improved. At the same time, the memory bank size is controlled by a memory bank compression strategy. Technical solution

[0005] A dual-channel attention memory enhancement system, comprising: - Short-term attention channel: Employs a self-attention mechanism to capture the contextual dependencies within the current input sequence and output a short-term contextual representation; - Long-term attention channel: Employs a cross-attention mechanism to enable the current input sequence to interact with historical information in the external memory bank, recall relevant information, and output a long-term memory representation; the output end of the long-term attention channel is equipped with a learnable gating unit to control the inflow intensity of historical information; - Dynamic fusion module: Receives short-term context representation and long-term memory representation, and generates a fusion representation according to a preset fusion strategy; the fusion strategy includes additive fusion, gated fusion, or adaptive fusion, wherein adaptive fusion can dynamically calculate the position-by-position fusion weights through a small neural network; - Memory Management Module: Used to store feature representations of historical fragments. When the memory size exceeds a preset threshold, the earliest historical memory is compressed to prevent the memory from growing indefinitely.

[0006] 1. Input the current input sequence into the short-term attention channel and the long-term attention channel respectively; 2. The short-time attention channel performs self-attention operations on the input sequence to obtain a short-time contextual representation; 3. The long-term attention channel performs cross-attention operations on the input sequence and the external memory bank, and outputs a long-term memory representation through a learnable gating unit; 4. The dynamic fusion module fuses short-term contextual representations and long-term memory representations to obtain a fused representation; 5. Use the fused representation for subsequent network layers or tasks, while updating the encoding result of the current input sequence to the memory. Beneficial effects

[0007] Compared with the prior art, the present invention has the following advantages: 1. Parallel dual-channel architecture: Short-term and long-term attention channels are processed in parallel to achieve real-time interaction between the current context and historical memory, improving the timeliness of information fusion; 2. Learnable gating mechanism: A learnable gating mechanism is set at the exit of the long-term channel and initialized with a negative value. In the early stage of training, the inflow of historical information is suppressed, and the gating mechanism is automatically opened as training progresses, which solves the problem of training instability. 3. Adaptive position-by-position fusion: The fusion weights for each position are dynamically calculated based on short-term and long-term representations using a small neural network, making the fusion process more flexible and precise; 4. Memory bank compression strategy: Average pooling compression is applied to early memories that exceed the threshold, which preserves key information while controlling the size of the memory bank, making it suitable for online processing of long sequences; 5. High versatility: It can be seamlessly embedded into various Transformer models and is suitable for a variety of tasks such as dialogue systems, text generation, and document understanding. Attached Figure Description

[0008] Figure 1 This is a diagram illustrating the overall architecture of the dual-channel attention memory enhancement system of the present invention.

[0009] Figure 2 This is a flowchart of the memory bank update and compression process of the present invention. Figure 3 is a diagram of the internal structure of a single layer of the present invention, including layer normalization, short-term attention channel, long-term attention channel, fusion module, residual connection and feedforward network. Detailed Implementation

[0010] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. Example

[0011] This embodiment provides a dual-channel attention memory enhancement system implemented in PyTorch. Its core modules include: ShortTermAttention, LongTermAttention, DualChannelAttentionFusion, and MemoryAugmentedTransformer. The code implementation details are as follows.

[0012] The short-term attention channel employs a standard multi-head self-attention mechanism to capture the contextual relationships within the current input sequence. Its input is a tensor of shape (batch_size, seq_len, embed_dim), and its output is a short-term context representation and attention weights of the same shape. The implementation must ensure that embed_dim is divisible by num_heads.

[0013] The long-term attention channel employs a cross-attention mechanism, allowing the current input sequence (as a query) to interact with an external memory (as keys and values). This module supports sparse attention (top-k), focusing only on the k most relevant historical vectors in the memory to reduce computational cost. Furthermore, a learnable gating parameter, `gate`, is set at the output. After sigmoid activation, it is multiplied by the attention output and initialized to a negative value (e.g., -3.0), ensuring the gating is close to 0 during the initial training phase, thus suppressing historical information.

[0014] The dynamic fusion module provides three fusion methods: - Addition fusion: Directly add the short-time output to the long-time output, with a fixed fusion weight of (0.5, 0.5); - Gated fusion: Learn two scalar gating systems, pass them through a sigmoid function, and normalize them to serve as fusion weights; - Adaptive fusion: The fusion weights are dynamically calculated using a small neural network (two linear layers + ReLU + Softmax). If per_token=True, the network receives the concatenated short-time and long-time representations ([short_out, long_out]) and outputs the two-dimensional weights for each position; if per_token=False, the average weights are first calculated over the sequence dimensions, then the global weights are calculated and broadcast to all positions.

[0015] The overall model consists of multiple stacked dual-channel attention layers, each containing short-term attention, long-term attention, a fusion module, and a feedforward network. The model employs residual connections and layer normalization. The memory management module calls the `update_memory_bank` function after each forward propagation, appending the output of the current segment (after detaching) to the end of the memory bank. If the memory bank size exceeds `max_memory_size` and compression is enabled, the earliest memories are compressed using average pooling (compression ratio `compress_ratio`), retaining the compressed vector and any remaining portions, and pruning it to the maximum size.

[0016] In a specific instance, the following hyperparameters are set: - Vocabulary size vocab_size = 10000 - Embedding dimension: embed_dim = 256 - Number of attention heads num_heads = 4 - Number of layers num_layers = 2 - Maximum memory size max_memory_size = 100 - The fusion method is set to 'adaptive', and per_token_fusion = True. - Enable memory compression (memory_compress = True), compression ratio (compress_ratio = 2) - Long-term attention sparse top-k = 20 - Gating initialization value: gate_init = -3.0 During training, long dialogues are input into the model in chunks (chunk_size=50) sequentially, with the memory passed between chunks. The optimizer is Adam, with a learning rate of 1e-4 and gradient clipping of 1.0. Cross-entropy loss is calculated for each chunk and backpropagated.

[0017] Experiments show that the model using the method of this invention outperforms the baseline model in terms of long text perplexity, dialogue consistency, and other metrics, and the training process is stable with the memory size always kept within a preset range.

[0018] This invention can be integrated as a core memory enhancement component into various dialogue systems or text generation services. Users send dialogue history via API, and the system internally maintains a memory bank for each session, updating it in real time and generating responses using dual-channel attention. Because the memory bank supports compression, the system can run for extended periods without exhausting its memory.

[0019] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A dual-channel attention-memory enhancement system, characterized in that, include: The short-term attention channel module is used to perform self-attention operations on the current input sequence, capture the contextual dependencies within the current input, and generate short-term attention output; The long-term attention channel module is used to perform cross-attention operation between the current input sequence and the external memory bank, and to use a learnable gating unit to adjust the output of the cross-attention to generate a long-term attention output; The dynamic fusion module is used to perform weighted fusion of the short-term attention output and the long-term attention output to generate a fused attention representation; The memory management module is used to concatenate the representation of the current input sequence into the external memory, and to compress the earliest memory when the size of the external memory exceeds a preset threshold. The output prediction module is used to map the fused attention representation to the prediction result.

2. The system according to claim 1, characterized in that, The short-time attention channel module includes: The first query projection unit is used to project the input sequence into a query matrix; The first key projection unit is used to project the input sequence into a key matrix; The first value projection unit is used to project the input sequence into a value matrix; The first attention calculation unit is used to calculate the self-attention score and output based on the query matrix, key matrix, and value matrix; The first output projection unit is used to perform a linear transformation on the self-attention output.

3. The system according to claim 1, characterized in that, The long-term attention channel module includes: The second query projection unit is used to project the current input sequence into a query matrix; The second key projection unit is used to project the external memory bank into a key matrix; The second value projection unit is used to project the external memory into a value matrix; The second attention calculation unit is used to calculate the cross-attention score and output based on the query matrix, key matrix, and value matrix; The learnable gating unit includes a trainable parameter initialized to a negative value, which is used to gating the cross-attention output through the sigmoid function.

4. The system according to claim 3, characterized in that, The long-term attention channel module also includes: The sparse attention unit is used to retain the k positions with the highest attention scores through a top-k selection mechanism when the external memory bank size exceeds a preset threshold or when sparse mode is enabled, and the remaining positions are set to negative infinity.

5. The system according to claim 1, characterized in that, The dynamic fusion module includes an adaptive fusion unit, which comprises: The feature splicing subunit is used to splice the short-term attention output and the long-term attention output along the feature dimension; A weight prediction network is used to generate fusion weights for the two channels based on the concatenated features. The weighted summation subunit is used to perform a weighted summation of the short-term attention output and the long-term attention output based on the generated fusion weights.

6. The system according to claim 5, characterized in that, The weight prediction network supports at least one of the following two modes: Global mode generates the same fusion weights for all positions in the input sequence; The position-by-position mode generates fusion weights for each position in the input sequence.

7. The system according to claim 1, characterized in that, The memory bank management module includes: The memory splicing unit is used to splice the representation of the current input sequence into the historical memory bank; The memory compression unit is used to compress the earliest part of the memory when the memory size exceeds the preset maximum memory length. The memory truncation unit is used to retain the most recent memory vector when the compressed memory still exceeds the maximum memory length.

8. A dual-channel attention memory enhancement method based on the system according to any one of claims 1 to 7, characterized in that, Includes the following steps: Step A: Perform a short-time self-attention operation on the current input sequence to generate a short-time attention output; Step B: Perform cross-attention operation on the current input sequence and the external memory bank, and use learnable gating to adjust the output of cross-attention to generate long-term attention output; Step C: Weight and fuse the short-term attention output and the long-term attention output to generate a fused attention representation; Step D: Concatenate the representation of the current input sequence into the external memory bank, and compress the earliest memory when the size of the external memory bank exceeds a preset threshold; Step E: Map the fused attention representation to the final prediction result.

9. The method according to claim 8, characterized in that, Step B further includes: Step B1: Project the current input sequence into a query matrix; Step B2: Project the external memory into a key matrix and a value matrix; Step B3: Calculate the attention scores for the query matrix and the key matrix; Step B4: When sparse mode is enabled and the size of the external memory exceeds a preset threshold, retain the top-k positions with the highest attention scores, and set the remaining positions to negative infinity; Step B5: Perform softmax normalization on the attention scores to obtain the attention weights; Step B6: Perform a weighted summation of the value matrix based on the attention weights to obtain the cross-attention output; Step B7: Adjust the cross-attention output by initializing it with a learnable gating system that is negative.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the dual-channel attention memory enhancement method as described in claim 8 or 9.

Images