Two-channel attention memory enhancing system V1.0

By using a dual-channel attention-memory enhancement system, the problems of information forgetting and infinite memory growth in the Transformer model during long sequence data processing are solved, realizing real-time information interaction and stable long-term memory, which is suitable for natural language processing tasks.

CN122114000APending Publication Date: 2026-05-29吴清昊
0 Cites 0 Cited by

Patent Information

Application Number
CN202610274852.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-03-09
Publication Date
2026-05-29

AI Technical Summary

Technical Problem

Existing Transformer models suffer from problems such as early information forgetting, increased computational cost, infinite growth of the memory bank, and unstable training when processing long sequence data.

Method used

It employs parallel short-term and long-term attention channels, combined with a dynamic fusion module and a memory bank compression strategy, to achieve real-time information interaction through self-attention and cross-attention mechanisms, control the inflow of historical information through a learnable gating mechanism, and control the size of the memory bank through a memory bank management module.

Benefits of technology

It improves the model's ability to model long sequences, achieves the timeliness and stability of information fusion, is applicable to a variety of natural language processing tasks, and has a controlled memory size, making it suitable for long-term operation.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention discloses a dual-channel attention memory enhancement system and method, and belongs to the technical field of artificial intelligence. According to the method, through a short-time attention channel and a long-time attention channel which are arranged in parallel, a context dependency relationship in a current input sequence and association with historical information in an external memory bank are respectively captured; performing adaptive weighted fusion on the output of the two channels through a dynamic fusion module, wherein the adaptive fusion adopts a neural network to dynamically calculate the fusion weight per position; a learnable gating unit is arranged at the output end of the long-time attention channel, and negative value initialization is adopted to control the inflow intensity of historical information; and the memory bank management module performs average pooling compression on the early memory exceeding the threshold value to prevent the memory bank from infinitely increasing. According to the method, real-time parallel interaction between the current context and the historical memory is realized, the problems of information forgetting, unstable training and memory bank expansion in long sequence processing are solved, and the method can be widely applied to intelligent dialogues, text generation and other scenes needing long-range memory.
Need to check novelty before this filing date? Find Prior Art