Long-term dialogue memory compression and dialogue model adaptation method

Abstract

The invention discloses a long-term dialogue memory compression and dialogue model adaptation method, which comprises the following steps: S1, dialogue history compression based on a memory model: the memory model adopts a Transform architecture, so that memory update is carried out in a mode of redistributing attention distribution, and the dialogue history compression is carried out on the basis of the dialogue history compression; determining an attention distribution condition by combining the global self-attention between the memory and the cross attention of the new dialogue information and the memory; s2, memory perception adaptation of the dialogue model: on the premise of not changing original model parameters, accessing a cross attention linear mapping layer, and carrying out adaptation among different semantic spaces; s3, attention output calculation: final attention output is obtained in combination with self-attention and cross attention calculation results after adaptation; according to the method, in a dialogue scene, the dialogue content is compressed into the memory with the fixed size by using the compression model, so that the memory is dynamically updated along with the increase of dialogue rounds, the calculation complexity can be reduced, and the memory overhead is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of computer data processing technology, and specifically relates to a long-term dialogue memory compression and dialogue model adaptation method, which aims to reduce computational complexity while enhancing the model's processing capability for long context sequences. Background Art

[0002] The emergence of the Transformer architecture has revolutionized the field of natural language processing. Its unique self-attention mechanism enables the model to simultaneously focus on various positions in a sequence, effectively capturing long-range dependencies while processing sequential data in parallel. Its powerful and efficient sequence modeling capabilities have made it the dominant architecture for large language models (LLMs). This architecture, consisting of an encoder and a decoder, was initially used for tasks requiring both encoding and decoding, such as machine translation. With the development of language models, encoder-only architectures, exemplified by BERT, and decoder-only architectures, exemplified by GPT, have evolved. The encoder in the encoder-only architecture utilizes a global attention mechanism, making it naturally suited for tasks like summarization and often used as the structure for embedding models. The decoder-only architecture, on the other hand, demonstrates unique advantages in generation tasks, making it a dominant architecture for large models.

[0003] The decoder-only model uses causal masks to predict the next word in an autoregressive manner, making it a natural fit for sequence generation tasks. In conversational scenarios requiring dynamic processing of multiple conversation turns while maintaining contextual coherence, it can also effectively leverage key-value caching (KV Cache), reducing the computational complexity of each step from O(n²) to O(n), thereby accelerating generative inference efficiency. With the widespread application and rapid development of generative models in interactive scenarios such as dialogue systems, the demand for longer contexts is increasing, prompting further exploration of how to improve models' ability to handle long contexts. To overcome the limitations of context length, existing methods often build on key-value caching to further optimize memory usage and dynamic context management, such as KV Cache Eviction, sparse attention, and dynamic masking. In conversational scenarios, the accumulation of long conversation turns exacerbates the context length limitation. However, these methods still face challenges in enabling models to effectively understand and utilize long conversational histories while accurately tracking and responding to user intent and topic changes across multiple rounds of interaction. Summary of the Invention

[0004] To address the above issues, the present invention proposes a method for compressing long-term dialogue memory and adapting it to a dialogue model. In a dialogue scenario, this method uses a compression model to compress the dialogue content into a fixed-size memory, so that the memory is dynamically updated as the number of dialogue turns increases. This method enables the dialogue model to effectively utilize the relevant information in the compressed historical memory to generate answers during the dialogue process, rather than a historical key-value cache that grows in length as the number of dialogue turns increases. This can reduce computational complexity, alleviate memory overhead, and achieve a long-term dialogue system that can handle approximately infinite dialogue turns.

[0005] To achieve the above object, the present invention adopts the following technical solutions: A method for compressing long-term dialogue memory and adapting a dialogue model includes the following steps: S1. Conversation history compression based on the memory model: After a new round of conversation between the user and the conversation model, the conversation content is obtained and the memory model is used to compress the conversation history into memory. The memory model adopts the Transformer architecture, which updates memory by redistributing attention. The attention allocation is determined by combining the cross-attention between memory and new conversation information and the global self-attention between memories. S2. Memory-aware adaptation of the dialogue model: Without changing the original model parameters, a cross-attention linear mapping layer is introduced to adapt between different semantic spaces. S3. Attention output calculation: Combine the self-attention and adapted cross-attention calculation results to obtain the final attention output.

[0006] Preferably, the specific process of step S1 is: S11. During the round-by-round dialogue process, the KV memory matrix is ​​used to store memory information. The calculation formula is: , ,in, is the compressed KV memory matrix; is the key vector; is a value vector; For collection; is the size of the memory matrix; is the dimension of memory space; is the first a key vector of elements; is the first A value vector of elements; S12, dialogue embedding encoding: Use the sentence encoder to encode the input and output of the dialogue model to preprocess the subsequent memory storage and update operations. The calculation formula is: , ,in, User input, i.e., input to the dialogue model; is the dialogue response, i.e., the output of the dialogue model; Embed(·) is the encoding operation of the text-level sentence encoder; is the output dimension of the sentence encoder; The result of encoding the dialogue model input through the text-level sentence encoder; The output of the dialogue model is encoded by the text-level sentence encoder; S13. Attention scores and mapping: input via K key vectors and new dialogue models The associated matching between them calculates the corresponding attention score; K key vector and the input of the new dialogue model , value vector V and the output of the new dialogue model , after mapping, the output is calculated according to the attention distribution, and the calculation formula is: , , , , , , , , , , ,in, For Calculate the parameter matrix for memory self-attention query mapping; For Calculate the parameter matrix for memory self-attention key mapping; For Calculation parameter matrix for memory self-attention value mapping; For and Calculate the parameter matrix for the cross-attention query mapping between memory and new dialogue; For and Calculate the parameter matrix for the cross-attention key mapping between memory and new dialogue; For and Calculate the parameter matrix for mapping the cross-attention values ​​between memory and new dialogues; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; For Calculation parameter matrix for mapping; For Calculation parameter matrix for mapping; for through The result of the mapping; for through The result of the mapping; The score calculation method for combining self-attention and cross-attention is: , , , ,in, Mark for memory Memory Mark Attention weight distribution; For the indivual through The result of the mapping; Mark for memory Input to the conversation model Attention weight distribution; Mark for memory Memory Mark Attention score; Mark for memory Input to the conversation model Attention score; is the index value; S14. Memory update mechanism: Dynamically update memory based on the global memory's attention allocation to new conversations and memory capacity limitations.

[0007] Preferably, the calculation formula of step S14 is: The total amount of attention allocated by the global memory to the new dialogue is calculated as: ,in, The sum of the attention allocated to the new dialogue by the global memory; The sum of attention allocations to new dialogues according to global memory , the lower limit of global memory attention allocation to new dialogues and the upper limit of memory matrix size ,when , the global memory allocates enough attention to the new dialogue, and the attention output is carried out as usual. The calculation formula is: , ,in, for Attention output; For the indivual through The result of the mapping; for Attention output; For the indivual through The result of the mapping; when , the global memory does not pay enough attention to the new dialogue, but there is still space in the memory capacity. A new memory marker is added to store the missed information. The calculation formula is: , , ,in, The newly added memory mark The attention output of the key vector of elements; The newly added memory mark The attention output of the value vector of elements; M = M+1, where Indicates assignment; when , the global memory does not allocate enough attention to the new dialogue, but the memory capacity is full. The memory markers with the lowest relevance to the new content will make up for the insufficient attention and correspondingly reduce the attention to other information. The calculation formula is: , , ,in, The memory marker that pays the least attention to the new conversation; For Take the minimum value; For K The attention output of each memory token; For V The attention output of each memory token; Input to the dialogue model for memory identification Attention score; Mark for memory Memory Mark Attention score.

[0008] Preferably, the calculation formula of step S2 is: , , , , , , , , ,in, The first a logo; is the dimension size; For Calculate the parameter matrix for sequential self-attention query mapping; For Calculate the parameter matrix for sequential self-attention key mapping; For Calculation parameter matrix for sequence self-attention value mapping; For 、 and Calculate the parameter matrix for cross-attention query mapping between sequence and history memory; For 、 and Calculate the parameter matrix for cross-attention key mapping between sequence and history memory; For 、 and Calculate the parameter matrix for mapping the cross-attention values ​​between sequence and history memory; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping.

[0009] Preferably, the calculation formula of step S3 is: , , ,in, Sequence ID Sequence ID The attention weight of For the indivual through The result of the mapping; Sequence ID Sequence ID The attention weight of for Attention output.

[0010] After adopting the above technical solution, the present invention has the following beneficial effects: 1. The present invention can improve computing efficiency and performance. On the one hand, it can reduce computational complexity: with traditional methods, as the number of conversation turns increases, the context length continues to grow, and the computational complexity is high; however, the present invention uses a compression model to compress the conversation content into a fixed-size memory, so that the memory is dynamically updated as the number of conversation turns increases, rather than simply accumulating a key-value cache of historical sequence identifiers. This greatly reduces computational complexity, accelerates the efficiency of generating inferences, and greatly shortens the delay generated during inference, allowing the model to respond to user requests more quickly and improve the real-time and smoothness of the conversation system. On the other hand, it can reduce memory overhead: by limiting the capacity threshold of the memory matrix and designing a dynamic update mechanism, the memory is continuously updated and evolved within a limited dimension, avoiding the unlimited growth of memory usage due to the increase in conversation turns. This allows the model to process longer conversation histories with limited memory resources, improves memory utilization, and reduces the memory requirements for hardware devices, which is conducive to the deployment and operation of conversation systems in resource-constrained environments.

[0011] 2. The present invention can enhance the performance of the dialogue model. On the one hand, it can improve the coherence and accuracy of the dialogue: the memory model adopts the Transformer architecture, combining the cross-attention between memory and new dialogue information and the global self-attention between memories to determine the attention allocation. This mechanism enables the dialogue model to better capture the long-term dependencies in the dialogue, more accurately understand and utilize long-history dialogue information, so that when generating answers, it can more accurately track and respond to the user's intentions and topic changes in multiple rounds of interaction, generate more coherent and more in line with the dialogue logic of the answers, and improve the quality of the dialogue. On the other hand, it can achieve effective management of long-term memory: through the memory weight decay algorithm, the recent dialogue content is highlighted, while the rest of the irrelevant historical information is correspondingly downplayed. This enables the dialogue model to maintain attention on key information in long-term dialogues, effectively manage long-term memory, avoid the decline in dialogue quality due to memory information overload or irrelevant information interference, and achieve long-term effective modeling of dialogue context, so that the dialogue system can better cope with long-content dialogue scenarios.

[0012] 3. The present invention can improve the applicability and scalability of the model. On the one hand, it can be applied to a variety of dialogue scenarios: the present invention introduces a lightweight cross-attention linear mapping layer into the dialogue model to achieve memory perception, without changing the parameter structure of the original dialogue model. It has good versatility and compatibility, and can be easily adapted to different Decoder-Only dialogue models without large-scale structural adjustment and retraining of the model, reducing the difficulty and cost of model adaptation, and meeting the needs of the model's long context processing capabilities in different dialogue scenarios. On the other hand, it can be easily expanded and optimized: the dialogue history compression method based on the memory model uses a key-value pair representation to construct a flexibly scalable KV memory matrix. This flexible storage structure allows the memory model to be easily expanded and optimized according to actual needs. For example, the memory update mechanism can be further optimized, the capacity threshold of the memory matrix can be adjusted, etc., to meet the requirements of different dialogue tasks and scenarios, providing convenience for the continuous improvement and optimization of the dialogue system.

[0013] 4. The present invention can improve the user experience. On the one hand, it can support nearly unlimited conversation turns: by dynamically updating and compressing memory, the dialogue system can handle nearly unlimited conversation turns, avoiding situations where the model cannot handle or performance degradation occurs due to excessively long conversations. It provides support for users to explore deeper and more complex conversation topics, and improves the freedom and satisfaction of users interacting with the dialogue system. On the other hand, it can enhance the personalization and depth of conversations: by being able to effectively utilize conversation history information over a long period of time, the dialogue model can have a deeper understanding and memory of the user's interests, preferences, and conversation context, thereby generating more personalized answers that better meet user expectations, improving the depth and quality of conversations, and enhancing the emotional resonance and trust between users and the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Flowchart of the present invention; Figure 2 This is a schematic diagram of attention calculation for the memory model of the present invention; Figure 3 This is a memory-aware adaptation diagram of the conversation model of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0016] like Figure 1 As shown, a method for compressing long-term dialogue memory and adapting a dialogue model includes the following steps: S1. Conversation history compression based on the memory model: After a new round of conversation between the user and the conversation model, the conversation content is obtained and the memory model is used to compress the conversation history into memory. The memory model adopts the Transformer architecture, which updates memory by redistributing attention. The attention allocation is determined by combining the cross-attention between memory and new conversation information and the global self-attention between memories. The specific process of step S1 is: S11. During the round-by-round dialogue process, the KV memory matrix is ​​used to store memory information. The calculation formula is: , ,in, is the compressed KV memory matrix; is the key vector; is a value vector; For collection; is the size of the memory matrix; is the dimension of memory space; is the first in the compressed KV memory matrix a key vector of elements; is the first A value vector of elements; S12, dialogue embedding encoding: Use the sentence encoder to encode the input and output of the dialogue model to preprocess the subsequent memory storage and update operations. The calculation formula is: , ,in, User input, i.e., input to the dialogue model; is the dialogue response, i.e., the output of the dialogue model; Embed(·) is the encoding operation of the text-level sentence encoder; is the output dimension of the sentence encoder; The result of encoding the dialogue model input through the text-level sentence encoder; The output of the dialogue model is encoded by the text-level sentence encoder; S13. Attention scores and mapping: input via K key vectors and new dialogue models The associated matching between them calculates the corresponding attention score; K key vector and the input of the new dialogue model , value vector V and the output of the new dialogue model , after mapping, the output is calculated according to the attention distribution, and the calculation formula is: , , , , , , , , , , ,in, For Calculate the parameter matrix for memory self-attention query mapping; For Calculate the parameter matrix for memory self-attention key mapping; For Calculation parameter matrix for memory self-attention value mapping; For and Calculate the parameter matrix for the cross-attention query mapping between memory and new dialogue; For and Calculate the parameter matrix for the cross-attention key mapping between memory and new dialogue; For and Calculate the parameter matrix for mapping the cross-attention values ​​between memory and new dialogues; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; For Calculation parameter matrix for mapping; For Calculation parameter matrix for mapping; for through The result of the mapping; for through The result of the mapping; The score calculation method for combining self-attention and cross-attention is: , , , ,in, Mark for memory Memory Mark Attention weight distribution; For the indivual through The result of the mapping; Mark for memory Input to the conversation model Attention weight distribution; Mark for memory Memory Mark Attention score; Mark for memory Input to the conversation model Attention score; is the index value; S14, memory update mechanism: dynamically update memory based on the global memory's attention allocation to new conversations and memory capacity limitations; The calculation formula of step S14 is: The total amount of attention allocated by the global memory to the new dialogue is calculated as: ,in, The sum of the attention allocated to the new dialogue by the global memory; The sum of attention allocations to new dialogues according to global memory , the lower limit of global memory attention allocation to new dialogues and the upper limit of memory matrix size ,when , the global memory allocates enough attention to the new dialogue, and the attention output is carried out as usual. The calculation formula is: , ,in, for Attention output; For the indivual through The result of the mapping; for Attention output; For the indivual through The result of the mapping; when , the global memory does not pay enough attention to the new dialogue, but there is still space in the memory capacity. A new memory marker is added to store the missed information. The calculation formula is: , , ,in, The newly added memory mark The attention output of the key vector of elements; The newly added memory mark The attention output of the value vector of elements; M = M+1, where Indicates assignment; when , the global memory does not allocate enough attention to the new dialogue, but the memory capacity is full. The memory markers with the lowest relevance to the new content will make up for the insufficient attention and correspondingly reduce the attention to other information. The calculation formula is: , , ,in, The memory marker that pays the least attention to the new conversation; For Take the minimum value; For K The attention output of each memory token; For V The attention output of each memory token; Input to the dialogue model for memory identification Attention score; Mark for memory Memory Mark Attention score; S2. Memory-aware adaptation of the dialogue model: Without changing the original model parameters, a cross-attention linear mapping layer is introduced to adapt between different semantic spaces. The calculation formula of step S2 is: , , , , , , , , ,in, The first a logo; is the dimension size; For Calculate the parameter matrix for sequential self-attention query mapping; For Calculate the parameter matrix for sequential self-attention key mapping; For Calculation parameter matrix for sequence self-attention value mapping; For 、 and Calculate the parameter matrix for cross-attention query mapping between sequence and history memory; For 、 and Calculate the parameter matrix for cross-attention key mapping between sequence and history memory; For 、 and Calculate the parameter matrix for mapping the cross-attention values ​​between sequence and history memory; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; S3, attention output calculation: Combine the self-attention and adapted cross-attention calculation results to obtain the final attention output; The calculation formula of step S3 is: , , ,in, Sequence ID Sequence ID The attention weight of For the indivual through The result of the mapping; Sequence ID Sequence ID The attention weight of for Attention output.

[0017] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for compressing long-term dialogue memory and adapting dialogue models, characterized in that: The following steps are involved: S1. Conversation history compression based on the memory model: After a new round of conversation between the user and the conversation model, the conversation content is obtained and the memory model is used to compress the conversation history into memory. The memory model adopts the Transformer architecture, which updates memory by redistributing attention. The attention allocation is determined by combining the cross-attention between memory and new conversation information and the global self-attention between memories. S2. Memory-aware adaptation of the dialogue model: Without changing the original model parameters, a cross-attention linear mapping layer is introduced to adapt between different semantic spaces. S3. Attention output calculation: Combine the self-attention and adapted cross-attention calculation results to obtain the final attention output.

2. The method for compressing long-term dialogue memory and adapting dialogue models according to claim 1, characterized in that: The specific process of step S1 is: S11. During the round-by-round dialogue process, the KV memory matrix is ​​used to store memory information. The calculation formula is: , ,in, is the compressed KV memory matrix; is the key vector; is a value vector; For collection; is the size of the memory matrix; is the dimension of memory space; is the first a key vector of elements; is the first A value vector of elements; S12, dialogue embedding encoding: Use the sentence encoder to encode the input and output of the dialogue model to preprocess the subsequent memory storage and update operations. The calculation formula is: , ,in, User input, i.e., input to the dialogue model; is the dialogue response, i.e., the output of the dialogue model; Embed(·) is the encoding operation of the text-level sentence encoder; is the output dimension of the sentence encoder; The result of encoding the dialogue model input through the text-level sentence encoder; The output of the dialogue model is encoded by the text-level sentence encoder; S13. Attention scores and mapping: input via K key vectors and new dialogue models The association match between them is used to calculate the corresponding attention score; the K key vector and the input of the new dialogue model , value vector V and the output of the new dialogue model , after mapping, the output is calculated according to the attention distribution, and the calculation formula is: , , , , , , , , , , ,in, For Calculate the parameter matrix for memory self-attention query mapping; For Calculate the parameter matrix for memory self-attention key mapping; For Calculation parameter matrix for memory self-attention value mapping; For and Calculate the parameter matrix for the cross-attention query mapping between memory and new dialogue; For and Calculate the parameter matrix for the cross-attention key mapping between memory and new dialogue; For and Calculate the parameter matrix for mapping the cross-attention values ​​between memory and new dialogues; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; For Calculation parameter matrix for mapping; For Calculation parameter matrix for mapping; for through The result of the mapping; for through The result of the mapping; The score calculation method for combining self-attention and cross-attention is: , , , ,in, Mark for memory Memory Mark Attention weight distribution; For the indivual through The result of the mapping; Mark for memory Input to the conversation model Attention weight distribution; Mark for memory Memory Mark Attention score; Mark for memory Input to the conversation model Attention score; is the index value; S14. Memory update mechanism: Dynamically update memory based on the global memory's attention allocation to new conversations and memory capacity limitations.

3. The method for compressing long-term dialogue memory and adapting dialogue models according to claim 2, characterized in that: The calculation formula of step S14 is: The total amount of attention allocated by the global memory to the new dialogue is calculated as: ,in, The sum of the attention allocated to the new dialogue by the global memory; The sum of attention allocations to new dialogues according to global memory , the lower limit of global memory attention allocation to new dialogues and the upper limit of memory matrix size ,when , the global memory allocates enough attention to the new dialogue, and the attention output proceeds as usual. The calculation formula is: , ,in, for Attention output; For the indivual through The result of the mapping; for Attention output; For the indivual through The result of the mapping; when , the global memory does not pay enough attention to the new dialogue, but there is still space in the memory capacity. A new memory marker is added to store the missed information. The calculation formula is: , , ,in, The newly added memory mark The attention output of the key vector of elements; The newly added memory mark The attention output of the value vector of elements; M = M+1, where Indicates assignment; when , the global memory does not allocate enough attention to the new dialogue, but the memory capacity is full. The memory markers with the lowest relevance to the new content will make up for the insufficient attention and correspondingly reduce the attention to other information. The calculation formula is: , , ,in, The memory marker that pays the least attention to the new conversation; For Take the minimum value; For K The attention output of each memory token; For V The attention output of each memory token; Input to the dialogue model for memory identification Attention score; Mark for memory Memory Mark Attention score.

4. The method for compressing long-term dialogue memory and adapting dialogue models according to claim 1, wherein: The calculation formula of step S2 is: , , , , , , , , ,in, The first a logo; is the dimension size; For Calculate parameter matrix for sequential self-attention query mapping; For Calculate the parameter matrix for sequential self-attention key mapping; For Calculation parameter matrix for sequence self-attention value mapping; For 、 and Calculate the parameter matrix for cross-attention query mapping between sequence and history memory; For 、 and Calculate the parameter matrix for cross-attention key mapping between sequence and history memory; For 、 and Calculate the parameter matrix for mapping the cross-attention values ​​between sequence and history memory; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping.

5. The method for compressing long-term dialogue memory and adapting dialogue models according to claim 1, wherein: The calculation formula of step S3 is: , , ,in, Sequence ID Sequence ID The attention weight of For the indivual through The result of the mapping; Sequence ID Sequence ID The attention weight of for Attention output.

Images