A long dialogue memory compression and dialogue model adaptation method

Abstract

The application discloses a long-term dialogue memory compression and dialogue model adaptation method, comprising the following steps: S1, dialogue history compression based on a memory model: the memory model adopts a Transformer architecture, so that the updating of the memory is performed in the form of re-distribution of attention distribution, and the attention distribution is determined by combining cross-attention between the memory and new dialogue information and global self-attention between the memory; S2, memory perception adaptation of the dialogue model: under the premise that the original model parameters are not changed, a cross-attention linear mapping layer is accessed to adapt between different semantic spaces; S3, attention output calculation: the final attention output is obtained by combining self-attention and the calculation result of the adapted cross-attention; in the dialogue scene, the method uses a compression model to compress dialogue content into memory of a fixed size, so that the memory is dynamically updated with the increase of dialogue turns, the calculation complexity can be reduced, and the memory overhead can be reduced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer data processing, and particularly relates to a long dialogue memory compression and dialogue model adaptation method, aiming to reduce the computational complexity while enhancing the processing capacity of the model for long context sequences. BACKGROUND

[0002] The emergence of the Transformer architecture has brought major changes to the field of natural language processing. Its unique self-attention mechanism allows the model to focus on each position in the sequence simultaneously, effectively capturing long-range dependencies in the sequence while processing sequence data in parallel. The powerful and efficient sequence modeling capability makes it the mainstream architecture of current large language models (LLM). The architecture consists of an encoder (Encoder) and a decoder (Decoder), which was originally used for tasks such as machine translation that require encoding and decoding processes. With the development of language models, Encoder-Only architectures represented by BERT and Decoder-Only architectures represented by GPT have emerged. The encoder of the Encoder-Only architecture uses a global attention mechanism, which is naturally suitable for tasks such as summarization and is often used as an embedding model structure. The Decoder-Only architecture has unique advantages in generation tasks and has become one of the mainstream large model architectures.

[0003] Decoder-Only models use causal masks to predict the next word in a self-recursive manner, which is naturally compatible with sequence generation tasks. In dialogue scenarios that require dynamic processing of multiple rounds of dialogue while maintaining context coherence, KV Cache can be efficiently utilized to reduce the computational complexity from O(n²) to O(n), thereby accelerating the generation inference efficiency. With the widespread application and rapid development of generation models in interactive scenarios such as dialogue systems, the demand for context length is increasing, and how to improve the model's long context processing capability is further explored. To break through the limitations of context length, existing methods often further optimize memory usage and dynamic context management based on KV Cache, such as KV Cache Eviction, sparse attention, and dynamic masking. In dialogue scenarios, the accumulation of long content dialogue rounds exacerbates the problem of context length limitations, but these methods still face certain challenges in enabling the model to effectively understand and utilize long historical dialogue information while accurately tracking and responding to changes in user intent and topics in multiple rounds of interaction. SUMMARY

[0004] To solve the above problems, the application provides a long-term dialogue memory compression and dialogue model adaptation method, which uses a compression model to compress dialogue content into a fixed-size memory in a dialogue scenario, so that the memory is dynamically updated as the dialogue round increases; the dialogue model can effectively use the relevant information in the compressed historical memory to generate answers during the dialogue process, instead of the historical key-value cache which increases in length as the dialogue round increases, so as to reduce the computational complexity, reduce the memory overhead, and realize a long-term dialogue system that can approximately handle an infinite number of dialogue rounds.

[0005] To achieve the above object, the application adopts the following technical solutions:

[0006] A long-term dialogue memory compression and dialogue model adaptation method, comprising the following steps:

[0007] S1, dialogue history compression based on a memory model: obtaining dialogue content after a new round of dialogue between a user and a dialogue model, and compressing the dialogue history into memory using a memory model; the memory model adopts a Transformer architecture, so that the update of the memory is performed by reallocating attention distribution, and the attention distribution is determined by combining cross-attention between the memory and new dialogue information and global self-attention between the memory;

[0008] S2, memory perception adaptation of the dialogue model: under the premise of not changing the original model parameters, accessing a cross-attention linear mapping layer to adapt between different semantic spaces;

[0009] S3, attention output calculation: combining the self-attention and the cross-attention calculation result after adaptation to obtain the final attention output.

[0010] Preferably, the specific process of step S1 is:

[0011] S11, in the process of round-by-round dialogue, a KV memory matrix is used to store memory information, and the calculation formula is: , wherein, is the compressed KV memory matrix; is the key vector; is the value vector; is the set; is the size of the memory matrix; is the dimension of the memory space; is the key vector of the first element in the compressed KV memory matrix; is the value vector of the first element in the compressed KV memory matrix;

[0012] ​​S12, dialogue embedding encoding: Use the sentence encoder to encode the input and output of the dialogue model to preprocess the subsequent memory storage and update operations. The calculation formula is: , ,in, User input, i.e., input to the dialogue model; is the dialogue response, i.e., the output of the dialogue model; Embed(·) is the encoding operation of the text-level sentence encoder; is the output dimension of the sentence encoder; The result of encoding the dialogue model input through the text-level sentence encoder; The output of the dialogue model is encoded by the text-level sentence encoder;

[0013] S13. Attention scores and mapping: input via K key vectors and new dialogue models The association match between them is used to calculate the corresponding attention score; the K key vector and the input of the new dialogue model , value vector V and the output of the new dialogue model , after mapping, the output is calculated according to the attention distribution, and the calculation formula is: , , , , , , , , , , ,in, For Calculate the parameter matrix for memory self-attention query mapping; For Calculate the parameter matrix for memory self-attention key mapping; For Calculation parameter matrix for memory self-attention value mapping; For and Calculate the parameter matrix for the cross-attention query mapping between memory and new dialogue; For and Calculate the parameter matrix for the cross-attention key mapping between memory and new dialogue; For and Calculate the parameter matrix for mapping the cross-attention values ​​between memory and new dialogues; for through The result of the mapping; For After the result of mapping; For After the result of mapping; For After the result of mapping; For After the result of mapping; For After the result of mapping; For the calculation parameter matrix for mapping; For the calculation parameter matrix for mapping; For After the result of mapping; For After the result of mapping;

[0014] The score calculation method combining self-attention and cross-attention is: , , , wherein, is the memory identifier the attention weight distribution of the memory identifier ; is the th After the result of mapping; is the attention weight distribution of the memory identifier to the input of the dialogue model; is the attention score of the memory identifier to the memory identifier ; is the attention score of the memory identifier to the input of the dialogue model; is the index value;

[0015] S14, memory updating mechanism: dynamically updating the memory according to the attention distribution of the global memory to the new dialogue and the memory capacity limit.

[0016] Preferably, the calculation formula of step S14 is:

[0017] The calculation formula of the sum of the attention distribution of the global memory to the new dialogue is: ,in, The sum of the attention allocated to the new dialogue by the global memory;

[0018] The sum of attention allocations to new dialogues according to global memory , the lower limit of global memory attention allocation to new dialogues and the upper limit of memory matrix size ,when , the global memory allocates enough attention to the new dialogue, and the attention output is carried out as usual. The calculation formula is: , ,in, for Attention output; For the indivual through The result of the mapping; for Attention output; For the indivual through The result of the mapping;

[0019] when , the global memory does not pay enough attention to the new dialogue, but there is still space in the memory capacity. A new memory marker is added to store the missed information. The calculation formula is: , , ,in, The newly added memory mark The attention output of the key vector of elements; The newly added memory mark The attention output of the value vector of elements; M = M+1, where Indicates assignment;

[0020] when , the global memory does not allocate enough attention to the new dialogue, but the memory capacity is full. The memory markers with the lowest relevance to the new content will make up for the insufficient attention and correspondingly reduce the attention to other information. The calculation formula is: , , ,in, The memory marker that pays the least attention to the new conversation; For Take the minimum value; For K The attention output of each memory token; For V The attention output of each memory token; Input to the dialogue model for memorization Attention score; Mark for memory Memory Mark Attention score.

[0021] Preferably, the calculation formula of step S2 is: , , , , , , , , ,in, The first a logo; is the dimension size; For Calculate the parameter matrix for sequential self-attention query mapping; For Calculate the parameter matrix for sequential self-attention key mapping; For Calculation parameter matrix for sequence self-attention value mapping; For 、 and Calculate the parameter matrix for cross-attention query mapping between sequence and history memory; For 、 and Calculate the parameter matrix for cross-attention key mapping between sequence and history memory; For 、 and Calculate the parameter matrix for mapping the cross-attention values ​​between sequence and history memory; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping.

[0022] Preferably, the calculation formula of step S3 is: , , wherein, is a sequence identifier an attention weight for the sequence identifier ; is the result of the mapping of the th sequence identifier ; is a sequence identifier an attention weight for the sequence identifier ; is an attention output of .

[0023] After the above technical solution, the application has the following beneficial effects:

[0024] 1. The application can improve the calculation efficiency and performance. On the one hand, it can reduce the calculation complexity: the traditional method has high calculation complexity as the context length increases with the increase of the dialogue round. The application compresses the dialogue content into a fixed size memory by using a compression model, so that the memory is dynamically updated as the dialogue round increases, rather than simply accumulating the key-value cache of the historical sequence identifier, which greatly reduces the calculation complexity, speeds up the generation of inference efficiency, highly shortens the delay in inference, makes the model respond to user requests more quickly, and improves the real-time performance and smoothness of the dialogue system. On the other hand, it can reduce the memory overhead: by limiting the capacity threshold of the memory matrix and designing a dynamic updating mechanism, the memory is continuously updated and evolved within a limited dimension, avoiding the unlimited growth of memory occupation caused by the increase of dialogue round, so that the model can process longer dialogue history under limited memory resources, improve the utilization rate of memory, reduce the memory requirement of hardware devices, and be beneficial to deploy and run the dialogue system in resource-limited environment.

[0025] 2、The application can enhance the performance of the dialogue model. On the one hand, the coherence and accuracy of the dialogue can be improved: the memory model adopts the Transformer architecture, combines the cross-attention of memory and new dialogue information and the global self-attention between memory to determine the attention allocation, which can make the dialogue model better capture the long-term dependency in the dialogue, more accurately understand and use the long historical dialogue information, so as to more accurately track and respond to the user's intention and topic change in multiple rounds of interaction when generating answers, generate more coherent and more consistent with the dialogue logic answers, and improve the dialogue quality. On the other hand, the effective management of long-term memory can be realized: through the memory weight decay algorithm, the recent dialogue content is highlighted, and the rest of the irrelevant historical information is correspondingly faded, which enables the dialogue model to maintain attention on key information in long-term dialogue, effectively manages long-term memory, avoids the decline of dialogue quality caused by memory information overload or irrelevant information interference, realizes long-term effective modeling of dialogue context, and enables the dialogue system to better cope with long content dialogue scenarios.

[0026] 3、The application can improve the applicability and expansibility of the model. On the one hand, it can be applied to various dialogue scenarios: the application introduces a lightweight cross-attention linear mapping layer into the dialogue model to realize memory awareness, without changing the parameter structure of the original dialogue model, which has good universality and compatibility, can be easily adapted to different Decoder-Only dialogue models, without the need for large-scale structural adjustment and retraining of the model, reducing the difficulty and cost of model adaptation, and meeting the demand for long context processing ability of the model in different dialogue scenarios. On the other hand, it can be easily extended and optimized: the dialogue history compression method based on the memory model adopts a key-value pair representation to construct a flexible KV memory matrix, which makes the memory model easy to extend and optimize according to actual needs, such as further optimizing the memory update mechanism, adjusting the capacity threshold of the memory matrix, etc., to adapt to the requirements of different dialogue tasks and scenarios, and provides convenience for the continuous improvement and optimization of the dialogue system.

[0027] 4、The application can improve the user experience. On the one hand, it can support approximately infinite dialogue rounds: by dynamically updating and compressing memory, the dialogue system can handle approximately infinite dialogue rounds, avoiding the situation that the model cannot handle or performance declines due to long dialogue, providing support for users to explore deeper and more complex dialogue topics, and improving the degree of freedom and satisfaction of user interaction with the dialogue system. On the other hand, it can enhance the personalization and depth of the dialogue: it can effectively use the dialogue history information for a long time, enabling the dialogue model to have a deeper understanding and memory of the user's interests, preferences and dialogue background, so as to generate more personalized answers that meet the user's expectations, improve the depth and quality of the dialogue, and enhance the emotional resonance and trust between the user and the system. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 Flowchart of the present application;

[0029] Figure 2 Schematic diagram of memory model attention calculation of the present application;

[0030] Figure 3 Memory perception adaptation diagram of the dialogue model of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0032] As shown in Figure 1 , a long-term dialogue memory compression and dialogue model adaptation method comprises the following steps:

[0033] S1, dialogue history compression based on memory model: after a new round of dialogue between the user and the dialogue model, the dialogue content is obtained, and the dialogue history is compressed into memory using the memory model; the memory model adopts the Transformer architecture, so that the update of the memory is performed by reallocating the attention distribution, and the attention distribution is determined by combining the cross-attention between the memory and the new dialogue information and the global self-attention between the memories;

[0034] The specific process of step S1 is as follows:

[0035] S11, in the process of round-by-round dialogue, the memory information is stored using the KV memory matrix, and the calculation formula is: , wherein, is the compressed KV memory matrix; is the key vector; is the value vector; is the set; is the size of the memory matrix; is the dimension of the memory space; is the key vector of the th element in the compressed KV memory matrix; is the value vector of the th element in the compressed KV memory matrix;

[0036] S12, dialogue embedding encoding: the input and output of the dialogue model are encoded and processed by the sentence encoder, and the subsequent memory storage and update operation is preprocessed, and the calculation formula is: , wherein, User input, i.e., input to the dialogue model; is the dialogue response, i.e., the output of the dialogue model; Embed(·) is the encoding operation of the text-level sentence encoder; is the output dimension of the sentence encoder; The result of encoding the dialogue model input through the text-level sentence encoder; The output of the dialogue model is encoded by the text-level sentence encoder;

[0037] S13. Attention scores and mapping: input via K key vectors and new dialogue models The association match between them is used to calculate the corresponding attention score; the K key vector and the input of the new dialogue model , value vector V and the output of the new dialogue model , after mapping, the output is calculated according to the attention distribution, and the calculation formula is: , , , , , , , , , , ,in, For Calculate the parameter matrix for memory self-attention query mapping; For Calculate the parameter matrix for memory self-attention key mapping; For Calculation parameter matrix for memory self-attention value mapping; For and Calculate the parameter matrix for the cross-attention query mapping between memory and new dialogue; For and Calculate the parameter matrix for the cross-attention key mapping between memory and new dialogue; For and Calculate the parameter matrix for mapping the cross-attention values ​​between memory and new dialogues; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for The result of the mapping; The result of the mapping; The result of the mapping; The result of the mapping; The result of the mapping; The result of the mapping; The result of the mapping; The result of the mapping; The result of the mapping; The result of the mapping; The result of the mapping; The result of the mapping; The result of the mapping; The result of the mapping; The result of the mapping; The result of the mapping; The result of the mapping; The result of the mapping; The result of the mapping;

[0038] The score calculation method combining self-attention and cross-attention is: , , , , is the attention weight distribution of the memory identifier to the input of the dialogue model; The result of the mapping; The result of the mapping; The result of the mapping; The result of the mapping; The attention score of the memory identifier to the input of the dialogue model; The attention score of the memory identifier to the memory identifier ; The attention score of the memory identifier to the input of the dialogue model; is an index value;

[0039] S14, memory updating mechanism: dynamically updating the memory according to the attention distribution of the global memory to the new dialogue and the memory capacity limit;

[0040] The calculation formula of step S14 is:

[0041] The calculation formula of the sum of the attention distribution of the global memory to the new dialogue is: , is the sum of the attention distribution of the global memory to the new dialogue;

[0042] According to the sum of the attention distribution of the global memory to the new dialogue , the lower limit of the attention distribution of the global memory to the new dialogue and the upper limit of the memory matrix size When The global memory allocates enough attention to the new dialogue, and the attention output is as usual, the calculation formula is: , Wherein, The attention output of ; The th mapped result; The attention output of ; The th mapped result; The attention output of ;

[0043] When The global memory allocates insufficient attention to the new dialogue, but there is still room for memory capacity, and the new memory identifier stores the missing information, the calculation formula is: , , Wherein, The attention output of the key vector of the th element of the new memory identifier; The attention output of the value vector of the th element of the new memory identifier; M = M+1, wherein

[0044] Indicates assignment; When The global memory allocates insufficient attention to the new dialogue, but the memory capacity is full, and the memory identifier with the lowest correlation with the new content will make up for the insufficient attention, and the attention to other information will be reduced accordingly, the calculation formula is: , , Wherein, The memory identifier with the minimum attention to the new dialogue; Take the minimum value; The attention output of the th memory identifier in K; The attention output of the th memory identifier in V; The attention score of the memory identifier to the input of the dialogue model;

[0045] S2, memory perception adaptation of the dialogue model: without changing the original model parameters, access cross-attention linear mapping layer to adapt between different semantic spaces;

[0046] The calculation formula of step S2 is: , , , , , , , , Wherein, is the i-th identifier of the sequence; is the dimension size; is the calculation parameter matrix for sequence self-attention query mapping of ; is the calculation parameter matrix for sequence self-attention key mapping of ; is the calculation parameter matrix for sequence self-attention value mapping of ; is the calculation parameter matrix for cross-attention query mapping of sequence and historical memory respectively to , and ; is the calculation parameter matrix for cross-attention key mapping of sequence and historical memory respectively to , and ; is the calculation parameter matrix for cross-attention value mapping of sequence and historical memory respectively to , and ; is the result of mapping; is the result of mapping; is the result of mapping; is the result of mapping; is the result of mapping; is the result of mapping; is the result of mapping; is the result of mapping; is the result of mapping;

[0047] ​S3, attention output calculation: combine the self-attention and the cross-attention calculation results after adaptation to obtain the final attention output;

[0048] The calculation formula of step S3 is: , , Wherein, is the sequence identifier The attention weight of the sequence identifier ; is the result of the first mapping; is the result of the second mapping; is the attention weight of the sequence identifier ; is the attention output of .

[0049] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.​

Claims

1. A method for long dialogue memory compression and dialogue model adaptation, characterized in that, The method comprises the following steps: S1, memory model-based dialogue history compression: obtaining dialogue content after a new round of dialogue between a user and a dialogue model, and compressing dialogue history into memory using a memory model; the memory model adopts a Transformer architecture, so that the updating of the memory is performed through a manner of re-distributing attention allocation, and the attention allocation is determined in combination with cross-attention between the memory and new dialogue information and global self-attention between the memory; S2, memory perception adaptation of the dialogue model: under the premise of not changing original model parameters, a cross-attention linear mapping layer is connected, and adaptation between different semantic spaces is performed; S3, attention output calculation: obtaining final attention output in combination with self-attention and the calculation result of the adapted cross-attention.

2. The long dialogue memory compression and dialogue model adaptation method of claim 1, wherein, The specific process of step S1 is as follows: S11, in the process of round by round dialogue, using KV memory matrix to store memory information, the calculation formula is: , , , , , , , , , , , , S12, dialogue embedding encoding: the input and output of the dialogue model are encoded by a sentence encoder for subsequent memory storage and update operation preprocessing, and the calculation formula is: , , wherein, is the user input, i.e. the input of the dialogue model; is the dialogue reply, i.e. the output of the dialogue model; Embed(·) is the encoding operation of the text-level sentence encoder; is the output dimension of the sentence encoder; is the encoding operation result of the input of the dialogue model through the text-level sentence encoder; is the encoding operation result of the output of the dialogue model through the text-level sentence encoder; S13, Attention score and mapping: through K key vector and input of new dialogue model , calculate the corresponding attention score; K key vector and input of new dialogue model , value vector V and output of new dialogue model , calculate the output according to the attention allocation after mapping, the calculation formula is: , , , , , , , , , , , wherein, is the calculation parameter matrix for memory self-attention query mapping of ; is the calculation parameter matrix for memory self-attention key mapping of ; is the calculation parameter matrix for memory self-attention value mapping of ; is the calculation parameter matrix for cross-attention query mapping of memory and new dialogue of and respectively; is the calculation parameter matrix for cross-attention key mapping of memory and new dialogue of and respectively; is the calculation parameter matrix for cross-attention value mapping of memory and new dialogue of and respectively; is the result of mapping; is the result of mapping; is the result of mapping; is the result of mapping; is the result of mapping; is the result of mapping; is the result of mapping; is the result of mapping; is the result of mapping; is the calculation parameter matrix for mapping of ; is the calculation parameter matrix for mapping of a matrix of computational parameters to perform the mapping; for a result of the mapping; a result of the mapping; for a result of the mapping; a result of the mapping; The score calculation method for combining self-attention and cross-attention is: , , , ,in, Mark for memory Memory Mark Attention weight distribution; For the indivual through The result of the mapping; Mark for memory Input to the conversation model Attention weight distribution; Mark for memory Memory Mark Attention score; Mark for memory Input to the conversation model Attention score; is the index value; S14, memory updating mechanism: dynamically updating the memory according to the attention allocation of the global memory to the new dialogue and the memory capacity limit.

3. The long dialogue memory compression and dialogue model adaptation method of claim 2, wherein, The calculation formula of step S14 is as follows: The formula for the sum of the global memory's attention allocation to the new dialog is: wherein, is the sum of the global memory's attention allocation to the new dialog; The sum of attention allocations to new dialogues according to global memory , the lower limit of global memory attention allocation to new dialogues and the upper limit of memory matrix size ,when , the global memory allocates enough attention to the new dialogue, and the attention output is carried out as usual. The calculation formula is: , ,in, for Attention output; For the indivual through The result of the mapping; for Attention output; For the indivual through The result of the mapping; When The global memory has enough capacity for new memories, but the attention of the new dialogue is not enough, so the new memory identifier stores the missing information, and the calculation formula is: , , Wherein, is the attention output of the key vector of the first element of the new memory identifier; is the attention output of the value vector of the first element of the new memory identifier; is M = M+1, wherein represents assignment; when , the global memory does not allocate enough attention to the new dialogue, but the memory capacity is full. The memory markers with the lowest relevance to the new content will make up for the insufficient attention and correspondingly reduce the attention to other information. The calculation formula is: , , ,in, The memory marker that pays the least attention to the new conversation; For Take the minimum value; For K The attention output of each memory token; For V The attention output of each memory token; Input to the dialogue model for memorization Attention score; Mark for memory Memory Mark Attention score.

4. The long dialogue memory compression and dialogue model adaptation method of claim 1, wherein, The calculation formula of step S2 is: , , , , , , , , ,in, The first a logo; is the dimension size; For Calculate the parameter matrix for sequential self-attention query mapping; For Calculate the parameter matrix for sequential self-attention key mapping; For Calculation parameter matrix for sequence self-attention value mapping; For 、 and Calculate the parameter matrix for cross-attention query mapping between sequence and history memory; For 、 and Calculate the parameter matrix for cross-attention key mapping between sequence and history memory; For 、 and Calculate the parameter matrix for mapping the cross-attention values ​​between sequence and history memory; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping; for through The result of the mapping.

5. The long dialogue memory compression and dialogue model adaptation method of claim 4, wherein, The calculation formula of step S3 is: , , ,in, Sequence ID Sequence ID The attention weight of For the indivual through The result of the mapping; Sequence ID Sequence ID The attention weight of for Attention output.

Images