A large language model for generating multi-session dialogues

Abstract

The application provides a large language model for generating multi-session dialogues, comprising an original large language model, a VAE unit, a LoRA unit and a hidden state generation unit; the VAE unit receives a current query or extracts a query in each round of dialogue from a dialogue history, then generates a hidden vector representation corresponding to the current query and the dialogue history, and transmits the hidden vector representation to the hidden state generation unit; the LoRA unit is connected after the output of each layer of the original large language model, superimposes a weight matrix on the output of each layer, and then transmits the result to the hidden state generation unit; the hidden state generation unit receives the output of the LoRA unit and the VAE unit and the parameters of each layer of the original large model, and adds the three to obtain a new hidden state of each layer of the original large language model. The application can generate replies consistent with the dialogue history, improve the dialogue capability of the large language model, enrich the dialogue experience of users and attract the dialogue interest of users.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and deep learning, and in particular to a large language model for generating multi-conversation dialogues. Background Technology

[0002] Multi-conversation dialogue generation has become a key issue and important challenge in open-domain dialogue systems. Its goal is to generate responses consistent with the dialogue history. Multi-conversation dialogues contain dialogue history from multiple sessions, significantly exceeding the limitation of traditional personalized dialogues to 16 rounds [1].

[0003] Most existing multi-conversation dialogue generation models currently focus only on the dialogue history of the current conversation. They still cannot establish long-term interaction with humans in multi-conversation dialogues, especially in multi-conversation personalized dialogue generation. In real-world dialogue scenarios, dialogue models usually need to have the ability to accompany and become familiar with each other for a long time [2].

[0004] Existing multi-conversation dialogue generation methods mainly focus on historical modeling, which can be further divided into summarization-based methods, extended context window methods, and retrieval-based methods. Summarization-based methods usually use prompts or instructions to make the generation model summarize the dialogue history. For example, reference [3] proposes to recursively use previous memories and subsequent context to generate new memories, thereby helping chatbots generate highly consistent responses in long-term dialogues. Extended context window methods increase the input capacity by changing the architecture of the generation model. For example, reference [4] designed different attention mask matrices based on long-term memory and short-term memory to expand the input scale, and the efficiency was significantly improved while ensuring performance. However, this method may face the risk of losing important historical information or introducing noise. In order to guide the model to focus on important historical information, many studies have explored the use of retrieval-based methods. They use retrieval models to obtain relevant dialogue information. For example, reference [5] proposes to use a memory pool and store the dialogue history in the memory pool. The memory pool accumulates information as the dialogue progresses. Before the dialogue is generated, retrieval tools such as DPR are used to retrieve personality memories or historical event memories to maintain the consistency of the dialogue model. While these methods can access relevant historical information, they may disrupt the dialogue structure and hinder the model's understanding of the dialogue history.

[0005] References:

[0006] [1] Huang, M., Zhu, X., & Gao, J. (2019). Challenges in BuildingIntelligent Open-domain Dialog Systems. ACM Transactions on InformationSystems (TOIS), 38, 1 - 32.

[0007] [2] Li, Hao, Chenghao Yang, An Zhang, Yang Deng, Xiang Wang and Tat-Seng Chua. “Hello Again! LLM-powered Personalized Agent for Long-termDialogue.” ArXiv abs / 2406.05925 (2024): n. when.

[0008] [3] Wang, Qingyue, Liang Ding, Yanan Cao, Zhiliang Tian, ​​Shi Wang,Dacheng Tao and Li Guo. “Recursively Summarizing Enables Long-Term DialogueMemory in Large Language Models.” ArXiv abs / 2308.15022 (2023): n. when.

[0009] [4] Kim, Seo Hyun, Keummin Ka, Yohan Jo, Seung-won Hwang, Dongha Leeand Jinyoung Yeo. “Ever-Evolving Memory by Blending and Refining the Past.” ArXiv abs / 2403.04787 (2024): n. when.

[0010] [5] Lee, Gibbeum, Volker Hartmann, Jongho Park, DimitrisPapailiopoulos and Kangwook Lee. “Prompted LLMs as Chatbot Modules for LongOpen-domain Conversation.” Annual Meeting of the Association for Computational Linguistics (2023). Summary of the Invention

[0011] To address the aforementioned issues and enhance the generation capabilities of large language models in multi-conversation dialogue generation tasks, this invention improves upon existing open-source large language models by adding VAE units, LoRA units, and hidden state generation units to each layer. This provides the large language model with useful prior knowledge in the context of multi-conversation dialogue generation, thereby enhancing its multi-conversation dialogue generation capabilities.

[0012] The large language model for generating multi-conversation dialogues proposed in this invention includes an original large language model and a hidden state improvement module. The hidden state improvement module, used to generate new hidden states for each layer of the original large language model, includes a VAE unit, a LoRA unit, and a hidden state generation unit.

[0013] The VAE unit is used to receive the current query or extract the query from each round of dialogue from the dialogue history, then generate the implicit vector representation corresponding to the current query and the dialogue history, and transmit the implicit vector representation to the hidden state generation unit.

[0014] LoRA units, which have LoRA weight matrices (i.e., low-rank matrices), are connected after the output of each layer of the original large language model. They are used to superimpose the weight matrices on the output of each layer and then pass the result to the hidden state generation unit.

[0015] The hidden state generation unit receives the outputs of the LoRA unit and the VAE unit, as well as the parameters of each layer of the original large model. It adds the three together to obtain the new hidden state of each layer of the original large language model.

[0016] Furthermore, it also includes a historical dialogue acquisition module, which processes the dialogue history, calculates the similarity score between each dialogue history and the current query, selects the k dialogue histories with the highest relevance scores as the dialogue history of the current query, and then inputs the dialogue history into the original large language model.

[0017] Furthermore, in the historical dialogue acquisition module, the methods for obtaining similarity scores include:

[0018] The time interval between all dialogue history entries in the dialogue and the current query is calculated, and a weight is assigned to each dialogue history entry according to the time decay formula:

[0019]

[0020] Where t is the time interval. Temperature coefficient is a hyperparameter that can be adjusted automatically.

[0021] The similarity between the current query and all dialogue history in the dialogue is calculated and multiplied by the time decay factor to obtain the final similarity score.

[0022] Furthermore, in the VAE unit, a variational autoencoder is used to encode and reconstruct the input data, and the query in the dialogue is encoded to better extract the implicit information in the current query, ultimately generating a vector containing the implicit information.

[0023] Furthermore, the VAE unit also includes an adapter to bridge the gap between the implicit vectors generated by the VAE module and the vector space of the original large language model.

[0024] Furthermore, in the VAE unit, during training, the current query and the standard answer are input to the encoder to obtain representation vectors; then, the reparameter technique is used to sample the two representation vectors to ensure that gradients can be calculated in all steps of training.

[0025] Furthermore, we approximate the two sampled vectors using KL divergence, with the following loss function:

[0026]

[0027] in, and These are the approximate posterior distribution and the approximate prior distribution, respectively. and It's a hyperparameter. For the representation of the query, As a representation of the standard answer, This represents the sampling distribution of the VAE.

[0028] Furthermore, it also includes a data cleaning module for data cleaning and segmentation, obtaining several dialogues and corresponding tags. Each piece of data in the processed text contains the user's personality information, the chatbot's personality information, the user's current query, the chatbot's standard response, and past dialogue history.

[0029] Furthermore, it also includes a training module for training the parameters of VAE units and LoRA units, and the training methods include:

[0030] S1. Clean and segment the dataset to obtain several dialogues and their corresponding labels;

[0031] S2. Retrieve the dialogue history related to the current query;

[0032] S3. Extract the queries from each round of dialogue and input them into the VAE unit to obtain implicit representations;

[0033] S4. Add the implicit representation, the parameters of the original large language model, and the output of the LoRA unit to set the training parameters, and update the training parameters using the backpropagation algorithm.

[0034] Furthermore, in the training module, the loss function is:

[0035]

[0036]

[0037] in, It is the generation loss of the original large language model. LoRA represents With hidden state The loss between them, MIM is a method to obtain the maximum mutual information. It's a hyperparameter.

[0038] The beneficial effects of this invention are as follows:

[0039] (1) Modeling the memory mechanism for the time interval between different sessions can effectively filter the dialogue history related to the current query and provide useful historical information for dialogue generation.

[0040] (2) Creatively combining VAE and LoRA is a new and efficient fine-tuning method that can effectively provide rich prior knowledge for large language models and improve their generation capabilities.

[0041] (3) This method has achieved the best results to date on the MSC dataset, indicating that this method can achieve good performance on multi-session dialogue generation tasks. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of the structure of a large language model according to an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of the improved structure of each layer of a large language model according to an embodiment of the present invention;

[0045] Figure 3 This is a flowchart illustrating a historical dialogue acquisition module according to an embodiment of the present invention;

[0046] Figure 4 This is a flowchart illustrating a VAE unit according to an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of the training process of a large language model according to an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be reviewed and described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] The large language model for generating multi-conversation dialogues provided by this invention, such as Figure 1 , 2 As shown, it includes the original large language model and the hidden state improvement module, among which,

[0050] The original large language model is an existing open-source large language model, such as ChatGLM and Llama2.

[0051] The hidden state improvement module is used to generate new hidden states for each layer of the original large language model. It includes VAE units, LoRA units, and a hidden state generation unit.

[0052] The VAE unit is used to receive the current query or extract the query from each round of dialogue from the dialogue history, then generate the implicit vector representation corresponding to the current query and the dialogue history, and transmit the implicit vector representation to the hidden state generation unit.

[0053] The LoRA unit, with a LoRA weight matrix (i.e., a low-rank matrix), is connected after the output of each layer of the original large language model. It is used to superimpose the output of each layer onto the parameter matrix, and then pass the result to the hidden state generation unit.

[0054] The hidden state generation unit receives the outputs of the LoRA unit and the VAE unit, as well as the parameters of each layer of the original large model. The three are added together to obtain the new hidden state of each layer of the original large language model.

[0055] The hidden state of the original large language model refers to the output of each layer, which is a series of vectors. The hidden state of the last layer, plus an additional network similar to MLP, is mapped onto the vocabulary of the large language model to produce the final generated result. This part is part of the original large language model's own structure and has not been changed, so it will not be elaborated further.

[0056] In one embodiment, each layer of the original large language model corresponds to a LoRA unit and a VAE unit.

[0057] In one embodiment, because the dialogue history is extremely long, it cannot be directly input into the large language model. Therefore, this invention designs a historical dialogue acquisition module to process the dialogue history. Specifically, it includes: using the DPR model and a time decay formula to calculate the relevance score of each dialogue history to the current query, selecting the k dialogue histories with the highest relevance scores as the dialogue history of the current query, and then transmitting the dialogue history to the original large language model to generate the hidden state of each layer.

[0058] In the historical dialogue acquisition module, a memory mechanism is established for the large language model to retrieve the dialogue history relevant to the current query from past dialogues, such as... Figure 3 As shown, the time interval between all dialogue history entries in the dialogue and the current query is first calculated, and then a weight is assigned to each dialogue history entry according to the time decay formula:

[0059]

[0060] Where t is the time interval. The temperature coefficient is a hyperparameter that can be adjusted automatically. Then, the similarity between all dialogue history in the dialogue and the current query is calculated (e.g., using a DPR model), and multiplied by the time decay factor to obtain the final similarity score. Based on this similarity score, the k dialogues with the highest similarity are selected as relevant dialogue history inputs to the VAE unit.

[0061] The DPR (Dense Passage Retrieval) model, proposed by Facebook AI Research, aims to improve the efficiency and accuracy of retrieval through dense vector representation, selecting the most relevant passages as search results.

[0062] In one embodiment, the historical dialogue acquisition module can use Hamming distance, cosine similarity, etc., to calculate the similarity between all historical dialogues in the dialogue and the current query.

[0063] In the VAE unit, a variational autoencoder is used to generate new data and compress and reconstruct the input data. The variational autoencoder combines the encoder and decoder parts, enabling efficient random sampling in the latent space to generate new samples similar to the training data. In this invention, as... Figure 4 As shown, an encoder using only variational autoencoders leverages its ability to encode and reconstruct input data to encode queries in the dialogue, better extracting implicit information such as topic, sentiment, and personality from the current query, ultimately generating corresponding vectors containing implicit information. During the training of the large language model, both the current query and the standard answer are input to the encoder to obtain representation vectors. The standard answer is introduced to provide effective prior knowledge to the original large language model, improving its ability to generate meaningful answers. For the encoder, any model with encoding capabilities can be used, such as BERT. Then, reparameterization is used to sample the implicit space of the two representation vectors to ensure that gradients can be calculated in all steps of training.

[0064]

[0065] Finally, to ensure the consistency of prior knowledge, KL divergence is used to approximate the two sampled vectors, and the loss function is as follows:

[0066]

[0067] in, and These are the approximate posterior distribution and the approximate prior distribution, respectively. and It's a hyperparameter. For the representation of the query, As a representation of the standard answer, This represents the sampling distribution for the VAE. During inference, it is only necessary to perform the same encoding and resampling on the current query to obtain the vector.

[0068] In one embodiment, the VAE unit further includes an adapter for connection after the VAE unit to bridge the gap between the latent representation vectors extracted by the VAE module and the vector space of the original large language model.

[0069] In the LoRA unit, LoRA (Low-Rank Adaptation) is an efficient model fine-tuning technique designed to optimize the adaptability of large-scale pre-trained models. It features a low-rank matrix (i.e., a weight matrix), enabling fast and efficient model adjustments, thereby significantly reducing the required computational resources and storage space while maintaining the original model performance. This allows for high-quality fine-tuning even with limited resources.

[0070] In the hidden state generation unit, the output of the VAE unit, the output of the LoRA unit, and the hidden state of the original large language model are received. Then, the three are added together to form the new hidden state of the original large language model, as shown in the following formula:

[0071]

[0072] in, It is a new hidden state of the original large language model. This is the initial hidden state of the original large language model. These are the parameters of the original large language model. and It is the weight matrix of the LoRA unit. and It is the parameter matrix of the adapter.

[0073] This invention creatively integrates the concepts of Variational Autoencoders (VAEs) and Low-Rank Adaptation (LoRA). While LoRA can effectively improve the performance of large language models in many tasks, its improvement is always limited in specific tasks. To further enhance the performance of LoRA, the parameters (weights, which are essentially matrices) of the LoRA units, the sampling results of the VAE or the result after multiplying with the adapter (also a matrix), and the output of each layer of the large language model (also a matrix) are added together to obtain a new hidden state of the large language model. This provides the large language model with more effective prior knowledge to improve its language model generation ability, while ensuring its performance in downstream tasks.

[0074] In one embodiment, a data cleaning module is also included for data cleaning and segmentation to obtain several dialogues and corresponding tags. The text input to the large model may contain dialogue data, including all dialogue history and time intervals between different sessions. All dialogue data is described in natural languages ​​such as English and Chinese. To facilitate subsequent model training, the input text is processed to segment the dialogue history of different sessions according to the time intervals in the text, and sensitive information appearing in open-domain dialogues is filtered out. Each piece of data in the processed text contains the user's personality information, the chatbot's personality information, the user's current query, the chatbot's standard response, and previous dialogue history.

[0075] In one embodiment, a training module is also included to set the hyperparameters of the VAE and LoRA units, including the learning rate, batch size, and number of training iterations (the parameters of the original large language model are frozen at this time). Then, the parameters of the VAE and LoRA units are updated using a backpropagation algorithm (such as the AdamW optimizer) based on the loss function. The loss function is as follows:

[0076]

[0077]

[0078] in, It is the generation loss of the original large language model. yes With the new hidden state The loss between them, MIM is a method to obtain the maximum mutual information. In fact, mean squared error and KL divergence can also be applied. It's a hyperparameter.

[0079] The specific training process is as follows: Figure 5 As shown, it includes:

[0080] S1. Clean and segment the dataset to obtain several dialogues and their corresponding labels;

[0081] S2. Retrieve the dialogue history related to the current query;

[0082] S3. Extract the queries from each round of dialogue and input them into the VAE unit to obtain implicit representations;

[0083] S4. Add the implicit representation, the parameters of the original large language model, and the output of the LoRA unit to obtain a new hidden state, set the training parameters, generate multi-conversation dialogues, and then use the backpropagation algorithm to update the training parameters.

[0084] S1 can be completed through the data cleaning module, S2 through the historical dialogue acquisition module, S3 through the VAE unit, and S4 through the LoRA unit and the hidden state generation unit. These will not be elaborated on here.

[0085] This invention improves the architecture of the original large language model. Based on the establishment of a memory mechanism, it trains VAE-LoRA units, which not only maintains the dialogue structure so that the large language model can fully understand the dialogue context of long-term dialogues, but also enables the large language model to deeply understand and mine user profiles, and generate responses consistent with dialogue history and personalized information.

[0086] Experimental verification:

[0087] The input text is the MSC dataset. The paper Beyond Goldfish Memory: Long-Term Open-Domain Conversation, presented at the 2022 ACL conference, first proposed open-domain dialogue with multi-session sessions and introduced this dataset.

[0088] This invention effectively enhances the performance of large language models on the MSC dataset, achieving state-of-the-art (SOTA) performance. As shown in Table 1, experiments were conducted on the MSC dataset using ChatGLM-6B and Llama2-7B, and the method of this invention. BlenderBot, HAHT, ChatGLM-6B (Base), Llama2-7B (Base), ChatGPT, CPD, StreamingDialogue, and LD-Agent were used as baseline models for comparison on BLEU-1, BLEU-2, BLEU-3, and ROUGE-L metrics.

[0089] BlenderBot is an advanced open-domain dialogue model developed through large-scale pre-training on massive datasets, combining a retrieval-refinement mechanism and optimized decoding strategies. HAHT is a model employing hierarchical encoding and attention mechanisms to maintain and utilize long-term historical dialogue memory, generating context-sensitive responses through a history-aware response generator. ChatGPT is a closed-source large-scale language model based on the GPT architecture; we utilize OpenAI's gpt-3.5-turbo model API service. CPD is a causal-aware multi-turn dialogue framework that employs a perturbation-based causal variable discovery method to extract highly causally relevant statements from historical dialogues, thereby enhancing the causal awareness of large-scale language models. StreamingDialogue is a model that effectively handles long-contextual dialogues by compressing dialogue history into a "dialogue attention pool" and employing short-term memory reconstruction (SMR) and long-term memory activation (LMR) learning strategies. LD-Agent is a dialogue agent framework that supports coherent dialogue by integrating event memory and personalized role modeling. It is the previous state-of-the-art method on the MSC dataset. As shown in Table 1, the best results are indicated in bold, and the second-best results are marked with an underline.

[0090] Table 1

[0091] Model BLEU-1 BLEU-2 BLEU-3 ROUGE-L BlenderBot - 4.91 1.53 16.06 HAHT - 5.10 1.59 16.58 ChatGLM(Base) 19.20 5.54 1.50 16.49 Llama2(Base) 17.34 4.37 1.21 10.29 ChatGPT 18.96 5.77 1.51 16.84 CPD 12.45 4.41 - 12.14 StreamingDialogue 19.33 - - 15.86 LD-Agent 19.54 7.31 2.51 18.44 Ours (ChatGLM) <![CDATA[ 19.87 ]]> <![CDATA[ 7.71 ]]> <![CDATA[ 2.55 ]]> 19.87 Ours (Llama2) 20.51 8.16 <![CDATA[ 2.59 ]]> <![CDATA[ 18.77 ]]>

[0092] Those skilled in the art will understand that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A large language model for generating multi-conversation dialogues, characterized in that, It includes the original large language model and a hidden state improvement module. The hidden state improvement module is used to generate new hidden states for each layer of the original large language model, and includes VAE units, LoRA units, and a hidden state generation unit. The VAE unit receives the current query or extracts queries from each round of dialogue from the dialogue history, then generates implicit vector representations corresponding to the current query and the dialogue history, and transmits these implicit vector representations to the hidden state generation unit. In the VAE unit, during training, the current query and the standard answer are input to the encoder to obtain representation vectors. Then, the two representation vectors are sampled using the reparameter technique to ensure that gradients can be calculated in all steps of training. To ensure the consistency of prior knowledge, KL divergence is used to approximate the two sampled vectors. The LoRA unit, with a LoRA weight matrix, is connected after the output of each layer of the original large language model. It is used to superimpose the weight matrix on the output of each layer and then pass the result to the hidden state generation unit. The hidden state generation unit receives the outputs of the LoRA unit and the VAE unit, as well as the parameters of each layer of the original large model. It adds the three together to obtain the new hidden state of each layer of the original large language model.

2. The large language model according to claim 1, characterized in that, It also includes a historical dialogue acquisition module, which processes dialogue history, calculates the similarity score between each dialogue history and the current query, selects the k dialogue histories with the highest relevance scores as the dialogue history of the current query, and then inputs the dialogue history into the original large language model.

3. The large language model according to claim 2, characterized in that, In the historical dialogue acquisition module, the methods for obtaining similarity scores include: The time interval between all dialogue history entries in the dialogue and the current query is calculated, and a weight is assigned to each dialogue history entry according to the time decay formula: Where t is the time interval. The temperature coefficient is a self-adjusting hyperparameter. The similarity between the current query and all dialogue history in the dialogue is calculated and multiplied by a time decay factor to obtain the final similarity score.

4. The large language model according to claim 1, characterized in that, In the VAE unit, a variational autoencoder is used to encode and reconstruct the input data, and the query in the dialogue is encoded to better extract the implicit information in the current query, ultimately generating a vector containing the implicit information.

5. The large language model according to claim 4, characterized in that, The VAE unit also includes an adapter to bridge the gap between the implicit vectors generated by the VAE module and the vector space of the original large language model.

6. The large language model according to claim 1, characterized in that, In the VAE unit, the loss function is as follows: in, and These are the approximate posterior distribution and the approximate prior distribution, respectively. and It's a hyperparameter. For the representation of the query, As a representation of the standard answer, This represents the sampling distribution of the VAE.

7. The large language model according to claim 1, characterized in that, It also includes a data cleaning module for data cleaning and segmentation, obtaining several dialogues and corresponding tags. Each piece of data in the processed text contains the user's personality information, the chatbot's personality information, the user's current query, the chatbot's standard response, and past dialogue history.

8. The large language model according to claim 1, characterized in that, It also includes a training module for training the parameters of VAE units and LoRA units. The training methods include: S1. Clean and segment the dataset to obtain several dialogues and their corresponding labels; S2. Retrieve the dialogue history related to the current query; S3. Extract the queries from each round of dialogue and input them into the VAE unit to obtain implicit representations; S4. Add the implicit representation, the parameters of the original large language model, and the output of the LoRA unit to set the training parameters, and update the training parameters using the backpropagation algorithm.

9. The large language model according to claim 8, characterized in that, In the training module, the loss function is: in, It is the generation loss of the original large language model. In VAE-LoRA, LoRA represents... With hidden state The loss between them, MIM is a method to obtain the maximum mutual information. It's a hyperparameter.

Images