Multi-round dialogue optimization method, system and equipment based on large language model and medium

Abstract

The invention relates to the technical field of natural language processing, and discloses a multi-round dialogue optimization method based on a large language model.The method comprises the steps that before the large language model outputs answers according to current questions input by a user, the current questions input by the user are subjected to question preference alignment, answer preference alignment and preferred item preference alignment in a memory list; and in combination with a self-attention reward iteration function and a model updating rule based on a strategy gradient, an optimal matching historical question and an optimal matching historical answer of the current question are found, an optimal historical question and answer pair of the current question is formed, context window limitation of a large language model can be overcome, accurate historical memory can be provided for the large language model, and the user experience is improved. The context understanding ability and reasoning ability of the large language model in multiple rounds of dialogues are greatly improved, so that the large language model can accurately output coherent and high-quality answers, the quality of the multiple rounds of dialogues is improved, and the user experience is ensured.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a multi-round dialogue optimization method, system, device and medium based on a large language model. Background Art

[0002] As a branch of the field of natural language processing (NLP), multi-round dialogue technology has developed rapidly in recent years with the rise of large language models. By learning a large amount of dialogue data, it is possible to better understand the context and generate coherent responses. Multi-round dialogues require large language models to have stronger context understanding and reasoning capabilities in order to maintain coherence and consistency during the dialogue process. However, due to the memory limitations of computing resources, computing costs, and the architectural design of large language models, when conducting multi-round dialogues, the length of the dialogue context that can be effectively processed by the large language model is limited by a threshold. As the number of dialogue rounds increases, the large language model may also forget the previous context information. The gradual loss of historical dialogue information may cause factual errors, failure of dialogue context association, and a significant decrease in context understanding and reasoning capabilities, resulting in poor quality of multi-round dialogues.

[0003] Therefore, there is an urgent need for a multi-round dialogue optimization method based on a large language model. Summary of the invention

[0004] The present invention provides a multi-round dialogue optimization method, system, device and medium based on a large language model, which are used to solve the defect that the existing multi-round dialogue method based on a large language model lacks an effective historical dialogue processing mechanism, resulting in poor quality of multi-round dialogue.

[0005] The present invention provides a multi-round dialogue optimization method based on a large language model, comprising: Receive user input for the current question; According to the current question, question preference alignment, answer preference alignment, and preference preference alignment are performed in the memory list, and this step is iterated until the preset stop condition is reached to obtain the optimal historical question-answer pair that matches the current question; Based on the current question and the best historical question-answer pair that matches the current question, the best answer to the current question is obtained through a large language model.

[0006] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the question preference alignment includes: The similarity score between each historical question in the memory list and the current question is obtained through the retriever, and all the historical questions in the memory list are sorted and labeled according to the similarity score, wherein the types of labels include preferred matching historical questions and non-preferred matching historical questions; According to the similarity score between each historical question in the memory list and the current question, the first loss function is used to obtain the first preference score of the current question under each query in the memory list through the large language model, and all historical questions in the memory list are sorted again according to the first preference score to obtain a set of historical questions that match the current question.

[0007] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the first preference score of the current question under each query in the memory list is obtained by using a first loss function based on the similarity score between each historical question and the current question in the memory list through the large language model, including: The preference score between each historical question and the current question in the memory list is obtained by using a large language model as a preference reward model; By combining the similarity scores between each historical question in the memory list and the current question and the preference scores between each historical question in the memory list and the current question, the first preference score of the current question under each query in the memory list is obtained through the first expression.

[0008] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the first expression is: , In the first expression, Indicates current problem In the memory list The first preference score under queries, Represents the first Historical Questions With current issues The preference score of Indicates the first Historical Questions With current issues The similarity score of Represents the first parameter weight.

[0009] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the expression of the first loss function is: , In the expression of the first loss function Indicates current problem The problem prefers alignment loss, represents the loss function, It means averaging all samples. These variables are distributed from preference Medium sampling, Respectively represent historical issues , Historical issues The types of labels include preferred matching history issues and non-preferred matching history issues. represents the probability distribution of the output, represents the set of historical questions in the iteration process, Represents the sigmoid function, which is used to map the input to values ​​between 0 and 1.

[0010] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the answer preference alignment includes: The similarity score between each historical answer in the memory list and the current question is obtained through the retriever, and all the historical answers in the memory list are sorted and labeled according to the similarity score, wherein the types of labels include preferred matching historical answers and non-preferred matching historical answers; According to the similarity score between each historical answer in the memory list and the current question, the second preference score of the current question under each query in the memory list is obtained through the large language model using the second loss function, and all historical answers in the memory list are re-sorted according to the second preference score to obtain a set of historical answers that match the current question.

[0011] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the second preference score of the current question under each query in the memory list is obtained by using a large language model according to the similarity score between each historical answer in the memory list and the current question, using a second loss function, including: The preference score between each historical answer and the current question in the memory list is obtained by using a large language model as a preference reward model. By combining the similarity score between each historical answer in the memory list and the current question and the preference score between each historical answer in the memory list and the current question, the second preference score of the current question under each query in the memory list is obtained through the second expression.

[0012] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the second expression is: , In the second expression, Indicates current problem In the memory list The second preference score under queries, Represents the first Historical Answers With current issues The preference score of Indicates the first Historical Answers With current issues The similarity score of Represents the second parameter weight.

[0013] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the expression of the second loss function is: , In the expression of the second loss function, Indicates current problem The answer prefers alignment loss, represents the number of samples in each batch, Represents the batch samples in a batch and The number of samples with the same label, including preferred matching historical answers and non-preferred matching historical answers, represents the indicator function, , , Represent the hidden representation of the word vector of the query, the preferred matching historical answer, and the non-preferred matching historical answer, respectively. Indicates the retrieval relevance adjustment parameter.

[0014] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the preference alignment of the first option includes: According to the set of historical questions matching the current question and the set of historical answers matching the current question, combined with the self-attention reward iteration function and the policy gradient-based model update rule, preference alignment is performed to obtain the optimal historical question-answer pair matching the current question.

[0015] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the expression of the self-attention reward iteration function is: , In the expression of the self-attention reward iteration function, represents the self-attention reward iteration function, represents the evaluation function, Represents the preferred historical conversation information for each iteration, Represents the non-preferred historical dialogue information of each iteration, where the historical dialogue information includes historical questions and historical answers.

[0016] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the expression of the model update rule based on the policy gradient is: , In the expression of the policy gradient-based model update rule, Indicates current problem The problem prefers alignment loss, Indicates current problem The answer prefers alignment loss, , They represent the adjustment parameters for the historical questions and answers in this iteration, represents the model parameters of the large language model, represents the learning rate, Represents the self-attention reward iteration function.

[0017] A multi-round dialogue optimization method based on a large language model provided by the present invention also includes: Update the current question and best answer to the memory list.

[0018] The present invention also provides a multi-round dialogue optimization system based on a large language model, comprising: The receiving module is used to: receive the current question input by the user; An alignment module is used to: perform question preference alignment, answer preference alignment, and option preference alignment in the memory list according to the current question, and iterate this step until a preset stop condition is reached to obtain the optimal historical question-answer pair that matches the current question; The output module is used to obtain the optimal answer to the current question through a large language model based on the current question and the optimal historical question-answer pair that matches the current question.

[0019] The present invention also provides an electronic device, comprising a processor and a memory storing a computer program, characterized in that when the processor executes the computer program, any of the multi-round dialogue optimization methods based on a large language model as described above is implemented.

[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multi-round dialogue optimization methods based on a large language model described above.

[0021] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute any of the above-mentioned multi-round dialogue optimization methods based on a large language model.

[0022] The present invention provides a multi-round dialogue optimization method, system, device and medium based on a large language model. Before the large language model outputs an answer according to the current question input by the user, the current question input by the user is first aligned in the memory list with the question preference, answer preference and option preference. The self-attention reward iteration function and the policy gradient-based model update rule are combined to find the preferred matching historical question and the preferred matching historical answer for the current question, and form the optimal historical question-answer pair for the current question. The context window limitation of the large language model can be overcome and accurate historical memory can be provided for the large language model, which greatly improves the context understanding and reasoning ability of the large language model in multi-round dialogues, so that the large language model can accurately output coherent and high-quality answers, improve the quality of multi-round dialogues, and ensure user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 One of the flow charts of a multi-round dialogue optimization method based on a large language model provided by the present invention.

[0025] Figure 2 The second flowchart of a multi-round dialogue optimization method based on a large language model provided by the present invention.

[0026] Figure 3 Shows the relationship between question preference alignment, answer preference alignment, and preference preference alignment.

[0027] Figure 4 A schematic diagram of the structure of a multi-round dialogue optimization system based on a large language model provided by the present invention.

[0028] Figure 5 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, and they should not be understood as limitations on the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In the description of the present invention, it should be understood that the terms used are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0030] Combine the following Figure 1-Figure 5 The multi-round dialogue optimization method, system, device and medium based on a large language model provided by the present invention are described. It should be noted that the execution subject of the multi-round dialogue optimization method based on a large language model provided by the present invention can be any network side device / terminal side device that meets the technical requirements, such as a multi-round dialogue optimization device based on a large language model.

[0031] Figure 1-Figure 2 Schematic diagram of the process of the multi-round dialogue optimization method based on the large language model provided by the present invention. Figure 1 The present invention provides a multi-round dialogue optimization method based on a large language model, which may include: Step S110: receiving a current question input by a user.

[0032] Step S120: According to the current question, question preference alignment, answer preference alignment, and preference preference alignment are performed in the memory list, and the step is iterated until the preset stop condition is reached to obtain the optimal historical question-answer pair matching the current question. The preset stop condition can be set according to the actual situation, such as the preset number of iterations, etc. The relationship between question preference alignment, answer preference alignment, and preference preference alignment can be seen in Figure 3 .

[0033] In one embodiment, question preference alignment includes: The similarity score between each historical question in the memory list and the current question is obtained through the retriever, and all the historical questions in the memory list are sorted and labeled according to the similarity score, wherein the types of labels include preferred matching historical questions and non-preferred matching historical questions, and the labeling standard can be set according to the actual situation. For example, when the similarity score between the historical question and the current question is greater than a preset threshold, the historical question is determined to be a preferred matching historical question, otherwise it is a non-preferred matching historical question; According to the similarity score between each historical question in the memory list and the current question, the first loss function is used to obtain the first preference score of the current question under each query in the memory list through the large language model, and all historical questions in the memory list are sorted again according to the first preference score to obtain a set of historical questions that match the current question.

[0034] Among them, the preference score of each historical question in the memory list and the current question can be obtained by using a large language model as a preference reward model, and then the similarity score of each historical question in the memory list and the current question and the preference score of each historical question in the memory list and the current question are combined, and the first preference score of the current question under each query in the memory list is obtained through the first expression.

[0035] The first expression is: , In the first expression, Indicates current problem In the memory list The first preference score under queries, Represents the first Historical Questions With current issues The preference score of Indicates the first Historical Questions With current issues The similarity score of Represents the first parameter weight.

[0036] The expression of the first loss function is: , In the expression of the first loss function, Indicates current problem The problem prefers alignment loss, represents the loss function, It means averaging all samples. These variables are distributed from preference Medium sampling, Respectively represent historical issues , Historical issues The types of labels include preferred matching history issues and non-preferred matching history issues. represents the probability distribution of the output, represents the set of historical questions in the iteration process, Represents the sigmoid function, which is used to map the input to values ​​between 0 and 1.

[0037] In one embodiment, the answer in the answer preference alignment may be the answer in each historical dialogue in the historical multi-round dialogue, that is, the historical dialogue output. The answer preference alignment includes: The similarity score between each historical answer in the memory list and the current question is obtained through the retriever, and all the historical answers in the memory list are sorted and labeled according to the similarity score, wherein the types of labels include preferred matching historical answers and non-preferred matching historical answers, and the labeling standard can be set according to the actual situation. For example, when the similarity score between the historical answer and the current question is greater than a preset threshold, the historical answer is determined to be a preferred matching historical answer, otherwise it is a non-preferred matching historical answer; According to the similarity score between each historical answer in the memory list and the current question, the second preference score of the current question under each query in the memory list is obtained through the large language model using the second loss function, and all historical answers in the memory list are re-sorted according to the second preference score to obtain a set of historical answers that match the current question.

[0038] Among them, the large language model can be used as a preference reward model to obtain the preference score of each historical answer in the memory list and the current question, and then combined with the similarity score of each historical answer in the memory list and the current question and the preference score of each historical answer in the memory list and the current question, the second preference score of the current question under each query in the memory list is obtained through the second expression.

[0039] The second expression is: , In the second expression, Indicates current problem In the memory list The second preference score under queries, Represents the first Historical Answers With current issues The preference score of Indicates the first Historical Answers With current issues The similarity score of Represents the second parameter weight.

[0040] The expression of the second loss function is: , In the expression of the second loss function, Indicates current problem The answer prefers alignment loss, represents the number of samples in each batch, Represents the batch samples in a batch and The number of samples with the same label, including preferred matching historical answers and non-preferred matching historical answers, represents the indicator function, , , Represent the hidden representation of the word vector of the query, the preferred matching historical answer, and the non-preferred matching historical answer, respectively. Indicates the retrieval relevance adjustment parameter.

[0041] In one embodiment, preference alignment includes: According to the set of historical questions matching the current question and the set of historical answers matching the current question, combined with the self-attention reward iteration function and the policy gradient-based model update rule, preference alignment is performed to obtain the optimal historical question-answer pair matching the current question.

[0042] Among them, the expression of the self-attention reward iteration function is: , In the expression of the self-attention reward iteration function, represents the self-attention reward iteration function, represents the evaluation function, Represents the preferred historical conversation information for each iteration, Represents the non-preferred historical dialogue information of each iteration, where the historical dialogue information includes historical questions and historical answers.

[0043] The expression of the model update rule based on policy gradient is: , In the expression of the policy gradient-based model update rule, Indicates current problem The problem prefers alignment loss, Indicates current problem The answer prefers alignment loss, , They represent the adjustment parameters for the historical questions and answers in this iteration, represents the model parameters of the large language model, represents the learning rate, Represents the self-attention reward iteration function.

[0044] Step S130: Based on the current question and the best historical question-answer pair that matches the current question, the best answer to the current question is obtained through the large language model. That is, the current question and the best historical question-answer pair that matches the current question form a prompt, and the large language model can generate an answer based on the prompt.

[0045] In one embodiment, the multi-round dialogue optimization method based on a large language model provided by the present invention may further include other steps, such as: Updating the current question and the best answer to the memory list helps improve the accuracy of the large language model in answering the user's next question; Regularly clean up historical question and answer pairs in the memory list to improve the effectiveness and efficiency of the memory list.

[0046] The multi-round dialogue optimization method based on a large language model provided by the present invention first performs question preference alignment, answer preference alignment, and option preference alignment on the current question input by the user in a memory list before the large language model outputs an answer according to the current question input by the user, and uses a self-attention reward iteration function and a policy gradient-based model update rule to find the preferred matching historical question and the preferred matching historical answer for the current question, thereby forming an optimal historical question-answer pair for the current question. This method can overcome the context window limitation of the large language model and provide the large language model with accurate historical memory, greatly improving the context understanding and reasoning capabilities of the large language model in multi-round dialogues, enabling it to accurately output coherent and high-quality answers, thereby improving the quality of multi-round dialogues and ensuring user experience.

[0047] The multi-round dialogue optimization system based on a large language model provided by the present invention is described below. The multi-round dialogue optimization system based on a large language model described below and the multi-round dialogue optimization method based on a large language model described above can be referenced to each other.

[0048] Reference Figure 4 The present invention provides a multi-round dialogue optimization system based on a large language model, which may include: The receiving module is used to: receive the current question input by the user; An alignment module is used to: perform question preference alignment, answer preference alignment, and option preference alignment in the memory list according to the current question, and iterate this step until a preset stop condition is reached to obtain the optimal historical question-answer pair that matches the current question; The output module is used to obtain the optimal answer to the current question through a large language model based on the current question and the optimal historical question-answer pair that matches the current question.

[0049] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the multi-round dialogue optimization method based on the large language model, and the method includes: Receive user input for the current question; According to the current question, question preference alignment, answer preference alignment, and preference preference alignment are performed in the memory list, and this step is iterated until the preset stop condition is reached to obtain the optimal historical question-answer pair that matches the current question; Based on the current question and the best historical question-answer pair that matches the current question, the best answer to the current question is obtained through a large language model.

[0050] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0051] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the multi-round dialogue optimization method based on a large language model provided by the above methods, the method comprising: Receive user input for the current question; According to the current question, question preference alignment, answer preference alignment, and preference preference alignment are performed in the memory list, and this step is iterated until the preset stop condition is reached to obtain the optimal historical question-answer pair that matches the current question; Based on the current question and the best historical question-answer pair that matches the current question, the best answer to the current question is obtained through a large language model.

[0052] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the multi-round dialogue optimization method based on a large language model provided by the above methods, the method comprising: Receive user input for the current question; According to the current question, question preference alignment, answer preference alignment, and preference preference alignment are performed in the memory list, and this step is iterated until the preset stop condition is reached to obtain the optimal historical question-answer pair that matches the current question; Based on the current question and the best historical question-answer pair that matches the current question, the best answer to the current question is obtained through a large language model.

[0053] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0054] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-round dialogue optimization method based on a large language model, characterized in that: include: Receive user input for the current question; According to the current question, question preference alignment, answer preference alignment, and preference preference alignment are performed in the memory list, and this step is iterated until the preset stop condition is reached to obtain the optimal historical question-answer pair that matches the current question; Based on the current question and the best historical question-answer pair that matches the current question, the best answer to the current question is obtained through a large language model.

2. The multi-round dialogue optimization method based on a large language model according to claim 1 is characterized in that: The problem preference alignment includes: The similarity score between each historical question in the memory list and the current question is obtained through the retriever, and all the historical questions in the memory list are sorted and labeled according to the similarity score, wherein the types of labels include preferred matching historical questions and non-preferred matching historical questions; According to the similarity score between each historical question in the memory list and the current question, the first loss function is used to obtain the first preference score of the current question under each query in the memory list through the large language model, and all historical questions in the memory list are sorted again according to the first preference score to obtain a set of historical questions that match the current question.

3. The multi-round dialogue optimization method based on a large language model according to claim 2 is characterized in that: The method uses a first loss function and a large language model to obtain a first preference score of the current question under each query in the memory list according to the similarity score between each historical question and the current question in the memory list, including: The preference score between each historical question and the current question in the memory list is obtained by using a large language model as a preference reward model; Combining the similarity scores between each historical question in the memory list and the current question and the preference scores between each historical question in the memory list and the current question, the first preference score of the current question under each query in the memory list is obtained through the first expression; Among them, the first expression is: ， In the first expression, Indicates current problem In the memory list The first preference score under queries, Represents the first Historical Questions With current issues The preference score of Indicates the first Historical Questions With current issues The similarity score of represents the first parameter weight; The expression of the first loss function is: ， In the expression of the first loss function, Indicates current problem The problem prefers alignment loss, Represents the loss function It means averaging all samples. These variables are distributed from preference Medium sampling, Respectively represent historical issues , Historical issues The types of labels include preferred matching history issues and non-preferred matching history issues. represents the probability distribution of the output, represents the set of historical questions in the iteration process, Represents the sigmoid function.

4. The multi-round dialogue optimization method based on a large language model according to claim 2 is characterized in that: The answer preference alignment includes: The similarity score between each historical answer in the memory list and the current question is obtained through the retriever, and all the historical answers in the memory list are sorted and labeled according to the similarity score, wherein the types of labels include preferred matching historical answers and non-preferred matching historical answers; According to the similarity score between each historical answer in the memory list and the current question, the second preference score of the current question under each query in the memory list is obtained through the large language model using the second loss function, and all historical answers in the memory list are re-sorted according to the second preference score to obtain a set of historical answers that match the current question.

5. The multi-round dialogue optimization method based on a large language model according to claim 4 is characterized in that: The method uses a second loss function and a large language model to obtain a second preference score of the current question under each query in the memory list according to the similarity score between each historical answer in the memory list and the current question, including: The preference score between each historical answer and the current question in the memory list is obtained by using a large language model as a preference reward model. Combining the similarity score between each historical answer in the memory list and the current question and the preference score between each historical answer in the memory list and the current question, the second preference score of the current question under each query in the memory list is obtained through the second expression; The second expression is: ， In the second expression, Indicates current problem In the memory list The second preference score under queries, Represents the first Historical Answers With current issues The preference score of Indicates the first Historical Answers With current issues The similarity score of represents the second parameter weight; The expression of the second loss function is: ， In the expression of the second loss function, Indicates current problem The answer prefers alignment loss, represents the number of samples in each batch, Represents the batch samples in a batch and The number of samples with the same label, including preferred matching historical answers and non-preferred matching historical answers, represents the indicator function, , , Represent the hidden representation of the word vector of the query, the preferred matching historical answer, and the non-preferred matching historical answer, respectively. Indicates the retrieval relevance adjustment parameter.

6. The multi-round dialogue optimization method based on a large language model according to claim 4 is characterized in that: The preference alignment includes: According to the set of historical questions matching the current question and the set of historical answers matching the current question, combined with the self-attention reward iteration function and the policy gradient-based model update rule, preference alignment is performed to obtain the optimal historical question-answer pair matching the current question.

7. The multi-round dialogue optimization method based on a large language model according to claim 6 is characterized in that: The expression of the self-attention reward iteration function is: ， In the expression of the self-attention reward iteration function, represents the self-attention reward iteration function, represents the evaluation function, Represents the preferred historical conversation information for each iteration, Indicates the non-preferred historical dialogue information of each iteration, where the historical dialogue information includes historical questions and historical answers; The expression of the model update rule based on policy gradient is: ， In the expression of the policy gradient-based model update rule, Indicates current problem The problem prefers alignment loss, Indicates current problem The answer prefers alignment loss, , They represent the adjustment parameters for the historical questions and answers in this iteration, represents the model parameters of the large language model, represents the learning rate, Represents the self-attention reward iteration function.

8. A multi-round dialogue optimization system based on a large language model, characterized in that: include: The receiving module is used to: receive the current question input by the user; An alignment module is used to: perform question preference alignment, answer preference alignment, and option preference alignment in the memory list according to the current question, and iterate this step until a preset stop condition is reached to obtain the optimal historical question-answer pair that matches the current question; The output module is used to obtain the optimal answer to the current question through a large language model based on the current question and the optimal historical question-answer pair that matches the current question.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the multi-round dialogue optimization method based on a large language model is implemented as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-round dialogue optimization method based on a large language model is implemented as described in any one of claims 1 to 7.

Images