A multi-round dialogue optimization method, system, device and medium based on a large language model

Abstract

The present invention relates to the technical field of natural language processing and discloses a multi-round dialogue optimization method based on a large language model. Before the large language model outputs an answer based on the current question input by the user, the present invention first aligns the question preference, answer preference, and option preference of the current question input by the user in a memory list. By combining a self-attention reward iteration function and a model update rule based on a policy gradient, the method finds the optimal matching historical questions and optimal matching historical answers for the current question, forming an optimal historical question-answer pair for the current question. This method can overcome the context window limitation of the large language model and provide the large language model with accurate historical memory, greatly improving the context understanding and reasoning capabilities of the large language model in multi-round dialogues, enabling the model to accurately output coherent and high-quality answers, improving the quality of multi-round dialogues, and ensuring user experience.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a multi-round dialogue optimization method, system, device and medium based on a large language model. Background Art

[0002] Multi-turn conversation technology, a branch of natural language processing (NLP), has experienced rapid development in recent years with the rise of large language models. By learning from large amounts of conversation data, it can better understand context and generate coherent responses. Multi-turn conversations require large language models to possess stronger contextual understanding and reasoning capabilities to maintain coherence and consistency throughout the conversation. However, due to memory limitations, computational costs, and the architectural design of large language models, the length of conversation context that large language models can effectively process during multi-turn conversations is limited. As the number of conversation turns increases, large language models may forget previous context information. The gradual loss of historical conversation information can lead to factual errors, loss of contextual association, and a significant decrease in contextual understanding and reasoning capabilities, resulting in poor multi-turn conversation quality.

[0003] Therefore, there is an urgent need for a multi-round dialogue optimization method based on a large language model. Summary of the Invention

[0004] The present invention provides a multi-round dialogue optimization method, system, device and medium based on a large language model, which is used to solve the defect that the existing multi-round dialogue method based on a large language model lacks an effective historical dialogue processing mechanism, resulting in poor quality of multi-round dialogue.

[0005] The present invention provides a multi-round dialogue optimization method based on a large language model, comprising:

[0006] Receive the current question from the user;

[0007] Based on the current question, question preference alignment, answer preference alignment, and option preference alignment are performed in the memory list, and this step is iterated until the preset stopping condition is reached to obtain the optimal historical question-answer pair that matches the current question;

[0008] Based on the current question and the best historical question-answer pair that matches the current question, the optimal answer to the current question is obtained through a large language model.

[0009] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the question preference alignment includes:

[0010] The similarity score between each historical question in the memory list and the current question is obtained through the retriever, and all historical questions in the memory list are sorted and labeled according to the similarity score. The label types include preferred matching historical questions and non-preferred matching historical questions;

[0011] Based on the similarity score between each historical question in the memory list and the current question, the first loss function is used to obtain the first preference score of the current question under each query in the memory list through the large language model, and all historical questions in the memory list are sorted secondary according to the first preference score to obtain a set of historical questions that match the current question.

[0012] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the method uses a first loss function based on the similarity score between each historical question in the memory list and the current question to obtain a first preference score of the current question under each query in the memory list through the large language model, including:

[0013] The preference score of each historical question and the current question in the memory list is obtained by using a large language model as a preference reward model;

[0014] Combining the similarity scores between each historical question in the memory list and the current question and the preference scores between each historical question in the memory list and the current question, the first preference score of the current question under each query in the memory list is obtained through the first expression.

[0015] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the first expression is:

[0016] ,

[0017] In the first expression, Indicates the current problem In the memory list The first preference score under the query, Indicates the first historical issues and current issues The preference score of Indicates the first historical issues and current issues The similarity score of Represents the first parameter weight.

[0018] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the expression of the first loss function is:

[0019] ,

[0020] In the expression of the first loss function Indicates the current problem The problem preference alignment loss, represents the loss function, It means averaging all samples. Denotes that these variables are distributed from preference Medium sampling, Respectively represent historical issues , historical issues The types of labels include preferred matching history issues and non-preferred matching history issues. represents the probability distribution of the output, represents the set of historical problems in the iterative process, Represents the sigmoid function, which is used to map the input to values ​​between 0 and 1.

[0021] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the answer preference alignment includes:

[0022] The similarity score between each historical answer in the memory list and the current question is obtained through the retriever, and all historical answers in the memory list are sorted and labeled according to the similarity score, where the label types include preferred matching historical answers and non-preferred matching historical answers;

[0023] Based on the similarity score between each historical answer in the memory list and the current question, the second loss function is used to obtain the second preference score of the current question under each query in the memory list through the large language model, and all historical answers in the memory list are sorted secondary according to the second preference score to obtain a set of historical answers that match the current question.

[0024] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the method uses a second loss function based on the similarity score between each historical answer in the memory list and the current question to obtain a second preference score of the current question under each query in the memory list through the large language model, including:

[0025] The preference score between each historical answer and the current question in the memory list is obtained by using a large language model as a preference reward model;

[0026] Combining the similarity score between each historical answer in the memory list and the current question and the preference score between each historical answer in the memory list and the current question, the second preference score of the current question under each query in the memory list is obtained through the second expression.

[0027] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the second expression is:

[0028] ,

[0029] In the second expression, Indicates the current problem In the memory list The second preference score under queries, Indicates the first Historical answers and current issues The preference score of Indicates the first Historical answers and current issues The similarity score of Represents the second parameter weight.

[0030] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the expression of the second loss function is:

[0031] ,

[0032] In the expression of the second loss function, Indicates the current problem The answer prefers alignment loss, represents the number of samples in each batch, Represents the batch samples in a batch and The number of samples with the same label, including preferred matching historical answers and non-preferred matching historical answers, represents the indicator function, 、 、 Represent the hidden representation of the word vector of the query, the preferred matching historical answer, and the non-preferred matching historical answer, respectively. Indicates the retrieval relevance adjustment parameter.

[0033] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the preference alignment includes:

[0034] Based on the set of historical questions and the set of historical answers that match the current question, combined with the self-attention reward iteration function and the policy gradient-based model update rule, preference alignment is performed to obtain the optimal historical question-answer pair that matches the current question.

[0035] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the expression of the self-attention reward iteration function is:

[0036] ,

[0037] In the expression of the self-attention reward iteration function, represents the self-attention reward iteration function, represents the evaluation function, Indicates the preferred historical conversation information for each iteration, Indicates the non-preferred historical conversation information for each iteration. The historical conversation information includes historical questions and historical answers.

[0038] According to a multi-round dialogue optimization method based on a large language model provided by the present invention, the expression of the model update rule based on policy gradient is:

[0039] ,

[0040] In the expression of the policy gradient-based model update rule, Indicates the current problem The problem preference alignment loss, Indicates the current problem The answer prefers alignment loss, 、 They represent the tendency adjustment parameters of this iteration for historical questions and historical answers, represents the model parameters of the large language model, represents the learning rate, Represents the self-attention reward iteration function.

[0041] A multi-round dialogue optimization method based on a large language model provided by the present invention also includes:

[0042] Update the current question and best answer to the memory list.

[0043] The present invention also provides a multi-round dialogue optimization system based on a large language model, comprising:

[0044] The receiving module is used to: receive the current question input by the user;

[0045] The alignment module is used to: align the question preference, answer preference, and option preference in the memory list according to the current question, and iterate this step until the preset stopping condition is reached to obtain the optimal historical question-answer pair that matches the current question;

[0046] The output module is used to obtain the optimal answer to the current question based on the current question and the optimal historical question-answer pair that matches the current question through a large language model.

[0047] The present invention also provides an electronic device, comprising a processor and a memory storing a computer program, characterized in that when the processor executes the computer program, it implements any of the above-mentioned multi-round dialogue optimization methods based on a large language model.

[0048] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned multi-round dialogue optimization methods based on a large language model.

[0049] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute any of the above-mentioned multi-round dialogue optimization methods based on a large language model.

[0050] The present invention provides a multi-round dialogue optimization method, system, device and medium based on a large language model. Before the large language model outputs an answer according to the current question input by the user, the current question input by the user is first aligned with the question preference, answer preference and option preference in the memory list. Combined with the self-attention reward iteration function and the policy gradient-based model update rule, the preferred matching historical question and the preferred matching historical answer for the current question are found to form the optimal historical question-answer pair for the current question. This can overcome the context window limitation of the large language model and provide the large language model with accurate historical memory, greatly improving the context understanding and reasoning ability of the large language model in multi-round dialogues, enabling it to accurately output coherent and high-quality answers, improve the quality of multi-round dialogues, and ensure user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 This is one of the flow charts of a multi-round dialogue optimization method based on a large language model provided by the present invention.

[0053] Figure 2 This is the second flow chart of a multi-round dialogue optimization method based on a large language model provided by the present invention.

[0054] Figure 3Shows the relationship between question preference alignment, answer preference alignment, and preference preference alignment.

[0055] Figure 4 This is a structural diagram of a multi-round dialogue optimization system based on a large language model provided by the present invention.

[0056] Figure 5 This is a schematic structural diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the embodiments described are part of the embodiments of the present invention, not all of the embodiments, and they should not be understood as limitations on the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In the description of the present invention, it should be understood that the terms used are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0058] The following combination Figure 1-Figure 5 The present invention describes the multi-turn dialogue optimization method, system, device, and medium based on a large language model. It should be noted that the multi-turn dialogue optimization method based on a large language model provided by the present invention can be executed by any network-side device or terminal-side device that meets the technical requirements, such as a multi-turn dialogue optimization device based on a large language model.

[0059] Figure 1-Figure 2 This is a flow chart of the multi-round dialogue optimization method based on a large language model provided by the present invention. Figure 1 The present invention provides a multi-round dialogue optimization method based on a large language model, which may include:

[0060] Step S110: receiving the current question input by the user.

[0061] Step S120: Based on the current question, perform question preference alignment, answer preference alignment, and option preference alignment in the memory list, and iterate this step until the preset stop condition is reached to obtain the optimal historical question-answer pair that matches the current question. The preset stop condition can be set according to actual conditions, such as the preset number of iterations. The relationship between question preference alignment, answer preference alignment, and option preference alignment can be seen in Figure 3 .

[0062] In one embodiment, question preference alignment includes:

[0063] The similarity score between each historical question in the memory list and the current question is obtained through the retriever, and all historical questions in the memory list are sorted and labeled according to the similarity score. The types of labels include preferred matching historical questions and non-preferred matching historical questions. The labeling standard can be set according to the actual situation. For example, if the similarity score between the historical question and the current question is greater than a preset threshold, the historical question is determined to be a preferred matching historical question, otherwise it is a non-preferred matching historical question.

[0064] Based on the similarity score between each historical question in the memory list and the current question, the first loss function is used to obtain the first preference score of the current question under each query in the memory list through the large language model, and all historical questions in the memory list are sorted secondary according to the first preference score to obtain a set of historical questions that match the current question.

[0065] Among them, the preference score of each historical question and the current question in the memory list can be obtained by using a large language model as a preference reward model, and then combined with the similarity score of each historical question and the current question in the memory list and the preference score of each historical question and the current question in the memory list, the first preference score of the current question under each query in the memory list is obtained through the first expression.

[0066] The first expression is:

[0067] ,

[0068] In the first expression, Indicates the current problem In the memory list The first preference score under the query, Indicates the first historical issues and current issues The preference score of Indicates the first historical issues and current issues The similarity score of Represents the first parameter weight.

[0069] The expression of the first loss function is:

[0070] ,

[0071] In the expression of the first loss function, Indicates the current problem The problem preference alignment loss, represents the loss function, It means averaging all samples. Denotes that these variables are distributed from preference Medium sampling, Respectively represent historical issues , historical issues The types of labels include preferred matching history issues and non-preferred matching history issues. represents the probability distribution of the output, represents the set of historical problems in the iterative process, Represents the sigmoid function, which is used to map the input to values ​​between 0 and 1.

[0072] In one embodiment, the answer in the answer preference alignment can be the answer in each historical dialogue in the historical multi-round dialogue, that is, the historical dialogue output. Answer preference alignment includes:

[0073] The similarity score between each historical answer in the memory list and the current question is obtained through the retriever, and all historical answers in the memory list are sorted and labeled according to the similarity score. The types of labels include preferred matching historical answers and non-preferred matching historical answers. The labeling standard can be set according to the actual situation. For example, when the similarity score between the historical answer and the current question is greater than a preset threshold, the historical answer is determined to be a preferred matching historical answer, otherwise it is a non-preferred matching historical answer.

[0074] Based on the similarity score between each historical answer in the memory list and the current question, the second loss function is used to obtain the second preference score of the current question under each query in the memory list through the large language model, and all historical answers in the memory list are sorted secondary according to the second preference score to obtain a set of historical answers that match the current question.

[0075] Among them, the large language model can be used as a preference reward model to obtain the preference score of each historical answer in the memory list and the current question, and then combined with the similarity score of each historical answer in the memory list and the current question and the preference score of each historical answer in the memory list and the current question, the second preference score of the current question under each query in the memory list is obtained through the second expression.

[0076] The second expression is:

[0077] ,

[0078] In the second expression, Indicates the current problem In the memory list The second preference score under queries, Indicates the first Historical answers and current issues The preference score of Indicates the first Historical answers and current issues The similarity score of Represents the second parameter weight.

[0079] The expression of the second loss function is:

[0080] ,

[0081] In the expression of the second loss function, Indicates the current problem The answer prefers alignment loss, represents the number of samples in each batch, Represents the batch samples in a batch and The number of samples with the same label, including preferred matching historical answers and non-preferred matching historical answers, represents the indicator function, 、 、 Represent the hidden representation of the word vector of the query, the preferred matching historical answer, and the non-preferred matching historical answer, respectively. Indicates the retrieval relevance adjustment parameter.

[0082] In one embodiment, preference alignment includes:

[0083] Based on the set of historical questions and the set of historical answers that match the current question, combined with the self-attention reward iteration function and the policy gradient-based model update rule, preference alignment is performed to obtain the optimal historical question-answer pair that matches the current question.

[0084] Among them, the expression of the self-attention reward iteration function is:

[0085] ,

[0086] In the expression of the self-attention reward iteration function, represents the self-attention reward iteration function, represents the evaluation function, Indicates the preferred historical conversation information for each iteration, Indicates the non-preferred historical conversation information for each iteration. The historical conversation information includes historical questions and historical answers.

[0087] The expression of the model update rule based on policy gradient is:

[0088] ,

[0089] In the expression of the policy gradient-based model update rule, Indicates the current problem The problem preference alignment loss, Indicates the current problem The answer prefers alignment loss, 、 They represent the tendency adjustment parameters of this iteration for historical questions and historical answers, represents the model parameters of the large language model, represents the learning rate, Represents the self-attention reward iteration function.

[0090] Step S130: Based on the current question and the optimal historical question-answer pair that matches the current question, the large language model is used to obtain the optimal answer to the current question. That is, the current question and the optimal historical question-answer pair that matches the current question are formed into a prompt, and the large language model can generate an answer based on the prompt.

[0091] In one embodiment, the multi-turn dialogue optimization method based on a large language model provided by the present invention may further include other steps, such as:

[0092] Updating the current question and the best answer to the memory list helps improve the accuracy of the large language model in answering the user's next question;

[0093] Regularly clean up historical question-and-answer pairs in the memory list to improve the effectiveness and efficiency of the memory list.

[0094] The multi-round dialogue optimization method based on a large language model provided by the present invention first performs question preference alignment, answer preference alignment, and option preference alignment on the current question input by the user in a memory list before the large language model outputs an answer based on the current question input by the user. The self-attention reward iteration function and the policy gradient-based model update rule are used to find the optimal matching historical questions and the optimal matching historical answers for the current question, forming the optimal historical question-answer pair for the current question. This method can overcome the context window limitation of the large language model and provide the large language model with accurate historical memory, greatly improving the context understanding and reasoning capabilities of the large language model in multi-round dialogues, enabling it to accurately output coherent and high-quality answers, improve the quality of multi-round dialogues, and ensure user experience.

[0095] The following describes the multi-round dialogue optimization system based on a large language model provided by the present invention. The multi-round dialogue optimization system based on a large language model described below and the multi-round dialogue optimization method based on a large language model described above can be referenced to each other.

[0096] Reference Figure 4 The present invention provides a multi-round dialogue optimization system based on a large language model, which may include:

[0097] The receiving module is used to: receive the current question input by the user;

[0098] The alignment module is used to: align the question preference, answer preference, and option preference in the memory list according to the current question, and iterate this step until the preset stopping condition is reached to obtain the optimal historical question-answer pair that matches the current question;

[0099] The output module is used to obtain the optimal answer to the current question based on the current question and the optimal historical question-answer pair that matches the current question through a large language model.

[0100] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a multi-turn dialogue optimization method based on a large language model, which includes:

[0101] Receive the current question from the user;

[0102] Based on the current question, question preference alignment, answer preference alignment, and option preference alignment are performed in the memory list, and this step is iterated until the preset stopping condition is reached to obtain the optimal historical question-answer pair that matches the current question;

[0103] Based on the current question and the best historical question-answer pair that matches the current question, the optimal answer to the current question is obtained through a large language model.

[0104] Furthermore, the logic instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0105] In another aspect, the present invention further provides a computer program product, comprising a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the multi-turn dialogue optimization method based on a large language model provided by the above methods, the method comprising:

[0106] Receive the current question from the user;

[0107] Based on the current question, question preference alignment, answer preference alignment, and option preference alignment are performed in the memory list, and this step is iterated until the preset stopping condition is reached to obtain the optimal historical question-answer pair that matches the current question;

[0108] Based on the current question and the best historical question-answer pair that matches the current question, the optimal answer to the current question is obtained through a large language model.

[0109] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for optimizing multi-turn dialogue based on a large language model provided by the above methods is implemented. The method includes:

[0110] Receive the current question from the user;

[0111] Based on the current question, question preference alignment, answer preference alignment, and option preference alignment are performed in the memory list, and this step is iterated until the preset stopping condition is reached to obtain the optimal historical question-answer pair that matches the current question;

[0112] Based on the current question and the best historical question-answer pair that matches the current question, the optimal answer to the current question is obtained through a large language model.

[0113] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0114] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A multi-round dialogue optimization method based on a large language model, characterized in that: include: Receive user input for the current question; Based on the current question, question preference alignment, answer preference alignment, and option preference alignment are performed in the memory list, and this step is iterated until the preset stopping condition is reached to obtain the optimal historical question-answer pair that matches the current question; Based on the current question and the best historical question-answer pair that matches the current question, the optimal answer to the current question is obtained through a large language model. The problem preference alignment includes: The similarity score between each historical question in the memory list and the current question is obtained through the retriever, and all historical questions in the memory list are sorted and labeled according to the similarity score. The label types include preferred matching historical questions and non-preferred matching historical questions; Based on the similarity score between each historical question in the memory list and the current question, the first loss function is used to obtain the first preference score of the current question under each query in the memory list through the large language model, and all historical questions in the memory list are sorted secondary according to the first preference score to obtain a set of historical questions that match the current question.

2. The multi-round dialogue optimization method based on a large language model according to claim 1 is characterized in that: The method uses a first loss function and a large language model to obtain a first preference score of the current question under each query in the memory list based on the similarity score between each historical question and the current question in the memory list, including: The preference score of each historical question and the current question in the memory list is obtained by using a large language model as a preference reward model; Combining the similarity scores between each historical question in the memory list and the current question and the preference scores between each historical question in the memory list and the current question, the first preference score of the current question under each query in the memory list is obtained through the first expression; Among them, the first expression is: s i =a·r θ (q,Q i )+(1-a)·s R (q,Q i ), In the first expression, s i represents the first preference score of the current question q under the i-th query in the memory list, r θ (q,Q i ) represents the i-th historical question Q in the memory list given by the large language model i The preference score of the current question q, s R (q,Q i ) represents the i-th historical question Q in the memory list given by the retriever i The similarity score with the current question q, a represents the first parameter weight; The expression of the first loss function is: In the expression of the first loss function, L question represents the problem preference alignment loss of the current problem q, E represents the loss function, Indicates the average of all samples, (q,Q w ,Q l ,y w ,y l )~Q pref Denote these variables from the preference distribution Q pref mid-sampling, y w 、y l Represents historical questions Q w ,,Historical QuestionsQ l The types of labels include preferred matching history issues and non-preferred matching history issues, ρ θ represents the probability distribution of the output, Q pref Represents the set of historical problems in the iterative process, and σ represents the sigmoid function.

3. The multi-round dialogue optimization method based on a large language model according to claim 1 is characterized in that: The answer preference alignment includes: The similarity score between each historical answer in the memory list and the current question is obtained through the retriever, and all historical answers in the memory list are sorted and labeled according to the similarity score, where the label types include preferred matching historical answers and non-preferred matching historical answers; Based on the similarity score between each historical answer in the memory list and the current question, the second loss function is used to obtain the second preference score of the current question under each query in the memory list through the large language model, and all historical answers in the memory list are sorted secondary according to the second preference score to obtain a set of historical answers that match the current question.

4. The multi-round dialogue optimization method based on a large language model according to claim 3 is characterized in that: The method uses a second loss function and a large language model to obtain a second preference score of the current question under each query in the memory list based on the similarity score between each historical answer and the current question in the memory list, including: The preference score between each historical answer and the current question in the memory list is obtained by using a large language model as a preference reward model; Combining the similarity score between each historical answer in the memory list and the current question and the preference score between each historical answer in the memory list and the current question, the second preference score of the current question under each query in the memory list is obtained through the second expression; The second expression is: s′ i =a′·r θ (q,A i )+(1-a′)·s R (q,A i ), In the second expression, s′ i represents the second preference score of the current question q under the i-th query in the memory list, r θ (q,A i ) represents the i-th historical answer A in the memory list given by the large language model i The preference score of the current question q, s R (q,A i ) represents the i-th historical answer A in the memory list given by the retriever i The similarity score with the current question q, a′ represents the second parameter weight; The expression of the second loss function is: In the expression of the second loss function, L answer represents the answer preference alignment loss for the current question q, N t Indicates the number of samples in each batch, N yi Represents the batch samples and y in a batch i The number of samples with the same label. The types of labels include preferred matching historical answers and non-preferred matching historical answers. I represents the indicator function, and h q 、h p 、h n They represent the hidden representation of the word vectors of the query, the preferred matching historical answer, and the non-preferred matching historical answer, respectively, and T represents the retrieval relevance adjustment parameter.

5. The multi-round dialogue optimization method based on a large language model according to claim 3 is characterized in that: The preference alignment includes: Based on the set of historical questions and the set of historical answers that match the current question, combined with the self-attention reward iteration function and the policy gradient-based model update rule, preference alignment is performed to obtain the optimal historical question-answer pair that matches the current question.

6. The multi-round dialogue optimization method based on a large language model according to claim 5 is characterized in that: The expression of the self-attention reward iteration function is: In the expression of the self-attention reward iteration function, R t represents the self-attention reward iteration function, f represents the evaluation function, y t Indicates the preferred historical conversation information for each iteration, Represents the non-preferred historical conversation information of each iteration, which includes historical questions and historical answers; The expression of the model update rule based on policy gradient is: In the expression of the model update rule based on policy gradient, L question represents the question preference alignment loss of the current question q, L answer represents the answer preference alignment loss for the current question q, C1 and C2 represent the tendency adjustment parameters for historical questions and historical answers in this iteration, θ represents the model parameters of the large language model, α represents the learning rate, and R t Represents the self-attention reward iteration function.

7. A multi-round dialogue optimization system based on a large language model, characterized by: include: The receiving module is used to: receive the current question input by the user; The alignment module is used to: align the question preference, answer preference, and option preference in the memory list according to the current question, and iterate this step until the preset stopping condition is reached to obtain the optimal historical question-answer pair that matches the current question; The problem preference alignment includes: The similarity score between each historical question in the memory list and the current question is obtained through the retriever, and all historical questions in the memory list are sorted and labeled according to the similarity score. The label types include preferred matching historical questions and non-preferred matching historical questions; Based on the similarity score between each historical question in the memory list and the current question, the first loss function is used to obtain the first preference score of the current question under each query in the memory list through the large language model. All historical questions in the memory list are then sorted again based on the first preference score to obtain a set of historical questions that match the current question. The output module is used to obtain the optimal answer to the current question based on the current question and the optimal historical question-answer pair that matches the current question through a large language model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the multi-round dialogue optimization method based on a large language model is implemented as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-round dialogue optimization method based on a large language model is implemented.

Images