A method and device for optimizing and evaluating multi-turn dialogue of a large language model based on reinforcement learning

Abstract

This application discloses a method and apparatus for optimizing and evaluating multi-turn dialogues using a large language model based on reinforcement learning. Belonging to the field of reinforcement learning, this application addresses the significant performance degradation of existing large language models in multi-turn dialogues by proposing a reinforcement learning-driven dialogue guidance mechanism. This mechanism enables the large language model to refuse to answer or actively discuss the question further when the instruction conditions are insufficient, thereby improving the answer accuracy of the large language model, reducing hallucination responses, and enhancing its multi-turn dialogue capabilities. Specifically, this application constructs a reward model to determine whether the output semantics and answer results of the large language model are consistent with the expected results, and performs reinforcement learning fine-tuning on the large language model based on a group relative policy optimization algorithm. Experimental results show that the method of this application significantly reduces the hallucination response rate of the model when conditions are insufficient, and effectively improves the multi-turn dialogue capabilities of multiple large language models.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically relating to a method and device for optimizing and evaluating multi-turn dialogues using a large language model based on reinforcement learning. Background Technology

[0002] Modern large language models (LLMs), such as ChatGPT, Gemini, and Claude, serve as conversational interfaces, allowing users to interact with the model through multi-turn dialogues. This interaction method is effective not only when users have clear needs (i.e., can fully express their needs through commands) but also when their needs are unclear. In such scenarios, users may initiate a dialogue with vague commands and then gradually clarify their needs through multiple rounds of interaction. In experiments conducted by other work (Laban P, Hayashi H, Zhou Y, Neville J. LLMs get lost in multi-turn conversation[J]. arXiv preprint arXiv:2505.06120,2025.), it was observed that the average performance of models participating in multiple rounds of incomplete dialogue was 65%—a decrease of 25 percentage points compared to the 90% performance of a single-turn dialogue when a complete command is obtained at the beginning of the dialogue. Notably, this performance degradation was observed in all tested large language models, from the lightweight open-source model (LLama3.1-8B-Instruct) to the state-of-the-art model (Gemini 2.5Pro).

[0003] Ambiguity in multi-turn interaction behaviors and user commands is common, so the model's ability in multi-turn interaction scenarios is particularly important. In view of the problems existing in the current large language models, this invention proposes a technical solution to solve the above problems. Summary of the Invention

[0004] The main objective of this invention is to overcome the shortcomings of existing technologies and provide a method and device for optimizing and evaluating multi-turn dialogues using a large language model based on reinforcement learning. To achieve the above objective, this invention adopts the following technical solution:

[0005] This invention relates to a method for optimizing and evaluating multi-turn dialogues using a large language model based on reinforcement learning. Specifically, the method includes the following main steps:

[0006] S1: Construct an original dataset containing multi-dimensional task instructions, generate an incomplete instruction set by removing at least one condition from the task instructions, merge the incomplete instruction set with the original dataset to form an augmented dataset, and divide the augmented dataset into a training set and a test set according to a predetermined ratio.

[0007] S2: Construct a reward model and fine-tune the target large language model using reinforcement learning;

[0008] S3: Systematically evaluate the multi-turn dialogue capabilities of the trained target large language model.

[0009] Through the above steps, the present invention aims to enhance the ability of a target large language model to refuse to answer or further discuss when faced with insufficient instructions, and thus systematically evaluate the model's multi-turn dialogue capability.

[0010] Specifically, in step S1, an original dataset is constructed from task instruction data of four different dimensions: code instructions, query instructions, service call instructions, or mathematical calculation instructions. Then, the large language model API is called, and all task instructions in the dataset are input in sequence, so that it returns the task instructions after removing at least one condition. Finally, all processed task instructions are manually reviewed. If no condition is removed, at least one condition of the task instruction is removed manually.

[0011] Furthermore, all processed data are added to the original dataset and merged into an augmented dataset. The answer field of data lacking conditions is set to a null identifier to indicate that the data is a task instruction with insufficient conditions. The augmented dataset is divided into training and test sets according to a preset ratio.

[0012] Specifically, in step S2, to guide the target large language model to refuse to answer or engage in further discussion when faced with insufficient task instructions, a reward model that includes both semantic and answer correctness rewards is designed. When the target large language model faces insufficient task instructions, the large language model API is called as a discriminant model to determine the semantic type of the model's answer. Subsequently, a semantic reward is given based on the task instruction type and the semantic type of the model's answer. The semantic reward S... semantics The calculation method is as follows:

[0013]

[0014] The correctness reward is used as the second component to evaluate the correctness of the target large language model's response. When the model faces a well-defined task instruction, only the correctness of the model's response is considered, and the correctness reward S is calculated. answer The calculation method is as follows:

[0015]

[0016] Furthermore, the Group Relative Policy Optimization (GRPO) algorithm is used to fine-tune the target large language model using reinforcement learning based on the above reward model. The specific process includes the following steps:

[0017] At the start of training, the reference model π will be used. ref Set as the target large language model π θ The parameters are kept constant during the iteration process, serving as a reference for the target large language model in subsequent optimizations to prevent excessive optimization; reward model Consisting of two reward functions S semantics S answer composition;

[0018] For each input instruction Q, use the current target large language model π θ Generate G distinct responses O1, O2…O G And record the probability π generated at each time step in each response. θ (o i,t |Q,o i,<t );

[0019] For each complete response O i Using a reward model Scoring is performed to obtain r1, r2...r G Further calculate relative advantage Its formula is defined as follows:

[0020]

[0021] Where mean(·) is the mean operation and std(·) is the standard deviation operation;

[0022] Finally, the GRPO algorithm was used to optimize the target large language model π. θ Its formula is defined as follows:

[0023]

[0024] in:

[0025]

[0026] Among them, D KL Let β represent the KL divergence, and πθ be a hyperparameter. old For the old target large language model during the iteration process, r i,t (θ) represents the probability ratio generated by the new and old target large language models, clip(·) is the clipping function, and ε is the clipping parameter.

[0027] Specifically, in step S3, the multi-turn dialogue capability of the fine-tuned target large language model is systematically evaluated. The specific process includes the following steps:

[0028] On a pre-defined test set, conditional instructions are input into the model in different rounds. First, the question is input, followed by one of the conditions of the question after each model's response. The model's hallucination response rate is evaluated, excluding the final round of responses. The hallucination response rate is determined by calling the Large Language Model API, and the formula for the hallucination response rate is defined as follows:

[0029]

[0030] Among them, N1 hallucination N1 represents the number of samples of the model's hallucination responses in the intermediate rounds. total The total number of samples that answered the questions in the intermediate rounds of the model;

[0031] After all conditions are entered, the final round of evaluation is conducted to determine whether the model's response meets expectations. The evaluation metric is the accuracy of the response, and its formula is defined as:

[0032]

[0033] Among them, N2 right N2 represents the number of correct samples in the last round of responses. total This represents the total number of samples in the final round of responses.

[0034] Optionally, the large language model APIs called in steps S1, S2 and S3 include, but are not limited to, calls to the APIs of DeepSeekV3 and Gemini 2.5Pro.

[0035] This invention also provides a device for optimizing and evaluating multi-turn dialogues using a large language model based on reinforcement learning, comprising the following modules:

[0036] The data construction module is used to build datasets containing both complete and incomplete task instructions;

[0037] The reward and optimization module is used to build a reward model and perform reinforcement learning fine-tuning on the target large language model.

[0038] The assessment module is used to evaluate multi-round dialogue capabilities.

[0039] The method of this invention constructs a multi-dimensional dataset containing insufficient condition instructions, builds a reward model using semantic rewards and answer correctness rewards, and fine-tunes the model through reinforcement learning to reduce phantom responses when faced with insufficient condition instructions. Experimental results show that this invention demonstrates a significant performance improvement in the evaluation of multi-turn dialogue capabilities. Attached Figure Description

[0040] Figure 1 A flowchart illustrating a multi-turn dialogue optimization and evaluation method based on reinforcement learning for a large language model provided by this invention;

[0041] Figure 2 This is a schematic diagram of the structure of a multi-turn dialogue optimization and evaluation device based on a large language model using reinforcement learning, provided by the present invention. Detailed Implementation

[0042] The present application will be described in detail below with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present application. Any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the protection scope of the present application.

[0043] Reference Figure 1 This invention proposes a method for optimizing and evaluating multi-turn dialogues using a large language model based on reinforcement learning. The steps are as follows:

[0044] S1: Construct an original dataset containing multi-dimensional task instructions, generate an incomplete instruction set by removing at least one condition from the task instructions, merge the incomplete instruction set with the original dataset to form an augmented dataset, and divide the augmented dataset into a training set and a test set according to a predetermined ratio.

[0045] S2: Construct a reward model and fine-tune the target large language model using reinforcement learning;

[0046] S3: Systematically evaluate the multi-turn dialogue capabilities of the trained target large language model.

[0047] Specifically, in step S1, an original dataset is constructed from task instruction data of four different dimensions: code instructions, query instructions, service call instructions, and mathematical calculation instructions. This dataset includes HumanEval&LiveCodeBench, Spider, Berkeley Function Calling Leaderboard, and GSM8K. Then, the DeepSeek V3 API is called, and all task instructions from the dataset are input sequentially. The prompt "This instruction consists of a question and multiple conditions. Please remove one of the conditions and then return the new instruction, outputting only the final result.\nThe instruction is:" prompts the API to return the task instruction after removing one condition. Finally, all processed task instructions are manually reviewed. If a condition is not met, such as if it was not removed, the condition for that task instruction is manually removed.

[0048] Furthermore, all processed data are added to the original dataset and merged into an augmented dataset. For data lacking conditions, the content of answer fields such as "answer" and "solution" is set to "null" to indicate that the data is a task instruction with insufficient conditions. The augmented dataset is then divided into training and test sets in a 9:1 ratio.

[0049] Specifically, in step S2, in order to guide the target large language model to refuse to answer or make further discussions when faced with task instructions with insufficient conditions, a reward model that includes both semantic and answer correctness rewards was designed.

[0050] When the model faces a task instruction with insufficient conditions, DeepseekV3 is used to discriminate the model. An API call is made to determine the semantic type of the model's response. The prompt is: "The following is a response to an incomplete question. If it mentions that the conditions of the question are incomplete and does not provide an answer, reply to me with 'yes'. If it still provides an answer despite the incomplete conditions, reply to me with 'no'. Only tell me 'yes' or 'no'. The response to an incomplete question:\n". Subsequently, a semantic reward is given based on the task instruction type and the semantic type of the model's response. The semantic reward S semantics The calculation method is as follows:

[0051]

[0052] The correctness reward is used as the second component to evaluate the correctness of the target large language model's response. When the model faces a well-defined task instruction, only the correctness of the model's response is considered, and the answer reward S... answer The calculation method is as follows:

[0053]

[0054] Furthermore, the Group Relative Policy Optimization (GRPO) algorithm is used to fine-tune the target large language model using reinforcement learning based on the above reward model. The specific process includes the following steps:

[0055] At the start of training, the reference model π will be used. ref Set as the target large language model π θ The parameters are kept constant during the iteration process, serving as a reference for the target large language model in subsequent optimizations to prevent excessive optimization; reward model. Consisting of two reward functions S semantics S answer composition;

[0056] For each input instruction Q, use the current target large language model π θ Generate G distinct responses O1, O2…O G And record the probability π generated at each time step in each response. θ (o i,t |Q,o i,<t );

[0057] For each complete response O i Using a reward model Scoring is performed to obtain r1, r2...r G Further calculate relative advantage Its formula is defined as follows:

[0058]

[0059] Where mean(·) is the mean operation and std(·) is the standard deviation operation;

[0060] Finally, the GRPO algorithm was used to optimize the target large language model π. θ Its formula is defined as follows:

[0061]

[0062] in:

[0063]

[0064] Among them, D KL Let β represent the KL divergence, and πθ be a hyperparameter. old For the old target large language model during the iteration process, r i,t (θ) represents the probability ratio generated by the new and old target large language models, clip(·) is the clipping function, and ε is the clipping parameter.

[0065] Specifically, in step S3, the multi-turn dialogue capability of the fine-tuned target large language model is systematically evaluated. The specific process includes the following steps:

[0066] On a pre-defined test set, complete conditional instructions were input into the model in different rounds. First, the question was input, followed by one of the conditions of the question after each model's response. The model's hallucination response rate was evaluated, excluding the final round. The hallucination response rate was determined by calling the DeepSeek V3 API, with the prompt: "The following is a response to an incomplete question. If it mentions that the conditions of the question are incomplete and does not provide an answer, reply to me with 'yes'. If it still provides an answer despite the incomplete conditions, reply to me with 'no'. Only tell me 'yes' or 'no'.\nthe response to an incomplete question:\n". The hallucination response rate formula is defined as follows:

[0067]

[0068] Among them, N1 hallucination N1 represents the number of samples of the model's hallucination responses in the intermediate rounds. total The total number of samples that answered the questions in the intermediate rounds of the model;

[0069] After all conditions are entered, the final round of evaluation is conducted to determine whether the model's response meets expectations. The evaluation metric is the accuracy of the response, and its formula is defined as:

[0070]

[0071] Among them, N2 right N2 represents the number of correct samples in the last round of responses. total The total number of samples in the final round of responses;

[0072] This embodiment of the invention uses PyTorch 2.6.0 in Python 3.10 to conduct experiments on two Nvidia A100-80G GPUs, and uses Qwen2.5-7B-Instruct and Qwen2.5-1.5B-Instruct as target large language models for fine-tuning. This embodiment directly trains the model for 600 steps with an initial learning rate of 1e-5, a maximum response length of 1024, a batch size of 2, and 8 inter-group generation numbers, and uses reinforcement learning to fine-tune all parameters of the model. For further comparison, this method performs 600 steps of ordinary reinforcement learning fine-tuning (VanillaGRPO) on two target large language models on unprocessed datasets of the same dimension, using only the correctness of the response as the reward function, and keeping other experimental settings the same.

[0073] In this embodiment, the method of the present invention is compared with ordinary reinforcement learning fine-tuning.

[0074] This invention uses the hallucination response rate (Hal.) and accuracy (Acc.) to evaluate the model's multi-turn dialogue ability.

[0075] Table 1 shows a comparison of the multi-turn dialogue capabilities between the model trained by the method of this invention and the original model fine-tuned using ordinary reinforcement learning.

[0076] Table 1: Comparison of the multi-turn dialogue capability between the present invention and the ordinary reinforcement learning fine-tuning original model.

[0077] Method Name Hal. Acc. Qwen2.5-7B-Instruct 76.46 35.35 +VanillaGRPO 74.10 44.18 +Ours 56.89 47.16 Qwen2.5-1.5B-Instruct 88.56 17.00 +VanillaGRPO 81.43 26.09 +Ours 67.84 28.43

[0078] Please see Figure 2 As shown, the present invention also provides a device for optimizing and evaluating multi-turn dialogues using a large language model based on reinforcement learning, the device comprising the following modules:

[0079] The data construction module is used to build datasets containing both complete and incomplete task instructions;

[0080] The reward and optimization module is used to build a reward model and perform reinforcement learning fine-tuning on the target large language model.

[0081] The assessment module is used to evaluate multi-round dialogue capabilities.

[0082] The operation and effect of the reinforcement learning-based large language model multi-turn dialogue optimization and evaluation device of the present invention are consistent with the above-mentioned reinforcement learning-based large language model multi-turn dialogue optimization and evaluation method. Therefore, the reinforcement learning-based large language model multi-turn dialogue optimization and evaluation device will not be described again here.

[0083] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can be implemented in the form of a computer program product on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Other embodiments may also be used. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimizing and evaluating multi-turn dialogues using a large language model based on reinforcement learning, characterized in that, Includes the following steps: S1: Construct an original dataset containing multi-dimensional task instructions, generate an incomplete instruction set by removing at least one condition from the task instructions, merge the incomplete instruction set with the original dataset to form an enhanced dataset, and divide the enhanced dataset into a training set and a test set according to a predetermined ratio. S2: Construct a reward model and fine-tune the target large language model using reinforcement learning; S3: Systematically evaluate the multi-turn dialogue capabilities of the trained target large language model.

2. The method for optimizing and evaluating multi-turn dialogues based on a large language model using reinforcement learning according to claim 1, characterized in that, In step S1: The multi-dimensional task instructions include at least one of code instructions, query instructions, service call instructions, or mathematical calculation instructions; The code instructions include program code generation task instructions in Python, Java, or C++ languages; The query instruction is a structured query language generation task instruction; The service call instruction is an application interface call to generate a task instruction; The mathematical calculation instructions are mathematical solution task instructions; The removal includes processing with a large language model and manual review, removing at least one condition of the task instruction to make it incomplete; The answer field of the incomplete instruction set is set to a null identifier to indicate that the task instruction is insufficient.

3. The method for optimizing and evaluating multi-turn dialogues based on a large language model using reinforcement learning according to claim 1, characterized in that, In step S2: The reward model includes semantic rewards and answer accuracy rewards; The semantic reward is given for task instructions with insufficient conditions. If the target large language model responds with a semantic meaning of refusing to answer or asking for further discussion, a reward is given. If the target large language model responds with a semantic meaning of answering directly, no reward is given. For task instructions with sufficient conditions, no semantic reward judgment is made. The answer accuracy reward is determined for task instructions with sufficient conditions. If the answer is correct, a reward is given; otherwise, no reward is given. For task instructions with insufficient conditions, the answer accuracy reward is not determined. The target large language model is fine-tuned using reinforcement learning based on the reward model using a group-relative strategy optimization algorithm.

4. The method for optimizing and evaluating multi-turn dialogues based on a large language model using reinforcement learning according to claim 1, characterized in that, In step S3: The evaluation method for the multi-turn dialogue ability is as follows: a complete instruction is input into the target large language model in different turns. First, the question of the instruction is input, and then one of the conditions of the question is input after the target large language model answers. The illusion response rate of the model is evaluated except for the last round of answer. After all conditions are input, the last round of answer of the target large language model is evaluated to see if it meets the expectations. The evaluation index is the answer accuracy. The hallucination response rate is the proportion of hallucination responses to all responses, where all responses exclude the last round of responses.

5. The method for optimizing and evaluating multi-turn dialogues based on a large language model using reinforcement learning according to claim 2, characterized in that, The process of the large language model processing task instructions is as follows: call the large language model API, input the original task instruction, and let it return the task instruction after removing at least one condition.

6. The method for optimizing and evaluating multi-turn dialogues based on a large language model using reinforcement learning according to claim 3, characterized in that, The reinforcement learning fine-tuning includes: At the beginning of training, the reference model π ref is set to the target large language model π θ , and the parameters are kept fixed during the iteration process, and the target large language model is referenced in subsequent optimization to prevent excessive optimization. For each input instruction Q, the current target large language model π is used. θ Generate G distinct responses O1, O2…O G And record the probability π generated at each time step in each response. θ (o i,t |Q,o i,<t ); For each complete response O i Using the reward model Scoring is performed to obtain r1, r2...r G Further calculate relative advantage Its formula is defined as follows: Where mean(·) is the mean operation and std(·) is the standard deviation operation; Finally, the target large language model π is optimized using a group-relative strategy optimization algorithm. θ Its formula is defined as follows: in: Among them, D KL Let KL divergence be denoted by β, and β be a hyperparameter. For the old target large language model during the iteration process, r i,t (θ) represents the probability ratio generated by the new and old target large language models, clip(·) is the clipping function, and ε is the clipping parameter.

7. A device for optimizing and evaluating multi-turn dialogues using a large language model based on reinforcement learning, characterized in that, Includes the following modules: The data construction module is used to build datasets containing both complete and incomplete task instructions; The reward and optimization module is used to construct the reward model and perform reinforcement learning fine-tuning on the target large language model. An evaluation module is used to evaluate the multi-turn dialogue capabilities.

Images