A dialogue optimization method and system through LLMs-RL cascaded inference

Abstract

The present application relates to a kind of dialogue optimization method and system by LLMs-RL cascade inference, method includes: obtaining the current dialogue state of user;LLMs and chain thought prompt are used to carry out domain relationship inference to current dialogue state, determine explicit relationship and implicit relationship, and output new dialogue state and confidence score;New dialogue state and confidence score are verified based on double security mechanism;When verification passes, new dialogue state is input into reinforcement learning RL decision module to generate dialogue action.Combination of LLMs and chain thought prompt is used to carry out domain relationship inference, so that inference path is traceable;Double security mechanism reduces the illusion output of LLMs, guarantees that result is verifiable;RL decision module uses reinforcement learning algorithm, avoids strategy drift, guarantees the global optimality of long-term dialogue strategy.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, and in particular to a dialogue optimization method and system using LLMs-RL cascaded reasoning. Background Technology

[0002] Task-Oriented Dialog Systems (TODS) play a crucial role in human-computer interaction and have been widely applied in fields such as intelligent customer service and smart home control. While TODS performs well in single-domain scenarios, its efficiency drops significantly in multi-domain scenarios. This performance degradation stems from two core challenges: i) the difficulty in effectively reusing cross-domain knowledge and achieving smooth domain transitions, defined as explicit constraints; and ii) the inability to reason about implicit common-sense relationships between cross-domains, defined as implicit constraints. Since real-world multi-domain dialogues are not simply a superposition of single-domain tasks but involve dynamic interactions of interdependent sub-tasks across multiple domains, the system needs to dynamically capture and process explicit and implicit cross-domain relationships to achieve coherent and efficient task coordination.

[0003] Fundamentally, these challenges require advanced decision-making capabilities, particularly in optimizing multi-domain dialogue strategies. Current methods for learning such strategies primarily follow two paradigms: the first is a policy optimization approach based on reinforcement learning (RL), which decomposes the task into independent single-domain problems or shares knowledge through a multi-task learning framework. However, the reliance on high-quality data or predefined knowledge limits their ability to adapt to domain diversity. The second approach leverages the commonsense reasoning capabilities of large language models (LLMs) for multi-domain decision-making. However, the black-box nature of LLMs compromises verifiability in long-term dialogues and leads to policy drift—the phenomenon where decisions gradually deviate from the global optimization objective, thus disrupting the balance between domain specialization and global coherence. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a dialogue optimization method and system based on LLMs-RL cascaded reasoning.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] Firstly, a dialogue optimization method based on LLMs-RL cascaded reasoning is provided, including:

[0007] Get the user's current conversation state;

[0008] Using large language models (LLMs) and chain of thought (CoT), we can perform domain relation reasoning on the current dialogue state, determine explicit and implicit relations, and output the new dialogue state and confidence score.

[0009] The new dialogue status and confidence score are verified based on a dual-guarantee mechanism;

[0010] Once the verification is successful, the new dialogue state is input into the reinforcement learning (RL) decision module to generate dialogue actions.

[0011] Furthermore, the dual safeguard mechanism includes a confidence filtering mechanism and a database retrieval verification mechanism.

[0012] Furthermore, the confidence filtering mechanism is as follows:

[0013] Compare confidence scores with confidence thresholds;

[0014] When the confidence score exceeds the confidence threshold, the confidence verification is considered successful.

[0015] When the confidence score does not exceed the confidence threshold, the confidence verification is determined to be unsuccessful, and a new dialogue state is generated.

[0016] Furthermore, the confidence threshold is dynamically adjustable, and the dynamic adjustment formula is as follows;

[0017] ;

[0018] in, It is the initial confidence threshold; This is the current training round; This is the training phase that begins with dynamic adjustments; It is the F1 score in round e; It is a preset minimum threshold; It is a preset adjustment coefficient.

[0019] Furthermore, the database retrieval verification mechanism is as follows:

[0020] Perform a structural consistency check between the new dialogue state and the current dialogue state;

[0021] After passing the structural consistency check, candidate results are retrieved from the known database;

[0022] The semantic similarity calculation formula is used to evaluate the similarity between the new dialogue state and the candidate results, and the semantic similarity value is obtained.

[0023] Compare semantic similarity values ​​with similarity thresholds;

[0024] When the semantic similarity value is not lower than the similarity threshold, the database retrieval verification is deemed successful.

[0025] When the semantic similarity value is lower than the similarity threshold, the database retrieval verification is deemed unsuccessful.

[0026] Furthermore, the current dialogue state includes dialogue history, intent, and slot;

[0027] Perform a structural consistency check between the new dialogue state and the current dialogue state, including:

[0028] The new dialogue state and the current dialogue state are structurally matched to find inconsistencies in the slots.

[0029] Candidate results were retrieved from known databases, including:

[0030] Retrieve multiple candidate slot values ​​that may exist for slot difference points from a known database;

[0031] The semantic similarity calculation formula is used to evaluate the similarity between the new dialogue state and the candidate results, and the semantic similarity value is obtained, including:

[0032] Using a semantic similarity calculation formula, the similarity value between the slot value of the corresponding slot difference point in the new dialogue state and each candidate slot value is calculated;

[0033] Select the largest similarity value from multiple similarity values ​​as the semantic similarity value.

[0034] Furthermore, the formula for calculating semantic similarity is:

[0035] ;

[0036] in, and Represents two semantic vectors; It is the dot product of vectors; Let be the vector magnitude.

[0037] Furthermore, the RL decision module employs either a proximal policy optimization algorithm or a deep Q-network algorithm.

[0038] Secondly, a dialogue optimization system based on LLMs-RL cascaded reasoning is provided, including:

[0039] The input unit is used to obtain the user's current dialogue state;

[0040] The domain reasoning unit is used to perform domain relation reasoning on the current dialogue state using large language models (LLMs) and chained thinking prompts, determine explicit and implicit relations, and output the new dialogue state and confidence score.

[0041] The dual-guarantee unit is used to verify the new dialogue state and confidence score based on the dual-guarantee mechanism;

[0042] The reinforcement learning unit is used to input the new dialogue state into the reinforcement learning (RL) decision module to generate dialogue actions after the verification is successful.

[0043] The beneficial effects achieved by this invention are as follows:

[0044] Obtain the user's current dialogue state; use large language models (LLMs) and chained thinking prompts to perform domain relation reasoning on the current dialogue state, determine explicit and implicit relations, and output the new dialogue state and confidence score; verify the new dialogue state and confidence score based on a dual guarantee mechanism; when the verification is successful, input the new dialogue state into the reinforcement learning (RL) decision module to generate dialogue actions;

[0045] By combining LLMs with chain-like thinking prompts for domain relationship reasoning, the reasoning path becomes traceable.

[0046] The dual safeguard mechanism requires verification of the new dialogue state and confidence score to reduce the hallucination output of LLMs and ensure that the results are verifiable.

[0047] The RL decision module employs reinforcement learning algorithms such as Proximal Policy Optimization (PPO) or Deep Q-Network (DQN) to avoid policy drift and ensure the global optimality of long-term dialogue policies. Attached Figure Description

[0048] Figure 1 The flowchart shows the dialogue optimization method of the present invention using LLMs-RL cascaded inference.

[0049] Figure 2 This is a flowchart of relational reasoning in the field of this invention;

[0050] Figure 3 This is a structural diagram of the dialogue optimization system based on LLMs-RL cascaded reasoning of the present invention. Detailed Implementation

[0051] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0052] like Figure 1 As shown, this embodiment of the invention provides a dialogue optimization method using LLMs-RL cascaded reasoning, including:

[0053] 101. Obtain the user's current conversation state;

[0054] In this embodiment, the user's current dialogue state is obtained from the RL process. The current dialogue state includes dialogue history, intent, and slot; the current dialogue state serves as the basis for subsequent reasoning and decision-making.

[0055] 102. Using large language models (LLMs) and chained thinking prompts, perform domain relation reasoning on the current dialogue state, determine explicit and implicit relations, and output the new dialogue state and confidence score.

[0056] In this embodiment, the current dialogue state is placed in a pre-designed Prompt for domain relationship reasoning, and Chained Thinking Hints (CoT) are used to enhance the reasoning ability of LLMs; the process of domain relationship reasoning is as follows: Figure 2 As shown, it includes:

[0057] 201, Domain Identification: Determine the target domain involved in the current dialogue state;

[0058] 202, Information Detection: Extract the constraints of known slots in the current dialogue state;

[0059] 203. Constructing Scenario Reasoning: Construct reasoning paths and find explicit and implicit relationships between different target domains as the reasoning results;

[0060] 204. Determine the modifiable slots and their corresponding slot values: Modify the current dialogue state based on the reasoning results to obtain a new dialogue state;

[0061] 205, output the new dialogue state and confidence score.

[0062] Specifically, the Prompt design divides problem-solving into approximately five steps: First, it identifies the target domains of the problem, specifically which domains have their state values ​​updated. Second, LLMs select known slots (already filled slots) from the domains with dialogue states as prior information for inference. Third, it constructs scenarios to perform inter-domain inference, seeking explicit and implicit relationships as much as possible. Fourth, it assigns modifiable slots and their corresponding values ​​based on the inference, modifies the current dialogue state according to the inference results, and obtains a new dialogue state. Fifth, it outputs the new dialogue state and a confidence score.

[0063] 103. Verify the new dialogue status and confidence score based on the dual guarantee mechanism;

[0064] In this embodiment, the dual protection mechanism includes a confidence filtering mechanism and a database retrieval verification mechanism;

[0065] The confidence filtering mechanism is as follows:

[0066] Compare confidence scores with confidence thresholds;

[0067] When the confidence score exceeds the confidence threshold, the confidence verification is considered successful.

[0068] When the confidence score does not exceed the confidence threshold, the confidence verification is determined to be unsuccessful, and a new dialogue state is generated.

[0069] The confidence threshold is dynamically adjustable, and the dynamic adjustment formula is as follows;

[0070] ;

[0071] in, It is the initial confidence threshold; This is the current training round; This is the training phase that begins with dynamic adjustments; It is the F1 score in round e; It is a preset minimum threshold; It is a preset adjustment coefficient.

[0072] The database retrieval verification mechanism is as follows:

[0073] The new dialogue state and the current dialogue state are structurally matched to find inconsistencies in the slots.

[0074] After passing the structural consistency check, multiple candidate slot values ​​that may exist for slot difference points are retrieved from the known database.

[0075] The semantic similarity calculation formula is used to calculate the similarity value between the slot value of the corresponding slot difference point in the new dialogue state and each candidate slot value; the semantic similarity calculation formula is:

[0076] ;

[0077] in, and Represents two semantic vectors; It is the dot product of vectors; Let be the vector magnitude.

[0078] Select the largest similarity value from multiple similarity values ​​as the semantic similarity value;

[0079] Compare semantic similarity values ​​with similarity thresholds;

[0080] When the semantic similarity value is not lower than the similarity threshold, the database retrieval verification is deemed successful.

[0081] When the semantic similarity value is lower than the similarity threshold, the database retrieval verification is deemed unsuccessful.

[0082] It should be noted that in this embodiment, a trial-and-error threshold is also provided. If the number of times the semantic similarity fails or the structural consistency fails is too large, then the reasoning will be skipped. This means that the existing known information cannot explore a meaningful action space, that is, LLMs cannot infer useful additional information from this known information.

[0083] It should be noted that this invention also sets up a self-confidence judgment mechanism for LLMs; when outputting a new dialogue state, a confidence score is also obtained; if no confidence score is obtained, then the output of LLMs is considered to be incorrect, and the error count is increased by one in the case of making an error; under this mechanism, LLMs will score the output, that is, perform self-evaluation.

[0084] Set an untrust threshold, which is the number of times the dialogue state is allowed to be regenerated. When the number of generated dialogue states and the confidence score are all lower than the confidence threshold, it is determined that the information in the current dialogue state is insufficient to support cross-domain reasoning. Therefore, the LLMs will be paused and other original models will continue the dialogue to supplement the necessary background information.

[0085] 104. Once the verification is successful, the new dialogue state is input into the reinforcement learning (RL) decision module to generate dialogue actions.

[0086] In this embodiment, the RL decision module adopts the PPO or DQN algorithm;

[0087] The objective function of PPO is:

[0088] ;

[0089] in, The ratio of dialogue state to action probability between the old and new strategies at time t; Indicates the strategy parameters; This represents the dialogue action at time t. This represents the dialogue state at time t; This represents the probability of the action output by the new strategy; This represents the probability of the action output by the old strategy; The dominant function; For hyperparameters;

[0090] The update formula for DQN is:

[0091] ;

[0092] in, The learning rate; Discount factor; Indicates the old network parameters; This represents the target network parameters, used for stable training to obtain the required dialogue states and dialogue actions.

[0093] In multi-turn dialogues, constructing a credible basic strategy is key to stabilizing the entire dialogue system. Therefore, credible decision-making based on RL was chosen. The RL decision-making module can optimize long-term goals. Unlike supervised learning, which optimizes the accuracy of a single turn, RL focuses on the cumulative reward of the entire dialogue process. This means that the dialogue strategy obtained by RL may "sacrifice" some short-term benefits in exchange for longer-term rewards. In other words, RL focuses on the overall task, which greatly mitigates the overexploration problem brought about by adding LLMs collaboration.

[0094] It should be noted that after generating the dialogue action, it is integrated into the dialogue system to continue the dialogue with the user according to the dialogue action.

[0095] The beneficial effects of the embodiments of the present invention are as follows:

[0096] By introducing chain-like thinking prompts, LLMs can establish explicit dependencies between domains and combine common sense reasoning to uncover implicit constraints, thereby effectively solving the problem of inconsistent constraints in multi-domain tasks and achieving more efficient cross-domain knowledge reuse.

[0097] By verifying the new dialogue state and confidence score through a dual guarantee mechanism, the "illusion" problem of LLMs in the reasoning process is effectively alleviated; in particular, the use of a dynamically adjustable confidence threshold can adaptively improve the output quality of LLMs and maintain stability and reliability without relying on large-scale manual annotation.

[0098] By inputting the validated new dialogue state into the RL decision module and using the PPO or DQN algorithm for policy updates, the policy drift problem in traditional methods can be avoided, ensuring global optimality and stability in multi-turn long sequence dialogues.

[0099] By introducing explicit reasoning paths and confidence scores, the decision-making process in the dialogue becomes traceable and explainable, thereby improving the system's transparency and user trust.

[0100] It should be noted that the dialogue optimization method of the present invention through LLMs-RL cascaded reasoning is not only applicable to single-scenario task-oriented dialogue systems, but can also be extended to multi-domain composite dialogue systems including hotel booking, restaurant recommendation, travel planning, medical consultation, and tourism Q&A, and has strong versatility and application value.

[0101] In summary Figure 1 The dialogue optimization method using LLMs-RL cascaded reasoning described in the previous section is illustrated below with examples of a dialogue optimization system using LLMs-RL cascaded reasoning.

[0102] like Figure 3 As shown, this embodiment of the invention provides a dialogue optimization system based on LLMs-RL cascaded reasoning, comprising:

[0103] Input unit 301 is used to obtain the user's current dialogue state;

[0104] Domain reasoning unit 302 is used to perform domain relation reasoning on the current dialogue state using large language models LLMs and chained thinking prompts, determine explicit and implicit relations, and output the new dialogue state and confidence score.

[0105] Dual safeguard unit 303 is used to verify the new dialogue state and confidence score based on the dual safeguard mechanism;

[0106] Reinforcement learning unit 304 is used to input the new dialogue state into the reinforcement learning (RL) decision module to generate dialogue actions after the verification is successful.

[0107] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0109] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0110] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0111] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of the claims of the present invention pending approval.

Claims

1. A dialogue optimization method using LLMs-RL cascaded reasoning, characterized in that, include: Get the user's current conversation state; Using Large Language Models (LLMs) and chained thinking prompts, domain relation reasoning is performed on the current dialogue state to determine explicit and implicit relations, and a new dialogue state and confidence score are output. The new dialogue state and the confidence score are verified based on a dual-guarantee mechanism, which includes a confidence filtering mechanism and a database retrieval verification mechanism. Once the verification is successful, the new dialogue state is input into the reinforcement learning (RL) decision module to generate dialogue actions. The confidence filtering mechanism is as follows: Compare the confidence score with the confidence threshold; When the confidence score exceeds the confidence threshold, the confidence verification is deemed successful. When the confidence score does not exceed the confidence threshold, the confidence verification is determined to be unsuccessful, and a new dialogue state is generated. The database retrieval verification mechanism is as follows: Perform a structural consistency check between the new dialogue state and the current dialogue state; After passing the structural consistency verification, candidate results are retrieved from the known database; The semantic similarity value is obtained by evaluating the similarity between the new dialogue state and the candidate result using a semantic similarity calculation formula. Compare semantic similarity values ​​with similarity thresholds; When the semantic similarity value is not lower than the similarity threshold, the database retrieval verification is deemed successful. When the semantic similarity value is lower than the similarity threshold, the database retrieval verification is determined to fail.

2. The dialogue optimization method using LLMs-RL cascaded reasoning according to claim 1, characterized in that, The confidence threshold is dynamically adjustable, and the dynamic adjustment formula is as follows: ； Among them, the It is the initial confidence threshold; the This is the current training round; the stated This is the training phase that begins with dynamic adjustment; the aforementioned It is the F1 score of e training rounds; It is a preset minimum threshold; the It is a preset adjustment coefficient.

3. The dialogue optimization method using LLMs-RL cascaded reasoning according to claim 1, characterized in that, The current dialogue state includes dialogue history, intent, and slot; The step of performing a structural consistency check between the new dialogue state and the current dialogue state includes: The new dialogue state and the current dialogue state are structurally matched to find inconsistencies in slot differences. The process of retrieving candidate results from a known database includes: Retrieve multiple candidate slot values ​​that may exist for the slot difference point from a known database; The step of evaluating the similarity between the new dialogue state and the candidate results using a semantic similarity calculation formula to obtain a semantic similarity value includes: Using a semantic similarity calculation formula, the similarity value between the slot value corresponding to the slot difference point in the new dialogue state and each candidate slot value is calculated; Select the largest similarity value from multiple similarity values ​​as the semantic similarity value.

4. The dialogue optimization method using LLMs-RL cascaded reasoning according to claim 3, characterized in that, The formula for calculating semantic similarity is: ； Among them, the and stated Representing two semantic vectors; the For vector dot product; Let be the vector magnitude.

5. The dialogue optimization method using LLMs-RL cascaded reasoning according to claim 1, characterized in that, The RL decision module employs either the near-end policy optimization algorithm or the deep Q-network algorithm.

6. A dialogue optimization system using LLMs-RL cascaded reasoning, characterized in that, include: The input unit is used to obtain the user's current dialogue state; The domain reasoning unit is used to perform domain relation reasoning on the current dialogue state using large language models (LLMs) and chained thinking prompts, determine explicit and implicit relations, and output the new dialogue state and confidence score. A dual-guarantee unit is used to verify the new dialogue state and the confidence score based on a dual-guarantee mechanism; the dual-guarantee mechanism includes a confidence filtering mechanism and a database retrieval verification mechanism. A reinforcement learning unit is used to input the new dialogue state into the reinforcement learning (RL) decision module to generate dialogue actions after the verification is passed. The confidence filtering mechanism is as follows: Compare the confidence score with the confidence threshold; When the confidence score exceeds the confidence threshold, the confidence verification is deemed successful. When the confidence score does not exceed the confidence threshold, the confidence verification is determined to be unsuccessful, and a new dialogue state is generated. The database retrieval verification mechanism is as follows: Perform a structural consistency check between the new dialogue state and the current dialogue state; After passing the structural consistency verification, candidate results are retrieved from the known database; The semantic similarity value is obtained by evaluating the similarity between the new dialogue state and the candidate result using a semantic similarity calculation formula. Compare semantic similarity values ​​with similarity thresholds; When the semantic similarity value is not lower than the similarity threshold, the database retrieval verification is deemed successful. When the semantic similarity value is lower than the similarity threshold, the database retrieval verification is determined to fail.

Images