Conversation planning method and system based on successful perception selection action

Abstract

The invention discloses a dialogue planning method and system based on successful perception selection action. The method comprises the following steps: step 1, continuously obtaining a dialogue state; step 2, strategy network prediction: obtaining action prior distribution and expected accumulated rewards based on a pre-trained agent; 3, performing Monte Carlo tree search simulation; 4, recording a success signal, and correcting a Monte Carlo tree search decision; 5, calculating a successful perception score, and adjusting the weight coefficients of the successful perception score and the number of visited times along with the extension of the dialogue time; and step 6, strategy network updating is carried out by minimizing the strategy loss function. A success perception factor is introduced in an action selection process, and a system tends to select a high-value action in planning through modeling and amplification of a success signal, so that the success rate of a dialogue task is improved.

Description

Technical Field

[0001] This invention belongs to the field of dialogue system technology, and in particular relates to a dialogue planning method and system based on successful perception of action selection. Background Technology

[0002] With the rapid development of artificial intelligence technology, proactive dialogue systems are playing an increasingly important role in human-computer interaction. However, how to formulate effective, goal-oriented dialogue strategies to ensure long-term, in-depth interaction in these systems remains a critical technical problem that urgently needs to be solved. While current large language models (LLMs) perform well in generating context-sensitive and fluent single-turn responses, their capabilities are still significantly limited in tasks requiring long-term planning and strategic thinking. This makes it difficult for dialogue systems to achieve complex dialogue goals and they are prone to deviating from the preset path in multi-turn dialogues, resulting in a poor user experience.

[0003] To address the shortcomings of LLMs in long-term planning, some scholars have drawn inspiration from cognitive decision theory, proposing to combine rapid-response strategy networks with slow and deliberate planning mechanisms. For example, Monte Carlo Tree Search (MCTS) can be used as a planning mechanism to enhance the strategic capabilities of dialogue by simulating and evaluating different dialogue paths. This paradigm aims to compensate for the shortcomings of LLMs in strategic decision-making by combining "intuition" (strategy networks) with "rationality" (MCTS), thereby improving the goal orientation and long-term planning capabilities of proactive dialogue systems to some extent.

[0004] However, the aforementioned policy planning method combining MCTS has a key technical challenge and a significant drawback. When the policy network assigns almost equal scores to multiple potential dialogue options, the exploration process of the MCTS planner becomes highly unfocused. This means that the planning mechanism cannot effectively identify and prioritize the most promising paths, but instead distributes computational resources and exploration effort almost evenly across all possibilities, resulting in severe waste of computational resources and inefficiency. This unfocused exploration not only directly weakens the quality of dialogue decisions, causing the system to potentially choose suboptimal dialogue actions, but more seriously, it also pollutes the model's learning process during the training phase, reinforcing the weights of suboptimal policies, thus creating a vicious cycle that makes it difficult for the system to effectively learn from errors and improve its long-term planning capabilities. Summary of the Invention

[0005] The purpose of this invention is to address the problem in existing dialogue systems mentioned in the background art that when the policy network scores multiple options similarly, the Monte Carlo Tree Search (MCTS) planner explores without focus, resulting in decreased decision quality and contamination of the training process. This application provides a dialogue planning method and system based on successful perception of action selection.

[0006] To achieve the objectives of this invention, the present invention provides a dialogue planning method based on success perception for action selection, the method comprising:

[0007] Step 1: Continuously obtain the dialogue status:

[0008] Obtain the current state during the dialogue. , It is a record of historical dialogues, represented by a sequence of historical interactions between the system and the user; Refers to the dialogue in the t-th round of the system. This refers to the conversation between users in round t, and each round of conversation... The system provider and the user provider each select a specific action from the fixed action set A. generate;

[0009] The dialogue state is continuously acquired until the dialogue goal is achieved or the maximum number of rounds T is reached; where A is a fixed set of optional actions designed for each dataset by existing methods; It is the specific action selected from action set A in the t-th round of dialogue. T is an integer used to limit the maximum number of dialogue rounds in the dialogue task. In this experiment, the maximum number of dialogue rounds is 8.

[0010] use This represents the dialogue state in round t, where the current dialogue state s is the sequence of actions selected from round 0 to round t: .

[0011] Step 2: Policy network prediction: Obtain the prior distribution of actions and expected cumulative reward based on the pre-trained agent;

[0012] Existing pre-trained agents consist of a policy network and a value network. The policy network is used to input the current dialogue state s to obtain the predicted prior probability distribution of actions. Output, where The parameter is The probability distribution output by the policy network, where a and s represent the specific action and state in the action set A, respectively, and the action prior probability distribution. Let represent the prior probability distribution of each action in state s. The expected cumulative reward of the corresponding action in state is obtained using the value network. The prior distribution of actions and the expected cumulative reward serve as initial guidance during the planning phase of the Monte Carlo tree search.

[0013] Step 3: Perform Monte Carlo Tree Search (MCTS) simulation:

[0014] Based on the prior distribution of actions output by the pre-trained agent and the expected cumulative reward, perform several Monte Carlo tree searches to select the action sequence that can be selected in the current dialogue state and the simulated dialogue corresponding to the recorded action sequence.

[0015] To ensure efficiency, 10 Monte Carlo tree search simulations were performed.

[0016] The Monte Carlo tree search process includes selection, expansion, simulation, and backtracking;

[0017] After the four stages of the Monte Carlo tree search process are completed, the following is obtained: Where N represents the total number of times the action was accessed after the Monte Carlo tree search process is completed. This represents the number of times each candidate action a has been explored, starting from the current state s. Traditional Monte Carlo tree search would simply select the action that has been explored the most times.

[0018] Step 4: Record the success signal and correct the Monte Carlo tree search decision;

[0019] To incorporate information from successful dialogue completion into the final decision, a post-processing step is added to refine the Monte Carlo tree search decision; instead of simply counting the number of explorations for each action, a success signal is recorded for each action. ;

[0020] The process is as follows:

[0021] In the existing Monte Carlo tree search process, the number of visits N(s,a) for each action a under state s is counted. After the system and the user select an action to simulate a round of dialogue between the system and the user, the judge played by the large model is used to determine whether the transaction has been completed in the current round based on the current historical dialogue, and the information on whether the dialogue has been completed is recorded.

[0022] If the action sequence starting with 'a' in state s is evaluated as completed in the dialogue record, then the success count of selecting action 'a' in state s is incremented by one, which is expressed as follows: Where Nsucc represents the total number of successful searches after the Monte Carlo tree search is completed. It records the number of times that an action sequence starting with action a in state s successfully ended the dialogue within the maximum dialogue round limit T.

[0023] Step 5: Calculate the success perception score, and adjust the weighting coefficients of the success perception score and the number of times the conversation is extended.

[0024] After adding a post-processing step to record a success signal, record the number of times each action is accessed in the current state. And the number of successful dialogues resulting from the selection of the corresponding action. ;

[0025] Based on these two key pieces of information, the success perception score (SAAS) for each action is calculated, and the action with the highest score is finally selected as the next decision.

[0026] The method for determining the success score is as follows:

[0027] ;

[0028] To strike a balance between exploration and leveraging known successful actions, dynamic adjustment is introduced. coefficient pair and The weights of the two signals are controlled.

[0029] The calculation method is as follows:

[0030] ;

[0031] The value range is from 0 to T-1. To prevent the success signal in round 0 from being ineffective, a small influence factor of 0.1 is added to the left. This represents the current round of dialogue. This is the preset maximum number of dialogue rounds; It is the size of action set A, representing the number of actions in action set A;

[0032] In the early stages of the dialogue Smaller, at this time The value is smaller; smaller mean The impact on the perceived success score is relatively weak, which makes the algorithm more inclined to explore actions that are accessed more often even if they are not successful in many cases in the early stages of the dialogue.

[0033] In the later stages of the dialogue, as the number of rounds increases, The value gradually increases, and the increase is... Make The weight of successful perception scores is increased. At this point, the algorithm will focus more on utilizing actions that have already demonstrated a high success rate.

[0034] A new action probability distribution is obtained using the success perception score. :

[0035] ;

[0036] The denominator is a subset of all actions. The values ​​are summed to ensure that the success perception score can be normalized and compared across all actions; the action with the highest probability is selected and output to the environment for execution.

[0037] Step 6: Update the policy network by minimizing the policy loss function.

[0038] Record the action selected by the policy network in step 5 and calculate the corresponding log probability in the policy network. The recorded log probabilities are combined with the action value signals calculated in existing reinforcement learning frameworks to form the policy loss function.

[0039] ;

[0040] in Expressing expectations, The actions recorded in the dialogue trajectory are in the parameter of The log probability output by the policy network. The action value signal is obtained from the existing value network; by minimizing the policy loss function, the parameters of the policy network are... Updated.

[0041] This application also provides a dialogue planning system based on success perception for selecting actions. The system is used to implement the method provided in this application, and the system includes:

[0042] The dialogue acquisition module is used to continuously acquire the dialogue status;

[0043] The policy network prediction module is used to obtain the action prior distribution and expected cumulative reward based on the pre-trained agent;

[0044] The simulation module is used to perform Monte Carlo tree search simulations.

[0045] The recording module is used to record success signals and correct Monte Carlo tree search decisions.

[0046] The success perception score calculation module is used to calculate the success perception score and adjust the weight coefficients of the success perception score and the number of times the conversation is accessed as the conversation time increases.

[0047] The update module is used to update the policy network by minimizing the policy loss function.

[0048] The methods and systems provided in this application correspond one-to-one. The technical details of the system will not be repeated here. For specific technical details, please refer to the methods provided in this application.

[0049] This invention generates a more exploratory and success-oriented action selection distribution through a dynamic SaaS mechanism. This distribution serves as an optimization source for action selection during training, indirectly influencing the policy network and its update direction, thereby improving the overall performance and decision-making efficiency of the policy network.

[0050] Compared with the prior art, the significant advancement of this invention lies in the introduction of a success perception factor into the action selection process. By modeling and amplifying the success signal, the system is more inclined to select high-value actions during planning, thereby improving the success rate of dialogue tasks.

[0051] A dynamic λ control mechanism is used when calculating the success perception factor. The parameters are adaptively adjusted according to the interaction process, which enables the system to flexibly balance exploration and utilization in different scenarios, thereby significantly reducing the number of dialogue rounds and improving interaction efficiency.

[0052] Use action distributions with success information during strategy optimization. By explicitly introducing success signals into the action distribution, the entropy of the policy distribution is effectively reduced, making the policy more focused and stable, thereby enhancing the stability of the dialogue policy and reducing volatility.

[0053] To more clearly illustrate the functional characteristics and structural parameters of the present invention, further explanation is provided below in conjunction with the accompanying drawings and specific embodiments. Attached Figure Description

[0054] Figure 1 This is a flowchart of the method provided in the embodiments of this application;

[0055] Figure 2 This is a diagram of the Monte Carlo Tree Search (MCTS) action selection method improved by success perception score provided in the embodiments of this application. Detailed Implementation

[0056] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] This application provides a new paradigm that, by introducing a Success-Aware Action Selection (SAAS) mechanism, assigns scoring rewards to actions that have been shown to produce positive dialogue outcomes during the simulation process, thereby more effectively guiding exploration and significantly improving planning efficiency.

[0058] Figure 1The flowchart illustrates an embodiment of the present invention that combines existing Monte Carlo Tree Search (MCTS) technology with the Successful Perception Score (SAAS Score) of the present invention to guide existing agents in making dialogue decisions. Figure 1 The paper describes how an agent learns from dialogue data and uses MCTS (Multi-Cut Response Theory) combined with the success perception score of this invention to make more optimized action selections, thereby improving the effectiveness of the dialogue strategy.

[0059] The following is an explanation of some of the technical terms used in this application:

[0060] Intelligent Agent: Representing the dialogue intelligent agent in the prior art, it typically includes a policy network and a value network for learning the prior probability distribution of the dialogue policy and predicting the value of actions. In this invention, the intelligent agent receives actions from the method of this invention to guide optimization decisions.

[0061] Dialogue data: This refers to the dataset used to train existing agents. It typically includes information such as the user's historical dialogue records with the system, corresponding states, actions taken, and rewards obtained. The "dialogue state" information will be passed to the existing agents.

[0062] Monte Carlo Tree Search (MCTS): Represents the Monte Carlo Tree Search Planner, a decision-making and planning algorithm. In this invention, existing agents output "prior probabilities / values ​​of actions" to provide guidance, while MCTS utilizes relevant information to construct and explore trees to generate potential action sequences.

[0063] Successful Perception Score (SAAS Score): This score receives the "action access count" and "success signal" output after the MCTS process ends, and is used to evaluate the probability of each action being selected in its corresponding state. It is one of the core innovations of this invention, and the SAAS score is a key basis for guiding subsequent action selection.

[0064] Training data -- "Dialogue state" --> Agent: This means that the training data provides the current dialogue state information for the existing agent, which then uses this information to predict actions.

[0065] Agent -- "Action Prior Probability and Value" --> Monte Carlo Tree Search: This means that the existing agent outputs the prior probability distribution and expected cumulative reward (action value) of each action based on the current state, which serves as the input to MCTS and guides the exploration and evaluation of MCTS.

[0066] Monte Carlo Tree Search -- "Action Visit Count and Success Signal --> Success Perception Score Calculation: This means that during the exploration process, MCTS records the number of visits for each action and the success signal after simulation, and passes the relevant information to the SAAS score calculation module.

[0067] Successful perception score calculation -- "Executing an action and generating a new state" --> Dialogue completion status: Executing an action involves interacting with the external environment. If the interaction is not yet complete, the computer device adds the executed action to the historical action sequence to obtain the next state, and passes the next state to the agent, thus forming a continuous decision-making loop. This continues until the environment determines that the dialogue has successfully ended or the maximum number of dialogue rounds has been triggered. The successful perception score is then passed to the agent to assist in updating it.

[0068] Figure 2 This embodiment provides an improved Monte Carlo Tree Search (MCTS) action selection method that utilizes success-aware scores. The method combines node success information and access frequency to more effectively guide the search direction.

[0069] Figure 2 This example illustrates the results of MCTS simulation in an embodiment of the present invention. In a certain "current state" of MCTS, there are three selectable actions: "Action 1," "Action 2," and "Action 3." Each action may guide the system to a different "state." The label for each "state" node (e.g., "State 1 0 / 1") represents "Number of Successes / Number of Visits," where the green number represents the number of successes achieved in that state, and the red number represents the total number of visits to that state. Green squares indicate that the dialogue successfully ended before triggering the maximum number of rounds; for example, "State 8," "State 9," and "State 10" are the endpoints of successful dialogue completion. Other white squares represent other states where the dialogue did not successfully end, which may include states such as triggering the maximum number of dialogue rounds, not triggering the maximum number of dialogue rounds, but the dialogue not successfully ending.

[0070] In existing dialogue system methods, action selection is typically based on access frequency, favoring actions frequently accessed through simulated paths as having greater exploration potential. For the three initial actions in the "current state," if action scores are calculated solely based on access frequency:

[0071] The score for action 1 is: 0.1

[0072] The score for action 2 is 0.5.

[0073] The score for action 3 is: 0.4

[0074] Based on the existing calculation method, the current approach will select action 2, which has the highest score.

[0075] In the method of this invention, a success perception score is calculated for the three actions in the "current state," combining success information and access frequency. The calculation result may be:

[0076] The success perception score for action 1 is: 0.075

[0077] The success perception score for action 2 is 0.377.

[0078] The success perception score for action 3 was 0.548.

[0079] Based on this success perception score, the method of the present invention will select action 3 with the highest score.

[0080] By introducing a success-perceived score, this invention can more intelligently evaluate the action selection of MCTS in a dialogue system. Even if the pure access frequency score (based on traditional formulas such as UCT) of an action is not high, if its subsequent paths contain more states that successfully end the dialogue or have a higher success rate, the method of this invention can still identify and prioritize the corresponding action. This helps MCTS to more effectively explore and utilize dialogue paths that contain high rewards but are not prioritized by traditional methods, thereby improving the success rate and efficiency of the dialogue system in achieving user goals. In the embodiments of this application, although existing methods may select action 2, this invention, by combining success information, identifies action 3 as having a higher perceived potential in ultimately achieving dialogue success and selects action 3. This shows that this invention can guide the dialogue system to discover better dialogue strategies, improving user experience and dialogue efficiency.

[0081] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A dialog planning method for selecting actions based on successful perception, characterized in that, The method includes: Step 1: Continuously acquire the conversation status; Step 2: Policy network prediction: Obtain the prior distribution of actions and expected cumulative reward based on the pre-trained agent; Step 3: Perform Monte Carlo tree search simulation: Step 4: Record the success signal and correct the Monte Carlo tree search decision; Step 5: Calculate the success perception score, and adjust the weighting coefficients of the success perception score and the number of times the conversation is extended. Step 6: Update the policy network by minimizing the policy loss function.

2. The method of claim 1, wherein, Step 1: Continuously acquire the conversation state, including: Acquiring the current state in the dialogue process, , is a historical dialogue record, represented by a historical interaction sequence of the system and the user; refers to the dialogue of the system in the tth round, refers to the dialogue of the user in the tth round, and each round of dialogue is generated by the system side and the user side selecting specific actions in a fixed action set A ; continuously acquiring the dialogue state until the dialogue goal is achieved or the maximum number of turns T is reached; wherein A is a fixed set of optional actions designed by the existing method for each dataset; is the specific action selected in the tth turn of the dialogue from the action set A, and T is an integer used to limit the maximum number of dialogue turns of the dialogue task, and the maximum number of dialogue turns is 8. With denotes the dialogue state of the t-th round, and the current dialogue state s is the specific action sequence selected from the 0-th round to the t-th round: .

3. The method of claim 2, wherein, Step 2: Policy network prediction, including: The pre-trained agent contains a policy network and a value network, and the policy network is used to obtain the predicted action prior probability distribution by inputting the current dialogue state s Output, wherein represents the probability distribution output by the policy network with parameters , a and s represent a specific action in the action set A and a state, respectively, and the action prior probability distribution represents the prior probability distribution of each action under the state s, and the value network is used to obtain the expected cumulative reward of the corresponding action under the state; the action prior distribution and the expected cumulative reward are used as the initial guidance of the planning period of the Monte Carlo tree search.

4. The method of claim 3, wherein, Step 3: Perform Monte Carlo tree search simulation, including: Based on the prior distribution of actions output by the pre-trained agent and the expected cumulative reward, perform several Monte Carlo tree searches to select the action sequence that can be selected in the current dialogue state and the simulated dialogue corresponding to the recorded action sequence. Perform 10 Monte Carlo tree search simulations; The Monte Carlo tree search process includes selection, expansion, simulation, and backtracking; At the end of the four phases of the Monte Carlo tree search procedure, we obtain where N denotes the total number of visits of actions after the end of the Monte Carlo tree search procedure, denotes the number of times each candidate action a has been explored from the current state s.

5. The method of claim 4, wherein, Step 4: Record success signals and correct Monte Carlo tree search decisions; including: Count a success signal for each action ; The process is as follows: After the system and user sides each select an action to simulate a round of dialogue, the large model acts as an evaluator to determine whether the transaction has been completed in the current round based on the current historical dialogue, and records the information on whether the dialogue has been completed. If the action sequence starting with a in state s corresponds to a dialogue record that is evaluated as complete, the success count of selecting action a in state s is incremented by one, formally denoted as wherein denotes the total success count that is statistically counted after the Monte Carlo tree search is finished, records the number of times that the action sequence starting with action a in state s eventually reaches a successful end of the dialogue within the maximum number of dialogue rounds T limit.

6. The method of claim 5, wherein, Step 5: Calculate the success perception score, and adjust the weighting coefficients of the success perception score and the number of times the conversation is conducted as the conversation progresses; including: After adding a post-processing step to record a success signal, record the number of times each action is accessed in the current state. And the number of successful dialogues resulting from the selection of the corresponding action. ; Calculate the success perception score (SAAS) for each action, and finally select the action with the highest score as the next decision. The method for determining the success score is as follows: ； To strike a balance between exploration and leveraging known successful actions, dynamic adjustment is introduced. coefficient pair and The weights of the two signals are controlled. The calculation method is as follows: ； The value range is from 0 to T-1. To prevent the success signal in round 0 from being ineffective, a small influence factor of 0.1 is added to the left. This represents the current round of dialogue. This is the preset maximum number of dialogue rounds; It is the size of action set A, representing the number of actions in action set A; In the early stages of the dialogue Smaller, at this time The value is smaller; smaller mean The impact on perceived success scores is relatively weak; In the later stages of the dialogue, as the number of rounds increases, The value gradually increases, and the increase is... Make Increased weighting in perceived success scores; A new action probability distribution is obtained using the success perception score. : ； The denominator is a subset of all actions. The values ​​are summed to ensure that the success perception score can be normalized and compared across all actions; the action with the highest probability is selected and output to the environment for execution.

7. The method according to claim 6, characterized in that, Step 6: Update the policy network by minimizing the policy loss function, including: Record the action selected by the policy network in step 5 and calculate the corresponding log probability in the policy network. The recorded log probabilities are combined with the action value signals calculated in existing reinforcement learning frameworks to form the policy loss function. ； in Expressing expectations, The actions recorded in the dialogue trajectory are in the parameter of The log probability output by the policy network. The action value signal is obtained from the existing value network; by minimizing the policy loss function, the parameters of the policy network are... Updated.

8. A dialogue planning system based on success perception for action selection, the system being used to implement any one of the methods of claims 1 to 7, characterized in that, The system includes: The dialogue acquisition module is used to continuously acquire the dialogue status; The policy network prediction module is used to obtain the action prior distribution and expected cumulative reward based on the pre-trained agent; The simulation module is used to perform Monte Carlo tree search simulations. The recording module is used to record success signals and correct Monte Carlo tree search decisions. The success perception score calculation module is used to calculate the success perception score and adjust the weight coefficients of the success perception score and the number of times the conversation is accessed as the conversation time increases. The update module is used to update the policy network by minimizing the policy loss function.

Images