Lightweight large language model dialogue planning method based on nested rollback strategy adaptation

Abstract

This invention relates to a dialogue policy planning method based on nested rollback policy adaptation. The method includes the following steps: formalizing the dialogue policy planning task as a Markov decision process; initializing the policy π; setting the number of iterations and the level depth to control the recursion depth of the nested rollback policy algorithm; performing dialogue policy planning using the nested rollback policy method, which optimizes the policy through multi-level exploration, updating the policy weights at each level by simulating dialogue paths and selecting the best sequence; and evaluating the quality of the final state of the simulated dialogue path using a reward function. Combined with the generative capabilities of large language models, this method enables efficient dialogue policy planning under a zero-shot learning paradigm. Through multi-level policy optimization and dynamic weight adjustment, this method can dynamically adjust the policy according to the dialogue progress, optimize dialogue behavior selection, and thus improve the response quality and task completion efficiency of the dialogue system.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence dialogue systems, specifically relating to a dialogue strategy planning method based on nested rollback strategy adaptation. Background Technology

[0002] With the rapid development of large language model technology, dialogue systems are increasingly being used in various fields, especially in scenarios such as emotional support, customer service, and educational tutoring.

[0003] Existing dialogue strategy planning methods mainly rely on supervised learning or reinforcement learning with large amounts of labeled data, which suffers from high data acquisition costs and poor domain adaptability. Traditional methods often employ static dialogue strategies or simple heuristic rules, making it difficult to achieve dynamic optimization in complex and ever-changing dialogue scenarios. Summary of the Invention

[0004] This invention introduces a Nested Reverse Policy (NRPA) adaptation algorithm, combined with the generative capabilities of a large language model, to achieve efficient dialogue policy planning under a zero-shot learning paradigm. Through multi-level policy optimization and dynamic weight adjustment, this method can dynamically adjust policies based on dialogue progress, optimizing dialogue behavior selection and thus improving the response quality and task completion efficiency of the dialogue system.

[0005] To achieve the above technical objectives, the technical solution adopted by this invention is: a dialogue strategy planning method based on nested rollback strategy adaptation, which includes the following steps: S1. The dialogue strategy planning task is formalized as a Markov decision process, in which there are t rounds of dialogue between the user and the system, and each system response is associated with a specific dialogue behavior. S2. Initialize the strategy π as a softmax probability distribution covering all actions. In each round, legal actions are first screened out, and then dialogue behaviors are selected from π by sampling. The large language model generates natural language responses based on the prompt template of the behavior and advances the state. The complete dialogue trajectory is obtained through playout. The path quality is measured by the task achievement and round penalty score. In the nested rollback strategy algorithm, multiple iterations are performed: recursively searching for candidates, scoring and selecting the best, and updating the logarithmic weight of the strategy with the best trajectory. At the same time, early stopping and robust convergence are combined to allow π to gradually focus on high-value dialogue decisions. S3. Set the number of iterations and the level depth to constrain the search overhead of the nested rollback strategy; S4. Dialogue strategy planning using nested rollback strategy: Set the level depth and the number of iterations per level. At each level, simulate dialogue multiple times starting from the current strategy π for legal bidding actions to obtain candidate negotiation paths. The large language model scores the complete state based on bidding features, selects the optimal sequence to update the strategy weights, and simulates, scores, selects the best, updates and rolls back upwards layer by layer to find a better strategy. S5. Evaluate the quality of the final state of the simulated dialogue path through a reward function, measure the degree of user problem-solving in emotional support scenarios, impose dialogue round penalties to implicitly introduce a discount factor, and incentivize the model to achieve the goal in fewer rounds.

[0006] Furthermore, the simulation process of the nested rollback strategy algorithm of S2 includes the following steps: S21. Starting from the current dialogue state s, first generate a set of legal actions based on the bidding rules and dialogue progress, map the weights corresponding to the current strategy to a softmax probability distribution, normalize only on legal actions, and sample the dialogue action a to be executed in this round according to the probability, so that high-value actions are tried more frequently, while retaining the necessary exploration space. Action sampling: ; In the above formula: A(s) is the action set containing all actions, w a w represents the weight of the current action. b b represents the weight of the remaining actions; b represents the remaining actions. S22. Map state s and action a to the corresponding bidding prompt template, call the large language model to generate system response u, and use the environment transition function to advance the dialogue: write u into the dialogue history, update the bid, round and termination flag and other state variables according to u and business rules to obtain the new state s; at the same time, add the action a of this round to the action sequence as the basis for subsequent evaluation and strategy update. Large Language Model Generation and State Update: ; ; ; In the above formula: S t For the current state of the entire conversation, a t In state S t The selected dialogue action, u t The dialogue generated by the current large model, The environment transition function; inputs the current state, action, and generated response, and outputs the next state S. t+1 seq represents the action sequence; S23. Repeat execution until the termination condition is triggered, and the final state s is obtained.T The simulation path is scored once with the complete action sequence, and the reward score and corresponding action sequence are returned. In the bidding scenario, the reward is based on whether a transaction is completed, and the penalty is combined with the dialogue round factor to encourage the completion of effective transactions in fewer rounds. ; In the above formula: R(s) T ) for the termination state s T Reward assessment; {Deal(s T )} is the indicator function; λ is the dialogue turn penalty coefficient, turns is the dialogue length per turn, and T is the total number of steps.

[0007] Furthermore, the update process of the nested rollback strategy algorithm includes the following steps: Receive high-quality action sequences and adjust policy weights; For each action 'a' in the sequence, a penalty-reward mechanism is applied, and a normalization factor is calculated. For all actions a'∈A, perform a weight update; For the current action 'a', add extra weight to enable the strategy to learn and optimize from successful experiences.

[0008] Furthermore, the process of optimizing the multi-level exploration strategy is as follows: Determine the current level. If level=0, directly execute the simulation operation and return the score and sequence. If level > 0, then enter the recursive process and execute N iterations. In each iteration, call the nested rollback strategy method of the lower level to obtain the path score and sequence, compare and update the optimal score and sequence, and then call the adaptive function to update the strategy π according to the optimal sequence.

[0009] Furthermore, the multi-level system also includes the following steps: By prompting a large language model during the decision-making process, multi-level exploration can be achieved without the need for large amounts of training data specific to a particular domain; Simulate user and system response interactions, assess current task progress, and optimize the probability of selecting the next dialogue action; A dynamic rollout strategy is adopted instead of a static uniform random strategy, and the strategy weights are adjusted in real time within the nested hierarchy.

[0010] Furthermore, the reward function design includes: Evaluate the quality of the final state s of the simulated dialogue path and measure whether the user's problem has been successfully resolved in an emotional support scenario; Focusing on the final state assessment, a discount factor is implicitly introduced by imposing a dialogue round penalty, which incentivizes the model to achieve the goal in fewer rounds.

[0011] On the other hand, the present invention also provides a system for a dialogue strategy planning method based on nested rollback strategy adaptation, the system comprising: The large language model module is used to generate system responses based on the current dialogue state and the selected action. The strategy management module is used to maintain and update the dialogue strategy weight vector π. The nested rollback strategy planner module implements any of the above methods to perform multi-level optimization of the dialogue strategy; The state tracking module is used to maintain the historical state of the dialogue and evaluate the progress of the dialogue. The reward evaluation module is used to calculate the final score of the dialogue path and provide feedback signals for policy updates.

[0012] Compared with the prior art, the beneficial effects of the present invention are: This invention utilizes a large language model to simultaneously simulate user and system behavior. NRPA-GD constructs a complete dialogue trajectory evaluation mechanism. Through a nested Monte Carlo simulation and policy adaptation optimization framework, the policy is dynamically adjusted during the dialogue process. Experiments on four typical goal-oriented dialogue datasets show that NRPA-GD significantly outperforms existing prompting engineering methods and dedicated pre-trained models, further highlighting the advantages and innovation of LLM-based planning methods in solving practical planning tasks. Attached Figure Description

[0013] Figure 1 This is a flowchart of the dialogue strategy planning method based on nested rollback strategy adaptation of the present invention; Figure 2 This is a framework diagram of the dialogue strategy planning method based on nested rollback strategy adaptation of the present invention; Figure 3 The results on the P4G and CIMA datasets were manually evaluated. Figure 4 This is the pseudocode for the rollback strategy algorithm of this invention; Figure 5 This is a flowchart of the algorithm operation of the present invention. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0015] Although the steps in this invention are arranged by reference numerals, this is not intended to limit the order of the steps. Unless the order of the steps is explicitly stated or the execution of a step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items.

[0016] The dialogue policy planning method based on the nested rollback policy adaptation algorithm can achieve efficient dialogue policy optimization under the zero-shot learning paradigm. Through multi-level exploration and dynamic weight adjustment, it can significantly improve the response quality and task completion efficiency of the dialogue system. Figure 1 This demonstrates the complete workflow of a dialogue strategy planning method that adapts to nested rollback strategies, such as... Figures 1-5 As shown.

[0017] The dialogue policy planning method based on the nested rollback policy adaptation algorithm includes the following steps: Step 1: Initialize the dialogue state and policy distribution. Starting from the initial state, set the current dialogue utterance as "current dialogue behavior: self-determination and comfort, information provision, other, suggestion, etc.", and initialize the policy distribution π as a uniform distribution.

[0018] Step 2: Execute the Level-1 Rollout process. In State +1, select the action sequence "Information Provision" based on the current policy distribution, and generate the corresponding system response using the large language model. Continue executing Level-1 to simulate the next round, selecting the action sequence "Information Provision...Suggestion Giving" and generating the corresponding dialogue content.

[0019] Step 3: Calculate the final score and adjust the strategy. After completing the simulation at Level-1, the final score is obtained based on the dialogue quality assessment. This score reflects the effectiveness of the dialogue sequence in solving the user's problem. Based on the obtained high-quality action sequence "information provided...suggestion given", the strategy distribution π is adjusted by increasing the weight of successful actions and decreasing the probability of other actions, making the strategy more inclined to generate high-quality dialogue paths.

[0020] Step 4: The nested rollback strategy-adaptive dialogue strategy planning method collects the best results from Level 1 at Level 2, compares the current simulated score with the historical best score, and updates the best score record if the current score is better. This process continuously optimizes the strategy quality through multiple iterations and nested levels, allowing each level's strategy to learn from the successful experiences of the next level, thus achieving gradual refinement and improvement of the strategy.

[0021] Step 5: Implement iterative iteration and policy convergence. The entire nested rollback policy-adaptive dialogue policy planning method forms a complete policy learning loop by repeatedly executing the Level-1 simulation, evaluation, and update process. In each iteration, the policy distribution π is continuously adaptively adjusted based on successful dialogue sequences, gradually converging to the optimal policy that can generate high-quality dialogue. Through this multi-level nesting and iterative optimization mechanism, the algorithm can efficiently search the dialogue policy space and find the policy configuration most suitable for the current dialogue scenario.

[0022] The following section provides a detailed explanation using the bidding scenario as an example.

[0023] S1. The dialogue strategy planning task is formalized as a Markov Decision Process (MDP), where there are t rounds of dialogue between the user and the system, and each system response is associated with a specific dialogue behavior. In the bidding scenario, the dialogue is modeled as a Markov Decision Process (MDP), where the user (seller) and the system (buyer's agent) engage in multi-round dialogue. In each round, the system selects a specific dialogue behavior, such as bidding, rejecting, or proposing conditions, which are provided by a predefined action space A. S2. Initialize the policy π as a softmax probability distribution covering all actions. In each round, legal actions are first screened out, and then dialogue behaviors are selected from π by sampling. The LLM generates natural language responses based on the prompt template of the behavior and advances the state. The complete dialogue trajectory is obtained through playout. The path quality is measured by the task achievement and round penalty scores. In the NRPA algorithm, multiple iterations are performed: recursively searching for candidates, scoring and selecting the best, and updating the log weight of the policy with the best trajectory (promoting the best action and suppressing others). At the same time, early stopping and robust convergence are combined to allow π to gradually focus on high-value dialogue decisions. The simulation process of the nested rollback strategy algorithm includes the following steps: S100. Starting from the current dialogue state s, first generate a set of legal actions (such as offering a price, counter-offering, making concessions, probing for a deal, confirming a deal, etc.) based on the bidding rules and dialogue progress. Map the weights corresponding to the current strategy to a softmax probability distribution, normalize only on legal actions, and sample the dialogue actions to be executed in this round according to the probability, so that high-value actions are tried more frequently, while retaining the necessary exploration space; Action sampling: Action sampling: ; In the above formula: A(s) is the action set containing all actions, w a w represents the weight of the current action. b b represents the weight of the remaining actions; b represents the remaining actions. S22. Map state s and action a to the corresponding bidding prompt template, call the large language model to generate system response u, and use the environment transition function to advance the dialogue: write u into the dialogue history, update the bid, round and termination flag and other state variables according to u and business rules to obtain the new state s; at the same time, add the action a of this round to the action sequence as the basis for subsequent evaluation and strategy update. Large Language Model Generation and State Update: ; ; ; In the above formula: S t For the current state of the entire conversation, a t In state S t The selected dialogue action, u t The dialogue generated by the current large model, The environment transition function; inputs the current state, action, and generated response, and outputs the next state S. t+1 seq represents the action sequence; S23. Repeat execution until the termination condition is triggered (such as reaching the transaction / rejection limit, or the round limit), and obtain the final state s. T The simulation path is scored once with the complete action sequence, and the reward score and corresponding action sequence are returned. In the bidding scenario, the reward is based on whether a transaction is completed, and the penalty is combined with the dialogue round factor to encourage the completion of effective transactions in fewer rounds. ; In the above formula: R(s) T ) for the termination state s T Reward assessment; {Deal(s T )} is the indicator function; λ is the dialogue turn penalty coefficient, turns is the dialogue length per turn, and T is the total number of steps.

[0024] S3. Set the number of iterations and the depth of the hierarchy to constrain the search cost of NRPA. The deeper the hierarchy and the more iterations per level, the more fully the strategy adapts and the higher the probability of finding high-quality dialogue paths, but the computational cost increases exponentially. S4. For bidding scenarios, use Nested Rollback Aspect (NRPA) for dialogue strategy planning: Set the layer depth and the number of iterations per layer. At each layer, simulate the dialogue multiple times starting from the current strategy π for legal bidding actions (such as opening bid, counter-bid, concession, and transaction) to obtain candidate negotiation paths. The large language model scores the complete state based on bidding features (such as bid difference, concession amount, anchoring strength, and counterparty's transaction signal). Select the optimal sequence to update the weights of the strategy pair (strengthen the preferred action and suppress the rest). Simulate, score, select, update, and roll back upward layer by layer to find a strategy with a shorter path and better fit for the bidding context. S5. In the bidding scenario, design a task-based reward for the simulated path of NRPA: use whether the question is concluded as the main indicator (if a conclusion signal is detected, a base score of 1.0 is given, otherwise 0 is given), and add a dialogue round penalty (the more rounds, the greater the penalty), so as to encourage the completion of negotiations in shorter rounds. Ultimately, under the constraints of prioritizing the achievement of goals and shorter and more stable paths, the strategy is driven to converge to a high success rate and efficient dialogue decision in the bidding scenario.

[0025] Furthermore, the simulation process of the NRPA algorithm includes: starting from the current state s, sampling action a according to probability using a softmax probability distribution; calling the large language model to generate a system response u based on state s and action a, updating state s and adding action a to the sequence; repeating the above process until the dialogue ends, and returning the reward score and action sequence; the policy adaptive update process includes: receiving a high-quality action sequence and adjusting the policy weights; for each action a in the sequence, using a penalty-reward mechanism to calculate the normalization factor; performing weight updates for all actions a'∈A; and adding additional weights to the current action a, enabling the policy to learn and optimize from successful experiences.

[0026] The pseudocode for the algorithm is as follows: Figure 4 As shown, starting from state s, a "trial sequence" is generated according to policy π, and the score of the sequence is calculated; Input: initial state s, policy π, generative large language model M θ Action space A; Logic: Initialize an empty sequence, reset state s=s i (Initial state); while state s has not reached the "termination state", the following loop is executed: calculate the sum of probabilities of all actions in the action space z = ∑ a'∈A e π(a') (Probability normalization); probability distribution according to strategy π ((1 / z)·e π(a) Randomly select an action 'a'; invoke the generative large language model Mθ to generate subsequent content u based on "state s + action a". sys (For example, in a text generation scenario, u is the text fragment generated by the model); Update state s = s∪{u} sys} (Incorporate the generated content into the state), and add action a to the sequence; after the sequence is generated, calculate the final score using the reward function R(s), and return (score, sequence).

[0027] It should be noted that the multi-level policy optimization process includes: determining the current level; if level=0, directly performing simulation operations to return the score and sequence; if level>0, entering the recursive process, performing N iterations, each iteration calling the NRPA method of the lower level to obtain the path score and sequence, comparing and updating the optimal score and sequence, and then calling the adaptive function to update the policy π according to the optimal sequence.

[0028] The pseudocode of the algorithm is as follows Figure 5 As shown, the core recursive logic of NRPA finds the optimal sequence through "nested iteration + policy optimization". Inputs include: level (recursion level, controlling optimization precision), π (current policy, describing the probability of action selection), s (initial state), and the generative large language model M. θ Given the number of iterations N and the action space A; if level=0, call PLAYOUT to directly generate a "trial sequence" from the current strategy π and state s and return the score.

[0029] The method further includes a zero-shot model training paradigm, which enables multi-level exploration through prompting a large language model (LLM) during the decision-making process without requiring a large amount of training data for a specific domain; it simulates user and system response interactions, evaluates the current task progress, and optimizes the selection probability of the next dialogue behavior; and it adopts a dynamic rollout strategy instead of a static uniform random strategy, adjusting the strategy weights in real time within the nested hierarchy.

[0030] The reward function design includes: evaluating the quality of the final state s of the simulated dialogue path, measuring whether the user's problem has been successfully resolved in the emotional support scenario; focusing on the final state evaluation, implicitly introducing a discount factor by imposing a dialogue round penalty to incentivize the model to achieve the goal in fewer rounds.

[0031] The pseudocode of the algorithm is as follows Figure 4 As shown, adjust the policy π based on the "high-scoring sequence" to make the policy more inclined to generate such high-quality sequences; input: current policy π, high-scoring sequence, learning rate α (controlling the policy update magnitude), state s; initialize the new policy π'=π, reset the state s=s i (Initial state), traverse each action a in the high-scoring sequence: calculate the sum of probabilities z = ∑ a'∈A e π(a')(Used for probability normalization); For each action a', adjust its probability: if a' is the currently traversed action a, increase its probability (π'(a)+α); otherwise, decrease its probability (π'(a')-α·(1 / z)·e π(a') (Ensure the probability sum remains z); select action a using the adjusted policy π', update state s (simulating the state transition after executing action a), and return the updated policy π'.

[0032] On the other hand, a dialogue strategy planning system based on the NRPA algorithm is provided. The system includes: a large language model module for generating system responses based on the current dialogue state and selected actions; a strategy management module for maintaining and updating the dialogue strategy weight vector π; an NRPA planner module for implementing any of the above methods to perform multi-level optimization of the dialogue strategy; a state tracking module for maintaining the dialogue history state and evaluating the dialogue progress; and a reward evaluation module for calculating the final score of the dialogue path and providing feedback signals for strategy updates.

[0033] Figure 3 For human evaluation of results, large language models may lack rigorous criteria for judging persuasive elements due to their preference for self-generated dialogues. Therefore, 50 dialogues were randomly selected from each of the P4G and CIMA datasets for multi-dimensional human evaluation. The results show that as search depth increases, the NRPA-GD model significantly improves performance in both tasks: it can more effectively provide emotional support to promote donation behavior (P4G) and also provide more accurate learning cues (CIMA).

[0034] Table 1 shows the effectiveness of the proposed framework under different model sizes.

[0035] As shown in Table 1, three different sizes of LLM were used for testing: gpt-4o-mini, llama-3.1-8b-chat, and qwen-3-0.6b. By evaluating the performance of NRPA-GD at the first and second levels respectively, when using gpt-4o-mini and llama-3.1-8b-chat, NRPA-GD showed a stable performance improvement trend on the CraigslistBargain dataset as the nesting level increased. For the qwen-3-0.6b model with the smallest number of parameters, it failed to complete the ESConv experimental task within the preset number of dialogue rounds due to parameter size limitations, but it still demonstrated the advantages of NRPA-GD on other datasets: on the CIMA dataset, the method achieved the expected high success rate performance in the AT (average number of dialogue rounds) and SR (success rate) indicators of the second level with a shorter number of rounds; while on the CraigslistBargain dataset, the SL (sell price to initial price ratio) value of the second level was significantly improved to 0.6307, which was comparable to the performance of gpt-4o-mini and gpt-3.5-turbo. Experiments have shown that even with a small number of parameters, NRPA-GD can still achieve performance levels that surpass those of pre-trained models with large parameters on specific tasks.

[0036] Table 2 compares the test results of NRPA-GD with GDP-Zero on the P4G and ESConv datasets for different nesting levels.

[0037] The experiments used gpt-4o-mini as the main model and gpt-3.5-turbo for evaluation. Results show that NRPA-GD's first level performs exceptionally well: achieving a 68.61% win rate on the P4G dataset with less time consumption; and maintaining the lowest time cost with a 63.40% win rate on the ESConv dataset. Although upgrading to the second level increases computational burden, it achieves optimal performance on both datasets.

[0038] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0039] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A lightweight large language model dialogue planning method based on nested rollback strategy adaptation, characterized in that, The method includes the following steps: S1. The dialogue strategy planning task is formalized as a Markov decision process, in which there are t rounds of dialogue between the user and the system, and each system response is associated with a specific dialogue behavior. S2. The initialization strategy is a softmax probability distribution covering all actions. In each round, legal actions are first screened out, and then dialogue behaviors are selected from the initialization strategy by sampling. The large language model generates natural language responses based on the prompt template of the behavior and advances the state. The complete dialogue trajectory is obtained through playout. The path quality is measured by the task achievement and round penalty scores. The nested rollback strategy algorithm iterates multiple times: recursively searching for candidates, scoring and selecting the best, and updating the log weight of the strategy with the best trajectory. At the same time, it combines early stopping and robust convergence, so that the initialization strategy gradually focuses on high-value dialogue decisions. The simulation process of the nested rollback strategy algorithm of S2 includes the following steps: S21. Starting from the current dialogue state s, first generate a set of legal actions based on the bidding rules and dialogue progress, map the weights corresponding to the current strategy to a softmax probability distribution, normalize only on legal actions, and sample the dialogue action a to be executed in this round according to the probability, so that high-value actions are tried more frequently, while retaining the necessary exploration space. Action sampling: ； In the above equation: A(s) is the action set containing all actions, w a is the current action weight, w b is the remaining action weight; b is the remaining action. S22. Map state s and action a to the corresponding bidding prompt template, call the large language model to generate system response u, and use the environment transition function to advance the dialogue: write u into the dialogue history, update the bid, round and termination flag and other state variables according to u and business rules to obtain the new state s; at the same time, add the action a of this round to the action sequence as the basis for subsequent evaluation and strategy update. Large Language Model Generation and State Update: ； ； ； In the above formula: S t For the current state of the entire conversation, a t In state S t The selected dialogue action, u t The dialogue generated by the current large model, The environment transition function; inputs the current state, action, and generated response, and outputs the next state s. t+1 seq represents the action sequence; S23. Repeat execution until the termination condition is triggered to obtain the final state s_T and the complete action sequence. Score the simulated path once and return the reward score and the corresponding action sequence. In the bidding scenario, the reward is based on whether a transaction is completed as the main objective signal, and is combined with the dialogue round factor for punishment, thereby encouraging the completion of effective transactions in fewer rounds. ； In the above formula: R(s T ) is the reward evaluation of the termination state s T ; {Deal(s_T)} is an indicator function; if the terminal state s T occurs Deal, it takes the value 1, otherwise 0, λ is the turn penalty coefficient (λ>0), turns: the length of the dialogue by turns, T is the total number of steps; S3. Set the number of iterations and the level depth to constrain the search overhead of the nested rollback strategy; S4. Dialogue strategy planning using nested rollback strategy: Set the level depth and the number of iterations per level. At each level, simulate the dialogue multiple times starting from the current strategy for the legal bidding action to obtain candidate negotiation paths. The large language model scores the complete state in combination with bidding features. Select the optimal sequence to update the strategy weights. Simulate, score, select the best, update and roll back up layer by layer to find a better strategy. S5. Evaluate the quality of the final state of the simulated dialogue path through a reward function, measure the degree of user problem-solving in emotional support scenarios, impose dialogue round penalties to implicitly introduce a discount factor, and incentivize the model to achieve the goal in fewer rounds.

2. The lightweight large language model dialogue planning method based on nested rollback strategy adaptation according to claim 1, characterized in that, Specifically, S1 involves modeling the dialogue as a Markov decision process, where the user and the system engage in multiple rounds of dialogue. In each round, the system selects a specific dialogue behavior, which is provided by a predefined action space A.

3. The lightweight large language model dialogue planning method based on nested rollback strategy adaptation according to claim 2, characterized in that, The update process of the nested rollback strategy algorithm includes the following steps: Receive high-quality action sequences and adjust policy weights; For each action 'a' in the sequence, a penalty-reward mechanism is applied, and a normalization factor is calculated. For all actions a'∈A, perform a weight update; For the current action 'a', add extra weight to enable the strategy to learn and optimize from successful experiences.

4. The lightweight large language model dialogue planning method based on a nested rollback strategy adaptation of claim 1, characterized in that, The process of achieving strategy optimization through multi-level exploration is as follows: Determine the current level. If level=0, directly execute the simulation operation and return the score and sequence. If level > 0, then enter the recursive process and execute N iterations. In each iteration, call the nested rollback strategy method of the lower level to obtain the path score and sequence, compare and update the optimal score and sequence, and then call the adaptive function to update the strategy according to the optimal sequence.

5. The lightweight large language model dialogue planning method based on nested rollback strategy adaptation according to claim 4, characterized in that, Multi-level systems also include the following steps: By prompting a large language model during the decision-making process, multi-level exploration can be achieved without the need for large amounts of training data specific to a particular domain; Simulate user and system response interactions, assess current task progress, and optimize the probability of selecting the next dialogue action; A dynamic rollout strategy is adopted instead of a static uniform random strategy, and the strategy weights are adjusted in real time within the nested hierarchy.

6. The lightweight large language model dialogue planning method based on a nested rollback strategy adaptation of claim 1, characterized in that, The reward function design includes: Evaluate the quality of the final state s of the simulated dialogue path and measure whether the user's problem has been successfully resolved in an emotional support scenario; Focusing on the final state assessment, a discount factor is implicitly introduced by imposing a dialogue round penalty, which incentivizes the model to achieve the goal in fewer rounds.

7. The system for lightweight large language model dialogue planning method based on nested rollback strategy adaptation according to any one of claims 1-6, characterized in that, The system includes: The large language model module is used to generate system responses based on the current dialogue state and the selected action. The strategy management module is used to maintain and update the dialogue strategy weight vector; A nested rollback strategy planner module is used to implement the method of any one of claims 1 to 6 to perform multi-level optimization of the dialogue strategy; The state tracking module is used to maintain the historical state of the dialogue and evaluate the progress of the dialogue. The reward evaluation module is used to calculate the final score of the dialogue path and provide feedback signals for policy updates.

Images