Generative dialogue optimization method and system based on multi-step intensified simulation

Abstract

The invention discloses a generative dialogue optimization method and system based on multi-step intensified simulation, and the method comprises the steps: generating a plurality of candidate replies based on the current dialogue history through a basic dialogue model; using the user simulator to generate a dialogue chain for each candidate reply; calculating a reward for each dialogue chain, selecting K dialogue chains with the highest reward value as high-quality samples, and taking the dialogue chain with the lowest reward value as a negative sample; updating parameters of the basic dialogue model by using the high-quality sample and the negative sample through a contrast learning and reinforcement learning method to obtain a trained basic dialogue model as a customer service model; and the customer service model generates a reply according to a statement input by the user. According to the conversation generation method, multi-round conversation dynamic planning, medical scene adaptability optimization and efficient strategy exploration can be fused, and double optimization of user trust establishment and business goal achievement is achieved while compliance is ensured.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a generative dialogue optimization method and system based on multi-step reinforcement simulation. Background Technology

[0002] With the deepening application of artificial intelligence technology in the healthcare field, intelligent customer service systems based on generative dialogue models are gradually becoming important auxiliary tools for doctor-patient communication. Such systems not only need to provide accurate medical consultation responses, but also need to build user trust during the dialogue and guide them to complete specific goals (such as making appointments and collecting contact information). However, existing technologies have significant limitations in optimizing multi-turn dialogue strategies, specifically manifested in the following problems:

[0003] (1) Short-sighted response generation. Current mainstream dialogue model training methods (such as supervised fine-tuning SFT and direct preference optimization DPO) typically optimize the quality of the current response based on the history of a single-turn dialogue, lacking an overall plan for long-term dialogue goals. For example, the model may generate immediate and credible medical advice, but does not consider how to naturally guide the user to actively provide contact information in subsequent conversations. This "single-step optimization" strategy can easily lead to the dialogue path deviating from the final goal, or require too many redundant turns to achieve business requirements, affecting the user experience.

[0004] (2) Insufficient policy robustness. Traditional reinforcement learning schemes (such as PPO optimization based on proximal policies) can introduce long-term goals through reward functions, but the acquisition of real user feedback data they rely on is costly, and sparse reward signals (such as giving positive feedback only when finally successful) make model convergence difficult. In addition, user intentions in medical scenarios are complex and diverse (such as anxiety about illness, privacy concerns, etc.), and existing models have difficulty dynamically predicting user reactions and adjusting their communication strategies, which can easily lead to user resistance due to inappropriate responses.

[0005] (3) Deviation between simulated environment and real-world scenario. Some studies have attempted to expand the training sample by generating virtual dialogue data through user simulators. However, existing simulators are mostly built based on rules or simple language models, which cannot realistically reflect the changes in users' psychological states in medical scenarios (such as the process of trust accumulation and differences in privacy sensitivity). This deviation may cause the model to perform well in the simulated environment, but fail in real dialogues due to insufficient policy generalization.

[0006] To address the aforementioned issues, there is an urgent need for a dialogue generation method that can integrate multi-turn dialogue dynamic planning, medical scenario adaptability optimization, and efficient strategy exploration, so as to achieve dual optimization of user trust building and business goal achievement while ensuring compliance. Summary of the Invention

[0007] The purpose of this invention is to solve the problems in the prior art.

[0008] The technical solution adopted by this invention to solve its technical problem is: to provide a generative dialogue optimization method based on multi-step reinforcement simulation, comprising the following steps:

[0009] Using a basic dialogue model, several candidate responses are generated based on the current dialogue history; a user simulator is used to generate a dialogue chain for each candidate response.

[0010] A reward is calculated for each dialogue chain, and the K dialogue chains with the highest reward values ​​are selected as high-quality samples, while the dialogue chain with the lowest reward value is selected as a negative sample. The reward value includes an immediate reward calculated based on candidate responses and a long-term reward calculated based on the dialogue chain.

[0011] By using contrastive learning and reinforcement learning methods, the parameters of the basic dialogue model are updated using high-quality samples and negative samples to obtain a well-trained basic dialogue model as a customer service model; the customer service model generates responses based on user input statements.

[0012] Preferably, the basic dialogue model uses a finely tuned Qwen2.5-7b-instruct, and the user simulator uses DeepSeek-V3.

[0013] Preferably, generating a dialogue chain for each candidate response using a user simulator includes the following steps:

[0014] The initial step is to use the candidate responses as the current response;

[0015] The user response generation step involves the user simulator generating a user response based on the current reply.

[0016] In the response generation step, the basic dialogue model generates a new response based on the user's response.

[0017] Repeat the steps, using the new response as the current response, to generate user responses and generate responses until the required number of user responses are obtained;

[0018] In the process of generating user responses in the user simulator, the user's focus is predicted by combining the content of the current reply and the preset application scenario, and a user response is generated based on the focus.

[0019] Preferably, the instant reward calculated based on candidate responses includes:

[0020] Determine whether the candidate response meets the current stage goal of the dialogue, and give the action intent matching degree based on the judgment result;

[0021] Determine whether the candidate responses express medical knowledge in a professional field, and assign a professional level rating based on the determination result.

[0022] Preferably, the long-term reward based on the dialogue chain computation is obtained by evaluating the following aspects of the dialogue chain:

[0023] Analyze the emotional changes among multiple user responses and assess trust levels based on these emotional changes;

[0024] The goal achievement rate is assessed based on whether the preset user goals are met among multiple user responses.

[0025] Predict users' willingness to continue participating in the next three rounds of dialogue to assess the sustainability of the conversation.

[0026] Preferably, the method of updating the parameters of the base dialogue model using high-quality and negative samples through contrastive learning and reinforcement learning includes the following steps:

[0027] The Direct Preference Optimization algorithm is used to update the parameters of the basic dialogue model by comparing the differences between high-quality samples and negative samples.

[0028] By using a user simulator as the adversary, the cumulative reward of the basic dialogue model in simulated dialogue is maximized.

[0029] Preferably, the Direct Preference Optimization algorithm is used to update the parameters of the basic dialogue model by comparing the differences between high-quality samples and negative samples. The loss function used is expressed as follows:

[0030]

[0031] in, This represents the mathematical expectation operator, where x represents the input context, and y represents the mathematical expectation operator. w Indicates a high-quality response sample, y l Represents a negative sample, π θ Let π represent the target model to be optimized. ref denoted as the reference model, σ represents the Sigmoid function, and β represents the temperature coefficient.

[0032] Preferably, the loss function used to maximize the cumulative reward of the base dialogue model in the simulated dialogue, with the user simulator as the adversary, is expressed as follows:

[0033]

[0034] in, The expression represents the mathematical expectation operator, k represents the dialogue turn, and N represents the number of turns to explore backward.

[0035] Preferably, the method of contrastive learning and reinforcement learning updates the parameters of the base dialogue model using high-quality and negative samples, and the overall loss function is expressed as:

[0036] L(total)=λ DPO L DPO +λ ADV L ADV ;

[0037] Where, λ DPO express The weighting coefficient, λ ADV express The weighting coefficients.

[0038] This invention also provides a generative dialogue optimization system based on multi-step reinforcement simulation, comprising:

[0039] The sample generation module uses a basic dialogue model to generate several candidate responses based on the current dialogue history; it uses a user simulator to generate a dialogue chain for each candidate response.

[0040] The sample partitioning module calculates a reward for each dialogue chain, selects the K dialogue chains with the highest reward values ​​as high-quality samples, and uses the dialogue chain with the lowest reward value as a negative sample; the reward value includes an immediate reward calculated based on candidate responses and a long-term reward calculated based on the dialogue chain.

[0041] The dynamic optimization module uses contrastive learning and reinforcement learning methods to update the parameters of the basic dialogue model using high-quality and negative samples, resulting in a well-trained basic dialogue model that serves as the customer service model. The customer service model generates responses based on user input.

[0042] The present invention has the following beneficial effects:

[0043] (1) Dynamic optimization of long-term strategy. This invention designs a closed-loop mechanism of “generation-simulation-evaluation”, which enables the model to make forward-looking decisions through multi-round dialogue path prediction and avoids short-sighted response generation; it integrates immediate rewards (professionalism, emotional comfort) and long-term rewards (speed of goal achievement, dialogue sustainability) to achieve dual-path alignment between medical consultation and business goals.

[0044] (2) Realism assurance driven by deep intelligent agents. This invention, based on the DeepSeek-V3 simulator, breaks through the limitations of traditional rules and achieves multimodal intent understanding (text + potential user state) through prompt words, generating responses that conform to the psychological laws of actual scenarios; the user profile dynamic update module supports personalized dialogue path simulation, such as automatically focusing on cost explanation for economically sensitive users.

[0045] (3) High-efficiency multi-candidate game mechanism. The candidate response generator of this invention controls the response by temperature parameter levels (conservative / balanced / aggressive strategies), balancing diverse generation with computational efficiency and avoiding candidate redundancy. Under the adversarial training framework, the bidirectional game between the customer service model and the DeepSeek-V3 simulator continuously improves the robustness of the strategy and enhances the generalization ability to deal with complex user intentions.

[0046] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the steps of a generative dialogue optimization method based on multi-step reinforcement simulation, according to an embodiment of the present invention.

[0048] Figure 2 This is a system architecture diagram of a generative dialogue optimization system based on multi-step reinforcement simulation, according to an embodiment of the present invention. Detailed Implementation

[0049] See Figure 1 The diagram shows the steps of a generative dialogue optimization method based on multi-step reinforcement simulation according to an embodiment of the present invention, including the following steps:

[0050] S101: Using a basic dialogue model, generate several candidate responses based on the current dialogue history; use a user simulator to generate a dialogue chain for each candidate response;

[0051] S102, calculate the reward for each dialogue chain, select the K dialogue chains with the highest reward values ​​as high-quality samples, and select the dialogue chain with the lowest reward value as a negative sample; the reward value includes the immediate reward calculated based on the candidate response and the long-term reward calculated based on the dialogue chain.

[0052] S103, by using contrastive learning and reinforcement learning methods, the parameters of the basic dialogue model are updated using high-quality samples and negative samples to obtain a trained basic dialogue model as a customer service model; the customer service model generates a response based on the user's input statement.

[0053] Specifically, step S101 includes the following steps:

[0054] S1011, Receive current dialogue history H t ={u1, c1, ..., u t}, where u i For user statements, c i For customer service statements;

[0055] S1012, use the basic dialogue model to simultaneously generate 3 candidate responses.

[0056] S1013, for each candidate response Use DeepSeek-V3 user simulator to generate user response And continue generating the next round of customer service responses. Forming a two-round simulated dialogue chain Specifically, it utilizes the features of DeepSeek-V3 to analyze user emotions (such as anxiety and trust) in conversations through its understanding capabilities; and captures potential intentions (such as privacy concerns and treatment needs) in historical conversations based on long context windows.

[0057] Specifically, the base dialogue model used is a fine-tuned Qwen2.5-7b-instruct. Specifically, it involves dialogue fine-tuning based on a vertical business scenario (medical marketing dialogue), employing full parameter fine-tuning using a transformers trainer. However, this fine-tuning only learns the dialogue style and is not effective in action planning. Therefore, subsequent reinforcement learning is needed for alignment and optimization of multi-turn action planning.

[0058] Specifically, in step S1013, DeepSeek-V3 is used to simulate user responses, and the example of the prompt word template used is as follows:

[0059] Example 1: You are a searcher who uses the search term "keyword" to conduct an online consultation with a department. Your consultation information is [dialog]. You will continue the chat using this information and existing conversation history. Please note that the doctor can only diagnose you based on the conversation history; they cannot see your personal information. When consulting online, please be timely and reply with short sentences (no more than 15 characters; you can simply reply with words like "uh-huh / yes / no" to answer the doctor's questions).

[0060] Example 2: You are here to inquire about the cause of your illness, etc. The doctor will answer your questions by asking about your symptoms. You need to answer his questions to cooperate with him. Please note that you are having a conversation with a doctor, and your replies need to be consistent with your persona. If the doctor gives you advice, you can follow it, and you can ask the doctor if you have any questions. If the doctor is explaining something to you, you can ask questions, but please note that you are not a doctor. Your replies need to be very conversational, do not address the other person by name, and be as casual as possible (typos are acceptable). Please reply with a short sentence that is different from the one in the chat history. The meaning of the returned sentence must not be the same as any sentence in [client_sentence], and the output must be entirely text. If you have no further questions and want to end the conversation, you can add a note to the end of your reply. <dialogover>For example: "I continued to reply: Okay" <dialogover>If the doctor wants to add your contact information, you can provide your WeChat or phone number, or refuse, such as: "I continue replying: My WeChat is XXXX", "I continue replying: My phone number is XXXXXXXXXXX", or "I continue replying: Okay, I'll add you", "I continue replying: I don't want to add you on WeChat", "I continue replying: Let's chat here". The chat history is [chat], pay special attention to the following chat history [last_sentence], please continue replying to the doctor. The return format is "I continue replying: XXXXX".

[0061] For example, in S1011, the user's current question is "My blood sugar has been fluctuating a lot recently, do I need to be hospitalized?"; in S1012, the following candidate responses are generated using a slightly modified Qwen2.5-7b-instruct: Candidate 1, "It is recommended to check fasting blood glucose and glycated hemoglobin first, and evaluate according to the results.", mainly providing professional guidance; Candidate 2, "Can you describe your specific diet and medication?", mainly for information gathering; Candidate 3, "Our hospital's endocrinology department has remote monitoring services, do you need to make an appointment for us?", mainly for goal orientation. In S1013 (taking candidate 3 as an example), DeepSeek-V3 simulates the user's response "Is remote monitoring free? How does it work?" (potential intent = price sensitivity), and the simulated customer service continues to generate the next round of responses: "The cost is 300 yuan per month. After you leave your mobile phone number, we will send you the operation guide." (triggering the goal). If the simulated user responds with their contact information, the goal is achieved.

[0062] Specifically, in S102, two types of rewards are calculated for each candidate response and its simulated dialogue chain, and then the candidate responses are classified according to... The samples are ranked, with the Top-K selected as high-quality positive samples and the bottom-ranked samples as negative samples. The two types of rewards include immediate rewards and long-term rewards.

[0063] In this embodiment, the immediate rewards include action intent matching and domain expertise. Action intent matching uses DeepSeekv3 to determine the intent of the current speech action, judging whether the candidate speech matches the current dialogue stage goal (e.g., rounds 1-3 should be "information gathering," rounds 3-5 "trust building," and rounds 5-7 "goal guidance"). Prompt: Based on the dialog_record above, determine the intent category of the current sentence: information gathering, trust building, goal guidance, or other. Select one of these four categories for output. A correct action match earns 1 point, and an incorrect match earns 0 points. Domain expertise uses DeepSeekv3 to determine the expertise of the current speech. Prompt: Based on the dialog_record above, determine the expertise of the current sentence. If it expresses professional medical knowledge, the expertise is high; if it reflects some professional relevance, the expertise is medium; if it does not reflect any medical knowledge at all, the expertise is low. High expertise earns 1 point, medium expertise earns 0.5 points, and low expertise earns 0 points.

[0064] In this embodiment, the long-term rewards (simulated rounds) include trust level, goal achievement rate, and dialogue sustainability. Trust level is analyzed using DeepSeekv3 to assess the emotional changes in subsequent user statements during the simulated dialogue (e.g., "Thank you, I understand" vs. "I don't quite believe it"). A positive change in emotion earns 1 point; no change earns 0.5 points; and a negative change earns 0 points. Goal achievement rate is assessed using DeepSeekv3 to determine whether the simulated dialogue guides the user to proactively provide contact information within a preset number of rounds (e.g., the user asks "How do I contact an expert?"). A visitor earns 1 point if they directly leave contact information or proactively inquire about customer service contact information during the simulated round; expressing a willingness to leave contact information earns 0.5 points; otherwise, 0 points are awarded. Dialogue sustainability is assessed by using DeepSeek-V3 to predict the user's willingness to participate in the next 3 rounds of dialogue (0-1 probability). A reward of +1 is given for a "probability of the user continuing to ask questions > 80%", and +0.5 is given for a "probability of the user continuing to ask questions > 50%".

[0065] Specifically, the reward calculation formula is expressed as follows:

[0066]

[0067] in, R represents the reward value of the i-th dialogue chain; imm Indicates an immediate reward, used to measure the language quality, informativeness, and positive sentiment of candidate responses; R long This indicates a long-term reward; This represents a candidate response in the i-th dialogue chain. This represents the second user's reply in the i-th dialogue chain; γ and δ are both hyperparameters, optimized through grid search.

[0068] Specifically, in S103, the contrastive learning training employs the DPO (Direct Preference Optimization) algorithm, which updates the dialogue model parameters by comparing the differences between high-quality samples and negative samples. The formula is as follows:

[0069]

[0070] in, This represents the DPO loss function value. By minimizing this loss, the model is made to favor generating high-quality responses over ordinary ones. In diabetes consultations, this prioritizes professional guiding language over directly soliciting contact information. The DPO loss is a static single-round script reward; the optimization goal is to generate fluent, doctor-style scripts, and the primary evaluation focuses on the quality of the scripts being optimized. This is the mathematical expectation operator, which means averaging all samples in the training dataset. It is calculated by averaging the mini-batch during batch training. σ is the sigmoid function, expressed as... Its function is to map the difference to the (0,1) probability interval, and its significance in the scenario is to calculate the probability that a high-quality response is better than a normal response. β is a temperature coefficient (hyperparameter), typically ranging from 0.1 to 1.0, used to control the model's sensitivity to differences in preferences. A larger β more strictly distinguishes between good and bad samples, while a smaller β allows for smaller quality differences; in medical scenarios, β = 0.5 is recommended (balancing professionalism and guidance). π θ This represents the target model to be optimized (with parameters θ), and the Transformer-based dialogue generation model (the generation model to be trained), whose parameters are dynamically updated during training. π ref This represents the reference model (with fixed parameters), derived from the SFT (Supervised Fine-tuning) base medical customer service model, serving as an optimization baseline to prevent the model from deviating from the original distribution. x represents the input context, which is the current dialogue history Ht. y w This indicates high-quality response samples, derived from top-K high-reward candidate phrases; y l This indicates a negative sample (ordinary sample) from a low-reward candidate statement.

[0071] The core comparison terms include: This indicates the advantage of high-quality samples; This indicates the advantage of the ordinary sample.

[0072] Specifically, in S103, the DeepSeek-V3 simulator is used as the adversary, maximizing the cumulative reward of the customer service model in the simulated dialogue. The Adversity (Adv) loss is the reward for the current round's action, and the optimization objective is to optimize the action strategy for the current round by exploring future action paths. The adversarial loss is expressed as:

[0073]

[0074] Where k represents the cumulative reward step size from the current customer service response (round 1) to the simulated next round of customer service response (round 2). N represents the number of rounds of exploration (exploration depth). N=1 means exploring one round downwards, N=2 means exploring two rounds downwards from the current round, and so on. Exploring one round means generating a customer service model c(t+1), deeplyseeking to simulate a visitor generating u(t+1), and then the customer service model generates c(t+2). Exploring two rounds means, based on c(t+2), deeplyseeking to simulate a visitor generating u(t+2), and then the customer service model generates c(t+3). The reason for designing N is to improve the model's multi-step exploration capability and increase diversity. The value of N is an integer greater than or equal to 1. In this embodiment of the invention, N=2 is chosen; k=1 represents the reward for the currently generated customer service script, and k=2 represents the reward for the next round of customer service script in the simulated dialogue after the visitor generates a sentence. Choosing N=2 is for performance reasons. Since each candidate needs to be simulated independently for two rounds (3 candidates × 2 rounds = 6 paths), if it is expanded to N=3 rounds, the number of paths will increase to 12, and the GPU memory usage will exceed the 24GB threshold, which will put a lot of pressure on the online system.

[0075] The total loss is expressed as:

[0076] L(total)=λ DPO L DPO +λ ADV L ADV ;

[0077] See Figure 2 The diagram shown is a structural diagram of a generative dialogue optimization system based on multi-step reinforcement simulation according to an embodiment of the present invention, comprising:

[0078] The sample generation module 201 uses a basic dialogue model to generate several candidate responses based on the current dialogue history; and uses a user simulator to generate a dialogue chain for each candidate response.

[0079] The sample partitioning module 202 calculates a reward for each dialogue chain, selects the K dialogue chains with the highest reward values ​​as high-quality samples, and uses the dialogue chain with the lowest reward value as a negative sample; the reward value includes an immediate reward calculated based on candidate responses and a long-term reward calculated based on the dialogue chain.

[0080] The dynamic optimization module 203 updates the parameters of the basic dialogue model using high-quality and negative samples through contrastive learning and reinforcement learning methods, and obtains a trained basic dialogue model as a customer service model; the customer service model generates a response based on the user's input statement.

[0081] As can be seen, the present invention can integrate a dialogue generation method that combines multi-turn dialogue dynamic programming, medical scenario adaptability optimization and efficient strategy exploration, so as to achieve dual optimization of user trust building and business goal achievement while ensuring compliance.

[0082] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.< / dialogover> < / dialogover>

Claims

1. A generative dialogue optimization method based on multi-step reinforcement simulation, characterized in that, Includes the following steps: Using a basic dialogue model, generate several candidate responses based on the current dialogue history; Use a user simulator to generate a dialogue chain for each candidate response; A reward is calculated for each dialogue chain, and the K dialogue chains with the highest reward values ​​are selected as high-quality samples, while the dialogue chain with the lowest reward value is selected as a negative sample. The reward value includes an immediate reward calculated based on candidate responses and a long-term reward calculated based on the dialogue chain. By using contrastive learning and reinforcement learning methods, the parameters of the basic dialogue model are updated using high-quality samples and negative samples to obtain a well-trained basic dialogue model as a customer service model; the customer service model generates responses based on user input.

2. The generative dialogue optimization method based on multi-step reinforcement simulation according to claim 1, characterized in that, The basic dialogue model uses a finely tuned Qwen2.5-7b-instruct, and the user simulator uses DeepSeek-V3.

3. The generative dialogue optimization method based on multi-step reinforcement simulation according to claim 1, characterized in that, For each candidate response, the process of generating a dialogue chain using a user simulator includes the following steps: The initial step is to use the candidate responses as the current response; The user response generation step involves the user simulator generating a user response based on the current reply. In the response generation step, the basic dialogue model generates a new response based on the user's response. Repeat the steps, using the new response as the current response, to generate user responses and generate responses until the required number of user responses are obtained; In the process of generating user responses in the user simulator, the user's focus is predicted by combining the content of the current reply and the preset application scenario, and a user response is generated based on the focus.

4. The generative dialogue optimization method based on multi-step reinforcement simulation according to claim 1, characterized in that, The instant reward calculated based on candidate responses includes: Determine whether the candidate response meets the current stage goal of the dialogue, and give the action intent matching degree based on the judgment result; Determine whether the candidate responses express medical knowledge in a professional field, and assign a professional level rating based on the determination result.

5. The generative dialogue optimization method based on multi-step reinforcement simulation according to claim 1, characterized in that, The long-term reward based on the dialogue chain computation is obtained by evaluating the following aspects of the dialogue chain, including: Analyze the emotional changes among multiple user responses and assess trust levels based on these emotional changes; The goal achievement rate is assessed based on whether the preset user goals are met among multiple user responses. Predict users' willingness to continue participating in the next three rounds of dialogue to assess the sustainability of the conversation.

6. The generative dialogue optimization method based on multi-step reinforcement simulation according to claim 1, characterized in that, The method of updating the parameters of the base dialogue model using contrastive learning and reinforcement learning, and utilizing high-quality and negative samples, includes the following steps: The Direct Preference Optimization algorithm is used to update the parameters of the basic dialogue model by comparing the differences between high-quality samples and negative samples. By using a user simulator as the adversary, the cumulative reward of the basic dialogue model in simulated dialogue is maximized.

7. The generative dialogue optimization method based on multi-step reinforcement simulation according to claim 1, characterized in that, The Direct Preference Optimization algorithm is employed, which updates the parameters of the basic dialogue model by comparing the differences between high-quality and negative samples. The loss function used is as follows: in, This represents the mathematical expectation operator, where x represents the input context, and y represents the mathematical expectation operator. w Indicates a high-quality response sample, y l Represents a negative sample, π θ Let π represent the target model to be optimized. ref denoted as the reference model, σ represents the Sigmoid function, and β represents the temperature coefficient.

8. The generative dialogue optimization method based on multi-step reinforcement simulation according to claim 1, characterized in that, The loss function used to maximize the cumulative reward of the base dialogue model in simulated dialogue, with the user simulator as the adversary, is expressed as follows: in, The expression represents the mathematical expectation operator, k represents the dialogue turn, and N represents the number of turns to explore backward.

9. The generative dialogue optimization method based on multi-step reinforcement simulation according to claim 1, characterized in that, The method described above, which combines contrastive learning and reinforcement learning, updates the parameters of the base dialogue model using high-quality and negative samples. The overall loss function is expressed as: L(total)=λ DPO L DPO +λ ADV L ADV ; Where, λ DPO express The weighting coefficient, λ ADV express The weighting coefficients.

10. A generative dialogue optimization system based on multi-step reinforcement simulation, characterized in that, include: The sample generation module uses a basic dialogue model to generate several candidate responses based on the current dialogue history. Use a user simulator to generate a dialogue chain for each candidate response; The sample partitioning module calculates a reward for each dialogue chain, selects the K dialogue chains with the highest reward values ​​as high-quality samples, and uses the dialogue chain with the lowest reward value as a negative sample; the reward value includes an immediate reward calculated based on candidate responses and a long-term reward calculated based on the dialogue chain. The dynamic optimization module uses contrastive learning and reinforcement learning methods to update the parameters of the basic dialogue model with high-quality and negative samples, resulting in a well-trained basic dialogue model that serves as the customer service model. The customer service model generates responses based on user input.

Images