A target guiding dialogue generation method based on joint strategy driving

Abstract

The application discloses a target guiding dialogue generation method based on a joint strategy driving, and specifically guides a system dialogue model to output corpus through a direction strategy network, a guiding strategy network and a reinforcement learning method; the application can guarantee the directionality of the dialogue, can pay attention to fine-grained guidance in a specific context environment, and improves the target directivity and achievement efficiency of the dialogue system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of active dialogue, and more particularly to a target guiding dialogue generation method based on joint strategy driving. BACKGROUND

[0002] Currently, in the active dialogue task, in order to simulate the behavior of human experts, the traditional method usually adopts corpus-based fine-tuning to predict the dialogue strategy. However, these methods rely on a large amount of labeled data and are difficult to generalize to new situations. Recently, some researches prompt large language models to perform self-thinking (Self-Thinking) of the next round of strategy, but ignore the long-term dialogue goal. Fu et al. use self-play (Self-Play) to simulate iterative optimization of strategy planning, but this method is only suitable for individual cases. The PPDPP method introduces a learnable language model plugin that can improve the strategy planning ability by fine-tuning the plugin and assist the large language model to achieve the dialogue goal, but it lacks effective guidance of the fine-grained expected behavior.

[0003] Therefore, how to provide a dialogue generation method which can not only guarantee the directionality of the dialogue, but also pay attention to the fine-grained guidance in the specific context environment, and improve the goal-oriented nature and efficiency of the dialogue system is a problem that those skilled in the art need to solve. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a target guiding dialogue generation method based on joint strategy driving.

[0005] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] A target guiding dialogue generation method based on joint strategy driving, comprising the following steps:

[0007] S1: inputting a historical dialogue s t to a direction strategy network to obtain a direction strategy v t ; wherein the historical dialogue s t = {(o1, u1), (o2, u2),..., (o t-1 , u t-1 ); t >= 2; o t-1 represents the output corpus of the system dialogue model LLM sys in the t-1 round of dialogue; u t-1 represents the output corpus of the user role model LLM user in the t-1 round of dialogue; wherein the direction strategy network represents a supervised fine-tuned direction strategy base model;

[0008] S2: input the historical dialogue s t and the direction strategy v t to a guidance strategy network to obtain a guidance keyword w t ; wherein the guidance strategy network represents a guidance strategy base model after supervised fine-tuning;

[0009] S3: input the historical dialogue s t , the direction strategy v t and the guidance keyword w t to a system dialogue model LLM sys to obtain an output corpus o sys of the system dialogue model LLM t in the tth round of dialogue;

[0010] S4: input the historical dialogue s t and the output corpus o t to a user role model LLM user to obtain an output corpus u user of the user role model LLM t in the tth round of dialogue;

[0011] S5: update the network parameters of the direction strategy network and the guidance strategy network using a reinforcement learning algorithm to obtain a direction strategy network and a guidance strategy network

[0012] S6: continuously repeat the above steps until a preset termination condition is met.

[0013] Preferably, the direction strategy network is obtained based on the following steps:

[0014] obtain a direction strategy dialogue data set D', wherein D' = {(s i ,v i )}; i = 1, 2,..., N'; (s i ,v i ) represents the ith direction strategy data pair in the direction strategy dialogue data set D'; s i represents the historical dialogue included in the ith direction strategy data pair; v i represents the direction strategy v i corresponding to the historical dialogue s i ; and N' represents the number of direction strategy data pairs included in the direction strategy dialogue data set D';

[0015] training the direction policy base model on the direction policy dialogue data set D' and adjusting network parameters of the direction policy base model by maximizing a first log-likelihood function to obtain the direction policy network

[0016] Preferably, the expression of the first log-likelihood function is:

[0017]

[0018] wherein p P (v i |s i ) represents inputting the historical dialogue s i to the direction policy base model, and the direction policy base model outputs a probability of the direction policy v i ; log represents taking a logarithm;

[0019] represents taking an average value of N' logp P (v i |s i ).

[0020] Preferably, the guide policy network is obtained based on the following steps:

[0021] obtaining a guide policy dialogue data set D"; wherein, represents the jth guide policy data pair in the guide policy dialogue data set D"; s j represents a historical dialogue included in the jth guide policy data pair; v j represents a corresponding direction policy of the historical dialogue s j . represents a guide keyword corresponding to the historical dialogue s j and the direction policy v j ; and N" represents a number of guide policy data pairs included in the guide policy dialogue data set D";

[0022] training the guide policy base model on the guide policy dialogue data set D" and adjusting network parameters of the guide policy base model by maximizing a second log-likelihood function to obtain the guide policy network

[0023] Preferably, the guide keyword is obtained based on the following steps:

[0024] obtaining a user response corpus in the last round dialogue in the historical dialogue s j .

[0025] extracting keywords from the user response corpus by using a vlt5-base-keywords model to obtain the guide keywords

[0026] Preferably, the expression of the second log-likelihood function is:

[0027]

[0028] wherein, represents inputting the historical dialogue s j and the direction strategy v j into the guide strategy base model, and the guide strategy base model outputs the probability of the guide keywords ; log represents taking the logarithm; E represents taking the average value of N" number of ; p(k m |k1,...,k m-1 ; s j , v j ) represents the probability of the guide strategy base model generating the mth token under the condition of knowing the historical dialogue s j , the direction strategy v j , and the first m-1 tokens k1,...,k m-1 , and k m represent tokens, which together constitute the guide keywords n represents the number of tokens included in the guide keywords .

[0029] Preferably, S5 specifically comprises the following steps:

[0030] S51: inputting the output corpus o t and the output corpus u t into an evaluation reward model LLM critic to obtain the reward value r t of the tth round of dialogue;

[0031] S52: maximizing the reward to update the network parameters of the direction strategy network and the guide strategy network , and obtain the direction strategy network and the guide strategy network wherein, represents taking the divergence between the guide strategy network and the guide strategy network ; and β represents a penalty coefficient.

[0032] Preferably, the following steps are repeatedly performed M times: inputting the output corpus o t and the output corpus u t to an evaluation reward model LLM critic to obtain M reward values;

[0033] calculating the average of the M reward values to obtain the reward value r t of the tth round of dialogue.

[0034] Preferably, the direction strategy base model is RoBERTa, and the guide strategy base model is GPT-2.

[0035] Preferably, the user role model LLM user , the system dialogue model LLM sys and the evaluation reward model LLM critic are simulated by ChatGPT.

[0036] According to the technical solution described above, compared with the prior art, the application provides a target guide dialogue generation method based on joint strategy driving, which can not only ensure the directionality of the dialogue, but also pay attention to the fine-grained guidance in a specific context environment, thereby improving the target orientation and efficiency of the dialogue system. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0038] Figure 1 A flowchart of a target guide dialogue generation method based on joint strategy driving provided by the application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some embodiments of the application, not all embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0040] As shown in Figure 1 , the embodiments of the application disclose a target guide dialogue generation method based on joint strategy driving, comprising the following steps:

[0041] S1: Historical Dialogues t Input to the direction policy network Obtain direction strategy v t ; wherein, the historical dialogues s t ={(o1,u1),(o2,u2),...,(o t-1 ,u t-1 );t≥2;o t-1 This represents the system dialogue model LLM in the (t-1)th round of dialogue. sys The output corpus; u t-1 This represents the user role model (LLM) in the (t-1)th round of dialogue. user The output corpus; where the direction policy network This represents the basic model of the direction strategy after supervision and fine-tuning;

[0042] What is understandable is: Directional Policy Network This represents the directional policy network obtained in the previous round of S5;

[0043] In one embodiment, the basic model of the direction policy adopts the Large version of RoBERTa;

[0044] This invention uses ChatGPT (gpt-3.5-turbo) to simulate the user role model LLM. user The system dialogue model LLM sys These two characters.

[0045] In one embodiment, the direction policy network Obtained based on the following steps:

[0046] Obtain the directional policy dialogue dataset D', where D'={(s i ,v i )};i=1.2..N';(s i ,v i ) represents the i-th direction policy data pair in the direction policy dialogue dataset D'; s i This indicates that the i-th direction strategy data pair includes historical dialogues; v i Indicates historical dialogues i Corresponding directional strategy v i N' represents the number of direction policy data pairs included in the direction policy dialogue dataset D';

[0047] The direction policy base model is trained using the direction policy dialogue dataset D', and the network parameters of the direction policy base model are adjusted by maximizing the first log-likelihood function to obtain the direction policy network.

[0048] It can be understood that the direction strategy dialogue dataset D' is different for different target tasks. If the dialogue task is emotional support, the ESConv dataset (for reference, Liu S, Zheng C, Demasi O, et al. Towards Emotional Support Dialog

[0049] Systems[C] / / Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021: 3469-3483.

[0050] ) training data and validation data (the present application divides the ESConv dataset into training data, validation data and test data, the ratio is 6:2:2) as the direction strategy dialogue dataset D'; if the dialogue task is a buying and selling dialogue negotiation, the improved CraiglistBargain dataset (for reference, Yang R, Chen J, Narasimhan K. Improving Dialog Systems for Negotiation with Personality Modeling[C] / / Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021: 681-693) can be used.

[0051] training data and validation data (the present application divides the improved CraiglistBargain dataset into training data, validation data and test data, the ratio is 6:2:2) as the direction strategy dialogue dataset D'.

[0052] In an embodiment, the expression of the first log-likelihood function is:

[0053]

[0054] wherein, p P (v i |s i ) represents inputting the historical dialogue s i to the direction strategy base model, and the direction strategy base model outputs the probability of the direction strategy v i ; log represents taking the logarithm;

[0055] represents taking the average value of N' logp P (v i |s i ).

[0056] S2: inputting the historical dialogue s t and the direction strategy v t to the guide strategy network to obtain the guide keyword w t ; wherein, the guide strategy network represents the guide strategy base model after supervised fine-tuning;

[0057] It can be understood that: the guide strategy network represents the guide strategy network obtained in the last round S5;

[0058] In an embodiment, the guide strategy base model adopts GPT-2;

[0059] In an embodiment, the guide strategy network is obtained based on the following steps:

[0060] obtain a guide strategy dialogue data set D”; wherein, represents the jth guide strategy data pair in the guide strategy dialogue data set D”; s j represents the historical dialogue included in the jth guide strategy data pair; v j represents the corresponding direction strategy of the historical dialogue s j ; represents the guide keyword corresponding to the historical dialogue s j and the direction strategy vj; N” represents the number of guide strategy data pairs included in the guide strategy dialogue data set D”;

[0061] train the guide strategy base model using the guide strategy dialogue data set D”, and maximize the second log-likelihood function to adjust the network parameters of the guide strategy base model, to obtain the guide strategy network

[0062] In an embodiment, the guide keyword Based on the following steps:

[0063] Obtain the user response corpus in the last round of dialogue in the historical dialogue s j ;

[0064] Extract keywords from the user response corpus using the vlt5-base-keywords model to obtain the guide keywords w

[0065] It can be understood that the guide strategy dialogue dataset D" is different for different target tasks. If the dialogue task is emotional support, the guide strategy dialogue dataset D" can be constructed based on the data in the ESConv dataset for keyword extraction. If the dialogue task is a buying and selling dialogue negotiation, the guide strategy dialogue dataset D' can be constructed based on the data in the improved CraiglistBargain dataset for keyword extraction.

[0066] In an embodiment, the expression of the second log-likelihood function is:

[0067]

[0068] wherein, represents inputting the historical dialogue s j and the direction strategy v j into the guide strategy base model, and the guide strategy base model outputs the probability of the guide keywords w ; log represents taking the logarithm; E represents taking the average of N" times ; p(k m |k1,...,k m-1 ; s j , v j ) represents the probability of the guide strategy base model generating the mth token under the condition of knowing the historical dialogue s j , the direction strategy v j , and the first m-1 tokens k1,...,k m-1 ; k m represents the token, which together constitute the guide keywords w n represents the number of tokens included in the guide keywords w .

[0069] S3: input the historical dialogue s t , the direction strategy v t , and the guide keywords w t into the system dialogue model LLM sys to obtain the tth round of dialogue in the system dialogue model LLMsys output corpus o t ;

[0070] S4: inputting the historical dialogue s t and the output corpus o t to the user role model LLM user to obtain an output corpus u user of the user role model LLM t in the tth round of dialogue;

[0071] S5: updating network parameters of the direction policy network and the guidance policy network by using a reinforcement learning algorithm to obtain the direction policy network and the guidance policy network

[0072] In an embodiment, S5 specifically comprises the following steps:

[0073] S51: inputting the output corpus o t and the output corpus u t to an evaluation reward model LLM critic to obtain a reward value r t of the tth round of dialogue;

[0074] In an embodiment, the evaluation reward model LLM critic is simulated by using ChatGPT.

[0075] Specifically, the evaluation reward model LLM critic can be prompted to generate evaluation feedback results for specific goals by asking it a selection question. The specific evaluation feedback results include three types: progress (score of 1), maintain the status quo (score of 0), and regression (score of -1). Further, the evaluation feedback results are mapped to a scalar by using a mapping function fmap(·) as the final reward value.

[0076] S52: maximizing the reward to update network parameters of the direction policy network and the guidance policy network to obtain the direction policy network and the guidance policy network wherein represents the divergence between the guidance policy network and the guidance policy network ; and β represents a penalty coefficient.

[0077] In an embodiment, the following steps are repeatedly performed M times: inputting the output corpus ot and the output corpus u t input to the evaluation reward model LLM critic , obtain M reward values;

[0078] calculate the average of the M reward values, and obtain the reward value r of the tth round of dialogue t .

[0079] S6: repeatedly repeat the above steps until the preset termination condition is met.

[0080] It can be understood that: the preset termination condition is to complete the dialogue goal or reach the preset maximum number of rounds of dialogue.

[0081] In order to verify the effectiveness of the present application, the present application (named JSD) and six baseline models DiagoGPT, Vanilla, Proactive, ProCoT, AnE, ICL-AIF and PPDPP are compared in performance in two tasks of emotional support dialogue and buying and selling negotiation dialogue.

[0082] Task 1: Emotional support dialogue task

[0083] The task aims to reduce the emotional distress of individuals and help people understand and solve the challenges they face. The task uses the ESConv dataset, which involves 1300 dialogues under 10 topic questions, and contains 8 support strategies (corresponding to the direction strategy of the present application) during the dialogue process. Each dialogue is labeled with problem category, emotion category, and basic situation of the seeker, etc. The present task divides ESConv into training set / verification set / test set in the ratio of 6:2:2. In the process of emotional support dialogue, if the average value of the M reward values output by the current round of dialogue evaluation reward model LLM critic is greater than 0.5, it is considered that the current dialogue goal is completed.

[0084] Task 2: Buying and selling negotiation dialogue task

[0085] In order to further verify the effectiveness of the model in non-cooperative dialogue (where the user and the system have conflicting goals), the present application uses the improved CraiglistBargain dataset. The dataset is based on negotiation dialogues of real goods, covering the negotiation process between buyers and sellers on the price of goods. Each negotiation sample contains the category of the goods, the description of the goods, the target price of the buyer and the target price of the seller. The present task selects 11 negotiation strategies (corresponding to the direction strategy of the present application) labeled in it, and divides the dataset into training set / verification set / test set in the ratio of 6:2:2.

[0086] All experiments and model training of the present application are completed on an Ubuntu server equipped with 2 Nvidia A800 GPUs, and the main environment used includes Python 3.10, PyTorch 2.4 and CUDA 12.1. In order to ensure the determinacy of the output of the large language model, the temperature is set to 0, the evaluation reward model LLM critic of each dialogue is set to 10. During the reinforcement learning training process, the learning rate of the policy model is set to 2e-5, the upper limit of the dialogue round is set to 7, and a total of 5 epochs are trained.

[0087] Table 1 shows the performance of the JSD of the present application and six baseline models in the emotional support dialogue and the buy-sell negotiation dialogue tasks.

[0088] Table 1 shows the performance of the JSD of the present application and six baseline models in the emotional support dialogue and the buy-sell negotiation dialogue tasks.

[0089]

[0090] In Table 1:

[0091] 1) The bold font represents the best result in each column, and the underlined font represents the second best result,

[0092] 2)

[0093] wherein N represents the total number of dialogues, t i represents the number of rounds used to achieve the goal in the ith dialogue, or the maximum number of rounds reached without achieving the goal; AT measures the efficiency of goal completion by calculating the average number of rounds to achieve the goal, and the smaller the value is, the better.

[0094] 3) SR% = SR@T = S T / N.

[0095] wherein S T represents the number of dialogues that achieve the dialogue goal before or equal to T rounds, and N is the total number of dialogues; SR@T measures the effectiveness of goal completion by calculating the success rate of achieving the goal within the predefined maximum round T, and the larger the value is, the better.

[0096] 4)

[0097] wherein d p represents the deal price, s p represents the seller target price, and b prepresents the target price of the buyer; SL% represents the proportion of the buyer's expected profit in the transaction relative to the two-way price, and if the transaction fails, SL% is set to 0.

[0098] From Table 1, it can be seen that:

[0099] On the ESConv dataset of the emotional support dialogue task, the JSD of the present application performs best with a success rate of 93.85%, which is significantly higher than other models, indicating that it has the highest target achievement rate. Compared with the second best performing model PDPDP, the JSD of the present application improves by 9.23 percentage points, most effectively guiding the dialogue to achieve the expected target. In terms of the average number of rounds to achieve the dialogue target, the JSD of the present application achieves the predetermined target with an average of 3.35 rounds, which is 1.21 rounds less than PDPDP, indicating that the JSD of the present application is the most efficient in achieving the target.

[0100] On the CraigslistBargain dataset of the buy-sell dialogue negotiation task, the JSD of the present application also performs well, with a success rate of 82.45%, which is significantly higher than other models, indicating that the JSD of the present application has the highest target achievement rate in the dialogue negotiation task and successfully guides the dialogue to achieve the expected target. Compared with the other best model PDPDP, the JSD of the present application improves by 21.28 percentage points. In terms of the average number of rounds, the JSD of the present application achieves the target with an average of 4.00 rounds, which is 1.62 rounds less than PDPDP, also indicating that it is more efficient in achieving the target in multiple tasks. Although the JSD of the present application is not the highest in the SL% index (38.88%), it still performs outstandingly in overall performance.

[0101] In addition, the present application also conducts an ablation test on the JSD provided by the present application on ESConv and CraigslistBargain, and the analysis results are shown in Table 2.

[0102] Table 2 Ablation test analysis results of the present application

[0103]

[0104] From Table 2, on the ESConv dataset, when removing the direction strategy module (w / o SP), the performance of the variant model decreased from 93.85% to 90.58% in target achievement rate, and increased to 4.23 in average turns per dialogue, which shows that the direction strategy module has a significant contribution to improving the target achievement rate and dialogue efficiency. When removing the guidance strategy module (w / o SG), the performance of the variant model decreased to 84.62% in target achievement rate, and increased to 4.56 in average turns per dialogue, which shows that the guidance strategy plays a key role in the fine-grained control of dialogue content. When removing the reinforcement learning module (w / o RL), the performance of the variant model decreased to 88.46% in target achievement rate, and increased to 4.35 in average turns per dialogue, which shows that reinforcement learning plays an important role in optimizing the overall strategy.

[0105] On the CraigslistBargain dataset, when removing the direction strategy module, the target achievement rate of the variant model decreased from 82.45% to 58.51%, and the average turns per dialogue increased to 5.17, which shows the importance of the direction strategy in the negotiation of buying and selling dialogue. When removing the guidance strategy module, the target achievement rate of the variant model decreased to 79.79%, and the average turns per dialogue increased slightly to 4.05, which shows that the guidance strategy also plays an important role in controlling the direction of the dialogue. When removing the reinforcement learning module, the target achievement rate of the variant model increased to 86.17%, the average turns per dialogue decreased to 3.61, and the SL% reached 42.79%, which shows that without reinforcement learning, the model may tend to adopt a more conservative and simplified strategy, avoiding complex negotiation processes and directly achieving the transaction goal.

[0106] Finally, the present application demonstrates the effectiveness of the JSD in guiding target dialogue from the aspects of direction strategy and guidance strategy cases.

[0107] 1) Direction strategy case analysis

[0108] To better analyze the ability of JSD in dialogue guiding tasks, the present application shows a complete emotional support dialogue case. As shown in Table 3, JSD provides decision support for system assistants (i.e., the system dialogue model described in the present application) in terms of strategy direction and guiding clues based on the dual-process theory through joint strategy prompting, forming a dynamically planned path solution. This solution gradually guides the dialogue to the intended goal and drives the model to actively move towards the target direction. In the specific case, the user (a patient with psychological problems) has negative emotions such as anxiety due to "personal health problems", and the system assistant (also known as a therapist) provides emotional treatment for the patient with psychological problems under the guidance of joint strategy. Among them, the yellow highlight represents the direction strategy of the dialogue and the corresponding generated content, and the red highlight represents the specific guiding strategy and the corresponding generated content. In the case shown, the model gradually leads the user from an anxious state to a more positive emotional state through three dialogue turns with the assistance of joint strategy. In each dialogue turn (Turn), the system selects different strategies based on the current state of the patient. In the first round, the therapist chooses the "affirmation and reassurance" strategy, affirms the patient's feelings, and emphasizes the importance of health and happiness, and the system guides the patient to focus on their own health priorities, gradually calming negative emotions. In the second round, the therapist adopts the "reflection of feelings" strategy, and the therapist expresses understanding in response to the patient's frustration, and proposes to work together to find a solution, helping the patient see hope from the emotional dilemma. In the last round of dialogue, the therapist adopts the "providing suggestions" strategy, and promises to support the patient in every step of action, further guiding the patient to find more possibilities and hope when facing challenges.

[0109] Table 3 Case analysis of direction strategy guiding dialogue

[0110]

[0111] 2) Guiding strategy case analysis

[0112] To better analyze the capabilities of the guidance policy network, the present application presents a series of typical cases. As shown in Table 4, the present application JSD can generate reasonable guiding keywords for each round of conversation through joint strategy prompts. The model will generate specific conversation details according to these keywords to guide the progress of the conversation. Under the guidance of the pre-planned conversation path, the system assistant (i.e., the system conversation model described in the present application) can better plan the details of the conversation and actively promote the conversation according to the target. In the three specific cases, different users have negative emotions such as anxiety due to "remote work", "unemployment", and "emotional companionship", and the guidance strategy is responsible for optimizing the specific details of the conversation, so that the system assistant can more efficiently alleviate the patient's distress. Among them, the yellow highlight represents the direction strategy of the conversation and the corresponding generated content, and the red highlight represents the specific guidance strategy and the corresponding generated content. In the cases shown, JSD generates content corresponding to the user's conversation in the current round through different guiding keywords. In the first conversation case, the patient expresses negative feelings about remote work and believes that "remote work is not conducive". The therapist uses the "self-disclosure" strategy and uses "remote work" and "functional work" as guiding keywords. This strategy allows the therapist to express understanding of the patient's feelings, such as "feeling disconnected from work can affect your work performance". This guidance strategy helps the patient focus on how to maintain work performance, especially in a remote work environment, providing constructive suggestions. In the second conversation case, the patient experiences continuous anxiety and frustration due to unemployment. The therapist uses the "affirmation and reassurance" strategy and uses "job opportunities" and "despair" as guiding keywords. The therapist guides the patient to recognize the root cause of the emotion, such as "you feel hopeless and overwhelmed due to the lack of job opportunities". This strategy helps the patient express their emotions and seek support and advice through empathetic guidance. The third conversation case explores the loneliness and stress the patient feels after losing a partner. The patient faces psychological difficulties due to the lack of emotional companionship. The therapist again uses the affirmation and reassurance strategy and uses "difficult" and "lonely" as guiding keywords. By guiding the patient to recognize that it is normal to feel lost and overwhelmed, the therapist helps the patient realize that their emotions are understandable and reminds the patient that they do not have to face difficulties alone. This strategy reduces the patient's sense of loneliness and provides emotional comfort through emotional support.

[0113] Table 4 Case analysis of guiding strategy guiding conversation

[0114]

[0115] The various embodiments described in this specification are implemented in a progressive manner, each embodiment focusing on the differences from other embodiments, and the same or similar parts between embodiments can be mutually referred to. For the apparatus disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0116] The above description of disclosed embodiments enables one of ordinary skill in the art to make or use the application. Various modifications to these embodiments will be readily apparent to those of ordinary skill in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a goal-oriented dialogue based on a joint strategy driving, characterized in that, The method comprises the following steps: S1: inputting a historical dialogue to a direction strategy network , to obtain a direction strategy ; wherein the historical dialogue ; represents the output corpus of a system dialogue model t in the first -1 round dialogue; represents the output corpus of a user role model t in the first -1 round dialogue; wherein the direction strategy network represents a direction strategy base model after supervised fine-tuning; S2: inputting the historical dialogue and the direction strategy into a guidance strategy network to obtain a guidance keyword ; wherein the guidance strategy network represents a supervised fine-tuned guidance strategy base model; S3: inputting the historical dialogue , the direction strategy , and the guide keyword to the system dialogue model to obtain the output corpus of the system dialogue model in the tth round of dialogue ; S4: inputting the historical dialogue and the output corpus to the user role model , obtaining the output corpus t of the user role model in the first round dialogue S5: updating the direction policy network using a reinforcement learning algorithm and network parameters of the guidance policy network to obtain a direction policy network and a guidance policy network ; S6: continuously repeat the above steps until a preset termination condition is met; The direction policy network Obtained on the basis of the following steps: A direction policy dialogue dataset is obtained , wherein ; represents an i-th direction policy data pair in the direction policy dialogue dataset ; represents a history dialogue included in the i-th direction policy data pair represents a history dialogue corresponding direction policy ; represents a number of direction policy data pairs included in the direction policy dialogue dataset ; The direction strategy dialogue data set is obtained by using the direction strategy dialogue data set generation method The direction strategy base model is trained, and a first log-likelihood function is maximized to adjust network parameters of the direction strategy base model, to obtain the direction strategy network ; The guiding policy network Obtained on the basis of the following steps: Acquiring a guiding policy dialogue dataset ; wherein, ; denotes the jth guiding policy data pair in the guiding policy dialogue dataset ; denotes a history dialogue included in the jth guiding policy data pair; denotes a history dialogue corresponding direction policy; denotes a history dialogue and a direction policy corresponding guiding keyword; denotes the number of guiding policy data pairs included in the guiding policy dialogue dataset ; Utilize the guiding strategy dialogue data set Train the guiding strategy base model, and maximize the second log-likelihood function to adjust the network parameters of the guiding strategy base model, to obtain the guiding strategy network ; The direction strategy base model adopts a Large version of RoBERTa; The guidance strategy base model adopts GPT-2.

2. The target-guided dialogue generation method based on joint strategy driving according to claim 1, characterized in that: The expression of the first log-likelihood function is: ； wherein, represents inputting the historical dialogue to the direction policy base model, and the direction policy base model outputs a probability of a direction policy ; log represents taking a logarithm; represents taking an average value of .​ 3. The target-guided dialogue generation method based on joint strategy driving according to claim 1, characterized in that: The guide keyword is obtained based on the following steps: acquiring a corpus of user responses in the last turn of the historical conversation ​ extracting keywords from the user response corpus using a vlt5-base-keywords model to obtain the guide keywords .

4. The target-guided dialogue generation method based on joint strategy driving according to claim 1, characterized in that: The expression of the second log-likelihood function is: ； ； wherein, denotes that the historical dialogue and the direction strategy are input into the guide strategy base model, and the guide strategy base model outputs the probability of the guide keyword ; log denotes taking the logarithm; E denotes taking the average value of ; ; denotes the probability of the guide strategy base model generating the mth token under the condition that the historical dialogue , the direction strategy and the previous m -1 tokens are known, denotes the token, which together constitute the guide keyword ; n denotes the number of tokens included in the guide keyword .

5. The method of claim 3, wherein the method further comprises: S5 specifically comprises the following steps: S51: The output corpus and the output corpus Input into the evaluation reward model , obtained the t Rewards for round-based dialogue ; S52: maximize the reward with the updated network parameters of the direction policy network and the guidance policy network , to obtain the direction policy network and the guidance policy network ; wherein, denotes the divergence between the guidance policy network and the guidance policy network ; and β denotes a penalty coefficient.

6. The method of claim 5, wherein the method further comprises: repeating the following steps M times: inputting the output corpus and the output corpus to the evaluation reward model to obtain M reward values; calculating an average of the M reward values to obtain a reward value of the tth round of dialogue .

7. The method of claim 5, wherein the method further comprises: The user role model The system dialogue model And the evaluation reward model are simulated by ChatGPT.

Images