Dialogue model training method, dialogue method, dialogue system, dialogue equipment and medium

Abstract

The invention provides a dialogue model training method, a dialogue method, a dialogue system, equipment and a medium, and relates to the technical field of man-machine interaction, the method comprises the following steps: generating current sample dialogue state information according to current sample user input information and historical sample dialogue information in a sample dialogue; performing tree search based on the to-be-trained strategy network and the to-be-trained value network to obtain current sample reply action information and a global reward value of a dialogue track corresponding to the current sample reply action information; according to the next sample dialogue state information, the current sample dialogue state information, the current sample reply action information and the global reward value which are generated by simulating the current sample reply action information, constructing sample data; and performing joint training on the to-be-trained strategy network and the to-be-trained value network according to the sample data to construct a dialogue model. According to the method, long-term return and strategy optimization are comprehensively considered during dialogues, so that the generation quality, the interaction capability and the user experience of multiple rounds of dialogues are improved.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a dialogue model training method, a dialogue method, a system, a device and a medium. Background Art

[0002] With the continuous development of artificial intelligence technology, natural language processing technology has gradually matured, and more and more industries have begun to use dialogue systems to replace existing traditional manual question-and-answer business scenarios.

[0003] Existing dialogue systems usually use adaptive technologies such as context modeling or reinforcement learning to improve the multi-round dialogue capabilities of large-scale pre-trained language models. However, such technologies can easily lead to the dialogue system's lack of long-term planning capabilities and easy fall into invalid repetitive or logically erroneous response patterns, thus affecting the multi-round dialogue effect and user experience. Summary of the invention

[0004] The present invention provides a dialogue model training method, a dialogue method, a system, a device and a medium, which are used to solve the defects in the prior art that the dialogue system lacks long-term planning ability and is prone to falling into invalid repetition or logically erroneous response modes, resulting in poor dialogue effects and affecting user experience, so as to improve the multi-round dialogue effects and user experience.

[0005] The present invention provides a dialogue model training method, comprising: Generate current sample dialogue state information based on current sample user input information and historical sample dialogue information in the sample dialogue; Performing a tree search based on the to-be-trained policy network and the to-be-trained value network to obtain current sample reply action information corresponding to the current sample dialogue state information, and a global reward value of the dialogue trajectory corresponding to the current sample reply action information; Simulating and generating next sample dialogue state information according to the current sample reply action information, and constructing sample data according to the current sample dialogue state information, the current sample reply action information, the global reward value, and the next sample dialogue state information; The strategy network to be trained and the value network to be trained are jointly trained according to the sample data, and a dialogue model is constructed according to the training results.

[0006] According to a dialogue model training method provided by the present invention, the current sample reply action information is searched based on the following steps: Based on the to-be-trained policy network, applying the current sample dialogue state information, generating reply information for the current sample user input information, and obtaining a target reply set; Based on the strategy network to be trained, the predicted reply information corresponding to the current sample user input information is evaluated as belonging to the action probability of each candidate reply information in the target reply set, and based on the value network to be trained, the value score corresponding to each candidate reply information is evaluated; Taking the current sample user input information as the root node, performing node selection, expansion, simulation and backtracking according to the action probability and the value score, to obtain a plurality of dialogue tracks corresponding to the current sample dialogue state information, and a value score of each reply action information corresponding to the current sample dialogue state information in the plurality of dialogue tracks; According to the value score, the current sample reply action information is searched in the multiple dialogue tracks.

[0007] According to a dialog model training method provided by the present invention, based on the to-be-trained policy network, applying the current sample dialog state information, generating reply information for the current sample user input information, and obtaining a target reply set, comprises: Inputting the current sample dialogue state information into the to-be-trained policy network to obtain a plurality of initial reply information corresponding to the current sample user input information; Pruning the multiple initial reply information to obtain the target reply set.

[0008] According to a dialogue model training method provided by the present invention, the global reward value of the dialogue trajectory corresponding to the current sample reply action information is calculated based on the following steps: According to the current sample dialogue state information and the reply information corresponding to the current sample reply action information, the dialogue quality of the current sample reply action information in the current round of the sample dialogue is evaluated to obtain a process reward value of the current sample reply action information; According to the user input information and reply information in the dialogue track corresponding to the current sample reply action information, and the dialogue target of the sample dialogue, the response degree of the current sample reply action information to the dialogue target is evaluated to obtain the result reward value of the current sample reply action information; The global reward value is determined according to the process reward value and the result reward value.

[0009] According to a dialogue model training method provided by the present invention, the next sample dialogue state information is generated by simulation based on the following steps: Determining a target language model according to the to-be-trained strategy network or a plurality of pre-trained language models; Generate user input prompt information according to the current sample dialogue state information; Inputting the current sample reply action information and the user input prompt information into the target language model to obtain the next sample user input information corresponding to the current sample reply action information; The next sample dialogue state information is generated according to the next sample user input information, the historical sample dialogue information, and the response information corresponding to the current sample user input information and the current sample response action information.

[0010] According to a dialog model training method provided by the present invention, determining a target language model according to the strategy network to be trained or a plurality of pre-trained language models includes: The strategy network to be trained is used as the target language model, or a pre-trained language model is randomly selected from a plurality of pre-trained language models as the target language model.

[0011] According to a dialog model training method provided by the present invention, determining a target language model according to the strategy network to be trained or a plurality of pre-trained language models includes: Taking the historical reply information corresponding to each historical sample user input information in the historical sample dialogue information as a sample, and taking the next historical sample user input information corresponding to each historical sample user input information in the historical sample dialogue information as a label, supervised training is performed on any one of the multiple pre-trained language models to obtain the target language model.

[0012] The present invention also provides a dialogue method, comprising: Generate current dialogue state information based on current user input information and historical dialogue information in the target dialogue; Inputting the current dialogue state information into the dialogue model to obtain reply information corresponding to the current user input information; Wherein, the dialogue model is constructed based on any of the dialogue model training methods described above.

[0013] The present invention also provides a dialogue model training system, comprising: A first state generating unit, configured to generate current sample dialogue state information according to current sample user input information and historical sample dialogue information in the sample dialogue; A search unit, configured to perform a tree search based on the to-be-trained policy network and the to-be-trained value network to obtain current sample reply action information corresponding to the current sample dialogue state information and a global reward value of a dialogue trajectory corresponding to the current sample reply action information; A sample construction unit, configured to simulate and generate next sample dialogue state information according to the current sample reply action information, and construct sample data according to the current sample dialogue state information, the current sample reply action information, the global reward value, and the next sample dialogue state information; The model building unit is used to jointly train the strategy network to be trained and the value network to be trained according to the sample data, and to build a dialogue model according to the training results.

[0014] The present invention also provides a dialogue system, comprising: A second state generating unit, configured to generate current dialog state information according to current user input information and historical dialog information in the target dialog; A dialogue unit, used to input the current dialogue state information into the dialogue model to obtain reply information corresponding to the current user input information; Wherein, the dialogue model is constructed based on any of the dialogue model training methods described above.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements any one of the above-described dialogue model training methods, or implements any one of the above-described dialogue methods.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements any of the above-mentioned dialogue model training methods, or implements any of the above-mentioned dialogue methods.

[0017] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned dialogue model training methods, or implements any of the above-mentioned dialogue methods.

[0018] The dialogue model training method, dialogue method, system, device and medium provided by the present invention generate current sample dialogue state information according to current sample user input information and historical sample dialogue information in the sample dialogue to comprehensively obtain dialogue context information; and perform tree search based on the to-be-trained strategy network and the to-be-trained value network to comprehensively explore the dialogue space and quickly generate high-quality sample data. Then, the to-be-trained strategy network and the to-be-trained value network are updated using reinforcement learning based on the high-quality sample data generated by the tree search, so as to utilize the technical advantages of reinforcement learning and tree search, and gradually train through multi-level modeling and dynamic strategy optimization to obtain a dialogue model with long-term planning ability and strategy optimization ability to conduct multiple rounds of dialogue, so that the dialogue model can consider long-term rewards and strategy optimization when generating replies, and avoid falling into invalid repetition or logically erroneous reply modes. Therefore, through the dialogue model, dialogue can be carried out to ensure that the dialogue system performs well in balancing context consistency, global goal achievement and generation quality, thereby improving the generation quality and interaction ability of multiple rounds of dialogue, as well as user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 This is one of the flow charts of the dialogue model training method provided by the present invention.

[0021] Figure 2 This is the second flow chart of the dialogue model training method provided by the present invention.

[0022] Figure 3 It is a flowchart of the next sample dialogue state information generation step provided by the present invention.

[0023] Figure 4 It is a flowchart of the dialogue method provided by the present invention.

[0024] Figure 5 It is a structural schematic diagram of the dialogue model training system provided by the present invention.

[0025] Figure 6 It is a structural schematic diagram of the dialogue system provided by the present invention.

[0026] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] With the continuous development of artificial intelligence technology, natural language processing technology has gradually matured, and more and more industries have begun to use dialogue systems to replace existing traditional manual question-and-answer business scenarios.

[0029] Large-scale pre-trained language models (hereinafter referred to as large language models or large models) rely on massive corpus data and powerful computing power to generate contextually coherent natural language output and perform well in tasks such as dialogue generation, code generation, and mathematical reasoning. Therefore, existing dialogue systems usually use large models for dialogue generation, but in multi-round dialogue scenarios, existing large models still lack long-term planning capabilities, are prone to invalid repetition or logically incorrect response modes, and are difficult to effectively adjust responses to meet changing user needs, thus affecting the multi-round dialogue effect and user experience.

[0030] In this regard, existing dialogue systems usually use context modeling or reinforcement learning (RL) to improve the multi-round dialogue capabilities of large models.

[0031] Among them, context modeling specifically uses technologies such as sequence pair input, window interception strategy, explicit context summary technology, and implicit memory mechanism to improve the context learning ability of large models. Among them, the sequence pair input method is simple, directly splicing the multi-round dialogue history into continuous text, but it is limited by the input length of the large model. In order to cope with the length limit, a window interception strategy will be introduced to retain the key content of the last few rounds of dialogue, and the most important information will be filtered through the weight mechanism. In addition, explicit context summary technology will be introduced to regularly summarize historical dialogues and extract key information, such as user intent or a summary of solved problems, to reduce redundant input. At the same time, an implicit memory mechanism will be introduced to utilize the attention mechanism to dynamically focus on the content in the historical dialogue that is most relevant to the current problem, without storing all information, further improving the multi-round dialogue capability.

[0032] Although context modeling can enhance the context learning ability of large models and thus help improve the multi-round dialogue ability of large models, in multi-round dialogues, historical information may be contradictory or inconsistent, causing large models to easily fall into invalid repetition or logically incorrect response patterns. For example, when the context contains semantically conflicting content, it is difficult for large models to identify conflicting content, thereby generating logically incorrect responses. In addition, large models are easily affected by misleading content in the context, especially when users ask ambiguous or contradictory questions, which will lead to the accumulation of semantic deviations. Moreover, since context modeling accumulates historical information in each round of dialogue, if there are semantic errors or logical loopholes in the responses generated early, subsequent rounds may continue to generate responses based on these errors, and the early deviations cannot be effectively corrected, resulting in error amplification and decreased coherence, making it easy to fall into invalid repetition or logically incorrect response patterns.

[0033] Among them, reinforcement learning is an optimization method based on reward signals. For example, through reinforcement learning from human feedback (RLHF), a reward model is designed to evaluate the performance of dimensions such as dialogue coherence, relevance, and politeness. In addition, in order to improve the global quality of multi-round dialogues, a multi-objective reward function is usually used to combine the above multiple evaluation dimensions to design a reward model. In addition, reinforcement learning also focuses on the balance between short-term and long-term benefits, that is, in the process of dialogue, it not only ensures the quality of the current response, but also reasonably plans the global dialogue strategy. In the optimization process of reinforcement learning, the large model will adjust its generation strategy based on the reward signal. Specifically, each round of generated dialogue will be scored and evaluated by the reward model, and then the model will optimize the generation parameters to maximize the reward. This method effectively solves the problem of a single objective function in supervised learning, thereby being able to more comprehensively and effectively improve the performance and quality of the dialogue system.

[0034] Although reinforcement learning can improve the quality of multi-round dialogue generation by large models, and thus assist in improving the multi-round dialogue capabilities of large models. However, since the state of each round of dialogue consists of the current context and implicit semantic information, the state space of reinforcement learning grows exponentially as the number of dialogue rounds increases. This high-dimensional state-action space significantly increases the computational complexity of reinforcement learning and makes policy optimization more difficult. In addition, RL usually relies on clear reward signals, which are often only provided at the end of the dialogue or at certain key nodes, such as user feedback or task completion marks, resulting in sparse rewards and difficulty in directly capturing the long-term goals and complex strategies of the dialogue, resulting in the lack of long-term planning capabilities of the dialogue system, which affects the multi-round dialogue effect and user experience.

[0035] In this regard, the present embodiment provides a dialogue model training method to solve the above-mentioned technology. The method models multi-round dialogues as decision trees, so that reinforcement learning and tree search can be combined to train the dialogue model in multi-round dialogues. The trained dialogue model can effectively combine long-term planning capabilities and strategy optimization capabilities to conduct multi-round dialogues, so as to ensure that the dialogue system performs well in balancing context consistency, achieving global goals, and generating quality, thereby improving the multi-round dialogue effect and user experience.

[0036] Figure 1 This is one of the flow charts of the dialogue model training method provided by the present invention. The method can be applied to a variety of dialogue scenarios, including but not limited to task-based dialogue scenarios or multi-round open reasoning dialogue scenarios, etc., which are not specifically limited in this embodiment.

[0037] like Figure 1 As shown, the method includes step 110 , step 120 , step 130 and step 140 .

[0038] Step 110: Generate current sample dialogue state information according to current sample user input information and historical sample dialogue information in the sample dialogue.

[0039] The dialogue here refers to multiple rounds of continuous human-computer dialogue used for model training.

[0040] The current sample user input information in the sample dialogue is the user input information of the current round of dialogue input by the sample user in the interactive interface during the sample dialogue, such as question information, requirement information, prompt information, etc. The so-called input can be command line interface input, graphical interface input, touch input, drop-down selection input, voice input, etc., which is not specifically limited in this embodiment.

[0041] The historical conversation information includes historical sample user input information generated in historical rounds of conversation under the sample conversation process (that is, user input information of historical rounds of conversation input by sample users in the interactive interface) and historical reply information corresponding to the historical sample user input information.

[0042] In the state representation process, after obtaining the current sample user input information and the historical sample conversation information in the sample conversation, the current sample user input information and the historical sample conversation information can be directly spliced ​​to form the current sample conversation state information, such as The corresponding current sample dialogue state information can be expressed as ,in and They are respectively The historical sample user input information generated in the round-by-round dialogue and the reply information corresponding to the historical sample user input information. Alternatively, the context information may be extracted from the historical sample dialogue information to splice the current sample user input information and the context information to form the current sample dialogue state information, which is not specifically limited in this embodiment.

[0043] The context information extraction here may be realized by serializing the historical sample conversation information into text representation, or by embedding the historical sample conversation information using a feature extraction layer of a large model, such as a Transformer encoding layer, etc. This embodiment does not specifically limit this.

[0044] Step 120 , performing a tree search based on the to-be-trained policy network and the to-be-trained value network to obtain the current sample reply action information corresponding to the current sample dialogue state information, and the global reward value of the dialogue trajectory corresponding to the current sample reply action information.

[0045] It should be noted that MCTS (Monte Carlo Tree Search, MCTS) is a decision optimization algorithm that is widely used in games and complex task planning. MCTS explores the optimal decision path by constructing a search tree, and combines sample simulation to evaluate long-term benefits, which can effectively make up for the shortcomings of traditional RL methods in long-term planning. Therefore, the tree search here can be implemented based on Monte Carlo tree search, so as to form a dynamic optimization framework by combining the search ability of MCTS with the learning ability of RL in multi-round dialogue scenarios, so as to better balance short-term and long-term goals in dialogue generation.

[0046] The strategy network to be trained here is used to make corresponding reply action decisions based on the dialogue state information. It is constructed based on the large model to be optimized with dialogue function. The large model to be optimized here is a large pre-trained language model to be optimized, such as a Transformer-based bidirectional encoder representation (Bidirectional Encoder Representations from Transformers, BERT) model, a generative pre-trained Transformer, a text-to-text conversion Transformer, and a human-computer interaction model deployed in various interactive systems for human-computer interaction. Large pre-trained language models, this embodiment does not specifically limit this.

[0047] The value network to be trained here is used to evaluate the global value of the dialogue action to provide guidance for the optimization of the policy network to be trained.

[0048] Optionally, after obtaining the current sample dialogue status information After that, the current sample dialogue state information can be applied based on the strategy network to be trained and the value network to be trained Perform a tree search to explore the current sample conversation state information The corresponding current sample reply action information , and explore the current sample reply action information The global reward value of the corresponding dialogue track. For example, the candidate reply actions, that is, all possible reply information, can be generated by using the policy network to be trained. For each candidate reply action, the policy network to be trained will evaluate its corresponding action probability, and the value network to be trained will evaluate the global reward value of its corresponding dialogue track, so as to judge the value score of the reply action in the overall dialogue through the global reward value. Thus, according to the action probability and value score, after a series of searches and evaluations, the current sample reply action information corresponding to the current sample dialogue state information and the global reward value of the dialogue track corresponding to the reply action information can be obtained.

[0049] The global reward value here includes at least a process reward value for describing the dialogue quality of the current sample response action information in the current round of the sample dialogue, and a result reward value for the degree of response to the dialogue goal to be achieved in the sample dialogue by conducting multiple rounds of dialogue according to the dialogue trajectory corresponding to the current sample response action.

[0050] Step 130, simulate and generate next sample dialogue state information according to the current sample reply action information, and construct sample data according to the current sample dialogue state information, the current sample reply action information, the global reward value, and the next sample dialogue state information.

[0051] Figure 2 This is the second flow chart of the dialogue model training method provided by the present invention.

[0052] like Figure 2 As shown, the next sample dialogue status information is During the generation process, the target language model (also called user model) can be used to apply the current sample to reply action information. Simulate and generate the next sample user input information (i.e., the hypothetical input information of the sample user in the next round of dialogue) based on the current sample dialogue state information , current sample reply action information and the next sample user input information , generate the next sample dialogue state information .

[0053] The target language model here can be a strategy network to be trained, or a language model randomly selected from multiple pre-trained language models, or a language model obtained by supervised training of a pre-trained language model based on historical sample user input information and historical reply information in historical sample dialogue information, etc. This embodiment does not specifically limit this.

[0054] After obtaining the next sample dialogue state information, the current sample dialogue state information, the current sample reply action information, the global reward value and the next sample dialogue state information may be integrated to obtain sample data.

[0055] Step 140: jointly train the strategy network to be trained and the value network to be trained according to the sample data, and construct a dialogue model according to the training results.

[0056] After obtaining the sample data, the sample data can be updated to the experience replay pool. If the number of sample data in the updated experience replay pool reaches a preset number, multiple sample data in the target prediction interval are sampled therefrom, and the policy network to be trained and the value network to be trained are jointly trained based on the sampled sample data to obtain the trained policy network and the trained value network. Based on the trained policy network, a dialogue model is constructed that can accurately output reply information with excellent performance in terms of context consistency, global goal achievement, and generation quality based on the dialogue state information in the dialogue, so that in actual applications, high-quality reply information can be generated for the current user input information in the target dialogue based on the trained dialogue model.

[0057] During the training process, sample data can be input into the policy network to be trained and the value network to be trained, and the output of the policy network to be trained and the output of the value network to be trained are obtained, so as to calculate the loss of the policy network to be trained through the output of the policy network to be trained and the action distribution generated by the tree search. , and calculate the loss of the value network to be trained through the long-term reward z generated by the tree search and the output of the value network to be trained, and then through the joint and To adjust the parameters of the back propagation to treat the model parameters of the training strategy network and the model parameters θ of the value network to be trained to obtain the policy network of the current iteration training and the value network of the current iteration training. If it is determined that the current iteration training reaches the preset convergence condition, such as the policy network of the current iteration training converges, or satisfies a certain number of training rounds, the training is stopped to determine the trained policy network with the best model performance among the policy networks after multiple iteration trainings to construct the dialogue model; if the current iteration training does not reach the preset convergence condition, the policy network of the current iteration training and the value network of the current iteration training are used as the new policy network to be trained and the new value network to be trained, and the next sample dialogue state information is used as the new current sample dialogue state information, and the model training steps of steps 120-140 are executed repeatedly until it is determined that any iteration training reaches the preset convergence condition, and the training is stopped to determine the trained policy network with the best model performance among the policy networks after multiple iteration trainings to construct the dialogue model.

[0058] Among them, the strategy network to be trained The training goal is to approximate the action probability distribution generated during the tree search process , which can be achieved by maximizing the log-likelihood of the action probability distribution, that is, minimizing the cross entropy loss. The specific calculation formula is as follows: ; In the formula, is the loss of the strategy network to be trained; is the probability of using sample reply action information a to conduct a dialogue under dialogue state information s, calculated by tree search; is the probability of using the sample reply action information a to conduct a dialogue under the dialogue state information s, as predicted by the strategy network to be trained; is the model parameter of the strategy network to be trained; this loss function is used to measure the predicted action probability distribution of the strategy network to be trained and the action probability distribution generated during the tree search process By minimizing this loss, the prediction of the policy network to be trained is made closer to the result of the tree search.

[0059] Among them, the model parameters of the strategy network to be trained The updated calculation formula is as follows: ; In the formula, is the learning rate, which is used to control the step size of parameter update. is the parameter gradient of the loss of the policy network to be trained.

[0060] Value network to be trained The training goal is to approximate the actual long-term return produced by tree search , which can be achieved through mean square error loss. The specific calculation formula is as follows: ; In the formula, is the loss of the value network to be trained; The value score output by the value network to be trained for the dialogue state information s; are the model parameters of the value network to be trained; is the average value of the global reward of the dialogue trajectory obtained by performing multiple searches on the dialogue state information s through tree search.

[0061] Then the value network to be trained The updated calculation formula is as follows: ; In the formula, is the learning rate, which is used to control the step size of parameter update. is the parameter gradient of the loss of the value network to be trained.

[0062] The method provided in this embodiment generates current sample dialogue state information based on current sample user input information and historical sample dialogue information in the sample dialogue to comprehensively obtain dialogue context information; and performs tree search based on the strategy network to be trained and the value network to be trained to comprehensively explore the dialogue space and quickly generate high-quality sample data. Then, based on the high-quality sample data generated by the tree search, the strategy network to be trained and the value network to be trained are updated using reinforcement learning to utilize the technical advantages of reinforcement learning and tree search. Through multi-level modeling and dynamic strategy optimization, a dialogue model with long-term planning capabilities and strategy optimization capabilities for multi-round dialogues is gradually trained, so that the dialogue model can consider long-term rewards and strategy optimization when generating responses, and avoid falling into invalid repetition or logically erroneous response modes. Therefore, through the dialogue model, it is possible to ensure that the dialogue system performs well in balancing context consistency, global goal achievement, and generation quality, thereby improving the generation quality and interaction capabilities of multi-round dialogues, as well as the user experience.

[0063] In some embodiments, the current sample reply action information in step 120 is obtained by searching based on the following steps: Based on the to-be-trained policy network, applying the current sample dialogue state information, generating reply information for the current sample user input information, and obtaining a target reply set; Based on the strategy network to be trained, the predicted reply information corresponding to the current sample user input information is evaluated as belonging to the action probability of each candidate reply information in the target reply set, and based on the value network to be trained, the value score corresponding to each candidate reply information is evaluated; Taking the current sample user input information as the root node, performing node selection, expansion, simulation and backtracking according to the action probability and the value score, to obtain a plurality of dialogue tracks corresponding to the current sample dialogue state information, and a value score of each reply action information corresponding to the current sample dialogue state information in the plurality of dialogue tracks; According to the value score, the current sample reply action information is searched in the multiple dialogue tracks.

[0064] Optionally, in the process of determining the action space, the current sample dialogue state information may be input into the policy network to be trained, so that the policy network to be trained applies the current sample dialogue state information to make a reply action decision, so as to generate initial reply information corresponding to the current sample user input information, that is, all possible reply information of the current sample user input information generated based on the analysis of the current sample dialogue state information and the current sample user input information, so as to form a target reply set.

[0065] It should be noted that, in the process of determining the target response set, all initial response information generated by the strategy network to be trained can be directly combined to form a target response set; or, the initial response information generated by the strategy network to be trained can be lightweight processed to form high-quality response information, and the high-quality response information can be combined to form a target response set, etc. This embodiment does not specifically limit this.

[0066] Then, the strategy network to be trained is used to evaluate the action probability that the predicted reply information corresponding to the current sample user input information belongs to each candidate reply information in the target reply set, and the value score corresponding to each candidate reply information is evaluated based on the value network to be trained. The current sample user input information is used as the root node, and a tree search process is performed based on the action probability and the value score, which specifically includes node selection, expansion, simulation and backtracking processes, so as to obtain multiple dialogue trajectories corresponding to the current sample dialogue state information, and the value score of each reply action information corresponding to the current sample dialogue state information in the multiple dialogue trajectories.

[0067] like Figure 2 As shown in the figure, in the tree search process, for each round of search, the current sample user input information can be used as the root node, and the child node selection is performed based on the Upper Confidence Bound (UCB) algorithm, as well as the action probability and value score, and the child node is moved down in the tree until the leaf node is reached. The calculation formula for child node selection is as follows: ; In the formula, and are the value scores of the dialogue using the sample reply action information a under the dialogue state information s output by the value network to be trained, and the action probability of the dialogue using the sample reply action information a under the dialogue state information s output by the strategy network to be trained; N is the number of node visits, For all candidate reply information (that is, all candidate reply actions), is a hyperparameter.

[0068] After node selection, MCTS generates unvisited reply actions (i.e., reply information) for the selected leaf nodes to expand new nodes; for the newly generated nodes, MCTS simulates and predicts future user input information until the termination conditions are met, such as the user actively ending the conversation, the model responding abnormally, the conversation goal being reached, etc., and uses a reward model (such as a global reward model that includes a process reward model and a result reward model) to assign the global reward value of the entire trajectory to the newly generated nodes; finally, the obtained global reward value is traced back from the leaf node to the root node, and the statistical information stored in each node is updated, such as the value score value Q and the number of visits N; among them, the value score value Q is updated based on the updated number of visits, the current value score value, and the global reward value obtained in the current simulation.

[0069] After obtaining the value scores of each reply action information corresponding to the current sample dialogue state information in multiple dialogue tracks, the reply action information with the highest value score may be selected as the current sample reply action information.

[0070] The method provided in this embodiment performs Monte Carlo tree search by using the policy network to be trained and the value network to be trained, so as to select, expand, simulate and backtrack nodes through the output of the policy network to be trained and the output of the value network to be trained, so as to guide the efficient search of the Monte Carlo tree through pruning to obtain high-quality state information and action information data pairs, so as to help improve the decision-making quality and dialogue effect of the dialogue model.

[0071] In some embodiments, based on the to-be-trained policy network, applying the current sample dialogue state information, generating reply information for the current sample user input information, and obtaining a target reply set includes: Inputting the current sample dialogue state information into the to-be-trained policy network to obtain a plurality of initial reply information corresponding to the current sample user input information; Pruning the multiple initial reply information to obtain the target reply set.

[0072] Optionally, in the process of determining the action space, the current sample dialogue state information may be input into the to-be-trained policy network, so that the to-be-trained policy network applies the current sample dialogue state information to make a reply action decision, so as to generate initial reply information corresponding to the current sample user input information. Furthermore, in order to reduce the amount of calculation, the initial reply information generated by the to-be-trained policy network may be pruned, so as to retain the top k reply information with better dialogue quality among the multiple initial reply information as candidate reply information to form a target reply set.

[0073] The conversation quality here can be obtained by jointly evaluating indicators including relevance, accuracy, diversity, and fluency of the reply information. The relevance indicator is used to measure the degree of association between the reply information and the current sample user input information and the current sample conversation status information; the accuracy indicator is used to evaluate whether the reply information can correctly answer the user's questions or meet the user's needs; the diversity indicator is used to ensure that the reply information has a certain degree of variation to avoid repetition and monotony; the fluency indicator is used to check whether the language expression of the reply information is smooth and natural.

[0074] The method provided in this embodiment, in the process of determining the action space, generates initial reply information by inputting the current sample dialogue state information into the strategy network to be trained, and prunes it based on the dialogue quality index to retain the top k reply information with better dialogue quality as candidate reply information to form a target reply set, thereby reducing the amount of calculation while improving the quality of the reply information, thereby improving the performance of the dialogue system and user experience.

[0075] In some embodiments, the global reward value of the dialogue track corresponding to the current sample reply action information in step 120 is calculated based on the following steps: According to the current sample dialogue state information and the reply information corresponding to the current sample reply action information, the dialogue quality of the current sample reply action information in the current round of the sample dialogue is evaluated to obtain a process reward value of the current sample reply action information; According to the user input information and reply information in the dialogue track corresponding to the current sample reply action information, and the dialogue target of the sample dialogue, the response degree of the current sample reply action information to the dialogue target is evaluated to obtain the result reward value of the current sample reply action information; The global reward value is determined according to the process reward value and the result reward value.

[0076] like Figure 2As shown in the figure, the global reward value of each dialogue track in the tree search is evaluated by the reward model. The reward model is divided into a process reward model (PRM) and an outcome reward model (ORM) according to the dialogue goal. The process reward model generates reward values ​​by evaluating the dialogue quality for each round of dialogue; the outcome reward model generates reward values ​​by evaluating the response degree of the dialogue goal for multiple rounds of dialogue in the dialogue track.

[0077] Optionally, during the reward evaluation process, the current sample conversation state information and the reply information corresponding to the current sample reply action information may be input into a process reward model, so that the process reward model can apply the current sample conversation state information and the reply information corresponding to the current sample reply action information to evaluate the conversation quality of the current sample reply action information in the current round of the sample conversation, so as to obtain the process reward value of the current sample reply action information based on the evaluation result.

[0078] The dialogue quality assessment here includes but is not limited to the assessment of indicators such as security, correctness, naturalness, diversity, personification, and semantic coherence. The naturalness indicator is used to check whether the language expression of the reply information is smooth and natural. Specifically, the naturalness can be evaluated by calculating the semantic score and grammatical score of the reply information through the language model; the diversity indicator is used to ensure that the reply information has a certain change to avoid repetition and monotony. Specifically, it can be evaluated by evaluating the score or entropy value of the change of parameters such as vocabulary, sentence structure, and expression in the reply information through the language model; the personification indicator is used to check whether the language expression of the reply information is smooth and natural. Specifically, it can be evaluated by evaluating the language expression, emotional color, tone change, etc. in the reply information through the language model to evaluate the personification; the semantic coherence indicator is used to evaluate whether the reply information is semantically coherent and consistent with the current sample user input information and the current sample dialogue state information. Specifically, it can be evaluated by calculating the semantic similarity or correlation score between the reply information and the dialogue context (that is, user input information, dialogue state information) to evaluate semantic coherence.

[0079] In the process of calculating the process reward value, the dialogue quality of the current sample reply action information in the current round of the sample dialogue can be evaluated by combining the above-mentioned multi-dimensional dialogue quality evaluation indicators to obtain the multi-dimensional dialogue quality evaluation results of the current sample reply action information, and the process reward value of the current sample reply action information can be obtained by fusing the multi-dimensional dialogue quality evaluation results. The fusion here can be direct addition or weighted addition, etc., which is not specifically limited in this embodiment.

[0080] In addition, the user input information and reply information in the dialogue trajectory corresponding to the current sample reply action information, and the dialogue goal of the sample dialogue are input into the result reward model, so that the result reward model selects a response degree evaluation index corresponding to the dialogue goal according to different dialogue goals, and evaluates the response degree of the current sample reply action to the dialogue goal according to the corresponding response degree evaluation index, and obtains the result reward value of the current sample reply action information based on the evaluation result.

[0081] The dialogue goal here is adaptively determined according to different dialogue scenarios. For example, when the dialogue scenario of the sample dialogue is a task-based dialogue scenario, the dialogue goal of the sample dialogue is to complete a specific dialogue task. Accordingly, the response level evaluation index corresponding to the dialogue goal can be the degree of task completion, such as task completion efficiency, consistency and correctness of task completion, etc. This embodiment does not specifically limit this. When the dialogue scenario of the sample dialogue is an open dialogue scenario, the dialogue goal of the sample dialogue is to meet user needs. Accordingly, the response level evaluation index corresponding to the dialogue goal can be user feedback score and user emotion score, etc. This implementation does not specifically limit this.

[0082] Finally, the process reward value and the result reward value are combined to calculate the global reward value of the dialogue trajectory corresponding to the current sample reply action information, so as to balance the quality of immediate response in multiple rounds of dialogue and the optimization of the overall dialogue strategy based on this global reward value. The combination here can be achieved by weighted addition or direct addition, etc., which is not specifically limited in this implementation.

[0083] Compared with the prior art that only focuses on the quality of the current round of dialogue, the method provided in this embodiment optimizes the model by jointly considering the long-term rewards of multiple rounds of dialogue and the quality of the current round of dialogue during the tree search process, so as to more accurately output reply information with excellent performance in terms of context consistency, global goal achievement, and generation quality. For example, in a task-based dialogue, a decision can be made to select a reply that currently appears to be suboptimal, which may lead to a reply that is easier to achieve the goal in future rounds as the current reply information; in an open dialogue scenario, there may be multiple possible reply paths, and the exploration mechanism of the tree search is used to explore potential high-quality paths to select the current reply information, rather than relying entirely on the existing distribution in the training data, so as to more effectively guide the large model to optimize in the direction of better strategies such as context consistency, global goal achievement, and generation quality, thereby improving the generation quality and interaction capabilities of multiple rounds of dialogue, as well as the user experience.

[0084] Figure 3 is a flowchart of the steps of generating the next sample dialogue state information provided by the present invention; Figure 3As shown, in some embodiments, the next sample dialog state information in step 130 is generated based on the following simulation steps: Step 131, determining a target language model according to the strategy network to be trained or multiple pre-trained language models; Step 132, generating user input prompt information according to the current sample dialogue state information; Step 133, inputting the current sample reply action information and the user input prompt information into the target language model to obtain the next sample user input information corresponding to the current sample reply action information; Step 134 , generating the next sample dialogue state information according to the next sample user input information, the historical sample dialogue information, and the response information corresponding to the current sample user input information and the current sample response action information.

[0085] The multiple pre-trained language models here can be a BERT model, a generative pre-trained Transformer, a text-to-text conversion Transformer, and multiple human-computer interaction models deployed in various interactive systems for human-computer interaction, which are not specifically limited in this embodiment.

[0086] Optionally, during the state transfer process, a model may be selected from the policy network to be trained or multiple pre-trained language models as the target language model, or a model may be selected from multiple pre-trained language models for supervised training as the target language model. The target language model is a language model that simulates the user and can perform high-quality simulation of the next user input information of the current sample reply action information based on the prompt information of the current sample dialogue state information and the current sample reply action information.

[0087] The selection here may be implemented by random selection, or by selection based on the degree of association with the dialogue scene corresponding to the sample dialogue, etc. This embodiment does not specifically limit this.

[0088] In addition, user input prompt information can also be generated based on the current sample dialogue state information, and the prompt information is intended to instruct the target language model to simulate the next question or input that the user may ask in a specific dialogue state. In the process of generating prompt information, prompt information can be automatically generated from the current sample dialogue state information using a prompt model that is iteratively trained using sample prompt information and sample dialogue state information to obtain user input prompt information; or key information such as topic information, dialogue task, and dialogue style can be extracted from the current sample dialogue state information to generate input prompt information based on the extracted key information to obtain user input prompt information.

[0089] Then, the current sample reply action information and the user input prompt information are input into the target language model, so as to guide the target language model through the user input prompt information, and simulate the user to generate input information according to the current sample reply action information, thereby obtaining the next sample user input information corresponding to the current sample reply action information.

[0090] After obtaining the next sample user input information, it can be based on the current sample dialogue state information , current sample reply action information and the next sample user input information , generate the next sample dialogue state information For specific steps, please refer to the steps for generating the current sample dialogue state information, which will not be described in detail here.

[0091] The method provided in this embodiment integrates multiple pre-trained language models and policy networks to be trained, and uses prompt engineering to guide the model to simulate the dialogue mode of user input, so as to generate the next sample dialogue state information with high quality, thereby improving the coherence and naturalness of the dialogue system in multiple rounds of dialogue, thereby enhancing the overall performance of the dialogue system and user experience.

[0092] In some embodiments, step 131 specifically includes: The strategy network to be trained is used as the target language model, or a pre-trained language model is randomly selected from a plurality of pre-trained language models as the target language model.

[0093] Optionally, in the process of selecting the target language model, in order to enhance the ability of the policy network to be trained to simulate user behavior and the diversity of dialogue generation, the policy network to be trained can be used as the target language model, so as to guide the policy network to be trained to play the role of the user through prompt engineering, and generate the user's hypothetical response by imitating the user's dialogue method to obtain the next sample user input information.

[0094] In addition, in the process of selecting the target language model, in order to simulate the user's diverse conversation methods as much as possible and improve the diversity, coherence and naturalness of the dialogue system in simulating the interaction between users and the system, multiple pre-trained language models can be selected as candidate language models, and a candidate language model can be randomly sampled as the target language model in each round of the sample dialogue. The randomly sampled candidate language model can be guided to play the role of the user through prompt engineering, and the user's hypothetical response can be generated by imitating the user's conversation method to obtain the next sample user input information.

[0095] In some embodiments, step 131 further includes: Taking the historical reply information corresponding to each historical sample user input information in the historical sample dialogue information as a sample, and taking the next historical sample user input information corresponding to each historical sample user input information in the historical sample dialogue information as a label, supervised training is performed on any one of the multiple pre-trained language models to obtain the target language model.

[0096] Optionally, in the process of selecting the target language model, the historical reply information corresponding to each historical sample user input information in the historical sample dialogue information can be used as a sample, and the next historical sample user input information corresponding to each historical sample user input information in the historical sample dialogue information can be used as a label to perform supervised training on any of the multiple pre-trained language models to optimize and obtain a target language model that is more suitable for the dialogue scenario and has stronger prediction ability, and guide the supervised trained language model to play the role of the user through prompt engineering, so as to generate a hypothetical response of the user by imitating the user's dialogue method, and obtain the next sample user input information, so as to improve the accuracy of the user input information, thereby enhancing the overall performance and user experience of the dialogue system.

[0097] In some embodiments, this embodiment also provides a dialogue method. Figure 4 It is a flow chart of the dialogue method provided by the present invention; Figure 4 As shown, the method includes step 410 and step 420.

[0098] Step 410, generating current dialog state information according to current user input information and historical dialog information in the target dialog.

[0099] The target dialogue here refers to the continuous multiple rounds of human-computer dialogue required in practical applications.

[0100] The current user input information in the target dialogue refers to the user input information of the current dialogue round input by the target user in the interactive interface during the target dialogue.

[0101] The historical dialogue information refers to the historical target user input information generated by the historical rounds of dialogue of the target dialogue and the reply information corresponding to the historical target user input information.

[0102] In the state representation process, the current user input information and the historical conversation information may be directly spliced ​​to form the current conversation state information of the target conversation, or the current user input information and the context information may be extracted from the historical conversation information and then spliced ​​to form the current conversation state information, etc. This embodiment does not specifically limit this.

[0103] Step 420, inputting the current dialogue state information into the dialogue model to obtain reply information corresponding to the current user input information; The dialogue model is constructed based on the dialogue model training method provided in the above embodiments.

[0104] Optionally, before executing step 420, a joint tree search and reinforcement learning training may be performed to obtain a dialogue model that can accurately output response information with excellent performance in terms of context consistency, global goal achievement, and generation quality based on dialogue state information during a dialogue. The specific training steps can be found in steps 110 to 140, which will not be repeated here.

[0105] After obtaining the dialogue model, the current dialogue state information is applied through the dialogue model to generate multiple candidate reply information for the current user input information, and the candidate reply information with the highest value score is decided to be output as the reply information corresponding to the current user input information among multiple possible reply information, thereby providing users with a more natural, smooth and accurate dialogue experience.

[0106] The method provided in this embodiment, by utilizing the technical advantages of reinforcement learning and tree search, gradually trains through multi-level modeling and dynamic strategy optimization to obtain a dialogue model with long-term planning capabilities and strategy optimization capabilities for multi-round dialogues, so that the dialogue model can consider long-term rewards and strategy optimization when generating responses, and avoid falling into invalid repetitive or logically erroneous response patterns. Therefore, through the dialogue model, it can be ensured that the dialogue system performs well in balancing context consistency, achieving global goals, and generating quality, thereby improving the generation quality and interaction capabilities of multi-round dialogues, as well as the user experience.

[0107] The dialogue model training system provided by the present invention is described below. The dialogue model training system described below and the dialogue model training method described above can be referenced to each other.

[0108] Figure 5 is a schematic diagram of the structure of the dialogue model training system provided by the present invention; Figure 5 As shown, the system includes: The first state generating unit 510 is used to generate current sample dialogue state information according to current sample user input information and historical sample dialogue information in the sample dialogue; The search unit 520 is used to perform a tree search based on the to-be-trained policy network and the to-be-trained value network to obtain the current sample reply action information corresponding to the current sample dialogue state information and the global reward value of the dialogue trajectory corresponding to the current sample reply action information; The sample construction unit 530 is used to simulate and generate the next sample dialogue state information according to the current sample reply action information, and construct sample data according to the current sample dialogue state information, the current sample reply action information, the global reward value, and the next sample dialogue state information; The model building unit 540 is used to jointly train the strategy network to be trained and the value network to be trained according to the sample data, and to build a dialogue model according to the training results.

[0109] The system provided in this embodiment generates current sample dialogue state information based on current sample user input information and historical sample dialogue information in the sample dialogue to comprehensively obtain dialogue context information; and performs tree search based on the strategy network to be trained and the value network to be trained to comprehensively explore the dialogue space and quickly generate high-quality sample data. Then, based on the high-quality sample data generated by the tree search, the strategy network to be trained and the value network to be trained are updated using reinforcement learning to utilize the technical advantages of reinforcement learning and tree search. Through multi-level modeling and dynamic strategy optimization, a dialogue model with long-term planning capabilities and strategy optimization capabilities for multi-round dialogues is gradually trained, so that the dialogue model can consider long-term rewards and strategy optimization when generating responses, and avoid falling into invalid repetition or logically erroneous response modes. Therefore, through the dialogue model, it is possible to ensure that the dialogue system performs well in balancing context consistency, achieving global goals, and generating quality, thereby improving the generation quality and interaction capabilities of multi-round dialogues, as well as the user experience.

[0110] In some embodiments, the search unit is specifically configured to: Based on the to-be-trained policy network, applying the current sample dialogue state information, generating reply information for the current sample user input information, and obtaining a target reply set; Based on the strategy network to be trained, the predicted reply information corresponding to the current sample user input information is evaluated as belonging to the action probability of each candidate reply information in the target reply set, and based on the value network to be trained, the value score corresponding to each candidate reply information is evaluated; Taking the current sample user input information as the root node, performing node selection, expansion, simulation and backtracking according to the action probability and the value score, to obtain a plurality of dialogue tracks corresponding to the current sample dialogue state information, and a value score of each reply action information corresponding to the current sample dialogue state information in the plurality of dialogue tracks; According to the value score, the current sample reply action information is searched in the multiple dialogue tracks.

[0111] In some embodiments, the search unit is further configured to: Inputting the current sample dialogue state information into the to-be-trained policy network to obtain a plurality of initial reply information corresponding to the current sample user input information; Pruning the multiple initial reply information to obtain the target reply set.

[0112] In some embodiments, the search unit is further configured to: According to the current sample dialogue state information and the reply information corresponding to the current sample reply action information, the dialogue quality of the current sample reply action information in the current round of the sample dialogue is evaluated to obtain a process reward value of the current sample reply action information; According to the user input information and reply information in the dialogue track corresponding to the current sample reply action information, and the dialogue target of the sample dialogue, the response degree of the current sample reply action information to the dialogue target is evaluated to obtain the result reward value of the current sample reply action information; The global reward value is determined according to the process reward value and the result reward value.

[0113] In some embodiments, the sample construction unit is specifically used to: Determining a target language model according to the to-be-trained strategy network or a plurality of pre-trained language models; Generate user input prompt information according to the current sample dialogue state information; Inputting the current sample reply action information and the user input prompt information into the target language model to obtain the next sample user input information corresponding to the current sample reply action information; The next sample dialogue state information is generated according to the next sample user input information, the historical sample dialogue information, and the response information corresponding to the current sample user input information and the current sample response action information.

[0114] In some embodiments, the sample construction unit is further used to: The strategy network to be trained is used as the target language model, or a pre-trained language model is randomly selected from a plurality of pre-trained language models as the target language model.

[0115] In some embodiments, the sample construction unit is further used to: Taking the historical reply information corresponding to each historical sample user input information in the historical sample dialogue information as a sample, and taking the next historical sample user input information corresponding to each historical sample user input information in the historical sample dialogue information as a label, supervised training is performed on any one of the multiple pre-trained language models to obtain the target language model.

[0116] In some embodiments, this embodiment also provides a dialogue system. Figure 6 is a schematic diagram of the structure of the dialogue system provided by the present invention; Figure 6 As shown, the system includes: The second state generating unit 610 is used to generate current dialogue state information according to current user input information and historical dialogue information in the target dialogue; The dialogue unit 620 is used to input the current dialogue state information into the dialogue model to obtain reply information corresponding to the current user input information; The dialogue model is constructed based on the dialogue model training method as described above.

[0117] The system provided in this embodiment, by utilizing the technical advantages of reinforcement learning and tree search, gradually trains through multi-level modeling and dynamic strategy optimization to obtain a dialogue model with long-term planning capabilities and strategy optimization capabilities for multi-round dialogues, so that the dialogue model can consider long-term rewards and strategy optimization when generating responses, and avoid falling into invalid repetitive or logically erroneous response patterns. Therefore, by conducting conversations through the dialogue model, it can be ensured that the dialogue system performs well in balancing context consistency, achieving global goals, and generating quality, thereby improving the generation quality and interaction capabilities of multi-round dialogues, as well as the user experience.

[0118] The system provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0119] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7As shown, the electronic device may include: a processor (processor) 710 , a communication interface (Communications Interface) 720 , a memory (memory) 730 and a communication bus 740 , wherein the processor 710 , the communication interface 720 , and the memory 730 communicate with each other through the communication bus 740 . The processor 710 may call the logic instructions in the memory 730 to execute the dialogue model training method, which includes: generating current sample dialogue state information according to current sample user input information and historical sample dialogue information in the sample dialogue; performing tree search based on the policy network to be trained and the value network to be trained to obtain current sample reply action information corresponding to the current sample dialogue state information, and a global reward value of the dialogue trajectory corresponding to the current sample reply action information; simulating and generating next sample dialogue state information according to the current sample reply action information, and constructing sample data according to the current sample dialogue state information, the current sample reply action information, the global reward value, and the next sample dialogue state information; jointly training the policy network to be trained and the value network to be trained according to the sample data, and constructing a dialogue model according to the training result; or, executing a dialogue method, which includes: generating current dialogue state information according to current user input information and historical dialogue information in the target dialogue; inputting the current dialogue state information into the dialogue model to obtain reply information corresponding to the current user input information.

[0120] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0121] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the dialogue model training method provided by the above methods, which includes: generating current sample dialogue state information according to current sample user input information and historical sample dialogue information in the sample dialogue; performing tree search based on the policy network to be trained and the value network to be trained to obtain current sample reply action information corresponding to the current sample dialogue state information, and a global reward value of the dialogue trajectory corresponding to the current sample reply action information; simulating and generating next sample dialogue state information according to the current sample reply action information, and constructing sample data according to the current sample dialogue state information, the current sample reply action information, the global reward value, and the next sample dialogue state information; jointly training the policy network to be trained and the value network to be trained according to the sample data, and constructing a dialogue model according to the training result; or executing a dialogue method, which includes: generating current dialogue state information according to current user input information and historical dialogue information in the target dialogue; inputting the current dialogue state information into the dialogue model to obtain reply information corresponding to the current user input information.

[0122] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when executed by a processor to execute the dialogue model training method provided by the above methods, the method comprising: generating current sample dialogue state information according to current sample user input information and historical sample dialogue information in the sample dialogue; performing tree search based on the strategy network to be trained and the value network to be trained to obtain current sample reply action information corresponding to the current sample dialogue state information, and a global reward value of the dialogue trajectory corresponding to the current sample reply action information; simulating and generating next sample dialogue state information according to the current sample reply action information, and constructing sample data according to the current sample dialogue state information, the current sample reply action information, the global reward value, and the next sample dialogue state information; jointly training the strategy network to be trained and the value network to be trained according to the sample data, and constructing a dialogue model according to the training result; or executing a dialogue method, the method comprising: generating current dialogue state information according to current user input information and historical dialogue information in the target dialogue; inputting the current dialogue state information into the dialogue model to obtain reply information corresponding to the current user input information.

[0123] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0124] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dialogue model training method, characterized in that: include: Generate current sample dialogue state information based on current sample user input information and historical sample dialogue information in the sample dialogue; Performing a tree search based on the to-be-trained policy network and the to-be-trained value network to obtain current sample reply action information corresponding to the current sample dialogue state information, and a global reward value of the dialogue trajectory corresponding to the current sample reply action information; Simulating and generating next sample dialogue state information according to the current sample reply action information, and constructing sample data according to the current sample dialogue state information, the current sample reply action information, the global reward value, and the next sample dialogue state information; The strategy network to be trained and the value network to be trained are jointly trained according to the sample data, and a dialogue model is constructed according to the training results.

2. The method for training a dialogue model according to claim 1, characterized in that: The current sample reply action information is obtained by searching based on the following steps: Based on the to-be-trained policy network, applying the current sample dialogue state information, generating reply information for the current sample user input information, and obtaining a target reply set; Based on the strategy network to be trained, the predicted reply information corresponding to the current sample user input information is evaluated as belonging to the action probability of each candidate reply information in the target reply set, and based on the value network to be trained, the value score corresponding to each candidate reply information is evaluated; Taking the current sample user input information as the root node, performing node selection, expansion, simulation and backtracking according to the action probability and the value score, to obtain a plurality of dialogue tracks corresponding to the current sample dialogue state information, and a value score of each reply action information corresponding to the current sample dialogue state information in the plurality of dialogue tracks; According to the value score, the current sample reply action information is searched in the multiple dialogue tracks.

3. The dialogue model training method according to claim 2, characterized in that: The method of generating reply information for the current sample user input information based on the to-be-trained policy network and applying the current sample dialogue state information to obtain a target reply set includes: Inputting the current sample dialogue state information into the to-be-trained policy network to obtain a plurality of initial reply information corresponding to the current sample user input information; Pruning the multiple initial reply information to obtain the target reply set.

4. The method for training a dialogue model according to any one of claims 1 to 3, characterized in that: The global reward value of the dialogue trajectory corresponding to the current sample reply action information is calculated based on the following steps: According to the current sample dialogue state information and the reply information corresponding to the current sample reply action information, the dialogue quality of the current sample reply action information in the current round of the sample dialogue is evaluated to obtain a process reward value of the current sample reply action information; According to the user input information and reply information in the dialogue track corresponding to the current sample reply action information, and the dialogue target of the sample dialogue, the response degree of the current sample reply action information to the dialogue target is evaluated to obtain the result reward value of the current sample reply action information; The global reward value is determined according to the process reward value and the result reward value.

5. The method for training a dialogue model according to any one of claims 1 to 3, characterized in that: The next sample dialogue state information is generated based on the following simulation steps: Determining a target language model according to the to-be-trained strategy network or a plurality of pre-trained language models; Generate user input prompt information according to the current sample dialogue state information; Inputting the current sample reply action information and the user input prompt information into the target language model to obtain the next sample user input information corresponding to the current sample reply action information; The next sample dialogue state information is generated according to the next sample user input information, the historical sample dialogue information, and the response information corresponding to the current sample user input information and the current sample response action information.

6. The method for training a dialogue model according to claim 5, characterized in that: The step of determining a target language model according to the strategy network to be trained or the plurality of pre-trained language models includes: The strategy network to be trained is used as the target language model, or a pre-trained language model is randomly selected from a plurality of pre-trained language models as the target language model.

7. The method for training a dialogue model according to claim 5, characterized in that: The step of determining a target language model according to the strategy network to be trained or the plurality of pre-trained language models includes: Taking the historical reply information corresponding to each historical sample user input information in the historical sample dialogue information as a sample, and taking the next historical sample user input information corresponding to each historical sample user input information in the historical sample dialogue information as a label, supervised training is performed on any one of the multiple pre-trained language models to obtain the target language model.

8. A conversation method, characterized in that: include: Generate current dialogue state information based on current user input information and historical dialogue information in the target dialogue; Inputting the current dialogue state information into the dialogue model to obtain reply information corresponding to the current user input information; Wherein, the dialogue model is constructed based on the dialogue model training method as described in any one of claims 1 to 7.

9. A dialogue model training system, characterized in that: include: A first state generating unit, configured to generate current sample dialogue state information according to current sample user input information and historical sample dialogue information in the sample dialogue; A search unit, configured to perform a tree search based on the to-be-trained policy network and the to-be-trained value network to obtain current sample reply action information corresponding to the current sample dialogue state information and a global reward value of a dialogue trajectory corresponding to the current sample reply action information; A sample construction unit, configured to simulate and generate next sample dialogue state information according to the current sample reply action information, and construct sample data according to the current sample dialogue state information, the current sample reply action information, the global reward value, and the next sample dialogue state information; The model building unit is used to jointly train the strategy network to be trained and the value network to be trained according to the sample data, and to build a dialogue model according to the training results.

10. A dialogue system, characterized in that: include: A second state generating unit, configured to generate current dialog state information according to current user input information and historical dialog information in the target dialog; A dialogue unit, used to input the current dialogue state information into the dialogue model to obtain reply information corresponding to the current user input information; Wherein, the dialogue model is constructed based on the dialogue model training method as described in any one of claims 1 to 7.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the dialog model training method according to any one of claims 1 to 7 is implemented, or the dialog method according to claim 8 is implemented.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the dialogue model training method according to any one of claims 1 to 7 is implemented, or the dialogue method according to claim 8 is implemented.

Images