User model determination method, device and equipment for dialogue model training and computer readable medium

Abstract

The application discloses a user model determination method, device and equipment for dialogue model training and a computer readable medium. The method comprises the following steps: constructing a user model set from an expert simulator by adopting neighborhood randomization, wherein the user model set comprises at least two user models; training a dialogue model by using different user models in the user model set respectively, and obtaining different dialogue tracks; evaluating the dialogue quality of the user models according to the dialogue tracks and by using an evaluator based on inverse reinforcement learning, and determining a target user model. The user model evaluation technical solution disclosed in the embodiment of the application can successfully improve the performance of the dialogue model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and in particular to a user model determination method and device for dialogue model training, an equipment and a computer readable medium. BACKGROUND

[0002] Task-oriented dialogue systems aim to help users complete real-life tasks through dialogue, such as helping users find goods, book hotels and restaurants, etc. Compared with open-domain dialogue systems, task-oriented dialogue systems focus more on completing certain specific tasks in one or more domains. Task-oriented dialogue systems use deep reinforcement learning (DRL) to learn strategies, train DRL dialogue models with different dialogue trajectories, and can improve the generalization ability of dialogue models. An effective diversification method is to let dialogue models interact with user models, but user models usually lack the language complexity of human dialogue and contain generation errors, and the design bias may reduce the performance of dialogue models. SUMMARY

[0003] In order to solve at least one technical problem existing in the prior art, the embodiments of the present application provide a user model determination method, device, equipment and computer readable medium for dialogue model training. The technical solution is as follows:

[0004] In the first aspect, a user model determination method for dialogue model training is provided, and the method comprises:

[0005] A user model set is constructed from an expert simulator using neighborhood randomization, and the user model set includes at least two user models;

[0006] The dialogue model is trained using different user models in the user model set to obtain different dialogue trajectories;

[0007] According to the dialogue trajectory and using an evaluator based on inverse reinforcement learning, the dialogue quality of the user model is evaluated to determine a target user model.

[0008] Further, the user model set is constructed from the expert simulator using neighborhood randomization, comprising:

[0009] A plurality of neural network models with different initial parameters are used to train user models corresponding to different dialogue environments, and the initial parameters are randomly generated, and the training target of the user model is:

[0010]

[0011] Wherein, θ is the parameter vector of the user model,

[0012] M θIt is a user model.

[0013] A u It is the number of dialogue behaviors available in the user model.

[0014] It is the state of the user model.

[0015] a t,i It is the action of the i-th user model.

[0016] t represents the number of dialogue rounds, and T is the maximum number of dialogue rounds.

[0017] Furthermore, the step of training the dialogue model using different user models from the user model set to obtain different dialogue trajectories includes:

[0018] The user model initializes the task objective, which includes constraints and requests.

[0019] The dialogue model engages in dialogue with the user model according to the task objectives generated by different user models, thereby obtaining the dialogue trajectory. The dialogue trajectory includes: the interaction tuple generated after the dialogue model and the user model complete the dialogue. The interaction tuple includes: the state of the dialogue model, the behavior of the dialogue model, the state of the user model, the behavior of the user model, and the dialogue reward value.

[0020] Furthermore, the step of training the dialogue model using different user models from the user model set to obtain different dialogue trajectories includes:

[0021] The diversification ratio is calculated using the following formula:

[0022]

[0023] Among them, T i T represents the interaction tuple at time i. j Let D represent the interactive tuple at time j. u D is used to store the interactive tuples of the expert simulator. s Interaction tuples used to store diverse user models;

[0024] The user model is evaluated based on the diversity ratio to determine whether it meets the diversity conditions. If it does, the dialogue trajectory corresponding to the user model is obtained; otherwise, the dialogue trajectory corresponding to the user model is not obtained.

[0025] Further, the step of determining the target user model by evaluating the dialogue quality of the user model based on the dialogue trajectory and utilizing an evaluator based on inverse reinforcement learning includes:

[0026] Predict dialogue rewards in human conversations based on human dialogue trajectories;

[0027] The target user model is determined by performing near-end policy optimization learning on the user model based on the dialogue reward.

[0028] Furthermore, the prediction of dialogue rewards in human dialogue based on human dialogue trajectories includes:

[0029] Constructing human dialogue trajectories as a Boltzmann distribution model;

[0030] The dialogue reward is predicted using gradient descent and based on the Boltzmann distribution model.

[0031] Further, the step of performing proximal policy optimization learning on the user model based on the dialogue reward to obtain a dialogue policy includes:

[0032] The optimal user model in the user model is determined based on the preset cumulative discount revenue;

[0033] Using the optimal user model as a benchmark, calculate the distance between the other user models in the user model set and the optimal user model;

[0034] The target user model to be trained is determined based on the distance.

[0035] Secondly, a user model determination device for training a dialogue model is provided, the device comprising:

[0036] The user model set construction module is used to construct a user model set from the expert simulator using neighborhood randomization. The user model set includes at least two user models corresponding to different dialogue environments.

[0037] The training module is used to train the dialogue model using different user models in the user model set to obtain different dialogue trajectories.

[0038] The target model determination module is used to determine the target user model based on the dialogue trajectory and by evaluating the dialogue quality of the user model using an inverse reinforcement learning-based evaluator.

[0039] Furthermore, the user model set construction module is specifically used for:

[0040] Using multiple neural network models with different initial parameters, user models corresponding to different dialogue environments are trained respectively. The initial parameters are randomly generated, and the training objective of the user models is:

[0041]

[0042] Where θ is the parameter vector of the user model.

[0043] Furthermore, the training module is specifically used for:

[0044] The user model initializes the task objectives, which include constraints and requests.

[0045] The dialogue model engages in dialogue with the user model according to the task objectives generated by different user models, and obtains the dialogue trajectory. The dialogue trajectory includes: the interaction tuple generated after the dialogue model and the user model complete the dialogue. The interaction tuple includes: the state of the dialogue model, the behavior of the dialogue model, the state of the user model, the behavior of the user model, and the dialogue reward value.

[0046] Furthermore, training module 402: a diverse evaluation module for user models, used for:

[0047] The diversification ratio is calculated using the following formula:

[0048]

[0049] Among them, D u D is used to store the interactive tuples of the expert simulator. s Interaction tuples used to store diverse user models;

[0050] The user model is evaluated based on the diversity ratio to determine whether it meets the diversity criteria. If it does, the dialogue trajectory corresponding to the user model is obtained; otherwise, the dialogue trajectory corresponding to the user model is not obtained.

[0051] Furthermore, the target model determination module includes:

[0052] The dialogue reward prediction module is used to predict dialogue rewards in human dialogue based on human dialogue trajectories.

[0053] The optimization learning module is used to perform near-end policy optimization learning on the user model based on dialogue rewards to determine the target user model.

[0054] Furthermore, the dialogue reward prediction module is specifically used for:

[0055] Constructing human dialogue trajectories as a Boltzmann distribution model;

[0056] We use gradient descent and Boltzmann distribution model to predict dialogue rewards.

[0057] Furthermore, the learning module is optimized, specifically for:

[0058] The optimal user model in the user model is determined based on the preset cumulative discount revenue;

[0059] Using the optimal user model as a benchmark, calculate the distance between other user models in the user model set and the optimal user model;

[0060] The target user model for training is determined based on distance.

[0061] Thirdly, an electronic device is provided, comprising:

[0062] One or more processors; and

[0063] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the method as described in any of the first aspects.

[0064] Fourthly, a computer-readable medium is provided having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any of the first aspects.

[0065] The beneficial effects of the technical solution provided by the embodiments of the present invention are as follows:

[0066] 1. The user model evaluation technology disclosed in this embodiment of the invention incorporates an evaluator based on inverse reinforcement learning to obtain the dialogue quality of the user model, so as to select high-quality dialogue trajectories.

[0067] 2. The user model evaluation technology disclosed in this embodiment of the invention selects high-quality user models to participate in policy learning by designing a sampling environment distribution, thereby maintaining diversity while effectively controlling the quality of training trajectories.

[0068] 3. The simulation experiments on the Multiwoz dataset show that the user model evaluation technology disclosed in this embodiment of the invention can successfully improve the performance of the dialogue model. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 This is a flowchart of the user model determination method for dialogue model training provided in an embodiment of the present invention;

[0071] Figure 2 These are the robustness test results of user models obtained using different determination methods provided in the embodiments of the present invention;

[0072] Figure 3These are experimental results of hyperparameters of user models obtained using different determination methods, provided in embodiments of the present invention.

[0073] Figure 4 This is a schematic diagram of the user model determination device for dialogue model training provided in an embodiment of the present invention;

[0074] Figure 5 This is a schematic diagram of the electronic device structure provided in an embodiment of the present invention. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0076] This invention provides a method, apparatus, device, and computer-readable medium for determining user models for dialogue model training. The specific technical solutions are as follows:

[0077] A method for determining the user model for training a dialogue model, comprising:

[0078] S1. Construct a set of user models from the expert simulator using neighborhood randomization. The set of user models includes at least two user models.

[0079] As described above, the expert simulator is a rule-based simulator used to generate dialogue trajectories for learning dialogue policies and to generate diverse user models. In this embodiment, the expert simulator is represented as M0, and the user model is represented as M. θi User models cannot fully mimic real human conversational behavior, lacking the complexity and flexibility of real user dialogue, resulting in poor performance in real human conversations. Compared to training in a single environment, applying neighborhood randomization can generate a sufficient number of user models in an expert simulator, improving the generalization ability of the user models.

[0080] In one embodiment, step S1 includes:

[0081] Using multiple neural network models with different initial parameters, user models corresponding to different dialogue environments are trained respectively. The initial parameters are randomly generated, and the training objective of the user models is:

[0082]

[0083] Here, θ represents the parameter vector of the user model. The user model learns θ using a parameterized multilayer perceptron neural network, improving the diversity of dialogue by learning multiple models.

[0084] M θ It is a user model.

[0085] A u It is the number of dialogue behaviors available in the user model.

[0086] It is the state of the user model.

[0087] a t,i It is the action of the i-th user model.

[0088] t represents the number of dialogue rounds, and T is the maximum number of dialogue rounds.

[0089] The above describes the use of neighborhood randomization to construct a diverse set of user simulators M from the expert simulator. θi Each user model is trained using a separate neural network with the same structure but different initial parameters. This diverse set of simulators is called a diverse set of user models (DUME). To flexibly build different user models, the training trajectory τ can be decomposed into individual state-action pairs. Policy learning is defined as choosing the correct dialogue action given a state, i.e., from... (User status) to (User state or user behavior) learning mapping, where θ is the model parameter vector in the optimization objective. The learning of the user model is achieved by a multilayer perceptron neural network parameterized by θ, which improves the diversity of dialogue by learning multiple models.

[0090] S2. Train the dialogue model using different user models from the user model set to obtain different dialogue trajectories.

[0091] In one embodiment, step S2 includes:

[0092] The user model initializes the task objectives, which include constraints and requests.

[0093] The dialogue model engages in dialogue with the user model according to the task objectives generated by different user models, obtaining dialogue trajectories. These trajectories include interaction tuples generated after the dialogue between the dialogue model and the user model completes the dialogue. Interaction tuples include: the state of the dialogue model. Behavior of dialogue models Dialogue reward value r t The state of the user model User model behavior

[0094] As described above, at the beginning of each dialogue session, the user simulator initializes a user goal, which includes constraints and requests. Constraints refer to the requirements that the system response must meet, and requests refer to the missing information needed to complete the user task. The user state is defined as follows:

[0095]

[0096] Where G represents the user goal, including constraints and requests, and A represents the historical actions of the dialogue model. In dialogue round t, the user will... and the actions taken by the dialogue model a t-1 Take action And transition to the next user state. User actions are defined as in This is a binary variable representing whether the dialogue behavior is active at the time of the dialogue. If the dialogue model achieves the user's goal, the dialogue session will be successfully terminated by the user. Otherwise, if the dialogue model gives an incorrect answer or the maximum number of dialogues is reached, the dialogue session will be identified as failed.

[0097] The dialogue state tracker monitors the dialogue session to update the dialogue state. It records constraint information from the user and the information the user wants to query. Taking the dialogue history and dialogue action 'a' as input, it updates the dialogue state 's'. The actions and states of the dialogue model are defined as follows:

[0098] State: The system state during a dialogue round. in, Indicates the current user's request action; a t-1 and b t This represents the system's actions at the previous moment and historical round information; q t This represents the results retrieved from the database.

[0099] Action: An action 'a' in a dialogue model is a subset of the dialogue behavior set A. In a multi-domain setting, a dialogue behavior is represented by a quadruple consisting of domain, intent, slot type, and slot value (e.g., [hotel, inform, area, east]).

[0100] Interaction tuples are tuples generated by the interaction between the dialogue model and the user model or expert simulator. For example, at time t, the interaction tuple is: A trajectory segment is a sequence of interacting tuples. For example, at time j to k, the trajectory segment is...

[0101] In one embodiment, step S2 further includes:

[0102] The diversification ratio is calculated using the following formula:

[0103]

[0104] Among them, D u D is used to store the interactive tuples of the expert simulator. s T is used to store interaction tuples for diverse user models. i T represents the interaction tuple at time i. j Represents the interaction tuple at time j;

[0105] The user model is evaluated based on the diversity ratio to determine whether it meets the diversity criteria. If it does, the dialogue trajectory corresponding to the user model is obtained; otherwise, the dialogue trajectory corresponding to the user model is not obtained.

[0106] The above describes how dialogue diversity is controlled by setting the ratio σ between the expert simulator and the interaction between the diverse user model and the dialogue model. u D is used to store the interactive tuples of the expert simulator. s The interaction tuples used to store diverse user models are used to calculate the diversity ratio σ. The larger the σ is, the more diverse the learning of the dialogue model.

[0107] S3. Based on the dialogue trajectory and using an evaluator based on inverse reinforcement learning to assess the dialogue quality of the user model, determine the target user model.

[0108] As mentioned above, user model diversification is achieved through randomized neural network parameter initialization. However, errors in user model learning can quickly propagate to policy learning. The dialogue evaluator is used to select dialogue trajectories from better user models to participate in policy learning, limiting the noise introduced by diversification. The goal of inverse reinforcement learning is to learn a policy network that mimics the black-box policy of human experts.

[0109] In one embodiment, step S3 includes:

[0110] Predict dialogue rewards in human conversations based on human dialogue trajectories;

[0111] Based on dialogue rewards, the user model is used to perform near-end policy optimization learning to determine the target user model.

[0112] The above-mentioned process of inverse reinforcement learning mainly uses human dialogue strategy as expert strategy, learns rewards from expert strategy through inverse reinforcement, and then infers human dialogue strategy based on rewards. In this way, the target user model can be determined based on human dialogue strategy, and thus the generalized user model can be determined.

[0113] In one embodiment, predicting dialogue rewards in human dialogue based on human dialogue trajectories includes:

[0114] Constructing human dialogue trajectories as a Boltzmann distribution model;

[0115] We use gradient descent and Boltzmann distribution model to predict dialogue rewards.

[0116] The above, based on maximum entropy IRL, shows the trajectory of human dialogue. Can be modeled f w (τ) follows a Boltzmann distribution and does not reveal any additional preferences in the dialogue. The reward estimator maximizes the log-probability of observed human dialogue sessions to infer potential targets:

[0117]

[0118]

[0119] r represents the reward for action a, and γ is the reward discount factor.

[0120] The evaluator distinguishes between real human dialogue and generated dialogue by minimizing the KL divergence between the user model's trajectory distribution and the Boltzmann distribution, and maximizing the KL divergence between the policy distribution and the Boltzmann distribution. The evaluator's initial loss function is:

[0121] J f (ω)=-KL(p D (τ)|q ω (τ))+KL(π θ (τ)||q ω (τ))

[0122] Therefore, the gradient of the evaluator is:

[0123]

[0124] The estimator uses the entire session τ, which is very inefficient due to sparsity issues and can have high variance depending on the varying length of the session. The dialogue τ can be decomposed into state-action pairs (s, a), and the loss function of the estimator is:

[0125]

[0126] Since the estimator evaluates a state-action pair, it can predict the reward in each dialogue round:

[0127]

[0128] Reward estimator f ωIt can be transformed into a reward approximator g ω and shaping item h ω Using the state-behavior-state triple (s, a, s) next Instead of the state-action pair (s, a), use it as the input to the reward estimator:

[0129] f ω (s,a,s next ) = g ω (s,a)+γh ω (s next )-h ω (s).

[0130] In one embodiment, a dialogue policy is obtained by performing proximal policy optimization learning on a user model based on dialogue rewards, including:

[0131] The optimal user model in the user model is determined based on the preset cumulative discount revenue;

[0132] Using the optimal user model as a benchmark, calculate the distance between other user models in the user model set and the optimal user model;

[0133] The target user model for training is determined based on distance.

[0134] As mentioned above, the diverse user simulators constructed using domain randomization exhibit model differences. The training environment may have varying impacts on the generalization ability and robustness of the policy. An evaluator is used to select the user model with the optimal quality. In a bounded reward function MDP, we define the expected cumulative discount reward η(π|M) θi )Measure the performance of the user model:

[0135]

[0136] Based on the above cumulative discount benefit η(π|M) θi This allows us to obtain the best-performing user model from the set of user models.

[0137]

[0138] The sequential decision problem in a fully observable stochastic environment with a Markov transition model and additional rewards is called a Markov decision process or MDP, which consists of a set of states (with an initial state s0); each state has a set of actions including a transition model P(s'|s,a) and a reward function R(s).

[0139] Using the optimal user model as a benchmark, calculate the distance between the current user model and the optimal user model:

[0140]

[0141] The target user model for participating in the training of the dialogue model is selected based on its distance from the optimal user model.

[0142] When calculating the differences between environment models, it is necessary to estimate the expected total distance between the optimal user model and every other sampled environment at each time step. Without a model, the analytical equations for environment transitions cannot be obtained, therefore, the total distance between two environments cannot be calculated. Under deterministic state transitions, the transition probability of the environment state is 1. We set the user model's state to the same state as the optimal user model's trajectory, and then compare the next state, progressively calculating the total distance.

[0143] For each environment p i ,calculate Then according to ε(p) i ,π)≥ε(p k (π) represents the selected user model. k is a hyperparameter used to control the expected cumulative discount reward. and distance In theory, a smaller k means that the selected model is closer to the optimal model.

[0144] The policy is optimized using sampled trajectories; the state-value function... State value function The advantage function A(s, a) = Q(s, a) - V(s) is used. Proximal Policy Optimization (PPO) learns dialogue policies and is a simple and stable policy-based RL algorithm.

[0145]

[0146] The above, E t It is the expectation of the trajectory τ sampled by the policy;

[0147] This represents the ratio of new to old strategies, ensuring that the agent takes the most possible steps to improve the strategy based on the data it currently has, without causing unexpected performance degradation.

[0148] μ is a hyperparameter;

[0149] clip constrains the dialogue strategy through a clipping probability ratio, clip(l t (θ), 1-μ, 1+μ) are eliminated by the pruning probability ratio The motivation to move outside the interval [1-μ, 1+μ];

[0150] A t Let A represent the dominance function at time t. t A value greater than 0 indicates that the current strategy is better, encouraging the user to perform this action more often. tA value less than 0 indicates that the strategy is worse, and it should be penalized by reducing the number of times this action is performed.

[0151] The entire MDPO algorithm is shown below:

[0152]

[0153] To more specifically demonstrate the beneficial effects of the user model determination technology disclosed in the embodiments of the present invention, the following comparative analysis is provided with specific experiments:

[0154] Dataset: The proposed method is evaluated using the MultiWOZ dataset. The MultiWOZ dataset is a multi-domain, multi-intent task-oriented dialogue corpus. MultiWOZ contains 7 domains, 13 intents, 25 slot types, and 10438 dialogues. Each dialogue is annotated with a series of dialogue states and corresponding system dialogue behaviors. The multi-domain dialogues cover 7 task domains: restaurants, hotels, tourist attractions, taxis, trains, hospitals, and police. Approximately 70% of the dialogues exceed 10 rounds, with each round involving at least 2 to 5 domains. A single dialogue involves multiple task domains, reflecting how people converse in real life.

[0155] Table 1. Dataset Statistics (Multiwoz)

[0156]

[0157]

[0158] All domains in the dataset have a script above the words in a complete ontology table to identify their respective domains: *:universal, 1:restaurant, 2:hotel, 3:attraction, 4:taxi, 5:train, 6:hospital, 7:police.

[0159] Evaluation metrics: The evaluation criteria for task-oriented dialogues are informF1, matchrate, turns, and success. InformF1 assesses whether the dialogue model provides all the information requested by the user. Matchrate assesses whether the registered entities meet all specified constraints; a score of 1 is given if the correct entity is registered, and 0 otherwise. Turns assess the total number of dialogue turns; fewer turns indicate better dialogue performance. Success is considered a success only if all information is provided and the entities are correctly registered.

[0160] Baseline: Comparing the performance of DRL dialogue models using different settings: a model-free approach, a method for generating complete trajectories from an environment model, a method for controlling the quality of trajectory generation from a user model, and a method for controlling the differences in user simulators.

[0161] PPO: PPO is a stochastic policy-based DRL algorithm that controls the magnitude of policy updates by limiting changes in the behavior distribution during policy updates when iteratively updating the Critic's parameters. During policy learning, it uses real-world experience from an expert simulator.

[0162] PPO (Dvs): Incorporating multiple environment models into the dialogue model to generate simulated experiences. The dialogue model interacts with different learning user models, generating complete trajectories from the environment models, and using the simulated experiences generated by the environment models to enhance the training of the dialogue model.

[0163] MDPO(Dvs): In diverse user models, the environment used for training has different effects on policy learning. MRPO selects high-quality user models for training.

[0164] PPO (ISEE): When user models generate simulated experiences, the initial state of the environment model is obtained from the expert simulator, controlling the quality of diverse user model dialogues.

[0165] MDPO (ISEE): Dialogue models use ISEE to gain simulation experience and MRPO to learn policies.

[0166] In the experiment, the Multiwoz agenda-based simulator was used as the expert simulator, with 20 diverse user models. The policy network, value network, and diverse user models were implemented using three-layer perceptrons (MLPs). The reward estimator was divided into two networks, g(s,a) and h(s), each implemented using two-layer perceptrons (MLPs), with ReLU activation layers in all MLPs. r The buffer size is set to 1024, and the maximum length of the simulated dialogue is 40. The dialogue model shown is trained using a hybrid of expert simulators and diverse user simulators, with an η ratio of 0.2.

[0167] Experimental results:

[0168] Table 2 Dialogue effects on MultiWOZ

[0169]

[0170] As shown in Table 2, for each dialogue model, its effectiveness was verified based on notification rate, matching rate, success rate, and average number of rounds. With an average of 8 repeated experiments, the proposed method MDPO improved the performance of the policy. MDPO(ISEE) performed best, outperforming other baselines in matching rate, success rate, number of rounds, and average number of rounds. The average number of rounds was 7.9462, the matching rate was 90.08%, the notification rate was 89.83%, and the success rate was 88.00%. In the comparison between PPO(Dvs) and MDPO(Dvs), the diversity of dialogues in PPO(Dvs) was uncontrolled, resulting in significant noise in the diverse user models. After applying MDPO, selecting higher-quality user models significantly improved the success rate of the dialogue model. In the comparison between PPO(Dvs) and MDPO(Dvs), the initial state of the environment model was obtained from an expert simulator, improving the quality of the user model data. Using MDPO to select trajectory segments that better reflect human dialogue behavior further improved the performance of the dialogue model.

[0171] To evaluate the robustness of the model, box plots are used to visualize the results, such as... Figure 2 As shown, MDPO can improve the stability of the model in the test environment, with lower fluctuations in results compared to other methods. It also shows excellent performance on each evaluation metric, thus improving the robustness of the model.

[0172] The experiment also investigated the relationship between the hyperparameters of the method disclosed in this invention and the success rate, which are responsible for the diversity of user models. These are the diversity ratio σ, model distance k, and user model set size E; as these parameters increase, the degree of diversification increases. According to... Figure 3 The learning curve of the dialogue model is plotted using three parameters, a, b, and c.

[0173] When the model achieves optimal success, each hyperparameter has an optimal value. The optimal value of σ needs to be determined by seeking the best trade-off between the quality of the world model and the number of simulated experiences useful for improving the dialogue model. Figure 3 In (a), σ = 0.1, 0.2, 0.5, 1, 2 was selected, and the diversification ratio of 0.2 showed the best performance. The distance between the K-control model and the optimal model is... Figure 3 In (b), k = 20, 30, 40, 50, 60 were selected, and the model distance of 30 showed the best performance. In the user model, E is a hyperparameter that controls the number of user models. A larger E indicates greater diversity, but too many user models may introduce noise and reduce the performance of the dialogue model. Figure 3 In (c), E = 5, 10, 15, 20, 25 were selected. Through comparative experiments, the dialogue model performed best when the number of user models was 20.

[0174] Based on the user model determination method for dialogue model training disclosed in the above embodiments of the present invention, such as Figure 4 As shown, this embodiment of the invention also provides a user model determination device for dialogue model training, comprising:

[0175] User model set construction module 401 is used to construct a user model set from the expert simulator using neighborhood randomization. The user model set includes at least two user models corresponding to different dialogue environments.

[0176] Training module 402 is used to train the dialogue model using different user models in the user model set to obtain different dialogue trajectories;

[0177] The target model determination module 403 is used to determine the target user model based on the dialogue trajectory and by evaluating the dialogue quality of the user model using an inverse reinforcement learning-based evaluator.

[0178] In one embodiment, the user model set construction module 401 is specifically used for:

[0179] Using multiple neural network models with different initial parameters, user models corresponding to different dialogue environments are trained respectively. The initial parameters are randomly generated, and the training objective of the user models is:

[0180]

[0181] Where θ is the parameter vector of the user model.

[0182] M θ It is a user model.

[0183] A u It is the number of dialogue behaviors available in the user model.

[0184] It is the state of the user model.

[0185] a t,i It is the action of the i-th user model.

[0186] t represents the number of dialogue rounds, and T is the maximum number of dialogue rounds.

[0187] In one embodiment, the training module 402 is specifically used for:

[0188] The user model initializes the task objectives, which include constraints and requests.

[0189] The dialogue model engages in dialogue with the user model according to the task objectives generated by different user models, and obtains the dialogue trajectory. The dialogue trajectory includes: the interaction tuple generated after the dialogue model and the user model complete the dialogue. The interaction tuple includes: the state of the dialogue model, the behavior of the dialogue model, the state of the user model, the behavior of the user model, and the dialogue reward value.

[0190] In one embodiment, the training module 402 includes: a diverse evaluation module for the user model, used for:

[0191] The diversification ratio is calculated using the following formula:

[0192]

[0193] Among them, T i T represents the interaction tuple at time i. j Let D represent the interactive tuple at time j. u D is used to store the interactive tuples of the expert simulator. s Interaction tuples used to store diverse user models;

[0194] The user model is evaluated based on the diversity ratio to determine whether it meets the diversity criteria. If it does, the dialogue trajectory corresponding to the user model is obtained; otherwise, the dialogue trajectory corresponding to the user model is not obtained.

[0195] In one embodiment, the target model determination module 403 includes:

[0196] The dialogue reward prediction module is used to predict dialogue rewards in human dialogue based on human dialogue trajectories.

[0197] The optimization learning module is used to perform near-end policy optimization learning on the user model based on dialogue rewards to determine the target user model.

[0198] In one embodiment, the dialogue reward prediction module is specifically used for:

[0199] Constructing human dialogue trajectories as a Boltzmann distribution model;

[0200] We use gradient descent and Boltzmann distribution model to predict dialogue rewards.

[0201] In one embodiment, the optimization learning module is specifically used for:

[0202] The optimal user model in the user model is determined based on the preset cumulative discount revenue;

[0203] Using the optimal user model as a benchmark, calculate the distance between other user models in the user model set and the optimal user model;

[0204] The target user model for training is determined based on distance.

[0205] In addition, this invention also provides an electronic device, including:

[0206] One or more processors; and

[0207] A memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, perform the user model determination method disclosed in the above embodiments.

[0208] Among them, such as Figure 5 As shown, computer device 12 is represented as a general-purpose computing device. Components of computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and a bus 18 connecting different system components (including system memory 28 and processing units 16). Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0209] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0210] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (not shown, commonly referred to as a "hard disk drive"). Although not shown in the figures, disk drives for reading and writing to removable non-volatile disks (e.g., "floppy disks") and optical disk drives for reading and writing to removable non-volatile optical disks (e.g., CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0211] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0212] The computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with the computer device 12, and / or with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via the input / output (I / O) interface 22. Furthermore, in this embodiment, the display 24 of the computer device 12 is not an independent entity, but is embedded in a mirror, so that when the display surface of the display 24 is not displayed, the display surface of the display 24 and the mirror surface visually blend together. Moreover, the computer device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via the network adapter 20. As shown, the network adapter 20 communicates with other modules of the computer device 12 via the bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0213] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28.

[0214] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0215] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

[0216] All the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of the present invention, and will not be described in detail here.

[0217] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for determining a user model for training a dialogue model, characterized in that, include: A set of user models is constructed from an expert simulator using neighborhood randomization, the set of user models including at least two user models; Using multiple neural network models with different initial parameters, user models corresponding to different dialogue environments are trained respectively. The initial parameters are randomly generated, and the training objective of the user models is: Where θ is the parameter vector of the user model. M θ It is a user model. A u It is the number of dialogue behaviors available in the user model. It is the state of the user model. a t,i It is the action of the i-th user model. t represents the number of dialogue rounds, and T is the maximum number of dialogue rounds. The dialogue model is trained using different user models from the user model set to obtain different dialogue trajectories; The user model initializes the task objective, which includes constraints and requests. The dialogue model engages in dialogue with the user model according to the task objectives generated by different user models, thereby obtaining the dialogue trajectory. The dialogue trajectory includes: the interaction tuple generated after the dialogue model and the user model complete the dialogue. The interaction tuple includes: the state of the dialogue model, the behavior of the dialogue model, the state of the user model, the behavior of the user model, and the dialogue reward value. The diversification ratio is calculated using the following formula: Among them, T i Let T represent the interaction tuple at time i. j Let D represent the interactive tuple at time j. u D is used to store the interactive tuples of the expert simulator. s Interaction tuples used to store diverse user models; The user model is evaluated based on the diversity ratio to determine whether it meets the diversity conditions. If it does, the dialogue trajectory corresponding to the user model is obtained. If it does not meet the conditions, the dialogue trajectory corresponding to the user model is not obtained. Based on the dialogue trajectory and by evaluating the dialogue quality of the user model using an inverse reinforcement learning-based evaluator, the target user model is determined. Predict dialogue rewards in human conversations based on human dialogue trajectories; Based on the dialogue reward, the target user model is determined by performing near-end policy optimization learning on the user model. Constructing human dialogue trajectories as a Boltzmann distribution model; The dialogue reward is predicted using gradient descent and based on the Boltzmann distribution model. The optimal user model in the user model is determined based on the preset cumulative discount revenue; Using the optimal user model as a benchmark, calculate the distance between the other user models in the user model set and the optimal user model; The target user model to be trained is determined based on the distance.

2. A user model determination device for dialogue model training, characterized in that, include: The user model set construction module uses neighborhood randomization to construct a user model set from the expert simulator, and the user model set includes at least two user models; Using multiple neural network models with different initial parameters, user models corresponding to different dialogue environments are trained respectively. The initial parameters are randomly generated, and the training objective of the user models is: Where θ is the parameter vector of the user model. M θ It is a user model. A u It is the number of dialogue behaviors available in the user model. It is the state of the user model. a t,i It is the action of the i-th user model. t represents the number of dialogue rounds, and T is the maximum number of dialogue rounds. The training module trains the dialogue model using different user models from the user model set to obtain different dialogue trajectories. The user model initializes the task objective, which includes constraints and requests. The dialogue model engages in dialogue with the user model according to the task objectives generated by different user models, thereby obtaining the dialogue trajectory. The dialogue trajectory includes: the interaction tuple generated after the dialogue model and the user model complete the dialogue. The interaction tuple includes: the state of the dialogue model, the behavior of the dialogue model, the state of the user model, the behavior of the user model, and the dialogue reward value. The diversification ratio is calculated using the following formula: Among them, T i Let T represent the interaction tuple at time i. j Let D represent the interactive tuple at time j. u D is used to store the interactive tuples of the expert simulator. s Interaction tuples used to store diverse user models; The user model is evaluated based on the diversity ratio to determine whether it meets the diversity conditions. If it does, the dialogue trajectory corresponding to the user model is obtained. If it does not meet the conditions, the dialogue trajectory corresponding to the user model is not obtained. The target model determination module determines the target user model based on the dialogue trajectory and by evaluating the dialogue quality of the user model using an inverse reinforcement learning-based evaluator. Predict dialogue rewards in human conversations based on human dialogue trajectories; Based on the dialogue reward, the target user model is determined by performing near-end policy optimization learning on the user model. Constructing human dialogue trajectories as a Boltzmann distribution model; The dialogue reward is predicted using gradient descent and based on the Boltzmann distribution model. The optimal user model in the user model is determined based on the preset cumulative discount revenue; Using the optimal user model as a benchmark, calculate the distance between the other user models in the user model set and the optimal user model; The target user model to be trained is determined based on the distance.

3. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the method as described in claim 1.

4. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in claim 1.

Images