Task-oriented dialogue model training method and device, equipment and computer readable medium

Abstract

The application discloses a kind of intelligent dialogue model training method, device, equipment and computer readable medium, belong to computer technical field.The method includes: generating user target using generator, user target includes: request and constraint, request is the missing information required to complete dialogue task, constraint is the requirement that must be met to complete dialogue task;Extract user target, extract dialogue action from dialogue action data set according to dialogue strategy using dialogue model;Whether dialogue strategy is completed and dialogue action meets user target, if yes, then generate the positive reward of generator according to the first reward strategy, if dialogue strategy is completed and dialogue action does not meet user target, then generate the negative reward of generator according to the first reward strategy.The application uses the generator network optimized using adversarial training to generate the task that is always at the appropriate difficulty level of dialogue model, with the progress of dialogue model, force generator to propose more and more challenging user target.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and computer-readable medium for training a task-oriented dialogue model. Background Technology

[0002] Task-oriented dialogue systems typically use deep reinforcement learning to learn dialogue strategies. However, one problem with reinforcement learning is the reward problem. Rewards in many dialogue domains require careful design, and as the demands for more complex objectives increase, the workload of manually designing reward functions becomes unsustainable due to the increasing complexity of the tasks. Summary of the Invention

[0003] To address at least one technical problem existing in the prior art, embodiments of the present invention provide a method, apparatus, device, and computer-readable medium for training a task-oriented dialogue model. The technical solution is as follows:

[0004] Firstly, a method for training a task-oriented dialogue model is provided, the method comprising:

[0005] A generator is used to generate user objectives, which include requests and constraints. The requests are missing information required to complete the dialogue task, and the constraints are requirements that must be met to complete the dialogue task.

[0006] Extract the user target and use the dialogue model to extract dialogue actions from the dialogue action dataset according to the dialogue strategy;

[0007] If the dialogue strategy is completed and the dialogue action satisfies the user's goal, a positive reward for the generator is generated according to the first reward strategy. If the dialogue strategy is completed but the dialogue action does not satisfy the user's goal, a negative reward for the generator is generated according to the first reward strategy.

[0008] Furthermore, the first reward strategy includes:

[0009]

[0010] Where, r T t represents the reward of the generator, +α and -β represent the positive and negative rewards of the generator, respectively. + Indicates the dialogue turn, t * The threshold range for dialogue rounds is defined in the dialogue strategy.

[0011] Furthermore, it also includes:

[0012] If the dialogue action satisfies the user's goal, then a positive reward for the dialogue model is generated according to the second reward strategy.

[0013] Furthermore, the second reward strategy includes:

[0014]

[0015] Where r represents the reward of the dialogue model.

[0016] Further, the step of extracting the user target, and extracting dialogue actions from the dialogue action dataset using a dialogue model according to a dialogue strategy, includes:

[0017] The dialogue actions in the dialogue action dataset are determined according to the dialogue strategy;

[0018] Update the slot value of the determined dialogue action and the belief state of the dialogue action slot type.

[0019] Furthermore, it also includes:

[0020] Record the interaction tuple between the dialogue model and the generator, the interaction tuple including: Multiple interactive tuples constitute a trajectory segment, wherein, For the state of the dialogue model, For the behavior of the dialogue model, r t For dialogue reward points, For the state of the user model, For the behavior of the user model;

[0021] Store the trajectory segment.

[0022] Furthermore, the method also includes:

[0023] The dialogue strategy is optimized by randomly selecting trajectory segments from the stored multiple trajectory segments and employing a near-end optimization strategy.

[0024] Secondly, a smart dialogue model training device is provided, the device comprising:

[0025] A generator is used to generate user goals, which include requests and constraints, wherein the requests are missing information required to complete the dialogue task, and the constraints are requirements that must be met to complete the dialogue task.

[0026] A dialogue model is used to extract the user's target and extract dialogue actions from the dialogue action dataset according to the dialogue strategy.

[0027] The reward module is used to determine whether the dialogue strategy is completed and whether the dialogue action satisfies the user's goal. If so, a positive reward for the generator is generated according to the first reward strategy. If the dialogue strategy is completed but the dialogue action does not satisfy the user's goal, a negative reward for the generator is generated according to the first reward strategy.

[0028] Thirdly, an electronic device is provided, comprising:

[0029] One or more processors; and

[0030] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the method as described in any of the first aspects.

[0031] Fourthly, a computer-readable medium is provided having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any of the first aspects.

[0032] The beneficial effects of the technical solution provided by the embodiments of the present invention are as follows:

[0033] The model training method proposed in this invention involves a generator proposing an intrinsic goal with adversarial motivation, which can train a goal-conditional "dialogue model" policy under sparse environmental rewards.

[0034] The model training method proposed in this invention uses a generator network to generate user goals. The generator can automatically discover the range of tasks to be performed in its environment. In each step, adversarial training is used to optimize the generator network to generate tasks that are always at an appropriate difficulty level for the dialogue model. As the dialogue model improves, the generator is forced to propose increasingly challenging user goals. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart of the intelligent dialogue model training method provided in the embodiments of the present invention;

[0037] Figure 2 The AMIGG training provided in this embodiment of the invention includes objective-generating teacher and student policies;

[0038] Figure 3This is a performance diagram of MultiWOZ and agenda-based user models;

[0039] Figure 4 This is a learning curve diagram of the dialogue agent provided in the embodiments of the present invention;

[0040] Figure 5 This is a schematic diagram of the intelligent dialogue model training device provided in an embodiment of the present invention;

[0041] Figure 6 This is a schematic diagram of the electronic device structure provided in an embodiment of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0043] In dialogue systems, model learning involves building user dialogue models. Deep dynamic-q (DDQ) incorporates a world model into the dialogue model, mimicking real user responses and generating simulated experiences. The dialogue model is then optimized using both real and simulated experiences. To minimize the difference between trajectories generated from the world model and those generated from the original expert simulator, those skilled in the art employ a discriminator from a Generative Adversarial Network (GAN) to distinguish between real and simulated experiences. To reduce errors in trajectories generated by the world model, Intermittent Short Extended Ensemble (I-SEE) controls the quality of diversity by constraining the interaction strength with the world model ensemble.

[0044] The reward function for a dialogue strategy requires careful design. Temporal Supervised Learning (TSL) learns directly from example dialog boxes by supervising expert dialogues, maximizing expected discounted accuracy by selecting the next dialogue action. Some studies combine multiple components, such as success rate and dialogue length, while others find the optimal reward function by searching for reward component weights. However, these methods still require annotation of the dialog boxes by real users. GDPL utilizes inverse reinforcement learning to obtain a reward estimator that guides the dialogue strategy by evaluating state-action pairs in each round of dialogue. It also integrates adversarial learning into the method to simultaneously optimize both the dialogue strategy and the reward estimator.

[0045] When environmental rewards are scarce, dialogue models struggle to discover new reward states. Therefore, solving problems through random exploration requires a large number of samples for learning, and the sample size increases exponentially with the number of reward steps. Intrinsic motivation has been shown to solve complex exploration problems. Intrinsic motivation formulas guide dialogue models to understand environmental dynamics through curiosity-driven exploration or count-based exploration. These methods can help dialogue models learn in tasks with sparse rewards, but for tasks with complex rewards, there is a lack of automatic reward calculation methods in related technologies.

[0046] To address the problems existing in the prior art, this invention provides a method, apparatus, device, and computer-readable medium for training an intelligent dialogue model. The specific technical solutions are as follows:

[0047] S1. Use a generator to generate user goals, which include requests and constraints. Requests are the missing information needed to complete the dialogue task, and constraints are the requirements that must be met to complete the dialogue task.

[0048] As described above, the dialogue model extracts the user's goal from the dataset and engages in dialogue based on that goal. The dialogue model attempts to help the user achieve their goal by selecting appropriate dialogue actions during the conversation. The user's dialogue state s at time t is... u,t Represented as s u,t = (A, G), User state s u,t The process is broken down into agenda A and objective G. The objective includes requests and constraints. Constraints are the requirements that the system response must meet, while requests are the missing information needed for the user to complete the task. For example, a user wants to book a Chinese restaurant but doesn't know its location. Here, the constraint is food = Chinese, and the request is location = ?. Agenda A represents the historical operations of the dialogue model, tracking the progress of the conversation.

[0049] S2. Extract user targets and use the dialogue model to extract dialogue actions from the dialogue action dataset according to the dialogue strategy.

[0050] The above refers to the user's dialogue action a. u,t Represented as in It is a binary variable, representing the activation of the i-th action in the t-th dialogue round, A. u These are all available dialogue actions. The dialogue is formed by the user's action a. u,1 The process involves the user's request and the reservation of an entity. If the dialogue model fulfills all requests and correctly reserves the entity, the user will successfully terminate the dialogue session. If the number of dialogues reaches the maximum number of dialogues T, the dialogue session will be considered a failure.

[0051] S3. Determine whether the dialogue strategy is completed and whether the dialogue action satisfies the user's goal. If yes, generate a positive reward for the generator according to the first reward strategy. If the dialogue strategy is completed but the dialogue action does not satisfy the user's goal, generate a negative reward for the generator according to the first reward strategy.

[0052] The above process of determining whether a dialogue strategy is complete and whether the dialogue action satisfies the user's goal can be divided into two steps. Specifically, it can first determine whether the dialogue strategy is complete. If the dialogue strategy is complete, it then determines whether the dialogue action satisfies the user's goal. If the user's goal is satisfied, a positive reward from the generator is generated according to the first reward strategy; if the user's goal is not satisfied, a negative reward from the generator is generated according to the first reward strategy. If the dialogue strategy is not complete, the dialogue strategy continues to be executed. However, the above approach only determines whether the user's goal is satisfied when the dialogue strategy is complete, which may result in a situation where the dialogue strategy is not complete but the user's goal has already been satisfied. Therefore, another approach is to determine whether the user's goal is satisfied after each round of dialogue. If the user's goal is satisfied, a positive reward from the generator is generated according to the first reward strategy; if the user's goal is not satisfied, it then determines whether the dialogue strategy is complete. If the dialogue strategy is complete, a negative reward from the generator is generated according to the first reward strategy; if the dialogue strategy is not complete, the dialogue strategy continues to be executed.

[0053] In one embodiment, the first reward strategy includes:

[0054]

[0055] Where, r T t represents the generator's reward, +α and -β represent the generator's positive and negative rewards, respectively. + Indicates the dialogue turn, t * This refers to the threshold range for dialogue turns defined in the dialogue strategy.

[0056] The generator G(θ) mentioned above g The generator is an independent network that, for automatic curriculum generation, uses a Generative Adversarial Network (GAN) to generate user goals. The generator takes the user's goal as its output, keeping it unchanged until a new goal is proposed, maximizing its expected reward by proposing goals achievable by the dialogue model. The generator's reward is r. T It is a function of the dialogue model's performance on the generated target, calculated at the end of a dialogue. Since the generator operates at different temporal frequencies, its reward is not discounted based on the number of steps the dialogue model takes.

[0057] The generator should generate user goals with a difficulty level that is just right for the current dialogue model strategy. If the dialogue model achieves the user goal after a certain effort, it means that the user goal generated by the generator meets our requirements. If the dialogue model cannot achieve the user goal or achieves the goal too quickly, it means that the user goal generated by the generator is too difficult or too easy. This embodiment of the invention uses the dialogue model strategy in dialogue rounds t. + To measure the performance of the dialogue model, a threshold interval t is defined. * When the number of dialogue turns taken by the dialogue model to achieve the user's goal is within the threshold range, the generator is given a positive reward. If the dialogue model takes fewer dialogue turns or fails to complete the task within the number of dialogue turns specified by the dialogue strategy, the generator is given a negative reward.

[0058] The method disclosed in this invention uses a course-based learning dialogue strategy. For example... Figure 2 AMIGS consists of two sub-modules: a goal-conditioned dialogue model policy for learning dialogue behaviors within the system, and a goal generator for generating user goals to guide the dialogue model network's training. During training, the generator proposes user goals, receiving a reward only when it proposes tasks of appropriate difficulty. Simultaneously, the dialogue model network receives a cumulative discounted reward for achieving the generator's goals. The generator and dialogue model networks work in tandem; the dialogue model maximizes its cumulative discounted reward by achieving the user goals as quickly as possible, while the generator maximizes its reward by proposing goals achievable by the dialogue model.

[0059] In one embodiment, the method disclosed in this invention further includes:

[0060] If the dialogue action satisfies the user's goal, then a positive reward for the dialogue model is generated according to the second reward strategy.

[0061] As described above, task-oriented dialogue is a Markov Decision Process (MDP). The MDP consists of states, actions, rewards, policies, and transitions, interacting with the user through a series of actions to achieve the user's goal. In each step, the dialogue model, based on policy π(a... t |s t ,θ π Select action a t The dialogue model will receive a reward r and will respond to the user's action a. u,t Update to the next state s t+1 .

[0062] State S: Dialogue states s include the current user's request action, the system's previous action, historical round information, and database query results. Each state s = [a u,t ,a t-1,b t ,q t ]∈S is the concatenation of these four vectors. Action A: A dialogue round may contain multiple intentions, so the system action a is a subset of the action set A. In a multi-domain setting, action a∈A consists of domain, intention, slot type, and slot value (e.g., [hotel, inform, stars, 4]). Reward R: The feedback signal obtained by the dialogue model after performing an action is called the reward. For example, at step time t, r t =R(s) t ,a t This refers to the immediate reward obtained by the dialogue model, also known as reinforcement.

[0063] In one embodiment, the second reward strategy includes:

[0064]

[0065] Where r represents the reward of the dialogue model, s t The user model state is represented by `agenda`, which indicates the dialogue agenda of the dialogue model. An empty dialogue agenda means that the dialogue model has used up all dialogue rounds.

[0066] The goal of the dialogue model, as described above, is to maximize the expected cumulative discount reward. The strategy is defined as follows: γ is the discount factor, which is related to the number of dialogue rounds. The more dialogue rounds the dialogue model uses, the larger the discount factor becomes, and r is the reward of the dialogue model.

[0067] In one embodiment, step S2 specifically includes:

[0068] The dialogue actions in the dialogue action dataset are determined according to the dialogue strategy;

[0069] Update the slot value of the determined dialogue action and the belief state of the dialogue action slot type.

[0070] As described above, a dialogue tracker updates the dialogue state by recording information about user constraints in the user's goal and indicating the content the user wants to query. Given a dialogue action, the dialogue tracker updates the belief state of its slot type based on its slot value, preparing state s for policy learning. t The dialogue policy π describes the behavior of the dialogue model and is generally defined as a probability density function. The policy is based on the observed state s. t Output the probability distribution of all possible actions.

[0071] In one embodiment, for a dialogue policy, an initial dialogue policy is obtained from the dataset using supervised learning before training begins. Compared to a model without pre-training, the pre-trained initial policy can accomplish some simple objectives, reduces the number of initial interactions, and accelerates the dialogue policy learning process. Multi-label classification loss is used for pre-training, and the loss function is:

[0072]

[0073] Where x and y represent the action of the dialogue model and the target dialogue action in the dialogue action dataset, respectively.

[0074] In one embodiment, the method disclosed in this invention further includes:

[0075] Record the interaction tuple between the dialogue model and the generator, the interaction tuple including: Multiple interactive tuples constitute a trajectory segment;

[0076] Store the trajectory segment.

[0077] As mentioned above, the trajectory segment is a series of interaction tuples between the dialogue model and the user model. For example, at dialogue turn t, the interaction tuple is... Each element in the tuple represents: For the state of the dialogue model, For the behavior of the dialogue model, r t For dialogue reward points, For the state of the user model, This represents the user model's behavior. The trajectory segment between dialogue turns j and l is... The state transition function determines the trajectory segment within a certain time period, and subsequent tuples depend on preceding tuples. A complete trajectory segment is... Where k is the length of the entire dialogue. We store the trajectory segments in the experience buffer D.

[0078] In one embodiment, the method disclosed in this invention further includes:

[0079] The dialogue strategy is optimized by randomly selecting trajectory segments from the stored multiple trajectory segments and employing a near-end optimization strategy.

[0080] As described above, this embodiment of the invention uses Proximal Policy Optimization (PPO) to optimize the dialogue policy. PPO consists of two neural networks: an Actor and a Critic. The Actor outputs the probability of each action, and the Critic outputs the action value Q. The Actor network updates its parameters to increase the output value of the Critic network, which is updated based on rewards from the environment. The state value function is defined as... The action value function is defined as: The advantage function is defined as: A π (s,a)=Q π (s,a)-V π (s).

[0081] PPO by incorporating the proportion of new and old strategies The goal is to enable the dialogue model to take the most possible policy improvement steps based on the current data without causing unexpected performance degradation. The objective function of the PPO algorithm is:

[0082] L(θ)=E t [min(l t (θ)A t ,clip(l t (θ), 1-μ, 1+μ)A t )]

[0083] Where μ is a hyperparameter, clip(l t (θ), 1-μ, 1+μ) eliminate l by clipping the probability ratio. t The motivation to move outside the interval [1-μ, 1+μ]. A t Let A represent the dominance function at time t. t A value greater than 0 indicates that the current strategy is better, encouraging the user to perform this action more often. t A value less than 0 indicates a worse strategy, and the strategy is penalized by reducing the number of times this action is performed. Within one epoch, the PPO network updates multiple times using different batches of trajectory segments randomly drawn from buffer D.

[0084] The following specific experiments illustrate the beneficial effects of the intelligent dialogue model training method disclosed in this invention:

[0085] The proposed AMIGG method was evaluated on the Multiwoz dataset. Multiwoz is a fully labeled dataset for multi-domain dialogue, encompassing seven task domains: restaurants, hotels, tourist attractions, taxis, trains, hospitals, and police. It is an order of magnitude larger than any previous annotated task-oriented corpus. Multiwoz contains 10,438 dialogues, with 7,032 multi-domain dialogues involving at least 2 to 5 domains. Each dialogue is annotated with a series of dialogue states and corresponding system dialogue behaviors. A single dialogue involves multiple task domains, reflecting real-life conversational patterns. Approximately 70% of the dialogues have more than ten turns, with average turns for single-domain and multi-domain dialogues being 8.93 and 15.39 respectively, indicating that the corpus reflects the complexity of real-world dialogue. Table 1 shows the statistics of the dataset.

[0086] Table 1. Dataset Statistics (Multiwoz)

[0087]

[0088]

[0089] The script above the words in the table indicates their respective domains: *:universal, 1:restaurant, 2:hotel, 3:attraction, 4:taxi, 5:train, 6:hospital, 7:police.

[0090] Evaluation metrics for task-oriented dialogues include the number of dialogues and the task success rate. The task success rate is the primary metric, encompassing the F1 score and the matching rate. A task is considered successful only if all requested information is provided and all booked entities meet the user's needs. A successful dialogue scores 1, otherwise 0. The definitions of F1 score, matching rate, and number of dialogues are explained below.

[0091] Inform F1 evaluates whether the dialogue model has informed the user of all requested information (e.g., restaurant prices, address). The formula for calculating F1 is... Here, pr and re represent the precision and recall of the dialogue model's response information, respectively.

[0092] Match assesses whether the booked entity meets the user-specified restrictions (e.g., hotels in the north of the city). In multi-domain conversations, a score of 1 is given if the correct entity is registered, and 0 otherwise. The average score across all domains represents the score for a single conversation.

[0093] Turns evaluates the total number of turns in a dialogue; fewer turns indicate higher dialogue efficiency.

[0094] In the experiment, four algorithms that performed well on the Multiwoz dataset were compared and set as (1) PPO, an algorithm that interacts directly with the dialogue model; (2) DDQ, an algorithm that incorporates a world model to simulate real user responses and generate simulated experiences; (3) I-SEE, an algorithm that controls the quality of dialogue diversity and resists noise; (4) GDPL, an algorithm that uses inverse reinforcement learning to design a reward estimator and guides the dialogue policy by evaluating state-action pairs; and (5) our proposed AMIGG algorithm. PPO is a policy-based algorithm that, in our setting, interacts directly with the dialogue model and learns the dialogue policy using the obtained trajectory segments. DDQ (Deep Dyna Q) incorporates a world model into the dialogue model and generates simulated experiences. The dialogue model generates complete dialogue trajectories and updates the learned dialogue policy using real and simulated experiences. I-SEE improves the quality of dialogue diversity and resists noise by limiting the intensity of interaction with the world model. GDPL uses inverse reinforcement learning to construct a reward function, which can effectively solve the problem of reward sparsity and integrates adversarial learning into the method, allowing the dialogue policy and reward estimator to be learned alternately and simultaneously. AMIGG utilizes curriculum learning, where the generator proposes "constructive adversarial" goals, enabling the dialogue model to learn realistic dialogue strategies.

[0095] This experiment uses the Multiwoz agenda-based simulator as the dialogue model, with a generator producing the user's goal. The input dimension of both the generator and the goal generator is 512, and the output dimension is 1224. The dialogue model uses an input layer of 553 units, the PPO policy network has a hidden layer of 200 units and 166 outputs, and the value network has a hidden layer of 50 units and 1 output. We use Adam as the optimization algorithm.

[0096] A good dialogue model should score highly on every metric, not just one. The successful conclusion of a dialogue should accurately inform each request, thus guiding the dialogue through user goals. In a comparison of 8 repeated experiments, our proposed method, ANIGG, improves policy performance, showing excellent results in matching rate, success rate, average number of turns, and notification rate (see Table 2). Compared to simply using PPO, ANIGG significantly improves learning performance. Except for the notification rate F1 score of 88.259, which is only 1.24 lower than PPO, it performs best on all other evaluation metrics, providing more accurate slot values ​​and matching entities than other baselines. Specifically, the average number of turns is 7.188, the matching rate is 93.169%, and the success rate is 93.163%. In the comparison with GDPL, although GDPL's reward function is carefully designed, ANIGG effectively overcomes sparse rewards through curriculum learning, even outperforming GDPL. By using user goals generated by a generator and training with curriculum learning, the dialogue model can better explore the state space and effectively improve dialogue performance.

[0097] like Figure 3 As shown, to assess the robustness of the model, we visualized the results using box plots. Compared to the other four baselines, while AMIG exhibited some volatility, all other baselines showed negative outliers, whereas AMIG showed no outliers across all four evaluation metrics.

[0098] Table 2. Dialogue Efficiency of Multiwoz

[0099]

[0100] To investigate the impact of incorporating course learning on the efficiency of learning dialogue strategies, we plotted learning curves for different baseline dialogue models based on success rate, matching rate, and number of dialogue turns. The results are as follows: Figure 4 As shown, AMIGG achieves faster success and matching rate performance and exhibits better stability after convergence than other baselines. PPO learns its dialogue policy from a simulator, relying on randomized user goals with uncontrolled difficulty. For the initial policy stages, learning the dialogue policy on a simple task is more suitable, while for later training, the dialogue policy should learn more difficult tasks. Without curriculum learning, the PPO dialogue model took approximately 15 epochs to reach a success and matching rate of around 85%, while with curriculum learning, it reached approximately 93% in just 10 epochs. In terms of dialogue turns, AMIGG can complete the task with fewer dialogues, and its learning efficiency is also higher than other baselines. Table 3 lists a comparison of dialogue sessions between AMIGG and PPO.

[0101] Table 3 Comparison of MDPO (ISEE) and PPO in terms of dialogue.

[0102]

[0103]

[0104]

[0105] Based on the intelligent dialogue model training method disclosed in the above embodiments of the present invention, such as Figure 5 As shown, this embodiment of the invention also provides an intelligent dialogue model training device, comprising:

[0106] Generator 501 is used to generate user objectives, which include: requests and constraints, wherein the requests are missing information required to complete the dialogue task, and the constraints are requirements that must be met to complete the dialogue task;

[0107] Dialogue model 502 is used to extract the user target and extract dialogue actions from the dialogue action dataset according to the dialogue strategy.

[0108] The reward module 503 is used to determine whether the dialogue strategy is completed and whether the dialogue action satisfies the user's goal. If so, a positive reward for the generator is generated according to the first reward strategy. If the dialogue strategy is completed and the dialogue action does not satisfy the user's goal, a negative reward for the generator is generated according to the first reward strategy.

[0109] In one embodiment, the first reward strategy includes:

[0110]

[0111] Where represents the generator's reward, +α and -β represent the generator's positive and negative rewards, respectively, t + Indicates the dialogue turn, t * The threshold range for dialogue rounds is defined in the dialogue strategy.

[0112] In one embodiment, the reward module 503 is further configured to:

[0113] If the dialogue action satisfies the user's goal, then a positive reward for the dialogue model is generated according to the second reward strategy.

[0114] In one embodiment, the second reward strategy includes:

[0115]

[0116] Where r represents the reward of the dialogue model, st The user model state is represented by `agenda`, which indicates the dialogue agenda of the dialogue model. An empty dialogue agenda means that the dialogue model has used up all dialogue rounds.

[0117] In one embodiment, extracting the user target by using a dialogue model to extract dialogue actions from the dialogue action dataset according to a dialogue strategy includes:

[0118] The dialogue actions in the dialogue action dataset are determined according to the dialogue strategy;

[0119] Update the slot value of the determined dialogue action and the belief state of the dialogue action slot type.

[0120] In one embodiment, the apparatus further includes:

[0121] A recording module is used to record the interaction tuples between the dialogue model and the generator, the interaction tuples including: Multiple interactive tuples constitute a trajectory segment, wherein, For the state of the dialogue model, For the behavior of the dialogue model, r t For dialogue reward points, For the state of the user model, For the behavior of the user model;

[0122] A storage module is used to store the trajectory segments.

[0123] In one embodiment, the apparatus further includes:

[0124] The strategy optimization module is used to randomly select the trajectory segments from the multiple stored trajectory segments and optimize the dialogue strategy using a near-end optimization strategy.

[0125] In addition, this invention also provides an electronic device, including:

[0126] One or more processors; and

[0127] A memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, perform the intelligent dialogue model training method disclosed in the above embodiments.

[0128] Among them, such as Figure 6As shown, computer device 12 is represented as a general-purpose computing device. Components of computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and a bus 18 connecting different system components (including system memory 28 and processing units 16). Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0129] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0130] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (not shown, commonly referred to as a "hard disk drive"). Although not shown in the figures, disk drives for reading and writing to removable non-volatile disks (e.g., "floppy disks") and optical disk drives for reading and writing to removable non-volatile optical disks (e.g., CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0131] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0132] The computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with the computer device 12, and / or with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via the input / output (I / O) interface 22. Furthermore, in this embodiment, the display 24 of the computer device 12 is not an independent entity, but is embedded in a mirror, so that when the display surface of the display 24 is not displayed, the display surface of the display 24 and the mirror surface visually blend together. Additionally, the computer device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via the network adapter 20. As shown, the network adapter 20 communicates with other modules of the computer device 12 via the bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0133] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28.

[0134] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0135] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

[0136] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of the present invention, and will not be described in detail here.

[0137] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for training a task-oriented dialogue model, characterized in that, include: A generator is used to generate user objectives, which include requests and constraints. The requests are missing information required to complete the dialogue task, and the constraints are requirements that must be met to complete the dialogue task. Extract the user target and use the dialogue model to extract dialogue actions from the dialogue action dataset according to the dialogue strategy; A dialogue tracker is used to update the dialogue state by recording information about user constraints in the user's goals and indicating what the user wants to query. Given a dialogue action, the dialogue tracker updates the belief state of its slot type according to its slot value. Determine whether the dialogue strategy is completed and whether the dialogue action satisfies the user's goal. If yes, generate a positive reward for the generator according to the first reward strategy. If the dialogue strategy is completed but the dialogue action does not satisfy the user's goal, generate a negative reward for the generator according to the first reward strategy. The first reward strategy includes ， Where, r T t represents the reward of the generator, +α and -β represent the positive and negative rewards of the generator, respectively. + Indicates the dialogue turn, t * The threshold range for dialogue turns defined in the dialogue strategy; The dialogue model receives a cumulative discount reward for completing the generator's goal. If the dialogue action satisfies the user's goal, a positive reward for the dialogue model is generated according to the second reward strategy. The second reward strategy includes ， Where r represents the reward of the dialogue model.

2. The method as described in claim 1, characterized in that, The step of extracting the user target, which involves using a dialogue model to extract dialogue actions from the dialogue action dataset according to a dialogue strategy, includes: The dialogue actions in the dialogue action dataset are determined according to the dialogue strategy; Update the slot value of the determined dialogue action and the belief state of the dialogue action slot type.

3. The method as described in claim 1, characterized in that, Also includes: Record the interaction tuple between the dialogue model and the generator, the interaction tuple including: Multiple interactive tuples constitute a trajectory segment, wherein, For the state of the dialogue model, The behavior of the dialogue model is represented by rt, where rt is the dialogue reward value. For the state of the user model, For the behavior of the user model; store the trajectory segments.

4. The method as described in claim 3, characterized in that, The method further includes: The dialogue strategy is optimized by randomly selecting trajectory segments from the stored multiple trajectory segments and employing a near-end optimization strategy.

5. An intelligent dialogue model training device, comprising: A generator is used to generate user goals, which include requests and constraints. The requests are missing information required to complete the dialogue task, and the constraints are requirements that must be met to complete the dialogue task. Extract the user target and use the dialogue model to extract dialogue actions from the dialogue action dataset according to the dialogue strategy; A dialogue tracker is used to update the dialogue state by recording information about user constraints in the user's goals and indicating what the user wants to query. Given a dialogue action, the dialogue tracker updates the belief state of its slot type according to its slot value. The reward module determines whether the dialogue strategy is completed and whether the dialogue action satisfies the user's goal. If so, it generates a positive reward for the generator according to the first reward strategy. If the dialogue strategy is completed but the dialogue action does not satisfy the user's goal, it generates a negative reward for the generator according to the first reward strategy. The first reward strategy includes: ， Where rT represents the generator's reward, +α and -β represent the generator's positive and negative rewards, t+ represents the dialogue round, and t* is the dialogue round threshold range defined in the dialogue strategy; The dialogue model receives a cumulative discount reward for completing the generator's goal. If the dialogue action satisfies the user's goal, a positive reward for the dialogue model is generated according to the second reward strategy. The second reward strategy includes: ， Where r represents the reward of the dialogue model.

6. An electronic device, characterized in that, include: One or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the method as described in any one of claims 1 to 4.

7. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 4.

Images