Task type dialogue strategy learning method and system based on reinforcement learning

Abstract

The invention belongs to the technical field of intelligent task-based dialogue, and particularly relates to a task-based dialogue strategy learning method and system based on reinforcement learning, which adopts a soft actor commentator algorithm to be combined with behavior cloning and maximum entropy inverse reinforcement learning to relieve a cold start problem. According to the method, maximum entropy inverse reinforcement learning is utilized, a reward value is calculated, and a user target in a dialogue is accurately deduced according to maximum log likelihood estimation of a human dialogue and a simulated dialogue; a potential reward function is extracted from a successful track through inverse reinforcement learning, manual reward design is replaced, accumulated reward maximization is pursued during strategy optimization, diversity and exploratory performance of the strategy are encouraged, and the dialogue strategy is prevented from being converged to a single mode too early.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent task-oriented dialogue, and particularly relates to a task-oriented dialogue strategy learning method and system based on reinforcement learning. BACKGROUND

[0002] With the increasing complexity of user expectation targets, traditional single-domain dialogue systems are insufficient to face complex human dialogue tasks. At the same time, traditional hand-designed reward functions are no longer sufficient to meet the complexity of the real world, and the manually set reward mechanism in single-domain dialogue systems is no longer applicable to complex multi-domain dialogue.

[0003] User interactions in single-domain and multi-domain task-oriented dialogue systems are different. For example, when facing the topic of "ordering food in a restaurant", a single-domain dialogue system only provides the content related to the restaurant and ends the dialogue, while a multi-domain dialogue system task involves multiple domains and can have multi-round interactions across domains. Therefore, when facing complex demand targets involving multiple domains, a reward function that can dynamically guide the dialogue strategy to complete the task during the entire dialogue is needed. SUMMARY

[0004] The application provides a task-oriented dialogue strategy learning method and system based on reinforcement learning.

[0005] The technical solution of the application is as follows: The application provides a task-oriented dialogue strategy learning method based on reinforcement learning, comprising: S1: obtaining user dialogue data, decomposing user dialogue targets into a constraint set and a request set, and constructing a structured state-action pair; S2: after constructing the user state according to the historical dialogue text and the user dialogue target, initializing the user simulator based on the agenda or the variational hierarchical model; The action probability is calculated by the chain rule, the user action corresponding to the state and the binary terminal signal are generated, and are transmitted to the state tracker; S3: the state tracker captures keywords according to the current system action, the user action and the state at the last time, generates a new state, and transmits it to the policy network; S4: the reward estimator calculates the reward function by using the maximum entropy inverse reinforcement learning according to the state at the last time, the current system action and the new state, and the policy probability corresponding to the current system action, trains against to distinguish the expert trajectory and the policy trajectory, minimizes the KL divergence, obtains the reward estimation value, and transmits it to the policy network; S5: the policy network calculates the target Q value according to the reward estimation value, the state at the last time and the current system action, and the behavior cloning regularization parameter, optimizes the Actor network, generates the updated policy network parameters, and updates the action probability; S6: The policy network generates new system actions based on the action probability and the new state, and then transmits them to the user simulator. It then generates user actions and binary terminal signals and transmits them to the state tracker. S3 is executed until the preset conditions are met.

[0006] During the S5 update process, the behavior cloning regularization parameter is used, as shown in the formula: ,accomplish; In the formula, Represent the objective function; This represents a behavior cloning regularization term; This indicates that the soft actor critic algorithm is used for strategy updates; Represents the regularization coefficient; Indicates hyperparameters; This represents the experience replay pool; Represents a state-action value function.

[0007] The reward estimator described in S4 calculates the reward function using maximum entropy inverse reinforcement learning, based on the previous state, the current system action, the new state, and the policy probability corresponding to the current system action. Adversarial training distinguishes between expert trajectories and policy trajectories, minimizing the KL divergence to obtain the reward estimate, which is derived from the formula: ; ; ,accomplish; In the formula, This represents the reward function value; , These represent the estimated immediate reward and the state value function value, respectively, used to measure the state. Long-term benefits; Representing state Corresponding actions The estimated reward value represents the cumulative discount reward; Indicates the instant reward value; The parameters representing the reward function; Represents the eigenvector; Represents the policy probability, dialogue policy The logarithmic probability represents the logarithmic probability of the current policy for the action.

[0008] S2 describes calculating the probability of an action using the chain rule, derived from the formula: ,accomplish; In the formula, Indicates the probability of an action; Represents the first step in the user action sequence. One action; Indicate the user's dialogue goal; Represents historical dialogue text; representing a dialogue turn.

[0009] The state tracker captures keywords according to the current system action, user action and the last time state, and generates a new state, specifically: The state tracker encodes the current system action, user action and the last time state, and obtains an aggregated sequence after processing by a feedforward layer, and obtains a probability distribution of each keyword after classification and state prediction, and decodes to generate a new state.

[0010] The target Q value is calculated, the Actor network is optimized, and the updated policy network parameters are generated. The Q network is updated, the value function is updated, and the policy network is updated in sequence, and the policy network update also includes adjustment of the regularization coefficient.

[0011] The preset condition is a preset turn or a binary terminal signal of 1.

[0012] The application also provides a task-based dialogue policy learning system based on reinforcement learning, comprising: A preprocessing module: obtaining user dialogue data, decomposing user dialogue targets into constraint sets and request sets, and constructing structured state-action pairs; A user simulator: constructing a user state according to historical dialogue text and user dialogue targets, and initializing the user simulator based on an agenda or a variational hierarchical model; The action probability is calculated by the chain rule, the user action corresponding to the state and the binary terminal signal are generated, and are transmitted to the state tracker; The state tracker: the state tracker captures keywords according to the current system action, user action and the last time state, and generates a new state, and transmits it to the policy network; The reward estimator: the reward estimator calculates the reward function by using maximum entropy inverse reinforcement learning according to the last time state, current system action and new state, and the policy probability corresponding to the current system action, and trains against to distinguish expert trajectories and policy trajectories, minimizes the KL divergence, and obtains the reward estimation value, and transmits it to the policy network; The policy update module: the policy network calculates the target Q value according to the reward estimation value, the last time state and the current system action, and the behavior cloning regularization parameter, optimizes the Actor network, generates the updated policy network parameters, and updates the action probability; The cycle interaction module: the policy network generates a new system action according to the action probability and the new state, and transmits it to the user simulator to generate user actions and binary terminal signals, and transmits them to the state tracker and executes the module operation until the preset condition is reached.

[0013] Advantages The present application adopts a soft actor critic algorithm combined with behavior cloning and maximum entropy inverse reinforcement learning to alleviate the cold start problem; wherein, the maximum entropy inverse reinforcement learning is used to calculate the reward value, and the user target in the dialogue is accurately inferred according to the maximization of the log-likelihood estimate of the human dialogue and the simulated dialogue; feedback is provided for the reinforcement learning algorithm by comparing the user simulator data and the real user target data, guiding the agent to learn the behavior consistent with the expert example, and then guiding the dialogue strategy optimization; and the potential reward function is extracted from the successful trajectory by inverse reinforcement learning to replace the artificial design reward, so that not only the maximization of cumulative reward is pursued when optimizing the strategy, but also the diversity and exploratory of the strategy are encouraged, avoiding the premature convergence of the dialogue strategy to a single mode, which effectively improves the response ability of the dialogue system in complex dialogue scenarios, enhances the generalization ability of the system, and significantly improves the learning efficiency of the dialogue strategy. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 The figure is a comparison chart of various indexes at Epoch = 10 and Epoch = 20.

[0015] Figure 2 The figure is a comparison chart of multi-domain task indexes.

[0016] Figure 3 The figure is an ablation experiment chart. DETAILED DESCRIPTION

[0017] The following examples are intended to illustrate the present application, but not to further limit the present application.

[0018] The present application provides a task-oriented dialogue strategy learning method based on reinforcement learning, comprising: S1: obtaining user dialogue data, decomposing user dialogue targets into a constraint set and a request set, and constructing a structured state-action pair.

[0019] S2: after constructing the user state according to the historical dialogue text and the user dialogue target, initializing the user simulator based on the agenda or the variational hierarchical model; The action probability is calculated by the chain rule, the user action corresponding to the state and the binary terminal signal are generated, and are transmitted to the state tracker.

[0020] In the task-oriented dialogue system, when applying reinforcement learning to learn the dialogue strategy, the system needs a real environment that can interact with the user. However, it is very expensive and time-consuming to apply real users to interact with the reinforcement learning agent. The huge training data cannot be modified with sufficient real data, so the present application uses a user simulator to simulate the interaction between real users and the system.

[0021] The present application is mainly based on an agenda-based user simulator, in addition, a data-driven neural model named variational hierarchical user simulator (VHUS) is used. It is a sequence-to-sequence (Seq2Seq) model, namely an encoder-decoder structure, the core task of the encoder is to convert the input sequence into a fixed-length context vector, the model uses a recurrent neural network (RNN) to encode the initial user goal and system interaction into a fixed-length representation.

[0022] The user simulator converts the stack-like historical dialogue text into a dialogue goal and encodes it into a user dialogue state pair , and then inputs it into the action selection model. The dialogue goal is composed of a request set and a constraint set , for example, the user goal may ask if there is a cheap restaurant in the south of the city, and the corresponding constraint set includes elements and elements , and the request set is the content of the user asking for the restaurant, i.e. the number of diners and whether there is a parking space . Correspondingly is shown in the following formula: .

[0023] The action selection is updated according to the chain rule, preferably realized by the formula: wherein represents the action probability; represents the th action in the user action sequence; represents the user dialogue goal; represents the historical dialogue text; represents the dialogue turn.

[0024] When new user behaviors are pushed onto the agenda, irrelevant behaviors are deleted. If the dialogue agent provides all the requests and correctly books the entity, the user will successfully terminate the dialogue session.

[0025] If the number of dialogues reaches the specified maximum number of dialogues, the dialogue session will be considered a failure. The decoder generates a new output sequence step by step according to the context vector output by the encoder and the previously generated output sequence, and the predicted output is shown in the following formula: ; wherein is the dialogue state information, and ​These are the decoder's weights and bias vectors, respectively, and the probability distribution for generating the next word is calculated using the SoftMax function; It is the decoder context vector.

[0026] S3: The state tracker captures keywords based on the current system action, user action, and the state at the previous moment, generates a new state, and passes it to the policy network.

[0027] The Dialogue State Tracker (DST) is primarily responsible for continuously recording and updating key information such as the user's goals and intentions during the conversation. This process integrates various fragmented pieces of information expressed by the user, transforming them into structured data that the system can understand and utilize. For example, if a user mentions "wanting to travel to a seaside city on a tight budget," the dialogue state tracker will capture relevant slot values ​​such as "destination" and "budget" and pass this information to the system's decision-making module. Based on this, the system recommends seaside cities that fit the user's budget, such as Yangjiang and Beihai, achieving efficient and user-relevant interaction.

[0028] In research on multi-domain dialogue, placeholders in the model are used to represent domain-slot pairs. Each time the model runs, a domain-slot pair is inserted into the placeholder; the model runs multiple times until all types of domain-slot pairs are covered. The Dialogue State Tracker (DST), within the standard TDS pipeline, is responsible for tracking information generated during user-dialogue agent interactions to update the dialogue state. It is a key component of multi-domain dialogue systems. The DST maintains the user's intent state by recording a set of slot-value pairs, slots containing information about user constraints, and requestable slots indicating the user's desired queries. In this invention, the core role of the DST is to track system actions and system states in the data and transform them into action and session states.

[0029] The state is that the DST is at time step Generation, Build Status The structured representation includes embedded results of entities returned by external database queries, the previous user action, the previous round of system actions, and the state of all beliefs in the DST. Actions are abstract representations of conversational behaviors through intents. The dialogue response problem is viewed as a multi-label prediction task. Dialogue behaviors in the dialogue policy are non-lexical, such as "taxi notification destination," and multiple atomic dialogue actions can occur and combine simultaneously in the same round of dialogue. Each action is composed of the domain name of the operation space, the operation type, and the slot name. Therefore, slot values ​​need to be replaced with count placeholders and the actual values ​​filled with entities selected from the external database.

[0030] In complex multi-turn dialogue scenarios, the importance of the dialogue state tracker is even more prominent. As the number of dialogue turns increases, information accumulates and changes, and it can effectively integrate contextual information to avoid information loss or misunderstanding. This enables the system to "remember" previous discussions and better understand the user's coherent intent, thereby providing consistent and accurate service in multi-turn interactions.

[0031] To improve the performance of the dialogue state tracker to adapt to more complex and diverse dialogue scenarios, the present invention regards the dialogue state as a clear fixed-size memory, and proposes a selective coverage mechanism to decompose the dialogue state tracking task into state operation prediction and slot value generation two sub-tasks, in this way to improve the efficiency and performance of dialogue state tracking, to realize more efficient dialogue state tracking.

[0032] Preferably, the state tracker according to the current system action, user action and last time state, captures keywords, generates a new state, specifically: After the state tracker encodes the current system action, user action and last time state, processes the aggregated sequence through the feedforward layer, classifies, obtains the probability distribution of each keyword and performs state prediction, decodes to generate a new state.

[0033] For example: After encoding the current system action, user action and last time state, processing the aggregated sequence through the feedforward layer, classifying, obtaining the probability distribution of each slot operation and performing state prediction, decoding to generate slot values, determining the specific value of each slot, obtaining a new state.

[0034] S4: The reward estimator calculates the reward function using maximum entropy inverse reinforcement learning based on the last time state, current system action and new state, and the policy probability corresponding to the current system action, and trains against to distinguish expert trajectories and policy trajectories, minimizes the KL divergence, obtains the reward value, and passes it to the policy network.

[0035] When the reinforcement learning algorithm of the dialogue policy interacts with the user simulator, the present invention constructs a reward estimator based on maximum entropy inverse reinforcement learning as an evaluation index to infer the user's goal in the dialogue. The process is briefly summarized as follows: first, compare the data given by the user simulator with the real user goal data After that, the feedback result will further train the reinforcement learning algorithm. In this process, the basis for comparison is the maximum log-likelihood estimation of human dialogue and simulated dialogue.

[0036] The specific process is as follows: After the model transmits the state-action pair , the reward estimator calculates the reward value from the initial state​ Start, follow the strategy , get a trajectory . Since the reward is sparse due to the complexity of the conversation, the reward estimator of the present application distinguishes the real human conversation from the generated conversation according to the artificial intelligence (AL) method from the dialogue strategy, which minimizes the KL divergence of the experience distribution while maximizing the KL divergence of the strategy distribution, so that the loss function of the reward estimator is more consistent with the historical experience, reduces the gap between the model and the reality, avoids overfitting or unnecessary bias, and helps to improve the stability and convergence of learning.

[0037] The reward function can make the trajectory distribution generated according to the strategy As close as possible to the expert trajectory distribution , a generative adversarial imitation learning (GAIL) framework is introduced to distinguish the expert trajectory from the strategy generated trajectory through the discriminator.

[0038] Preferably, the reward estimator of S4 calculates the reward function according to the state of the last moment, the current system action and the new state, and the strategy probability corresponding to the current system action, uses the maximum entropy inverse reinforcement learning, and obtains the reward estimation value by minimizing the KL divergence and distinguishing the expert trajectory from the strategy trajectory through adversarial training, which is: , realize In the formula, Represents the value of the reward function; , Represents the immediate reward estimation value, the state value function value, respectively, for measuring the long-term benefits of the state ; Represents the reward estimation value of the state Corresponding to the action , representing the cumulative discounted reward; Represents the immediate reward value; Represents the parameter of the reward function; Represents the feature vector; Represents the strategy probability, the logarithmic probability of the dialogue strategy , indicating the logarithmic probability of the action under the current strategy.

[0039] The present application converts the reward estimator into a reward approximator And a shaping term To recover the interpretable and robust reward from real human dialogue; the strategy Is then updated by maximizing , so as to indirectly optimize the consistency with the expert behavior. This design solves the difficulty of explicit modeling of the reward function in traditional inverse reinforcement learning.​​

[0040] S5: The policy network calculates the target Q value according to the reward estimation value, the state at the last moment and the current system action, and the behavior cloning regularization parameter, optimizes the actor network, generates updated policy network parameters, and updates the action probability.

[0041] In response to high complexity tasks, the present application selects to use a soft actor critic algorithm (Soft actor critic Policy, SAC). The SAC algorithm can provide sample stability, which helps to overcome the shortcomings of other algorithms in terms of sample efficiency, stability and hyperparameter sensitivity, and provides a more effective way to solve complex tasks such as dialogue strategy optimization.

[0042] Preferably, S5 sequentially performs Q network updating, value function updating, and policy network updating, and the policy network updating further includes adjustment of the regularization coefficient.

[0043] In the dialogue strategy SAC algorithm, there is an actor network and four critic networks, and the goal is to maximize the sum of the expected reward of the policy and the policy entropy.

[0044] When the input state is input, the system outputs the action policy . The action policy uses a double Q network, that is, two independent Q networks and are used to reduce the bias of the value function estimation, and the updating method is as follows: First, the Q function is updated, and the goal of the Q function is to estimate the long-term return of the state-action pair, which is updated by the following Bellman equation as shown in the following formula 1: ; Wherein, is the current Q function estimate. is the immediate reward value of the reward estimator. is the reward estimation value, representing the cumulative discounted reward. is the regularization coefficient of the target action selection.

[0045] Then, the Q function loss is calculated according to the following formulas 2-4: , formula 2; , formula 3; , formula 4; is the value of the target Q network, is the target policy network. By minimizing these two loss functions, the and parameters of the network.

[0046] SAC also uses a value function to approximate the function. The goal is to make the value function approximate the optimal value of the function, as shown in the following formula 5: ; Finally, the policy is updated, further, in order to make the SAC algorithm more adaptive, optimize the extrapolation error, therefore, the application adds a behavior cloning regularization term in the standard policy update step of the algorithm, in order to push the policy to the operation contained in the data set .

[0047] Preferably, the policy is updated by using the behavior cloning regularization term, which is realized by formula 6: , wherein, represents the target function; represents the behavior cloning regularization term; represents the policy update using the soft actor critic algorithm; represents the regularization coefficient; represents the hyperparameter; represents the experience replay pool; represents the state-action value function.

[0048] The newly added behavior cloning regularization term of the application prompts the policy to select the corresponding action in the data set as much as possible when encountering the state in the data set, thereby reducing the extrapolation error and enhancing the adaptability of the algorithm.

[0049] wherein, the regularization coefficient is automatically adjusted, the application minimizes the loss function , and the regularization coefficient dynamically adjusts the balance between policy entropy and reward maximization, so that the entropy of the policy is close to the target entropy. The update target of the regularization coefficient is shown in formulas 7-9: formula 7; formula 8; formula 9; wherein, is the learning rate, is the target entropy, which is set to-dim( ), is the policy entropy. The regularization coefficient is updated by gradient descent , ensure that the policy entropy always approximates the target entropy, so as to realize stable and adaptive exploration-exploitation trade-off.

[0050] The code of the above updating process is shown in Table 1.

[0051] Table 1: Pseudo code of policy updating algorithm S6: The policy network generates a new system action according to the action probability and the new state, and passes it to the user simulator, then generates a user action and a binary terminal signal, and passes them to the state tracker; S3 is executed until a preset condition is reached.

[0052] In addition, the preset condition can be a preset round or a binary terminal signal being 1.

[0053] The soft actor critic algorithm is combined with behavior cloning and maximum entropy inverse reinforcement learning to alleviate the cold start problem; wherein, the maximum entropy inverse reinforcement learning is used to calculate the reward value, and the maximum log-likelihood estimation of human dialogue and simulated dialogue is used to accurately infer the user goal in the dialogue. By comparing the user simulator data and the real user goal data, feedback is provided for the reinforcement learning algorithm to guide the agent to learn the behavior consistent with the expert example, and then guide the dialogue strategy optimization; and the potential reward function is extracted from the successful trajectory by inverse reinforcement learning to replace the artificial design reward, so that not only the maximization of cumulative reward is pursued, but also the diversity and exploration of the strategy are encouraged, avoiding the dialogue strategy converging to a single mode too early. This method effectively improves the response ability of the dialogue system in complex dialogue scenarios, enhances the generalization ability of the system, and significantly improves the learning efficiency of the dialogue strategy.

[0054] The application also provides a task-based dialogue strategy learning system based on reinforcement learning, comprising: A preprocessing module: obtaining user dialogue data, decomposing user dialogue goals into constraint sets and request sets, and constructing structured state-action pairs; A user simulator: after constructing the user state according to the historical dialogue text and the user dialogue goal, initializing the user simulator based on the agenda or variational hierarchical model; An action probability is calculated by the chain rule, a user action corresponding to the state and a binary terminal signal are generated, and are passed to the state tracker; A state tracker: the state tracker captures keywords according to the current system action, the user action and the state at the last time, generates a new state, and passes it to the policy network; Reward estimator: The reward estimator calculates the reward function using maximum entropy inverse reinforcement learning based on the previous state, the current system action, the new state, and the policy probability corresponding to the current system action. It also uses adversarial training to distinguish between expert trajectories and policy trajectories, minimizes the KL divergence, obtains the reward estimate, and passes it to the policy network. Policy update module: The policy network calculates the target Q value based on the reward estimate, the previous state, the current system action, and the behavior cloning regularization parameters, optimizes the Actor network, generates updated policy network parameters, and updates the action probabilities. Looping Interaction Module: The policy network generates new system actions based on action probabilities and new states, and then transmits them to the user simulator. This generates user actions and binary terminal signals, which are then transmitted to the state tracker to execute the module's operations until the preset conditions are met.

[0055] Therefore, this invention proposes a system that mainly includes a reward estimator constructed using inverse reinforcement learning, a dialogue policy guided by reinforcement learning, a state tracker, a user simulator, and a dataset. The specific implementation process is as follows: firstly, the collected dialogue information... State-action pairs, as the environment, are decomposed into the elements required for reinforcement learning. and And provide feedback, user simulator Based on user conversation status Send a response ,in This represents a binary termination signal, indicating whether the user wishes to end the conversation. Then the corresponding... and The Dialogue State Tracker (DST) acts as a recorder, intermediary, and decision-maker, transmitting information and data. Dialogue Strategies Next, dialogue strategy According to the reward estimator The REF value was compared and evaluated with samples of real dialogues from the corpus. (Ref 193817638 \r \h \* MERGEFORMAT)

[91] Feedback actions to the conversation tracker Interact with the user simulator to obtain the next state. Finally, the dialogue strategy is updated based on the reward value obtained.

[0056] Experimental setup 1. Data and Evaluation Experiment In previous related research, researchers often choose MultiWoz1.0 dataset to carry out experiments. However, with the progress of the times and the complexity of research content, this dataset has certain limitations in data size and coverage.

[0057] In order to break through these limitations and improve the reliability and comprehensiveness of the evaluation experiment, the present application uses MultiWoz2.0 dataset for experiment. Compared with MultiWoz1.0 dataset, MultiWoz2.0 dataset has been significantly improved in data size and coverage, and it is more extensive and comprehensive. MultiWoz2.0 adds a large number of dialogue samples, which can learn more rich language patterns and dialogue scenarios; in terms of coverage, MultiWoz2.0 further expands the domain category and dialogue theme, covering more application scenarios in real life, which helps to better adapt to the complex and variable real dialogue environment. In the data preprocessing stage, the present application divides the training set 70%, the validation set 15% and the test set 15%, ensuring consistent domain distribution. The detailed comparison information is shown in Table 2: Table 2 Comparison of MultiWoz1.0 dataset and MultiWoz2.0 dataset In this experiment, the dialogue strategy and the value function are both implemented by two hidden layer MLPs. The reward estimator is divided into two networks according to the proposed algorithm and , wherein each network is a hidden layer MLP. The activation function of the hidden layer MLP is Relu function, and Adam is used as the optimization algorithm.

[0058] The evaluation of task-oriented dialogue mainly includes the following aspects: (1) success rate : the success rate of completing the dialogue in the given test multi-turn dialogue is calculated, and the number of dialogue tasks successfully completed is used to calculate the proportion of the total number of given dialogues, the formula is as follows: ; wherein represents the number of successfully completed samples, and represents the total number of given samples.

[0059] (2) : whether all requested information (such as the address and phone number of the hotel) has been informed is evaluated. The score is a comprehensive embodiment of accuracy ( ) and recall ( ). The specific formula is as follows: ; ; In the above formula, TP represents True Positive, FP represents False Positive, FN represents False Negative.

[0060] (3) Matching rate : is used to estimate whether the dialogue system has successfully identified and processed all relevant reservation entities, and can estimate whether the reservation entity matches all instructions. A higher matching rate means that the system can effectively identify and process key information in user input. The constraint is that only when all information has been provided (F1 = 1) and the entity has been correctly reserved (matching rate = 1) to all domains, the dialogue is considered successful. The high performance and low computational complexity of this algorithm are important indicators for evaluating the proposed method. The number of training rounds required to achieve the highest success rate represents the performance of the algorithm in terms of time.

[0061] (4) Average number of dialogue rounds : is the maximum number of rounds of human-computer interaction. When it is very obvious to deviate from the overall purpose of the dialogue, fall into an infinite general sentence reply cycle, and the task is successfully completed, the dialogue state can be determined to be terminated. The dialogue success rate represents the proportion of the number of successfully completed tasks in the overall number of dialogues. The dialogue success rate is the core indicator for evaluating the pros and cons of dialogue strategy algorithms. When comparing the pros and cons of algorithms, first compare the dialogue success rate. When the dialogue success rates of two algorithms are close, the average number of dialogue rounds can be compared. The smaller the average number of rounds, the better the algorithm.

[0062] 2. Experimental parameter design This experiment uses Python 3.8 version and Python model framework, and uses specific parameters as shown in Table 3: Table 3 Experimental parameters 3. Baseline comparison experiment The present application selects the following 5 baselines for comparison experiments, and the corresponding model is introduced as follows: ACER (Actor-Critic RL policy with Experience Replay): Actor-Critic RL policy with Experience Replay is a confidence domain policy optimization method that can be trained using non-strategy, uses truncated importance sampling to control variance, and adapts well to discrete action space.

[0063] PPO (Proximal Policy Optimization) is a proximal policy optimization algorithm that interacts directly with the user simulator in the experimental setup and learns the dialogue policy using the obtained trajectory segments.

[0064] ALDM (Adversarial Learning Dialogue Model) uses Bi-LSTM to encode dialogue sequences as discriminators to learn dialogue rewards.

[0065] GAIL (Generative Adversarial Imitation Learning): combines the GAN idea of inverse reinforcement learning method with the direct learning strategy of agent, effectively solving the problems of high computational cost and low learning efficiency of inverse reinforcement learning.

[0066] BC (Behavior Cloning) is a dialogue model that uses behavior cloning learning to learn dialogue strategies by extracting state-action pairs to construct new sets and imitating the actions of the set.

[0067] In this experiment, two forms of user simulators based on agenda (Agenda) and neural network user simulators (VHUS) were used for testing, with Epoch = 10 and 20. In the experiment, in order to verify the effect of the dialogue strategy model, a user simulator was introduced to simulate the interaction between the real user and the system, and the number of experiments was used to compensate for the difference between the simulator and the real user. Among them, the comparison of experiments at different Epoch is shown in Table 4, Figure 1 As shown in Table 4, SDPL-bc represents the model without the behavior cloning item, SDPL represents the model using the method of the present application, and Human represents the interaction of artificial participation, which is the real interaction data collected in the actual experiment.

[0068] Table 4 Interaction results of agenda-based user simulator From the interaction experiment results of the agenda-based user simulator, it can be seen that the two agents related to SDPL (SDPL-bc and SDPL) perform well. In the Inform index, both Epoch 10 and 20 maintain a high value, indicating stable performance in information provision capability. SDPL-bc also maintains a high level of Success index in the two experiments, showing that its dialogue strategy has strong effectiveness.

[0069] Compared with SDPL-bc, SDPL has a slight gap in the Turns index, but does not affect the overall advantage of SDPL. SDPL does not reach the level of other baselines in Success, but compared with some algorithms, SDPL still shows obvious advantages in most key indicators. Although the Success index fluctuates, considering multiple indicators, SDPL still shows strong overall strength and adaptability in complex task execution and dialogue interaction process.

[0070] Compared with other agents, the SDPL series of agents can better balance matching degree and success rate while improving interaction efficiency, which shows that the design concept and training method of SDPL have certain advantages in multi-domain task dialogue systems. However, there is still a gap compared with human data, which can be further optimized in the future to approach the level of human interaction.

[0071] The present application also realizes interaction with each baseline through a neural network simulator (VHUS), and the experimental results are shown in Table 5.

[0072] Table 5: Interaction results of neural network user simulator From the VHUS experimental result table, it can be observed that different methods show diverse performance under different training rounds: at training rounds Epoch=10 and Epoch=20, as the training rounds increase, the dialogue rounds (Turns) decrease, and the success rate (Success) also decreases. The method of the present application (SDPL) is higher than other comparative methods in the Success index at Epoch=20, indicating that the method has more advantages in achieving the target. The method of the present application is also better than other methods in the Match index, indicating that it has better effect in matching related information.

[0073] 4. Multi-domain comparison experiment Based on multi-domain tasks, the present application involves gradually increasing the number of task domains for experiments. Four models are respectively experimented in five domains of [restaurant Document, hotel Hotel, taxi Taxi, travel travel, and shopping Shopping]. In the experiment, the domain name is replaced by the first letter for observation. The experimental results are shown in Table 6. Figure 2 As can be clearly seen from Table 6, when the model is experimented in multiple domains (D+Hotel represents two task domains of restaurant and hotel, DH+Taxi represents three task domains of restaurant, hotel and taxi, DHT+travel represents four task domains of restaurant, hotel, taxi and travel, and DHTT+Shopping represents five task domains of restaurant, hotel, taxi, travel and shopping), SDPL has higher success rate than other methods, which shows that the method of the present application has better adaptability in multi-domain task dialogue system. Rate Match rate Rates are generally stable. However, with the increase in the number of domains, PPO and ACER experience a sharp decline in performance. ALDM gradually stabilizes in some aspects with the increase in domains, but its performance is still inferior to SDPL. In conclusion, overall, SDPL offers better adaptability and stability.

[0074] 5. Ablation test This application employs a supervised learning pre-training strategy, using the cross-entropy loss function on the MultiWoz 2.0 dataset for ablation experiments to determine whether behavioral cloning needs to be introduced into the experiments. The experimental results are shown in the double Y-curve graph. Figure 3 As shown. Compared with methods with manually designed reward functions, the matching rate of SDPL-bc is... The accuracy rate was 24%, lower than SDPL's 33%. Behavioral cloning, initialized through a pre-trained strategy, constrains the action space exploration direction, thereby improving the matching accuracy of target entities. Adding behavioral cloning increases the algorithm's matching rate and success rate. Therefore, the conclusion drawn from the figure is that adding behavioral cloning to the dialogue strategy helps improve the dialogue matching rate.

Claims

1. A task-oriented dialogue policy learning method based on reinforcement learning, characterized in that, The method comprises the following steps of: S1: obtaining user dialogue data, decomposing a user dialogue target into a constraint set and a request set, and constructing a structured state-action pair; S2: constructing a user state according to historical dialogue text and the user dialogue target, and initializing a user simulator based on an agenda or a variational hierarchical model; calculating an action probability through a chain rule, generating a user action and a binary terminal signal corresponding to the state, and delivering them to a state tracker; S3: the state tracker captures keywords according to a current system action, a user action and a previous state, generates a new state, and delivers it to a policy network; S4: a reward estimator calculates a reward function by using maximum entropy inverse reinforcement learning according to the previous state, the current system action and the new state, and a strategy probability corresponding to the current system action, performs adversarial training to distinguish between an expert trajectory and a strategy trajectory, minimizes KL divergence, obtains a reward estimation value, and delivers it to the policy network; S5: the policy network calculates a target Q value according to the reward estimation value, the previous state and the current system action, and a behavior cloning regularization parameter, optimizes an Actor network, generates updated policy network parameters, and updates the action probability; S6: the policy network generates a new system action according to the action probability and the new state, delivers it to the user simulator, generates a user action and a binary terminal signal, and delivers them to the state tracker; and S3 is performed until a preset condition is reached. 2.The reinforcement learning based task-oriented dialogue policy learning method according to claim 1, characterized in that, In the updating process S5, the behavior cloning regularization parameter is utilized to realize the formula: ; wherein represents the objective function; represents the behavior cloning regularization term; represents the policy update with soft actor critic algorithm; represents the regularization coefficient; represents the hyperparameters; represents the experience replay pool; represents the state-action value function. 3.The reinforcement learning based task-oriented dialogue policy learning method according to claim 1, wherein, The reward estimator of S4 calculates a reward function by using maximum entropy inverse reinforcement learning according to the state of the previous moment, the current system action and the new state, and the policy probability corresponding to the current system action, trains against to distinguish the expert trajectory and the policy trajectory, minimizes the KL divergence, and obtains a reward estimation value, which is represented by the formula: ; ； , implementation; In the formula, represents the reward function value; , respectively represent the immediate reward estimate value, the state value function value, and are used to measure the long-term benefits of the state ; represents the reward estimate value of the state for the action , representing the cumulative discounted reward; represents the immediate reward value; represents the parameter of the reward function; represents the feature vector; represents the policy probability, the logarithmic probability of the dialogue policy , representing the logarithmic probability of the current policy for the action. 4.The reinforcement learning based task-oriented dialogue policy learning method according to claim 1, wherein, S2 said by chain rule to calculate the probability of action, by the formula: , implementation; In the formula, Indicates the probability of an action; Represents the first step in the user action sequence. One action; Indicate the user's dialogue goal; Represents historical dialogue text; Indicates the turn of the dialogue. 5.The reinforcement learning based task-oriented dialogue policy learning method according to claim 1, wherein, The state tracker captures keywords according to a current system action, a user action and a previous state, generates a new state, and specifically comprises the following steps of: After the state tracker encodes the current system action, the user action and the previous state, obtains an aggregated sequence through a feedforward layer, classifies the aggregated sequence, obtains a probability distribution of each keyword, performs state prediction, decodes, and generates a new state. 6.The reinforcement learning based task-oriented dialogue policy learning method according to claim 1, wherein, The method for calculating a target Q value, optimizing an Actor network, and generating updated policy network parameters comprises the following steps of: Q network updating, value function updating and policy network updating are performed in sequence, and the policy network updating further comprises adjusting a regularization coefficient. 7.The reinforcement learning based task-oriented dialogue policy learning method according to claim 1, wherein, The preset condition is a preset number of rounds or a binary terminal signal being 1.

8. A task-oriented dialogue policy learning system based on reinforcement learning, characterized by, The method comprises the following steps of: a preprocessing module: obtaining user dialogue data, decomposing a user dialogue target into a constraint set and a request set, and constructing a structured state-action pair; a user simulator: constructing a user state according to historical dialogue text and the user dialogue target, and initializing the user simulator based on an agenda or a variational hierarchical model; calculating an action probability through a chain rule, generating a user action and a binary terminal signal corresponding to the state, and delivering them to a state tracker; a state tracker: the state tracker captures keywords according to a current system action, a user action and a previous state, generates a new state, and delivers it to a policy network; a reward estimator: the reward estimator calculates a reward function by using maximum entropy inverse reinforcement learning according to the previous state, the current system action and the new state, and a strategy probability corresponding to the current system action, performs adversarial training to distinguish between an expert trajectory and a strategy trajectory, minimizes KL divergence, obtains a reward estimation value, and delivers it to the policy network; Policy update module: the policy network calculates the target Q value according to the reward estimation value, the state at the last moment, the current system action and the behavior cloning regularization parameter, optimizes the actor network, generates the updated policy network parameters, and updates the action probability; Cyclic interaction module: the policy network generates a new system action according to the action probability and the new state, and transmits it to the user simulator to generate a user action and a binary terminal signal, which are transmitted to the state tracker and the module operation is executed until the preset condition is reached.

Images