Systems and methods for improving security policies for task-oriented dialogue

Abstract

The embodiments described herein provide safe policy improvement (SPI) in a batch reinforcement learning framework for task-oriented dialog. Specifically, a batch reinforcement learning framework for dialog policy learning is provided that improves the performance of dialog and learns to form a reward that makes the invention the reason behind human responses, rather than just imitating human demonstrations.

Description

[0001] Inventors: G.S. Ramachandran, Kazuma Hashimoto, Caiming Xiong, and R. Zoher

[0002] Cross-referencing

[0003] The published text claims priority to U.S. non-provisional application No. 17 / 500855, filed October 13, 2021, which is a non-provisional application of U.S. provisional application No. 63 / 148861, filed February 12, 2021, and also claims priority to co-pending and common U.S. non-provisional application No. 17 / 105,262, filed November 25, 2020, which is a non-provisional application of U.S. provisional application No. 63 / 034653, filed June 4, 2020, and claims priority to it under 35 USC 119.

[0004] All of the above applications are explicitly incorporated herein by reference. Technical Field

[0005] The publicly available text generally deals with machine learning models and neural networks, and more specifically, with improvements to security policies for task-oriented dialogues. Background Technology

[0006] Neural networks have been used to generate conversational responses, enabling dialogue with human users to complete tasks. For example, a human user can converse with a smart assistant to book a travel flight, reserve a restaurant, and so on. To accomplish complex tasks, smart assistants often need to learn to collectively complete multiple sub-tasks. For instance, an assistant might need to book a hotel and a flight to allow sufficient commuting time between arrival and hotel check-in. To enable smart assistants to learn such complex tasks, they learn conversational strategies to choose among sub-tasks or options at a given time, which is typically accompanied by a state tracker that monitors the state of the sub-tasks.

[0007] Task-oriented dialogue systems typically learn from offline data collected using human demonstrations (e.g., past conversations), but collecting and annotating diverse demonstrations can be costly. Furthermore, such offline task-oriented dialogue systems often involve different systems, such as belief state trackers, dialogue policy management, response generation, etc. Besides the need for sample efficiency in effective dialogue policy learning, these different systems can introduce stochasticity and its associated challenges.

[0008] Therefore, efficient policy learning is required in task-oriented dialogue systems. Attached Figure Description

[0009] Figure 1AA diagram is provided illustrating an exemplary task-oriented dialogue described by a Markov decision process according to one embodiment described herein, on which security policy improvements can be applied.

[0010] Figure 1B One embodiment described herein is provided. Figure 1A The diagram illustrates an exemplary task-oriented dialogue involving multiple rounds of conversation between a user and an agent.

[0011] Figure 2 A simplified diagram is provided illustrating an exemplary architecture for training a policy network using reward learning according to one embodiment described herein.

[0012] Figure 3A One embodiment described herein is provided. Figure 2 The diagram shows a simplified representation of various aspects of the workflow within the reward learning module.

[0013] Figure 3B One embodiment described herein is provided. Figure 2 The diagram shows a simplified representation of the network architecture of the reward learning module.

[0014] Figure 4 This is a simplified diagram of a computing device for implementing security policy improvements and reward learning for task-oriented dialogue, based on some implementation schemes.

[0015] Figures 5A to 5B An exemplary logic flowchart illustrating a method for improving security policies based on MDP according to an implementation scheme is provided.

[0016] Figure 6A An exemplary pseudocode snippet illustrating an algorithm for Causal Aware Security Policy Improvement (CASPI) according to an implementation described herein is provided.

[0017] Figures 6B to 6C Provided are embodiments for use according to the description herein. Figure 6A An exemplary logic flowchart of the CASPI algorithm method is shown.

[0018] Figure 7 This is a simplified block diagram illustrating a hybrid human-machine collaboration and automated evaluation metric score for pairwise reward learning according to the implementation scheme described herein.

[0019] Figures 8 to 16 Data charts are provided to illustrate an example performance comparison between security policy improvements based on one implementation scheme and existing methods.

[0020] In the accompanying drawings and appendices, elements with the same name have the same or similar functions. Detailed Implementation

[0021] Task-oriented dialogue systems typically learn from offline data collected using human demonstrations (e.g., past conversations), but collecting and annotating diverse demonstrations can be costly. Furthermore, such offline task-oriented dialogue systems often involve different systems, such as belief state trackers, dialogue policy management, response generation, etc. Besides the need for sample efficiency in effective dialogue policy learning, these different systems can introduce stochasticity and its associated challenges.

[0022] Some existing systems employ non-policy-based reinforcement learning (Batch-RL) methods to solve complex tasks. Batch-RL methods typically use historical annotated data rather than simulators, which can be sampling-efficient because inexpensive simulators can often easily sample policy data. However, these techniques may not be so efficient to implement due to the nature of dialogue policy learning. For example, non-policy-based learning may often require estimating the behavioral policy for a given state (e.g., belief state) of the underlying Markov decision process (MDP). In real life, belief states do not always capture the true state of the MDP, and the latent states of the MDP (e.g., prosody, etc.) may lead to randomness in the agent's response in each round. Furthermore, semantic information may be lost when generating dialogue actions from natural language text. If each component of a complex action is given equal attention, then simply using policy imitation for dialogue actions may not be sufficient to provide fair reasoning for a particular outcome.

[0023] Given the need for efficient policy learning in task-oriented dialogue systems, the implementation described in this paper provides safe policy improvements within a batch reinforcement learning framework for task-oriented dialogue. Specifically, a dialogue policy is trained on dialogue outflows generated from a potential behavior policy with performance guarantees, for example, through reinforcement, where the new policy outperforms the old behavior policy by at least a positive margin. The training loss objective is then defined by minimizing the expected discount of future rewards, provided that the KL divergence between the old behavior policy and the target policy is no greater than a predefined hyperparameter. In this way, training bias on another policy outflow can be significantly reduced, resulting in "safe" policy improvements.

[0024] Furthermore, pairwise causal reward learning is provided to form rewards that infer the intent of human utterances, rather than mimicking human demonstrations in a batch reinforcement setting. The combination of improved security policies and pairwise causal reward learning can achieve sample efficiency when learning complex tasks.

[0025] As used herein, the term “network” can include any hardware- or software-based framework, including any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented thereon or with it.

[0026] As used herein, the term "module" can include a hardware- or software-based framework that performs one or more functions. In some implementations, the module may be implemented on one or more neural networks.

[0027] Figure 1A A diagram 100 is provided illustrating an exemplary task-oriented dialogue described by a Markov decision process according to one embodiment described herein, on which security policy improvements can be applied. Diagram 100 illustrates a dialogue round of a task-oriented dialogue between a user and an intelligent agent 120. For example, the user could provide the user utterance 101 “Book me a flight to London,” and the agent 120 could respond with the system response “When do you want to leave?” 102. The dialogue between the user and the intelligent agent 120 can form a task-oriented dialogue to complete the task of planning a trip to London.

[0028] Task-oriented dialogue can be modeled as a Markov Decision Process (MDP), illustrated by the connected graph structure 110. The MDP is described by tuples {S, A, P, R, γ} consisting of states S, actions A, transition probabilities P, rewards R, and discount factors γ. State S represents the dialogue context, which is the agent's interpretation of the environment. Action A is the possible communication behavior available to the agent in each state. The transition probability P defines the probability that state S transitions to another set of states S' given action A. For example, given state S at time step t... t The intelligent agent 120 can formulate target strategies based on the environment. e (a) t |s t ) Perform compound action a t This has a transition probability to the next state P(S' / S, A). For example, in state 105 s1 after user utterance 101, the departure city (for each user's location) is confirmed, and the destination city "London" is obtained from user utterance 101, but the departure date and departure time are unknown. Therefore, based on the target policy π... e (a2|s1) Execute dialogue action 106 to request information about the departure date, where agent 120 replies to the user with system response 102. After dialogue action 106, the dialogue state transitions from state s1 to s2.

[0029] A latent reward function R(a,s) with a discount factor γ∈[0,1] is associated with the MDP, defining the reward value for a given state and action set. For example, given state s1 and dialogue action a1, a positive reward r115 of "20" is assigned. In one implementation, the latent reward function R(a,s) and the discount factor γ can be predefined for the MDP. In another implementation, the latent reward function R(a,s) and the discount factor γ can be learned by referring to the pairwise causal reward learning mechanism described in Figure 3.

[0030] In one implementation, given a reward function and a discount factor, the objective is to optimize the target policy π. e (a) t |s t It maximizes the expected discount sum of future rewards on the MDP, and can be written as a state-action function Q. πe (a t |s t ) = , where r(s) t’ ,a t’ Let be the future reward at future time t', which can be similarly defined using the reward function R(a,s). To achieve this, refer to... Figure 2 Figure 5 illustrates the "safe" strategy improvement mechanism.

[0031] Figure 1B One embodiment described herein is provided. Figure 1A The diagram illustrates an exemplary task-oriented dialogue involving multiple rounds of conversation between a user and an agent. Figure 1B The dialogue shown corresponds to objective 122, for example, related to booking a train departing at a specific time to a specific destination. The dialogue comprises four rounds, each including delexified user utterances 125a-d, proxy dialogue actions 126a-d, and delexified proxy utterances / responses 127a-d. These four rounds may suggest that simply using policy imitation for dialogue-action may not be sufficient for reasoning about outcomes; rather, it requires equal attention to each component of the complex action. For example, rounds 3 and 4 are rich in semantic information: round 3 provides the key to the booking process, while round 4 is used least in the success of the conversation and is given equal weight to the other semantically rich rounds. These details are lost in imitation policy learning.

[0032] Figure 2A simplified illustration 200 is provided illustrating an exemplary architecture of a policy network trained with reward learning according to one embodiment described herein. Illustration 200 shows a training dataset 210 being input to a policy network 220 and a reward learning module 260. Specifically, dataset 210 includes multiple launches 212a-n from dialogue. These launches 212a-n can be generated by a human agent performing actions based on a latent stochastic behavior policy.

[0033] For example, in offline batch-RL, intelligent agents cannot interact with the environment. Instead, it is possible to obtain intelligence based on a policy π with latent random behavior. b Offline dataset D210 records human agents performing actions. Offline dataset D210 includes multiple launches 212a-n of a dialogue, each launch consisting of τ i ∈D represents. Each output τ i = Each ot is an observation at round t, determined by o. t = Composition. Here, b t It is the belief state of the agent at turn t. and These are the user and agent utterances at times t and t-1, respectively. Therefore, batch-RL requires training an exit strategy generated from the underlying behavioral policy.

[0034] However, directly optimizing the training objective, such as a discount to future rewards, when introducing a new strategy can lead to large biases in the value function estimation, poor generalization features, and low sample efficiency. Therefore, a "safe" policy improvement can be implemented, ensuring that the performance of the new strategy is bounded compared to the old one. Specifically, the new objective policy π... e Value function and potential behavioral strategies π b The value function satisfies: Pr(V) πe ≥V πb -ζ) ≥ 1-δ, where V πe and V πb These are the value functions of the target policy and the behavioral policy, respectively. Here, 1-δ and ζ are the high-probability and approximate meta-parameters, respectively.

[0035] Therefore, based on the input observations from dataset 210 t =(b t , , The policy network 220 can be based on the target policy π e The target action distribution π is generated by the parameters θ of the policy network. e (s) t Then, the stochastic loss target L for security policy improvement can be calculated at loss module 230.sto (θ):

[0036] (1)

[0037] In some implementations, belief state b can be used. t Replace s in equation (1) t To calculate the random loss target L sto (θ). The belief state is a random variable because it cannot capture all information. Calculate the policy π. e (b) t ;θ), to optimize the random loss function.

[0038] Traditionally, the update mechanism provided in Schulman et al.'s "Trust Region Policy Optimization," Proceedings of the International Conference on Machine Learning, pp. 1889-1897, 2015, provides a bounded error, provided that the constraint (1) is satisfied, where D KL (. / / .) is the KL divergence, and η is a hyperparameter. However, the Schulman update rule requires access to the behavior policy π. b (a) t |s t This is difficult to estimate. Conversely, in the state of belief b... t π b (b) t The behavioral policy conditional on () can be relative to s in (1). t To estimate, this leads to a stochastic behavioral strategy. Belief state b t It is the observation at turn t. t A portion of it can be obtained from a specific derivation in dataset D210. Therefore, in one implementation, when calculating the random loss target in (1), π b (s) t ) can be derived from π b (b) t (Approximately, π) b (b) t (This can be obtained from the derivation in dataset 210. For example, it can be obtained by considering a given b.) t Dialogue action a t The number of occurrences divided by the number of occurrences in a given b t action a t The total number of π is given. b (b) t (estimated).

[0039] Based on observation t The availability of more evidence (which contains more than belief state b) t(More information) The pattern of the policy can collapse into near-deterministic actions. To incorporate this factor into policy learning, an additional deterministic loss can be computed at loss module 240:

[0040] (2)

[0041] in, It is the discounted sum of future rewards for a single trajectory / event, for example, starting from time step t, referred to as having a target g. i The launch of τ i The discount factor is a function of parameter θ1; given parameter θ2, R θ2 (g, s, a) is the reward function for the state, action, and goal. R is learned from the reward learning module 260. θ2 (g, s, a) and discount factor Therefore, the combined loss module 250 calculates the policy optimization loss function as follows:

[0042] L(θ) = L sto (θ)+L det (θ) (3)

[0043] In one implementation, only random loss L can be used. sto (θ) or using only deterministic loss L det Network 220 is trained using (θ). Alternatively, as described below, network 220 is trained using the sum of two losses, L(θ).

[0044] In one implementation, the combined loss module 250 can implement the loss function (3) via two forward passes on the policy network 220. For example, in the first pass, only the belief states {b} from dataset 210 are used. t} is input into policy network 220, such that the first transmission captures only in belief state {b t The randomness of the policy is conditional. During the first pass, the random loss module 230 outputs π using the action distribution from the policy network 220. e (s) t ;θ) to calculate the random loss in (1). In the second pass, all observation information from dataset 210 { The action distribution π of the deterministic loss module 240 is input into the policy network 220 to obtain the action distribution π. e (o) t ), thus calculating the deterministic loss in (2). The second transmission of other potential information in a given state (e.g., u) u and u aIn the case of ), the pattern collapses. After two passes, the combined loss module 250 calculates the loss target in (3), which can be used to update the policy network 220 via backpropagation. See reference Figures 5A to 5B Find further details on the workflow for implementing security policy improvements using Policy Network 220.

[0045] As shown above, the random loss objective (1) for security policy improvement requires the Q function of the underlying behavioral policy, which can be estimated using Monte Carlo sampling on dataset D given a known reward R(s, a, g). The reward learning module 260 provides a mechanism for learning rewards based on causal reasoning of human demonstrator intent. The reward learning module 260 provides the reward function R(s, a, g) and the discount parameter γ to the random loss module 230 and the deterministic loss module 240. Further details of the reward learning module 260 are described below with reference to Figure 3.

[0046] Figure 3A Provided according to one implementation scheme Figure 2 The diagram shows a simplified representation of various aspects of the workflow within the reward learning module 260. Specifically, dialogue policy learning is typically accompanied by metrics M to evaluate the performance of the learned policy. While these metrics can act as proxies for reward functions, directly incorporating them into reward learning can be challenging. For example, these metrics functions often return a metric score for the entire dialogue. Given the complex state-action space of a dialogue management system, dialogue-level scores are not sufficiently explicit for rewarding actions performed in each dialogue round.

[0047] To address this ambiguous feedback, preference learning can be adapted from an online setup to an offline one. For example, preference learning was initially proposed as causal reasoning in feature selection by Paul et al.: Experiments with text classification, published in the proceedings of the 21st Conference on Computational Natural Language Learning, pp. 163-172, 2017. For each time step t, the reward can be parameterized as r(o t , a t (g). Given a pair, τ is derived. 1 τ 2 ∈D, having strategies for learning from each. and The action of each state in the sampling process, let P[τ] 1 >τ 2 ] is to capture preferences Exceed The probability metric is such that the preference is true when the sum of the rewards from each of the two dialogues satisfies the following condition:

[0048]

[0049] Such as about Figure 3B Further description, Defined as the preference probability, it is expressed as:

[0050]

[0051] Here, Φ() can be either exp() or identity 1(). For example, hyperparameters can be used to calculate probabilities:

[0052]

[0053] Therefore, the preference probability P[τ] can be minimized. 1 >τ 2 The reward R is optimized using a binary cross-entropy loss between the normalized metric score μ(τ) and the output of a pair. For example, based on the first dialogue τ from the pair... 1 The first metric score and the second dialogue τ from the pair 2 The normalized metric score is calculated using the second metric score, and both the first and second metric scores are generated by the same metric function M(), for example, In this way, the network (with a reward) is trained to generate dialogues with a performance metric that closely reflects the preferences between the pairs of responses. The loss objective for pairwise reward learning can be computed as follows:

[0054] (4)

[0055] in,

[0056] (5)

[0057] Here, θ1 and θ2 correspond to the parameters of the reward R(a,s,g;θ1) and the discount factor γ(θ2), respectively. Specifically, the discount factor γ can be predefined or learned during training.

[0058] Therefore, the reward learning module 260 receives the dataset D and splits it into K-fold training and validation subsets 261. For example, the dataset 210 is split into complementary subsets 261, training is performed on one subset, and the trained network is validated on the other (test) subset. At each epoch of training, K-baseline models 261a-n are trained using K training subsets based on cross-entropy loss (instead of (3)). The trained K-baseline models 261a-n are used to predict the corresponding validation subsets, and each baseline model can be similar to the neural model used by the policy network 220. The predicted action distributions from the K-baseline models are used to generate output dialogues 264a-n, each output dialogue being scored by a selected metric 263. Thus, dialogue pairs with corresponding score functions from the predicted dialogues 264a-n can be used to compute pairwise reward losses (4) at the pairwise causal reward learning module 265. The pairwise reward losses (4) can then be used to backpropagate the neural network to update parameters θ1, θ2. In this way, the pairwise causal reward learning module 265 outputs a reward function reward R(a,s,g;θ1) and a discount factor γ(θ2). For example, the neural network of the pairwise causal reward learning module 265 can be a bi-LSTM layer that embeds actions, states, and objectives, followed by a pair of multilayer perceptron (MLP) layers.

[0059] In another implementation, let θ = (θ1, θ2), then the parameter θ can be updated in the following way:

[0060] (6)

[0061] The learned reward is analogous to the sample weights for each instance of the data, which helps to redistribute the gradient update budget among samples based on their contribution to the overall success of the task-oriented dialogue (ToD) system. Therefore, the learned reward can be used as sample weights for any existing ToD dialogue system to reap the benefits of its sample efficiency.

[0062] In one implementation, dialogue outtakes are generated by an expert latent policy. The data (dialogue outtakes) can be distributed according to the optimal latent policy and transition probabilities. The process of learning a policy that maximizes the likelihood of the data can be a course of state actions for a pairwise reward learning objective (5). The process of fitting a maximum likelihood (MLE) policy may introduce useful perturbations due to the randomness of the optimizer. After the output dialogues 264a-n are selected for a metric 263 score, upon convergence of the MLE process, the learned outtake pairs with the corresponding metric scores can be used to train a priority optimization (5), which in turn learns a fine-grained reward R(a,s,g;θ1).

[0063] Figure 3B One embodiment described herein is provided. Figure 2 The diagram shows a simplified representation of the network architecture 300 of the reward learning module 260. In one implementation, three single bi-LSTM layers, each encoding the goal, belief state, and dialogue action or response sequence for each dialogue turn in each of the sampled push-out pairs. For example, bi-LSTM layer 301a encodes the goal of the sampled predicted push-out τ1; bi-LSTM layer 302a encodes the belief state for each dialogue turn of push-out τ1; and bi-LSTM layer 303a encodes the dialogue action for each dialogue turn of push-out τ1. Similarly, bi-LSTM layer 301b encodes the goal of the sampled predicted push-out τ2; bi-LSTM layer 302b encodes the belief state for each dialogue turn of push-out τ2; and bi-LSTM layer 303b encodes the dialogue action for each dialogue turn of push-out τ2.

[0064] In one implementation, three bi-LSTM layers can be used to encode both ejections τ1 and τ2. In another implementation, two sets of parallel bi-LSTM layers 301a, 302a, and 303a, as well as 301b, 302b, and 303b, can be used to encode the sampled ejection pairs in parallel.

[0065] In 305a, the three encoded representations from bi-LSTM layers 301a, 302a, and 303a are concatenated. Or in 305b, the three encoded representations from bi-LSTM layers 301b, 302b, and 303b are concatenated.

[0066] Then, the representation is fed into a pair of feedforward layers, followed by a bounded reward prediction for each round of yielding τ1 or τ2 using a sigmoid function. or For example, over all rounds of each launch, the rewards for each round are summed to form the global reward R(τ1) or R(τ2) for that launch pair.

[0067] Using a pair of dialogue rewards R(τ1) and R(τ2), the probabilistic preference between exits can be calculated using standard normalization or a softmax function, for example,

[0068]

[0069] Here, the Φ() function can be either a standard normalization function or a softmax function. The output of this preference probability 307 can be optimized using the cross-entropy loss described in equation (4).

[0070] Figure 4This is a simplified diagram of a computing device for implementing task-oriented dialogue security policy improvements and reward learning, based on some implementation schemes. For example... Figure 4 As shown, computing device 400 includes a processor 410 coupled to memory 420. The operation of computing device 400 is controlled by processor 410. And although computing device 400 is shown to have only one processor 410, it should be understood that processor 410 may represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), graphics processing units (GPUs), and / or the like in computing device 400. Computing device 400 may be implemented as a standalone subsystem, added to a board of computing devices, and / or a virtual machine.

[0071] Memory 420 may be used to store software executed by computing device 400 and / or one or more data structures used during operation of computing device 400. Memory 420 may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disks, floppy disks, hard disks, magnetic tape, any other magnetic media, CD-ROMs, any other optical media, punched cards, paper tape, any other physical media with a perforated pattern, RAM, PROM, EPROM, flash memory EPROM, any other memory chip or cassette memory, and / or any other media suitable for reading by a processor or computer.

[0072] Processor 410 and / or memory 420 can be arranged in any suitable physical configuration. In some embodiments, processor 410 and / or memory 420 may be implemented on the same board, in the same package (e.g., system in a package), on the same chip (e.g., system-on-a-chip), etc. In some embodiments, processor 410 and / or memory 420 may include distributed, virtualized, and / or containerized computing resources. Consistent with these embodiments, processor 410 and / or memory 420 may reside in one or more data centers and / or cloud computing facilities.

[0073] In some embodiments, memory 420 may include a non-transitory, tangible, machine-readable medium comprising executable code that, when run by one or more processors (e.g., processor 410), can cause the one or more processors to perform methods further described in detail herein. For example, as shown, memory 420 includes instructions for security policy improvement module 430 and reward learning module 435, which can be used to implement and / or simulate systems and models, and / or implement any methods further described herein. In some embodiments, security policy improvement module 430 and reward learning module 435 receive input 440 via data interface 415 and can generate output 450.

[0074] For example, input 440 can include, for example, Figure 2 The training dataset 210 shown in Figure 3 is referenced. Data interface 215 may include a communication interface for receiving dataset input 440 from a remote database via a communication network. In another embodiment, data interface 215 may include a user interface through which a user can select dataset input 440 and load it into processor 410. Output 450 may include action distributions, optimized strategies, etc., for dialogue.

[0075] Security policy improvement module 430 may include Figure 2 The policy network 220, random loss module 230, deterministic loss module 240, and combined loss module 250 are shown. The reward learning module 435 can be similar to... Figure 2 The module 260 shown is further detailed in Figure 3. As described with reference to Figure 3, the reward learning module 435 may include a K-basis model 262a-n and a pairwise causal reward learning module 265.

[0076] Figures 5A to 5B An exemplary logic flowchart illustrating a method 500 for improving a security policy based on MDP according to an implementation scheme is provided. One or more of the processes 502-524 of method 500 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium, which, when run by one or more processors, can cause one or more processors to execute one or more of the processes 502-524. In some implementations, method 500 may correspond to the method used by module 430.

[0077] In process 502, a training dataset (e.g., dataset 210) is received, comprising multiple dialogue launches (e.g., launches 212a-n) generated by a potential random behavior policy. Each launch includes a time series of observations representing information about the corresponding dialogue at multiple dialogue rounds.

[0078] In process 504, during the first pass to the neural model, belief states derived solely from observations of the training dataset (e.g., {b}) t The input is fed into a neural model (e.g., policy network 220).

[0079] In process 506, according to the target strategy, for example, π e (s) t ;θ), generate the first predicted action distribution based on the current state of the corresponding dialogue.

[0080] In process 508, based on the latent behavioral policy, a first discount sum is applied to the future reward based on the discount parameter and the reward function of the corresponding dialogue's action and state. Specifically, during the first transmission, the action distribution is conditioned on the belief state according to the latent stochastic behavioral policy, and the belief state is obtained from the observed time series.

[0081] In process 510, a first loss target is calculated based on a first discount to the future reward, a first expectation, and a first predicted action distribution. Specifically, a first expectation is taken on the probability distribution of the state and action according to the potential stochastic behavior policy, for example, according to (1).

[0082] In process 512, during the second pass, all observations are input into the neural model. For example, all observation information from dataset 210, except for belief states, is input. t =(b t , , )} is input into policy network 220.

[0083] In process 514, a second predicted action distribution is generated based on the current observations from the time series of observations, according to the target policy. For example, an action distribution π is generated. e (o) t ).

[0084] In process 516, a second discount sum for future rewards is calculated based on the discount parameters and reward function for a specific launch, for example, G(τ). i ,t)= Specifically, the second discount sum of future rewards is a collapsed deterministic approximation of the first discount sum of future rewards.

[0085] In process 520, a second loss target is calculated based on a second discount to future rewards, a second expectation, and a second predicted action distribution. Specifically, the second expectation is taken from the average of the observations on the training dataset. For example, the second loss target is calculated by the deterministic loss module 240 according to (2).

[0086] In process 522, for example, according to (3), the combined loss target is calculated by adding the first loss target and the second loss target.

[0087] In process 524, the neural model is updated based on the combined loss objective, provided that the KL divergence between the latent stochastic behavior policy and the objective policy conditioned on the current state of the corresponding dialogue is less than a predefined hyperparameter.

[0088] Figure 6A An exemplary pseudocode snippet illustrating an algorithm for Causal Aware Security Policy Improvement (CASPI) according to an implementation described herein is provided. The (training) dataset is sampled as a K-fold training set D. T and verification set D V A K-baseline model is trained using cross-entropy loss to fit the distribution of expert-generated data. During the process of fitting the data distribution, at each epoch of training, a statically learned K-policy is used to predict their corresponding K-fold validation subsets. During training, each dialogue is scored according to a selected dialogue level metric. On the convergence of the supervised learning process, the dialogue prediction pairs generated by the above process and their corresponding metric scores are used to train the priority optimization objective equation (4), which in turn learns a fine-grained reward R(a;s;g;θ). Using K-fold sampling and the K-baseline model helps to generate randomness in the generated samples. It also helps to use the data effectively and makes the method sampling efficient.

[0089] Figures 6B to 6C Provided are embodiments for use according to the description herein. Figure 6A An exemplary logic flowchart of the CASPI algorithm method is shown. One or more of the procedures 602-626 of method 600 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium, which, when run by one or more processors, can cause one or more processors to execute one or more of the procedures 602-626. In some embodiments, method 500 may correspond to the method used by module 430.

[0090] In process 602, a training dataset (e.g., dataset 210) is received, which includes multiple dialogue launches (e.g., launches 212a-n) generated by a potential random behavior policy.

[0091] In process 604, the training dataset is sampled multiple times to generate multiple training subsets and multiple validation subsets. For example, as referenced... Figure 3A The dataset D is described as being split into K-fold training subsets Di. T and verification subset D V261. For example, dataset 210 is divided into complementary subsets 261, training is performed on one subset, and the trained network is validated on another (test) subset.

[0092] In process 606, for (D) T D V For each dataset in the dataset, a task-oriented dialogue model is trained using training data from the first training subset of multiple training subsets, based on cross-entropy loss. For example, from multiple training subsets or multiple validation subsets (D... T D V The process involves acquiring a dataset, using dialogue data from that dataset, and updating the task-oriented dialogue model based on the target policy by minimizing the entropy of predicted dialogue actions conditioned on the current state of the dialogue. The entropy loss can be expressed as:

[0093]

[0094] Where, π m (s) represents the policy π conditioned on the dialogue state s. m And the predicted dialogue actions.

[0095] In step 608, for the same corresponding dataset from step 606, the task-oriented dialogue model generates predicted dialogues from the dialogue data in the first validation subset of multiple validation subsets.

[0096] In step 610, the predicted dialogue push is added to the pairwise causal learning subset D. P Starting from step 612, if another training epoch exists, steps 608-610 can be repeated. If no other training epoch exists in step 612, method 600 can determine in step 616 that (D T D V Does another dataset exist? If another dataset exists, method 600 continues from step 606, repeating with the other dataset. If no other dataset exists, method 600 proceeds to step 618.

[0097] In step 618, pairs can be derived from sampling dialogues from a subset of pairwise causal learning.

[0098] In step 620, a task-oriented dialogue model can be trained based on the preference probabilities between dialogue output pairs and a binary cross-entropy loss between normalized metric scores based on those dialogue output pairs. For example, step 620 can be based on a reference... Figure 3B The described processing flow is used to illustrate this.

[0099] In step 622, method 600 uses data D PDetermine if training convergence has been achieved. If not, method 600 proceeds from step 618 to resample another pair of sampled dialogues and repeats the process. If data D is used... P Convergence has been achieved; method 600 proceeds to step 624.

[0100] In step 624, a task-oriented dialogue model can be trained based on a policy optimization loss that optimizes the target policy using the training dataset. For example, refer to... Figures 5A to 5B Method 500 discusses strategy optimization.

[0101] In step 626, method 600 uses data D to determine whether training convergence has been achieved. If not, method 600 repeats from step 624. If convergence has been achieved using data D, method 600 can terminate.

[0102] Figure 7 This is a simplified block diagram illustrating a hybrid human-machine collaboration and automated evaluation metric score for pairwise reward learning according to the implementation described herein. Automated evaluation metrics have their own biases. The true goal of ToD is the human experience when interacting with a dialogue system, which automated evaluation metrics may fail to capture. Therefore, the quality of the generated responses can be manually evaluated. Quality can be defined by the following criteria: (a) appropriateness, e.g., is the generated response suitable for the given context in the dialogue round? (b) fluency, e.g., is the generated response coherent and intelligible?

[0103] Therefore, as Figure 7 As shown, at point 710, K models make predictions on K validation sets at each training epoch, and after pairwise causal reward learning at point 702, dialogue rounds are randomly selected from the test set. Human evaluators are shown the context leading to the round and are given an evaluation score at point 730 of the dialogue round. Predictions for the rounds by different models are anonymized and shown to the evaluators. For example, human evaluators are asked to rate appropriateness and fluency on a scale of 1 to 5, with 5 being the best and 1 being the worst. 100 randomly selected dialogue rounds were presented to 10 participants.

[0104] Then use as about Figures 6A to 6C The described pairwise causal reward learning is used to train the ToD model for reward R(s, a, g), wherein mini-batch instances are randomly sampled from instances rated by humans 730 or instances rated by an automated evaluation metric 740.

[0105] Note that it runs through. Figures 1A to 7The described implementation involves dialogue policy learning. However, similar implementations can be applied to different tasks in similar settings, such as, but not limited to, end-to-end dialogue system training (e.g., dialogue state trackers, dialogue policies and response generation, etc.), and / or the like.

[0106] Example performance

[0107] In one implementation, the training dataset (e.g., 2^10) could be the MultiWoz2.0 dataset, a multi-round, multi-domain dataset spanning seven domains, including attractions, hospitals, hotels, police, taxis, trains, and an additional domain for general greetings. This dataset is created based on real human conversations between tourists and information center staff. Each dialogue is generated by a user with a defined objective, which can cover 1-5 domains, and consists of up to 13 rounds in a single conversation. The dataset contains 10,438 dialogues, split into 8,438 dialogues for the training set and 1,000 dialogues, each for the validation and test sets.

[0108] In one implementation, the policy network 220 and / or reward learning network 260 can adopt the neural model proposed in “Task-Oriented Dialogue System Considering Multiple Appropriate Responses in the Same Context” by Zhang et al., arXiv preprint arXiv: 1911.10484, 2019, as a baseline (hereinafter referred to as “DAMD”). For the pairwise causal reward learning network 260, a bi-LSTM layer embedding actions, states, and goals can be used, followed by a pair of MLP layers. The DAMD consists of three seq2seq generative models using GRU. These three seq2seq models are used for the belief state module, the dialogue action module, and the response generation module, respectively. An attention layer is then used to focus on the output of the seq2seq model, which has a context vector from the previous round used for the replication mechanism. The output is then used as a representation of the token series predicting its corresponding module. Random L sto Loss function and deterministic L det The loss function is used for dialogue actions. For DST and response generation, the cross-entropy loss from DAMD is used as is.

[0109] In one implementation, the reward learning network 260 includes another, more complex model, which is the task-oriented dialogue model MinTL described in Lin et al., “MinTL: Minimal Transfer Learning for Task-Oriented Dialogue Systems,” arXiv preprint arXiv: 2009.12005, 2020. MinTL uses the large pre-trained language model BART, which serves as a standard encoder-decoder converter architecture with a bidirectional encoder and an autoregressive decoder. It is pre-trained on the task of denoising corrupted documents. BART is trained using the cross-entropy loss between the decoder output and the original document. MinTL does not explicitly predict dialogue actions. Therefore, a deterministic loss L is directly used for the generated responses. det Furthermore, for DST, we retain the loss from Minto as is.

[0110] In one implementation, the database results are represented as a heat vector. To reduce the variability of surface levels in the response, a domain-adaptive delexification preprocessing is employed, and a delexified response is generated with placeholders of specific values ​​that can be filled based on the current utterance, which refers to some slot values ​​provided by the system in the previous round.

[0111] In one implementation, a context-to-response generation task using Multi-woz2.0 can be implemented, and corresponding evaluation metrics are used to measure the quality of the response. These include notification rate and success rate, which measure the proportion of conversations in which the system has provided the requested information and the proportion of conversations in which the system has answered all requested information, respectively, and BLEU is used to measure the fluency of the generated response. Both settings use three evaluation metrics. These evaluation metrics include: 1) notification rate – measuring the proportion of conversations in which the system has provided the correct entity, 2) success rate – measuring the proportion of conversations in which the system has answered all requested information, and 3) BLEU – measuring the fluency of the generated response. A combined score (notification + success) × 0:5 + BLEU is also used. All reported CASPI figures are the median of 5 runs with different seeds.

[0112] For the metric M used in pairwise causal reward learning, the following metric is used:

[0113] M: = Notification + Success + λ × BLEU

[0114] This is very similar to the combinatorial score used in the evaluation, and they are equal when λ=2. The hyperparameter λ is used to normalize the achievable range of BLEU. If used as is, the success rate will result in a non-Markovian and stochastic per-round reward function, since the reward for the current state will depend on the performance of future states. Therefore, a soft version of the metric M is used. 软Success rate measures the proportion of requested information provided in a conversation. The original metric using a discrete variant of success rate is called M. 硬 Reward function R(s) t a t The action selection in (g) can be a dialog action or a generated response; we refer to the corresponding variants of the metric as M(action) and M(response). To demonstrate the versatility of this method in adapting to different metrics, all the variants of the metric discussed are used.

[0115] exist Figure 8 This paper compares Causal Awareness Security Policy Improvement (CASPI) with existing methods for context-to-response generation tasks in Multiwoz 2.0. Existing methods include:

[0116] DAMD, introduced by Zhang et al., is a domain-aware multi-decoder network. This method also leverages the randomness of dialogue actions through a data augmentation technique called multi-action data augmentation. DAMD with data augmentation is referred to here as DAMD+multi-action.

[0117] HDSA (Chen et al., Generating Semantically Conditioned Dialogue Responses via Hierarchical Disentangled Self-Attention (HDSA), arXiv preprint arXiv: 1905.12866, 2019) proposes using hierarchical graph representations for dialogue actions. It uses a pre-trained 12-layer BERT model to represent dialogue actions. A disentangled self-attention model, specifically a three-layer self-attention model, is used to transform the predicted dialogue actions into a hierarchical graph structure.

[0118] SOLOIST (Peng et al., Soloist: Few-shot Task-Oriented Dialogue with a Single Pre-trained Autoregressive Model, arXiv preprint arXiv: 2005.05298, 2020). These methods are trained on turn-level data without generating belief states and system actions in the dialogue history.

[0119] MinTL-BART (Lin et al.) introduces the Levenshtein Belief Span Framework, which predicts only incremental changes in the dialogue state for each turn. It utilizes pre-trained T5 and BART as the backbone of its model architecture.

[0120] HDNO, proposed by Wang et al. ("Option Framework for Task-Oriented Dialogue Systems: Hierarchical Modeling Between Dialogue Policies and Natural Language Generators," arXiv preprint arXiv: 2006.06814, 2020), is a dialogue policy learning method and a solution to the context-to-response generation task of Multiwoz2.0 (Budzianowski et al., 2018b). It leverages the hierarchy of dialogue actions and response generation tasks, proposing an option-based hierarchical RL framework and variational model to learn latent dialogue actions corresponding to natural language responses. Unlike CASPI, HDNO, while emphasizing the risk of sparsity in metrics such as success rate as reward functions, tends to form a proxy reward function. It uses a Markov language model as the proxy reward function. The language model is learned independently of the metric function. CASPI avoids reward formation and is independent of the properties of any unspecified metric function.

[0121] First, CASPI is compared with existing techniques for context-to-response generation tasks defined in MultiWoz2.0. The results are as follows... Figure 8 As shown. CASPI (DAMD) is an adaptive version of DAMD used for this task. CASPI (DAMD) outperforms other methods in three of the four performance metrics (i.e., success rate, notification rate, and combinatorial score). HDSA has a better BLEU score. HDSA's rich representation of natural language stems from its use of a large 12-layer BERT model.

[0122] Secondly, the adaptation of CASPI (DAMD) and CASPI (MinTL) was compared on the end-to-end dialogue task defined by MultiWoz2.0. The results are as follows: Figure 9 As shown, CASPI (DAMD) has a lightweight model architecture that does not require pre-training on any external corpus and outperforms all other previous methods on all evaluation metrics. This demonstrates that using CASPI to guide the gradient update process is effective because the sample weights for each dialogue turn result in a model that is perfectly aligned with the true objective of the task. CASPI (MinTL) has a robust pre-trained model that significantly outperforms CASPI (DAMD). This demonstrates the adaptability of existing methods to CASPI.

[0123] Inverse reinforcement learning, connecting non-policy policy learning, and evaluation have proven to be sample-efficient. CASPI is competitive with other sample-efficient techniques, such as data augmentation and transfer learning performed by (Zhang et al.) and (Lin et al.), respectively. To demonstrate this hypothesis, CASPI was baseline-tested under low sample complexity. For the experimental setup, the low-resource test policy was from (Lin et al.). The CASPI model was trained on 5%, 10%, and 20% of the training data and compared with other baselines for end-to-end dialogue and context-to-response generation tasks. Figures 10 to 11 The results are presented. In the end-to-end task, CASPI (MinTL), trained on only 10% of the data, outperforms previous state-of-the-art methods by training MinTL on 100% of the data in two of the three performance metrics. In the context-to-response generation task, CASPI (DAMD), trained on 75% of the data, matches HDNO's 100% data performance. This demonstrates that, in extremely limited resource settings, having the correct reward function to guide the gradient update process budget to achieve the true goal is crucial.

[0124] Figure 12 Examples of responses generated by different ToD models (e.g., MinTL, CASPI (MinTL), DAMD, and Simple ToD) are shown.

[0125] Figure 13 The evaluation results show human assessments of standard appropriateness and fluency. The mean and variance of the scores are shown. The appropriateness scores of MinTL 1301, Simple TOD 1302, and DAMD 1304 are compared to CASPI (MinTL) appropriateness 1303. The fluency scores of MinTL 1311, Simple TOD 1312, and DAMD 1314 are compared to CASPI (MinTL) fluency 1313. The results show that CASPI (MinTL) 1303 outperforms all other models 1301, 1302, and 1304 in terms of appropriateness score. The fluency scores of CASPI (MinTL) 1313, MinTL 1311, and Simple TOD 1312 are comparable.

[0126] Because automatic dialogue evaluation metrics are biased and do not truly reflect human goals, but in CASPI, these exact same dialogue evaluation metrics are used to learn the reward R(s, a, g). To bridge this gap, the following human-machine collaborative (HITL) experiment was conducted: a pair of CASPI (MINTL) models with different seeds were trained on 5% of the Multiwoz2.0 dataset. These pairs of models were then used to make predictions on 0.5% of the Multiwoz2.0 training data (40 dialogues), and human ratings were applied to the generated responses of these pairs relative to each other. Then, the results were used as referenced. Figures 6A to 6C The described pairwise causal reward learning is used to train a model for reward R(s, a, g), where mini-batch instances are randomly sampled from instances of human ratings or instances of ratings measured by automated evaluation, such as... Figure 13 As shown in the diagram. A new CASPI (MINTL) model was then trained on the original 5% of the data and the learned R(s, a, g). The trained model was manually evaluated using 3 participants in 24 test dialogues. Figure 14 The performance was demonstrated. By using HITL scores in reward learning, the performance of both human evaluation criteria (appropriateness and fluency) was improved. The human appropriateness score for 5% of the CASPI (MINTL) data increased from 1401 to 1402, now comparable to DAMD for 100% of the data. The fluency score also improved from 1411 to 1412. This demonstrates the versatility of pairwise causal reward learning. By using sufficiently rich neural networks, pairwise causal reward learning can be generalized to evaluation criteria for unknown dialogues.

[0127] Figure 15 It shows Figure 1B The same session between the visitor and the information center agent shown has example reward values ​​R(s) predicted for each round using pairwise causal reward learning. t a t (g). Observation shows that round #3 received the highest reward. Looking back, we realize this was the round where the transaction occurred—a crucial and risk-averse round in the dialogue, captured by the success rate metric of automatic evaluation. Round #2 received the next best reward, capturing the key information needed for the transaction to occur in round #3. Round #4 received a lower reward than rounds #3 & 2 because, aside from politeness, it didn't contribute much to the success of the conversation. It should be noted that round #4 will typically appear in almost all dialogues, and in supervised learning, it will receive the highest share of gradients. The learning reward redistribution aligns with the gradient budget for the success of the dialogue objective.

[0128] Figure 16This illustrates the different types of behavior that CASPI agents sometimes exhibit, especially when trained with low sample sizes. Greedy agents: In some domains, agents tend to book services before gathering all the necessary information or before the user requests or agrees to book the service. Figure 16 The first embodiment in the example demonstrates this behavior. Here, a user has requested a taxi, but the agent books it before gathering sufficient information (such as destination or departure time). This is because of a gap in the automated evaluation metrics. A low BLEU score and relatively high notification and success rates may indicate greedy agent behavior. Other reasons for a low BLEU score include a lack of diversity in responses or distorted responses.

[0129] Cautious agents: Agents tend to be cautious, providing lengthy responses that contain more information than is needed. Agents do this to avoid the risk of losing rewards due to information overload. This behavior is prevalent in... Figure 16 The second embodiment demonstrates this. These subtle behaviors illustrate gaps in automated evaluation metrics, which can be reduced by using evaluation in human-machine collaboration, such as... Figure 7 As shown in the image.

[0130] Some embodiments of a computing device (such as computing device 400) may include a non-transitory, tangible, machine-readable medium comprising executable code that, when run by one or more processors (e.g., processor 410), causes one or more processors to perform the process of method 200. For example, some common forms of machine-readable media that may include the process of method 200 may be floppy disks, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punched cards, paper tapes, any other physical media with a perforated pattern, RAM, PROMs, EPROMs, flash memory EPROMs, any other memory chips or cassettes, and / or any other media suitable for reading by a processor or computer.

[0131] This specification and accompanying drawings, which illustrate aspects, embodiments, implementation methods, or applications of the invention, should not be considered limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this specification and claims. In some cases, well-known circuits, structures, or techniques are not shown or described in detail to avoid obscuring the embodiments in the disclosure. Similar numbers in two or more figures represent the same or similar elements.

[0132] This specification sets forth specific details of some embodiments consistent with the disclosed text. Numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments can be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative but not restrictive. Those skilled in the art can implement other elements, although not specifically described herein, within the scope and spirit of the disclosed text. Furthermore, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments, unless otherwise specifically described, or if one or more features would render the embodiment inoperable.

[0133] Although illustrative embodiments have been shown and described, a wide range of modifications, alterations, and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be employed without correspondingly using other features. Many variations, substitutions, and modifications will be recognized by those skilled in the art. Therefore, the scope of the invention should be limited only by the appended claims, and it is appropriate that the claims be interpreted broadly and in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. A method for improving causal-aware safety policies in task-oriented learning, comprising: Receive a training dataset created from real human conversations, the training dataset comprising multiple dialogues in the form of natural language text, wherein the multiple dialogues include a first dialogue generated according to a potential random behavior strategy; The training dataset is sampled multiple times to generate multiple training subsets and multiple validation subsets; Using training data from the first training subset of the plurality of training subsets, a task-oriented dialogue model is trained based on cross-entropy loss. Using the task-oriented dialogue model, a predicted dialogue is generated based on the dialogue data in the first verification subset of the multiple verification subsets. The predicted dialogue push is added to the pairwise causal learning subset; The pair is derived from the sampled dialogue of the aforementioned pairwise causal learning subset; and The task-oriented dialogue model is trained using a binary cross-entropy loss between the preference probabilities between the dialogue output pairs and the normalized metric scores based on the dialogue output pairs.

2. The method according to claim 1, further comprising: Obtain the dataset from the plurality of training subsets or the plurality of validation subsets; and Using dialogue data from the acquired dataset, the task-oriented dialogue model is trained according to the target policy by minimizing the cross-entropy of predicted dialogue actions conditioned on the current state of the dialogue.

3. The method of claim 2, wherein the predicted dialogue is repeatedly generated according to the target policy by iterating over the plurality of validation subsets.

4. The method of claim 1, wherein training the task-oriented dialogue model based on the binary cross-entropy loss is performed by repeatedly sampling different dialogue pairs from the pairwise causal learning subset and retraining the task-oriented dialogue model based on the binary cross-entropy loss until convergence is achieved during training.

5. The method according to claim 1, wherein training the task-oriented dialogue model based on binary cross-entropy loss further comprises: Through three bi-LSTM layers, the goal, belief state, and dialogue action or response sequence at each dialogue round of each dialogue out in the sampled dialogue out pair are encoded into three coded representations respectively. The three encoded representations are concatenated; The concatenated coded representation is fed into one or more feedforward layers, which generate reward predictions for each dialogue round; The generated reward predictions are summed to obtain the dialogue reward for each dialogue in the sampled dialogue output pair; Based on the dialogue reward corresponding to the sampled dialogue output pair, calculate the preference probability between the dialogue output pairs; and Calculate the binary cross-entropy loss between the preference probabilities between the dialogue exit pairs and the normalized metric scores based on the dialogue exit pairs.

6. The method of claim 5, wherein the preference probability between the dialogue exit pairs is calculated using a normalization or softmax function.

7. The method according to claim 1, further comprising: Based on the policy optimization loss obtained by optimizing the target policy using the training dataset, the task-oriented dialogue model is repeatedly trained until training convergence is achieved.

8. The method of claim 7, wherein the strategy optimization loss is calculated by: Using the task-oriented dialogue model, a first predicted action distribution is generated based on the target policy and the current state of the dialogue. Based on the potential random behavior strategy, and using the discount parameter and the reward function of the dialogue's actions and states, calculate the first discount sum for future rewards; A first loss target is calculated based on the first discount and the first expectation of the future reward and the first predicted action distribution, wherein the first expectation takes a value on the probability distribution of the state and the action according to the potential random behavior strategy; Using the task-oriented dialogue model, a second predicted action distribution is generated based on the current observation from the time series of observations, according to the target policy. Based on the discount parameters and reward function for a specific launch, a second discount sum for future rewards is calculated, wherein the second discount sum for future rewards is a collapsed deterministic approximation of the first discount sum for future rewards; A second loss objective is calculated based on the second discount to the future reward, the second expectation, and the second predicted action distribution, wherein the second expectation takes a value at the average of the observations on the training dataset; and Calculate the sum of the first loss target and the second loss target.

9. The method according to claim 8, further comprising: The gradient update component is calculated based on the reward learned from the reward function of the actions and states of the dialogue, and the gradient of the target policy of the actions conditioned on the states and parameters of the task-oriented dialogue model. and The gradient update component is used to update the parameters of the task-oriented dialogue model.

10. The method according to claim 1, further comprising: During the validation of the trained task-oriented dialogue model, dialogue rounds are randomly selected. and A set of manually created evaluation scores are received from multiple evaluators to predict the dialogue rounds.

11. A system for improving causal-aware security policies in task-oriented learning, the system comprising: A communication interface that receives a training dataset created from real human conversations, the training dataset comprising multiple dialogues in the form of natural language text, wherein the multiple dialogues include a first dialogue generated according to a potential random behavior strategy. Memory, which stores multiple processor-executable instructions; and A processor that reads the plurality of processor-executable instructions from the memory to perform operations, the operations including: The training dataset is sampled multiple times to generate multiple training subsets and multiple validation subsets; Using training data from the first training subset of the plurality of training subsets, a task-oriented dialogue model is trained based on entropy loss. The task-oriented dialogue model generates a predicted dialogue from the dialogue data in the first verification subset of the plurality of verification subsets. The predicted dialogue push is added to the pairwise causal learning subset; From the sampled dialogues of the aforementioned pairwise causal learning subset, pairs are derived; and The task-oriented dialogue model is trained using a binary cross-entropy loss between the preference probabilities between the dialogue output pairs and the normalized metric scores based on the dialogue output pairs.

12. The system of claim 11, wherein the operation further comprises: Obtain the dataset from the plurality of training subsets or the plurality of validation subsets; and Using dialogue data from the acquired dataset, the task-oriented dialogue model is trained according to the target policy by minimizing the entropy of the predicted dialogue actions conditioned on the current state of the dialogue.

13. The system of claim 12, wherein the predicted dialogue is repeatedly generated and launched according to the target policy by iterating the plurality of verification subsets.

14. The system of claim 11, wherein the operation of training the task-oriented dialogue model based on the binary cross-entropy loss is performed by repeatedly sampling different dialogue pairs from the pairwise causal learning subset and retraining the task-oriented dialogue model based on the binary cross-entropy loss until convergence is achieved during training.

15. The system of claim 11, wherein the operation of training the task-oriented dialogue model based on binary cross-entropy loss further includes: Through three bi-LSTM layers, the goal, belief state, and dialogue action or response sequence of each dialogue round in each dialogue round of the sampled dialogue output pair are encoded into three coded representations respectively. The three encoded representations are concatenated; The concatenated coded representation is fed into one or more feedforward layers, which generate reward predictions for each dialogue round; The generated reward predictions are summed to obtain the dialogue reward for each dialogue in the sampled dialogue output pair; Based on the dialogue reward corresponding to the sampled dialogue output pair, calculate the preference probability between the dialogue output pairs; and Calculate the binary cross-entropy loss between the preference probabilities between the dialogue exit pairs and the normalized metric scores based on the dialogue exit pairs.

16. The system of claim 15, wherein the preference probability between the dialogue exit pairs is calculated using a normalization or softmax function.

17. The system of claim 11, wherein the operation further comprises: Using the task-oriented dialogue model, a first predicted action distribution is generated based on the target policy and the current state of the dialogue. Based on the potential random behavior strategy, and using the discount parameter and the reward function of the dialogue's actions and states, calculate the first discount sum for future rewards; A first loss target is calculated based on the first discount and the first expectation of the future reward and the first predicted action distribution, wherein the first expectation takes a value on the probability distribution of the state and the action according to the potential random behavior strategy; Using the task-oriented dialogue model, a second predicted action distribution is generated based on the current observation from the time series of observations, according to the target policy. Based on the discount parameters and reward function for a specific launch, a second discount sum for future rewards is calculated, wherein the second discount sum for future rewards is a collapsed deterministic approximation of the first discount sum for future rewards; A second loss objective is calculated based on the second discount to the future reward, the second expectation, and the second predicted action distribution, wherein the second expectation takes a value at the average of the observations on the training dataset; and Calculate the sum of the first loss target and the second loss target.

18. The system of claim 17, wherein the operation further comprises: The gradient update component is calculated based on the reward learned from the reward function of the actions and states of the dialogue, and the gradient of the target policy of the actions conditioned on the states and parameters of the task-oriented dialogue model. and The gradient update component is used to update the parameters of the task-oriented dialogue model.

19. The system of claim 11, wherein the operation further comprises: During the validation of the trained task-oriented dialogue model, dialogue rounds are randomly selected. and A set of manually created evaluation scores are received from multiple evaluators to predict the dialogue rounds.

20. A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for improving causal-aware security policies in task-oriented learning, the instructions being executed by a processor to perform operations, the operations including: Receive a training dataset created from real human conversations, the training dataset comprising multiple dialogues in the form of natural language text, wherein the multiple dialogues include a first dialogue generated according to a potential random behavior strategy; The training dataset is sampled multiple times to generate multiple training subsets and multiple validation subsets; Using training data from the first training subset of the plurality of training subsets, a task-oriented dialogue model is trained based on entropy loss. The task-oriented dialogue model generates a predicted dialogue from the dialogue data in the first verification subset of the plurality of verification subsets. The predicted dialogue push is added to the pairwise causal learning subset; The pair is derived from the sampled dialogue of the aforementioned pairwise causal learning subset; and The task-oriented dialogue model is trained using a binary cross-entropy loss between the preference probabilities between the dialogue output pairs and the normalized metric scores based on the dialogue output pairs.

Images