System and method for improving safety policies in task-oriented interactions - Patents.com

Abstract

The embodiments described herein provide safety policy improvement (SPI) in a batch reinforcement learning framework for task-oriented dialogue. Specifically, a batch reinforcement learning framework for dialogue policy learning is provided, which learns to improve dialogue performance and shape the rewards that the present invention discusses behind human responses rather than simply mimicking human performance.

Description

[Technical field]

[0001] [Cross reference] This disclosure is a nonprovisional application claiming priority to U.S. Provisional Application No. 63 / 148,861, filed February 12, 2021, and claims priority to U.S. Nonprovisional Application No. 17 / 500,855, filed October 13, 2021, which is a continuation-in-part of and claims priority to co-pending and commonly owned U.S. Nonprovisional Application No. 17 / 105,262, filed November 25, 2020, which is a nonprovisional application claiming priority under 35 U.S.C. 119 to U.S. Provisional Application No. 63 / 034,653, filed June 4, 2020.

[0002] All of the aforementioned applications are hereby expressly incorporated by reference in their entirety.

[0003] [Technical field] The present disclosure relates generally to machine learning models and neural networks, and more particularly to improving safety strategies for task-oriented interactions. [Background technology]

[0004] Neural networks have been used to generate conversational responses and thus interact with human users to accomplish tasks. For example, a human user can engage in a conversation with an intelligent assistant to book travel tickets, make restaurant reservations, etc. To accomplish a complex task, an intelligent assistant typically needs to learn to collectively complete multiple subtasks. For example, an assistant needs to book a hotel and book a flight such that enough time remains for commuting between arrival and hotel check-in. For an intelligent assistant to learn such a complex task, it learns a dialogue strategy for selecting among subtasks or options at a given time, which is often accompanied by a state tracker that tracks the status of the subtasks.

[0005] Task-oriented dialogue systems are typically trained from offline data collected using human performances (e.g., past interactions), but collecting diverse performances and annotating them can be expensive. Furthermore, such offline task-oriented dialogue systems often involve heterogeneous systems, such as belief state trackers, dialogue policy management, response generation, etc. These heterogeneous systems can induce stochasticity and related challenges in addition to the need for sample efficiency in effective dialogue policy learning.

[0006] Therefore, there is a need for efficient policy learning in task-oriented dialogue systems. [Brief description of the drawings]

[0007] [Figure 1A] 1 provides a diagram illustrating an example task-oriented dialogue described by a Markov decision process to which safety policy improvements may be applied, according to one embodiment described herein. [Figure 1B] 1A provides an illustration of an exemplary task-oriented dialogue of multiple dialogue turns between a user and an agent shown in FIG. 1A, according to one embodiment described herein. [Diagram 2] FIG. 1 provides a simplified diagram illustrating an example architecture for training a policy network using reward learning, according to one embodiment described herein. [Figure 3A] 3 provides a simplified diagram illustrating aspects of the workflow within the reward learning module shown in FIG. 2, according to one embodiment described herein. [Figure 3B] 3 provides a simplified diagram illustrating a network architecture for the reward learning module shown in FIG. 2 according to one embodiment described herein. [Figure 4] 1 is a simplified diagram of a computing device for implementing safety strategy improvement and reward learning for task-oriented interactions, according to some embodiments. [Figure 5A] 1 provides an exemplary logic flow diagram illustrating a method for MDP-based safety policy improvement, according to one embodiment. [Figure 5B] 1 provides an exemplary logic flow diagram illustrating a method for MDP-based safety policy improvement, according to one embodiment. [Figure 6A] 1 provides an example pseudo-code segment illustrating an algorithm for Causality-Aware Safety Strategy Improvement (CASPI), according to one embodiment described herein. [Figure 6B] 6B provides an example logic flow diagram illustrating a method for the CASPI algorithm shown in FIG. 6A according to one embodiment described herein. [Figure 6C] 6B provides an example logic flow diagram illustrating a method for the CASPI algorithm shown in FIG. 6A according to one embodiment described herein. [Figure 7] FIG. 1 is a simplified block diagram illustrating a mixed human-in-the-loop and automatic evaluation metric score for pairwise reward learning, according to embodiments described herein. [Figure 8] 1 provides a data chart illustrating an exemplary performance comparison of safety strategy improvements with existing methods, according to one embodiment. [Figure 9] 1 provides a data chart illustrating an exemplary performance comparison of safety strategy improvements with existing methods, according to one embodiment. [Figure 10] 1 provides a data chart illustrating an exemplary performance comparison of safety strategy improvements with existing methods, according to one embodiment. [Figure 11] 1 provides a data chart illustrating an exemplary performance comparison of safety strategy improvements with existing methods, according to one embodiment. [Figure 12] 1 provides a data chart illustrating an exemplary performance comparison of safety strategy improvements with existing methods, according to one embodiment. [Figure 13] 1 provides a data chart illustrating an exemplary performance comparison of safety strategy improvements with existing methods, according to one embodiment. [Figure 14] 1 provides a data chart illustrating an exemplary performance comparison of safety strategy improvements with existing methods, according to one embodiment. [Figure 15]1 provides a data chart illustrating an exemplary performance comparison of safety strategy improvements with existing methods, according to one embodiment. [Figure 16] 1 provides a data chart illustrating an exemplary performance comparison of safety strategy improvements with existing methods, according to one embodiment.

[0008] In the drawings and accompanying documents, elements having the same name have the same or similar functions. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Task-oriented dialogue systems are typically trained from offline data collected using human performances (e.g., past interactions), but collecting diverse performances and annotating them can be expensive. Furthermore, such offline task-oriented dialogue systems often involve heterogeneous systems, such as belief state trackers, dialogue policy management, response generation, etc. These heterogeneous systems can induce stochasticity and related challenges in addition to the need for sample efficiency in effective dialogue policy learning.

[0010] Some existing systems employ off-policy based reinforcement learning (batch RL) methods in solving complex tasks. Batch RL methods typically use historically annotated data instead of simulators, which can be sample-efficient since inexpensive simulators are usually readily available to sample on-policy data. However, these techniques may not work efficiently due to the nature of dialogue policy learning. For example, off-policy based learning may often require the estimation of behavior policies for a given state, e.g., belief state, of the underlying Markov Decision Process (MDP). In reality, the belief state does not necessarily capture the true state of the MDP, but MDP latent states such as prosody, among others, may induce stochasticity in the agent response at each turn. Furthermore, semantic information may be lost when dialogue acts are generated for natural language texts. The use of mere policy imitation for dialogue acts may be insufficient to provide unbiased inference for a particular outcome when each component of a compound behavior is equally focused on.

[0011] Considering the need for efficient policy learning in task-oriented dialogue systems, the embodiments described herein provide a safe policy improvement in a batch reinforcement learning framework for task-oriented dialogue. Specifically, a dialogue policy is trained against dialogue rollouts generated by potential behavior policies with performance guarantees, for example, by enforcing that the performance of the new policy is at least better than the old behavior policy for positive gaps. Then, a training loss objective is defined by minimizing the expected discounted sum of future rewards subject to the condition that the KL divergence between the old behavior policy and the target policy is not larger than a predefined hyperparameter. In this way, the bias in training against the rollout of another policy can be significantly reduced, thus resulting in a "safe" policy improvement.

[0012] Furthermore, pairwise causal reward learning is provided to shape rewards to infer the intent of human speech instead of mimicking human performance in a batch reinforcement setting. The combination of safety policy improvement and pairwise causal reward learning can achieve sample efficiency in learning complex tasks.

[0013] As used herein, the term "network" may include any hardware or software-based framework, including any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented thereon or in conjunction therewith.

[0014] As used herein, the term "module" may include a hardware or software-based framework that performs one or more functions. In some embodiments, a module may be implemented on one or more neural networks.

[0015] 1A provides a diagram 100 illustrating an example task-oriented dialogue described by a Markov decision process to which safety policy improvements may be applied, according to one embodiment described herein. Diagram 100 illustrates a dialogue turn of a task-oriented dialogue between a user 110 and an intelligent agent 120. For example, the user 110 can provide a user utterance 101, "Book me a flight to London," and the agent 120 can respond with a system response, "when do you want to leave?" 120. The dialogue between the user 110 and the intelligent agent 120 can form a task-oriented dialogue to complete the task of planning a trip to London.

[0016] Task-oriented dialogue can be modeled as a Markov Decision Process (MDP), represented by a connected graph structure 110. An MDP is described by a tuple {S, A, P, R, γ} of states S, actions A, transition probabilities P, rewards R, and discount factors γ. State S is the dialogue context, which is the agent's interpretation of the environment. Actions A are the possible communication behaviors available to the agent in each state. Transition probabilities P define the probability that state S transitions to another set of states S′ given action A. For example, intelligent agent 120 may transition from state s t At time step t, we define a target policy π e (a t |s t ) by compound action a t For example, in state 105s1 after user utterance 101, the origin city is confirmed (per user location) and the destination city "London" is obtained from user utterance 101, but the departure date and departure time are unknown. Therefore, dialogue act 106 executes target policy π e After the dialogue act 106, the dialogue state transitions from state s1 to state s2.

[0017] A latent reward function R(a,s) with discount factor γ ∈ [0,1] is associated with the MDP 120 and defines a reward value given a set of states and actions. For example, given state s1 and dialogue act a1, a positive reward r 115 of "20" is assigned. In one embodiment, the latent reward function R(a,s) and discount factor γ may be predefined for the MDP. In another embodiment, the latent reward function R(a,s) and discount factor γ may be learned through a pair-wise causal reward learning mechanism described in connection with FIG. 3.

[0018] In one embodiment, given a reward function and a discount factor, the objective is to find a target policy π e (a t |st ), which maximizes the expected discounted sum of future rewards over the MDP, which is the state-action function

number

[0019] FIG. 1B provides a diagram illustrating an exemplary task-oriented dialogue of multiple dialogue turns between the user and the agent shown in FIG. 1A, according to one embodiment described herein. The dialogue shown in FIG. 1B corresponds to a goal 122, for example, regarding reserving a train to depart at a particular time and go to a particular destination. The dialogue includes four dialogue turns, each of which includes a delexicalized user utterance 125a-d, an agent dialogue act 126a-d, and a delexicalized agent utterance / response 127a-d, respectively. The four dialogue turns may indicate that the mere use of strategy imitation for dialogue acts may be insufficient to reason on the outcome, but rather, focus equally on each component of the composite behavior. For example, turns 3 and 4 are rich in semantic information. Turn 3 provides the key to the transaction of the reservation process, while turn 4, which is of minimal use in the success of the conversation, gets equal weighting with other semantically rich turns. Such details are lost in imitation policy learning.

[0020] 2 provides a simplified diagram 200 illustrating an example architecture for training a policy network using reward learning, according to one embodiment described herein. Diagram 200 shows that a training dataset 210 is input to a policy network 220 and a reward learning module 260. Specifically, dataset 220 includes multiple rollouts 212a-n from a dialogue. The rollouts 212a-n may be generated by a human agent that executes actions based on a latent stochastic behavior policy.

[0021] For example, in offline batch RL, an intelligent agent does not get to interact with the environment. Instead, it computes a set of potential probabilistic behavior policies π b A set of offline data D 210 may be obtained that has been logged by a human agent performing actions based on τ. The set of offline data D 210 may include multiple rollouts 212a-n of interactions, each of which may include a rollout τ i ∈D. Each rollout τ i =((o0 i ,a0 i ), ···, (o T-1 i ,a T-1 i )), where each o t is the observation at turn t, and o t =(b t ,u t u ,u t-1 a ), where b t is the agent's belief state at turn t, and u t u and u t-1 a are the user and agent utterances at times t and t-1, respectively. Thus, batch RL involves training a policy against the rollouts generated by the potential behavior policy.

[0022] However, directly optimizing a training objective, e.g., the discounted sum of future rewards, for the rollout of a different policy leads to large bias in the value function estimates, poor generalization properties, and sample inefficiency. Therefore, a "safe" policy improvement can be implemented such that the new policy performance is bounded relative to the old policy. Specifically, the new target policy π e Value function of π and potential behavior policy b The value function of is

number

number

[0023] Therefore, the input observation o from the data set 210 t =(b t ,u t u ,u t-1 a ), the policy network 220 generates a target policy π e and the target action distribution π e (s t ;θ) can then be generated. sto (θ) may be calculated in the loss module 230 for improving the safety policy.

number

[0024] In some implementations, the probabilistic loss objective L sto (θ) is s in equation (1). t To replace the belief state b tThe belief state is a probabilistic variable because it does not capture all information. e (b t ;θ) is calculated to optimize a stochastic loss function.

[0025] Conventionally, the update mechanism provided by Schulman et al. in Trust Region Policy Optimization, in Proceedings of International conference on machine learning, pages 1889-1897, 2015, provides a bounded error as long as the constraint in (1) is satisfied, where D KL (.||.) is the KL divergence and η is a hyperparameter. However, the Schulman update rule is a behavioral policy π that is intractable to estimate. b (a t |s t ) instead, we need access to the belief state b t Behavior policy π subject to b (b t ) in Eq. (1) t , which results in a probabilistic behavior policy. t is the observation o at turn t that can be obtained from a particular rollout in the dataset D210. t Therefore, in one implementation, when calculating the probabilistic loss objective in (1), π b (s t ) can be obtained from the rollout in the dataset 210. b (b t ) can be approximated by, for example, π b (b t ) is estimated by b t Given a dialogue act a t The number of occurrences of b t Given action a t may be given by dividing by the total number of

[0026] Observation o t(This is the belief state b t Based on the availability of more evidence (containing more information), the mode of the policy may collapse to a near deterministic behavior. To factor this into the policy learning, a further deterministic loss may be computed in the loss module 240.

number

number

number

number

number

number

[0027] In one embodiment, the network 220 detects the probabilistic loss L sto (θ) alone, or the deterministic loss L detAlternatively, the network 220 can be trained with the sum of two losses L(θ), as described below.

[0028] In one embodiment, the combined loss module 250 can achieve the loss function (3) through two forward passes on the policy network 220. For example, in the first pass, the belief states {b t} is input to the policy network 220, so that the first pass is t During the first pass, the probabilistic loss module 230 captures the stochasticity of the policy conditional only on the action distribution output π e (s t ;θ) to calculate the probabilistic loss in (1). In the second pass, all the observation information {o t =(b t ,u t u ,u t-1 a )} is used to calculate the deterministic loss of (2) by the deterministic loss module 240. e (o t ) is input to the strategy network 220. The second pass is u and u a 5A-5B. After the two passes, the combined loss module 250 computes a loss objective in (3), which can be used to update the policy network 220 via backpropagation. Further details of the workflow for implementing safety policy improvements using the policy network 220 can be found in relation to FIGS. 5A-5B.

[0029] As shown above, the probabilistic loss objective (1) for safety policy improvement requires a Q-function of the potential behavior policy, which can be estimated using Monte Carlo sampling on the dataset D, assuming the reward R(s, a, g) is known. The reward learning module 260 provides a mechanism for learning rewards that are causally inferred to the intentions of a human actor. The reward learning module 260 provides the reward function R(s, a, g) and the discount parameter γ to the probabilistic loss module 230 and the deterministic loss module 240. Further details of the reward learning module 260 are described below in connection with FIG. 3.

[0030] FIG. 3A provides a simplified diagram illustrating aspects of the workflow inside the reward learning module 260 shown in FIG. 2, according to one embodiment. Specifically, dialogue policy learning is usually accompanied by a metric M to evaluate the performance of the learned policy. Although these metrics can serve as proxies for reward functions, directly combining them into learning rewards can be difficult. For example, these metric functions usually return a metric score for the entire dialogue. Given the complex state-action space of a dialogue management system, the scores at the dialogue level are under-specified to reward the actions performed in each dialogue turn.

[0031] To address this underspecified feedback, preference learning can be adapted from the online setting to the offline setting. For example, preference learning was originally proposed by Paul et al. in Feature selection as causal inference: Experiments with text classification, in Proceedings of the 21st Conference on Computational Natural Language Learning, pages 163-172, 2017. The reward is calculated as r(o t ,a t ,g). The learned policy πe 1 and π e 2 A rollout τ with actions for each state in the rollout sampled from each 1 , τ 2 Given a pair ∈D,

number

number

[0032] As will be further described in relation to FIG.

number

number

number

[0033] Therefore, the reward R is the preference probability

number

number

number

number

[0034] where θ1 and θ2 correspond to the parameters of the reward R(a,s,g;θ1) and the discount factor γ(θ2), respectively. Specifically, the discount factor γ may be predefined or learned during training.

[0035] Thus, the reward learning module 260 receives the dataset D and splits it into K-fold training and validation subsets 261. For example, the dataset 210 may be partitioned into complementary subsets 261, training performed on one subset, and validating the trained network on another (test) subset. For each training epoch, K baseline models 261a-n are trained based on the cross-entropy loss (instead of (3)) using the K training subsets. The trained K baseline models 261a-n are used to predict on the corresponding validation subset, where each baseline model may be similar to the neural model used by the policy network 220. The predicted behavior distributions from the K baseline models are used to generate output interactions 264a-n, each of which is scored by the selected metric 263. Thus, pairs of interactions from the predicted interactions 264a-n with corresponding score functions may be used to calculate the pairwise reward loss (4) in the pairwise causal reward learning module 265. The pairwise reward loss (4) can then be used to back-propagate through the neural network to update the parameters θ1, θ2. In this way, the pairwise causal reward learning module 265 outputs the reward function reward R(a,s,g;θ1) and the discount factor γ(θ2). For example, the neural network for the pairwise causal reward learning module 265 can be one bi-LSTM layer that embeds the actions, states, and goals, followed by several multilayer perceptron (MLP) layers.

[0036] In another embodiment, if θ=(θ1, θ2), then the parameter θ can be updated by:

number

[0037] The learned rewards are analogous to sample weights for each instance of data, which helps to redistribute the gradient update budget among samples based on their contribution to the overall success of the Task-oriented Dialogue (ToD) system. To this end, the learned rewards can be used as sample weights for any existing ToD dialogue system to benefit from the sample efficiency it brings.

[0038] In one embodiment, the dialogue rollouts are generated by an expert latent policy. The data (dialogue rollouts) may be distributed with optimal latent policies and transition probabilities. The process of learning a policy that maximizes the likelihood of the data may be a curriculum for searching state actions for a pairwise reward learning objective (5). The process of fitting a maximum likelihood (MLE) policy may induce useful perturbations due to the stochastic nature of the optimizer. After the output dialogues 264a-n are scored by the selected metric 263, upon convergence of the MLE process, pairs of learned rollouts with corresponding metric scores may be used to train a preference optimizer (5), which then learns a fine-grained reward R(a,s,g;θ1).

[0039] 3B provides a simplified diagram illustrating a network architecture 300 for the reward learning module 260 shown in FIG. 2, according to one embodiment described herein. In one embodiment, three single bi-LSTM layers are each used to encode the goal, belief state, and dialogue act or response sequence in each dialogue turn in each sampled rollout pair. For example, bi-LSTM layer 301a is used to encode the goal of the sampled predicted rollout τ1, bi-LSTM layer 302a is used to encode the belief state of each dialogue turn of rollout τ1, and bi-LSTM layer 303a is used to encode the dialogue act of each dialogue turn of rollout τ1. Similarly, bi-LSTM layer 301b is used to encode the goal of the sampled predicted rollout τ2, bi-LSTM layer 302b is used to encode the belief state of each dialogue turn of rollout τ2, and bi-LSTM layer 303b is used to encode the dialogue act of each dialogue turn of rollout τ2.

[0040] In one embodiment, three bi-LSTM layers may be used to encode both rollouts τ1 and τ2. In another embodiment, two sets of parallel bi-LSTM layers 301a, 302a, and 303a, and 301b, 302b, and 303b may be used to encode each pair of sampled rollouts in parallel.

[0041] The three encoded representations from the bi-LSTM layers 301a, 302a, and 303a are concatenated at 305a, or the three encoded representations from the bi-LSTM layers 301b, 302b, and 303b are concatenated at 305b.

[0042] The concatenated representation is then used to calculate the constrained reward prediction R(s1 τ1 ,a1 τ1 )...R(s nτ1 ,a n τ1 ) or R(s1 τ2 ,a1 τ2 )...R(s n τ2 ,a n τ2 ), the rewards per turn are summed over all turns of each rollout to form a global reward R(τ1) or R(τ2) for the pair of rollouts.

[0043] Using a pair of interaction rewards R(τ1) and R(τ2), we can estimate the probabilistic preference between rollouts using either a standard normalization or a softmax function, e.g.,

number

[0044] 4 is a simplified diagram of a computing device for implementing safety policy improvement and reward learning for task-oriented interaction, according to some embodiments. As shown in FIG. 4, the computing device 400 includes a processor 410 coupled to a memory 420. The operation of the computing device 400 is controlled by the processor 410. Although the computing device 400 is shown with only one processor 410, it is understood that the processor 410 may represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs), and the like within the computing device 400. The computing device 400 may be implemented as a standalone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0045] Memory 420 may be used to store software executed by computing device 400 and / or one or more data structures used during operation of computing device 400. Memory 420 may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tape, any other physical media with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other media adapted to be read by a processor or computer.

[0046] The processor 410 and / or memory 420 may be arranged in any suitable physical location. In some embodiments, the processor 410 and / or memory 420 may be implemented on the same board, in the same package (e.g., a system in package), on the same chip (e.g., a system on a chip), etc. In some embodiments, the processor 410 and / or memory 420 may comprise distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, the processor 410 and / or memory 420 may be located in one or more data centers and / or cloud computing facilities.

[0047] In some examples, memory 420 may include a non-transitory, tangible, machine-readable medium that includes executable code that, when executed by one or more processors (e.g., processor 410), may cause the one or more processors to perform methods described in further detail herein. For example, as shown, memory 420 includes instructions for a safety strategy refinement module 430 and a reward learning module 435 that may be used to implement and / or emulate the systems and models and / or implement any of the methods described further herein. In some examples, safety strategy refinement module 430 and reward learning module 435 may receive input 440 via data interface 415 and generate output 450.

[0048] For example, the input 440 may include a training dataset 210 as shown in Figures 2-3. The data interface 215 may include a communications interface to receive the dataset input 440 from a remote database over a communications network. In another example, the data interface 215 may include a user interface through which a user can select and load the dataset input 440 into the processor 410. The output 450 may include an action distribution for an interaction, an optimized strategy, etc.

[0049] The safety policy improvement module 430 may include the policy network 220, the probabilistic loss module 230, the deterministic loss module 240, and the coupling loss module 250 shown in Figure 2. The reward learning module 435 may be similar to the module 260 shown in Figure 2, which is further detailed in Figure 3. The reward learning module 435 may include the K-based models 262a-n and the pairwise causal reward learning module 265, as described in connection with Figure 3.

[0050] 5A-5B provide an example logic flow diagram illustrating a method 500 of MDP-based safety policy improvement, according to one embodiment. One or more of the processes 502-524 of the method 500 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when executed by one or more processors, may cause the one or more processors to perform one or more of the processes 502-524. In some embodiments, the method 500 may correspond to a method used by the module 430.

[0051] In process 502, a training dataset (e.g., dataset 210) is received that includes multiple dialogue rollouts (e.g., rollouts 212a-n) generated by a latent probabilistic behavior strategy. Each rollout includes a time series of observations that represent information for a respective dialogue in multiple dialogue turns.

[0052] In process 504, belief states (e.g., {b t}) is input to the neural model (e.g., policy network 220) in the first pass through the neural model.

[0053] In process 506, the first predicted behavior distribution is matched to a target strategy, e.g., π e (s t ;θ) based on the current state of each interaction.

[0054] In process 508, a first discounted sum of future rewards is based on discount parameters and reward functions for the actions and states of each interaction according to a latent behavior policy. Specifically, during the first pass, the action distributions are conditioned on belief states according to a latent probabilistic behavior policy, where the belief states are obtained from a time series of observations.

[0055] In process 510, a first loss objective is calculated based on a first expectation of a first discounted sum of future rewards and a first predicted action distribution. Specifically, the first expectation is obtained over a probability distribution of states and actions according to the underlying probabilistic behavior strategy, e.g., according to (1).

[0056] In process 512, the complete observations are input to the neural model in a second pass. For example, in addition to the belief states, all the observation information from the dataset 210 is input. t =(b t ,u t u ,u t-1 a )} is input to the policy network 220.

[0057] In process 514, a second predicted behavior distribution is generated based on the current observation from the time series of observations according to the target strategy. For example, the behavior distribution π e (o t ) is generated.

[0058] In process 516, a second discounted sum of future rewards based on the discount parameters and reward function for the particular rollout is calculated, e.g.,

number

[0059] In process 520, a second loss objective is calculated based on a second expectation of a second discounted sum of future rewards and the second predicted behavior distribution. Specifically, the second expectation is taken on an average of the observations across the training dataset. For example, the second loss objective is calculated by the deterministic loss module 240 according to (2).

[0060] In process 522, a combined loss objective is calculated by summing the first loss objective and the second loss objective, for example according to (3).

[0061] In process 524, the neural model is updated based on a combination loss objective, subject to the condition that the KL divergence between the potential probabilistic behavior policies and the target policy conditional on the current state of each interaction is less than a predefined hyperparameter.

[0062] FIG. 6A provides an example pseudocode segment illustrating an algorithm for causal aware safe policy improvement (CASPI) according to one embodiment described herein. The (training) dataset is divided into K-fold training D T and validation set D VThe K-fold subsampling is performed. K baseline models are trained to fit the data distribution generated by the experts using cross-entropy loss. During the process of fitting the data distribution, the still-learning K policies are used to predict for their corresponding K-fold validation subsets for each training epoch. Each of the interactions is scored by an interaction-level metric selected during training. At the convergence of the supervised learning process, the pairs of interaction predictions generated by the above process, along with their corresponding metric scores, are used to train for the preference optimization objective (4), which then learns the fine-grained reward R(a;s;g;θ). The K-fold subsampling and the use of K baseline models help to generate stochasticity in the generated samples. It also helps in using the data effectively and making the method sample-efficient.

[0063] 6B-6C provide an example logic flow diagram illustrating a method for the CASPI algorithm shown in FIG. 6A, according to one embodiment described herein. One or more of processes 602-626 of method 600 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when executed by one or more processors, may cause the one or more processors to perform one or more of processes 602-626. In some embodiments, method 500 may correspond to a method used by module 430.

[0064] In process 602, a training dataset (eg, dataset 210) is received that includes a number of interaction rollouts (eg, rollouts 212a-n) generated by a potential probabilistic behavior strategy.

[0065] In process 604, the training data set is repeatedly sampled multiple times to generate multiple training subsets and multiple validation subsets. For example, as described in connection with FIG. 3A, the data set D may be sampled into K-fold training D T and the validation subset D V 261. For example, the dataset 210 may be partitioned into complementary subsets 261, performing training on one subset and validating the trained network on another (test) subset.

[0066] In process 606, (D T ,D V For each dataset in , the task-oriented dialogue model is trained based on the cross entropy loss using the training data in a first training subset of the multiple training subsets. For example, if the dataset is trained using multiple training subsets or multiple validation subsets (D T ,D V ) and the task-oriented dialogue model is updated by minimizing the entropy of the predicted dialogue behavior conditional on the current state of the dialogue according to the target policy using dialogue data from the retrieved dataset. The entropy loss can be expressed as:

number

number

[0067] In step 608, for the same respective data set from step 606, the task-oriented interaction model generates a predicted interaction rollout from the interaction data in a first validation subset of the multiple validation subsets.

[0068] In step 610, the predicted dialogue rollout is calculated based on the pairwise causal learning subset D P From step 612, if there is another training epoch, steps 608-610 can be repeated. If there is not another training epoch at step 612, method 600 continues at step 616 with (D T ,D V ) if there is another dataset, method 600 proceeds to repeat from step 606 with the other dataset. If there is not another dataset, method 600 proceeds to step 618.

[0069] In step 618, pairs of interaction rollouts may be sampled from the pairwise causal learning subset.

[0070] In step 620, the task-oriented dialogue model may be trained based on a binary cross-entropy loss between preferred probabilities between pairs of dialogue rollouts and normalized metric scores based on the pairs of dialogue rollouts. For example, step 620 may be illustrated by the process flow described in connection with FIG. 3B.

[0071] In step 622, the method 600 performs a training convergence test on the data D P If not, the method 600 resamples another pair of the sampled pairs of interaction rollouts and repeats from step 618. P If convergence is reached using , then the method 600 proceeds to step 624 .

[0072] In step 624, the task-oriented dialogue model may be trained based on a policy optimization loss that optimizes for a target policy using the training dataset. For example, optimization over policies is discussed in connection with method 500 of FIGS. 5A-5B.

[0073] In step 626, method 600 determined whether training convergence was reached using data D. If not, method 600 repeats from step 624. If convergence was reached using data D, method 600 may end.

[0074] FIG. 7 is a simplified block diagram illustrating a mixed human-in-the-loop and automatic evaluation metric score for pair-wise reward learning according to an embodiment described herein. Automatic evaluation metrics have their own biases. The real objective of ToD is the human experience while interacting with a dialogue system, and automatic evaluation metrics may be insufficient to capture this. To this end, human evaluation can be done on the quality of the generated response. Quality can be defined by the following criteria: (a) appropriateness, e.g., is the generated response appropriate for a given context within a dialogue turn? (b) fluency, e.g., is the generated response coherent and easy to understand?

[0075] Thus, as shown in FIG. 7, after predictions on the K validation set by the K models at each epoch of training in 710 and pairwise causal reward learning in 702, a dialogue turn in the test set is picked randomly. Human raters were shown the context leading to this turn and gave a rating score for the dialogue turn in 730. The predictions of the turn by the different models were anonymized and displayed to the raters. For example, the human raters were asked to give scores between 1 and 5 for appropriateness and fluency, with a score of 5 being the best and 1 being the worst. 100 randomly selected dialogue turns were presented to 10 participants.

[0076] The ToD model is then trained on rewards R(s, a, g) using pairwise causal reward learning as described in connection with Figures 6A-6C, where mini-batch examples are randomly sampled from either human-scored examples 730 or examples scored by an automated evaluation metric 740.

[0077] It is noted that the embodiments described throughout Figures 1A-7 relate to dialogue strategy learning, however, similar embodiments may be applied to different tasks in similar settings, such as, but not limited to, end-to-end dialogue system training (e.g., dialogue state tracker, dialogue strategy and response generation, etc.).

[0078] Exemplary Performance In one embodiment, the training dataset (e.g., 210) may be a multi-turn multi-domain dataset spanning seven domains including the MultiWoz2.0 dataset, attractions, hospitals, hotels, police, taxis, trains, and an additional domain for general greetings. The dataset is created from real human conversations between travelers and shop assistants at an information center. Each dialogue is generated by a user with a defined goal that may cover 1-5 domains with up to 13 turns in the dialogue. The dataset has 10438 dialogues split into 8438 dialogues for the training set and 1000 dialogues each for the validation and test sets.

[0079] In one embodiment, the policy network 220 and / or the reward learning network 260 may adopt the neural model proposed in Zhang et al., Task-oriented dialog systems that consider multiple appropriate responses under the same context, arXiv preprint arXiv:1911.10484, 2019, as a baseline (called "DAMD"). In the pairwise causal reward learning network 260, one bi-LSTM layer for embedding actions, states, and goals, followed by a couple of MLP layers, may be used. DAMD is composed of three seq2seq generation models using GRUs. The three seq2seq models are one each for the belief state, dialogue act, and response generation modules. Then, an attention layer is used to note the output of the seq2seq models with the context vector of the previous turn for a copy over mechanism. The output is then used as a representation to predict the sequence of tokens for their respective modules. A probabilistic L sto and deterministic L det Both loss functions are used for dialogue acts. For DST and response generation, the cross entropy loss is used as from DAMD.

[0080] In one embodiment, the reward learning network 260 includes other models with additional complexity, including MinTL, a task-oriented dialogue model described in Lin et al., Mintl: Minimalist transfer learning for task-oriented dialogue systems, arXiv preprint arXiv:2009.12005, 2020. MinTL uses a large pre-trained language model BART, which uses a standard encoder-decoder transformer architecture with a bidirectional encoder and an autoregressive decoder. It is pre-trained on the task of denoising corrupted documents. BART is trained using the cross-entropy loss between the decoder output and the original documents. MinTL does not explicitly predict dialogue acts. Thus, a deterministic loss L det is used directly on the generated response, and DST preserves the losses as they are from MintTL.

[0081] In one embodiment, the database results are represented as one-hot vectors. To reduce surface-level variability in the responses, a domain-adaptive delexicalization preprocessing is employed, where delexicalized responses are generated with placeholders for specific values ​​that can be filled according to the current utterance that reference some slot values ​​provided by the system in the previous turn.

[0082] In one embodiment, the context-to-response generation task of Multi-woz2.0 can be implemented and corresponding evaluation metrics are used to measure the quality of the responses. These include notification rate and success rate, which measure the percentage of interaction, the system provided the requested information and the percentage of interaction, the system answered all the requested information, respectively, and BLEU is used to measure the fluency of the generated responses. Both of these settings use three evaluation metrics. These include 1) notification rate - measuring the percentage of interaction, the system provided the correct entity; 2) success rate - the percentage of interaction, the system answered all the requested information; and 3) BLEU - measuring the fluency of the generated responses. Additionally, a combined score (notification + success) x 0:5 + BLEU is used. All reported CASPI numbers are the median of 5 runs with different seeds.

[0083] For the metric M used in pairwise causal reward learning, the following metric is used: M:=Notification+Success+λ×BLEU. This is quite similar to the combined score used in the evaluation, and both are equal when λ=2. The hyperparameter λ is used to normalize the achievable scale of BLEU. If the success rate is used as is, it will result in a non-Markovian and stochastic turn-by-turn reward function, since the reward of the current state will depend on the performance of future states. Hence, the metric M soft A soft version of the success rate is used, where the success rate measures the proportion of requested information provided within a dialogue. The original metric, which uses a discrete variant of the success rate, is M hard It is called the reward function R(s t ,a tThe choice of action in,g) can either be a dialogue act or generate a response, and the corresponding variants of the metric are called M(act) and M(resp). All discussed variants of the metric are used to demonstrate the versatility of the method for adapting to different metrics.

[0084] Causal Aware Safety Strategy Improvement (CASPI) is compared against existing methods on the context-to-response generation task of Multiwoz2.0 in Figure 8. Existing methods include:

[0085] DAMD: Introduced by Zhang et al. is a domain-aware multi-decoder network. The method also exploits the probabilistic nature of dialogue acts by using a data augmentation technique called multi-action data augmentation. DAMD with data augmentation is denoted here as DAMD+multi-action.

[0086] (Chen et al., Semantically conditioned dialog response generation via hierarchical disentangled self-attention. (HDSA), arXiv preprint arXiv:1905.12866, 2019) HDSA proposes to use a hierarchical graph representation for dialogue acts. It uses a pre-trained 12-layer BERT model to represent dialogue acts. The predicted dialogue acts are converted into a hierarchical graph structure using a disentangled self-attention model, a 3-layer self-attention model.

[0087] SOLOIST (Peng et al., Soloist: Few-shot task-oriented dialog with a single pre-trained auto-regressive model, arXiv preprint arXiv:2005.05298, 2020). These methods are trained on turn-level data, without belief states and system actions generated in the dialogue history.

[0088] MinTL-BART (Lin et al.) introduces a Levenshtein belief spans framework that predicts only incremental changes in dialogue state per turn. It leverages pre-trained T5 and BART as the backbone for the model architecture.

[0089] HDNO (proposed by Wang et al., Modelling hierarchical structure between dialogue policy and natural language generator with option framework for task-oriented dialogue system. arXiv preprint arXiv:2006.06814, 2020) is a dialogue policy learning method for solving the context-to-response generation task of Multiwoz2.0 (Budzianowski et al., 2018b). It exploits the hierarchical nature of the dialogue act and response generation task by proposing an option-based framework of hierarchical RL and variational models to learn potential dialogue acts corresponding to natural language responses. Unlike CASPI, HDNO emphasizes the risk of sparsity in metric functions such as success rate as reward functions, but resorts to shaping a proxy reward function. It uses a Markov language model as a proxy reward function. The language model is learned independently of the metric function. CASPI forgoes reward shaping and is independent of the nature of any underspecified metric function.

[0090] CASPI is first compared against current state-of-the-art methods on the context-to-answer generation task defined by MultiWoz2.0. The results are tabulated in Figure 8. A CASPI adaptation of DAMD, CASPI(DAMD), is used for this task. CASPI(DAMD) performs better than other methods on three of the four performance criteria: success rate, notification rate, and combination score. HDSA has a better BLEU score. This rich representation of natural language by HDSA comes from the use of a large 12-layer BERT model.

[0091] Secondly, the adaptation of both CASPI(DAMD) and CASPI(MinTL) is compared on the end-to-end dialogue task defined by MultiWoz2.0. The results are tabulated in Fig. 9. CASPI(DAMD) with its lightweight model architecture without pre-training on any external corpus is able to outperform all other previous methods in all evaluation criteria. This shows that using CASPI to guide the gradient update process when the sample weights for each dialogue turn result in a model that is well aligned with the true objective of the task. CASPI(MinTL) with its robust pre-trained model outperforms CASPI(DAMD) by a large margin. This shows the ease of adapting existing methods to CASPI.

[0092] Inverse reinforcement learning combined with policy-off-policy learning and evaluation has been proven to be sample-efficient. CASPI competes with other sample-efficient techniques such as data augmentation and transfer learning as implemented by (Zhang et al.) and (Lin et al.), respectively. To validate the hypothesis, CASPI is tested against baselines in the low sample complexity regime. For the experimental setup, we use the low-resource testing strategy from (Lin et al.). CASPI models are trained on 5%, 10%, and 20% of the training data and compared with other baselines for end-to-end dialogue and context-to-response generation tasks, and Figures 10-11 list the results. For the end-to-end task, CASPI trained on only 10% of the data (MinTL) was able to outperform previous state-of-the-art methods. MinTL was trained on 100% of the data for two of the three performance metrics. For the context-to-response generation task, CASPI trained on 75% of the data (DAMD) was able to match HDNO's 100% data performance. This goes to show that having the right reward function to guide the budget of the gradient update process to reach the true objective is important in extremely low resource settings.

[0093] FIG. 12 shows examples of responses generated by different ToD models such as MinTL, CASPI(MinTL), DAMD, and SimpleTOD.

[0094] FIG. 13 shows human evaluation of the appropriateness and fluency of the criteria. Mean and variance of scores are shown. Appropriateness scores of MinTL 1301, SimpleTOD 1302, and DAMD 1304 are compared against CASPI(MinTL)Appropriateness 1303. Fluency scores of MinTL 1311, SimpleTOD 1312, and DAMD 1314 are compared against CASPI(MinTL)Fluency 1313. Evaluation results. CASPI(MinTL) 1303 outperforms all other models 1301, 1302, and 1304 in appropriateness scores. Meanwhile, fluency scores of CASPI(MinTL) 1313, MinTL 1311, and SimpleTOD 1312 are comparable to each other.

[0095] When automatic dialogue evaluation metrics are biased and do not truly reflect human objectives, however, in CASPI, these fairly same dialogue evaluation metrics are used to learn rewards R(s,a,g). To fill this gap, the following human-in-the-loop (HITL) experiment is conducted: Paired CASPI (MINTL) models with different seeds are trained on 5% of the Multiwoz2.0 dataset. Pairs of these models are then used to predict on 0.5% of the Multiwoz2.0 training data (40 dialogues), with human scores for these pairs of responses generated relative to each other. The models are then trained on rewards R(s,a,g) using pairwise causal reward learning as described in connection with Figures 6A-6C, where examples in a mini-batch are randomly sampled from either human-scored examples or examples scored by the automatic evaluation metrics, as shown in Figure 13. A new CASPI (MINTL) model is then trained on the original 5% of data and the learned R(s,a,g). Human evaluation of the trained model is performed on 24 interactions forming the test using three participants. Figure 14 shows the performance. With the HITL score in reward learning, there is an increase in performance on both human evaluation criteria: appropriateness and fluency. The human appropriateness score for 5% data CASPI (MINTL) is increased from 1401 to 1402, now comparable to 100% data DAMD. The fluency score also increased from 1411 to 1412. This shows the versatility of pairwise causal reward learning. With sufficient richness of the neural network used, pairwise causal reward learning can be generalized to unknown interaction evaluation criteria.

[0096] FIG. 15 shows the example reward values ​​R(s) predicted for each turn by pairwise causal reward learning for the same conversation between the traveler and the information center agent shown in FIG. 1B. t ,a t,g). We observe that turn #3 received the highest reward and retrospectively realize that this is the turn where the trade occurs, which is an important and risk-averse turn in the dialogue, and it is captured by the success rate of the automatic evaluation metric. Turn #2 gets the next best reward, which captures the important information need for the trade to occur in turn #3. Turn #4 gets a lower ranking reward than turns #3 and #2 because it does not contribute much to the success of the conversation, other than finesse. Note that turn #4 appears in almost all conversations and in supervised learning, and it is typical that it receives the highest share of the gradient. The learned reward redistributes the gradient budget to be aligned with the success of the dialogue objective.

[0097] Figure 16 illustrates different types of behaviors that CASPI agents sometimes exhibit, especially when trained in low sample regimes. Greedy Agent: In certain domains, agents have a tendency to book a service before it has collected all the necessary information or before the user has requested or agreed to book the service. The first example in Figure 16 demonstrates this behavior. Here, the user requests for a taxi and the agent books a taxi before enough information, such as the destination or time of departure, has been collected. This occurs because there are gaps in the automatic evaluation metrics. Low BLEU scores and relatively high notification and success rates may indicate greedy agent behavior. Other reasons for low BLEU scores include a lack of diversity in responses or malformed responses.

[0098] Cautious Agent: The agent tends to be cautious by providing long-lasting replies packed with more information than is required. The agent tends to do this so as not to risk losing reward through information rate. This behavior is demonstrated in the second example in Figure 16. These subtle behaviors demonstrate gaps in the automatic evaluation metrics, which can be reduced by using human-in-the-loop evaluation as shown in Figure 7.

[0099] Some examples of computing devices, such as computing device 100, may include a non-transitory, tangible, machine-readable medium that includes executable code that, when executed by one or more processors (e.g., processor 110), may cause the one or more processors to perform the processes of method 200. Some common forms of machine-readable media that may include the processes of method 200 are, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, a punch card, a paper tape, any other physical medium having a pattern of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other memory chip or cartridge, and / or any other medium adapted to be read by a processor or a computer.

[0100] This description and the accompanying drawings illustrating aspects, embodiments, implementations, or applications of the invention should not be construed as limiting. Various mechanical, compositional, structural, electrical, and operational changes can be made without departing from the spirit and scope of this description and claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail so as not to obscure the embodiments of the present disclosure. Like numbers in two or more figures represent the same or similar elements.

[0101] In this description, certain details are described to describe some embodiments consistent with the present disclosure. Numerous specific details are described to provide a thorough understanding of the embodiments. However, it will be apparent to one skilled in the art that some embodiments can be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative rather than limiting. Those skilled in the art can realize other elements that are not specifically described herein but are within the scope and spirit of the present disclosure. Moreover, to avoid unnecessary repetition, one or more features illustrated and described in connection with one embodiment may be incorporated in other embodiments, unless specifically noted otherwise or if one or more features render the embodiment non-functional.

[0102] Although exemplary embodiments have been shown and described, a wide range of modifications, changes, and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be employed without the corresponding use of other features. Those skilled in the art will recognize many variations, alternatives, and modifications. Accordingly, the scope of the present invention should be limited only by the scope of the following claims, which claims should be appropriately construed broadly and consistent with the scope of the embodiments disclosed herein.

Claims

1. 1. A method for improving causal-aware safety policies in task-oriented learning, comprising: receiving a training dataset including a plurality of interactions, the plurality of interactions including a first interaction rollout generated according to a potential probabilistic behavior strategy; repeatedly sampling the training data set a number of times to generate a number of training subsets and a number of validation subsets; training a task-oriented dialogue model based on cross-entropy loss using training data in a first training subset of the plurality of training subsets; generating a predicted interaction rollout based on interaction data in a first validation subset of the plurality of validation subsets using the task-oriented interaction model; adding the predicted interaction rollout to a pairwise causal learning subset; Sampling pairs of interaction rollouts from the pairwise causal learning subset; training the task-oriented dialogue model based on a binary cross-entropy loss between a favorable probability between the pair of dialogue rollouts and a normalized metric score based on the pair of dialogue rollouts; The method includes:

2. Extracting a dataset from the plurality of training subsets or the plurality of validation subsets; training the task-oriented dialogue model by minimizing a cross-entropy of predicted dialogue behaviors conditional on a current state of dialogue according to a target strategy using dialogue data from the retrieved dataset; The method of claim 1 further comprising:

3. The method of claim 2 , wherein the predicted interaction rollout is iteratively generated according to the targeting strategy by iterating through the multiple validation subsets.

4. 2. The method of claim 1 , wherein training the task-oriented dialogue model based on a binary cross-entropy loss is performed by repeatedly sampling different pairs of dialogue rollouts from the pairwise causal learning subset until convergence in training is reached, and retraining the task-oriented dialogue model based on the binary cross-entropy loss.

5. Training the task-oriented dialogue model based on binary cross entropy loss includes: encoding the goals, belief states, and dialogue acts or response sequences in each dialogue turn of each of the sampled pairs of the dialogue rollout into three coded representations through three bi-LSTM layers, respectively; concatenating the three encoded representations; feeding the concatenated encoded representation to one or more feedforward layers that generate reward predictions for each dialogue turn; summing the generated reward predictions into an interaction reward for each one of the sampled pairs of the interaction rollout; calculating the favorable probability between the pair of interaction rollouts based on interaction rewards corresponding to the sampled pairs of interaction rollouts; Calculating the binary cross-entropy loss between the favorable probability between the pair of interaction rollouts and the normalized metric score based on the pair of interaction rollouts; The method of claim 1 further comprising:

6. The method of claim 5 , wherein the favorable probability between the pair of interaction rollouts is calculated using a normalization or softmax function.

7. iteratively training the task-oriented dialogue model based on a policy optimization loss that optimizes for a target policy using the training dataset until training convergence is reached; The method of claim 1 further comprising:

8. The policy optimization loss is generating a first predicted behavior distribution based on a current state of the interaction according to a target strategy with the task-oriented interaction model; calculating a first discounted sum of future rewards based on discount parameters and reward functions of the actions and states of the interaction according to the potential behavior strategy; calculating a first loss objective based on a first expectation of the first discounted sum of future rewards and the first predicted action distribution, the first expectation being obtained over a probability distribution of the states and the actions according to the underlying probabilistic behavior strategy; generating, with the task-oriented dialogue model, a second predicted behavior distribution based on a current observation from a time series of observations according to the target strategy; calculating a second discounted sum of future rewards based on the discount parameters and the reward function for a particular rollout, the second discounted sum of future rewards being a collapsed near-deterministic approximation of the first discounted sum of future rewards; calculating a second loss objective based on a second expectation of the second discounted sum of future rewards and the second predicted behavior distribution, the second expectation being taken on an average of the observations across the training data set; and calculating a sum of the first loss objective and the second loss objective; computing a gradient update component based on the learned rewards from the reward function of the actions and states of the dialogue and gradients of the target policy of the actions conditioned on the states and parameters of the task-oriented dialogue model; updating the parameters of the task-oriented dialogue model using the gradient update components; and The method of claim 7, wherein the calculated value is:

9. 1. A system for improving causal-aware safety policies in task-oriented learning, comprising: a communications interface for receiving a training dataset including a plurality of interactions, the plurality of interactions including a first interaction rollout generated according to a potential probabilistic behavior strategy; a memory storing a plurality of processor-executable instructions; a processor that reads the plurality of processor-executable instructions from the memory and performs operations on the instructions; and said operation includes repeatedly sampling the training data set a number of times to generate a number of training subsets and a number of validation subsets; training a task-oriented dialogue model based on an entropy loss using training data in a first training subset of the plurality of training subsets; generating a predicted interaction rollout from interaction data in a first validation subset of the plurality of validation subsets using the task-oriented interaction model; adding the predicted interaction rollout to a pairwise causal learning subset; Sampling pairs of interaction rollouts from the pairwise causal learning subset; training the task-oriented dialogue model based on a binary cross-entropy loss between a preferred probability between the pair of dialogue rollouts and a normalized metric score based on the pair of dialogue rollouts; Including, the system.

10. 1. A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for improving causal-aware safety strategies in task-oriented learning, the instructions being executable by a processor to perform operations, the operations comprising: receiving a training dataset including a plurality of interactions, the plurality of interactions including a first interaction rollout generated according to a potential probabilistic behavior strategy; repeatedly sampling the training data set a number of times to generate a number of training subsets and a number of validation subsets; training a task-oriented dialogue model based on an entropy loss using training data in a first training subset of the plurality of training subsets; generating a predicted interaction rollout from interaction data in a first validation subset of the plurality of validation subsets using the task-oriented interaction model; adding the predicted interaction rollout to a pairwise causal learning subset; Sampling pairs of interaction rollouts from the pairwise causal learning subset; training the task-oriented dialogue model based on a binary cross-entropy loss between a preferred probability between the pair of dialogue rollouts and a normalized metric score based on the pair of dialogue rollouts; 16. A non-transitory processor-readable storage medium comprising:

Images