Expert mixing method for dialogue management based on reinforcement learning

Abstract

Systems and methods for dialog response prediction may utilize multiple machine learning language models to generate multiple candidate outputs, which may be processed by a dialog management model to determine a predicted dialog response. The plurality of machine learning language models may include a plurality of experts trained for different intent, emotion, and / or task. The selected particular candidate output may be selected by the dialog management model based on semantics determined from the language representation. The language representation may be a representation generated by processing a session history of the session to determine a session semantic.

Description

[0001] Related Applications

[0002] This application claims priority to and the benefit of U.S. Non-Provisional Patent Application No. 18 / 173,495, filed on February 23, 2023, which claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 343,766, filed on May 19, 2022. U.S. Non-Provisional Patent Application No. 18 / 173,495 and U.S. Provisional Patent Application No. 63 / 343,766 are incorporated herein by reference in their entireties. Technical Field

[0003] The present disclosure generally relates to generating a dialog response using a dialog management model that selects candidate responses from multiple expert model outputs. More specifically, the present disclosure relates to obtaining a conversation history, processing the conversation history using a language encoding model, multiple expert language models, and a dialog management model to generate a dialog response. Background Art

[0004] Advances in language models include the generation of large language models with billions of learned parameters. Despite having a large corpus of learned parameters, language models cannot correctly generate responses taking into account the semantics of the conversation history. Language models can struggle with dialogue management and the ability to conduct rich, engaging conversations. Additionally, processing and retraining of large language models can be computationally expensive. Summary of the invention

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0006] An example aspect of the present disclosure relates to a computing system. The system may include one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations may include obtaining input data. The input data may include session data describing a conversation. The operations may include processing the input data with an encoder model to generate a language representation. In some implementations, the language representation may include a potential embedding associated with the conversation. The encoder model may have been trained to map the encoded conversation to a potential distribution. The operations may include processing the language representation with one or more machine learning language models to generate one or more candidate outputs. The one or more candidate outputs may include one or more candidate utterances. The operations may include processing the one or more candidate outputs and the language representation with a dialogue management model to generate dialogue planning data. The operations may include generating a predicted dialogue response based on the dialogue planning data. In some implementations, the predicted dialogue response may include one or more predicted words associated with the one or more candidate utterances.

[0007] In some implementations, processing the language representation with the one or more machine learning language models to generate the one or more candidate outputs may include processing the language representation with multiple expert language models to generate multiple candidate outputs. The multiple candidate outputs may be associated with multiple candidate utterances. The predicted dialogue response may include a selected candidate utterance associated with a selected candidate output from the multiple candidate outputs. The selected candidate output may be selected by a dialogue management model. In some implementations, the one or more machine learning language models may include an expert language model trained on a specific skill so that the one or more candidate utterances indicate the specific skill. The dialogue management model may have been trained with reinforcement learning. Reinforcement learning may optimize predictions for complete conversations. The predicted dialogue response may include predicted utterances. The predicted utterances may be responsive to input data.

[0008] In some implementations, the language representation may describe the semantics of a conversation history of a conversation. The conversation history may include multiple exchanged text strings. The language representation may be associated with a latent space distribution of a learned latent space. In some implementations, the language representation may be associated with a learned distribution of the latent space. The learned distribution may be associated with a particular emotion. In some implementations, the dialog management model may be configured to determine a conversation intent based on the language representation and select a particular candidate output based on its association with the conversation intent.

[0009] Another example aspect of the present disclosure relates to a computer-implemented method. The method may include obtaining, by a computing system including one or more processors, session data. The session data may describe a session history. The method may include processing, by the computing system, the session data with a language encoding model to generate a language representation. The language representation may describe semantics associated with the session history. The method may include processing, by the computing system, the language representation with a plurality of machine learning language models to generate a plurality of candidate outputs. The plurality of machine learning language models may have been trained based on a learned emotion distribution associated with a latent space. The method may include processing, by the computing system, the language representation and the plurality of candidate outputs with a dialog management model to determine a dialog response.

[0010] In some implementations, a first machine learning language model in the plurality of machine learning language models may have been trained for a first skill. A second machine learning language model in the plurality of machine learning language models may have been trained for a second skill. The dialog management model may have been trained to select a particular candidate output in the plurality of candidate outputs based at least in part on a language representation. The plurality of machine learning language models may have been trained with benchmark real training data. In some implementations, the language encoding model may include a random encoder model. The random encoder model may include an encoder and a latent space distribution. The random encoder model may map a lemma conversation history to a latent space to generate a parameterized Gaussian distribution. In some implementations, the plurality of machine learning language models may include a plurality of expert models associated with a plurality of emotions. The plurality of machine learning language models may include a plurality of expert models associated with a plurality of tasks.

[0011] Another example aspect of the present disclosure relates to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations. The operations may include obtaining training data. The training data may include training session data. The operations may include processing the training session data with a language encoding model to generate a language representation. The operations may include processing the language representation with multiple machine learning language models to generate multiple candidate utterances. The operations may include processing the multiple candidate utterances with a dialogue management model to determine a predicted dialogue response. The operations may include providing a predicted dialogue response to a user computing system and receiving additional conversation data from the user computing system. The additional conversation data may describe a conversation that occurs after the predicted dialogue response. The operations may include adjusting one or more parameters of the dialogue management model based on the additional conversation data.

[0012] In some implementations, the operations may further include determining satisfaction data based at least in part on additional conversation data. The satisfaction data may describe a level of satisfaction with a predicted conversation response. The satisfaction data may be determined based at least in part on conversation engagement. The operations may include adjusting one or more parameters of a conversation management model based on the satisfaction data. In some implementations, the training data may include one or more benchmark real utterances. The operations may further include evaluating a loss function that evaluates the difference between a particular candidate utterance in the plurality of candidate utterances and the one or more benchmark real utterances, and adjusting one or more parameters of at least one of the plurality of machine learning language models based at least in part on the loss function. The language encoding model may include a variational autoencoder model. In some implementations, the plurality of machine learning language models may include a plurality of language decoder models. The language representation may include an embedding sequence.

[0013] Another example aspect of the present disclosure relates to a computing system. The system may include one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations may include obtaining sequence data. The sequence data may describe a sequence. The operations may include processing the sequence data with a coding model to generate a coded representation. The coded representation may describe the determined semantics of the sequence. The operations may include processing the coded representation with multiple machine learning expert models to generate multiple candidate outputs. The multiple machine learning expert models may be associated with multiple different intents. The operations may include processing the multiple candidate outputs and the coded representation with a management model to generate planning data and generating predicted outputs based on the planning data. The predicted output may include at least a portion of one or more of the multiple candidate outputs.

[0014] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0015] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] A detailed discussion of the embodiments for those skilled in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0017] Figure 1A A block diagram of an example computing system that performs dialog response prediction according to an example embodiment of the present disclosure is depicted.

[0018] Figure 1B A block diagram of an example computing device performing dialog response prediction is depicted according to an example embodiment of the present disclosure.

[0019] Figure 1C A block diagram of an example computing device performing dialog response prediction is depicted according to an example embodiment of the present disclosure.

[0020] Figure 2 Depicted is a block diagram of an example mixture-of-experts language model according to an example embodiment of the present disclosure.

[0021] Figure 3 Depicted is a block diagram of an example mixture of experts language model according to an example embodiment of the present disclosure.

[0022] Figure 4 Depicted is a block diagram of an example language model training according to an example embodiment of the present disclosure.

[0023] Figure 5 Depicted is a block diagram of an example expert language model training according to an example embodiment of the present disclosure.

[0024] Figure 6 A flow chart is depicted of an example method of performing dialog response prediction according to an example embodiment of the present disclosure.

[0025] Figure 7 A flow chart of an example method of performing dialog response prediction according to an example embodiment of the present disclosure is depicted.

[0026] Figure 8 Depicted is a flow chart of an example method of performing language model training according to an example embodiment of the present disclosure.

[0027] Fig. 9 Depicted is a flow chart of an example method of performing sequence training according to an example embodiment of the present disclosure.

[0028] Reference numerals that are repeated across multiple drawings are intended to identify like features in various implementations. DETAILED DESCRIPTION

[0029] Overview

[0030] In general, the present disclosure relates to generating a dialog response using a dialog management model that selects a candidate response from a plurality of expert model outputs. More specifically, the present disclosure relates to obtaining a conversation history, processing the conversation history using a language encoding model, a plurality of expert language models, and a dialog management model to generate a dialog response. For example, systems and methods may include obtaining input data. The input data may include a language representation. Alternatively and / or additionally, the input data may include conversation data describing a conversation, and the conversation data may be processed by an encoder model (e.g., a language encoding model) to generate a language representation. In some implementations, the language representation may be associated with the conversation. The language representation may be processed with one or more machine learning language models to generate one or more candidate outputs. The one or more candidate outputs may include one or more candidate utterances. The one or more candidate outputs and the language representation may be processed with a dialog management model to generate dialog planning data. In some implementations, a predicted dialog response may be generated based on the dialog planning data. The predicted dialog response may include one or more predicted terms associated with the one or more candidate utterances.

[0031] Systems and methods may obtain input data. The input data may include a language representation. Alternatively and / or additionally, the input data may include session data. The session data may be processed by an encoder model (e.g., a stochastic encoder trained to map encoded sessions to a latent distribution) to generate a language representation. The language representation may be associated with a session. In some implementations, the language representation may describe the semantics of a session history of a session. The session history may include multiple exchanged text strings. Additionally and / or alternatively, the language representation may be associated with a latent space distribution. In some implementations, the language representation may have been generated by processing the session history with a language encoding model.

[0032] The language representation may be processed with one or more machine learning language models to generate one or more candidate outputs. The one or more candidate outputs may include one or more candidate utterances. In some implementations, the one or more machine learning language models may include an expert language model trained on a specific skill, so that the one or more candidate utterances may indicate the specific skill. Each of the one or more machine learning language models may include a plurality of expert sub-models (and / or blocks) that may be trained to change emotions and / or tasks. Additionally and / or alternatively, the one or more machine learning language models may generate a plurality of candidate utterances, wherein each candidate utterance is associated with one or more emotions and / or tasks of an expert sub-model (and / or block (e.g., expert head)).

[0033] In some implementations, processing the language representation with the one or more machine learning language models to generate the one or more candidate outputs may include processing the language representation with multiple expert language models to generate multiple candidate outputs. The multiple candidate outputs may be associated with multiple candidate utterances. In some implementations, the predicted dialog response may include a selected candidate utterance associated with a selected candidate output from the multiple candidate outputs. The selected candidate output may be selected by the dialog management model.

[0034] The one or more candidate outputs and the language representation may be processed with a dialog management model to generate dialog planning data. The dialog management model may have been trained using reinforcement learning. Reinforcement learning may optimize predictions for complete conversations. In some implementations, the dialog management model may be configured to determine the intent of the conversation based on the language representation and to select a particular candidate output based on its association with the intent of the conversation. In some implementations, the reinforcement learning method may include an offline reinforcement learning method. Alternatively and / or additionally, the offline reinforcement learning method may include conservative Q-learning (Kumar et al., Conservative q-learning for offline reinforcement learning, Advances in Neural Information Processing Systems, 33:1179–1191, 2020). Conservative Q-learning may include a regularization scheme that learns a "conservative" Q function that reduces the bounds of the true Q function.

[0035] A predicted dialogue response may then be generated based on the dialogue planning data. The predicted dialogue response may include one or more predicted terms associated with the one or more candidate utterances. The predicted dialogue response may include the predicted utterance. In some implementations, the predicted utterance may be responsive to the input data.

[0036] Alternatively and / or additionally, the systems and methods may include obtaining conversation data. The conversation data may describe a conversation history. The conversation data may be processed with a language encoding model to generate a language representation. In some implementations, the language representation may describe semantics associated with the conversation history. The language representation may be processed with a plurality of machine learning language models to generate a plurality of candidate outputs. In some implementations, the language representation and the plurality of candidate outputs may be processed with a dialog management model to determine a dialog response based on the plurality of candidate outputs.

[0037] Specifically, the system and method can obtain session data. The session data can describe the session history.

[0038] The session data may be processed with a language encoding model to generate a language representation. The language representation may describe semantics associated with the session history. In some implementations, the language encoding model may include a stochastic encoder model. The stochastic encoder model may include an encoder and a latent space distribution. In some implementations, the stochastic encoder model may map the lemmatized session history to a latent space to generate a parameterized Gaussian distribution.

[0039] The language representation may be processed with a plurality of machine-learned language models to generate a plurality of candidate outputs. In some implementations, a first machine-learned language model in the plurality of machine-learned language models may have been trained for a first skill (e.g., a first intent), and a second machine-learned language model in the plurality of machine-learned language models may have been trained for a second skill (e.g., a second intent). Alternatively and / or additionally, the plurality of machine-learned language models may include a plurality of expert models associated with a plurality of emotions. In some implementations, the plurality of machine-learned language models may include a plurality of expert models associated with a plurality of tasks.

[0040] The language representation and the plurality of candidate outputs may be processed with a dialog management model to determine a dialog response based on the plurality of candidate outputs. The dialog management model may be trained to select a particular candidate output from the plurality of candidate outputs based at least in part on the language representation. In some implementations, the plurality of machine learning language models may be trained with benchmark real training data.

[0041] For example, systems and methods may include obtaining training data. The training data may include training session data describing a set of example sessions.

[0042] The training session data may be processed with a language encoding model to generate a language representation. In some implementations, the language encoding model may include a variational autoencoder model. The language representation may include an embedding sequence.

[0043] The language representation may be processed with a plurality of machine learning language models to generate a plurality of candidate utterances. The plurality of machine learning language models may include a plurality of language decoder models.

[0044] The plurality of candidate utterances may be processed with a dialog management model to determine a predicted dialog response.

[0045] The predicted conversation responses may then be provided to a user computing system.The user computing system may be associated with the users involved in the conversation.

[0046] The system and method may then receive additional session data from the user computing system. The additional session data may describe the session that occurred after the predicted dialog response.

[0047] One or more parameters of the dialog management model may be adjusted based on the additional session data.

[0048] In some implementations, the systems and methods may determine and / or generate satisfaction data based at least in part on the additional conversation data. The satisfaction data may describe a level of satisfaction with the predicted conversation response. In some implementations, the satisfaction data may be determined based at least in part on conversation engagement. The systems and methods may then adjust one or more parameters of the conversation management model based on the satisfaction data.

[0049] Additionally and / or alternatively, the training data may include one or more benchmark real utterances. The systems and methods may then evaluate a loss function that evaluates the difference between a particular candidate utterance in the plurality of candidate utterances and the one or more benchmark real utterances. One or more parameters of at least one of the plurality of machine learning language models may be adjusted based at least in part on the loss function.

[0050] Although the present disclosure discusses systems and methods for dialog responses, the systems and methods disclosed herein may generally be applied to other tasks including other sequence processing tasks. For example, the systems and methods may include obtaining sequence data. The sequence data may describe a sequence. The sequence data may be processed with a coding model to generate a coded representation. In some implementations, the coded representation may describe the determined semantics of the sequence. The coded representation may be processed with multiple machine learning expert models to generate multiple candidate outputs. In some implementations, the multiple machine learning expert models may be associated with multiple different intents. The systems and methods may include processing the multiple candidate outputs and the coded representation with a management model to generate planning data. A predicted output may then be generated based on the planning data. In some implementations, the predicted output may include at least a portion of one or more candidate outputs of the multiple candidate outputs. In some implementations, the input sequence data may include waveform data (e.g., waveform data describing audio such as speech), and the output may be waveform data generated in response to the input waveform data. Alternatively and / or additionally, the sequence data may include a statistical sequence, a waveform sequence, a pixel sequence, and / or various other sequences. The sequence may include a stream of conversation (verbal and / or via text), a news feed, a numerical sequence, a music sequence, a light sequence, a social media stream, and / or a sensor data sequence.

[0051] The system and method can be used for chatbots, communication applications, help services and / or search applications. For example, a user may input one or more messages, which may be processed by an encoder model to generate a language representation. The language representation may describe a task and / or emotion associated with one or more messages. The language representation may be processed by one or more language models to generate a plurality of candidate utterances, which may include a plurality of expert models (and / or machine learning heads). The dialogue management module may then process the language representation and the plurality of candidate utterances to generate a predicted dialogue response. The dialogue management block may determine one or more specific candidate utterances of the plurality of utterances to be utilized based on the emotion and / or task associated with the language representation (e.g., the language representation may be an embedding that falls within a learned potential distribution associated with a particular emotion and / or task). The candidate utterance may be based on one or more manually configured emotion sequences (e.g., the dialogue management module may be configured to provide additional weights to interrogative candidate utterances once a threshold number of messages are received). Alternatively and / or additionally, the candidate utterance determination may be based on one or more emotion sequences learned by one or more parameters of the machine learning model of the training dialogue management block.

[0052] The one or more machine learning models may be trained via unsupervised and / or semi-supervised learning based on training and / or via retraining based on sessions that meet one or more criteria. The one or more criteria may include an improvement in the sentiment of messages received from the user. Additionally and / or alternatively, the one or more criteria may include the user performing a particular action (e.g., making a reservation, marking a service complete, purchasing an item, etc.).

[0053] In some implementations, contextual information associated with the user and / or a particular topic associated with the message can be obtained and used to generate a predicted conversation response.

[0054] The one or more language models may include one or more generative models, which may include a Transformer model and / or one or more autoregressive language models. In some implementations, systems and methods may include end-to-end black-box optimization.

[0055] The multiple expert language models and / or dialogue management models can be trained to provide natural conversationalist responses. The one or more machine learning models can be trained for word-by-word utterance generation. Reinforcement learning can be used to determine the conversation level. Additionally and / or alternatively, the dialogue management model can be trained to provide empathetic utterance sequence patterns.

[0056] The systems and methods of the present disclosure provide a variety of technical effects and benefits. As an example, the systems and methods can be used to generate conversation history-aware automatic dialogue responses. For example, the systems and methods can be trained and / or configured to process conversation histories between at least two entities and generate semantically-aware dialogue responses. In addition, the use of the multiple machine learning expert language models can utilize multiple candidate responses associated with different emotions, intentions, or tasks to provide the dialogue management model with multiple options to consider when determining dialogue responses.

[0057] Another example technical effect and benefit may include utilizing multiple expert models and management models to provide semantically aware outputs without the computational cost of a large Transformer model. Specifically, the systems and methods disclosed herein may process sequence data to generate encoded data. The encoded data may be processed by multiple expert models to generate multiple candidate outputs associated with multiple different tasks and / or intents. The multiple candidate outputs and the encoded representations may then be processed by the management model to determine one or more candidate outputs to be used for predicted outputs. The predicted output may be based on the semantics of the sequence determined from the encoded representation. The use of the expert model in conjunction with the management model may reduce the amount of parameters that are retrained for each new context or meaning.

[0058] Another example technical effect and benefit relates to improved computing efficiency and improvements in the operation of computing systems. For example, a technical benefit of the systems and methods disclosed herein is the ability to reduce the computing resources required to retrain and use one or more machine learning language models. Specifically, the systems and methods disclosed herein can generate conversation-aware dialog responses without the heavy computational costs of large language models. The systems and methods disclosed herein can allow expert language models and dialog management models to be centrally trained and / or retrained without the computational cost of retraining billions of parameters. For example, instead of retraining the parameters of a large language model for each new semantic context, the systems and methods disclosed herein can utilize multiple expert models and dialog management models to generate semantically-aware dialog responses without the computational burden of retraining.

[0059] In some implementations, the systems and methods disclosed herein can process a session history and output a session-aware response and / or a set of responses that can result in reduced round-trips, thereby reducing computing resources (e.g., bandwidth) utilized by a computing system.

[0060] Referring now to the accompanying drawings, example embodiments of the present disclosure will be discussed in greater detail.

[0061] Example Apparatus and Systems

[0062] Figure 1AA block diagram of an example computing system 100 that performs dialog response prediction according to an example embodiment of the present disclosure is depicted. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 that are communicatively coupled via a network 180.

[0063] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smart phone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0064] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 executed by the processor 112 to cause the user computing device 102 to perform operations.

[0065] In some implementations, the user computing device 102 may store or include one or more expert mixture language models 120. For example, the expert mixture language model 120 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Figures 2 to 4 An example expert mixture language model 120 is discussed.

[0066] In some implementations, one or more expert mixture language models 120 may be received from the server computing system 130 via the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single expert mixture language model 120 (e.g., to perform parallel dialog response prediction across multiple instances of a conversation prompt).

[0067] More specifically, the expert mixture language model may process conversation data (eg, conversation data describing a conversation history) to generate predicted dialog responses that are semantically aware of the conversation history.

[0068] Additionally or alternatively, one or more expert hybrid language models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the expert hybrid language model 140 may be implemented by the server computing system 140 as part of a web service (e.g., a dialog response prediction service). Thus, one or more models 120 may be stored and implemented at the user computing device 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0069] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other devices by which a user can provide user input.

[0070] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 executed by the processor 132 to cause the server computing system 130 to perform operations.

[0071] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. Where the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0072] As described above, the server computing system 130 may store or otherwise include one or more machine learning expert mixture language models 140. For example, the model 140 may be or may otherwise include a variety of machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Figures 2 to 4 Example model 140 is discussed.

[0073] User computing device 102 and / or server computing system 130 may train models 120 and / or 140 via interaction with training computing system 150 communicatively coupled via network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0074] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or multiple processors operatively connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0075] The training computing system 150 may include a model trainer 160 that trains the machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques (e.g., such as error back propagation). For example, a loss function may be back propagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update parameters over multiple training iterations.

[0076] In some implementations, performing error back-propagation may include performing truncated back-propagation through time.The model trainer 160 may perform a variety of generalization techniques (eg, weight decay, dropout, etc.) to improve the generalization capabilities of the model being trained.

[0077] Specifically, the model trainer 160 may train the expert mixture language model 120 and / or 140 based on a set of training data 162. The training data 162 may include, for example, training session data (e.g., data describing example session histories associated with different semantics), benchmark real utterances, and / or one or more benchmark real labels.

[0078] In some implementations, if the user has provided consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 based on user-specific data received from the user computing device 102. In some cases, this process may be referred to as personalizing the model.

[0079] The model trainer 160 includes computer logic for providing the desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software that controls a general purpose processor. For example, in some implementations, the model trainer 160 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer executable instructions stored in a tangible computer readable storage medium such as RAM, a hard disk, or an optical or magnetic medium.

[0080] The network 180 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over the network 180 may be conducted over any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0081] The machine learning models described in this specification can be used for a variety of tasks, applications, and / or use cases.

[0082] In some implementations, the input of the machine learning model of the present disclosure may be text or natural language data. The machine learning model may process the text or natural language data to generate an output. As an example, the machine learning model may process the natural language data to generate a language encoding output. As another example, the machine learning model may process the text or natural language data to generate a potential text embedding output. As another example, the machine learning model may process the text or natural language data to generate a translation output. As another example, the machine learning model may process the text or natural language data to generate a classification output. As another example, the machine learning model may process the text or natural language data to generate a text segmentation output. As another example, the machine learning model may process the text or natural language data to generate a semantic intent output. As another example, the machine learning model may process the text or natural language data to generate an amplified text or natural language output (e.g., text or natural language data with higher quality than the input text or natural language, etc.). As another example, the machine learning model may process the text or natural language data to generate a prediction output.

[0083] In some implementations, the input to the machine learning model of the present disclosure may be speech data. The machine learning model may process the speech data to generate an output. As an example, the machine learning model may process the speech data to generate a speech recognition output. As another example, the machine learning model may process the speech data to generate a speech translation output. As another example, the machine learning model may process the speech data to generate a potential embedding output. As another example, the machine learning model may process the speech data to generate an encoded speech output (e.g., an encoded and / or compressed representation of the speech data, etc.). As another example, the machine learning model may process the speech data to generate an amplified speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a predicted output.

[0084] In some implementations, the input of the machine learning model of the present disclosure may be latent coded data (e.g., a latent space representation of an input, etc.). The machine learning model may process the latent coded data to generate an output. As an example, the machine learning model may process the latent coded data to generate a recognition output. As another example, the machine learning model may process the latent coded data to generate a reconstruction output. As another example, the machine learning model may process the latent coded data to generate a search output. As another example, the machine learning model may process the latent coded data to generate a re-clustering output. As another example, the machine learning model may process the latent coded data to generate a prediction output.

[0085] In some implementations, the input to the machine learning model of the present disclosure may be statistical data. The machine learning model may process the statistical data to generate an output. As an example, the machine learning model may process the statistical data to generate an identification output. As another example, the machine learning model may process the statistical data to generate a prediction output. As another example, the machine learning model may process the statistical data to generate a classification output. As another example, the machine learning model may process the statistical data to generate a segmentation output. As another example, the machine learning model may process the statistical data to generate a segmentation output. As another example, the machine learning model may process the statistical data to generate a visualization output. As another example, the machine learning model may process the statistical data to generate a diagnostic output.

[0086] Figure 1A An example computing system that can be used to implement the present disclosure is shown. Other computing systems may also be used. For example, in some implementations, the user computing device 102 may include a model trainer 160 and a training data set 162. In such implementations, the model 120 may be both trained and used locally at the user computing device 102. In some of such implementations, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0087] Figure 1B Depicted is a block diagram of an example computing device 10 performing in accordance with an example embodiment of the present disclosure. Computing device 10 may be a user computing device or a server computing device.

[0088] Computing device 10 includes multiple applications (e.g., Application 1 to Application N). Each application contains its own machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0089] like Figure 1B As shown, each application can communicate with multiple other components of the computing device (e.g., such as one or more sensors, context managers, device state components, and / or additional components). In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to the application.

[0090] Figure 1C Depicted is a block diagram of an example computing device 50 performing in accordance with an example embodiment of the present disclosure. Computing device 50 may be a user computing device or a server computing device.

[0091] The computing device 50 includes a plurality of applications (e.g., Application 1 to Application N). Each application communicates with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a public API across all applications).

[0092] The central intelligence layer includes multiple machine learning models. For example, Figure 1C As shown, a corresponding machine learning model (e.g., model) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included in the operating system of the computing device 50 or is otherwise implemented by the operating system.

[0093] The central intelligence layer may communicate with the central device data layer. The central device data layer may be a centralized data repository for the computing device 50. Figure 1C As shown, the central device data layer can communicate with multiple other components of the computing device (e.g., such as one or more sensors, context managers, device state components, and / or additional components). In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0094] Example Model Layout

[0095] Figure 2 A block diagram of an example expert mixture language model 200 is depicted according to an example embodiment of the present disclosure. Specifically, Figure 2 A tokenized conversation history 202 is depicted being processed by a stochastic encoder 204 that maps the tokenized conversation history 202 to a latent space. The mapped data may be used to construct a parameterized d-dimensional Gaussian distribution. The decoder model may then generate a plurality of candidate utterances 214. The plurality of candidate utterances 214 may be associated with different emotions and / or different purposes. From the plurality of candidate utterances 214, a predicted utterance 218 may be selected by a dialog management model 216. The selection may be based on a predicted satisfaction score for each candidate utterance. In some implementations, the predicted utterance 218 may include one or more candidate utterances.

[0096] For example, conversation data describing a conversation history 202 may be obtained. The conversation history 202 may include one or more messages associated with one or more emotions. The conversation history may be processed with an encoder model (e.g., a stochastic encoder 204) to generate a potential embedded language representation 206 describing the conversation history. Multiple Gaussian distributions 208 may then be sampled (and / or utilized) to generate multiple sampled embeddings 210 associated with multiple learned distributions, which may be associated with multiple different emotions. The multiple sampled embeddings 210 and / or language representation 206 may then be decoded with a decoder model 212 to generate multiple candidate utterances 214, which may be associated with multiple candidate utterances in response to at least one message of the conversation history 202. The multiple candidate utterances 214 and the language representation 206 may then be processed with a dialog management model 216 to generate and / or determine a predicted utterance 218 to be utilized in a response.

[0097] The example pipeline 220 may include obtaining a message from a user. The message may be encoded and then processed by multiple expert language models and / or by multiple expert heads of a large language model to generate multiple candidate utterances in response to the message. Different expert language models may be associated with different emotions, which may include "inquiring" (e.g., a language model trained to ask direct and / or specific questions), "exploratory" (e.g., a language model trained to explore options and / or ask general questions), "optimistic" and / or "angry". The multiple candidate utterances and language representations may be processed by a dialog management model, which may determine the candidate utterances to be used based on one or more learned sequences that result in satisfaction. In some implementations, the dialog management model may utilize a subset of the multiple candidate utterances, and may rank the selected utterances based on one or more learned sequences associated with the determined emotion of the user input message.

[0098] Figure 3 A block diagram of an example expert mixture language model 300 is depicted according to an example embodiment of the present disclosure. Specifically, the expert mixture model 300 can process conversation history data 302 to generate a predicted dialogue response 322. For example, the conversation history 302 can be processed by a language encoding model 304 to generate a latent representation, which can then be processed by multiple expert models (e.g., 310, 312, 314, and / or 316) to generate multiple candidate outputs. The multiple candidate outputs can then be processed with a dialogue management model 320 to generate a predicted dialogue response 322.

[0099] Throughout the communication sequence between the user and the chatbot, the conversation history 302 can be recursively embedded. The conversation history 302 can be processed by one or more encoder models (e.g., one or more latent encoding models 304) to generate a language representation associated with the embedding in the latent space. For example, the encoder model can be trained to map the conversation history to a latent space including multiple learned latent distributions associated with multiple emotions and / or multiple tasks.

[0100] The language representation may then be processed with multiple experts (e.g., multiple expert language models, multiple expert heads of a large language model, and / or multiple sampled expert embeddings associated with multiple learned distributions) to generate multiple candidate utterances. For example, the language representation may be processed with a first expert block 310 (e.g., an expert language model trained on training examples sampled from a first learned distribution (e.g., an embedding distribution associated with interrogative responses)) to generate a first candidate utterance. Additionally and / or alternatively, the language representation may be processed with a second expert block 312 (e.g., an expert language model trained on training examples sampled from a second learned distribution (e.g., an embedding distribution associated with exploratory responses)) to generate a second candidate utterance. The language representation may be processed with a third expert block 314 (e.g., an expert language model trained on training examples sampled from a third learned distribution (e.g., an embedding distribution associated with optimistic responses)) to generate a third candidate utterance. In some implementations, the language representations may be processed with an nth expert block 316 (e.g., an expert language model trained on training examples sampled from an nth learned distribution (e.g., an embedding distribution associated with degraded responses)) to generate an nth candidate utterance.

[0101] A plurality of candidate utterances including a first candidate utterance, a second candidate utterance, a third candidate utterance, and an nth candidate utterance may be processed with a dialog management model 320 to generate a predicted dialog response 322. The predicted dialog response 322 may include one or more selected candidate utterances from the plurality of candidate utterances. The predicted dialog response 322 may be generated based on one or more learning parameters of the dialog management model 320 trained according to one or more sequences and / or goals. The dialog management model 320 may be adjusted based on reinforcement learning.

[0102] Figure 4A block diagram of an example language model training 400 according to an example embodiment of the present disclosure is depicted. Specifically, an expert mixture model may be trained to process conversation history data to generate a semantically-aware predicted dialogue response 422. For example, conversation history 402 may be processed by language encoding model 404 to generate a latent representation, which may then be processed by multiple expert models (e.g., 410, 412, 414, and / or 416) to generate multiple candidate outputs. The multiple candidate outputs may then be processed with a dialogue management model 420 to generate a predicted dialogue response 422. The predicted dialogue response 422 may be processed to determine satisfaction data 426 (e.g., a satisfaction level determined based on engagement after providing the predicted dialogue response) and / or sentiment data 428 (e.g., data describing whether the sentiment of the predicted dialogue response is associated with the sentiment of the conversation history). One or more parameters of the dialogue management model 420 may then be adjusted based on the satisfaction data and / or sentiment data.

[0103] Throughout the communication sequence between the user and the chatbot, the conversation history 402 can be recursively embedded. The conversation history 402 can be processed by one or more encoder models (e.g., one or more latent encoding models 404) to generate a language representation associated with the embedding in the latent space. For example, the encoder model can be trained to map the conversation history to a latent space including multiple learned latent distributions associated with multiple emotions and / or multiple tasks.

[0104] The language representation may then be processed with multiple experts (e.g., multiple expert language models, multiple expert heads of a large language model, and / or multiple sampled expert embeddings associated with multiple learned distributions) to generate multiple candidate utterances. For example, the language representation may be processed with a first expert block 410 (e.g., an expert language model trained on training examples sampled from a first learned distribution) to generate a first candidate utterance. Additionally and / or alternatively, the language representation may be processed with a second expert block 412 (e.g., an expert language model trained on training examples sampled from a second learned distribution) to generate a second candidate utterance. The language representation may be processed with a third expert block 414 (e.g., an expert language model trained on training examples sampled from a third learned distribution) to generate a third candidate utterance. In some implementations, the language representation may be processed with an nth expert block 416 (e.g., an expert language model trained on training examples sampled from an nth learned distribution) to generate an nth candidate utterance.

[0105] The plurality of candidate utterances including the first candidate utterance, the second candidate utterance, the third candidate utterance, and the nth candidate utterance may be processed with the dialog management model 420 to generate a predicted dialog response 422. The predicted dialog response 422 may include one or more selected candidate utterances from the plurality of candidate utterances.

[0106] Additional user data 424 may be obtained (e.g., one or more additional messages in response to the provided predicted dialog response 422 and / or data associated with actions performed by the user after providing the predicted dialog response 422). The additional user data 424 may include updated conversation history embeddings and / or embeddings associated with subsequent messages from the user. One or more parameters of the dialog management model 420 may be adjusted based on the additional user data 424 and / or the predicted dialog response 422. For example, satisfaction data 426 (e.g., a satisfaction level determined based on engagement after providing the predicted dialog response) and / or sentiment data 428 (e.g., data describing whether the sentiment of the predicted dialog response is associated with the sentiment of the conversation history) may be determined and / or generated based on the additional user data 424 and / or the predicted dialog response 422. The satisfaction data 426 may be determined based on the user performing one or more desired actions associated with one or more goals of the task. For example, the user makes a reservation and / or purchase, ends a conversation, closes a help ticket, and / or provides a closing message. A loss function (e.g., a first loss function) may be evaluated that incentivizes the completion of a goal or penalizes a determined regression (e.g., a user repeating comments and / or questions). Sentiment data 428 may be determined based on encoding additional user data 424 to map additional user inputs to a latent space. Sentiment data 428 may describe an embedding associated with a particular sentiment distribution of the latent space. A loss function (e.g., a second loss function) may be evaluated that incentivizes improvement in sentiment and / or incentivizes matching sentiment to a desired sequence and / or penalizes negative sentiment changes and / or deviations from a desired sequence. One or more parameters of the dialog management model 420 may be adjusted based on one or more gradient descents generated by evaluating one or more loss functions evaluated based on predicted dialog responses 422, conversation history 402, satisfaction data 426, sentiment data 428, and / or additional user data 424.

[0107] Figure 5 A block diagram of an example expert language model training 500 is depicted according to an example embodiment of the present disclosure. Specifically, the expert language model training 500 may include processing a conversation history 502 with a language encoding model 504 and an expert language model 506 to generate candidate utterances 508. The candidate utterances 508 may then be compared to a reference true utterance 510 to evaluate a loss function 512. One or more parameters of the expert language model 506 may then be adjusted based on the loss function 512.

[0108] For example, multiple example conversation histories 502 (and / or training language representations) and benchmark real utterances 510 training pairs can be used to train one or more expert language models 506 and / or one or more language encoding models 504. In some implementations, the language encoding model 504 can be pre-trained and / or trained separately. The language encoding model 504 can be trained to map conversation data to an embedding space. By sampling embeddings from a potential embedding space and then classifying the output of processing the sampled embeddings, one or more emotional distributions and / or one or more task distributions of the embedding space can be learned. Based on the learned distribution, each of the multiple expert language models can be trained by selecting training examples from a specific learned distribution to generate an expert associated with the specific learned distribution.

[0109] The conversation history 502 and the benchmark real speech 510 can be associated with embeddings in the learned distribution, which is associated with the assigned emotions and / or tasks of the expert language model 506. For training, the conversation history 502 can be processed with the language encoding model 504 to generate a language representation. The language representation can then be processed with the expert language model 506 to generate candidate utterances 508. The candidate utterances 508 and the benchmark real speech 510 can be used to evaluate the loss function 512 to generate a gradient descent. The gradient descent can then be back-propagated to the expert language model 506 to adjust one or more parameters of the expert language model 506. The candidate utterance 508 can describe a predicted candidate response to one or more user messages of the conversation history 502. The candidate utterance 508 can include a text string. The benchmark real speech 510 can include a text string in response to one or more messages of the conversation history 502.

[0110] Example Method

[0111] Figure 6 Depicted is a flowchart of an example method performed in accordance with an example embodiment of the present disclosure. Figure 6 The steps are depicted as being performed in a particular order for purposes of illustration and discussion, but the method of the present disclosure is not limited to the particular order or arrangement shown. The various steps of method 600 may be omitted, rearranged, combined and / or adjusted in various ways without departing from the scope of the present disclosure.

[0112] At 602, a computing system may obtain input data. The input data may include a language representation. The language representation may be associated with a conversation. In some implementations, the language representation may describe the semantics of a conversation history of the conversation. The conversation history may include multiple exchanged text strings. Additionally and / or alternatively, the language representation may be associated with a latent space distribution of a latent space. In some implementations, the language representation may have been generated by processing the conversation history with a language encoding model. The language representation may be associated with a learned distribution of the latent space, and the learned distribution may be associated with a particular emotion.

[0113] In some implementations, the input data may include session data describing a session (e.g., session data describing a history of a session that may include a sequence of responses). The session data (i.e., input data) may be processed with an encoder model to generate a language representation. The language representation may include a latent embedding associated with the session. The encoder model may have been trained to map the encoded session to a latent distribution.

[0114] At 604, the computing system may process the language representation with one or more machine learning language models to generate one or more candidate outputs. The one or more candidate outputs may include one or more candidate utterances. In some implementations, the one or more machine learning language models may include an expert language model trained on a particular skill, such that the one or more candidate utterances may indicate the particular skill.

[0115] In some implementations, processing the language representation with the one or more machine learning language models to generate the one or more candidate outputs may include processing the language representation with multiple expert language models to generate multiple candidate outputs. The multiple candidate outputs may be associated with multiple candidate utterances. In some implementations, the predicted dialog response may include a selected candidate utterance associated with a selected candidate output from the multiple candidate outputs. The selected candidate output may be selected by the dialog management model.

[0116] At 606, the computing system may process the one or more candidate outputs and the language representation with a dialog management model to generate dialog planning data. The dialog management model may have been trained using reinforcement learning. Reinforcement learning may optimize predictions for complete conversations. In some implementations, the dialog management model may be configured to determine a conversation intent based on the language representation and select a particular candidate output based on its association with the conversation intent.

[0117] At 608, the computing system may generate a predicted dialogue response based on the dialogue planning data. The predicted dialogue response may include one or more predicted terms associated with the one or more candidate utterances. The predicted dialogue response may include the predicted utterance. In some implementations, the predicted utterance may be responsive to the input data.

[0118] Figure 7 Depicted is a flowchart of an example method performed in accordance with an example embodiment of the present disclosure. Figure 7 The steps are depicted in a specific order for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific illustrated order or arrangement. The various steps of method 700 can be omitted, rearranged, combined and / or adjusted in various ways without departing from the scope of the present disclosure.

[0119] At 702, the computing system can obtain session data. The session data can describe a session history.

[0120] At 704, the computing system may process the session data with a language encoding model to generate a language representation. The language representation may describe semantics associated with the session history. In some implementations, the language encoding model may include a stochastic encoder model. The stochastic encoder model may include an encoder and a latent space distribution. In some implementations, the stochastic encoder model may map the lemmatized session history to a latent space to generate a parameterized Gaussian distribution.

[0121] At 706, the computing system may process the language representation with a plurality of machine-learned language models to generate a plurality of candidate outputs. In some implementations, a first machine-learned language model in the plurality of machine-learned language models may have been trained for a first skill (e.g., a first intent or a first emotion), and a second machine-learned language model in the plurality of machine-learned language models may have been trained for a second skill (e.g., a second intent or a second emotion). Alternatively and / or additionally, the plurality of machine-learned language models may include a plurality of expert models associated with a plurality of emotions. In some implementations, the plurality of machine-learned language models may include a plurality of expert models associated with a plurality of tasks. Additionally and / or alternatively, the plurality of machine-learned language models may have been trained based on a learned emotion distribution associated with a latent space.

[0122] At 708, the computer system may process the language representation and the plurality of candidate outputs with a dialog management model to determine a dialog response. The dialog management model may be trained to select a particular candidate output from the plurality of candidate outputs based at least in part on the language representation. In some implementations, the plurality of machine learning language models may be trained with benchmark real training data.

[0123] Figure 8 Depicted is a flowchart of an example method performed in accordance with an example embodiment of the present disclosure. Figure 8 The steps are depicted in a specific order for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific illustrated order or arrangement. The various steps of method 800 can be omitted, rearranged, combined and / or adjusted in various ways without departing from the scope of the present disclosure.

[0124] At 802, a computing system can obtain training data. The training data can include training session data describing a set of example sessions.

[0125] At 804, the computing system can process the training session data with a language encoding model to generate a language representation. In some implementations, the language encoding model can include a variational autoencoder model. The language representation can include an embedding sequence.

[0126] At 806, the computing system may process the language representation with a plurality of machine learning language models to generate a plurality of candidate utterances. The plurality of machine learning language models may include a plurality of language decoder models.

[0127] At 808 , the computing system may process the plurality of candidate utterances with a dialog management model to determine a predicted dialog response.

[0128] At 810, the computing system may provide the predicted dialog response to the user computing system and receive additional session data from the user computing system. The user computing system may be associated with the user involved in the session. The additional session data may describe the session that occurred after the predicted dialog response.

[0129] At 812 , the computing system may adjust one or more parameters of the dialog management model based on the additional conversation data.

[0130] In some implementations, the computing system may determine and / or generate satisfaction data based at least in part on the additional conversation data. The satisfaction data may describe a level of satisfaction with the predicted conversation response. In some implementations, the satisfaction data may be determined based at least in part on the conversation engagement. The computing system may then adjust one or more parameters of the conversation management model based on the satisfaction data.

[0131] Additionally and / or alternatively, the training data may include one or more benchmark real utterances. The computing system may then evaluate a loss function that evaluates a difference between a particular candidate utterance in the plurality of candidate utterances and the one or more benchmark real utterances. One or more parameters of at least one of the plurality of machine learning language models may be adjusted based at least in part on the loss function.

[0132] Fig. 9 Depicted is a flowchart of an example method performed in accordance with an example embodiment of the present disclosure. Fig. 9 The steps are depicted as being performed in a particular order for purposes of illustration and discussion, but the method of the present disclosure is not limited to the particular order or arrangement shown. The various steps of method 900 may be omitted, rearranged, combined and / or adjusted in various ways without departing from the scope of the present disclosure.

[0133] At 902, the computing system may obtain sequence data. The sequence data may describe a sequence. For example, the sequence data may describe a sequence of conversation text strings and / or a sequence of emotions associated with the conversation.

[0134] At 904, the computing system may process the sequence data with the encoding model to generate an encoded representation. The encoded representation may describe the determined semantics of the sequence.

[0135] At 906, the computing system may process the encoded representation with a plurality of machine learning expert models to generate a plurality of candidate outputs. The plurality of machine learning expert models may be associated with a plurality of different intents.

[0136] At 908, the computing system may process the plurality of candidate outputs and the encoded representations with the management model to generate planning data.

[0137] At 910, the computing system may generate a predicted output based on the planning data. The predicted output may include at least a portion of one or more candidate outputs of the plurality of candidate outputs.

[0138] Example implementation and example experiments

[0139] Despite recent advances in language models (LMs), their application to the problem of dialogue management (DM) and their ability to conduct rich content conversations remain challenging. The systems and methods disclosed herein can exploit reinforcement learning (RL) to develop conversational agents that avoid short-sightedness (e.g., outputting generalized utterances) and can maximize overall user satisfaction. Most existing RL methods for DM train agents at the word level and, therefore, must handle combinatorially complex action spaces even for moderately large vocabularies. Thus, even if they are warm-started with a pre-trained LM, they may have difficulty generating successful and engaging conversations. RL-based DMs can exploit a novel mixture of experts language model (MoE-LM) that includes (i) a LM capable of learning different semantics of conversation history, (ii) multiple specialized LMs (or experts) capable of generating utterances corresponding to specific attributes or personalities, and (iii) an RL-based DM that performs dialogue planning with utterances generated by experts. Mixed-of-expert approaches can provide greater flexibility to generate perceptual utterances with different intents and can allow RL to focus on conversation-level DMs.

[0140] With advances in natural language understanding and generation, more and more attention is being directed toward building intelligent conversational agents that can have rich and engaging conversations with human users. Interactions can be open-ended, can include multiple different topics, and can involve underlying tasks (e.g., building rapport, negotiation, information exchange, and / or recommendation). Therefore, in order to satisfy users, a good conversational agent should not only generate natural responses, but also be able to achieve task goals and adapt to dynamic feedback from users.

[0141] The standard solution can use behavioral cloning to train a conversational agent, where the agent is a language model (LM) that mimics the utterances in the training set. By leveraging deep neural networks (e.g., RNN and / or Transformer), the LM can encode the conversation into a low-dimensional dialogue state and can predict utterances.

[0142] Dialogue management (DM) can be viewed as a control problem and reinforcement learning (RL) can be used to optimize the agent's policy (which can be a language model).

[0143] The system and method may include an RL-based DM agent using a novel mixture of experts (MoE) approach. The mixture of experts approach may utilize a mixture of experts language model (MoE-LM), which may include three main components: 1) LM (probabilistic encoder and decoder), which is able to learn different semantics of conversation history and thus generate different utterances, which may be represented as a primitive LM or , 2) Multiple dedicated LMs (or experts) , each of which can be used by The latent space is constructed from the learned latent space, and may have been trained to enable the expert to generate utterances corresponding to a certain intent or personality, and 3) an RL-based dialogue manager (DM), which, at each turn, given the utterances generated by the expert The shared latent states and the utterance actions they suggest, among which one can be selected for the agent to perform. The systems and methods can learn LMs (experts) and DMs. Additionally and / or alternatively, the DM in the MoE-LM can be a policy conditioned on both the latent states and the actions suggested by the expert. Different semantics of conversation history can be learned and allow the agent to generate multiple utterances. The diversity can be shared with dedicated LMs (experts) and can give the model flexibility in generating dedicated utterances. The system and method can utilize modularity that facilitates the addition and removal of dedicated LMs (experts). In some implementations, the layered architecture can allow the system to solve RL problems with much smaller state and action spaces. Since the candidate utterances are generated by experts with different intentions, the DM agent can focus on optimizing the specific goal of the conversation task instead of combining all agent-user signals into a single RL reward.

[0144] The systems and methods disclosed herein may include a seq2seq LM to generate the next utterance in a conversation. The systems and methods may access the form A data set where each yes Round Session History and is its next utterance. Every word in The upper limit of the length (number of words) can be expressed as LM can be used to record the session history. As a condition, , to predict the next utterance (include In the Transformer architecture, LM can first use the encoder Session History Encoding as embedding of length sequence, where each is a vector in the latent space. For ease of description, the systems and methods may concatenate these embeddings into a single embedding And the overall size of the latent space can be expressed as In the RNN architecture, the encoder of the LM You can directly log the session history Mapping to latent states In one or both architectures, the decoder (Right now Word by word, the next utterance Sampling is performed, where can be a fixed initial (sentence start) token, and the latent state can be represented as .

[0145] Markov Decision Process (MDP) may have been used to model the dialogue management problem in a variety of settings. MDP can be used Representation. State space Can represent the lemma session history, and the initial state It can be the query of the initial user. Action space It can be a lemma language space, where each action is the agent's next utterance (which can be of fixed length Transformation kernel The user's response to the actions taken by the agent (robot) can be modeled. Additionally and / or alternatively, the reward function User satisfaction can be measured. In these MDPs, we can view the entire LM as a policy that maps the conversation history to the next utterance, and can be solved by finding the policy with the maximum expected discounted reward (i.e. Strategy to solve them. The size of the tokenized state space and action space can grow exponentially with the size of the vocabulary. Even for moderately sized vocabularies, this growth can make the data intractable for solving MDPs.

[0146] The systems and methods can use open domain conversations as running examples to show how MoE-LM-based agents can improve measured user satisfaction through improvements in emotion or engagement. The DM agent can have different behaviors (e.g., inquiring, exploratory, relevant, soothing, empathetic, complimentary, provocative), and can quickly decide which intent to use to guide the conversation, build rapport, attract user interest, improve user mood, etc. To achieve this goal, the language model can utilize language representations that capture different semantics (primitive discovery) in order to encode different conversations and avoid generating boring and repetitive responses. Machines (expert construction) can embed different intents into sub-models of the LM so that they can behave accordingly and respond effectively when prompted. Additionally and / or alternatively, when there are various candidate utterances available, the DM module of the LM can understand the current level of user satisfaction and determine which response is most appropriate. Inspired by these observations, such as Figure 2 As shown, the MoE-LM can be constructed in three steps.

[0147] The systems and methods may include primitive discovery. For example, the systems and methods may first employ a dataset , and learns the random encoder (i.e., the encoder that maps the encoded session to the latent distribution and the latent space distribution ) and decoder Composition of language models . Random Encoder This may include tokenizing the conversation history Mapping to latent space (Right now, Encoder , this latent space can then be used to construct Parameterized on Dimensional Gaussian distribution The decoder can predict the points sampled from the latent distribution is the condition (i.e., The next word (Token by token). In practice, the systems and methods may use two latent states as input to the decoder model. Primitives can be represented and learned using a loss function that encourages the learned latent space to be accurate in addition to predicting the next utterance. The loss function may include an unsupervised learning method for discovering primitive skills in trajectories used by one or more downstream RL tasks.

[0148] Expert construction can include learning one or more latent distributions. For example, given the latent space learned in primitive discovery , Encoder and decoder , systems and methods can learn Potential distribution , each is defined as Each corresponds to an attribute (e.g., intent or personality (in the case of a chatbot)) and can be expressed in the latent space This configuration can result in samples with LM , each of which corresponds to the original LM and can be used as an expert in our MoE-LM. Each expert The next utterance can be generated in some part of the language space that is compatible with its attributes (personality) A candidate (or multiple candidates) for each Can use encouragement corresponding LM The loss function is used to learn to generate utterances consistent with its attributes (e.g., Equation 2).

[0149] Additionally and / or alternatively, the systems and methods may include a dialogue manager (DM). The dialog manager represented by and by experts The generated candidate action utterances are taken as input, and one of them can be selected as the action to be performed by the robot (i.e., .

[0150] In some implementations, the systems and methods may learn the primitive LM in the MoE-LM by solving the following KL-constrained optimization problem designed to capture different semantics: :

[0151]

[0152] in It is a dataset Zhongzai Expectations of experience, The encoded conversation history and target discourse is the distribution on the conditional latent space, and is a positive real-valued threshold. Using (1), the system and method can learn by maximizing the log-likelihood , while at the same time Predicted latent variables and KL-constrained Enforce consistency between distributions. Can be Gaussian distribution ,in is used for the target discourse The pre-trained encoder of and variance is a trainable model. Using a separate encoder One reason could be to avoid overfitting (i.e., avoid of The back-propagated gradients of are taken as input).

[0153] The systems and methods may include the KL constraint in (1) as a penalty weighted by a suitably chosen coefficient. Primitive discovery may include learning an encoder-decoder , the encoder-decoder can be selected by to modulate (i.e., change Fixed This may result in different distributions on the generated utterances.) The goal in (1) can be adjusted by and The potential variables on both (Right now, Additionally and / or alternatively, the KL constraint can be used to ensure that the random encoder of the primitive LM For different does not vary too much, thus having a limiting effect on diversity. When (or, , when used as a regularizer), there may be no Although This may be an extreme case, but when If the data can be modeled by a single LM, the VAE optimal decoder Can be simply ignored , resulting in a degradation of the latent space. In some implementations, the systems and methods may weaken the KL constraint , or equivalently, ). Weakening can make this method more reliable to ensure that only Indicates that each different session , which captures different semantic modalities and enables easier downstream specialization.

[0154] The log-likelihood objective in (1) can be obtained by specializing the latent space To recover the learning of any arbitrary expected LM Arbitrary LM in conditional session data distribution Coverage on can be characterized based on one or more definitions.

[0155] Definition 1 may include ,if , then it can be data of Theoretical results can characterize our primitive encoder-decoder pair About the data representative capacity.

[0156] Definition 2 can make Become (1) (where ). Then, Can be configured so that ,in and Can be a constant, depending on .if only exist In Shared, there is potential space specialization, when combined with When paired, this latent space can be approximately restored The quality of this approximation can be (e.g., how well the objective in (1) is optimized) and By Replace with (Right now, ) to construct the primitive, because Can be regarded as The system and method can use a trained encoder-decoder pair To learn from different latent distributions Parameterized spectrum of experts in different specialties.

[0157] In some implementations, the MoE framework may include different experts The entire domain, , where each expert generates candidate utterances with different intents. By treating each expert as conversation data The distribution of specific behaviors in the system and method can use a universal encoder-decoder among all experts Therefore, each expert Through any dimensional latent distribution (e.g., Gaussian), and the systems and methods can be used to parameterize the latent space Experts can all be classified as plug-and-play language models (PPLMs). Creating a new expert may include learning a new latent distribution, while switching between experts is equivalent to sampling different distributions.

[0158] Depend on The real-valued labels represented by The system and method can train an expert by solving an optimization problem Potential distribution of

[0159] (2)

[0160] With the weight Assigned to the corresponding expert Unlike the weighted maximum likelihood method of training samples, the system and method can learn each expert by maximizing the reward and Seen as an expert on what to maximize The method can be linked to reinforcement learning (RL), where both the “state” space and the “action” space are latent spaces. , and the “strategy” can be the potential distribution The systems and methods may not need to rely on data from Target words , and therefore some intentions may not be susceptible to For example, in some implementations, the reward maximization problem may be short-sighted (i.e., the above RL problem has a discount factor ). Unlike dialogue management, which can be a sequential decision problem, the systems and methods can be configured so that each expert has a specific behavior, and this configuration can be easily done via greedy maximization. Long-term dialogue optimization can be handled by the dialogue manager instead of these experts.

[0161] For example, in Gauss In the case of Model parameters , specifically based on:

[0162]

[0163] in is the learning rate. To reduce the variance of these estimates, the systems and methods may employ baseline reduction techniques in the policy gradient. This can be accomplished by Replace with To quantify the experts performance, which can produce suboptimal results.

[0164] No. The reward maximization objective can be expressed as .Target The optimal LM can be found in In Additionally and / or alternatively, You can use (2) Then, in the expert and In the case of . The system and method can be optimized relative to (2) , which encourages experts Captured by This inference can provide the following two additional implications: (i) Suboptimality compared to oracle Received quantity This proves that the use of expert construction optimization The original language is reasonable; (ii) suboptimality further depends on , thereby quantifying In the original dataset The degree of goodness expressed in .

[0165] The systems and methods disclosed herein may utilize one or more dialogue managers (e.g., one or more dialogue managers (DMs) for MoE-LM), and may utilize one or more RL techniques to train the respective dialogue managers. The DM may include processing the encoded conversation history. and by experts Generated strategy for candidate action utterances , and one of them can be randomly selected to execute (i.e. ). Each expert Can be a strategy LM , and each session history can be Mapping to discourse This architecture can be used to address the large size of the state space and action space in the original MDP, which grows exponentially with the size of the vocabulary. The state space and action space of the original MDP can be the tokenized session history and the tokenized language space, respectively, while the DM can be in the latent space of a given encoded session. In the case of The MoE-LM can be different from other hierarchical architectures where the decision at the higher level is to select a lower level controller based solely on the current state of the system. In some MoE-LMs, the DM can observe the current state and the actions suggested by the experts and can then select one of them.

[0166] Systems and methods can learn DM policies by solving an MDP, which can be referred to as a MoE-MDP , and available To represent MoE-MDP. The state space of MoE-MDP can be the learned latent space and The product of the joint action spaces of the experts (i.e., ). The corresponding action space may include experts, namely, The initial state may be an encoding of the initial user query and the utterances suggested by the expert in response to the query. The transformation may continue to model the user's response, but now possibly on the joint space of latent states and expert actions. The reward function may be the same as in the original MDP (i.e., ),in , and ,and is an expert selected by the DM. Since a MoE-MDP may have a finite number of actions, learning a DM is equivalent to solving a finite-action MDP and finding The strategy with maximum expected cumulative discounted return in the 2-dimensional simplex method .

[0167] Systems and methods can use two RL techniques to solve MoE-MDP and learn DM policies The first technique may include conservative Q-learning (CQL) (Kumar et al., “Conservative q-learning for offline reinforcement learning,” Advances in Neural Information Processing Systems, 1179–1191 (December 2020)). In settings where this technique is applicable, the collected conversation data may be used to CQL can be a conservative approach to learning a lower bound on the true function. Given offline session data , system and method parameters available To parameterize function and learns by minimizing the Bellman error using behavioral regularization : in is composed of the primitive LM (which can be approximated Behavioral strategies) Suggested actions, is the regularization parameter, and is the target function parameters. CQL regularization minimizes our DM and primitive According to the CQL technique, the system and method can set DM to correspond to entropy regularization The optimal strategy for learning (Schulman et al., “Equivalence Between Policy Gradients and Soft Q-Learning,” Arxiv (October 14, 2018), https: / / arxiv.org / pdf / 1704.06440.pdf.).

[0168] A second RL technique may include model-based RL (MBRL) (Shah et al., "Bootstrapping aneural conversational agent with dialogue self-play, crowdsourcing and on-line reinforcement learning", In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. 3 (Industry Papers), pp. 41-51, 2018). The system and method may use maximum likelihood to model the user's speech. , then generate the data , the next state of the data The next session and the corresponding candidate actions generated by rolling out are encoded, and the Bellman error minimization in MoE-MDP can be solved: The benefit of MBRL over CQL may be twofold. First, by simply fine-tuning the large LM, the system can easily obtain a high-fidelity user utterance model. Second, with enough dialogue rollouts to capture many different scenarios, MBRL may not rely on behavioral regularization and may be less conservative.

[0169] An implementation of the MoE approach for dialogue management is evaluated on the following two benchmark open-domain tasks. The first benchmark is Cornell (Danescu-Niculescu-Mizil et al., "Chameleons in imagined conversations: A new approach to understanding coordination of linguisticstyle in dialogs", Arxiv (June 15, 2011), https: / / arxiv.org / pdf / 1106.3077.pdf.), which consists of conversations between speakers in different movie scripts and has The second benchmark is Reddit (Ghandeharioun et al., “Approximating interactive human evaluation with self-play for open-domain dialog systems,” Advances in Neural Information Processing Systems, 32, 2019), which is about the median conversation length of at least A corpus of small talk conversations on various topics, including The median number of utterances in a conversation.

[0170] The evaluation consists of conducting several experiments to test the efficacy of different parts of MoE-LM, namely (i) the predictive power and diversity of primitives, (ii) the quality of experts, and (iii) the overall DM performance. For each metric, we report The mean of the evaluation set of sessions Standard deviation. Ablation studies were performed on the Transformer-based MoE-LMs (i.e., MoE-1, MoE-2, MoE-3, MoE-4) to understand the impact of different model architectures, language encoders, and latent generators on performance. MoE-1 and MoE-2 can use simpler architectures, while MoE-3 and MoE-4 can use the same encoder architecture as BERT. MoE-1 can use a much smaller latent distribution model than MoE-2. ; MoE-3 can use the pre-trained BERT encoder, while MoE-4 can train the encoder from scratch.

[0171] A first experiment may include comparing primitive models.

[0172] This experiment can be compared with the MoE-LM The quality of the latent representation learned by the implementation is related to baselines (e.g., standard Transformer (Wolf et al., "Huggingface's transformers: State-of-the-artnatural language processing", Arxiv (July 14, 2020), https: / / arxiv.org / pdf / 1910.03771.pdf.) and VHRED (Serban et al., "A hierarchical latent variable encoder-decoder model for generating dialogues", In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 31, 2017b)). To evaluate their quality, for each test session, the experiment generates utterances and reportable measures: (i) diversity: the singular values ​​of the embedded utterances Sparsity (i.e., ,in ,and is a pre-trained sentence encoder); (ii) : The only one in the generated utterance (iii) perplexity. This metric measures both accuracy and semantic diversity.

[0173] The results of the above experiments are reported in Table 1. In comparison with the baselines (Transformer and VHRED), in general, (i) the Transformer-based LM outperforms VHRED due to its attention mechanism that explicitly encodes sequential semantic information, and (ii) MoE-LM achieves better diversity without sacrificing much accuracy (i.e., the perplexity score is still quite low). Qualitatively, the sample utterances generated by Transformer are closer to the target than those generated by MoE-2 and MoE-4, which may be due to the Transformer's tendency to memorize the corpus. MoE-LM can generate utterances with similar contexts to the target with paraphrases or similar structures but different contexts, demonstrating its generalization.

[0174] Among the different MoE-LMs, MoE-2 and MoE-4 have the best performance, especially MoE-4 has better diversity, while MoE-2 has lower perplexity. The results can be confirmed by the following hypotheses: (i) joint training of the encoder and decoder using Equation 1 can be used to encourage semantic diversity (in contrast to using a pre-trained BERT encoder that maximizes likelihood), and (ii) Use enough representation power to match the posterior distribution , in order to capture Different semantics in . Samples can be taken for both Transformer and MoE-4 The latent space of MoE-4 is likely to be more scattered in the embedding space, which means that most sessions are uniquely encoded. In contrast, the latent space of Transformer may have many clusters, which indicates that the latent space is more inclined to generate similar utterances even with different input sessions and may therefore not generalize well.

[0175]

[0176] Table 1 describes the accuracy (perplexity) and diversity of the language primitive experts trained on Reddit.

[0177]

[0178] Table 2 describes the performance of the RL-based DM trained with Reddit (in terms of average user satisfaction in a session).

[0179] Experiment 2 can evaluate the quality of experts.

[0180] This experiment can be compared with The performance of the experts learned by MoE-LM (where the experts are trained separately by optimizing Eq. 2) is The evaluation includes two baselines (WD (Holtzman et al., “Learning to write with cooperative discriminators”, Arxiv (May 16, 2018), https: / / arxiv.org / pdf / 1805.06087.pdf.) and PPLM (Dathathri et al., “Plug and play language models: A simple approach to controlled text generation,” Arxiv (March 3, 2020), https: / / arxiv.org / pdf / 1912.02164.pdf.)). To study the suboptimal gap, the evaluation can include the performance of expert end-to-end Transformer-based LMs that are individually optimized with REINFORCE (Li et al., “Deep reinforcement learning for dialogue generation”, Arxiv (September 29, 2016), https: / / arxiv.org / pdf / 1606.01541.pdf.) and the expert labels. The system and method can use the following labeling function to define the expert's intention: (i) , , , , , Quantification different emotional tones and can be constructed through a sentiment detector based on RoBERTa (Liao et al., “An improved aspect-category sentiment analysis model for text sentiment analysis based onroberta,” Applied Intelligence, 51(6):3522–3533, 2021), which predicts whether the utterance is positive or negative, and whether it belongs to Any of the following more specific emotions: {joy, optimism, sadness, anger}; (ii) Measuring empathy, i.e., the coherence of the robot’s emotions with the user’s emotions; (iii) when a robot problem is detected Output , otherwise output (iv) Quantify exploration (i.e., the tendency to avoid repeated context).

[0181] The results of the above experiments can be reported in Table 3. Compared with the baseline LM, in general, under the MoE-LM framework, especially under MoE-2 and MoE-4, the created experts can better capture all different language intentions (where WD and PPL appear to capture negative emotions and sentiments much more effectively than behavior), thus demonstrating the efficacy of some implementations of the method of constructing specialized experts on different language spaces via reward maximization (rather than weighted MLE).

[0182] Similar to the ablation study in Experiment 1, all experts associated with MoE-2 and MoE-4 may perform best in capturing language intent. Experts in MoE-4 may perform best on Reddit data, while the best experts on much less data (Cornell) may be built on the simpler MoE-2 architecture. This difference may be due to the overfitting problem faced by the larger LM (MoE-4) when there is not enough data for expert fine-tuning.

[0183] The latent distributions of the emotion experts may be clearly separated (since positive and negative emotions have opposite behaviors), whereas the latent distributions of the sentiment experts may have a more gradual separation, or even some overlap (since, for example, joy is very similar to optimism, while joy is very different from anger). This distribution validates that the MoE-LM represents different behaviors in separate regions of the latent space, and justifies the structural prior for modeling each expert as a specialized version of a primitive LM whose latent distribution is focused on a specific region.

[0184]

[0185] Table 3 describes the quality of each expert PPLM trained on the Reddit dataset with respect to the training labels.

[0186] Experiment 3 evaluates MoE-RL on DialoGPT simulated users.

[0187] This experiment compares the dialogue management performance of MoE-LM (its DM trained with different methods (BC, DQN, CQL, MBRL) and We present three baselines (REINFORCE, KL control (Jaques et al., “Wayoff-policy batch deep reinforcement learning of implicit human preferences in dialog”, Arxiv (July 8, 2019), https: / / arxiv.org / pdf / 1907.00456.pdf.), and VHRL (Saleh et al., “Hierarchical reinforcement learning for open-domain dialog”, In Proceedings of the AAAIConference on Artificial Intelligence, Vol. 34, pp. 8741-8748, 2020)). Based on the expert quality results in Experiment 2, MoE-2 and MoE-4 frameworks are selected for evaluation on the Cornell and Reddit tasks, respectively. For the system evaluation, the evaluation includes testing the most The experiment was conducted by having these RL agents interact with a user environment simulated by DialoGPT (Zhang et al., “Dialogpt: Large-scale generative pre-training for conversational response generation”, Arxiv (May 2, 2020), https: / / arxiv.org / pdf / 1911.00536.pdf.) (a large LM capable of natural open-domain conversations). The DM task can be to maximize the overall user satisfaction at the conversation level, which can be measured by (i) the user's overall emotion and (ii) the user's emotion transition. In order to construct an immediate reward used as a proxy for user satisfaction, the system and method can set , where the linear combination weights Related to (Ghandeharioun et al., “Approximating interactive humanevaluation with self-play for open-domain dialog systems,” Advances in NeuralInformation Processing Systems, 32, 2019), and can be the same RoBerTa-based sentiment labeler as in Experiment 2, which will be derived from Assign a score that is proportional to the predicted probability of positive sentiment and inversely proportional to the predicted probability of negative sentiment. To ensure that the baseline RL DM method also has certain robot-level features, such as questions, positive sentiment, etc., in addition to the above RL rewards for user satisfaction, we also optimize a linear combination of robot-based rewards when training the baseline model. Since the DM problem may last up to rounds, the experiment uses it as the effective range and sets To evaluate the fluency of the LM, experiments report GPT-Perplexity, which computes the perplexity score of the final utterance of the DialoGPTLM.

[0188] The results of the above experiments are reported in Table 2. The experiments show that MoE-LM outperforms most baseline methods in terms of DM performance. The results can describe the impact of the following two factors: (i) MoE-MDP restricts the action space to a smaller set of candidate utterances generated by experts (whose quality is verified in EXP2), and the corresponding RL problem becomes simpler and requires less data (especially in Cornell) to solve; (ii) Unlike baseline RL methods that need to optimize both robot signals and user signals, MoE DM agents focus on optimizing user satisfaction goals and are therefore more effective. MoE-LM can have lower (better) GPT-perplexity scores than other methods. The results can be based on MoE-LM using a pre-trained encoder and decoder from the primitive LM, which is optimized for generalization and accuracy, while other RL methods may distort their language representation to create utterances that maximize rewards but become less natural. Among the different MoE-RL strategies, a user utterance model is first learned (the model uses the same encoder from the primitive LM and learns a separate decoder for user utterance prediction) and then The learned MBRL performs the best of the tested model configurations. CQL modestly improves the primitive LM (behavior policy), while DQN seems less effective in solving offline RL problems.

[0189] In some implementations, the systems and methods may utilize a KL contrast constraint. The KL constraint may enhance the consistency of the latent variables predicted by the semantic generator and the posterior. However, due to and are all models to be learned. If the data can be modeled by a single fixed LM that does not depend on the latent space, a weakly degraded solution (satisfying the KL constraint) may cause both models to output very small values, which may hinder the generalization (and diversity) of the LM embedding, especially for unprepared sessions. Using similar arguments to KL distribution matching and mutual information maximization in connection representation learning, this problem can be solved by considering the KL contrast constraint, that is, let , we replace the constraints in the optimization problem in (1) with

[0190]

[0191] in is a trade-off factor in the KL contrast constraint. Although the first part of the constraint restricts , but the second part can be in the positive sample (for example, relative to the reference true next utterance) to a much higher value than in the negative samples (e.g., relative to other next-utterance candidates). This constraint prevents the degradation problem mentioned above and injects flexibility to control the size of the latent representation space.

[0192] The systems and methods can take advantage of one or more bottlenecks. and to encourage better generalization to out-of-distribution inputs. In some implementations, the systems and methods may parameterize ,in The output is A continuous-valued vector of the form , and the discretized operator The following configurations are available:

[0193] Given a vector ,Will Split into vectors, each of which has a size of .

[0194] For each , based on the adoption of The softmax classification distribution of the given logit is One-hot vector of sampling.

[0195] Concatenate vectors To generate size Multi-hot vectors .

[0196] When calculating When backpropagating on , we use the straight-through gradient; that is, .

[0197] Systems and methods can be discrete Output The system and method may first is parameterized as a Gaussian and can then be discretized to generate the multi-heat ( hot, ) vector. In some implementations, the systems and methods may be used to and prior The same discretization is used.

[0198] In order to avoid convergence to The systems and methods can employ an entropy regularizer on the discrete distribution sampled from it to find the local minimum of . The systems and methods can use an adaptive entropy regularizer with a fixed target entropy as a hyperparameter.

[0199] Example model parameters are described below for some implementations of MoE-1, MoE-2, MoE-3, and MoE-4. The model may represent a mixture of experts and may include one or more Transformers. A Transformer may be an encoder-decoder based model that uses self-attention to capture relationships between elements of a sequence. In some implementations, the systems and methods may utilize multi-head attention (e.g., an implementation similar to https: / / www.tensorflow.org / text / tutorials / transformer#point_wise_feed_forward_network).

[0200] MoE-1 and MoE-2 can use a simple Transformer architecture, while MoE-3 and MoE-4 can use the same encoder architecture as BERT. MoE-1 can use a much smaller latent distribution model than MoE-2. ; MoE-3 can use the pre-trained BERT encoder (https: / / tfhub.dev / tensorflow / bert_en_uncased_L-12_H-768_A-12 / 4), while MoE-4 can train the model from scratch.

[0201] The Transformer model parameters for a simple Transformer architecture can be summarized in Table 4:

[0202]

[0203] Table 4 describes the simple Transformer architecture.

[0204] The BERT-based Transformer model can be similar to the structure of Table 4. Table 5 lists the differences of the pre-trained BERT model.

[0205]

[0206] Table 5 describes the BERT-based Transformer architecture.

[0207] Table 6 lists the differences for a BERT model trained from scratch.

[0208]

[0209] Table 6 describes a smaller trainable “BERT” based Transformer architecture.

[0210] Potential Distribution It can be implemented as a FFN that models the mean and variance of a normal distribution. MoE-1, MoE-2, MoE-3, MoE-4 can use different values ​​for the hidden layer size of the neural network. Additionally and / or alternatively, MoE-1 and MoE-3 can use Target entropy, MoE-2 and MoE-4 can use Table 7 lists the common parameters of FFN (e.g., FFN may have a final layer without activation).

[0211]

[0212] Table 7 Table 7 can be described FFN architecture.

[0213] Table 8 may show various parameters that are different for each of the MoE models.

[0214]

[0215] Table 8 may describe the MoE parameters.

[0216] Training and evaluation can be performed on 8 GPU instances with 20GB of RAM and NVIDIA Tesla P100 graphics cards.

[0217] Each batch of training data includes a subset of such sessions.

[0218] During Experiment 2 above, several types of language experts were defined whose utterance output can be used to smooth the robot responses when used in a conversational interaction with a user.

[0219] The first type of experts generated can be emotion-based, which can provide fluent emotional expression, thereby creating a sense of understanding in human conversation. To quantify the emotional tone of robot utterances, the system and method can utilize one or more emotion detectors, which can output the following Group prediction probabilities – (i) whether the robot utterance is positive, neutral, or negative; (ii) whether the robot utterance is Any of the more specific emotions {joy, optimism, sadness, anger}.

[0220] Ignoring neutral sentiment outputs, the systems and methods may define the following Emotion labeling function: , , , , , , which can output a score that depends on the predicted probability of emotion for any candidate robot utterance. The system can generate emotional empathy experts whose utterances will be consistent with the user's emotions.

[0221] In some implementations, RoBERTa may give a score for each of the six categories listed above, and then weight these scores by the coefficients in Table 9 to obtain a sentiment-based score:

[0222]

[0223] Table 9 may describe the sentiment-based label coefficients.

[0224] To quantify the emotional coherence between the user and the robot, the labeling function The cosine similarity of the user-bot sentiment embeddings (corresponding to the logit vectors of the first set of RoBERTa predictions) in the conversation history can be calculated.

[0225] The second type of expert used may be engagement-based, which may be configured to encourage the user to participate in the conversation. One candidate may be a question expert, which may be used to provide a robot that asks questions appropriately to demonstrate the system's attention to the user and, therefore, also enhance the user's responsiveness. To characterize whether a robot utterance is linked to a question, the system and method may define a label function , which outputs when both the question word and the question token are present , otherwise output Another engagement skill of interest could be exploration (i.e., the expert is able to change the tone or topic of the conversation to avoid a boring conversation). To measure the degree of exploration, in the label function In the present invention, the systems and methods can utilize a sentence encoder to encode both the conversation history and the robot utterance and output the negative cosine similarity between the two embeddings.

[0226] For Experiment 3, the reward for RL-based dialogue management can be defined as , where the linear combination weights Related to (Ghandeharioun et al., “Approximating interactive human evaluation with self-play for open-domain dialog systems,” Advances in Neural InformationProcessing Systems, 32, 2019), and can be the same RoBerTa-based sentiment labeler as in Experiment 2, which will be derived from Assign a score that is proportional to the predicted probability of positive emotions and inversely proportional to the predicted probability of negative emotions. Can respond to the next user Assign weights to the sentiment-based scores. A weight is assigned to the transition in user sentiment, which the system may define as the difference between the sentiment-based score of the next user response and the discounted sentiment-based score of the current session.

[0227] Systems and methods may include a mixture of experts (MoE) approach for RL-based dialogue management (DM). A MoE language model (MoE-LM) may include the following three components: (i) a LM that can generate different semantics for a conversation history, (ii) multiple specialized LMs (or experts) that can generate utterances corresponding to specific attributes or intents, and (iii) an RL-based DM that performs dialogue planning using utterances generated by experts.

[0228] Additional disclosure

[0229] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and partitions of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system, or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0230] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation, rather than limitation of the present disclosure. Those skilled in the art, after understanding the foregoing, may easily produce changes, modifications, and equivalents to these embodiments. Therefore, the present disclosure does not exclude such modifications, changes, and / or additions to the subject matter that would be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment may be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover these changes, modifications, and equivalents.

Claims

1. A computing system, the system comprising: one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: obtaining input data, wherein the input data includes session data describing a session; processing the input data with an encoder model to generate a language representation, wherein the language representation includes a latent embedding associated with the conversation, wherein the encoder model is trained to map encoded conversations to a latent distribution; processing the language representation with one or more machine learning language models to generate one or more candidate outputs, wherein the one or more candidate outputs include one or more candidate utterances; processing the one or more candidate outputs and the language representation with a dialog management model to generate dialog planning data; and A predicted dialog response is generated based on the dialog planning data, wherein the predicted dialog response includes one or more predicted terms associated with the one or more candidate utterances.

2. The system of any preceding claim, wherein processing the language representation with the one or more machine learning language models to generate the one or more candidate outputs comprises: processing the language representation with a plurality of expert language models to generate a plurality of candidate outputs, wherein the plurality of candidate outputs are associated with a plurality of candidate utterances; and wherein the predicted dialog response comprises a selected candidate utterance associated with a selected candidate output among the plurality of candidate outputs, and wherein the selected candidate output is selected by the dialog management model.

3. A system as described in any preceding claim, wherein the one or more machine-learned language models include an expert language model trained on a specific skill, so that the one or more candidate utterances are indicative of the specific skill.

4. The system of any preceding claim, wherein the dialog management model is trained using reinforcement learning, wherein the reinforcement learning optimizes predictions for complete conversations; and Wherein the predicted dialog response comprises a predicted utterance, and wherein the predicted utterance is responsive to the input data.

5. The system of any preceding claim, wherein the language representation describes semantics of a session history of the session, wherein the session history comprises a plurality of exchanged text strings.

6. The system of any preceding claim, wherein the language representation is associated with a latent space distribution of the learned latent space.

7. The system of any preceding claim, wherein the language representation is associated with a learned distribution of a latent space, wherein the learned distribution is associated with a specific emotion.

8. A system as claimed in any preceding claim, wherein the dialog management model is configured to: determining a conversation intent based on the language representation; and The particular candidate output is selected based on being associated with the conversation intent.

9. A computer-implemented method, the method comprising: acquiring, by a computing system including one or more processors, session data, wherein the session data describes a session history; processing, by the computing system, the session data using a language encoding model to generate a language representation, wherein the language representation describes semantics associated with the session history; processing, by the computing system, the language representation with a plurality of machine-learned language models to generate a plurality of candidate outputs, wherein the plurality of machine-learned language models are trained based on learned sentiment distributions associated with a latent space; as well as The language representation and the plurality of candidate outputs are processed by the computing system using a dialog management model to determine a dialog response.

10. The method of claim 9, wherein a first machine learning language model of the plurality of machine learning language models has been trained for a first skill, and wherein, A second machine learning language model among the plurality of machine learning language models has been trained for a second skill.

11. A method as claimed in any preceding claim, wherein the dialog management model is trained to select a particular candidate output from the plurality of candidate outputs based at least in part on the language representation, and wherein the plurality of machine learning language models are trained with benchmark real training data.

12. A method as claimed in any preceding claim, wherein the language encoding model comprises a stochastic encoder model, wherein the stochastic encoder model comprises an encoder and a latent space distribution, and wherein the stochastic encoder model maps a tokenization conversation history to a latent space to generate a parameterized Gaussian distribution.

13. A method as claimed in any preceding claim, wherein the plurality of machine learning language models comprises a plurality of expert models associated with a plurality of emotions.

14. A method as claimed in any preceding claim, wherein the plurality of machine learning language models comprises a plurality of expert models associated with a plurality of tasks.

15. One or more non-transitory computer-readable media collectively storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising: obtaining training data, wherein the training data comprises training session data; processing the training session data with a language encoding model to generate a language representation; processing the language representation using a plurality of machine learning language models to generate a plurality of candidate utterances; processing the plurality of candidate utterances with a dialog management model to determine a predicted dialog response; providing the predicted dialog response to a user computing system; receiving additional session data from the user computing system, wherein the additional session data describes a session occurring after the predicted dialog response; as well as One or more parameters of the dialog management model are adjusted based on the additional conversation data.

16. The one or more non-transitory computer-readable media of claim 15, wherein the operations further comprise: determining satisfaction data based at least in part on the additional conversation data, wherein the satisfaction data describes a level of satisfaction with the predicted conversation response, wherein the satisfaction data is determined at least in part based on conversation engagement; and One or more parameters of the dialog management model are adjusted based on the satisfaction data.

17. The one or more non-transitory computer-readable media of any preceding claim, wherein the training data comprises one or more benchmark real utterances, and wherein the operations further comprise: evaluating a loss function that evaluates a difference between a particular candidate utterance among the plurality of candidate utterances and the one or more reference true utterances; as well as One or more parameters of at least one of the plurality of machine learning language models are adjusted based at least in part on the loss function.

18. The one or more non-transitory computer-readable media of any preceding claim, wherein the language encoding model comprises a variational autoencoder model.

19. The one or more non-transitory computer-readable media of any preceding claim, wherein the plurality of machine learning language models comprises a plurality of language decoder models.

20. The one or more non-transitory computer-readable media of any preceding claim, wherein the language representation comprises an embedded sequence.

21. A computing system, the system comprising: one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: obtaining sequence data, wherein the sequence data describes a sequence; processing the sequence data with a coding model to generate a coded representation, wherein the coded representation describes the determined semantics of the sequence; processing the encoded representation with a plurality of machine learning expert models to generate a plurality of candidate outputs, wherein the plurality of machine learning expert models are associated with a plurality of different intents; processing the plurality of candidate outputs and the encoded representation with a management model to generate planning data; and A predicted output is generated based on the planning data, wherein the predicted output includes at least a portion of one or more candidate outputs of the plurality of candidate outputs.

Images