Reinforcement learning agent for multi-dimensional dialog action selection

Abstract

The present disclosure relates to reinforcement learning agents for multi-dimensional dialog action selection. A computer-implemented method for generating a response in a dialog system, wherein the dialog system is for conducting a dialog with a user, the method comprising: receiving an utterance from the user; updating a dialog state; a first dialog action agent selecting a first candidate action from a first set of candidate actions using a first machine learning model, wherein the first set of candidate actions is associated with a first dialog dimension; a second dialog action agent selecting a second candidate action from a second set of candidate actions using a second machine learning model, wherein the second set of candidate actions is associated with a second dialog dimension; an evaluation agent selecting an output action using a third machine learning model, wherein the output action comprises: a combination of the first candidate action and / or the second candidate action; and generating a system response.

Description

Technical Field

[0001] The embodiments described herein generally relate to a method and apparatus for generating responses in a dialogue system. Background Technology

[0002] Dialogue systems, such as task-oriented spoken dialogue systems (SDS), include systems that can continuously interact with humans to assist them in performing tasks through the use of speech. These tasks can include information retrieval, customer support, e-commerce, physical environment control, and human-robot interaction.

[0003] A dialogue system may include components that track what happens in a dialogue by incorporating information about current and previous user inputs (i.e., user turns) and system outputs (i.e., system turns) into a “dialogue state”.

[0004] The action selection component of a dialogue system selects actions from a fixed set of actions based on the dialogue strategy and dialogue state.

[0005] Discourses within a dialogue typically serve more than one communicative function. For example, they may move tasks forward, provide and elicit feedback, and manage who has the right to speak (i.e., turn management). Discourses with more than one communicative function are called multifunctional discourses. The ISO 24617-2 Annotation Standard is designed to support multifunctional discourses and defines an annotation scheme that includes multiple dimensions, each dedicated to a different aspect of the dialogue process.

[0006] Multidimensional dialogue systems that drive decision-making processes through several dialogue strategies, rather than a single one, have been developed. This approach reflects the multifunctional nature of human dialogue behavior. In this case, each dialogue action (DA) agent has its own dialogue strategy with a set of actions, and each DA agent generates at least one candidate dialogue action.

[0007] Multifunctional discourse requires a combination of dialogue actions from several different dimensions. However, some dimensions cannot be considered sufficiently independent. For example, dialogue actions from different dimensions may conflict with each other, making it possible for only one of them to be output. Therefore, it is necessary to evaluate the final combination of candidate dialogue actions selected by the agent (EA). Summary of the Invention

[0008] According to a first aspect, a computer-implemented method for generating a response in a dialogue system is provided, wherein the dialogue system is used to engage in dialogue with a user. The method includes: receiving a utterance from a user; updating a dialogue state based on the utterance; generating information identifying the dialogue state; a first dialogue action agent selecting a first candidate action from a first candidate action set using a first machine learning model and the information identifying the dialogue state, wherein the first candidate action set is associated with a first dialogue dimension; a second dialogue action agent selecting a second candidate action from a second candidate action set using a second machine learning model and the information identifying the dialogue state, wherein the second candidate action set is associated with a second dialogue dimension; an evaluation agent selecting an output action using a third machine learning model, the first candidate action, and the second candidate action, wherein the output action includes a combination of the first candidate action and / or the second candidate action; and generating a system response based on the output action.

[0009] In one embodiment, the dialogue system includes a microphone for receiving the speech.

[0010] In one embodiment, the dialogue system includes a speaker, and the method further includes outputting a system response via the speaker.

[0011] In this embodiment, the output action includes the first candidate action and the second candidate action.

[0012] In an embodiment, the output action includes either the first candidate action or the second candidate action.

[0013] In an embodiment, the output action does not include the first candidate action and the second candidate action, and preferably the output action indicates that no response will be generated.

[0014] In one embodiment, the first machine learning model, the second machine learning model, and the third machine learning model are trained independently using a single, identical reward signal.

[0015] In this embodiment, the first machine learning model implements a first dialogue strategy, the second machine learning model implements a second dialogue strategy, and the third machine learning model implements a third dialogue strategy.

[0016] In this embodiment, the natural language generator is a template-based natural language generator.

[0017] In one embodiment, the dialogue state includes a data structure that includes items that have been mentioned during the dialogue between the system and the user.

[0018] In an embodiment, each dialogue dimension is associated with a different aspect of the dialogue process and with a plurality of actions for responding to said aspect of the dialogue process.

[0019] In this embodiment, the dialogue dimension includes dimensions defined according to the ISO 25617-2 standard.

[0020] In this embodiment, the information used to identify the dialogue state is the dialogue state itself.

[0021] In this embodiment, the information for identifying the dialogue state is a set of features extracted from the dialogue state.

[0022] In an embodiment, generating information identifying the dialogue state includes: generating a first instance of information identifying the dialogue state; generating a second instance of information identifying the dialogue state; and wherein a first candidate action is selected based on the first instance of information identifying the dialogue state; and a second candidate action is selected based on the second instance of information identifying the dialogue state.

[0023] In an embodiment, the first instance of information identifying the dialogue state and the second instance of information identifying the dialogue state include different sets of dialogue state features.

[0024] In one embodiment, the system response is generated by a natural language generator.

[0025] In this embodiment, the system response is a speech.

[0026] In an embodiment, selecting the output action includes: generating a value vector that includes an indication of an estimated cumulative reward for each combination of the first candidate action and / or the second candidate action, in a given dialogue state, associated with selecting the corresponding combination as the system response; and selecting the output action based on the value vector.

[0027] In one embodiment, the value vector is generated using a function parameterized by a first set of trainable weights, wherein the input to the function is information identifying the dialogue state, and the output of the function is an estimated cumulative reward for each combination.

[0028] In this embodiment, the function is a linear function.

[0029] In one embodiment, the third machine learning model includes a linear layer configured to implement the linear function.

[0030] In this embodiment, a (artificial) neural network is used to implement the linear layer.

[0031] In one embodiment, selecting the output action based on the value vector includes: converting the value vector into a probability distribution to generate an action probability vector; and selecting the output action based on the action probability vector.

[0032] In one embodiment, the value vector is converted into a probability distribution by applying the softmax function to the value vector.

[0033] In this embodiment, the probability distribution is a normalized probability distribution.

[0034] In an embodiment, converting the value vector into the probability distribution to generate the action probability vector includes: converting the value vector into a converted action value vector, wherein the converted action value vector includes values ​​greater than or equal to zero; forming a masked converted action value vector by setting a first action value in the converted action value vector to zero based on a predetermined criterion; and normalizing the masked converted action value vector to form the action probability vector.

[0035] In an embodiment, the converted action value vector contains only values ​​greater than or equal to zero.

[0036] In an embodiment, the masking probability vector is formed by a masking layer configured to take the transformed action value vector as input and provide the masking probability vector as output.

[0037] In an embodiment, the predetermined criteria include at least one of the following: a list of meaningless combinations, and / or a list of combinations not supported by downstream components (such as a natural language generator).

[0038] In one embodiment, setting the first action value to zero disables the combination of the first candidate action and / or the second candidate action.

[0039] In this embodiment, the first dialogue dimension is different from the second dialogue dimension.

[0040] In this embodiment, the user is a human user.

[0041] In one embodiment, the dialogue system includes a microphone, and receiving the utterance from the user includes generating a representation of the human speech received via the microphone.

[0042] In one embodiment, the dialogue state is updated based on the utterance and the previous instance of the dialogue state.

[0043] In one embodiment, the dialogue system includes a speaker, and the method further includes outputting the system response via the speaker such that the system response can be heard by the user.

[0044] According to a second aspect, a computer implementation method for training a dialogue system is provided, the method comprising: generating a system response according to the above method; providing the system response to the user; and training a first machine learning model, a second machine learning model, and a third machine learning model using reinforcement learning.

[0045] In this embodiment, the user is a simulated user.

[0046] In one embodiment, using reinforcement learning to train the first machine learning model, the second machine learning model, and the third machine learning model includes: determining a cumulative reward; and training the first machine learning model, the second machine learning model, and the third machine learning model based on the cumulative reward.

[0047] In this embodiment, Monte Carlo control with linear value function approximation is used to train the first machine learning model, the second machine learning model, and the third machine learning model.

[0048] In this embodiment, the first machine learning model, the second machine learning model, and the third machine learning model are trained separately.

[0049] In one embodiment, the cumulative reward is the sum of individual rewards obtained between providing the system response and completing the dialogue.

[0050] In one embodiment, the first machine learning model is configured to estimate a first estimated cumulative reward for the first candidate output action based on a first function parameterized by a first set of trainable weights; and wherein training the first machine learning model based on the cumulative reward includes: determining a difference between the cumulative reward and the first estimated cumulative reward; and updating the first set of trainable weights based on the difference.

[0051] In this embodiment, the difference is the mean squared error.

[0052] In one embodiment, updating the first trainable weight set includes: using gradient descent and determining the weight change of each weight in the first weight set based on the difference.

[0053] In one embodiment, the first set of trainable weights is updated to minimize the difference between the cumulative reward (i.e., the “true” cumulative reward) and the first estimated cumulative reward.

[0054] In one embodiment, the third machine learning model is configured to estimate a second estimated cumulative reward for the output action based on a second function parameterized by a second set of trainable weights; and training the third machine learning model based on the cumulative reward includes: determining the difference between the cumulative reward and the second estimated cumulative reward; and updating the second set of trainable weights based on the difference.

[0055] In one embodiment, the cumulative reward is based on the sum of rewards, which includes a first reward generated for providing the system response.

[0056] In one embodiment, the first reward comprises the sum of a first component and a second component, wherein the first component provides a penalty to prevent long conversations, and the second component provides a reward for completing a predefined goal.

[0057] In an embodiment, the method further includes: receiving a second utterance in response to providing the system response; generating a second system response according to the method for generating a response in a dialogue system, wherein the utterance in the method for generating a response in a dialogue system is the second utterance, and the system response generated by the method for generating a response in a dialogue system is the second system response; generating a second reward based on the second system response; and calculating the cumulative reward based on the sum of the first reward and the second reward.

[0058] In one embodiment, the sum of the first reward and the second reward is a discount sum, such that the value of the second reward in the sum is reduced by a discount factor.

[0059] In an embodiment, the dialogue system includes a first hyperparameter configured to control the exploration amount of the following dialogue strategies: a first dialogue strategy implemented by a first machine learning model, a second dialogue strategy implemented by a second machine learning model, and a third dialogue strategy implemented by a third machine learning model, wherein: the first hyperparameter has a first value during a first dialogue, the first dialogue including the utterance and the system response, and the method further includes: determining whether the first dialogue is completed, and in response to determining that the first dialogue is completed: modifying the first hyperparameter such that the exploration amount of the first dialogue strategy, the second dialogue strategy, and the third dialogue strategy in the state-action space is less than the exploration amount during the first dialogue; and retraining the dialogue system with a second dialogue.

[0060] In an embodiment, each of the first machine learning model, the second machine learning model, and the third machine learning model includes a transformation layer configured to convert an action value vector into a transformed action value vector, wherein the transformation layer is controlled according to a temperature hyperparameter, and wherein modifying the first hyperparameter includes reducing the value of the temperature hyperparameter to make it closer to 1.

[0061] In this embodiment, one-hot encoding is used to represent the output of the first machine learning model.

[0062] In an embodiment, the output of the first machine learning model includes a first vector, wherein the first vector includes a plurality of entries, wherein each entry is associated with a different action in a first action set, and wherein the entry associated with the first candidate action includes information indicating that the candidate action has been selected by the first machine learning model, and other entries among the plurality of entries include information indicating that candidate actions associated with the other entries have not yet been selected by the first machine learning model.

[0063] According to a third aspect, an apparatus is provided for generating a response in a dialogue system, wherein the dialogue system is used to engage in dialogue with a user, the apparatus being configured to: receive utterances from the user; update a dialogue state based on the utterances; generate information identifying the dialogue state; a first dialogue action agent selects a first candidate action from a first candidate action set using a first machine learning model and the information identifying the dialogue state, wherein the first candidate action set is associated with a first dialogue dimension; a second dialogue action agent selects a second candidate action from a second candidate action set using a second machine learning model and the information identifying the dialogue state, wherein the second candidate action set is associated with a second dialogue dimension; an evaluation agent selects an output action using a third machine learning model, the first candidate action, and the second candidate action, wherein the output action includes: a combination of the first candidate action and / or the second candidate action; and generate a system response based on the output action.

[0064] In an embodiment, the device is further configured, when selecting the output action, to: generate a value vector including an indication of an estimated cumulative reward for each combination of the first candidate action and / or the second candidate action, in a given dialogue state, associated with selecting the corresponding combination as the system response; and to select the output action based on the value vector.

[0065] In an embodiment, the device is further configured to generate the value vector using a function parameterized by a first set of trainable weights, wherein the input to the function is information identifying the dialogue state, and the output of the function is an estimated cumulative reward for each combination.

[0066] In this embodiment, the function is a linear function.

[0067] In an embodiment, the device is further configured to: convert the value vector into a probability distribution to generate an action probability vector when selecting the output action based on the value vector; and select the output action based on the action probability vector.

[0068] In an embodiment, the device is further configured to: convert the value vector into a converted action value vector, wherein the converted action value vector includes values ​​greater than or equal to zero; form a masked converted action value vector by setting a first action value in the converted action value vector to zero based on a predetermined criterion; and normalize the masked converted action value vector to form the action probability vector.

[0069] In this embodiment, the first dialogue dimension is different from the second dialogue dimension.

[0070] According to a fourth aspect, a training system for training a dialogue system is provided, the training system including the means discussed above for generating a response in the dialogue system, the training system being configured to: generate a system response using the means discussed above for generating a response in the dialogue system; provide the system response to the user; and train a first machine learning model, a second machine learning model, and a third machine learning model using reinforcement learning.

[0071] In an embodiment, the training system is further configured to: determine a cumulative reward; and train the first machine learning model, the second machine learning model, and the third machine learning model using reinforcement learning.

[0072] In one embodiment, the first machine learning model is configured to: estimate a first estimated cumulative reward for the first candidate output action based on a first function parameterized by a first set of trainable weights; and wherein the training system, when training the first machine learning model based on the cumulative reward, is further configured to: determine the difference between the cumulative reward and the first estimated cumulative reward; and update the first set of trainable weights based on the difference.

[0073] In one embodiment, the third machine learning model is configured to: estimate a second estimated cumulative reward for the output action based on a second function parameterized by a second set of trainable weights; and wherein the training system, when training the third machine learning model based on the cumulative reward, is further configured to: determine the difference between the cumulative reward and the second estimated cumulative reward; and update the second set of trainable weights based on the difference.

[0074] In one embodiment, the cumulative reward is based on the sum of rewards, which includes a first reward generated for providing the system response.

[0075] In an embodiment, the training system is further configured to: receive a second utterance in response to providing the system response; generate a second system response using the means discussed above for generating responses in a dialogue system, wherein the utterance received by the means is the second utterance, and the system response generated by the means is the second system response; generate a second reward based on the second system response; and calculate the cumulative reward based on the sum of the first reward and the second reward.

[0076] In an embodiment, the dialogue system includes a first hyperparameter configured to control the exploration amount of the following dialogue strategies: a first dialogue strategy implemented by a first machine learning model, a second dialogue strategy implemented by a second machine learning model, and a third dialogue strategy implemented by a third machine learning model, wherein: the first hyperparameter has a first value during a first dialogue, the first dialogue including the utterance and the system response, and the training system is further configured to: determine whether the first dialogue is completed, and in response to determining that the first dialogue is completed: modify the first hyperparameter such that the exploration amount of the first dialogue strategy, the second dialogue strategy, and the third dialogue strategy in the state-action space is less than the exploration amount during the first dialogue; and retrain the dialogue system with a second dialogue.

[0077] According to a fifth aspect, a non-transitory computer-readable medium is provided, the non-transitory computer-readable medium comprising computer program instructions suitable for execution by a processor, the instructions being configured, when executed by the processor, to cause the processor to perform the methods discussed above. Attached Figure Description

[0078] The arrangement will be more fully understood and appreciated from the following detailed description, which is given only as an example and in conjunction with the accompanying drawings:

[0079] Figure 1 This demonstrates a dialogue system based on an example;

[0080] Figure 2 This demonstrates a multi-dimensional action selection component based on an example;

[0081] Figure 3A An example implementation of a reinforcement learning (RL) machine learning model is shown;

[0082] Figure 3B A second example implementation of a reinforcement learning (RL) machine learning model is shown;

[0083] Figure 4A The implementation of the evaluation agent 205 is shown based on an example;

[0084] Figure 4B The representation of the action probability vector is shown based on the example;

[0085] Figure 4C The implementation of the evaluation agent 205 is shown based on the second example;

[0086] Figure 5 This demonstrates a method for generating system discourse based on examples;

[0087] Figure 6 This demonstrates a method for selecting components based on multidimensional actions in an example;

[0088] Figure 7 A comparison of the dialogue success rates of a multidimensional system using a manually coded evaluation agent and an evaluation agent based on the examples described herein is shown.

[0089] Figure 8A This demonstrates a transfer learning approach using multidimensional action selection components, based on examples.

[0090] Figure 8B This illustrates a transfer learning approach using a multidimensional action selection component, based on the second example.

[0091] Figure 9 A schematic diagram of hardware that can be used to implement the method according to the embodiment is shown. Detailed Implementation

[0092] Figure 1 A dialogue system is shown as an example. The dialogue system includes a dialogue action marker component 100, a dialogue state tracking component 101, an action selection component 102, and a natural language generator 103.

[0093] The dialogue action marker component 100 is configured to acquire information that identifies user input (optionally, spoken user input, such as that received via a microphone). Alternatively, the user input may be in text form.

[0094] In the example where spoken user input is received via a microphone, an automatic speech recognition component (not shown) performs automatic speech recognition (ASR) to recognize the user's speech and convert it into text. In one example, the automatic speech recognition component uses... This can be achieved through the Web Speech API.

[0095] The input to the dialogue action tagger component 100 is text input by the user. The dialogue action tagger component 100 is configured to recognize one or more dialogue action tags based on the user-input text and output the recognized dialogue action tags to the dialogue state tracking component 101. The dialogue action tagger component 100 is also configured to output the user-input text to the dialogue state tracking component 101. The dialogue action tagger component is used to identify domain-independent dialogue actions required by the action selection component 101, such as greetings and thanks. Optionally, the dialogue action tagger component 100 is a general-purpose dialogue action tagger.

[0096] The dialogue state tracking component 101 is configured to generate and update the dialogue state after each received utterance. In this example, the dialogue state tracking component 101 takes user text and identified dialogue action tags (both output by the dialogue action tagger 100) as input and generates an updated dialogue state that includes the identified dialogue action tags. In this example, the dialogue state tracker component 101 is configured to consider the user utterance (delivered by the dialogue action tagger component 100) and the current dialogue state (stored from the previous round) to identify updates for the user's goal.

[0097] In this example, the representation of the dialogue state takes the form of a data structure that includes items that have been mentioned during the dialogue between the system and the user. In a further example, the representation of dialogue state 101 includes structured information about the dialogue so far, such as the user's goals.

[0098] In another example, the dialogue state stores information by providing slots. In a specific example where the dialogue system is used in the restaurant search domain, the dialogue state includes "target" constraints represented by three slots: food, region, and price range (e.g., cheap, medium, expensive). At the start of the dialogue, each slot is empty, and as more information is gathered from the user, the slots are filled by the dialogue state tracking component 101.

[0099] In another example, the dialogue state tracking component 101 is an action state update model for dialogue state tracking, as described by reference in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 7398-7402, by Svetlana Stoyanchev, Simon Keizer and Rama Doddipatla, “Action state update approach to dialogue management”.

[0100] More generally, the dialogue state maintained by the dialogue state tracking component 101 includes information that identifies the system’s view on the user’s goals and optionally includes dialogue history, which includes the items discussed above.

[0101] The dialogue state tracking component 101 is configured to output a representation of the dialogue state to the action selection component 102.

[0102] Action selection component 102 is configured to receive a representation of the dialogue state and generate information indicative of a system response based on the dialogue strategy of the action selection component and the received representation of the dialogue state. In this example, action selection component 102 is implemented according to an example of a multidimensional action selection component described below. Action selection component 102 is configured to output information identifying dialogue actions that will form a system response. As will be discussed in more detail below, the system response can take the form of multi-functional utterances.

[0103] The natural language generator 103 is configured to receive output information from the action selection component 102 (i.e., information identifying dialogue actions that will form a system response) and is configured to generate a system response in the form of utterances.

[0104] Optionally, the natural language generator 103 is a template-based natural language generator (NLG). In the example, the template-based natural language generator 103 is implemented as described in "Reiter, E., & Dale, R. (2000). Building Natural Language Generation Systems (Studies in Natural Language Processing). Cambridge: Cambridge University Press".

[0105] In one example, the natural language system response (i.e., the generated utterance) is output as an audio waveform via a speaker so that the system response can be received by a human user. Optionally, when the output of the natural language generator 103 is natural language utterance in text form, the output of the natural language generator 103 may optionally be transmitted via a speaker before being output. The Web Speech API converts text into speech.

[0106] exist Figure 1 In the example shown, the dialogue system is used in the restaurant search domain and receives the input statement: "I'm looking for cheap restaurants." Using the functionality of the dialogue state tracking component 101, the action selection component 102, and the natural language generator 103, the dialogue system provides the user with restaurant recommendations (e.g., the dialogue system provides the system response "Prezzo is a popular Italian restaurant").

[0107] While the above discussion of dialogue systems in the context of restaurant search highlights the fact that dialogue systems can be modified for numerous applications. One possible use is information retrieval. However, other applications are possible, such as information gathering, troubleshooting, customer support, e-commerce, physical environment control, and human-robot interaction.

[0108] Figure 2 This demonstrates a multi-dimensional action selection component based on an example. Figure 2 The illustrated multidimensional action selection component includes (e.g., received from the dialogue state tracking component 101) information identifying the dialogue state 201. As discussed above, the information identifying the dialogue state 201 includes a systemic view of the user's goals and a dialogue history, which includes items discussed earlier (e.g., previous user utterances and system utterances).

[0109] In a further example, the information for identifying dialogue state 201 is a set of features extracted from the complete dialogue state. In this context, features are properties or characteristics of the complete dialogue state.

[0110] In a further example, the information identifying dialogue state 201 includes multiple feature sets (e.g., a first feature set, a second feature set, and a third feature set), where each feature set is associated with a different dialogue action (DA) agent (discussed in more detail below). Optionally, the feature sets among the multiple feature sets may overlap. That is, two feature sets among the multiple feature sets may include common features.

[0111] The multi-dimensional action selection component further includes a first dialogue action (DA) agent 202, a second dialogue action (DA) agent 203, and a third dialogue action (DA) agent 204.

[0112] The first dialogue action (DA) agent 202, the second dialogue action (DA) agent 203, and the third dialogue action (DA) agent 204 are each configured to access information identifying the dialogue state 201, as discussed above, which includes a set of features extracted from the complete dialogue state and uniquely associated with the respective dialogue action (DA) agent.

[0113] The first Dialogue Act (DA) agent 202, the second Dialogue Act (DA) agent 203, and the third Dialogue Act (DA) agent 204 are each also associated with different dimensions of the Dialogue Act taxonomy described in the ISO 24617-2 labeling standard. A brief introduction to that standard is provided below. However, a more comprehensive description is also provided by reference in “Bunt, Harry & Petukhova, Volha & Traum, David & Alexandersson, Jan. (2017). Dialogue Act Annotation with the ISO 246172 Standard. 10.1007 / 978-3-319-42816-1_6”, which is incorporated herein by reference.

[0114] The ISO 24617-2 annotation standard is designed to annotate spoken, written, and multimodal dialogues using information about the dialogical actions that constitute a dialogue. A dialogical action is defined as a communicative activity of the participants in the dialogue, which is interpreted as having a specific communicative function and semantic content. As discussed above, utterances in a dialogue typically have more than one communicative function. To capture the different aspects of the dialogue process, the ISO 24617-2 annotation standard defines the concept of "dimensions."

[0115] The dimensions specified in the ISO 24617-2 standard include, but are not limited to, the task dimension, the automatic feedback dimension, and the social responsibility management (SOM) dimension. Other dimensions include round transitions and task management.

[0116] Task dimensions are associated with conversational actions that incentivize dialogue and move tasks or activities forward. For example, when a multi-dimensional action selection component is used in the restaurant search domain, the task dimension is responsible for recommending restaurants. More generally, task dimensions are associated with actions that focus on underlying tasks or activities (e.g., information navigation, teaching, negotiation, etc.).

[0117] The automatic feedback dimension relates to dialogical actions that provide or elicit information about the current speaker's processing and the listener's previous utterances (e.g., asking the user to repeat / paraphrase their previous input after a processing question). The automatic feedback dimension can be used to indicate that the user input is not understood when the speech recognition component (upstream of the multidimensional action selection component) does not return any results. An example output from an action in the automatic feedback dimension could be: "I don't quite understand, could you please repeat that?". Actions in the automatic feedback dimension can also be used to provide clear feedback in the form of explicit or implicit confirmation when the system is unsure about the user input. An example of this could be: "You mean it's expensive?".

[0118] The Social Responsibility Management (SOM) dimension relates to conversational actions used to address social norms, such as greetings, introducing oneself, apologies, and thanks.

[0119] exist Figure 2 In this system, each dimension is associated with a set of possible summary actions (collectively referred to as actions). Summary actions are dialogic actions to be specified, which may require additional information from the dialogic state to be mapped to fully specified dialogic actions. For example, the summary action “REQUEST” is mapped to the dialogic action of type “request” by determining which area is requested based on the dialogic state (e.g., request (area)). Similarly, the summary action “OFFER” is mapped to a fully specified dialogic action by determining which location should be offered. The mapping from summary actions to dialogic actions is performed by a dialogic action (DA) agent. In this example, summary actions are mapped to dialogic actions using a fixed heuristic (e.g., using a predefined mapping). The dialogic actions generated by the dialogic action (DA) agent are used by the natural language generator 103 to generate system utterances.

[0120] As discussed above, each Dialogue Action (DA) agent generates a dialogue action based on a selected action and optionally using additional information from the dialogue state. In some cases, the DA agent's policy selection directly corresponds to the action of the dialogue action (no additional information required, e.g., ACCEPT_THANKING). In other cases, the DA agent selects an action that requires additional information to form a summary action of the complete dialogue action (e.g., REQUEST).

[0121] Advantageously, using summary actions instead of fully specified dialogue actions reduces the size of the action set used by the dialogue action (DA) agent's strategy, thus improving the ease and speed of training the dialogue action (DA) agent.

[0122] In the example, the dialogue actions implemented by the system are at least partially based on a standard such as ISO 24617-2. However, these standards tend to focus only on dialogue action types / communication functions and are intended to be domain-independent. Consequently, in a further example, the system implements dialogue actions specific to the domain in which the system operates. In this example, the dialogue actions implemented by the system require semantic content, which may optionally be represented as a list of interval-value pairs (e.g., food = Chinese food, region = city center). In this example, the range of interval-value pairs is described in the domain ontology. The specific dialogue actions implemented by the system are a matter of system design, as several dialogue system components must support this set of dialogue actions, such as components for natural language understanding and generation, and state tracking and action selection (policy) components.

[0123] In specific examples: the task dimension is associated with a set of actions including “OFFER”, “ANSWER”, “REQUEST”, and “NONE”; the automatic feedback dimension is associated with a set of actions including “IMPL-CONFIRM”, “EXPL_CONFIRM”, “AUTO_NEGATIVE”, and “NONE”; and the Social Responsibility Management (SOM) dimension is associated with a set of actions including “ACCEPT_THANKING”, “RETURN_GOODBYE”, and “NONE”.

[0124] In this example, the "OFFER" action can produce output statements that include a suggestion to satisfy the goal (e.g., in the case of a multidimensional action selection component being used in the restaurant search domain, the "OFFER" action could result in the output statement: "How about Rice Boat?", where "Rice Boat" is the name of the restaurant). The "ANSWER" action can produce output statements that include an answer to the question (e.g., "The address of Rice Boat is..."). The "REQUEST" action can produce output statements that request more information (e.g., "What price range are you looking for?"). The "IMPL-CONFIRM" action can be used to produce output statements that implicitly confirm the system has understood the input (e.g., "Okay. Italian food...", implicitly confirming that the input request is related to Italian food). The "EXPL_CONFIRM" action can be used to produce output statements that implicitly confirm the system has understood the user input (e.g., "You want Italian food, right?"). The "AUTO_NEGATIVE" action can be used to produce output statements that convey the system's uncertainty about the user input (e.g., "I don't quite understand, could you please rephrase it?"). The “ACCEPT_THANKING” and “RETURN_GOODBYE” actions are self-explanatory. Example utterances based on the “ACCEPT_THANKING” action could include “You’re welcome,” and example utterances based on “RETURN_GOODBYE” could include “Have a good time.” In the examples, the “NONE” action causes a predetermined dialogic action to be selected (e.g., in this example, the “NONE” action causes the dialog system to select the “autoNegative” dialogic action).

[0125] In the specific examples above, the action set includes a particular action. However, to avoid any ambiguity, it is emphasized that the corresponding action set may include other actions not explicitly discussed above.

[0126] exist Figure 2 In the multidimensional action selection component, the first dialogue action (DA) agent 202 is associated with the "task" dimension, the second dialogue action (DA) agent 203 with the "automatic feedback" dimension, and the third dialogue action (DA) agent 204 with the "Social Responsibility Management (SOM)" dimension. Advantageously, these three agents (corresponding to the three dimensions) are considered to be the minimum required for a task-oriented multidimensional dialogue system. Although Figure 2 Three Dialogue Action (DA) agents are shown, but it is emphasized that any number of Dialogue Action (DA) agents, more than one, can be provided.

[0127] Consider the first dialogue action (DA) agent 202 in more detail. The first dialogue action (DA) agent 202 is associated with the "task" dimension and is configured to generate candidate actions based on information identifying the dialogue state 201 and a dialogue policy implemented by the first dialogue action (DA) agent 202. The dialogue policy associated with the first dialogue action (DA) agent 202 is used to select candidate actions from the set of possible actions based on the input information identifying the dialogue state 201.

[0128] As discussed above, the set of possible actions includes multiple summary actions. Summary actions are dialog actions to be specified, which may require additional information from the dialog state to be mapped to fully specified dialog actions. In the example, the action is mapped to a full dialog action using additional information from the dialog state. In a further example, the actions in the set of possible actions correspond to communication functions from the ISO 24617-2 standard.

[0129] The first dialogue action (DA) agent 202 includes a reinforcement learning (RL) machine learning model, or in other words, a machine learning model trained using reinforcement learning (RL). As is known in the art, reinforcement learning (RL) is a type of machine learning that learns a policy (i.e., which action is chosen when given a state) from rewards (also known as reinforcements) generated from previous segments.

[0130] The training of the various machine learning models in the Dialogue Action (DA) agent will be discussed in more detail below. However, in brief, during training, the first Dialogue Action (DA) agent 202 is configured to learn an action value function that estimates the cumulative reward obtained when starting in state S (corresponding to information recognizing dialogue state 201) and selecting an output action from a set of possible actions for a specific dimension (i.e., the "task" dimension). As will be discussed in more detail below, the reinforcement learning (RL) machine learning model of the first Dialogue Action (DA) agent 202 uses (e.g., based on reward signals generated from interactions with the (simulated) user) to train the parameters of the action value function in order to better estimate the cumulative reward associated with taking an action from a given state.

[0131] During inference, the first dialogue action (DA) agent 202 is configured to generate candidate actions for output from a set of possible actions associated with the "task dimension" based on the input dialogue state 201 and the learned dialogue policy. Specifically, the machine learning model of the first dialogue action (DA) agent 202 is configured to determine the cumulative reward associated with selecting each action from the set of actions given the current state S (corresponding to information identifying dialogue state 201). The machine learning model is further configured to select a (single) candidate action for the task dimension based on the estimated cumulative reward.

[0132] Regarding Figure 3A Let's discuss in more detail the example implementation of the first reinforcement learning (RL) machine learning model (also known as the first machine learning model).

[0133] Figure 3A An example implementation of a reinforcement learning (RL) machine learning model is shown.

[0134] In the example, the reinforcement learning (RL) machine learning model of the first dialogue action (DA) agent 202 can contain several stages. In one example, the machine learning model of the first dialogue action (DA) agent 202 includes a linear layer 302 with trainable parameters.

[0135] The input to the linear layer 302 includes information that identifies the dialogue state 301 (e.g., a feature set representing the complete dialogue state, and optionally, a feature set representing the complete dialogue state uniquely associated with the first dialogue action (DA) agent 202).

[0136] Linear layer 302 is configured to generate the expected long-term cumulative reward associated with selecting an action from the set of actions given the current state (i.e., given the current dialogue state). In this case, linear layer 302 implements an action-value function. In one example, the linear layer is implemented using an (artificial) neural network. In one example, the linear layer is a neural network with a single hidden layer. In a further example, the linear layer is implemented using more than one hidden layer.

[0137] The output of linear layer 302 includes a value vector 303 (also referred to as an action value vector), where each value indicates the extent to which an action (from the set of actions associated with the relevant dimension) maximizes the long-term cumulative reward. Optionally, the value indicates the long-term cumulative reward associated with taking each action.

[0138] As will be discussed in more detail below, during the training of a reinforcement learning (RL) machine learning model, the reward signal is used to update the trainable parameters of the linear layer so that the expected return (i.e., the cumulative reward) is closer to the actual return received.

[0139] In the example, the machine learning model of the first dialogue action (DA) agent 202 also includes a transformation layer 304. The output of the linear layer 302 (i.e., the action value vector 303) is the input to the transformation layer 304.

[0140] Transformation layer 304 is configured to transform the action value vector (where each value can take values ​​between negative and positive infinity) into a transformed action value vector 305 that takes any value greater than or equal to 0. In this example, transformation layer 304 transforms each input action value y in action value vector 303. i calculate In fact, all action values ​​are converted to values ​​greater than or equal to zero.

[0141] In this embodiment, the conversion layer 304 processes each input action value y in the action value vector 303. i calculate Where τ is the temperature hyperparameter. As is known in the art, hyperparameters are parameters chosen to control the learning process. Specifically, the temperature hyperparameter is used to control the level of exploration during the training of a dialogue policy implemented through a reinforcement learning (RL) machine learning model. A higher temperature is used during the early training phase and causes actions (from the set of possible actions associated with a dimension) to be selected more randomly. A lower temperature is used during the later training phase and causes the estimated values ​​to be followed more closely (i.e., high-value actions are selected with a higher probability). The temperature hyperparameter thus controls the exploration vs. exploitation of the dialogue policy.

[0142] In the equations above, e is the standard exponential function, and τ is the temperature hyperparameter, taking values ​​between 0 and infinity. As the temperature hyperparameter approaches infinity, the probability distribution becomes uniform. As will be discussed in more detail below, specifically when discussing the training of the multidimensional action selection component, the temperature hyperparameter can be changed throughout the training period. In one example, the temperature hyperparameter is fixed during inference, optionally set to 1.

[0143] The output of the transformation layer 304 is a transformed motion value vector 305. The transformed motion value vector 305 is input into the masking layer 306.

[0144] Masking layer 306 is coupled to the output of transformation layer 304. Masking layer 306 is configured to disable actions in the transformed action value vector 305 (e.g., by setting their associated values ​​to zero) to produce a masked transformed action value vector 307. In this example, if, for example, the selected summary action cannot be mapped to a full dialogue action because the required information is lost in the full dialogue state, or the output action is not supported by downstream components (e.g., natural language generator 103), then masking layer 306 in the first dialogue action (DA) agent disables the action. The output of masking layer 306 is coupled to the input of normalization layer 308.

[0145] Normalization layer 308 is configured to receive the masked, transformed action value vector 307. Normalization layer 308 is configured to normalize the values ​​in the masked, transformed action value vector 307 to produce an action probability vector in which the sum of probabilities equals 1. In other words, the masked, transformed action values ​​307 (which can take any value greater than or equal to zero) are normalized to a probability distribution in which the masked action has a probability of zero.

[0146] In the example, normalizing the values ​​in the masked transformed action value vector 307 involves dividing each action value in the masked transformed action value vector 307 by the sum of the action values ​​in the masked transformed action value vector 307. The output of the normalization layer 308 is an action probability vector.

[0147] The first dialogue action (DA) agent 202 is further configured to select candidate output actions from the action set based on action probability vectors.

[0148] In one example, the output of the first dialogue action (DA) agent 202 includes a one-hot code representation indicating the action selected from the set of possible actions (e.g., a binary "1" indicates the selected action, and a binary "0" indicates the unselected action). Actions can be selected based on values ​​in the action probability vector (e.g., by selecting the action associated with the highest probability, or by sampling actions based on a probability distribution in the action probability vector).

[0149] about Figure 3A The examples discussed illustrate a machine learning model that includes a masking layer 306. However, it is emphasized that the masking layer 306 is optional. In this case, the transformed action value vector 305 is fed into the normalization layer 308. Optionally, when the masking layer is absent, the functionality of the transformation layer 304 and the normalization layer 308 can be combined in a softmax layer.

[0150] As is known in the art, the softmax function is a function that transforms a vector of K real values ​​into a vector of K real values ​​that sum to 1. Input values ​​can be positive, negative, zero, or greater than 1, but softmax transforms them to values ​​between 0 and 1 so that they can be interpreted as probabilities. In this case, the softmax layer is configured to transform action values ​​(i.e., action value vector 303) that indicate long-term cumulative reward into a normalized probability distribution. The output of the softmax layer comprises an action probability vector, where each probability is associated with a distinct action in the set of possible actions, and each probability indicates the likelihood that a given action maximizes the long-term cumulative reward.

[0151] In the example, the softmax layer implements the softmax equation: Where σ is the softmax function, y is the input vector (i.e., the action value vector 303), n is the magnitude of the input vector, and e is the standard exponential function. In this example, the temperature hyperparameter is used as discussed above, and in this case, the softmax layer is implemented as follows: Where τ is a temperature hyperparameter that takes values ​​between 0 and infinity.

[0152] In a further example where masking layer 306 exists, the functionality of masking layer 306 and transformation layer 304 (i.e., converting action values ​​to numbers greater than or equal to zero) is combined in one layer.

[0153] Figure 3B This shows a second example implementation of a reinforcement learning (RL) machine learning model. Figure 3B In, with Figure 3A The same number is used to represent the same component. Figure 3B Further, a modified masking layer 350 is included. The modified masking layer 350 is configured to receive an action value vector 303, which includes an estimated cumulative reward Q(s,a) for taking action a in state s. First, exp(Q(s,a) / τ) values ​​are calculated for all actions (where τ is the temperature hyperparameter and "exp" is an exponential function). Then, a mask (mask(s,a)) (i.e., mask(s,a)*exp(Q(s,a) / τ)) is applied, setting some of these values ​​to zero, thereby generating a second action value vector 352. Finally, the resulting values ​​(i.e., the values ​​in the second action value vector 352) are normalized to a probability distribution (where the masked actions have a probability of zero) through a normalization layer as discussed above.

[0154] exist Figure 2In the example, the output of the reinforcement learning (RL) machine learning model of the first dialogue action (DA) agent includes information that the "OFFER" summary action has been responded to in response to the identification of dialogue state 201, and an indication that it has been selected from the set of possible actions as the system response in the "task" dimension. In other words, the output of the first dialogue action (DA) agent 202 includes an indication that the action in the "task" dimension is the "OFFER" summary action.

[0155] In addition to outputting an indication of the selected candidate action, the first dialogue action (DA) agent 202 is also configured to output a candidate dialogue action associated with the selected candidate action. In this example, the first dialogue action (DA) agent 202 generates the candidate dialogue action based on the selected candidate action and information identifying the dialogue state 201.

[0156] As discussed above, Figure 1 The multidimensional dialogue system also includes a second dialogue action (DA) agent 103 and a third dialogue action (DA) agent 104.

[0157] The second dialogue action (DA) agent 103 and the third dialogue action (DA) agent 104 comprise reinforcement learning (RL) machine learning models similar to those discussed above with respect to the first dialogue action (DA) agent 102. However, it will be appreciated that the actions are selected from different sets of actions because each dialogue action (DA) agent is associated with a different dimension of the ISO 24617-2 annotation standard; therefore, candidate actions are selected from different sets of possible actions. Furthermore, with each dialogue action (DA) agent using a different set of features extracted from the dialogue state, it will be appreciated that the information for each dialogue action (DA) agent to identify the dialogue state 101 will be a different set of features (optionally with overlapping features).

[0158] Consider in more detail the Second Dialogue Action (DA) Agent 203.

[0159] Training the second dialogue action (DA) agent 203 involves learning an action value function that estimates the cumulative reward obtained when starting in state S (corresponding to information recognizing dialogue state 201) and selecting an output action from a set of possible actions for a specific dimension (i.e., the "auto-feedback" dimension). As will be discussed in more detail below, the reinforcement learning (RL) machine learning model of the second dialogue action (DA) agent 203 uses the reward signal (i.e., the same reward signal as the first dialogue action (DA) agent 202) to train the parameters of the linear layer implementing the action value function. Although training the second dialogue action (DA) agent 203 is based on a common reward signal, it is emphasized that the training is independent.

[0160] During inference, the second dialogue action (DA) agent 203 is configured to generate candidate actions for output from a set of possible actions associated with the "automatic feedback" dimension, based on input information recognizing the dialogue state 201 and a learned dialogue policy. Specifically, the machine learning model of the second dialogue action (DA) agent 203 is configured to determine the cumulative reward associated with each action selected in the action selection process given the current state (i.e., given the current dialogue state). The machine learning model is further configured to select a (single) candidate action for the "automatic feedback" dimension based on the estimated cumulative reward.

[0161] exist Figure 2 In the example, the output of the second dialogue action (DA) agent 203 includes an indication that an "IMPL-CONFIRM" (i.e., implicit confirmation) action has been selected as the system response in response to information recognizing dialogue state 201, within the "Auto-feedback" dimension. The summary action selected by the second dialogue action (DA) agent 203 is output to the evaluation agent 205, which determines whether the candidate action becomes final. As discussed above, the second dialogue action (DA) agent 202 is also configured to base its decisions on candidate actions (in... Figure 2 In the example, the candidate dialogue action is generated and output using information from the "IMPL-CONFIRM" action and the identification of the dialogue state 201.

[0162] The third dialogue action (DA) agent 204 is configured in a similar manner to the first dialogue action (DA) agent 202 and the second dialogue action (DA) agent 203, but is associated with the "Social Responsibility Management (SOM)" dimension.

[0163] exist Figure 2 In the example, the output of the third dialogue action (DA) agent 204 includes information that the "ACCEPT_THANKING" action has been responded to in response to the identification of dialogue state 201, and an indication that it has been selected as the system response in the "Social Responsibility Management (SOM)" dimension. The summary action selected by the third dialogue action (DA) agent 204 is output to the evaluation agent 205, which determines whether the candidate action becomes final. The third dialogue action (DA) agent 204 is also configured to base its decisions on the generated candidate actions (in... Figure 2 In the example, candidate dialogue actions are output using the "ACCEPT_THANKING" action and information from the identified dialogue state 201.

[0164] Figure 2The multidimensional action selection component also includes an evaluation agent 205. The evaluation agent 205 is configured to select a combination of actions suggested by the dialogue action (DA) agents (i.e., the first dialogue action (DA) agent 202, the second dialogue action (DA) agent 203, and the third dialogue action (DA) agent 204). By providing a means of selecting and combining actions among candidate actions, Figure 2 The system can generate multifunctional system discourse (i.e., discourse that conveys more than one dialogical action), thereby improving both the naturalness and effectiveness of the system response.

[0165] exist Figure 2 In this example, information identifying each selected candidate action in the Dialogue Action (DA) agent is provided as input to the evaluation agent 205. Each DA agent is configured to generate at most one candidate action per round. The evaluation agent 205 is configured to generate a combination of these candidate actions as the system output.

[0166] Figure 4A An implementation of evaluation agent 205 based on an example is shown. Evaluation agent 205 includes a fourth reinforcement learning (RL) machine learning model, which includes a linear layer 402, a transformation layer 404, a masking layer 406, and a normalization layer 408. Evaluation agent 205 is communicatively coupled to the output of each dialogue action (DA) agent (i.e., the first dialogue action (DA) agent 202, the second dialogue action (DA) agent 203, and the third dialogue action (DA) agent 204).

[0167] The output of each Dialogue Action (DA) agent includes a vector, using one-hot encoding, indicating which action from the set of possible actions (as associated with the DA agent's dimension) has been selected as a candidate action. In this case, the magnitude of the output vector from a particular DA agent corresponds to the number of actions in the set of possible actions for the associated dimension. Evaluation agent 205 is configured to concatenate the output vectors from each of the DA agents to form input vector 401.

[0168] In the presence of three Dialogue Action (DA) agents Figure 1 In the example, input vector 401 comprises three concatenated one-hot code vectors. In this case, there exist 2 that can be selected as the output of the evaluation agent 205. 3 =8 possible dialogue combinations.

[0169] The fourth reinforcement learning (RL) machine learning model for evaluating agent 205 has a structure similar to that discussed above for machine learning models of dialogue action (DA) agents. Specifically, the fourth reinforcement learning (RL) machine learning model includes a linear layer 402 with trainable parameters, which transforms the input vector 401 into an action value vector 403 that estimates the long-term cumulative reward for each possible combination of dialogue action candidates for the input.

[0170] The output of linear layer 402 (i.e., including action value vector 403) is input to transformation layer 404. Transformation layer 404 is configured to convert action values ​​in action value vector 403, which can take any value between negative and positive infinity, into transformed action value vector 405, which includes values ​​greater than or equal to zero. The output of transformation layer 404 (i.e., transformed action value vector 405) is coupled to the input of masking layer 406 (i.e., masking layer 406 is configured to receive transformed action value vector 405). Transformation layer 404 can implement [something related to...]. Figure 3A The transformation discussed in transformation layer 304 is the same transformation.

[0171] Masking layer 406 is configured to disable candidate action combinations of several outputs by setting the values ​​of these combinations to zero to produce a masked, transformed action value vector 407. In one example, the masking layer is configured to disable candidate action combinations that cause conflict. For example, asking a question and saying goodbye would be meaningless. Therefore, some action combinations are determined to be meaningless, thus representing a conflict based on common speaking conventions. In one example, the action combinations disabled by the masking layer are predetermined and / or hard-coded.

[0172] In another example, the masking layer is configured to ensure that evaluation agent 205 only allows a single dialogue action, or a combination of the "IMPL-CONFIRM" (in the "Auto Response" dimension) and the "OFFER" or "REQUEST" actions (in the "Task" dimension). The following illustrates the specific combinations allowed by masking layer 406 for this particular example. Examples of utterances that can be generated based on action combinations are also provided.

[0173]

[0174] While the above illustrates specific combinations of actions from different dimensions, it is important to emphasize that other combinations of other actions are permissible. In a further example, the set of actions associated with the "Social Responsibility Management (SOM)" dimension includes the "APOLOGY" action, and permissible combinations include "APOLOGY" + "AUTO_NEGATIVE," which can elicit the statement: "Sorry, I don't quite understand."

[0175] The masking layer 406 of the evaluation agent 205 represents a constraint on the range of dialogue combinations the agent can choose, for example, in a clever way to match the input requirements of downstream components (such as the natural language generator 103). Furthermore, the masking layer also accelerates the training of the multidimensional action selection component, as this component does not need to learn conflicting combinations (e.g., meaningless combinations).

[0176] The output of masking layer 406 is a masked, transformed action value vector 407. The output of masking layer 406 is coupled to normalization layer 408. Normalization layer 408 is configured to normalize the action values ​​present in the masked, transformed action value vector 407 such that their sum equals 1, thus indicating the existence of an action that maximizes long-term cumulative reward. The output of normalization layer 408 is an output vector of action probabilities.

[0177] Figure 4B This shows a representation of the output vector based on the action probabilities of the example. (As for...) Figure 2 The three dimensions discussed for generating candidate dialogue actions include: “tasks”, “automatic feedback”, and “Social responsibility management (SOM)”. Figure 4B This shows the probability associated with the combination of candidate actions for each of the corresponding dimensions. For example, Figure 4B The combination of candidate actions for the “task” dimension (i.e., “OFFER”) and candidate actions associated with the “automatic feedback” dimension (i.e., “IMPL-CONFIRM”) is shown to be associated with the highest probability of maximizing long-term cumulative reward.

[0178] Figure 4B The entry associated with the blank state (i.e., "---") is also shown. This is the probability associated with no candidate action being selected. This is analogous to no response being produced. Finally, Figure 4B It also shows multiple action combinations associated with no (i.e., "0") probability (e.g., "autofeedback" + "SOM") due to the masking layer 406.

[0179] Such as about Figure 3A The masking layer 406 discussed is optional. In this case, the functionality of the transformation layer 404 and the normalization layer 408 can be combined in a softmax layer, which optionally implements the temperature hyperparameter. Alternatively, if a masking layer is present, the functionality of the transformation layer 404 and the masking layer 406 can be combined in a modified masking layer.

[0180] Figure 4C The implementation of the evaluation agent 205 is shown based on the second example. Figure 4C In, with Figure 4A The same number is used to represent the same component. Figure 4C Further, a modified masking layer 450 is included. The modified masking layer 450 is configured to receive an action value vector 403, which includes an estimated cumulative reward Q(s,a) for taking action a in state s. First, exp(Q(s,a) / τ) values ​​are calculated for all actions (where τ(tau) is the temperature hyperparameter and "exp" is an exponential function). Then, a mask (mask(s,a)) (i.e., mask(s,a)*exp(Q(s,a) / τ)) is applied, setting some of these values ​​to zero, thereby generating a second action value vector 452. Finally, the resulting values ​​(i.e., the second action value vector 452) are normalized to a probability distribution (where the masked actions have a probability of zero) by a normalization layer 408 as discussed above.

[0181] After generating an output vector of action probabilities, the evaluation agent 205 is configured to select a combination of one or more candidate actions by sampling from the resulting probability distribution (i.e., the probability distribution of the output vector).

[0182] consider Figure 2 For example, this led the evaluation agent 205 to select the combination of the action "OFFER" from the "Task" dimension and "IMPL-CONFIRM" from the "Automatic Feedback" dimension, instead of the action "ACCEPT_THANKING" from the "Social Responsibility Management (SOM)" dimension.

[0183] After selecting candidate action combinations, the evaluation agent 205 is configured to generate dialogue action combinations by retrieving dialogue actions generated by the dialogue action (DA) agent that are associated with the actions generated by the evaluation agent 205 to form the final combination. Figure 2 In the example, the candidate action combination is "OFFER" from the "Task" dimension and "IMPL-CONFIRM" from the "Automatic Feedback" dimension. In this case, the evaluation agent 205 is configured to generate a dialogue action combination by retrieving and combining the candidate dialogue action (associated with the "OFFER" action) output by the first dialogue action (DA) agent 202 with the candidate dialogue action (associated with the "IMPL-CONFIRM" action) output by the second dialogue action (DA) agent 203. The dialogue action combination is output by the evaluation agent 205.

[0184] The output of evaluation agent 205 is coupled to a natural language generator (e.g., natural language generator 103), which is configured to generate multi-functional system utterances based on a combination of one or more dialogues selected and output by evaluation agent 205.

[0185] For example, the multi-functional utterance generated by the Natural Language Generation (NLG) module could be: a response to the user input “I’m looking for cheap Italian restaurants” such as “Prezzo is a popular Italian restaurant”. In this case, the utterance generated by the system includes both the “OFFER” dialogical action (i.e., “Prezzo”) and the “IMPL-CONFIRM” dialogical action (“…is a popular Italian restaurant”, implicitly confirming to the user that the dialogical system has understood that the user wants Italian food).

[0186] As discussed above, like each of the other reinforcement learning (RL) machine learning models, the fourth reinforcement learning (RL) machine learning model includes a linear layer 402 with trainable model parameters. As will be discussed in more detail below, the model parameters of the reinforcement learning (RL) machine learning models for each of the dialogue action agent and evaluation agent 205 are optimized based on online interactions with the (simulated) user. Specifically, the model parameters of each of the reinforcement learning (RL) machine learning models are optimized based on a common reward signal obtained from the (simulated) user. During interactions with the (simulated) user, the four agents (i.e., the first dialogue action (DA) agent 202, the second dialogue action (DA) agent 203, the third dialogue action (DA) agent 204, and the evaluation agent 205) receive the same shared reward and update their parameters (i.e., the model parameters for the linear layer) based on the reward, with the goal of obtaining a policy that maximizes long-term cumulative rewards. Furthermore, like each of the other reinforcement learning (RL) machine learning models, the fourth reinforcement learning (RL) machine learning model includes a transition layer 404 associated with a temperature hyperparameter. In the example, the agents (i.e., the first dialogue action (DA) agent 202, the second dialogue action (DA) agent 203, the third dialogue action (DA) agent 204, and the evaluation agent 205) use the same value of the temperature hyperparameter in their respective transformation layers.

[0187] Figure 5 A method for generating system utterances is illustrated based on an example. The method begins in step 501 with receiving user utterances. In one example, the utterances originate from a human user and are captured via a microphone. In this case, automatic speech recognition is performed on the user input to identify the user's speech and convert it into text. In one example, The Web Speech API is used for automatic speech recognition.

[0188] In step 502, the dialogue state is updated based on the user utterance received in step 501. In one example, the dialogue state is updated by the dialogue state tracking component 101.

[0189] In step 503, a first candidate action in the first dimension is generated. In this example, the first candidate action is generated by the first dialogue action (DA) agent 202 as discussed above.

[0190] In step 504, a second candidate action in the second dimension is generated. In this example, the second candidate action is generated by the second dialogue action (DA) agent 203 as discussed above.

[0191] In step 505, a third candidate action in the third dimension is generated. In this example, the third candidate action is generated by the third dialogue action (DA) agent 204 as discussed above.

[0192] Figure 5 The method demonstrates that the first, second, and third candidate actions are generated in parallel. This indicates that the generation of each candidate action is independent. However, to avoid ambiguity, it is emphasized that steps 503, 504, and 505 can be performed serially or in parallel.

[0193] In step 506, one or more final dialogue actions are generated by selecting a combination of a first candidate action, a second candidate action, and a third candidate action. To avoid any ambiguity, one combination of these actions does not include candidate actions. In this example, the final dialogue action is selected by the evaluation agent 205 as discussed above. Specifically, a combination of candidate actions is selected, and one or more final dialogue actions are generated based on the dialogue actions associated with the selected actions.

[0194] As discussed above, the first dialogue action (DA) agent 202, the second dialogue action (DA) agent 203, the third dialogue action (DA) agent 204, and the evaluation agent 205 each include a machine learning model with trainable parameters. Figure 5 The method shown involves the inference phase. Therefore, the parameters of the machine learning model are fixed (i.e., they do not change during use). In the example, Figure 5 The inference phase shown occurs after various machine learning models have been trained (discussed in more detail below). In another example, trainable parameters are retrieved from storage, where the trainable parameters were stored after the training phase (at different times).

[0195] In step 507, a system response is generated based on the final dialogue action selected in step 506. In this example, a natural language generator 103 is used to convert the final dialogue action into a natural language system response.

[0196] In step 508, a natural language response is output. In one example, the output of the natural language generator 103 is in text form, and optionally, it is achieved by using... The Web Speech API converts text into speech. The system response is then output via a speaker.

[0197] In the above example, the first dialogue action (DA) agent 202, the second dialogue action (DA) agent 203, the third dialogue action (DA) agent 204, and the evaluation agent (EA) 205 are pre-trained. That is, the trainable parameters of the machine learning model are obtained during the training phase (optionally at different times) and used by the machine learning model for inference. The trainable parameters for training the multidimensional action selection component will now be discussed in detail.

[0198] The first dialogue action (DA) agent 202, the second dialogue action (DA) agent 203, the third dialogue action (DA) agent 204, and the evaluation agent (EA) 205 are each trained using reinforcement learning.

[0199] In short, reinforcement learning (RL) is a framework that enables agents to make decisions in an order of optimization processes. The problem is formulated as a Markov decision process (MDP) with a state S and a set of agent actions A. The basic reinforcement learning (RL) agent interacts with its environment in discrete time steps. At each time t, the agent receives the current state s. t and reward r t The strategy then selects action a from the set of available actions. t Action a t It is then sent to the environment. The environment moves to the new state s. t+1 , and with transformation (s t ,a t ,s t+1 The associated reward r t+1 It has been determined. The goal of the reinforcement learning (RL) agent is to learn the policy: π: A×S→[0,1], π(a,s)=Pr(a t =a|s t =s), this strategy maximizes the expected cumulative reward. A more detailed discussion of reinforcement learning is provided by reference to "Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018", which is incorporated in this paper.

[0200] As discussed above, both the Dialogue Action (DA) agent and the evaluation agent include a linear layer. In the example, the linear layer is a single-layer (artificial) neuron and is configured to produce output based on a linear combination of inputs. The linear layer is configured to produce an indication of the expected return (i.e., cumulative reward) for each action, starting from state S (i.e., the input to the machine learning model), following the current policy π, and taking action a.

[0201] For a Dialogue Action (DA) agent, state S corresponds to the dialogue state of the system (i.e., the dialogue state maintained by the dialogue state tracking component 101), and actions are candidate actions from a set of actions for a specific dimension (i.e., the dimension for which the machine learning model is configured to provide candidate actions). In a further example, state S (i.e., the input to the machine learning model) is a set of features extracted from the complete dialogue state maintained by the dialogue state tracking component 101.

[0202] For the evaluation agent, state S corresponds to the candidate action selected by the Dialogue Action (DA) agent, and the action is a combination of candidate actions.

[0203] The linear layer is configured to produce a value for the action-value function. In other words, the linear layer is configured to produce the expected return (i.e., cumulative reward) associated with each action in the action set based on the input state.

[0204] Therefore, the output of the linear layer includes a value for each action in the action set, indicating the expected return (i.e., cumulative reward) for taking that action. For example, for the linear layer associated with the “task” dimension in the first dialogue action (DA) agent 202, the output of the linear layer includes the value of the action value function (i.e., the expected return / cumulative reward associated with taking the action) for each of the “OFFER”, “ANSWER”, “REQUEST”, and “NONE” actions.

[0205] As discussed above, both the Dialogue Action (DA) agent and the Evaluation Agent (EA) further include a transition layer (or softmax layer) controlled by a temperature hyperparameter. In the example, candidate actions are generated by sampling a probability distribution resulting from at least normalizing the action values ​​of the transition (optionally after masking the action values), where the probability distribution indicates the likelihood that each action in the action set maximizes the long-term cumulative reward.

[0206] In reinforcement learning, there is a trade-off between exploitation and exploration (i.e., whether the machine learning model continues to do what it has already done, or whether it tries new actions in a state, thus exploring the state-action space). In one example, the machine learning models for conversational action (DA) agents and evaluation agents use Boltzmann exploration.

[0207] In Boltzmann exploration, exploration actions are sampled from a softmax distribution applied to the estimated action-value function. As discussed above, the transformation layer is controlled by a temperature hyperparameter. A high temperature is used during the early training phase and causes actions (from the set of possible actions associated with the dimension) to be selected more randomly. A lower temperature is used during the later training phase and causes the estimated values ​​to be followed more closely (i.e., high-value actions are selected with higher probability). In this example, the temperature hyperparameter decays linearly, thus gradually reducing the exploration level until the policy (of the reinforcement learning machine learning model) selects only actions with the highest expected return (e.g., associated with the highest cumulative reward of the estimated value).

[0208] Figure 6 This demonstrates a method for selecting components based on examples of training multidimensional actions.

[0209] In the example, the trainable weights of the machine learning model (specifically, the weights of the linear layers used for each dialogue action (DA) agent and the evaluation agent) are initialized, optionally with random values.

[0210] In step 610, a dialogue is initiated. In this example, a dialogue is initiated between the system and a simulated user. The simulated user is configured to generate utterances and / or dialogue actions based on a goal.

[0211] In the example, initiating the dialogue in step 601 includes: randomly initializing a user goal from the domain ontology and providing that user goal to the simulated user. For example, in the restaurant search domain, the user goal could be to find cheap Italian restaurants and obtain their phone numbers. Generally, user goals include constraints (e.g., interval-value pairs) and requests (e.g., obtaining an interval of values).

[0212] In the example, an agenda-based user simulator is used, as described by reference in Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics: Companion Volume, Short Papers, Rochester, New York, April 2007, pp. 149-152, Association for Computational Linguistics, Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, Steve Young, “Agenda-Based User Simulation for Bootstrapping a POMDP Dialogue System”.

[0213] In step 602, a simulated user generates utterances, and the generated utterances are provided to the dialogue system being trained (i.e., as about...). Figure 1 The dialogue system under discussion includes a dialogue state tracking component 101, an action selection component 102 (implemented using a multidimensional action selection component that is being trained), and a natural language generator 103.

[0214] In step 603, in response to receiving a utterance from the simulated user, the dialogue state is updated (e.g., by the dialogue state tracking component 101).

[0215] In the example, the dialogue action is generated by a simulated user, and the simulated dialogue action is passed to a retrieval-based natural language generator to produce user utterances, which are then passed to an Action State Update (ASU) model to update the dialogue state.

[0216] In step 604, a system response is generated. As discussed above, each Dialogue Action (DA) agent obtains information identifying the dialogue state (e.g., a set of features extracted from the complete dialogue state) and estimates the expected return (i.e., cumulative reward) associated with each action in the action set taken when starting from the current state. An action value function is used to estimate the expected return. The action value function is implemented using linear layers (e.g., artificial neural networks), where the output of the action value function (i.e., the cumulative reward) is produced based on a linear combination of the input feature set. This linear combination is based on trainable weights, which are initialized weights for the first training iteration. Optionally, the estimated cumulative reward associated with the candidate actions is stored for future use.

[0217] Each dialogue action agent generates candidate actions, and the agent evaluates the combinations of candidate actions generated. After generating combinations of candidate actions, the agent evaluates the dialogue action combinations by combining the dialogue actions associated with each action into the candidate action combinations as discussed above.

[0218] In the first example, generating a system response in step 604 includes generating utterances. In this case, a combination of dialogue actions associated with a combination of candidate actions is then provided to a natural language generator (e.g., natural language generator 103) that generates natural language utterances. In this example, generating a system response includes performing actions such as... Figure 5 Steps 503 to 508 are discussed.

[0219] In step 605, the system response generated in step 604 is provided to the simulated user.

[0220] In step 606, a reward signal is calculated, and the reward generated when starting from state s and taking action a (e.g., selecting a candidate action from the action set) is optionally stored in memory. This reward value will then be used to determine the cumulative reward for the dialogue.

[0221] The reward signal is calculated after each round of the system (i.e., after each time the system responds to a simulated user producing a utterance). As is common in reinforcement learning, the resulting reward is used to train the machine learning model (i.e., to encourage / deter a certain behavior of the machine learning model).

[0222] In the example, the reward signal consists of two components: a reward generated by the simulated user and an internally generated reward.

[0223] In the example, upon completion of the task, the simulated user awards a reward of +100. In the context of restaurant search, this occurs when the dialogue system has already recommended restaurants that match the user's preferences (as specified in the user objectives provided to the simulated user in step 601) and has provided all the requested information about those restaurants.

[0224] Additionally, the user can be penalized when the system fails to respond to a gesture of gratitude, or a social action can be inserted when it is not needed (-5 for each occurrence). This forces the system to learn basic reactive social behaviors. In a further example, more complex social patterns can be encouraged through the design of reward signals.

[0225] The second component of the reward signal is an internally generated reward. Internally, a penalty of -1 is applied for each round (to encourage shorter dialogues), and a penalty of -25 is applied when a "handling problem" is encountered and the system does not signal the user with a feedback action to that "handling problem." Handling problems are recorded in the dialogue state when speech recognition or natural language understanding fails (i.e., no result is returned). During training, such handling problems are randomly simulated in 5% of the user rounds, where the original simulated user actions are discarded.

[0226] The reward signal for each round (e.g., when choosing action a in state s) is calculated by summing the user-generated rewards and internal rewards.

[0227] In step 607, it is determined whether the dialogue has been completed. In reinforcement learning terminology, a dialogue represents a segment (i.e., a sequence of states, actions, and rewards that ends in a terminating state). In one example, the dialogue is completed when the simulated user says "goodbye" and the system responds with the "returnGoodbye" dialogue action. In this example, the simulated user is configured to say "goodbye" upon completion of the objective. Alternatively, "goodbye" may be said after an initial thank you to the system.

[0228] In another example, the conversation ends when the user "hangs up." When using a phone-based application to train a multidimensional action selection component, the user "hangs up" by terminating the phone call (i.e., disconnecting the communication link from the user to the system). Alternatively, when training a multidimensional action selection component in a software system, the user does so by clicking a button on the user interface or typing a command (e.g., "..."). <hangup>Enter the text ") into the interface to "hang up".

[0229] If the dialogue has been completed, the method proceeds to step 608. In step 608, the learnable weights of the multidimensional action selection component are trained.

[0230] The weights of the linear layer (implementing the action-value function) are updated after each dialogue / segment based on the reward signal received throughout the dialogue. The policies (of each machine learning model) coordinate their actions only indirectly through the shared reward; that is, each policy operates independently without any direct communication with other policies. In other words, the dialogue action (DA) agent and the evaluation agent (EA) are each trained separately based on a common reward signal.

[0231] In one example, Monte Carlo control reinforcement learning is used to simultaneously train the dialogue action (DA) agent and the evaluation agent (EA) using linear value function approximation. Monte Carlo control reinforcement learning is described by reference in "Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, the MIT Press, second edition, 2018", which is incorporated in this paper.

[0232] In short, Monte Carlo controlled reinforcement learning uses the Monte Carlo method (i.e., repeated random sampling to obtain numerical estimates). In Monte Carlo controlled reinforcement learning, for each state-action pair (s, a) appearing in a segment, a return (i.e., cumulative reward) is computed. The cumulative reward for that state-action is appended to a list of cumulative rewards for the same state-action pair (i.e., if the segment produces the same state-action pair twice, there will be a list of two values ​​including the cumulative reward). The cumulative rewards are then averaged to obtain the "true" cumulative reward associated with choosing action a in state s.

[0233] As discussed above, in step 606, only the instantaneous value of the reward signal is calculated and stored. As a result, during step 608, a cumulative reward is generated for each state-action pair when the dialogue (in natural language processing terms) or, in this example, equivalently, the segment (in reinforcement learning terms) has terminated.

[0234] To do this, the sum of rewards received in the sequence between the corresponding state-action pair and the terminal state is calculated. Optionally, a discounted reward formula is used, in which a discount factor is applied to reduce the impact of future rewards on the value of the cumulative reward (e.g., to prevent infinitely large cumulative rewards).

[0235] The trainable parameters of the Dialogue Action (DA) agent and the Evaluation Agent (EA) are then updated using cumulative rewards for state-action pairs based on reward signals.

[0236] For ease of understanding, we define two terms. "Real" cumulative reward is the cumulative reward obtained during a conversation / segment with a simulated user. "Estimated" cumulative reward is the cumulative reward estimated through an action-value function implemented by a linear layer.

[0237] As discussed above, the machine learning model (specifically, the linear layer) is configured to produce an estimate of the cumulative reward for taking a given action from a given state. In the example, a linear function (specified by trainable weights) is used to produce an action value for each action given an input state.

[0238] After calculating the "true" cumulative return, a loss value can be obtained (i.e., the difference between the "true" cumulative reward and the "estimated" cumulative reward). The loss is then used to update the trainable parameters of the machine learning model (specifically, the weights of the linear layers).

[0239] In this example, training the machine learning model for each conversational action (DA) agent and evaluation agent (EA) involves minimizing the mean squared error between the "real" cumulative reward and the "estimated" cumulative reward. In this example, the mean squared error is calculated in step 608, and stochastic gradient descent is used to update the trainable weights of the machine learning model (specifically, the linear layers).

[0240] consider Figure 2 The example shown, and assume Figure 2 This represents the dialogue during the training phase. In this example, the action generated by the first dialogue action (DA) agent 202 (associated with the "Task" dimension) is "OFFER". The action generated by the second dialogue action (DA) agent 203 (associated with the "Automatic Feedback" dimension) is "IMPL-CONFIRM". The action generated by the third dialogue action (DA) agent 204 (associated with the "Social Responsibility Management" dimension) is "ACCEPT_THANKING". The action combination generated by the evaluation agent 205 is "OFFER" + "IMPL-CONFIRM".

[0241] During training, “real” cumulative rewards are generated based on the following rewards: 1) instantaneous rewards (generated by the simulated user in response to “OFFER” + “IMPL-CONFIRM”), and 2) subsequent instantaneous rewards received for other utterances in the conversation until the conversation ends.

[0242] The Dialogue Action (DA) agent and the Evaluation Agent (EA) are each trained separately based on a common reward signal (i.e., each agent is trained using the same "real" cumulative reward).

[0243] Consider the first dialogue action (DA) agent 202. The machine learning model (specifically, a linear layer) of the first dialogue action (DA) agent 202 estimates the cumulative reward for the "OFFER" action given an input state (e.g., features extracted from the full dialogue state). The weights of the machine learning model (representing the "estimated" action value function, since it produces the "estimated" cumulative reward) are then trained based on the difference between the previously generated "estimated" cumulative reward computed for the "OFFER" action at a given input state and the "true" cumulative reward computed above.

[0244] Consider the third dialogue action (DA) agent 204. The machine learning model (specifically, a linear layer) of the third dialogue action (DA) agent 204 estimates the cumulative reward for the "ACCEPT_THANKING" action based on the input state (e.g., features extracted from the complete dialogue state graph) according to the trainable weights of a linear function.

[0245] From the perspective of the third dialogue action (DA) agent 204, the DA agent "believes" that the "true" cumulative reward calculated above was generated for the "ACCEPT_THANKING" action (i.e., it is unaware that the evaluation agent 205 did not select the "ACCEPT_THANKING" action). As a result, the weights of the machine learning model in the third dialogue action (DA) agent are then trained based on the difference between the "true" cumulative reward and the "estimated" cumulative reward estimated for the "ACCEPT_THANKING" action (representing the "approximation" action value function for the "Social Responsibility Management (SOM)" dimension).

[0246] Consider evaluation agent (EA) 205. Evaluation agent (EA) 205 has a state different from any of the dialogue action (DA) agents (each of which may have different sets of features as input, but all are derived from the common complete dialogue state representation as discussed above). Specifically, the state of evaluation agent (EA) 205 is based on candidate actions output by various dialogue action (DA) agents (e.g., first dialogue action (DA) agent 202, second dialogue action (DA) agent 203, and third dialogue action (DA) agent 204).

[0247] Therefore, the machine learning model of the evaluation agent (EA) is configured to generate an "estimated" cumulative reward for the state {'OFFER'·'IMPL-CONFIRM'·'ACCEPT_THANKING'} and the action {'OFFER'+'IMPL-CONFIRM'}. Note that the action combination (i.e., 'OFFER'+'IMPL-CONFIRM') is then used to generate a combination of dialogue actions as discussed above, which is then used to generate the reward. The difference between the "estimated" cumulative reward and the "real" cumulative reward generated above is used to train learnable weights that approximate a linear function of the action-value function in the evaluation agent (EA) 205.

[0248] In summary, in this example, the "real" cumulative reward generated above is associated with the following (state, action) pairs: (Identifying the information of dialogue state_1, "OFFER") – for the first dialogue action (DA) agent 202, (Identifying the information of dialogue state_2, "IMPL-CONFIRM") – for the second dialogue action (DA) agent 203, (Identifying the information of dialogue state_3, "ACCEPT_THANKING") – for the third dialogue action (DA) agent 204, and ("'OFFER"·'IMPL-CONFIRM'·'ACCEPT_THANKING', 'OFFER+IMPL-CONFIRM') – for the evaluation agent 205.

[0249] As discussed above, the information for identifying the dialogue state in one example can be a different set of features extracted from a common (i.e., the same) complete dialogue state representation, as indicated by labels 1, 2 and 3.

[0250] Based on the difference between the (common) "real" cumulative reward and the "estimated" cumulative reward generated by each of the Dialogue Action (DA) agent and Evaluation Agent (EA), weight updates are generated for each weight in the machine learning model of each of the DA and EA agents. As discussed above, the weights of each linear layer are updated to minimize the mean squared error between the "real" and "estimated" cumulative rewards. Optionally, stochastic gradient descent is used to compute the weight updates for each linear layer for each weight.

[0251] In step 608, the weights of each linear layer in each of the Dialogue Action (DA) agent and Evaluation Agent (EA) are updated according to the calculated weight update. The method then proceeds to step 609.

[0252] In step 609, the temperature hyperparameter is controlled. As discussed above, in this example, the temperature hyperparameter of each transition layer (e.g., the transition layer of the first dialogue action (DA) agent 202, the transition layer of the second dialogue action (DA) agent 203, the transition layer of the third dialogue action (DA) agent 204, and the transition layer of the evaluation agent (EA) 205) decays linearly, thereby gradually reducing the level of inquiry as the number of dialogues (or equivalently, fragments in reinforcement learning terms) increases.

[0253] In this example, the weights are updated (in step 608) based on the learning rate hyperparameter, which controls the amount of change in weights caused by a given, determined error. In this case, step 609 also includes gradually decreasing the value of the learning rate hyperparameter as the number of dialogues increases.

[0254] In the example, both hyperparameters (i.e., the temperature hyperparameter and the learning rate hyperparameter) decay to a minimum after each training dialogue.

[0255] After adjusting the hyperparameters, the method continues to step 610.

[0256] In step 610, it is determined whether the number of training dialogues equals the maximum number of training dialogues. Essentially, step 610 includes determining whether the training phase has been completed. In this example, the maximum number of training dialogues is predetermined (i.e., at the beginning...). Figure 6 (Settings between methods).

[0257] If it is determined in step 610 that the number of dialogues equals the maximum number of training dialogues, then training is complete as shown in step 611. Optionally, upon completion of training, the trainable weights of each of the Dialogue Action (DA) agent and Evaluation Agent (EA) are stored for future use. As will be further explained below, the trainable weights of each Dialogue Action (DA) agent can be stored along with an indication of which dimension they were generated for, thereby enabling the creation of a modular system in which pre-generated weights for various dimensions can be reused.

[0258] If it is determined in step 610 that the number of dialogues is not equal to the maximum number of dialogues, the method proceeds to step 601, in which a new dialogue is started with the simulated user. In this example, step 601 further includes incrementing the value of the number of training dialogues.

[0259] In the training method discussed above, steps 601 (starting the dialogue), 602 (generating user utterances), and 603 (updating the dialogue state) involve passing user utterances / text to the dialogue system and then updating the model using action states. Similarly, in the example discussed above, step 604 includes generating a system response in the form of utterances.

[0260] In another example, a multidimensional action selection component is trained by using a rule-based state update model to pass (simulated) user dialogue actions to the dialogue system.

[0261] In this scenario, in steps 601 and 602, the (simulated) user generates a user dialogue action, and in step 603, the dialogue state is updated based on the user dialogue action according to a rule-based state update model. In this example, the simulated user uses a semantic error model to generate a dialogue action that becomes a hypothesis with a confidence score. The dialogue state is then updated with the first hypothesis based on manually defined rules.

[0262] In this example, the simulated user takes dialogue actions as input. Therefore, generating a system response in step 604 involves providing the simulated user with a combination of dialogue actions associated with the candidate action combinations (i.e., generating utterances without based on the dialogue action combinations). In this case, the functionality of the natural language generator is not required as part of the training method.

[0263] Using dialogue actions directly (i.e., without generating and / or processing utterances) can accelerate the training of multidimensional action selection components and use fewer resources because no action state update model or natural language generator component needs to be trained.

[0264] Although the training methods discussed above use simulated users, it is important to emphasize that, to avoid any doubt, the multidimensional action selection component could instead be trained based on dialogue with physical users (i.e., actual humans) or information representing interactions between human users (e.g., a dataset). However, using simulated users advantageously accelerates the training process.

[0265] Furthermore, while the training methods discussed above use specific approaches for reinforcement learning (i.e., Monte Carlo control with linear value function approximation), it is emphasized that, to avoid any doubt, other reinforcement learning methods can be used for training reinforcement learning models. Some of these methods are described by reference to "Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018," which is incorporated in this paper. For example, the State-Action-Reward-State-Action (SARSA) algorithm can be used to train machine learning models. Specifically, other reinforcement learning (RL) techniques can be used to train multidimensional action selection components, such as Gaussian process State-Action-Reward-State-Action (GP-SARSA) and / or Deep Q-Learning (DQN).

[0266] Finally, in the training methods discussed above, a specific reward function is used (i.e., to generate a reward signal). Other reward functions can also be used.

[0267] Figure 7 This paper presents a comparison of the dialogue success rates of a multidimensional system using a manually coded evaluation agent and an evaluation agent based on the examples described herein.

[0268] Tests are performed to estimate the impact that trainable evaluation agents (such as those described herein) can have on the overall dialogue policy optimization process (of the entire system). Testing involves training a set of 10 dialogue policies (i.e., completing 10 training runs). Each training run involves training 3 policies (i.e., the policies of the first dialogue action (DA) agent 202, the second dialogue action (DA) agent 203, and the third dialogue action (DA) agent 204) in a system using evaluation agents according to the examples described herein, and 4 policies (i.e., the policies of the first dialogue action (DA) agent 202, the second dialogue action (DA) agent 203, the third dialogue action (DA) agent 204, and the evaluation agent 205) in a system using evaluation agents according to the examples described herein. Figure 7 The learning curves for policy optimization over 40,000 simulated training dialogues for both systems are shown. Each curve represents the sliding window success rate, averaged over 10 training runs, over the last 100 dialogues during training.

[0269] As from Figure 7 It will be noted that both systems converge to comparable performance levels of approximately 95% average success rate. However, between 5,000 and 15,000 dialogues, the version using the trainable evaluation agent (i.e., the embodiment described herein) performs significantly better.

[0270] The table below shows the results of evaluating a fully trained policy on 3,000 test dialogues (e.g., after 40,000 training dialogues). The table indicates that a system with a trained evaluation agent (e.g., the system in the embodiments described herein) achieves a better success rate than a system with a manually coded evaluation agent.

[0271] Evaluation agent for training 96.8％ 9.52 85.56 Manually coded evaluation agent 95.4％ 9.42 84.98

[0272] In human user evaluations based on population data, systems with trained evaluation agents (i.e., systems like those described in the examples herein) are tested as part of a spoken dialogue system. Results for a fully trained system (including the trained evaluation agent) are shown below, where subjective partial and full completion rates (%), as well as average ratings (1–6) for perceived comprehension, speech recognition, appropriateness of system responses, and naturalness are obtained from questionnaires completed by the subjects after each dialogue.

[0273] average length 5.88 Partially completed 78.89％ Completed 79.90％ understand 3.76 Identification 3.86 System Response 3.93 natural 3.71

[0274] As will become apparent from the above, the examples described herein offer better performance than systems with manually coded evaluation agents. Furthermore, the action selection components implemented according to the examples described herein have the advantage of being easier to train as dialogue systems (e.g., due to the inclusion of more dialogue action (DA) agents to support additional dimensions, or by expanding the action set of a single dialogue action (DA) agent in a multidimensional system) become more complex. Unlike the examples described herein, scaling manually coded / rule-based evaluation agents becomes increasingly verbose and requires increasingly deeper understanding of dialogue action taxonomy. Using an evaluation agent (EA) that can be automatically trained from online interactions and is associated with the dialogue action (DA) agents in the system provides a more scalable and reliable approach for building multidimensional systems. A general approach for automatically learning the composition process (as described herein) is therefore preferred in long-term operation.

[0275] Furthermore, manually coded evaluation agents typically select only a single dialogue action candidate from a single dimension / dialogue action (DA) agent (i.e., some manually coded evaluation agents are not configured to select combinations of actions from different dimensions). Therefore, these manually coded evaluation agents do not generate multi-functional utterances (i.e., utterances conveying more than one dialogue action) in the regular manner provided by the ISO 24617-2 annotation standard, thus limiting the system response in terms of both naturalness and effectiveness. In contrast, the embodiments described herein are based on strategies learned using reinforcement learning (RL) to select combinations of one or more dialogue actions. Furthermore, the masking mechanisms described herein are used to control the range of possible combinations, particularly to adapt to downstream components.

[0276] Advantageously, the evaluation agent (EA) described herein opens up opportunities to learn directly from dialogue interactions about policy and style aspects of selecting combinations of dialogue actions. More generally, the embodiments described herein help advance the agenda for developing interoperable dialogue system components compliant with ISO 24617-2, which can be used in the production pipeline of dialogue systems for a wide range of applications.

[0277] In summary, multi-dimensional policy models facilitate training multiple policies / agents to perform action selection in dialogue, unlike models using a single policy. Such models facilitate the output of dialogue action combinations, where multiple agents are trained to handle various aspects of the dialogue. In the example above, a trainable evaluation agent (EA) is provided, trained together with multiple dialogue action (DA) agents to determine the next action based on a common reward signal. The selected action can be an action determined by a single DA agent or a combination of actions from multiple DA agents in a multi-dimensional system. In the example described in this paper, the evaluation agent (EA) is trained, unlike manually created / manually encoded ones. The experimental results provided above demonstrate that, compared to manually created evaluation agents, trainable evaluation agents (EAs) reduce complexity and help improve dialogue success.

[0278] Another problem with previous dialogue systems was their scalability to new domains and applications. Previous dialogue systems generally scalable poorly because the domains expanded with new intervals (i.e., new information was tracked by the dialogue state) and because new applications emerged requiring new, task-specific system actions. This was partly due to the lack of availability of labeled datasets for the domain.

[0279] One approach to this problem is to use transfer learning techniques to efficiently develop dialogue systems for new domains using limited data from the target domain, or even without data from the target domain.

[0280] Advantageously, the multidimensional action selection component presented in this paper is particularly well-suited for transfer learning. Specifically, the multidimensional action selection component based on the examples presented in this paper opens up opportunities for dialogue managers to efficiently adapt to new tasks and domains.

[0281] For example, the automatic feedback dimension and the social responsibility management (SOM) dimension are considered domain- and task-independent, so dialogue strategies for these dimensions can be passed on (from previously trained models), while strategies for the "task" dimension are trained from scratch in the target domain.

[0282] Figure 8A This demonstrates a transitive learning approach that uses a multidimensional action selection component based on an example.

[0283] We will discuss new use cases for adapting the multi-dimensional action selection component to the domain (e.g., restaurant search). Figure 8A The method. Specifically, it will be discussed by extending the set of actions for "task" agents to include examples of interval-specific actions. Figure 8A The method is described below. However, it is emphasized that the transfer learning method can be used to adapt multidimensional action selection components to new domains (e.g., from hotel search to restaurant search). Furthermore, the transfer learning method discussed below can be used when adding new actions to extend the behavior of the system (e.g., extending a restaurant search system to a restaurant reservation system).

[0284] In the example, the action set of the "Task" agent is expanded by replacing the summary action "REQUEST" used to request a range in the dialog state with a separate request action for each range. In the example where the dialog state includes ranges for food type (TYPE), price range (PRICERANGE), and area (AREA) within them, the action set is expanded to include "REQUEST-AREA", "REQUEST-FOOD", and "REQUEST-PRICERANGE".

[0285] For ease of understanding, we define the following terms. A source scenario is the scenario in which the trainable weights of the multidimensional action selection component were initially generated (e.g., a use case within the domain, or the domain itself). A target scenario is a new scenario in which the multidimensional action selection component is being used (e.g., a new domain or a new use case).

[0286] In the source scenario, the strategy can select the REQUEST summary action, after which heuristics determine which area is requested. In the target scenario, based on the example, the dialogue strategy can select the request action for a specific area, such as selecting the "REQUEST-AREA" action to request more information about the area. Therefore, in a target scenario with an expanded set of actions, the system automatically learns which area to request, rather than relying on heuristics from the source scenario.

[0287] As discussed above, in the restaurant search domain, this means that the "REQUEST" summary action is replaced with three range-specific request actions corresponding to the range of food, region, and price range (i.e., "REQUEST-AREA", "REQUEST-FOOD", and "REQUEST-PRICERANGE").

[0288] Returning to the original set of actions outlined in the table discussed above, the action set for the "task" agent thus grows from 4 actions (OFFER, REQUEST, ANSWER, NONE) to 6 actions (OFFER, REQUEST-FOOD, REQUEST-AREA, REQUEST-PRICERANGE, ANSWER, NONE), and the set of input features used to evaluate the agent (EA) is also expanded to cover the new set of actions for the "task" agent.

[0289] Therefore, both the "task" agent and the evaluation agent (EA) must be retrained for the target scenario, especially considering the new actions available in the "task" dimension.

[0290] In step 801, a multidimensional action selection component is generated. This step may include initiating the multidimensional action selection component in the software.

[0291] In step 802, trainable weights for the Dialogue Action (DA) agents are obtained. In this example, the DA agents are associated with dimensions that are not specific to the source scenario (e.g., the dimensions are domain-independent). As discussed above, domain-independent dimensions could include an "automatic feedback" dimension and a "Social Responsibility Management (SOM)" dimension. Optionally, when the machine learning model has been trained for the source scenario, the trainable weights for these agents are retrieved from storage.

[0292] In another example, obtaining pre-trained weights for the dialogue action (DA) agent includes training a multi-dimensional system for the source scenario (e.g., based on...). Figure 6 In this context, the "Task" Dialogue Action (DA) agent uses a summary action (i.e., "REQUEST") to request a range. This results in four trained policies corresponding to the three conversational dimensions plus the evaluation agent. In addition, domain-independent dialogue policies for "Auto-Feedback" and "SOM" are subsequently reused in the target scenario.

[0293] In step 803, a multi-dimensional action selection component is trained for the target scenario. In this case, the task agent uses an expanded set of actions (including "REQUEST-AREA", "REQUEST-FOOD", and "REQUEST-PRICERANGE"). The dialogue strategy for the domain / use case independent dimension is reused, and the multi-dimensional action selection component is retrained (e.g., based on...). Figure 6 This allows the system to use an expanded set of actions set in the "task" dimension.

[0294] It has been observed that by using such a repetitive strategy, the multidimensional action selection component can improve training speed (i.e., achieve a higher session success rate with fewer training dialogues).

[0295] Figure 8B The following illustrates a transferable learning method using a multidimensional action selection component, based on the second example. Step 850 includes training the multidimensional action selection component for the source scene. Step 851 includes storing the trained policy that can be transferred to the target scene. Step 852 includes training the multidimensional action selection component for the target scene, reusing and modifying the pre-trained transferable policy.

[0296] Figure 9 A schematic diagram of hardware that can be used to implement the method according to an embodiment is shown. The hardware includes a computing system 900. In this particular example, the components of the system will be described together. However, it will be appreciated that the components are not necessarily co-located.

[0297] The components of the computing system 900 include, but are not limited to, a processing unit 913 (such as a central processing unit, CPU, otherwise referred to as a "processor"), system memory 901, and a system bus 911 that couples various system components, including the system memory 901, to the processing unit 913. The system bus 911 can be any of several types of bus architectures, including a memory bus or memory controller, a peripheral bus and a local bus using any of various bus architectures, etc. The computing system 900 may also include a memory 915 connected to the system bus 911. The memory 915 may be external memory.

[0298] System memory 901 includes a computer storage medium in the form of volatile and / or non-volatile memory (such as read-only memory (ROM)). System memory 901 stores computer-readable instructions that, when executed by a processing unit 913 (e.g., a processor), cause the processing unit 913 (also referred to as the "processor") to perform the functionality described herein, particularly at least one of the following: as per [the relevant information]. Figure 1 The systems discussed, such as those concerning Figure 2 The described multi-dimensional action selection component, Figure 5 Method (reasoning) Figure 6 Methods (training) and / or Figure 8A or Figure 8B The method (transfer learning).

[0299] System memory 901 may also include operating system 905, application program 907, and program data 909 used by processing unit 913. In an embodiment, system memory 901 also includes basic input / output system (BIOS) 903, which contains routines that help transfer information between components within the computer, such as those typically stored in system memory 901 during startup.

[0300] The computing system further includes an interface 925 communicatively connected to a system bus 911. In embodiments, interface 925 includes functionality specifically for receiving and sending information from and to other devices via wired and / or wireless connections. Alternatively, interface 925 includes audio input (e.g., a microphone) for receiving user input (e.g., speech) and audio output (e.g., a speaker) for providing system responses to the user. Alternatively, interface 925 includes means for receiving text input data from a device configured to generate text data (e.g., a keyboard). In another embodiment, interface 925 may be various forms of user interface that allow a user to interact with the system, for example, by generating commands and / or questions and receiving responses.

[0301] exist Figure 9 In this example, a video interface 917 is provided. The video interface 917 includes a graphics processing unit (CPU) 919 communicatively connected to a graphics processing memory 921.

[0302] The graphics processing unit (GPU) 919 is particularly well-suited for training machine learning models, such as neural network training, due to its adaptability to data parallel operations. Therefore, in this embodiment, the processing for training the machine learning model can be divided between the CPU 913 and the GPU 919.

[0303] It should be noted that in some embodiments, different hardware can be used to train machine learning models and perform state updates. For example, training of the machine learning model can occur on one or more local desktop or workstation computers, or on a device in a cloud computing system that may include one or more discrete desktop or workstation GPUs, one or more discrete desktop or workstation CPUs (e.g., processors with PC-oriented architectures), and a large amount of volatile system memory, such as 16 GB or more. Alternatively, for example, the execution of a dialogue can use mobile or embedded hardware that may be part of a system-on-a-chip (SoC) or does not include a GPU; one or more mobile or embedded CPUs, such as processors with mobile-oriented or microcontroller-oriented architectures, and less volatile memory (e.g., less than 1 GB). For example, the hardware performing the dialogue could be a voice assistance system 120, such as a smart speaker, or a mobile phone including a virtual assistant.

[0304] Hardware used to train machine learning models for dialogue action (DA) agents and evaluate those agents can have significantly more computing power, for example, more operations per second and more memory compared to hardware used to perform tasks with the agents.

[0305] The computing system 900 can be implemented as part of a mobile phone, smart speaker, embedded device, desktop computer, laptop computer, or server, including a virtual assistant.

[0306] although Figure 9 A specific example of hardware that can be used to implement the methods described herein is shown, but it will be appreciated that this is just one example and other arrangements may also be used.

[0307] While certain arrangements have been described, they are presented merely as examples and are not intended to limit the scope of protection. The inventive concepts described herein can be implemented in various other forms. Furthermore, various omissions, substitutions, and changes may be made to the specific implementations described herein without departing from the scope of protection defined in the claims.< / hangup>

Claims

1. A computer-implemented method for generating a response in a dialogue system, wherein the dialogue system is used to engage in dialogue with a user, the method comprising: Receive messages from users; Update the dialogue state based on the utterance; Generate information that identifies the state of the dialogue; The first dialogue action agent selects a first candidate action from a first candidate action set using a first machine learning model and information that identifies the dialogue state, wherein the first candidate action set is associated with a first dialogue dimension. The second dialogue action agent uses a second machine learning model and information that identifies the dialogue state to select a second candidate action from a second candidate action set, wherein the second candidate action set is associated with a second dialogue dimension. The evaluation agent uses a third machine learning model to disable a first output action, the first output action including a first combination of the first candidate action and / or the second candidate action; The evaluation agent uses the third machine learning model, the first candidate action, and the second candidate action to select a second output action, wherein the second output action includes: a second combination of the first candidate action and / or the second candidate action; as well as The system response is generated based on the second output action.

2. The computer implementation method according to claim 1, wherein selecting the second output action includes: Generate a value vector, the value vector including an indication of the estimated cumulative reward for each combination of the first candidate action and / or the second candidate action, in a given dialogue state, associated with selecting the corresponding combination as the system response; as well as The second output action is selected based on the value vector.

3. The computer implementation method of claim 2, wherein the value vector is generated using a function parameterized by a first set of trainable weights, wherein the input to the function is information identifying the dialogue state, and the output of the function is an estimated cumulative reward for each combination.

4. The computer implementation method according to claim 3, wherein the function is a linear function.

5. The computer implementation method according to any one of claims 2 to 4, wherein selecting the second output action based on the value vector comprises: The value vector is converted into a probability distribution to generate an action probability vector; as well as The second output action is selected based on the action probability vector.

6. The computer implementation method according to claim 5, wherein converting the value vector into the probability distribution to generate the action probability vector comprises: The value vector is converted into a transformed action value vector, wherein the transformed action value vector includes values ​​greater than or equal to zero; A masked transformed action value vector is formed by setting the first action value in the transformed action value vector to zero based on a predetermined criterion. as well as The masked, transformed action value vector is normalized to form the action probability vector.

7. The computer implementation method according to claim 1, wherein the first dialogue dimension is different from the second dialogue dimension.

8. A computer implementation method for training a dialogue system, the method comprising: The method according to claim 1 is used to generate a system response; Provide the system response to the user; as well as Reinforcement learning is used to train the first machine learning model, the second machine learning model, and the third machine learning model.

9. The computer implementation method of claim 8, wherein training the first machine learning model, the second machine learning model, and the third machine learning model using reinforcement learning comprises: Determine the cumulative reward; as well as The first machine learning model, the second machine learning model, and the third machine learning model are trained based on the cumulative reward.

10. The computer implementation method according to claim 9, wherein: The first machine learning model is configured to estimate a first estimated cumulative reward for the first candidate action based on a first function parameterized by a first set of trainable weights; and wherein: Training the first machine learning model based on the cumulative reward includes: Determine the difference between the accumulated reward and the first estimated accumulated reward; and The first set of trainable weights is updated based on the differences.

11. The computer implementation method according to claim 10, wherein: The third machine learning model is configured to estimate a second estimated cumulative reward for the second output action based on a second function parameterized by a second set of trainable weights; and wherein: Training the third machine learning model based on the cumulative reward includes: Determine the difference between the accumulated reward and the second estimated accumulated reward; and The second set of trainable weights is updated based on the differences.

12. The computer implementation method of claim 9 or 10, wherein the cumulative reward is based on a sum of rewards, the sum of rewards including a first reward generated for providing the system response.

13. The computer implementation method according to claim 12, further comprising: In response to providing the system response, a second message is received; The method of claim 1 is used to generate a second system response, wherein the utterance in the method of claim 1 is the second utterance, and the system response generated by the method of claim 1 is the second system response; A second reward is generated based on the response of the second system. The cumulative reward is calculated based on the sum of the first reward and the second reward.

14. The computer-implemented method according to any one of claims 8 to 11, wherein: The dialogue system includes a first hyperparameter configured to control an exploration quantity of the following dialogue strategies: a first dialogue strategy implemented by a first machine learning model, a second dialogue strategy implemented by a second machine learning model, and a third dialogue strategy implemented by a third machine learning model, wherein: The first hyperparameter has a first value during the first dialogue, the first dialogue including the utterance and the system response, and the method further includes: Determine whether the first conversation has been completed, and in response to determining that the first conversation has been completed: Modify the first hyperparameter so that the exploration amount of the first dialogue policy, the second dialogue policy, and the third dialogue policy in the state-action space is less than the exploration amount during the first dialogue; and The dialogue system was retrained using a second dialogue session.

15. An apparatus for generating a response in a dialogue system, wherein the dialogue system is used to engage in dialogue with a user, the apparatus being configured to: Receive messages from users; Update the dialogue state based on the utterance; Generate information that identifies the state of the dialogue; The first dialogue action agent selects a first candidate action from a first candidate action set using a first machine learning model and information that identifies the dialogue state, wherein the first candidate action set is associated with a first dialogue dimension. The second dialogue action agent uses a second machine learning model and information that identifies the dialogue state to select a second candidate action from a second candidate action set, wherein the second candidate action set is associated with a second dialogue dimension. The evaluation agent uses a third machine learning model to disable a first output action, the first output action including a first combination of the first candidate action and / or the second candidate action; The evaluation agent uses the third machine learning model, the first candidate action, and the second candidate action to select a second output action, wherein the second output action includes: a second combination of the first candidate action and / or the second candidate action; as well as The system response is generated based on the second output action.

16. The apparatus of claim 15, wherein the apparatus is further configured to: Generate a value vector, the value vector including an indication of the estimated cumulative reward for each combination of the first candidate action and / or the second candidate action, in a given dialogue state, associated with selecting the corresponding combination as the system response; and The second output action is selected based on the value vector.

17. The apparatus of claim 16, wherein the apparatus is further configured to generate the value vector using a function parameterized by a first set of trainable weights, wherein the input to the function is information identifying the dialogue state, and the output of the function is an estimated cumulative reward for each combination.

18. The apparatus of claim 17, wherein the function is a linear function.

19. The apparatus according to any one of claims 16 to 18, wherein the apparatus is further configured to: select the second output action based on the value vector. The value vector is converted into a probability distribution to generate an action probability vector; and The second output action is selected based on the action probability vector.

20. The apparatus of claim 19, wherein the apparatus is further configured to: The value vector is converted into a transformed action value vector, wherein the transformed action value vector includes values ​​greater than or equal to zero; A masked transformed action value vector is formed by setting the first action value in the transformed action value vector to zero based on a predetermined criterion; and The masked, transformed action value vector is normalized to form the action probability vector.

21. The apparatus according to any one of claims 15 to 18, wherein the first dialogue dimension is different from the second dialogue dimension.

22. A training system for training a dialogue system, the training system comprising the apparatus of claim 15, the training system being configured to: Use the apparatus of claim 15 to generate a system response; Provide the system response to the user; and Reinforcement learning is used to train the first machine learning model, the second machine learning model, and the third machine learning model.

23. The training system of claim 22, wherein the training system is further configured to: when using reinforcement learning to train the first machine learning model, the second machine learning model, and the third machine learning model. Determine the cumulative rewards; and The first machine learning model, the second machine learning model, and the third machine learning model are trained based on the cumulative reward.

24. The training system according to claim 23, wherein: The first machine learning model is configured to: estimate a first estimated cumulative reward for the first candidate action based on a first function parameterized by a first set of trainable weights; and wherein the training system is further configured to: Determine the difference between the accumulated reward and the first estimated accumulated reward; and The first set of trainable weights is updated based on the differences.

25. A non-transitory computer-readable medium comprising computer program instructions suitable for execution by a processor, the instructions being configured, when executed by the processor, to cause the processor to perform the method of claim 1.

Images