DEEP LEARNING OF BOTS THROUGH EXAMPLES AND EXPERIENCE

Description

[0001] Tsung-Hsien Wen et al in "A Network-based End-to-End Trainable Task-oriented Dialogue System", arXiv:1604.04562v1, describe a neural network based text-in, text-out end-to-end trainable dialogue system along with a new way of collecting task-oriented dialogue data based on a novel pipe-lined Wizard-of-Oz framework. The approach allows a dialogue system to be easily developed and the model can converse with human subjects naturally whilst helping them to accomplish tasks in a restaurant search domain.

[0002] International Patent Application WO2013 / 010262A1 describes processing natural language user queries for commanding a user interface used to perform functions. Individual user queries are classified in accordance with the types of functions and a plurality of user queries may be related to define a particular command. To assist with classification, a query type for each user query is determined where the query type is one of a functional query requesting a particular new command to perform a particular type of function, an entity query relating to an entity associated with the particular new command having the particular type of function and a clarification query responding to a clarification question posed to clarify a prior user query having the particular type of function. Functional queries may be processed using a plurality of natural language processing techniques and scores from each technique combined to determine which type of function is commanded.

[0003] Ryan Lowe et al, in "The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems", arXiv: 1506.08909v3, describe Ubuntu Dialogue Corpus, a dataset containing almost 1 million multi-turn dialogues, with a total of over 7 million utterances and 100 million words. This provides a unique resource for research into building dialogue managers based on neural language models that can make use of large amounts of unlabeled data. The dataset has both the multi-turn property of conversations in the Dialog State Tracking Challenge datasets, and the unstructured nature of interactions from microblog services such as Twitter. They also describe two neural learning architectures suitable for analyzing this dataset, and provide benchmark performance on the task of selecting the best next response.SUMMARY

[0004] The invention is set out in the appended set of claims

[0005] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to exclusively identify key features or essential features of the claimed subject matter, nor is it intended as an aid in determining the scope of the claimed subject matter.

[0006] The embodiments of the disclosure include methods and systems for using, creating and maintaining goal-oriented, dialog systems (i.e., "transactional bots" or "bots") that provide interfaces to application functionality such as, for example, interfaces to application functionality for ordering a taxi, controlling devices in the home, banking, or shopping. The methods and systems of the embodiments provide a bot that may learn in supervised learning and reinforcement learning from conversational examples provided by domain experts and from interaction with users. The embodiments provide conversational bots that may be created to interact using both text and / or application programming interface (API) calls. Use of the embodiments allows a developer to configure a bot that interfaces with an application back-end and allows the behavior of the bot to be configured by use of masking actions. Use of the embodiments also allows flexible design of a specification for the bot that specifies how developer code may be organized, for example, as masking operations on the possible actions the bot may execute. Additionally, the methods and systems may automatically infer the best state representation during a dialog so a state variable need not be predefined.

[0007] In an embodiment the methods and systems may be implemented as a bot using a Long Short-Term memory (LSTM) recurrent neural network model. The LSTM recurrent neural network may be optimized with supervised learning and / or using reinforcement learning via a specialized policy gradient method. In the implementation bot behavior may be specified with example dialogs. Bot behavior also may be improved autonomously from interactions with users without human intervention using a controller that makes use of business rules to gate exploration.

[0008] The claimed embodiments comprise a system according to claim 1..

[0009] In an implementation, the system and recurrent neural network may perform operations in a loop type behavior while the interactive dialog occurs by varying the configuration of the set of features provided to the recurrent neural network. For example, the system may include features generated by the system, features returned from an API call (if the previous action was an API call), or an indication of a previous action taken by the system (if there was a previous action), when providing the set of features to the recurrent neural network.

[0010] Another example implementation the system may include one or more processors that control a supervised learning process. The system may include memory that includes programs or code, that when executed, causes the one or more processors to control the system to receive one or more sample dialogs created by a developer, determine if any action in the one or more sample dialogs is masked out, and, in response to a determination that no action in the one or more sample dialogs is masked out, incorporate the one or more sample dialogs into a training set. The system may also create a log including a set of features and entity extractions for the training. The system may then apply training to a recurrent neural network using the training set and the log, score the one or more sample dialogs using the recurrent neural network to generate a score result based on the training, and, determine if a target action in the one or more sample dialogs was not assigned a highest probability in the score result. The determination of whether a target action was not assigned a highest probability may then be used to further refine the training. For example, if it is determined that a target action in the one or more sample dialogs was not assigned a highest probability in the score result, the system may provide an indication of a dialog of the one or more sample dialogs in which the target action was not assigned a highest probability to the developer. If it is determined that each target action in the one or more sample dialogs was assigned a highest probability in the score result, the system may provide an indication that all of the one or more sample dialogs were reconstructed successfully to the developer.

[0011] In a further example implementation the system may include one or more processors that control a reinforcement learning process. The system may include memory that includes programs or code, that when executed, causes the one or more processors to control the system to receive a definition for a reward signal and a return for one or more dialogs, perform the one or more dialogs using a recurrent neural network, create a log of rewards, features, available actions, and selected actions from the performed one or more dialogs, provide the return for the performed one or more dialogs to the neural network for use in improvement of the neural network, and provide the log for output at a user interface.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIGURE 1 is a simplified block diagram illustrating example functions according to an embodiment of the disclosure; FIGURE 2 is a flow diagram illustrating example signal flows between functional blocks of an operational loop in an implementation according to FIGURE 1; FIGURE 3 is a flow diagram illustrating runtime operations performed by an example system; FIGURE 4 is a flow diagram illustrating supervised learning operations performed by an example system; FIGURE 5A is a flow diagram illustrating reinforcement learning operations performed by an example system; FIGURE 5B is a flow diagram illustrating improvement operations for reinforcement learning operations performed by an example system; and, FIGURE 6 is a simplified block diagram illustrating an example computing device. DETAILED DESCRIPTION

[0013] The system and method will now be described by use of example embodiments. The example embodiments are presented in this disclosure for illustrative purposes, and not intended to be restrictive or limiting on the scope of the disclosure or the claims presented herein.

[0014] The disclosed embodiments provide a technical advantage as compared to currently used methods for using and creating transactional bots (bots) for dialog systems. The embodiments provide a system that allows a neural network to be optimized both with supervised learning and using reinforcement learning via a specialized policy gradient method. In an implementation bot behavior may be specified with example dialogs. Bot behavior also may be improved autonomously from interactions with users without human intervention using a controller that reinforces business rules to gate exploration by the neural network. Additionally, use of the embodiments with a recurrent neural network allows inference of a latent representation of the state of the system.

[0015] Use of the embodiments provides an advantage over currently used rule-based methods. Rule languages used to define behavior in rule based systems are often not easy to use by non-experts. It is often difficult for a program manager, designer, or marketing executive to build or maintain a bot without support from a software developer expert in rules languages. Also, as the number of rules for defining a system grows, the rules begin to interact and, as a result, rule changes have unforeseen consequences, so fixing one bug can introduce many other bugs. This makes maintenance of non-trivial rule based systems slow and extremely difficult. Additionally, rule-based methods do not learn automatically from experience. Even if a bot conducts a million dialogs a day, no automatic improvement is possible. Improvements to bot behavior can only be made through time-consuming human analysis. Use of the embodiments of the disclosure allows bot behavior to be specified using example dialogs without the need to know a specialized rules language. For example, designers, program managers, or marketing executives may configure the system without help from specialized software developers who know the rules language. Use of the embodiments also allows bot behavior to be improved autonomously through interactions with users without human intervention. The embodiments also provide the advantage that exploration performed by a neural network can be gated or restricted in scope by optional code that enforces rules for bot behavior.

[0016] The capability of the embodiments for bot behavior to be improved autonomously through interactions with users of the embodiments also provides an advantage over currently used supervised learning (SL) methods. In SL, a domain expert provides example dialogs that the bot should imitate, and a machine learning algorithm ingests these and builds a model which attempts to generalize to new dialogs. SL methods do not have the ability to make improvements automatically through experience or through reinforcement learning. In order to make improvements in bot behavior, a domain expert must manually examine and label dialogs.

[0017] The capability of the embodiments of the disclosure to infer a latent representation of state without the need to manually define the state also provides an advantage over currently used supervised learning SL methods. Current SL methods require the careful design of a "state" variable, which the SL algorithm uses as the basis for choosing actions. Design of the state variable is problematic. Including too little information in the state variable definition prevents the SL algorithm from successfully learning to reconstruct the example dialogs. Including too much information in the state variable definition causes over-fitting or narrowing of behavior, which means the bot will not generalize to new situations. In the embodiments, use of a recurrent neural network allows inference of a latent representation of state. This inference of state substantially reduces the effort developers need to put into hand-engineering the state required by current SL methods.

[0018] Also, current SL methods do not allow actions to be masked by a developer. The capability of the embodiments to mask actions allows support of business logic, such as only allowing an action like transferring funds after an action confirming the transfer with the user has succeeded.

[0019] The embodiments of the disclosure also provide advantages over currently used reinforcement learning (RL) methods in which an RL algorithm explores different actions in different states, and over time makes progress toward finding the mapping from state to action that maximizes a reward signal. RL learns automatically, without input from system designers. RL systems are difficult to create and, typically, for the first 100s or 1000s of dialogs, RL systems usually perform very badly because an RL system explores all possible actions, including spurious actions. In the embodiments of the disclosure the capability to gate exploration using optional code that enforces business rules may be used to prevent exploration of spurious actions and improve system behavior as compared to the current RL systems. Also, in an RL system, as with an SL system, careful definition of the state variable is needed. The embodiments of the disclosure remove the need to define the state variable that is present in current RL systems.

[0020] Referring now to FIGURE 1, therein is a simplified block diagram illustrating example functions according to an embodiment of the disclosure. FIGURE 1 illustrates functional blocks of an example network 100 including an example system 104. In the example implementations system 104 may implement functionality for a bot and include entity extractor 106, recurrent neural network 108, and controller 110. System 104 may also include a developer interface 112.

[0021] System 104 interacts with device 102 and application server 114. Device 102 may include an application that allows a user to interact with and receive services from application server 114 through system 104. System 104 may communicate with a user of device 102 via one or more channels. The one or more channels may be any type of communication channels carrying communications for applications including short messaging services (SMS), email services, messaging platforms such as conferencing platforms, social network platforms, text messaging platforms, or any other type of application using text communication. The applications may also include any type of application using voice or spoken communications, or web browsers. In various implementations text may be typed in by the user, or transcribed text produced by a speech recognizer, or, also may be translated from another language using an automatic translation service. The user could engage in a one to one dialog with the bot, or the bot could be participating in a dialog with a plurality of users. The bot could be addressed directly, or the bot could monitor a conversation and respond when it determines a response would be relevant

[0022] In the claimed embodiments of the invention, the device 102 is a home appliance. In unclaimed embodiments, the applications may include, for example, reservation applications, retail / purchasing applications, information retrieval applications, or any other type of application that may interact with a user through a dialog. The functions of entity extractor 106, recurrent neural network 108 and controller 110 of system 104 may be implemented using one or more servers. In alternative embodiments of FIGURE 1, the functions described for a particular functional block in relation to FIGURE 1 may be performed in another of the functional blocks of FIGURE 1 or divided up between the functional blocks in a different way.

[0023] Referring now to FIGURE 2, therein is a flow diagram illustrating example signal flows for data exchange between functions of an operational loop 200 in an implementation according to FIGURE 1. The functions of FIGURE 1 may be described with reference to FIGURE 2. Block text input / text output 202 may represent functions performed at device 102, blocks entity input 206, action mask 222, entity output 228, and API call 218, may represent function performed by controller 110, and block entity extraction 204 may represent functions performed at entity extractor 106. Long short-term memory recurrent neural network (LSTM) may represent functions performed at recurrent neural network 108. Softmax 224 represents a conversion of LSTM 220 output values into action probabilities for communication to controller 110.

[0024] The process begins when device 102 provides a text input signal 238 to entity extractor 106. At 204, entity extraction is performed and entities are identified in the text. For example, in the query "Will it rain in Sacramento on Friday?", entity extraction at 204 may identify "Sacramento" as a city, "Friday" as a day, and "rain" as a weather event. The entity extraction of 204 may also use custom models specific to the user's domain, or may use generic models suitable for many domains. In the claimed embodiments, generic models extract entities such as dates, times, and locations, etc. the entity extraction at 204 may optionally resolve entities to a machine interpretable form. For example, "January" might be resolved to "month=1". In unclaimed embodiments, generic models may also extract monetary amounts.

[0025] Next, entity extractor 106 sends signal 208 to pass marked up text at 206 to the controller 110. Entity extractor 106 may also pass the input text and entities to recurrent neural network 108 through signal 208 at block 210. Controller 110 is configured to perform a defined set of operations and the developer may be further guided in what to write through the user-interface design. The controller 110 may resolve the text of entities to ontology within the domain of a particular application. For example, it might resolve the user's input of "town car" to a canonical indication like "CAR_TYPE=UBER_LIVERY". The controller 110 code may also maintain its own internal state, for example, tracking entities that have been input or output over the course of the dialog. Controller 110 implemented in a variety of ways, including as a web service (e.g. in C# or node.js), or run locally.

[0026] The controller 110 returns a "mask" over actions, i.e., an indication of which actions in the text are allowed and disallowed at action mask 222 through signal 223 to recurrent neural network 108. The code also returns other features at block 210 which can be used by recurrent neural network 108. The other features input to neural network by signals 214 and 216 depend on whether any previous operations were performed in the loop 200 and what those operations were. When controller 110 returns an indication of which actions are available, the list of actions returned may include wildcards. For example, the indication of available actions may indicate that any text output action is allowed with a wildcard indication for text output, but only certain API calls are allowed.

[0027] Masking in the masking over available actions takes into account which entity types are available to controller 110. For example, if the entity for "DEPARTURE_CITY" hasn't been received, actions that ask the user to confirm "DEPARTURE_CITY" (like "Leaving from Boston, is that right?") may be masked out. If controller 110 returns additional features to neural network 108, these can be implemented as a programmatic dictionary, like { "estimated_wait_minutes": 10, "user_logged_in": false }

[0028] At block 210, the set of features is then passed to recurrent neural network 108 along with the mask over available actions, if provided at 222. The set of features includes the text of the user's input at 202, an indication of which entities or types of entities were detected at 204, features returned from the developer code at 206, features returned from an API call (if the previous action was an API call), and an indication of the previous system action taken (if a previous system action was taken). The features can also include other items from previous time steps, however, the stateful nature of the recurrent neural network should prevent the need to do this.

[0029] At 220, the recurrent neural network 108 performs the function of a LSTM recurrent neural network and generates a distribution over actions to take. The distribution may be based on the received set of features and a received mask over available actions. The output actions may be represented as a "flat" list, or as a generative process. For example, a generative process might use a second neural network which can generate text, along with the probability of that text.

[0030] The output actions may include references to entity types, such as "CAR_TYPE", without specific values populated (like "town car"). Using entity types in actions rather than entity values may substantially reduce the number of actions, and allow the system to generalize over entity values it has never seen before. This may be an advantage when an entity may take on many values, or when the possible values may change over time, such as "RESTAURANT_NAME" or "EMAIL_SUBJECT". These references may be populated, before they are output.

[0031] If the output actions are represented as a flat list, the mask may be implemented by multiplying the raw (unmasked) action probabilities by an array of 1s and 0s, with 1s for allowed actions and 0s for disallowed actions. This array may then be renormalized so the non-zero actions sum to 1. The gradients of weights with respect to these masked outputs may be well-defined, so gradient descent methods may be applied.

[0032] Recurrent neural network 108 then samples an action from the (masked) distribution of actions. The action may be sampled from the distribution in the traditional sense. This type of sampling may provide an advantage when exploring for reinforcement learning in the neural network, at the expense of some performance. In another implementation, the action with the maximum probability may be selected. This type sampling may provide an advantage for maximizing performance, at the expense of not exploring.

[0033] At block 228, the controller 110 is sent an indication of which action has been selected. Controller 110 may update its internal state at this point. Controller 110 may also send an indication of the selected action to recurrent neural network through signal 216 for use as a feature. If the chosen action contains references to entity types, they are populated by the developer code at 228. For example, DESTINATION_CITY in "Going to DESTINATION_CITY, is that right?" may be changed to "Seattle". The selected action, with entity substitution, is then executed.

[0034] At 230 the action type is determined. If the action is a textual action, it is output to the user as text output at 202 through signal 234. If the action is an API call, the API call is invoked at 218 through signal 232. The API call at 218 may optionally return features related to the API call to neural network 108 through signal 214. If the action is a special "LISTEN" action, control returns to 202 and the system waits for user input. Other special actions may be defined, such as "QUIT" (which causes the system to abandon the conversation), "ESCALATE" (which causes a human agent to step in on behalf of the agent). The special actions may be defined to include any other appropriate action.

[0035] In other implementations, the flow described above may be modified to allow for asynchronous input, for example, if the user types input while the system is waiting on a response from an API call, the system may queue up user input and execute the loop when user input is received. In embodiments, API calls may interact with external services, or may interact with the user. For example, an API call could show a map, picture, or contact card to the user.

[0036] Following is a simple example of the type of dialog sequence that is possible with this loop: How can I help you? "Call Jason at his home" [API call to look up phone numbers] Sorry, I don't have a home number for Jason Williams. I have mobile or work. Which would you like? "Mobile" Calling Jason Williams, mobile. [API call to choose mobile phone number] [API call to place telephone call]

[0037] Referring now to FIGURE 3, is a flow diagram illustrating runtime operations performed by example system 104. FIGURE 3 illustrates operations performed by system 104 of FIGURE 1 during execution of runtime loop 200. FIGURE 3 is similar to FIGURE 2 but shows operations in a linear manner.

[0038] The process begins at 302 where user input is received. At 304, entity extraction is performed on the user input by entity extractor 106. At 306, marked up text is passed to the controller 110. The input text and entities may also be passed to the recurrent neural network (LSTM). At 308, controller 110 determines the mask over actions, and at 310 the set of features and the mask are passed to the recurrent neural network 108. The set of features may include features as described for FIGURE 2. At 312 the recurrent neural network 108 provides controller 110 with the selected action based on the set of features and the mask. At 314 controller110 updates its internal state. Controller 110 may also send an indication of the selected action to recurrent neural network 108 for use as part of a feature set, or, if the selected action contains references to entity types, populate the references.

[0039] At 322, controller 110 determines if the selected action includes entity references. If the selected action includes entity references, controller 110 populates the references at 324 as was described in relation to FIGURE 2 and moves to operation 326. If the selected action does not include entity references the process moves from 322 to 326.

[0040] At 326 it is determined if the selected action is a textual action. If the selected action is a textual action, at 328, controller 110 initiates the performance of text output at device 102 and returns to operation 308 at 330. If the selected action is not a textual action the process moves from 326 to 332.

[0041] At 332 it is determined if the selected action is an API call. If the selected action is an API call, at 334, controller 110 performs the appropriate API call. Controller 110 may also send an indication of the API call to recurrent neural network as part of the feature set. Next at 336 the process returns to operation 308. If the selected action is not an API call the process moves to 338.

[0042] At 338 it is determined if the selected action is a "listen" action. If the selected action is a listen action, the controller 110 initiates listening for user input at 340. When user input is received the process returns from 342 to 302.

[0043] If the selected action is not a listen action the process moves to 344 and determines a quit action was received. The process then ends at 346.

[0044] In alternative implementations, any number of actions may be defined for use in the process, and controller 110 may direct the process appropriately depending on which of the actions is selected.

[0045] Referring now to FIGURE 4, therein is a flow diagram illustrating supervised learning operations performed by an example system according to the embodiments. The process of FIGURE 4 may be performed for supervised learning operations in the system 104 of FIGURE 1. The process of FIGURE 4 may be used improved performance of system 104 by providing example dialogs, and training the recurrent neural network 108 to mimic the example dialogs.

[0046] The process begins at 402 where one or more sample dialogs are created. Because the embodiments do not use rule-based methods, the dialogs maybe created by persons who are non-experts in rules languages. For example, a domain expert such as a designer, program manager, marketing executive, or developer may create the one or more new sample dialogs. The sample dialogs may include an indication of where API calls are made and may be created in entirety by the domain expert through interaction with the current bot. The one or more sample dialogs also may be drawn from an existing corpus of interactions with real users or may be synthesized through interaction with a simulated user. The one or more sample dialogs may also be created by interacting with crowd workers, or taking existing dialogs and scrambling them.

[0047] At 404 a supervised learning training flag is passed to controller 110.

[0048] The run-time loop, which was described in relation to FIGURE 2, is run on the one or sample dialogs in a modified loop form in performing the supervised learning (SL) of FIGURE 4. In the modified loop, actions are not sampled, but rather, a check is done to see if each action in the sample dialog is masked out or not. At 406 the controller determines if any action in the one or more sample dialogs is masked out. At 408 if an action in the one or more sample dialogs is masked out the process moves to 410. If any action in the sample dialog is masked out, this means the dialog could not be produced with the existing developer code, i.e., the dialog is inconsistent with the developer code. At 410 the inconsistency is reported, for example, to the developer. The developer may then edit the dialogs or controller code at 412. The process may then return from 414 to 402 to begin again.

[0049] If at 408 no action in the one or more sample dialogs is masked out the process moves to 416. When no actions in the sample dialogs are masked out, this indicates the dialog may be produced by the existing developer code. In this case, the dialog is incorporated into the training set. In this modified loop, at 418, a log is created for the features reported by the developer code and entity extraction model. The log created at 418 may be used by the supervised learning algorithm in the SL training.

[0050] At 420 the SL training is then applied using recurrent neural network 108 and the entire corpus of training dialogs. The inputs are as described for the runtime loop of FIGURE 2, and the target outputs are the actions which appear in the training dialogs.

[0051] The recurrent neural network 108 may use gradient descent to train the model. If the model is a distribution over a flat list of actions, categorical cross-entropy between the model output and the one-hot vector encoding of the target action may be used by the recurrent neural network as the loss function. After SL training is applied, at 422, all of the dialogs in the training corpus are then scored using the new sequence model. At 424, a check is then performed to see whether any of the target actions in the training dialogs were not assigned the highest probability by the model.

[0052] Next, at 432 if any action in a training dialog was not assigned the highest probability by the new sequence model, the disagreement indicates that the new sequence model has failed to re-construct the training dialogs and the process moves to 428. At 428, the dialog turns of any disagreement are provided to the developer. The developer may then resolve the disagreement at 430 by changing, deleting, or adding an example dialog in the training corpus, modifying the developer code, modifying the SL learning algorithm or parameters, or modifying the entity extraction model. If however if any action in a training dialog was not assigned the highest probability by the new sequence model, the process moves from 432 to 434. At 434, the developer may be provided with an indication that the new sequence model has successfully re-constructed the training dialogs. The SL learning cycle may then be repeated. At any time, the developer may "deploy" the trained bot, so it is available for interaction with users in the runtime loop.

[0053] In addition to SL training, use of the embodiments also allows the use of reinforcement learning (RL) training to improve performance of a bot. Referring now to FIGURE 5A, therein is a flow diagram illustrating reinforcement learning operations performed by an example system. FIGURE 5A shows operations to optimize performance of system 104 of FIGURE 1 in the runtime loop of FIGURE 2 through interaction with users, and adjustment of the model using reinforcement learning.

[0054] The process of FIGURE 5A, begins at 502 where the developer defines a reward signal. The reward signal may be a real-valued number that indicates how "good" a system action is in a particular context. The rewards signal may be inferred from usage. For example, the signal may indicate whether the user achieved a desired task. The rewards signal may also be provided by the user providing an answer to a question at the end of a dialog such as "Did I do OK?" The reward signal may also be exogenous to the dialog, for example, whether the user later made a purchase sometime after the dialog completed. Alternately, the rewards signal may be provided by a labeler who examines dialogs and scores their quality. The reward signal may indicate a property of the whole dialog, a portion of a dialog, or a specific turn of dialog. For example, a reward at the end of the dialog might indicate a task completion for the whole dialog and a reward at a specific turn in the middle of a dialog might indicate the quality for that specific action.

[0055] At 504 the developer defines a return for the dialog. The overall return for the whole dialog may be a discounted sum of the rewards at each turn. The discount factor may be defined by the developer, or may be set to a standard value. For example the discount factor may be set to a value such as 0.95.

[0056] At 506, system 104 conducts a batch of dialogs. The batch may include one or more dialogs with one or more users. The users may be real users, crowd workers, or user simulations. At 508, the rewards, features, available actions, and actions selected from the batch are logged. At 510, recurrent neural network 108 makes improvements based on the rewards received during the batch. The improvement at 510 may be made using a policy gradient. Next, at 512 the recurrent neural network 108 is updated and deployed. At 514, it is determined if the training if finished. If the training is finished the process ends and performance is reported to the developer at 516. If the training is not finished the process moves back to 506 and continues conducting the training dialogs through the process of FIGURE 5A until training is finished.

[0057] In an alternate embodiment, the batch of dialogs used for improvement may include dialogs collected in previous batches. In a further embodiment, some iterations of FIGURE 5A may skip collecting new dialogs, and instead rely on dialogs collected from past batches.

[0058] Referring to FIGURE 5B, is a flow diagram illustrating improvement operations for RL operations performed by an example system at operation 510 of FIGURE 5A. At 520 the probabilities of masked actions are set to zero. At 522, a constant is added to all of the probabilities. The constant is added to prevent the logarithm and its derivative of the masked actions from being zero and undefined later in the process. The constant may be added to all action probabilities prior to taking the logarithm. This causes weights to have a zero derivative, i.e., no effect, with respect to masked actions, and other actions to be unaffected because the derivative of an additive constant is zero.

[0059] At 524, the gradients of the action probabilities for each turn with respect to weights are determined. At 526, adjustment of the weights is performed in view of gradients, the return of dialog, and the estimated average of the current model. The gradients for each dialog may be multiplied by a "step". The step may correspond to the quantity (R_n-B) where R_n is the observed return for dialog n, and B is the estimated average return of the current neural network. B may be computed by averaging R_n in the current batch, choosing a constant, or using some form of importance sampling, such as weighted importance sampling. When importance sampling is used, either the dialogs in the current batch may be used, the most recent K dialogs, or all dialogs observed to date.

[0060] When the weights have been adjusted the updated recurrent neural network is deployed as neural network 108 at operation 512 of FIGURE 5A.

[0061] Performance of reinforcement learning may be periodically reported to the domain expert. Also, constraints may be added which ensure that the training dialogs in the corpus are always re-constructed. For example, if the updated neural network 108 fails to re-generate a training dialog, then SL gradient descent can be applied on that dialog until it is re-generated.

[0062] Referring now to FIGURE 6, therein is a simplified block diagram of an example computing device 600. System 104 of FIGURE 1 may be implemented on a device such as computing device 600. Computing device 600 may include a server 604 having processing unit 606, a memory 610, network interfaces 602, and developer interfaces 608. Memory 610 may be implemented as any type of computer readable storage media, including non-volatile and volatile memory. Memory 610 is shown as including extraction programs 612, controller code 614, and neural network code 616. Server 604 and processing unit 606 may comprise one or more processors, or other control circuitry, or any combination of processors and control circuitry. Server 604 and processing unit 606 provide overall control of computing device 600 to implement the functions of an entity extractor, recurrent neural network, and controller according to the disclosed embodiments. Computing device 600 may also be configured to provide developer interface 608 which may be configured, for example, as developer interface 112 of FIGURE 1.

[0063] Developer interface 608 may be configured to allow a developer overall control of management and training of computing device 600. Developer interface 608 may be a user interface, such as a web interface, or any other application which guides the developer. In one implementation, developer interface 608 allows the developer to enter a new dialog. As the dialog is entered, the developer interface 608 may indicate what the next system response under the current model would be, or may indicate a ranking of multiple system responses ordered by their scores. Differences between the model output and the desired output help the developer to understand the strengths and weaknesses of the current model. Another section of the developer interface 608 may allow the developer to browse through the dialogs which have been entered so far, highlighting dialogs which disagree with the current model.

[0064] Developer interface 608 may also handle entity extraction. In one implementation, the developer interface 608 may provide a pointer to an external entity extraction service. In another implementation, entity extraction labeling and refinement may be performed, for example by labeling entities using the same developer interface 608 used for entering sample dialogs.

[0065] The developer interface 608 may also allow a developer to interface computing device 600 with custom computer code. In one implementation, the custom code can be entered directly into the developer interface 608 on server 604. In another implementation, the custom code may run on a separate server, for example, on a server hosted by the developer. This latter implementation involves the developer hosting their own webserver, but provides additional control, and allows the developer to avoid disclosing the implementation of their code at the developer interface. In either alternative, example starter code may show how to structure the code and what functions to implement.

[0066] The developer interface 608 may also allow the developer to manage the RL training. The developer may specify properties of reward signals, indicate whether the RL should be active or not, view graphs of performance over time, or mange other functions of the RL training.

[0067] Additionally, the developer interface 608 may allow the developer to set configuration options. These configuration options may specify the set of API calls that are available. (The API calls may also be provided or discovered programmatically). The configuration options may also include details about the HTTP endpoint on which the bot is available, authentication and subscription options, and general administration configuration options, such as which users have access to edit the bot.

[0068] System 600 is shown as an implementation that includes server 604 as a single server for performing operations of the embodiments according to programs and code in memory 610. However, server 604 and memory 610 may be understood as representative of server functions or memory provided by one or more servers or computing devices, or storage devices, that may be co-located or geographically dispersed and that may provide the functions of the entity extraction 106, the neural network 108, and the controller 110 for other implementations of system 104. For example, the controller code may be implemented on a separate server separate from the server on which the extractor and neural network code are implemented. The term server as used in this disclosure is used generally to include any computing devices or communications equipment.

[0069] The example embodiments disclosed herein may be described in the general context of processor-executable code or instructions stored on memory that may comprise one or more computer readable storage media (e.g., tangible non-transitory computer-readable storage media such as memory 610). As should be readily understood, the terms "computer-readable storage media" or "non-transitory computer-readable media" include the media for storing of data, code and program instructions, such as memory 610, and do not include portions of the media for storing transitory propagated or modulated data communication signals.

[0070] While implementations have been disclosed and described as having functions implemented on particular computing devices, server devices, and / or wireless devices operating in a network, one or more of the described functions for the devices may be moved between the devices and implemented on a different one of the devices than shown in the figures, or on different types of equipment.

[0071] While the functionality disclosed herein has been described by illustrative example using descriptions of the various components and devices of embodiments by referring to functional blocks and processors or processing units, controllers, and memory including instructions and code, the functions and processes of the embodiments may be implemented and performed using any type of processor, circuitry or combinations of processors and / or circuitry and code. This may include, at least in part, one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc. Use of the term processor or processing unit in this disclosure is meant to include all such implementations.

[0072] Although the subject matter has been described in language specific to structural features and / or methodological operations or acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features, operations, or acts described above. Rather, the specific features, operations, and acts described above are disclosed as example embodiments, implementations, and forms of implementing the claims and these example configurations and arrangements may be changed significantly without departing from the scope of the present invention, which is defined by the appended claims.

Claims

1. A dialog-based system configured for providing an interface to application functionality of a device (102) that allows a user to interact with and receive services from an application server (114), wherein the device (102) comprises a home appliance, the system comprising: one or more processors (606); and, memory (61) in communication with the one or more processors, the memory comprising code (612, 614, 616), that when executed, causes the one or more processor to control the system to: control an interactive dialog with a user; determine marked-up text from an input text (306) of the user wherein an entity extractor (106) extracts one or more entities from the input text (306) of the user to form the marked-up text, wherein the one or more entities comprise one or more of dates, times, and locations; determine, by a controller implemented by the execution of the code on the one or more processors (606), a mask over available actions from the marked-up text (308), wherein the mask indicates which actions of a set of actions are allowed and which are disallowed based on which entity types were extracted from the input text (306); provide, by the controller, the mask and a set of features to a recurrent neural network (310), wherein the set of features include the input text (306) of the user, an indication of the types of entities extracted from the input text, an indication of a previous action if a previous action took place and at least one feature associated with the previous action if the previous action took place (214, 216); receive, by the controller, an indication of a selected action from the recurrent neural network (312); update an internal state of the controller based on the indication of a selected action (314); and, initiate, by the controller, the selected action (326, 332, 338) to interface with an application back-end to control the home appliance.

2. The system of claim 1, wherein the selected action comprises a first selected action and the first selected action comprises an application programming interface call (334), the set of features comprises a first set of features, and the code further causes the one or more processors to control the system to: provide a second set of features to the recurrent neural network, the second set of features including a feature returned from the application programming interface call (214); receive an indication of a second selected action from the recurrent neural network (216); update the internal state based on the indication of the second selected action; and, initiate the second selected action.

3. The system of claim 1, wherein the set of features comprises a first set of features and the one or more processors are further caused to control the system to: provide a second set of features to the recurrent neural network, the second set of features including an indication of the selected action (216).

4. The system of claim 1, wherein the at least one feature associated with the previous action includes features that are associated with actions based on a previous set of features provided to the recurrent neural network (214).

5. The system of claim 1, wherein the recurrent neural network comprises a long short term memory neural network (220).

6. The system of claim 1, wherein the recurrent neural network comprises a standard recurrent neural network.

7. The system of claim 1, wherein the processors are caused to control the system to initiate the selected action by: determining that the selected action includes one or more entity references (322); populating the one or more entity references (324); and, initiating the selected action based on the populated references.

8. A method for providing an interface to application functionality of a device (102) that allows a user to interact with and receive services from an application server (114), using a dialog based system, wherein the device (102) comprises a home appliance, the method comprising: controlling an interactive dialog with a user; determining marked-up text from an input text (306) of the user wherein an entity extractor (106) extracts one or more entities from the input text (306) of the user to form the marked-up text, wherein the one or more entities comprise one or more of dates, times, and locations; determining, by a controller, a mask over available actions from the marked-up text (308), wherein the mask indicates which actions of a set of actions are allowed and which are disallowed based on which entity types were extracted from the input text (306); providing, by the controller, the mask and a set of features to a recurrent neural network (310), wherein the set of features include the input text (306) of the user, an indication of the types of entities extracted from the input text, an indication of a previous action if a previous action took place and at least one feature associated with the previous action if the previous action took place (214, 216); receiving, by the controller, an indication of a selected action from the recurrent neural network (312); updating an internal state of the controller based on the indication of a selected action (314); and, initiating, by the controller, the selected action (326, 332, 338) to interface with an application back-end to control the home appliance.

9. The method of claim 8, wherein the selected action comprises a first selected action and the first selected action comprises an application programming interface call, the set of features comprises a first set of features, and the method further comprises: providing a second set of features to the recurrent neural network, the second set of features including a feature returned from the application programming interface call (214); receiving an indication of a second selected action from the recurrent neural network (216); updating the internal state based on the indication of the second selected action; and, initiating the second selected action.

10. The method of claim 8, wherein the set of features comprises a first set of features and the method further comprises: providing a second set of features to the recurrent neural network, the second set of features including an indication of the selected action (216).

11. The method of claim 8, wherein the at least one feature associated with the previous action includes features that are associated with actions based on a previous set of features provided to the recurrent neural network (214).

12. The method of claim 8, wherein the recurrent neural network comprises a long short term memory neural network (220).

13. The method of claim 8, wherein the recurrent neural network comprises a standard recurrent neural network.

14. The method of claim 8, wherein initiating the selected action comprises: determining that the selected action includes one or more entity references (322); populating the one or more entity references (324); and, initiating the selected action based on the populated references.

15. A computer readable storage media containing instructions that when executed by a system according to any of claims 1-7 cause the system to perform a method according to any of claims 8 to 14.