Error recovery for session systems

Abstract

A method includes identifying user utterances for processing. The method further includes generating a data stream procedure based on the user utterance using a previously trained code generator, the data stream procedure being configured to produce a return value upon successful execution. The method also includes initiating execution of the data stream procedure. In response to reaching an error condition resulting from execution of the data stream procedure, the method further includes pausing execution of the data stream procedure before the data stream procedure produces the return value. The method also includes generating an error handling data stream procedure using the previously trained code generator, wherein the error handling data stream procedure is configured to: produce the return value; initiate execution of the error handling data stream procedure to produce the return value; and output the return value.

Description

Background Technology

[0001] A conversational system processes user utterances and responds by automatically performing actions, such as answering questions, invoking application programming interfaces (APIs), or otherwise assisting the user based on the utterances. Conversational systems are limited to processing a predefined set of hard-coded templates, which restricts the actions that can be performed by an automated assistant. Summary of the Invention

[0002] This overview is provided to introduce a set of concepts in a simplified form, which are further described in the detailed description below. This overview is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that address any or all of the shortcomings pointed out in any part of this disclosure.

[0003] A method includes identifying user utterances for processing. The method further includes generating a dataflow procedure based on the user utterances using a previously trained code generator, the dataflow procedure being configured to produce a return value upon successful execution. The method also includes initiating execution of the dataflow procedure. In response to reaching an error condition resulting from execution of the dataflow procedure, the method further includes pausing execution of the dataflow procedure before the dataflow procedure produces the return value. The method also includes generating an error-handling dataflow procedure using the previously trained code generator, wherein the error-handling dataflow procedure is configured to: produce the return value; initiate execution of the error-handling dataflow procedure to produce the return value; and output the return value. Attached Figure Description

[0004] Figure 1A-1D An exemplary data flow architecture for an automation assistant is shown.

[0005] Figure 1E An example dialogue between a user and an automated assistant is shown.

[0006] Figure 2 A method for handling ambiguous user utterances is shown.

[0007] Figures 3A-3C An exemplary user utterance containing ambiguity and a corresponding data stream procedure for processing the exemplary user utterance are shown.

[0008] Figure 4 A method for handling errors during the processing of user utterances is shown.

[0009] Figures 5A-5FAn exemplary user utterance is shown, a corresponding data stream procedure for processing the exemplary user utterance that may cause an error, and an alternative data stream procedure for resolving the error.

[0010] Figure 6 An exemplary computing system is shown. Detailed Implementation

[0011] Conversational computing interfaces can be used to interact with users via natural language (e.g., via voice and / or submitted text). As an example, an automated assistant can be used to assist a user through natural language interaction. Although this disclosure uses an automated assistant as an exemplary conversational computing interface, this example is non-limiting, and conversational computing interfaces can be implemented according to this disclosure for any suitable purpose, such as allowing a user to interact with any suitable computer hardware and / or computer software using natural language. Thus, every reference to an automated assistant in this disclosure also applies to any other conversational computing interface or other computing framework configured to respond to voice or text input.

[0012] Automated assistants can use Natural Language Processing (NLP) techniques (e.g., machine learning classifiers) to process input user utterances (e.g., user speech and / or submitted text) to perform predefined, hard-coded actions associated with the input user utterances. For example, an automated assistant may support multiple predefined, hard-coded templates, each with multiple time slots that can be populated to parameterize the hard-coded actions. As an example, an automated assistant may support predefined interactions to invoke application programming interfaces (APIs), such as booking a table at a restaurant, calling a ride-hailing service, or checking the weather. However, while an automated assistant may support multiple different predefined actions via predefined templates, an automated assistant that only supports predefined actions via templates may not be configured to perform more complex or novel behaviors.

[0013] This disclosure relates to an automated assistant that uses a dataflow program in a dataflow programming language to process user utterances (e.g., in addition to or as an alternative to using templates). The automated assistant uses a previously trained code generator to generate and / or output a dataflow program for the user utterances, wherein the dataflow program uses a plurality of predefined functions to define individual steps for processing the user utterances.

[0014] Processing user utterances using a dataflow program generated by a previously trained code generator can result in improved user experience, improved efficiency (e.g., improved storage usage and / or improved processing time) of automated assistants or other interactive computer services, and / or improved ability to respond to different user utterances. As an example, the dataflow program can encode various processing strategies for different user utterances, including performing computations based on the user utterance, accessing APIs for responding to the user utterance, etc. The code generator can generate a dataflow program specific to the user utterance being processed, enabling more efficient (e.g., avoiding irrelevant steps) and improved user satisfaction (e.g., by generating a program that better addresses the request expressed in the user utterance). Furthermore, the dataflow program can be paused when encountering one or more errors to handle the errors using an error recovery program, such as a modified version of the dataflow program or an alternative dataflow program. Thus, the approach using a previously trained code generator and dataflow program is more robust to different scenarios where errors may occur, as different types of errors can be handled by running the error recovery dataflow program.

[0015] therefore, Figure 1A A dataflow architecture for an automation assistant system 100 is illustrated. The automation assistant system 100 is configured to process user utterances 102 by operating a previously trained code generator 104, which is configured to output a dataflow program 106 for the user utterances 102. Although this disclosure focuses on interactions via natural language (e.g., speech and submitted text), the conversational computing interface of the automation assistant system 100 can also allow interactions via any other suitable input method, such as touchscreen input and / or button presses. Similarly, although this disclosure focuses on processing user utterances 102, other inputs such as button press events can be processed in a similar manner. For example, the previously trained code generator 104 can be configured to output a dataflow program 106 for one or more button press events. The previously trained code generator 104 can be trained to recognize different kinds of non-verbal input events, such as based on a specific button being pressed, timing of the button press relative to other input events, etc.

[0016] Dataflow program 106 is shown as a graph comprising multiple function nodes, wherein the function nodes are depicted using inputs and outputs indicated by arrows. The dataflow program is configured to produce a return value indicated by the bottommost arrow. A previously trained code generator 104 is configured to add any function from a plurality of predefined functions 110 to the dataflow program based on the user utterance. Each predefined function defines one or more individual steps for processing the user utterance 102. Dataflow program 106 is operable to cause the automated assistant to respond to the user utterance, for example, by performing any suitable response action. The predefined functions of dataflow program 106 may be operable to cause the automated assistant to perform any suitable response action, for example, outputting the response as speech and / or text (e.g., output as in...). Figure 1A The assistant response 120 shown in the diagram calls an API to perform an action (e.g., ordering food from a restaurant, arranging a ride using a ride-hailing service, scheduling a meeting in a calendar service, or making a phone call). Although this disclosure focuses on examples where the automated assistant responds to speech by outputting an assistant response (e.g., as voice and / or text), these examples are non-limiting, and the predefined functions and dataflow procedures described herein can be configured to cause the automated assistant to respond to speech in any suitable manner, such as by performing one or more actions using an API in lieu of or to supplement the output assistant response.

[0017] The previously described code generator 104 described herein can be used to respond to user utterances in any suitable manner. For example, the previously described code generator 104 can be configured to recognize user utterances and, based on the user utterances, generate a dataflow program that defines an operable plan for responding to the user utterances. The resulting dataflow program can be run, for example, by an automation assistant to process the user utterances. In some examples, the dataflow program for responding to the user utterances can be run to respond to the user utterances without requiring the code generator 104 to generate any additional code. In other words, the code generator 104 is configured to output a complete plan for processing user utterances. Alternatively or additionally, the code generator 104 can be configured to output a dataflow program for responding to user utterances, wherein some or all of the dataflow program is run before the code generator 104 is used to generate additional code that defines an additional plan for completing the processing of the user utterances. The code generator 104 can be used to plan and run the dataflow program in any suitable manner, including completing the planning before execution and / or interleaving the planning and execution in any suitable manner.

[0018] The previously trained code generator 104 can be based on any suitable technology, such as existing or future machine learning (ML), artificial intelligence (AI), and / or natural language processing (NLP) technologies. In some examples, the previously trained code generator 104 includes: an encoder machine configured to encode user utterances 102 into semantic features (e.g., vectors in a semantic vector space learned by the previously trained code generator 104); and a decoder machine configured to decode the semantic features by outputting one or more functions from a plurality of predefined functions 110. In some examples, the decoder machine is configured to output one or more functions based on typed syntax for combining functions from the plurality of predefined functions 110, thereby restricting the output of the decoder machine to a well-typed, fully runnable dataflow program. The plurality of predefined, composable functions 110 support a range of different automated assistant behaviors, such as calling APIs, answering questions, retrieving user data, and / or utilizing historical context from a context-specific dialogue history 130 maintained for the automated assistant. Figure 1A The context-specific dialogue history 130, indicated by the dashed arrow, can include any suitable aspect of the dialogue, such as user utterances 102, data flow procedures 106, and / or the received assistant responses 120.

[0019] Therefore, user utterance 102, the data stream procedure 106 generated in response to user utterance 102, and / or any relevant assistant response 120 of the automated assistant can be stored in a context-specific dialogue history 130. Thus, the context-specific dialogue history 130 defines multiple concepts (e.g., concept 130A, and any suitable number of concepts including concept 130N). The term "concept" is used herein to refer to any relevant or potentially relevant aspect of the interaction between the user and the automated assistant. For example, a concept may include entities (e.g., person, place, thing, number, date), the intent of a user query (e.g., the intent to order food, the intent to check the weather, the intent to schedule a meeting), actions performed by the automated assistant (e.g., ordering food, checking the weather, calling an API, finding information related to an entity, recognizing a specific user utterance, performing a composite action consisting of more than one action), or any other suitable characteristic. Concepts can be defined in any suitable manner, such as based on the text content of user utterance 102. In some examples, concept 130A is defined according to data stream procedure fragment 132A. For example, data flow program fragment 132A may include one or more functions configured to look up information related to a specific entity, and / or one or more functions configured to cause the automation assistant to perform a specific action.

[0020] In some examples, multiple predefined functions 110 include one or more history access functions configured to access a context-specific dialogue history 130. Therefore, the dataflow procedure 106 may include such a history access function. For example, both the multiple predefined functions 110 and the dataflow procedure 106 include a history access function 112 configured to access a context-specific dialogue history 130, as indicated by the arrows. In some examples, the multiple predefined functions 110 include a search history function configured to look up concepts (e.g., the getSalient() function, configured to look up previously discussed entities or previously performed actions) from the context-specific dialogue history 130, such as those related to... Figure 1B , 2 As discussed further in 3A-3F. In some examples, multiple predefined functions 110 include program rewriting functions parameterized by a specified concept stored in the context-specific dialogue history and configured to generate new dataflow program fragments related to the specified concept based on the concept from the context-specific dialogue history 130 (e.g., the Clobber() function, configured to generate new rewritten programs based on the specified concept, e.g., to write new programs for performing previous actions about new entities or for performing new actions about previously discussed entities). In some examples, the specified concept stored in the context-specific dialogue history includes a target sub-concept, and the program rewriting function is also parameterized by a replacement sub-concept for replacing the target sub-concept. Thus, the new dataflow program fragment corresponds to a specified concept that replaces the target sub-concept by replacing the replacement sub-concept. Specifically, in some examples, the new dataflow procedure fragment includes a rewritten dataflow procedure fragment based on a historical dataflow procedure fragment corresponding to the specified concept, wherein the subroutine fragment corresponding to the target subconcept is replaced by a different subroutine fragment corresponding to the replacement subconcept. (The remaining text appears to be incomplete and requires further context.) Figure 1C , 2 The program rewriting function will be discussed further with 3F-3G.

[0021] By searching and / or rewriting concepts from the context-specific dialogue history 130, the automated assistant can repeat actions, perform modified actions, search for related entities, or other details related to previous actions. For example, the automated assistant can repeat actions or perform modified actions by rerunning code based on a program fragment 132A corresponding to a concept 130A from the context-specific dialogue history 130. Furthermore, if any error condition is encountered during the execution of the data stream program 106 before the output assistant response 120, the data stream program 106 and / or other program fragments stored in the context-specific dialogue history 130 can be rerun (e.g., modified or unmodified) to recover from the error, such as regarding... Figure 1D , 4 Further descriptions of 5A-5H.

[0022] therefore, Figure 1B Different views of the automated assistant system 100 focusing on a search history function 112 among multiple predefined functions 110 are shown. The search history function 112 is configured to handle ambiguous user utterances 102 by resolving any ambiguity using concepts from a context-specific dialogue history 130. As indicated by the dashed arrows leading from the context-specific dialogue history 130 to the search history function 112, the search history function 112 is configured to access one or more concepts from the context-specific dialogue history 130 to determine a disambiguation concept 134.

[0023] In some examples, the disambiguation concept 134 is defined by the program fragment 136. As an example, if user utterance 102 refers to an ambiguous entity (e.g., by a pronoun or part of a name, such as "Tom"), the search history function 112 can be configured to search the ambiguous entity (e.g., based on the part of the name "Tom") within a context-specific dialogue history 130 to find a clarifying entity that matches the ambiguous entity (e.g., an entity with a related name such as "Tom" or "Thomas"). In some examples, the clarifying entity may be defined by its full name (e.g., "Thomas Jones"). In other examples, the clarifying entity may be defined by code configured to look up the clarifying entity (e.g., code that looks up a person named "Tom" in the user's address book).

[0024] In some examples, disambiguation concept 134 indicates a bridging reference between a user's utterance and a previous user utterance from the context-specific dialogue history. As an example, if a user asks, "When will I have lunch with Charles?" and then follows up with, "How long will it take to get there?", the word "there" in the subsequent user utterance refers to the location of having lunch with Charles. Therefore, search history function 112 can be configured to search for concepts corresponding to the location of having lunch with Charles. For example, the concept may include a dataflow procedure that recursively uses search history function 112 to find salient events (i.e., meetings with Charles), and further instructions for obtaining the location of the salient event. More generally, the concept found by search history function 112 may include a dataflow procedure that recursively calls search history function 112 to find any suitable sub-concepts, for example, to define a salient concept based on the search for other salient sub-concepts.

[0025] Regarding Figure 2 and Figures 3A-3C Let's further describe the search history function 112.

[0026] Figure 1C Another different view of the automated assistant system 100, focusing on the program rewriting function 112', is shown. The program rewriting function 112' is configured to generate a rewriting concept 140 (e.g., defined by the rewriting program fragment 142) starting with a specified concept 152 from a context-specific dialogue history 130, wherein the specified concept 152 includes at least one alternative target subconcept 154 to be modified / replaced using different alternative subconcepts 156. As an example, the specified concept 152 may refer to an action of the automated assistant, including "making a reservation at a local sushi restaurant." Thus, the target subconcept 154 may be a description of "sushi restaurant," and the alternative subconcept 156 may be an alternative description of "burrito restaurant." Specifically, the specified concept 152 may be defined by a program fragment for calling an API to make a restaurant reservation, wherein the API is parameterized by a program fragment corresponding to the target subconcept 154 indicating the restaurant to be reserved at. Therefore, the program rewrite function 112' is configured to output a rewrite concept 140 corresponding to "make a reservation at the local burrito restaurant," so that the rewrite concept 140 can be used to perform a new action consisting of making a reservation at the local burrito restaurant (e.g., by running program fragment 142). [The remaining text appears to be unrelated and possibly machine-generated.] Figure 2 and Figure 3C Further describe the rewritten function 112'.

[0027] Figure 1DThis illustrates another view of the automation assistant system 100 focusing on handling errors that may occur during the execution of the data flow program 106. Although in Figure 1D Not shown in, but as in Figure 1A-1C As shown, user utterances 102, data flow procedures 106, and / or assistant responses 120 can be stored in context-specific dialogue history 130 while processing user utterances.

[0028] If an error condition is reached during the execution of the data stream program 106 before outputting the assistant response 120, the data stream program can be paused and saved as a paused execution 160. The error handling executor 170 is configured to resume from the error condition in order to generate the assistant response 120'. For example, the error handling executor 170 can implement method 400, as described below regarding... Figure 4 Method 400 is described. To recover from the error condition, the error handling executor 170 may modify and / or re-execute the suspended run 160. Alternatively or additionally, the error handling executor 170 may run an alternative program fragment 180. For example, the error handling executor 170 may modify the suspended run 160 by using an alternative program fragment 180 to replace a program fragment of the dataflow program 106 with the alternative program fragment 180 using a program rewrite function, and / or by running the alternative program fragment instead of the dataflow program 106. The alternative program fragment 180 may be derived from context-specific dialogue history 130 (e.g., the alternative program fragment 180 may be a previously run program fragment), constructed according to multiple predefined functions 110, and / or output by a previously trained code generator 104. In some examples, the previously trained code generator 104 is configured to identify error conditions, and the output is configured to be a new program fragment for recovering from the error. For example, the previously trained code generator 104 may be trained with respect to one or more training examples, each training example including an exemplary error condition and an exemplary dataflow program fragment for responding to the error. For example, the previously trained code generator 104 can be trained on a large amount of annotated dialogue history (described below), wherein some or all of the annotated dialogue history includes the occurrence of error conditions.

[0029] In some examples, the error condition may arise due to ambiguity in user utterance 102, where the information provided by user utterance 102 is insufficient to fully serve the user based on the user utterance. As an example, if the user utterance is "Schedule a meeting with Tom", but there is more than one "Tom" in the user's address book, it may be unclear who to schedule the meeting with. Therefore, in some examples, error handling executor 170 is configured to run code to generate an initial assistant response 120' containing a clarifying question 122. The clarifying question 122 is output for the user to respond to the new, clarifying user utterance 102. Thus, the previously trained code generator 104 can be used to process the clarifying user utterance to generate a new dataflow procedure 106 for responding to the clarifying user utterance. Error recovery containing the clarifying question will be discussed below. Figure 4 and Figure 5A -5H is discussed further. The previously trained code generator 104 can be trained via supervised training on multiple annotated dialogue histories. In many examples, the previously trained code generator 104 is trained on a large number of annotated dialogue histories (e.g., hundreds, thousands, tens of thousands, or more). The annotated dialogue history describes state information associated with a user interacting with the automation assistant, annotated with exemplary dataflow procedures used to respond to said user interactions. For example, the annotated dialogue history may include information according to this disclosure (as described below). Figures 3A-3C and Figures 5A-5F The context-specific dialogue history described herein utilizes an exemplary dataflow program suitable for being run by an automation assistant in response to a context established within the context-specific dialogue history. In the example, the context-specific dialogue history includes multiple events arranged chronologically (e.g., timestamped events), including user utterances, dataflow programs run by the automation assistant, responses output by the automation assistant, and / or error conditions reached while running the dataflow program.

[0030] As a non-limiting example, the annotated dialogue history may include a context-specific dialogue history, where the most recent event is an exemplary user utterance annotated with an exemplary dataflow procedure for responding to the exemplary user utterance with respect to the context established by the context-specific dialogue history. Therefore, a previously trained code generator 104 can be trained to reproduce the exemplary dataflow procedure given an exemplary user utterance and a context-specific dialogue history. The exemplary dataflow procedure may contain any suitable functions in any suitable sequence / arrangement, thereby, through training, the code generator is configured to output suitable functions in a suitable sequence. For example, the exemplary dataflow procedure may include a search history function, and therefore, the code generator can be trained to output the search history function and other functions in a suitable sequence for responding to user utterances (e.g., to cause an automation assistant to perform any suitable response action, such as outputting the response as text and / or speech, or calling an API). In some examples, the annotated dialogue history includes a context-specific dialogue history where the most recent event is the occurrence of an error condition (e.g., not a recent user utterance), annotated with a dataflow procedure for recovery from the error. Therefore, the previously trained code generator 104 can be trained with respect to such an annotated dialogue history to generate a suitable data flow procedure for recovery from error conditions.

[0031] Annotated dialogue history can be obtained, for example, from a human presenter in any suitable manner. For instance, a context-specific dialogue history (e.g., a context-specific dialogue history derived from usage data obtained from human interaction, and / or a machine-generated context-specific dialogue history) can be shown to one or more human presenters, and for each context-specific dialogue history, a suitable dataflow procedure is required to respond to said context-specific dialogue history. The dataflow procedure provided by the human presenter can perform a wide range of tasks in any suitable manner using predefined functions in response to user utterances and / or error conditions.

[0032] For example, based on shown exemplary user utterances or exemplary error conditions, a human presenter can provide exemplary dataflow programs that perform any appropriate computations, output responses (e.g., ask the user a clarifying question, or answer a query issued by the user), listen to utterances from the user (e.g., obtain clarification from the user), call APIs, etc. Furthermore, the exemplary dataflow program may include a search history function (e.g., “getSalient()”) and / or a program rewriting function (e.g., “Clobber()”) that is invoked with any appropriate parameters, for example, to execute a dataflow program fragment based on the context-specific dialogue history. Therefore, by training on multiple annotated dialogue histories, the code generator 104 can be trained to generate dataflow programs similar to those provided by the human presenter, in order to respond to user utterances and / or recover from errors.

[0033] The dataflow program (e.g., a dataflow program generated by a previously trained code generator and / or an exemplary dataflow program) is constructed based on a plurality of predefined composable functions. These predefined composable functions can be combined into a program that calls the predefined composable functions in any suitable order and parameterizes them in any suitable manner. Therefore, based on the exemplary dataflow program provided during training, the previously trained code generator can be trained to output a suitable dataflow program for user utterances. The previously trained code generator is not limited to hard-coded behavior. For example, instead of responding to the exemplary user utterances seen during training, the previously trained code generator is configured to process novel user utterances (which may also not have been provided during training) by generating corresponding, novel dataflow programs (which may not have been provided during training).

[0034] In order to generalize from specific training examples seen during training and to respond to novel user discourse using novel dataflow procedures, this can be done in any suitable manner (e.g., as will be discussed below). Figure 6As described, the previously trained code generator is trained on any suitable training data (e.g., a large amount of annotated dialogue history, using any suitable ML, AI, and / or NLP model). In some examples, the previously trained code generator can be trained with respect to a loss function to evaluate whether the dataflow procedure is a suitable response to user utterances, wherein the loss function is configured to indicate zero or relatively small loss (e.g., no adjustment required, or relatively small adjustment to the training parameters) when the code generator successfully reproduces the training example (e.g., by generating the same dataflow procedure as provided by a human annotator given a context-specific dialogue history). However, while the loss function can be configured for relatively small loss when the code generator successfully reproduces the training example, it can also be configured for relatively small loss when the code generator reproduces different dataflow procedures, e.g., when a dataflow procedure with similar effects is run, and / or a dataflow procedure indicated as satisfactory by a human user (e.g., a human presenter and / or the end user of an automated assistant device). These exemplary schemes used for generalization during training are non-limiting, and any suitable AI, ML, and / or NLP techniques can be used to properly train the code generator to generate appropriate dataflow programs in response to a variety of user utterances.

[0035] In some examples, error conditions can be identified by manipulating a previously trained error detection model. For instance, the previously trained error detection model can be trained via supervised training on multiple annotated dialogue histories, where the annotated dialogue histories are annotated to indicate when an error occurred. For example, the annotated dialogue history can be obtained by showing one or more human presenters a context-specific dialogue history and asking the human presenters to indicate when the context-specific dialogue history indicated an error state.

[0036] In addition to history access functions such as search history functions and program rewrite functions, the plurality of predefined composable functions may include any suitable functions, such as a listener function configured to listen for a specific user utterance before continuing execution of the data stream program, a response function configured to output a description of a value determined during the execution of the data stream program, and / or a primitive computation function for processing values ​​obtained from user utterances and / or values ​​computed during the execution of the data stream program (e.g., data structure operations, such as forming tuples from data, or arithmetic operations).

[0037] In some examples, the plurality of predefined composable functions include external functions configured to call external (i.e., third-party) APIs. For example, external APIs can be called to interact with real-world services (e.g., arranging a car in a ride-hailing service, ordering food, or making a restaurant reservation). In some examples, the plurality of predefined composable functions include inference functions configured to perform calculations about the results of the external functions. The inference function can encapsulate high-level behavior about the API that would otherwise require the use of multiple different low-level functions of the API. For example, an external ride-hailing API might support functions for arranging a car, adding stops along the route, and ultimately determining the route to be arranged. Therefore, the inference function can be configured to: receive a destination and arrange a car based on that destination, add stops for the user corresponding to the pick-up location, add stops corresponding to the destination, and ultimately determine the route including the stops corresponding to the pick-up location and the destination. By encapsulating high-level behavior using inference functions, the code generator can easily output well-typed code for performing high-level behavior using external APIs without needing to output individual steps using the low-level functions of the external APIs. In some examples, an inference function may be defined with respect to one or more constraints, and running the inference function may include running a constraint-satisfying program to satisfy the one or more constraints, and then calling an external API using parameters defined by the solution for the constraints. In some examples, the one or more constraints may include “fuzzy” or “soft” constraints, and therefore, resolving the constraints may include running an inference program suitable for “fuzzy” logic inference, such as a Markov logic inference program.

[0038] In some examples, the plurality of predefined composable functions includes user-customized functions configured to access user-customized settings and perform calculations based on those settings. For example, the user-customized function could be configured to determine whether a user is available based on a user-customized schedule (e.g., calendar data). The user-customized function can be implemented using an external function configured to call an external API (e.g., an API for retrieving calendar data).

[0039] In some examples, the plurality of predefined composable functions include intelligent decision functions, wherein the intelligent decision functions are configured to perform computations using a previously trained machine learning model. As an example, the search history function could be an intelligent decision function configured to search the context-specific dialogue history using a previously trained relevance detector. As another example, the plurality of predefined composable functions could include an intelligent decision function configured to evaluate whether it is “morning” in a user-specific and / or crowd-specific manner. For example, the function could be configured to recognize that the time a user considers “morning” may vary depending on the day of the week or the time of year. For instance, the intelligent decision function could be configured to evaluate “morning” between 6 AM and 11 AM on weekdays, but could be configured to evaluate morning between 9 AM and 12 PM on weekends. The intelligent decision function can be trained in any suitable manner, such as based on labeled time examples and whether the user considers the time to be morning. Similar to user-customized functions, the intelligent decision function may consider auxiliary information such as the user's work schedule, calendar, and / or mobile phone usage, for example, to determine whether it is "morning" based on whether the user may have woken up on a given day. In some examples, the intelligent decision function may be configured to evaluate ambiguity (e.g., ambiguous user utterances or ambiguous constraints) and select a disambiguation data stream procedure to respond to ambiguity.

[0040] In some examples, the plurality of predefined composable functions include macro functions, wherein the macro functions include a plurality of other predefined composable functions and are configured to run the plurality of other predefined composable functions. For example, macro functions can be used to sort and organize related low-level steps of a high-level task using low-level functions. By using macro functions to encapsulate high-level behavior, the code generator can easily output well-typed code for performing the high-level behavior without using low-level functions to output individual steps.

[0041] Briefly switch to Figure 1E , Figure 1E A first example of a dialogue between a user and an automated assistant is shown, in which the automated assistant processes unambiguous user utterance 102'. A context-specific dialogue history 130 is shown as including concepts 130A, 130B, and additional concepts up to 130N. However, in this example, user utterance 102' is unambiguous and can be processed without reference to the context-specific dialogue history 130. The user asks, "When is my next meeting with Tom Jones?" and receives the output dataflow program 106' from a previously trained code generator.

[0042] Dataflow program 106' is illustrated with a non-restrictive example syntax, where square brackets indicate the return value of an expression; for example, [events] indicates the return value of an expression used to find all events that match a standard set, and [time] indicates the start time of the first such event in said set. The exemplary syntax includes various functions, including search history functions (e.g., “getSalient()”) and program rewrite functions (e.g., “Clobber()”), as well as other functions (such as primitive functions, API functions, etc., as described herein). The exemplary functions are illustrated with function call syntax indicating function names (e.g., “Find”, “getSalient”, “Clobber”, and other named functions) and parentheses containing parameters for calling said functions. The exemplary function call syntax is non-restrictive, and functions can be called and parameterized in any suitable manner (e.g., using any suitable formal language syntax). Exemplary function names represent predefined functions with implementations not shown herein. For example, each predefined function can be implemented by any suitable sequence of one or more instructions operable by the automation assistant to perform any suitable step (e.g., to produce the behavior indicated by the function name, the behavior described in this disclosure, and / or any other suitable behavior, such as performing a calculation, calling an API, outputting audio, visually presenting information via a display, etc.). For example, the “Find” function can be implemented in any suitable manner, for example, by calling an API to find information stored in the user’s calendar.

[0043] As shown, the dataflow program looks for events with a participant named "Tom Jones". The dataflow program calculates values ​​such as [events] and [time], and then outputs a response 120' using a "description" function configured to output a description of the value [time], for example, as voice from the automated assistant device's speaker. Therefore, response 120' indicates the next meeting time with Tom Jones, i.e., "at 12:30".

[0044] Figure 2 An exemplary method 200 for processing ambiguous user utterances is illustrated. The method uses a search history function to leverage information from context-specific dialogue history to resolve the ambiguity. (As in...) Figure 3A As shown, the context-specific dialogue history may include any suitable information tracked for the dialogue, such as concept 334 indicating previous user utterances 304 and concept 324 indicating previously executed data stream program segments 326.

[0045] At point 202, method 200 includes identifying ambiguous user utterances. For example, Figure 3A An exemplary ambiguous user utterance 302 is shown, in which a user asks, “What comes after that?” The utterance itself does not provide enough information to respond to the user, for example, because “that” does not refer to any specific event without further context.

[0046] In some examples, at point 204, method 200 includes identifying constraints related to ambiguity. For example, returning... Figure 3A The utterance includes the phrase "after that," and therefore, method 200 includes identifying constraints that the user is referring to regarding the relevant time. For example, the previously trained code generator may be trained on one or more training examples in which the user's utterance includes time-related words, such as "after" when the user is referring to time. For example, the time may be the start time defined by a schedule for an arranged event, such as an event scheduled in the user's calendar.

[0047] Therefore, let's briefly return to... Figure 2 At 206, method 200 includes using the previously trained code generator to generate a data stream procedure for responding to the user utterance, wherein the data stream procedure includes the search history function, which in Figure 3A The exemplary syntax shown is referred to as "getSalient". The search history function can be run to retrieve relevant information from the context-specific dialogue history to resolve the ambiguity. As described at 208, the search history function is configured to select the highest-confidence disambiguation concept from one or more concepts in the context-specific dialogue history to resolve the ambiguity using the highest-confidence disambiguation concept.

[0048] For example, in Figure 3A In the context of a user discussing their meeting with Tom Jones, the context-specific dialogue history 130 includes a concept 334 that includes a previous user utterance 304 in which the user asks, “When is my next meeting with Tom Jones?” and a data stream procedure fragment 326 that was previously executed in response to the previous user utterance 304 (e.g., as in...). Figure 1EThe dataflow procedure 322 (shown as '106') defines concept 324. Dataflow procedure 322 is configured to: determine a specific event (e.g., the next meeting with Tom), determine the start time for said event, and describe said time. Therefore, the search history function "getSalient" is configured to search for relevant times from the context-specific dialogue history to find events that occurred after that time and output a response 330 indicating what event occurred after that time. The relevant time can be a [time] value defined in dataflow procedure 326 related to the meeting with Tom. Therefore, response 330 indicates that there will be another meeting with Richard Brown after the meeting with Tom.

[0049] Return to Figure 2 In some examples, as described at 210, the search history function is configured to search within the context-specific dialogue history for a subset of one or more candidate concepts that satisfy constraints related to the ambiguity, wherein the disambiguated concept with the highest confidence is found within that subset. For example, in Figure 3A In the context of the utterance 302, which includes the phrase "after that," the identified constraints may include constraints on the relevant time defined by the timeline of the scheduled event (as described above), and therefore the relevant concepts from the context-specific dialogue history 130 should be values ​​of type "time." Thus, the search history function "getSalient(Time())" is configured to search for a subset of relevant values ​​of type "time" from the context-specific dialogue history 130, for example, including [time] values.

[0050] In some examples, such as those described at 212, the search history function is configured to use a previously trained relevance detector to identify the constraints of the ambiguity, and to select a disambiguation data stream procedure fragment corresponding to the disambiguation concept based on such identified constraints. For example, as in Figure 3A As shown, based on the "time" type constraint, data stream procedure fragment 326 can be selected for disambiguation. The previously trained correlation detector can include any suitable combination of existing and / or future ML, AI, and / or NLP technologies.

[0051] In some examples, the previously trained relevance detector can be trained via supervised training over multiple annotated dialogue histories, where the annotated dialogue histories include unresolved search history functions labeled with disambiguation concepts that will resolve the unresolved search history functions. Therefore, given an unresolved search history function, the previously trained relevance detector can be trained to predict an appropriate disambiguation concept. For example, the annotated dialogue history may include dataflow snippets using the search history functions (e.g., dataflow snippet 322 including "getSalient") and exemplary disambiguation concepts that will resolve the ambiguity (e.g., concept 324 including dataflow snippet 326 defining [time] = [vents][0].start).

[0052] In some examples, the annotated dialogue history can be provided by a human presenter, who may be shown ambiguous user utterances and corresponding dataflow procedures including the search history function, and be asked to provide exemplary dataflow procedure fragments that appropriately match the search history function within the context of the ambiguous user utterances. In some examples, the human presenter selects a disambiguation concept from the context-specific dialogue history that will resolve the unresolved search history function. In some examples, the disambiguation concept is an exemplary dataflow procedure fragment received from the human presenter. For example, the human presenter may be asked to provide the dataflow procedure fragment by combining one or more composable functions from the predefined composable functions and / or one or more dataflow procedure fragments from the context-specific dialogue history. For example, the human presenter may be asked to select one or more disambiguation concepts from the context-specific dialogue history using a graphical user interface (GUI), and / or to combine new procedures using such concepts and predefined composable functions selectable from menus. In some examples, the disambiguation concept is independent of the context-specific dialogue history. For instance, a human presenter might instruct that the salient date used to resolve a search history function constrained to find a date is "today," regardless of the concept in the context-specific dialogue history.

[0053] After generating the data stream procedure including the search history function, at 214, method 200 optionally further includes running the data stream procedure. Therefore, at 216, method 200 optionally further includes outputting the response obtained by running the data stream procedure. For example, as in... Figure 3AAs shown, a data stream procedure 322 can be run, including running the search history function to search for significant "time" type values ​​from the history to determine the value [time2], and to find events that begin after [time2] (e.g., by calling an API to access the user's calendar). Therefore, a response 330 can be output to tell the user, "After your meeting with Tom, you have a meeting with Richard Brown, starting at 1:30."

[0054] Figure 3B-3C This illustrates another example of resolving ambiguous user utterances. For example, in Figure 3B As shown, in some examples, the constraints on the ambiguity indicate the ambiguous entity and the entity attributes of the ambiguous entity, and a subset of one or more candidate concepts includes only those candidate entities from the context-specific dialogue history that have the entity attributes. For example, in user utterance 302', the user requests "arrange a meeting with him at 2:30". The descriptor "he" may be ambiguous without further context because it could refer to anyone using the male pronoun "he". However, the context-specific dialogue history 130 includes potentially relevant entities, including Tom Jones indicated by concept 350A and Jane Smith indicated by concept 350B. As described above (and not shown in detail here), concepts 350A and 350B can be represented in any suitable form; for example, concept 350A could correspond to a data stream procedure fragment used to access a user's address book using an API to look up an individual named "Tom Jones". Based on the user's utterance, the data flow procedure 322' is configured to search for salient individuals based on the attribute containing a male pronoun ("he"). Therefore, if the history includes relevant entities "Tom Jones" and "Jane Smith," and "Tom Jones" uses a male pronoun while "Jane Smith" uses a female pronoun, then the search history function ("getSalient") can find the more relevant entity "Tom Jones." Thus, in response to the 360 ​​instruction, the meeting is scheduled for 2:30 and Tom Jones is being invited.

[0055] In some examples, such as in Figure 3CAs shown, the constraints indicate ambiguous references to actions performed by the automated assistant and constraint attributes associated with those actions. Therefore, a subset of the one or more candidate concepts includes multiple candidate actions with constraint attributes, where each candidate action is defined by a candidate dataflow procedure fragment. For example, user utterance 302" indicates that the user wishes to alternatively invite "Jane," but user utterance 302" is ambiguous regarding what event Jane is invited to. Therefore, the constraint attribute associated with the action is that the action is related to inviting an individual to an event. Thus, there may be related actions performed by the automated assistant, which are stored as concept 306 in the context-specific dialogue history 130. As shown, concept 306 includes a dataflow procedure 308 corresponding to the action performed by the assistant, for example, with... Figure 3B The code corresponding to the meeting set in the interaction shown. The data stream procedure 308 for resolving the ambiguity is configured to use the search history function (“getSalienf”) to search for relevant events.

[0056] In some examples, the concepts indicated by the user in their speech may be based on external and / or contextual attributes and / or on the source of the concept within the conversation between the user and the automated assistant. As an example, the user's speech may refer to a meeting event by mentioning "second meeting," which could refer to a meeting via external and / or contextual attributes, such as a second meeting on the user's schedule. Alternatively or additionally, "second meeting" could refer to a second meeting discussed in the conversation between the user and the automated assistant. As a result of supervised training in this paper, the code generator can be configured to appropriately recognize from the context what the user is referring to when using phrases such as "second meeting."

[0057] In some examples, as in Figure 3CAs shown, the dataflow procedure 322 includes a procedure rewriting function (referred to as "Clobber" in the exemplary syntax) parameterized by a designated concept ("[designated]") stored in a context-specific dialogue history and configured to generate new dataflow procedure fragments related to the designated concept based on concepts from the context-specific dialogue history 130 (e.g., writing new procedures to perform previous actions about new entities). In some examples, the designated concept ("[designated]", i.e., the arrangement of a new meeting with Tom Jones) includes a target subconcept ("replacementTarget", i.e., the invitee of the event, Tom Jones). Therefore, the procedure rewriting function is further parameterized by a replacement subconcept used to replace the target subconcept; that is, "[replacing]" indicates the new invitee for the meeting, Jane. Thus, the new dataflow procedure fragment corresponds to the designated concept in which the target subconcept is replaced using the replacement subconcept. In other words, the procedure rewriting function produces a new dataflow procedure configured to invite Jane Smith to a previous meeting with Tom Jones. The same event set by Jones. Specifically, in some examples, the new dataflow procedure fragment includes a rewritten dataflow procedure fragment based on a historical dataflow procedure fragment corresponding to the specified concept, wherein the subroutine fragment corresponding to the target subconcept is replaced by a different subroutine fragment corresponding to the replacement subconcept.

[0058] The program rewriting function can be parameterized in any suitable manner. For example, the specified concept, the replacement target subconcept, and the replacement subconcept can be derived from context-specific dialogue history 130 (e.g., the alternative program fragment 180 could be a previously executed program fragment), constructed from multiple predefined functions 110, and / or output by a previously trained code generator 104. For example, the previously trained code generator 104 can be trained with respect to one or more training examples, which include an exemplary program rewriting function and exemplary dataflow program fragments for each parameter of the program rewriting function. In some examples, the previously trained code generator is trained with respect to a large number of different training examples that include the program rewriting function.

[0059] In the examples above, the context-specific dialogue history is used to process the user utterance, resulting in an output automated assistant response. However, in some examples, errors may occur during the processing of the user utterance. However, the dataflow procedure according to this disclosure can be configured for condition-free evaluation of one or more data values ​​including return values. Therefore, processing the dataflow procedure may include running the dataflow procedure to obtain the return values. However, although the dataflow procedure can be configured for condition-free evaluation of the data values ​​including the return values, errors may occur during the processing of the dataflow procedure, such as due to ambiguity in the user utterance, which may prevent the dataflow procedure from being fully resolved based on the user utterance. Therefore, in response to detecting any error condition while running the dataflow procedure, the execution of the dataflow procedure can be paused. After pausing the execution of the dataflow procedure, the previously trained code generator can be used to generate an alternative error-handling dataflow procedure based on the paused execution of the dataflow procedure. Therefore, the alternative error-handling dataflow procedure can be run to recover from the error condition.

[0060] Figure 4 An exemplary method 400 for processing user utterances by an automated assistant is shown when errors may occur during such processing. At 402, method 400 includes identifying the user utterances used for processing. For example, Figure 5A An exemplary dialogue including user utterance 502 is shown, in which the user asks, “Who is Dan’s manager?”. As in the previous example, when processing the user utterance, the automated assistant maintains a context-specific dialogue history 130, which is used to track concepts related to previous interactions between the user and the automated assistant, such as concepts 540A, 540B, and 540C, which are not shown in detail in this example.

[0061] At 404, method 400 includes using a previously trained code generator to generate a data stream procedure based on the user utterance (e.g., ...). Figure 5A The data flow procedure shown is 522. This data flow procedure is configured to produce a return value upon successful execution, for example, a description of Dan's manager. Therefore, the data flow procedure includes calculating a first value [rl], configured as a list of all relevant individuals named, for example, "Dan," where the expected list of results is a single-item list containing exactly one entry, for example, a unique result for an individual named, for example, "Dan." Assuming a single individual named "Dan" is found and saved as [rl], the data flow procedure also includes calculating another value [r2] by finding the manager of [rl] and describing the result.

[0062] At 405, method 400 includes running the data stream program. Therefore, at 406, method 400 includes initiating the execution of the data stream program. Briefly back to... Figure 5A The evaluation [rl] includes, for example, searching for an individual named "Dan" by accessing context-specific conversation history 130 and / or accessing the user's address book. However, when evaluating [rl], error condition 532 is reached because there is more than one individual named, for example, "Dan," namely "Dan A" and "Dan B," so the list of people named, for example, "Dan" is not a single item. Therefore, this discrepancy is detected as error condition 532, which includes a data stream procedure fragment describing the event in which the error condition was detected. The error event indicates that the error is due to ambiguity.

[0063] At 408, in response to an error condition resulting from the execution of the data stream program, method 400 includes handling the error condition. Handling the error condition includes: at 410, pausing execution of the data stream program (e.g., as in...) before the data stream program generates the return value, and before calculating and describing the return value (e.g., return value [r2]). Figure 5A The data flow program 522 shown is executed. To resolve the error condition (e.g., error condition 532), the automation assistant is configured to describe the error, for example, to obtain disambiguation information from the user. Thus, optionally at 418, processing the error condition may include outputting a clarifying question and identifying clarifying user utterances, for example, by looping back to 402 to identify further user utterances. Figure 5A As shown, the automated assistant output includes a response 534 that clarifies the question "Do you mean Dan A or Dan B?".

[0064] Now go to Figure 5B After pausing the execution of the data stream procedure and outputting the response including the clarification question, the automation assistant can receive subsequent user utterances to obtain disambiguation information for handling the error. Therefore, the erroneous data stream procedure 522 and error condition 532 are (respectively) stored as concepts 520 and 530 in the context-specific dialogue history, while processing the new user utterance 502' indicating "Dan A".

[0065] Therefore, at 412, method 400 further includes using the previously trained code generator to generate an error handling data stream procedure, wherein the error handling data stream procedure 552 is configured to produce the return value (e.g., the error handling data stream procedure 552 is an alternative means of calculating the expected value corresponding to [r2].

[0066] In some examples, reaching the error condition includes detecting an error when a problematic program segment is running the data stream program, for example, in Figure 5A The error condition shown occurs when evaluating a problematic code snippet that defines the value [rl], namely, "Find(Person(name:like("Dan"))).results.singleton()". Therefore, generating the error handling dataflow routine involves outputting a new dataflow routine based on the problematic code snippet [rl]. As shown in the example, error handling dataflow routine 552 uses the search history function ("getSalient") to find a relevant person named "Dan A" and saves the result as [r3]. In some examples, the dataflow routine may include an "intensionOf" function configured to return an "intent", which herein refers to a reference to a dataflow routine snippet. Error handling dataflow routine 552 uses the dataflow routine snippet corresponding to the [error] event, namely the "intensionOf[rl]" defined in the dataflow routine snippet of error condition 532.

[0067] As in Figure 5B As shown, the new dataflow procedure based on the problematic program fragment includes a program rewrite function (e.g., "Clobber") configured to replace the problematic program fragment with an alternative program fragment in the dataflow procedure. The error handling dataflow procedure 552 defines a new computation [newCompl] by using the program rewrite function ("Clobber") to rewrite the original return value [r2] defined by [r4] by replacing the target subconcept (i.e. the problematic program fragment that caused the error) with a different subconcept [r3].

[0068] In some examples, such as in Figure 5B As shown, the alternative program fragment includes a search history function configured to search for the alternative program fragment in the context-specific dialogue history based on its relationship to the problematic program fragment. Therefore, the sub-concept [r3] is defined by code used to find a relevant individual named "Dan A" using the search history function.

[0069] At 414, method 400 includes initiating execution of the error handling dataflow procedure to produce a return value. For example, error handling dataflow procedure 522 is configured to return the value [r2'] by running a new computation [newCompl]. At 420, method 400 includes outputting the return value. After outputting the return value, at 422, method 400 further includes outputting a response based on the return value, for example, Figure 5BThe example shows response 514, which states that "Dan's manager is Tom Jones".

[0070] In some examples, running a dataflow procedure can result in more than one different error condition being reached. For example, after a first error condition is reached and fully resolved, another error condition may be reached. Alternatively or additionally, the automation assistant can be configured to detect more than one simultaneous error condition and handle any such error condition simultaneously by generating an error handling dataflow procedure for all such error conditions.

[0071] In some examples, method 400 includes identifying additional errors during the execution of the error handling data stream procedure and running an error recovery loop, including generating and running additional error handling data stream procedures until one of the additional error handling data stream procedures produces the return value. Therefore, handling the error condition in response to reaching the error condition at 408 includes returning to 408 in response to detecting an additional error condition at 416 to handle such an additional error condition. By sequentially detecting each error and generating an error handling data stream procedure to recover from one or more errors, any number of errors can be resolved sequentially and / or simultaneously.

[0072] Figure 5C Examples of handling two different error conditions are shown. Figure 5C This is shown as a series of user utterances and responses, and the context-specific dialogue history and data flow procedures associated with the utterances / responses are not shown. In the first user utterance 504, the user requests to arrange a meeting with Dan and Tom. However, there may be more than one associated Dan, and similarly, there may be more than one associated Tom. Therefore, similar to... Figures 5A-5B As shown, the automation assistant is configured to output response 534', which asks the user which Dan they wish to meet with. The user responds in user statement 506, meaning "Dan A". The automation assistant is then configured to output a subsequent response 536, which asks the user which Tom they wish to meet with. The user responds in user statement 508, meaning "Tom J". Therefore, after "Dan" and "Tom" are deambiguously resolved, the automation assistant is configured to schedule the meeting and output a response 538 indicating that the meeting has been scheduled.

[0073] In some examples, such as in Figure 5DAs shown, in response to user utterance 510, the data stream program generated by the previously trained code generator uses a search history function to specify the concept to be resolved, i.e., to find the individual [r1] by searching for salient matches of “he” mentioned in user utterance 510. However, the search history function may find two or more related concepts, as indicated by error condition 534. For example, context-specific dialogue history 130 may include a concept 530A corresponding to “Dan A” and another concept 530B corresponding to “Tom J”. For example, the concept corresponding to “Dan A” may be derived from, for example, the concept in the context of… Figure 5B The error recovery dataflow procedure shown is defined by the dataflow procedure fragment “[r2'] = Execute([newCompl])”, or by any other suitable dataflow procedure fragment such as “[r3] = getSalient(Person(name:like(“Dan A”)))”, and a concept corresponding to “Tom J” can be defined similarly. Therefore, the automation assistant is configured to output a response 536 containing a clarification question that requires the user to provide a clarification response indicating one of two or more related concepts (e.g., indicating “Dan A” or “Tom J”). The error handling dataflow procedure is also configured to receive a clarification user statement 512 and resolve the ambiguity based on the clarification statement. Therefore, after resolving the user's reference to “Dan A” in the original user statement 510, the automation assistant is configured to perform the action [r2] to dial the phone number of “Dan A” and output a response 538 describing the action.

[0074] In some examples, such as in Figure 5E As shown, the error is ambiguity, wherein the dataflow procedure uses a search history function to specify the concept to be resolved, the search history function finds zero relevant concepts, and the error handling dataflow procedure can run to output a clarification question requiring the user to: provide a clarification response indicating the relevant concepts, receive a clarification user utterance, and resolve the ambiguity based on the clarification utterance. For example, in Figure 5EIn the user's utterance 524, the user requests to "call him to confirm the meeting." The resulting data stream procedure 526 can be run to find the relevant person and call that person. However, the context-specific dialogue history 130 may not contain any concepts corresponding to the relevant person (e.g., because no male person has been discussed recently). Therefore, error condition 534 indicates that no match was found to find the salient person to calculate [rl]. Therefore, the automated assistant is configured to run an error handling data stream procedure (not shown), which includes obtaining a clarification response 582 to find out who to call and call that person (e.g., by creating a new calculation based on [r1] and / or [r2] using the program rewrite function, similar to in Figures 5A-5B (Example shown). After the user clarifies that they mean "Dan A" in user utterance 516, error handling data stream procedure 552 is configured to dial the phone number "Dan A" based on the clarification provided by the user in order to complete the same steps as the original data stream procedure 522. Therefore, the data stream procedure is configured to output a response 584 indicating that the call is being dialed.

[0075] In some examples, such as in Figure 5F As shown, the error is a lack of confirmation, where the dataflow procedure specifies an action to be performed only after receiving a user confirmation response, and the user utterance does not include such a user confirmation response. The error handling dataflow procedure can run to output a confirmation question that requires the user to provide the user confirmation response before performing the action. For example, a user requests a ride to a meeting in user utterance 518. Dataflow procedure 562 is able to find a significant meeting and determine that a ride is now available to reach the meeting, storing the obtained value in [r2]. For example, [r2] may include any suitable information related to arranging a ride, such as a resource descriptor published by the ride-hailing service API. Dataflow procedure 562 is configured to arrange a ride based on the obtained value, but only after obtaining confirmation from the user. No confirmation can be detected as error condition 554. Therefore, the error handling dataflow procedure (not shown) can be configured to output a response 556 indicating the cost of the ride and requiring user confirmation. In user utterance 520, the user indicates that they do indeed want a ride arranged. Therefore, the error handling data stream procedure can continue to schedule the ride, for example by rerunning the calculation [r3], and output a response 558 indicating that the action has been performed.

[0076] In some examples, the error handling mechanisms described herein can be used to determine the information needed to process the user's utterance by asking the user one or more follow-up questions. For example, if the user's utterance includes a request to perform a specific action parameterized by one or more constraints, and the user's utterance specifies some, but not all, of the constraint values ​​used to specify the constraints, the action may not be performed before specifying the remaining constraints. Therefore, the missing constraint parameters can be detected as errors, and accordingly, the previously trained code generator is configured to output an error handling dataflow procedure. For example, the error handling dataflow procedure can run to ask questions to guide the user to provide further information for specifying the remaining constraints, and use such further information received from the user to perform the requested action. Alternatively or additionally, the error handling dataflow procedure can run to specify one or more of the remaining constraints using default values. As an example, if the user requests "schedule a meeting with Charles next Thursday at 10 a.m.," the resulting dataflow procedure can be configured to perform the meeting arrangement parameterized by the date, time, invitee, and duration. Since the user utterance mentions "Charles," "Thursday," and "10 a.m.," constraints can be specified for the date, time, and invitee. However, constraints for the duration may not be specified, leading to an error condition. Therefore, the previously trained code generator can be configured to output an error-handling dataflow procedure to specify the remaining constraints for the duration. The error-handling dataflow procedure can be configured to ask the user a follow-up question, "How long should the meeting last?" to obtain another user utterance that can determine the meeting duration. Alternatively, the error-handling dataflow procedure can be configured to determine a default value for specifying the constraints, such as a default meeting duration of 30 minutes. The default value can be determined by the code generator based on supervised training on exemplary default values, using the search history function, using any other intelligent decision function, using a user-customized function (e.g., to access the user's default meeting duration preference), or in any other suitable manner. In some examples, the actions defined by the dataflow procedure are implemented using an API inference function, which can be parameterized by one or more constraints. Therefore, if any such parameter is not defined by user utterance, some or all of the parameters can be inferred by running a constraint-satisfying procedure. For example, for an API inference function used to schedule a meeting that requires a start time and an end time, the end time can be inferred from the start time and the duration. Similarly, the start time can be inferred from the end time and the duration.Therefore, before asking questions, assuming default values, or otherwise handling missing constraints, the error handling dataflow procedure can be configured to infer as many constraints as possible against the API inference function. In some examples, instead of generating a single default value to specify a constraint, the error handling dataflow procedure can be configured to generate multiple distinct candidate values ​​to specify the constraint and prompt the user to select a particular candidate value. In some examples, the error handling dataflow procedure can parameterize with multiple distinct candidate values ​​and run multiple distinct API calls (e.g., using the API inference function), prompting the user to select a candidate value based on the results of the API calls. For example, the error handling dataflow procedure can attempt to schedule multiple distinct meetings with different durations and prompt the user to select one of the resulting arrangements. In some examples, the error handling dataflow procedure can filter candidate options based on the results of API calls, for example, attempting to schedule multiple distinct meetings with different durations and prompting the user to choose between meetings that do not cause scheduling conflicts, while omitting candidates with durations that would cause scheduling conflicts, as indicated by the meeting scheduling API.

[0077] In some examples, the alternative program fragments are output by the previously trained code generator. Therefore, the previously trained code generator may be trained using supervised training based on multiple training examples to output the alternative program fragments, wherein the training examples include exemplary problematic program fragments that will lead to an error condition, and resolving alternative program fragments that will not lead to an error condition. In other words, the previously trained code generator may be trained in a manner similar to generating code for user utterances and generating code for responding to errors. In either case, the previously trained code generator is configured to generate programs using multiple predefined, composable functions arranged in any suitable sequence to handle user utterances and / or errors. In some examples, the previously trained code generator is trained with respect to a large number of training examples that reach an error condition and wherein the alternative program fragments can run to recover from the error, for example, by successfully generating a return value.

[0078] The methods and processes described herein can be attached to a computing system of one or more computing devices. Specifically, such methods and processes can be implemented as a runnable computer application, a network-accessible computing service, an application programming interface (API), a library, or a combination of the above and / or other computing resources.

[0079] Figure 6A simplified representation of a computing system 600 is illustrated schematically. The computing system 600 is configured to provide any to all computing functions described herein. The computing system 600 may take the form of one or more personal computers, network-accessible server computers, tablet computers, home entertainment computers, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones), virtual / augmented / mixed reality computing devices, wearable computing devices, Internet of Things (IoT) devices, embedded computing devices, and / or other computing devices.

[0080] The computing system 600 includes a logic subsystem 602 and a storage subsystem 604. The computing system 600 may optionally include an input / output subsystem 606, a communication subsystem 608, and / or... Figure 6 Other subsystems not shown.

[0081] Logical subsystem 602 includes one or more physical devices configured to execute instructions. For example, the logical subsystem may be configured to execute instructions as part of one or more applications, services, or other logical constructs. The logical subsystem may include one or more hardware processors configured to execute software instructions. Alternatively or additionally, the logical subsystem may include one or more hardware or firmware devices configured to execute hardware or firmware instructions. The processor of the logical subsystem may be single-core or multi-core, and the instructions running thereon may be configured for sequential, parallel, and / or distributed processing. Individual components of the logical subsystem may optionally be distributed across two or more separate devices that may be remotely located and / or configured for coordinated processing. Aspects of the logical subsystem may be virtualized and run by remotely accessible networked computing devices configured in a cloud computing configuration.

[0082] Storage subsystem 604 includes one or more physical devices configured to temporarily and / or permanently store computer information, such as data and instructions executable by the logical subsystem. When the storage subsystem includes two or more devices, the devices may be co-located and / or remotely located. Storage subsystem 604 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location-addressable, file-addressable, and / or content-addressable devices. Storage subsystem 604 may include removable and / or built-in devices. The state of storage subsystem 604 may be changed—for example, to store different data—when the logical subsystem executes instructions.

[0083] Various aspects of the logic subsystem 602 and the storage subsystem 604 can be integrated together into one or more hardware logic components. For example, such hardware logic components may include program-specific and application-specific integrated circuits (PASIC / ASIC), program-specific and application-specific standard products (PSSP / ASSP), system-on-a-chip (SOC), and complex programmable logic devices (CPLD).

[0084] The logical subsystem and the storage subsystem may cooperate to instantiate one or more logical machines. As used herein, the term "machine" is used collectively to refer to a combination of hardware, firmware, software, instructions, and / or any other components that cooperate to provide computer functions. In other words, "machine" is never an abstract concept and always has a tangible form. A machine may be instantiated by a single computing device, or a machine may include two or more sub-components instantiated by two or more different computing devices. In some implementations, a machine includes a local component (e.g., a software application running by a computer processor) that cooperates with a remote component (e.g., a cloud computing service provided by a network of server computers). The software and / or other instructions that give functions to a particular machine may optionally be stored as one or more unexecuted modules on one or more suitable storage devices. For example, the previously trained code generator, the previously trained correlation detection machine, and / or error handling executor are examples of machines according to this disclosure.

[0085] The machine can be implemented using any suitable combination of existing and / or future machine learning (ML), artificial intelligence (AI), and / or natural language processing (NLP) techniques. For example, the previously trained code generator and / or the previously trained correlation detector can be combined with any suitable ML, AI, and / or NLP techniques, including any suitable language model.

[0086] Non-limiting examples of techniques that can be incorporated into implementations of one or more machines include support vector machines, multilayer neural networks, convolutional neural networks (e.g., including spatial convolutional networks for processing images and / or videos, temporal convolutional neural networks for processing audio signals and / or natural language sentences, and / or any other suitable convolutional neural networks configured to convolve and pool features across one or more temporal and / or spatial dimensions), recurrent neural networks (e.g., long short-term memory networks), associative memories (e.g., lookup tables, hash tables, Bloom filters, neural Turing machines, and / or neural random access memories), word embedding models (e.g., GloVe or Word2Vec), unsupervised spatial and / or clustering methods (e.g., nearest neighbor algorithms, topological data analysis, and / or k-means clustering), graphical models (e.g., (hidden) Markov models, Markov random fields, (hidden) conditional random fields, and / or AI knowledge bases), and / or natural language processing techniques (e.g., tokenization, stemming, region selection and / or dependency parsing and / or intent recognition, segmentation models, and / or super-segmentation models (e.g., implicit dynamic models)).

[0087] In some examples, the methods and processes described herein can be implemented using one or more differentiable functions, wherein the gradient of the differentiable function can be computed and / or estimated with respect to the inputs and / or outputs of the differentiable function (e.g., with respect to training data, and / or with respect to a target function). Such methods and processes can be determined at least in part by a set of trainable parameters. Therefore, the trainable parameters for a particular method or process can be tuned by any suitable training procedure to continuously improve the functionality of the method or process.

[0088] Non-limiting examples of training procedures for tuning trainable parameters include: supervised training (e.g., using gradient descent or any other suitable optimization method), zero-shot, few-shot, unsupervised learning methods (e.g., classification based on categories derived from unsupervised clustering methods), reinforcement learning (e.g., feedback-based deep Q-learning) and / or generative adversarial neural network training methods, belief propagation, RANSAC (random sample consensus), contextual gambling machine methods, maximum likelihood methods, and / or expectation maximization. In some examples, components of multiple methods, processes, and / or systems described herein can be trained simultaneously with respect to an objective function that measures the performance of the collective function of multiple components (e.g., with respect to reinforcement feedback and / or with respect to labeled training data). Simultaneous training of multiple methods, processes, and / or components can improve such collective function. In some examples, one or more methods, processes, and / or components can be trained independently of other components (e.g., offline training on historical data).

[0089] The previously trained code generator and / or the previously trained correlation detector can be incorporated into any suitable language model. The language model can utilize lexical features to guide the sampling / searching of words for speech recognition. For example, the language model can be defined at least in part by the statistical distribution of words or other lexical features. For instance, the language model can be defined by the statistical distribution of an n-gram, with the transition probabilities between candidate words defined according to lexical statistics. The language model can also be based on any other suitable statistical features, and / or utilize one or more machine learning and / or statistical algorithms to process the results of the statistical features (e.g., confidence values ​​generated by such processing). In some examples, such as based on the assumption that the words in the audio signal come from a specific vocabulary, the statistical model can constrain which words can be identified for the audio signal.

[0090] Alternatively or additionally, the language model may be based on one or more neural networks previously trained to represent audio input and words in a shared latent space, such as a vector space learned by one or more audio and / or word models (e.g., wav2letter and / or word2vec). Therefore, finding candidate words may include searching the shared latent space based on vectors encoded by an audio model for the audio input to find candidate word vectors for decoding using a word model. The shared latent space can be used to evaluate the confidence of one or more candidate words in representing the candidate words in speech.

[0091] The language model can be used in conjunction with an acoustic model configured to evaluate the confidence that a candidate word is included in speech within an audio signal based on the acoustic features of the word (e.g., mel-frequency cepstral coefficients, formants, etc.). Optionally, in some examples, the language model can be incorporated into the acoustic model (e.g., the evaluation and / or training of the language model can be based on the acoustic model). The acoustic model defines, for example, a mapping between labeled speech and basic sound units such as phonemes in an acoustic signal. The acoustic model can be based on any suitable combination of existing or future machine learning (ML) and / or artificial intelligence (AI) models, such as: deep neural networks (e.g., long short-term memory, temporal convolutional neural networks, restricted Boltzmann machines, deep belief networks), hidden Markov models (HMMs), conditional random fields (CRFs) and / or Markov random fields, Gaussian mixture models, and / or other graphical models (e.g., deep Bayesian networks). Audio signals to be processed using an acoustic model can be preprocessed in any suitable manner, such as encoding at any suitable sampling rate, Fourier transform, bandpass filtering, etc. The acoustic model can be trained to recognize the mapping between acoustic signals and sound units based on training using labeled audio data. For example, the acoustic model can be trained based on labeled audio data including speech and corrected text to learn the mapping between speech signals and sound units represented by the corrected text. Therefore, the acoustic model can be continuously improved to enhance its utility for accurately recognizing speech.

[0092] In some examples, the language model can incorporate any suitable graphical model, such as a Hidden Markov Model (HMM) or a Conditional Random Field (CRF), in addition to statistical models, neural networks, and / or acoustic models. Given speech and / or other words identified so far, the graphical model can utilize statistical features (e.g., transition probabilities) and / or confidence values ​​to determine the probability of recognizing a word. Therefore, the graphical model can utilize statistical features, previously trained machine learning models, and / or acoustic models to define the transition probabilities between states represented in the graphical model.

[0093] When included, the input / output subsystem 606 may include one or more displays that can be used to present a visual representation of the data stored by the storage subsystem 604. This visual representation may take the form of a graphical user interface (GUI). The input / output subsystem 606 may include one or more display devices that utilize virtually any type of technology. In some implementations, the display subsystem may include one or more virtual reality, augmented reality, or mixed reality displays. When included, the input / output subsystem 606 may also include one or more speakers configured to output speech, for example, to present an audible representation of the data stored by the storage subsystem 604, such as an automated assistant response.

[0094] When included, the input / output subsystem 606 may include or interface with one or more input devices. Input devices may include sensor devices or user input devices. Examples of user input devices include keyboards, mice, touchscreens, or game controllers. In some embodiments, the input subsystem may include or interface with a selected Natural User Input (NUI) component. Such components may be integrated or peripheral, and the translation and / or processing of input actions may be performed on-machine or off-machine. Exemplary NUI components may include microphones for speech and / or speech recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; and head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition.

[0095] When included, the communication subsystem 608 can be configured to communicatively couple the computing system 600 to one or more other computing devices. The communication subsystem 608 may include wired and / or wireless communication devices compatible with one or more different communication protocols. The communication subsystem can be configured to communicate via a personal area network, a local area network, and / or a wide area network.

[0096] This disclosure is presented by way of example and with reference to the accompanying drawings. Components, process steps, and other elements that may be substantially the same in one or more of the drawings are identified in a coordinated manner and described with minimal repetition. However, it will be noted that the elements identified in a coordinated manner may also differ to some extent. It will also be noted that some drawings may be schematic and not drawn to scale. Various drawing scales, aspect ratios, and numbers of components shown in the drawings may be intentionally distorted to make particular features or relationships easier to see.

[0097] In one example, a method includes: identifying user utterances for processing by an automation assistant; generating a dataflow program based on the user utterances using a previously trained code generator, the dataflow program being configured to: generate a return value upon successful execution; initiating execution of the dataflow program; in response to reaching an error condition resulting from execution of the dataflow program: pausing execution of the dataflow program before the dataflow program generates the return value; generating an error-handling dataflow program using the previously trained code generator, wherein the error-handling dataflow program is configured to: generate the return value; initiating execution of the error-handling dataflow program to generate the return value; and outputting the return value. In this example or any other example, reaching the error condition includes detecting an error when a problematic program segment of the dataflow program is executed, and wherein generating the error-handling dataflow program includes outputting a new dataflow program based on the problematic program segment. In this example or any other example, the new dataflow program based on the problematic program segment includes a program rewriting function configured to replace the problematic program segment in the dataflow program with an alternative program segment. In this example or any other example, the method further includes maintaining a context-specific dialogue history, wherein the alternative program fragment is a search history function configured to search for the alternative program fragment in the context-specific dialogue history based on its relationship to the problematic program fragment. In this example or any other example, the alternative program fragment is output by the previously trained code generator, and wherein the previously trained code generator is trained using supervised training based on a plurality of training examples to output the alternative program fragment, wherein the training examples include: an exemplary problematic program fragment that would cause an error condition, and a resolving alternative program fragment that would not cause the error condition. In this example or any other example, the method further includes: identifying additional errors during the execution of the error handling dataflow program, and running an error recovery loop, the error recovery loop including generating and running additional error handling dataflow programs until one of the additional error handling dataflow programs produces the return value. In this example or any other example, the error is ambiguity, wherein the dataflow procedure uses a search history function to specify the concept to be resolved, the search history function finds two or more related concepts, and the error handling dataflow procedure can run to: output a clarification question that requires the user to provide a clarification response indicating one of the two or more related concepts, receive a clarification user utterance, and resolve the ambiguity based on the clarification utterance.In this example or any other example, the error is ambiguity, wherein the dataflow procedure uses a search history function to specify the concept to be resolved, the search history function finds a zero-relevance concept, and the error handling dataflow procedure can run to: output a clarification question, the clarification question requiring the user to provide a clarification response indicating a relevant concept, receive a clarification user utterance, and resolve the ambiguity based on the clarification utterance. In this example or any other example, the error is lack of confirmation, wherein the dataflow procedure specifies an action to be performed only after receiving a user confirmation response and the user utterance does not include such a user confirmation response, and the error handling dataflow procedure can run to output a confirmation question, the confirmation question requiring the user to provide the user confirmation response before performing the action.

[0098] In one example, an automated assistant system includes: a microphone; a speaker; a logic subsystem; and a storage subsystem storing instructions executable by the logic subsystem to: recognize user utterances for processing by the automated assistant; generate a dataflow program based on the user utterances using a previously trained code generator, the dataflow program being configured to: generate a return value upon successful execution; begin execution of the dataflow program; in response to reaching an error condition resulting from execution of the dataflow program: suspend execution of the dataflow program before the dataflow program generates the return value; generate an error-handling dataflow program using the previously trained code generator, wherein the error-handling dataflow program is configured to: generate the return value; begin execution of the error-handling dataflow program to generate the return value; and output the return value. In this example or any other example, reaching the error condition includes detecting an error when a problematic program segment of the dataflow program is executed, and wherein generating the error-handling dataflow program includes outputting a new dataflow program based on the problematic program segment. In this example or any other example, the new dataflow program based on the problematic program fragment includes a program rewriting function configured to replace the problematic program fragment in the dataflow program with an alternative program fragment. In this example or any other example, the instructions may also run to maintain a context-specific dialogue history, wherein the alternative program fragment is a search history function configured to search for the alternative program fragment within the context-specific dialogue history. In this example or any other example, the alternative program fragment is output by the previously trained code generation machine, and wherein the previously trained code generation machine is trained using supervised training based on multiple training examples to output the alternative program fragment, wherein the training examples include: an exemplary problematic program fragment that would cause an error condition, and a resolving alternative program fragment that would not cause the error condition. In this example or any other example, the instructions may also be executed to: identify additional errors during the execution of the error handling data stream procedure; and run an error recovery loop, the error recovery loop including generating and running additional error handling data stream procedures until one of the additional error handling data stream procedures produces the return value.In this example or any other example, the error is ambiguity, wherein the dataflow program uses a search history function to specify the concept to be resolved, the search history function finding two or more related concepts, and the error handling dataflow program can operate to: output a clarifying question via the speaker as audio, the clarifying question requesting the user to provide a clarifying response indicating one of the two or more related concepts; receive clarifying user utterances from the microphone; and resolve the ambiguity based on the clarifying utterances. In this example or any other example, the error is ambiguity, wherein the dataflow program uses a search history function to specify the concept to be resolved, the search history function finding zero related concepts, and the error handling dataflow program can operate to: output a clarifying question via the speaker as audio, the clarifying question requesting the user to provide a clarifying response indicating a related concept; receive clarifying user utterances from the microphone; and resolve the ambiguity based on the clarifying utterances. In this example or any other example, the error is a lack of acknowledgment, wherein the dataflow procedure specifies an action to be performed only after a user acknowledgment response is received and the user utterance does not include such a user acknowledgment response, and the error handling dataflow procedure outputs an acknowledgment question that requires the user to provide the user acknowledgment response before performing the action.

[0099] In one example, a method includes: identifying user utterances for processing by an automation assistant; generating a dataflow program based on the user utterances using a previously trained code generator, the dataflow program being configured to: generate a return value upon successful execution; initiating execution of the dataflow program; detecting an error condition when a problematic code segment of the dataflow program is executed; in response to detecting the error condition: pausing execution of the dataflow program before the dataflow program generates the return value; generating an error-handling dataflow program based on the problematic code segment using the previously trained code generator, wherein the error-handling dataflow program includes a program rewrite function configured to replace the problematic code segment in the dataflow program with an alternative code segment, and wherein the error-handling dataflow program is configured to: generate the return value; initiating execution of the error-handling dataflow program to generate the return value; and outputting the return value. In this example or any other example, the alternative program fragment is output by the previously trained code generator, and wherein the previously trained code generator is trained using supervised training based on a plurality of training examples to output the alternative program fragment, wherein the training examples include: an exemplary problematic program fragment that will cause an error condition, and a corrective alternative program fragment that will not cause the error condition.

[0100] It will be understood that the configurations and / or methods described herein are exemplary in nature, and these particular embodiments or examples should not be considered limiting, as many variations are possible. The particular routines or methods described herein may represent one or more of any number of processing strategies. Therefore, the various actions shown and / or described may be performed in the shown and / or described sequence, in other sequences, in parallel, or omitted. Similarly, the order of the above processes may be changed.

Claims

1. A method comprising: Recognize user utterances intended for processing by the automated assistant; A data stream procedure is generated based on the user utterance using a previously trained code generator, wherein the data stream procedure is constructed from a plurality of predefined composable functions and configured to produce a return value upon successful execution; Begin running the data stream program; In response to reaching an error condition resulting from the execution of the data stream program, wherein reaching the error condition includes detecting an error from a problematic program segment running the data stream program: Pause the execution of the data stream program before it generates the return value; The previously trained code generator is used to generate an error handling data stream procedure, wherein generating the error handling data stream procedure includes outputting a new data stream procedure based on the problematic program fragment, and wherein the error handling data stream procedure is configured to produce the return value; Start running the error handling data stream procedure to produce the return value; and Output the returned value.

2. The method according to claim 1, wherein, The new dataflow program based on the problematic program fragment includes a program rewriting function configured to replace the problematic program fragment with an alternative program fragment in the dataflow program.

3. The method of claim 2, further comprising maintaining a context-specific dialogue history, wherein, The alternative procedural fragment is a search history function configured to search for the alternative procedural fragment in the context-specific dialogue history based on its relationship to the problematic procedural fragment.

4. The method according to claim 2, wherein, The alternative program fragment is output by the previously trained code generator, wherein the previously trained code generator is trained using supervised training based on a plurality of training examples to output the alternative program fragment, wherein the training examples include: an exemplary problematic program fragment that will cause an error condition, and a corrective alternative program fragment that will not cause the error condition.

5. The method of claim 1, further comprising identifying additional errors during the execution of the error handling data stream procedure and running an error recovery loop, the error recovery loop comprising generating and running additional error handling data stream procedures until one of the additional error handling data stream procedures produces the return value.

6. The method according to claim 1, wherein, The error is ambiguous, wherein the dataflow procedure uses a search history function to specify the concept to be resolved, the search history function finding two or more related concepts, and the error handling dataflow procedure can run to: Output a clarification question, which requests a clarification response from the user, the response indicating one of the two or more related concepts. Receive clarification from users, and The ambiguity is resolved based on the clarification provided.

7. The method according to claim 1, wherein, The error is ambiguous, wherein the dataflow procedure uses a search history function to specify the concept to be resolved, the search history function finds a zero-relevance concept, and the error handling dataflow procedure can run to: Output a clarification question, which requests the user to provide a clarification response indicating the relevant concepts. Receive clarification from users, and The ambiguity is resolved based on the clarification provided.

8. The method according to claim 1, wherein, The error is a lack of acknowledgment, wherein the data stream procedure specifies an action to be performed only after a user acknowledgment response is received and the user utterance does not include such a user acknowledgment response, and the error handling data stream procedure can run to output an acknowledgment question that requires the user to provide the user acknowledgment response before performing the action.

9. An automated assistant system, comprising: microphone; speaker; Logical subsystem; as well as The storage subsystem stores instructions that can be executed by the logic subsystem to: Recognize user utterances intended for processing by the automated assistant; A data stream procedure is generated based on the user utterance using a previously trained code generator, wherein the data stream procedure is constructed from a plurality of predefined composable functions and configured to produce a return value upon successful execution; Begin running the data stream program; In response to reaching an error condition resulting from the execution of the data stream program, wherein reaching the error condition includes detecting an error from a problematic program segment running the data stream program: Pause the execution of the data stream program before it generates the return value; The previously trained code generator is used to generate an error handling data stream procedure, wherein generating the error handling data stream procedure includes outputting a new data stream procedure based on the problematic program fragment, and wherein the error handling data stream procedure is configured to produce the return value; Start running the error handling data stream procedure to produce the return value; and Output the returned value.

10. The automated assistant system according to claim 9, wherein, The new dataflow program based on the problematic program fragment includes a program rewriting function configured to replace the problematic program fragment with an alternative program fragment in the dataflow program.

11. The automated assistant system according to claim 10, wherein, The instructions can also be executed to maintain a context-specific dialogue history, wherein the alternative procedural fragment is a search history function configured to search for the alternative procedural fragment in the context-specific dialogue history based on its relationship to the problematic procedural fragment.

12. The automated assistant system according to claim 10, wherein, The alternative program fragment is output by the previously trained code generator, wherein the previously trained code generator is trained using supervised training based on a plurality of training examples to output the alternative program fragment, wherein the training examples include: an exemplary problematic program fragment that will cause an error condition, and a corrective alternative program fragment that will not cause the error condition.

13. The automated assistant system according to claim 9, wherein, The instructions can also be executed to: Additional errors are identified during the execution of the error handling data stream procedure; as well as An error recovery loop is run, which includes generating and running additional error handling data stream procedures until one of the additional error handling data stream procedures produces the return value.

Images