Streaming real-time dialogue management

Abstract

Systems and methods provide real-time, non-turn-based, dialog management. An example method includes receiving an initial or next block in a stream of blocks; generating a first candidate response based on receiving the block; updating a ranked list of candidate responses with at least one of the first candidate responses, the candidate responses accepted or pending at the time of receiving the block, the updated ranked list of candidate responses including one or more backend requests and one or more system responses; performing backend requests to two or more dialog modes based on the updated ranked list; and after performing the backend requests: generating backend responses based on information obtained for the two or more dialog modes in response to the backend requests; deriving a composite candidate response from the two or more dialog modes based on the generated backend responses; further updating the updated ranked list based on the composite candidate response; and pruning the further updated ranked list based on an updated ranking of candidate responses of the further updated ranked list.

Description

[0001] This application is a divisional application of the application for patent application number 201711035649.7, filed on October 30, 2017, entitled "Streaming Real-Time Dialog Management."

[0002] Related Applications

[0003] This application is a non-provisional application of U.S. Provisional Patent Application No. 62 / 459,820, filed on February 16, 2017, entitled "Streaming Real-Time Dialog Management," and claims priority to the provisional application, the disclosure of which is incorporated herein by reference. BACKGROUND

[0004] Dialog management is the problem of tracking a user's conversation with an electronic assistant. The assistant waits for the user to finish speaking, computes a response, and then provides the response. Such a flow is called turn-taking because the user takes a turn, then the assistant takes a turn, then the user takes a turn, and so on. Most electronic assistants handle one schema at a time, where the schema is similar to a task or a topic of a conversation, such as cooking, weather, setting a timer, and so on. Assistants that can handle multiple schemas still operate in a turn-taking flow. SUMMARY

[0005] Embodiments provide real-time, as opposed to turn-taking, dialog management. Systems using a real-time dialog management framework are able to engage in natural, real-time, two-way conversations with a user. Real-time dialog management differs from turn-taking because the system is constantly listening to streaming audio and deciding what the user intends to say, what response might be appropriate, and when to provide the response appropriately. In turn-taking, the system generates and provides a response in accordance with a specific request by the user; there is no uncertainty about whether to give a response. In contrast, in real-time dialog management, the system can attempt to predict an appropriate response before receiving a complete request from the user (as implied by the user's silence for a period of time). In predicting, the system must deal with multiple dialog paths that are updated, pruned, and generated while the system is still receiving input from the user. The real-time dialog management framework balances processing resources (wasted processing cycles for computing responses that are never provided). The real-time dialog management framework can enable an electronic assistant to form a response more quickly than turn-taking, to provide backward channel feedback at an appropriate time, and can provide an auxiliary response, or in other words, a predictive response that completes the user's thought. Thus, the real-time dialog management framework better simulates a natural conversation than turn-taking dialog. The real-time dialog management framework, including a conversation mixer that handles multiple schemas, enhances the simulation of a more natural conversation. Real-time, two-way, and predictive dialog improves the electronic assistant interface.

[0006] According to certain aspects, a method, implementable by one or more processors, includes generating a first candidate response to a trigger event. The trigger event can be a real-time stream of a dialog or a backend response received to a previous backend request for a dialog mode. A "backend response" is a response from a backend system and / or a dialog manager. The method further includes updating a list of accepted or pending candidate responses using at least the first candidate response, and determining whether the list of candidate responses includes a candidate response having a confidence score that satisfies a trigger threshold for the trigger event. The method further includes waiting for a next trigger event without providing a candidate response when the list does not include a candidate response having a confidence score that satisfies the trigger threshold.

[0007] According to certain aspects, a method, implementable by one or more processors, includes providing a block from a real-time dialog stream to a dialog mixer in response to receiving the block, receiving response candidates for the block from the dialog mixer, each response candidate being a system response for a dialog mode or a backend request for a dialog mode, and updating a rotating list of response candidates using at least one of the response candidates for the block. The method further includes ordering the response candidates in the list, each response candidate having a respective confidence score, determining whether the rotating list includes a response candidate having a confidence score that satisfies a trigger threshold, and initiating a backend request represented by a response candidate in the list having a confidence score that satisfies an ordering threshold but is not yet in an accepted dialog state when the rotating list does not include a response candidate having a confidence score that satisfies the trigger threshold.

[0008] According to certain aspects, a method, implementable by one or more processors, includes receiving a trigger event for a real-time dialog having a dialog bundle associated with a first path, the dialog bundle representing a dialog state for the real-time dialog with a user, determining that the trigger event opens a new path in the dialog bundle, and backtracking to an ancestor node in the dialog bundle in the first path. The method further includes generating a response candidate by using a base state represented by the ancestor node and information from the trigger event, thereby opening a new path in the dialog bundle from the ancestor node, wherein the paths in the dialog bundle include one or more accepted or pending response candidates, the response candidate being a system response generated by a dialog mode or a backend request for a dialog mode.

[0009] According to certain aspects, a computing system includes at least one processor and a memory storing instructions that, when executed by the at least one processor, cause the computing device to perform any of the methods, operations, or processes disclosed herein.

[0010] According to certain aspects, a computer program product embodied on a computer readable storage device includes instructions that, when executed by at least one processor formed in a substrate, cause a computing device to perform any of the disclosed methods, operations, or processes. Another general aspect includes a system and / or method for streaming real-time, multi-modal dialogue management to enhance real-time dialogue with a user, substantially as shown in, and / or described in connection with, at least one of the figures, and as set forth more completely in the claims.

[0011] According to certain aspects, a computing device configured to manage real-time dialogue with a user includes: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the computing device to: receive an initial or next block in a stream of blocks representing a real-time speech stream from a user; generate a first candidate response based on receiving the block; update a ranked list of candidate responses with at least one of the first candidate responses, the candidate responses being accepted or pending at the time of receiving the block, the updated ranked list of candidate responses including one or more backend requests for one or more dialogue modes and one or more system responses generated by one or more of the dialogue modes; perform backend requests to two or more dialogue modes based on the updated ranked list; and after performing the backend requests: generate backend responses based on information obtained for the two or more dialogue modes in response to the backend requests; derive a composite candidate response from the two or more dialogue modes based on the generated backend responses; further update the updated ranked list based on the composite candidate response; and prune the further updated ranked list based on an updated ranking of the candidate responses of the further updated ranked list.

[0012] According to certain aspects, a method includes: receiving an initial or next block in a stream of blocks representing a real-time speech stream from a user; generating a first candidate response based on receiving the block; updating a ranked list of candidate responses with at least one of the first candidate responses, the candidate responses being accepted or pending at the time of receiving the block, the updated ranked list of candidate responses including one or more backend requests for one or more dialogue modes and one or more system responses generated by one or more of the dialogue modes; performing backend requests to two or more dialogue modes based on the updated ranked list; and after performing the backend requests: generating backend responses based on information obtained for the two or more dialogue modes in response to the backend requests; deriving a composite candidate response from the two or more dialogue modes based on the generated backend responses; further updating the updated ranked list based on the composite candidate response; and pruning the further updated ranked list based on an updated ranking of the candidate responses of the further updated ranked list.

[0013] According to certain aspects, a non-transitory computer-readable storage medium comprises instructions that, when executed, cause one or more processors to: receive an initial block or a next block in a stream of blocks representing a real-time stream of speech from a user; generate a first candidate response based on receiving the block; update a ranked list of candidate responses with at least one of the first candidate responses, the candidate responses being accepted or pending at the time the block is received, the updated ranked list of candidate responses comprising one or more backend requests for one or more dialog modes and one or more system responses generated by one or more of the dialog modes; perform backend requests for two or more dialog modes based on the updated ranked list; and after performing the backend requests: generate backend responses based on information obtained in response to the backend requests for the two or more dialog modes; derive a composite candidate response from the two or more dialog modes based on the generated backend responses; further update the updated ranked list based on the composite candidate response; and prune the further updated ranked list based on an updated ranking of candidate responses of the further updated ranked list.

[0014] One or more implementations of the subject matter described herein can be implemented so as to realize one or more of the following advantages. For example, implementations compute responses faster than turn-based dialog managers. In some implementations, a system can generate a candidate in 10 milliseconds. This is much faster than traditional turn systems, which often wait a period of time (e.g., 0.5 seconds) before starting to process a user's turn of a dialog. As another example, implementations provide more accurate conversations with a user because the system can provide backchannel feedback, can provide help, and can perform operations faster than turns. Thus, a call from a user can be completed faster compared to using a turn-based dialog manager because the time to generate a candidate response is shortened and / or because the need for the user to repeat or rephrase parts of their speech due to being misinterpreted or misunderstood by the electronic assistant is eliminated (or at least significantly reduced). This reduces the length of time that network resources are devoted to the call, reduces power consumption of a user device (especially advantageous when the user has a battery-powered device), etc. Providing more accurate / natural conversations also makes the electronic assistant more user-friendly and easy to use. Moreover, because the system can track and maintain different paths, the system can have multiple long-term conversations. Implementations are also non-blocking on remote procedure calls, further improving device performance.

[0015] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1is a block diagram illustrating an example system according to the disclosed subject matter.

[0017] Figure 2 is a block diagram illustrating another example system according to the disclosed subject matter.

[0018] Figure 3 is a flow diagram illustrating an example process for managing a real-time conversation according to disclosed embodiments.

[0019] Figure 4 is a block diagram illustrating an example real-time conversation beam managed by a conversation host according to the disclosed subject matter.

[0020] Figure 5 An example of a computer device that can be used to implement the described technology is shown.

[0021] Figure 6 An example of a distributed computer device that can be used to implement the described technology is shown.

[0022] Like reference numbers in different drawings indicate the same element. DETAILED DESCRIPTION

[0023] Embodiments include systems and methods that read real-time streaming input in, for example, chunks, maintain a list of response candidates for the input, and decide when to provide one of the response candidates back to the user. The list of response candidates is a dynamic list in that it is constantly updated by adding one or more new response candidates and / or removing (or "pruning") one or more response candidates from the list, and is referred to as a "rotating" list. The conversation host invokes the conversation mixer at a triggering event, which can be a return backend request, receipt of new streaming input, or expiration of a time window (in the absence of other triggering events within the window). The conversation host maintains one or more paths in a conversation beam, manages branching paths, prunes paths with lower posterior probabilities, and backtracks in order to open new paths when needed. Streaming input is input that is received in real-time, and can include incomplete requests. In other words, embodiments start generating response candidates even before the user finishes speaking. Because the conversation host starts formulating a reply before the user finishes speaking, the conversation host improves the speed at which the electronic assistant can respond to the user. The conversation host includes ranking and triggering capabilities to decide which, if any, conversation response to provide to the user as part of the conversation. Deciding when to respond, i.e., deciding not to respond to a particular triggering event, is an important function so that the electronic assistant does not inappropriately interrupt or provide premature suggestions. Embodiments track the conversation state for each path of the conversation beam, and are able to backtrack or open new paths for a conversation when additional input changes the context of the conversation.

[0024] Figure 1 is a block diagram of a real-time dialog management system according to example embodiments. The system 100 can be used to more accurately simulate natural conversations with users to provide more helpful responses and to provide responses more quickly than traditional round-robin dialog managers. The system 100 can also be configured to provide candidate responses that combine multiple dialog patterns of a pattern in time. The system 100 is able to process real-time streaming input from a user, rather than waiting to process the input after the user has finished a command or query. Figure 1 The depiction of the system 100 in is a single computing device, but embodiments can also move some components to a server, making the system 100 a client-server system, as in Figure 2 shown in more detail in. Furthermore, one or more components can be combined into a single module or engine, and some of the capabilities of the illustrated components can be performed by separate engines. In some embodiments, a user of the computing device can instruct that some processing be performed at a server. Thus, embodiments are not limited to the precise configuration shown.

[0025] The real-time dialog management system 100 includes a computing device 105. The computing device can be implemented in a personal computer (e.g., a laptop), a smartphone, a wearable device (a smartwatch, smart glasses, etc.), a game console, a home appliance, etc. The computing device 105 can be an example of the computer device 500 as shown in Figure 5 The computing device 105 can include one or more processors formed in a substrate (not shown) that are configured to run various pieces of one or more machine-executable instructions, or software, firmware, or a combination thereof. The processors can be semiconductor-based, that is, the processors can include semiconductor materials that can perform digital logic. The computing device 105 can also include one or more computer memories. The memories, e.g., main memory, can be configured to store one or more pieces of data temporarily, permanently, semi-permanently, or a combination thereof. The memories can include any type of storage device that stores information in a format that can be read and / or run by the one or more processors. The memories can include volatile memory, non-volatile memory, or a combination thereof, and storage modules or engines that, when the modules or engines are run by the one or more processors, perform certain operations. In some embodiments, the modules can be stored in an external storage device and loaded into the memories of the computing device 105.

[0026] The computing device 105 can include a dialog input / output device 110. The dialog input / output device 110 can include hardware that enables the dialog host 120 to receive input from the user 180 or provide responses to the user 180. The user's input can be vocal, for example, in the form of speech. The speech can be provided as a stream input using conventional techniques, such as chunking. The user's input can also be non-vocal, such as text provided by the user, taps, etc. Similarly, the output can be speech-based or text-based. Examples of the input / output device 110 can include a microphone and a speaker. Another example of the input / output device 100 can be a keyboard (virtual or physical) and a display. The input / output device 110 can also include a module to convert the sound captured by the microphone into a stream input. The real-time dialog management system 100 is primarily discussed in the context of a spoken conversation using a microphone and a speaker, but embodiments include other modes of conversation, such as those maintained in a messaging application.

[0027] The modules of the real-time dialog management system 100 can include a dialog host 120. The dialog host 120 can be configured to obtain or receive input from the input / output device 110. The input can include a stream input. The stream input captures the user's speech (talk) as a series of chunks (e.g., a few seconds long) and provides the chunks as files to the dialog host 120. The stream input is considered a spoken input. The dialog host 120 treats each new file as a triggering event and invokes the dialog mixer 130 for each new input. The input can include a sliding window of chunks. For example, the window can include the newly received file and a certain number of previously received files, if they exist. The window can represent a duration of input for which the system has not yet committed to semantic understanding or response. In other words, the window can represent an "unstable" portion of the input that the system is using to determine different paths, so the system can still backtrack or open new paths, etc. Once the system provides a response, the system commits to the provided input and the input becomes "stable." In some embodiments, the window can be defined as any input chunk received after providing the most recent response. In some embodiments, the window can be defined in units of time period (e.g., seconds, fractions of seconds), etc. Thus, older files become too old to be included in the window.

[0028] The dialog host 120 can be configured to recognize non-verbal input as a trigger event. Non-verbal input can include a text string, a tap input, or a selection obtained from a user using the input / output device 110. The dialog host 120 treats such non-verbal input as a trigger event and is configured to invoke the dialog blender 130 for each new non-verbal input. The dialog host 120 can also treat a rewrite candidate as a trigger event. In some embodiments, the system can provide the context of the current input to an engine that performs various types of decisions on the input (e.g., anaphora resolution, ellipsis, etc.). The engine can be a function provided by the dialog host 120 or one of the dialog managers 170. The engine can provide a rewrite candidate that the dialog host 120 can treat as a backend response. The dialog host 120 is configured to invoke the dialog blender 130 using the rewrite candidate as a new input.

[0029] The dialog host 120 also recognizes the receipt of a backend response as a trigger event. The dialog host 120 is configured to invoke the dialog blender 130 for each backend response received. A "backend response" represents data generated using a dialog manager 170 that can be based on one or more searchable data stores, such as the backend system 190. The data is intended for output to the input / output device 110. The backend response can be provided by the dialog manager 170 in response to a request sent to the dialog manager 170. Thus, a backend response represents a search result provided by a pattern operated by a particular dialog manager 170. In other words, in this embodiment, a "backend request" by a dialog manager 170 uses the input to initiate a search of a pattern managed by the dialog manager 170. The "backend response" returned by the dialog manager 170 includes the results of the search. The backend response can be for a request solicited by the dialog host 120. The backend response can also be for a request that is not solicited by the dialog host 120. For example, in some embodiments, the dialog manager 170a can provide resources (e.g., information or data obtained in response to a request) to one or more other dialog managers (e.g., 170b and / or 170n), and the other dialog managers can use some or all of the resources to provide additional backend responses. The backend response includes a proposed system response to the backend request. The system response can be a verbal output to be provided to the user by the input / output device 110. If the response is provided, the system response can alternatively or also be associated with an action to be performed by the computing device. For example, the system response can cause the computing device to open an application and perform certain functions in the application, such as adding a new calendar event.

[0030] The dialog host 120 can be configured to invoke the dialog mixer 130 periodically in the absence of other triggering events. For example, if no new input and backend responses are received for a period of time (e.g., 100 milliseconds), the dialog host 120 can treat the passage of the period of time as a triggering event and invoke the dialog mixer 130. This causes the dialog host 120 to update the rotating list of candidates and make a new decision about whether to provide one of the candidates to the user as a response via the dialog input / output device 110.

[0031] The dialog host 120 manages a rotating list of candidate responses 150. Each candidate response can be referred to as a dialog. In a real-time streaming dialog environment, a dialog can be represented as a path in a dialog beam. A dialog beam is a beam search in which dialog responses are mapped to dialog states. Paths in a dialog beam represent dialog states that are generated from the same base state for the same input (e.g., a query). Since the system monitors input in real-time, the user’s intended dialog is not always known. Thus, the dialog host 120 manages several possible dialogs represented in the candidate list 150 at a time. The dialog host 120 prunes paths in the dialog beam that become irrelevant or outdated and adds new paths as needed. Each candidate is associated with a dialog state. The state can be represented by a data structure. The state data structure can include, for example, the question being answered taken from the input (e.g., an input window). The state data structure can include the context of the current session, a history of user inputs / requests, a system interpretation of the input, a history of responses provided to the user, other relevant events such as incoming notifications, data related to task prediction (e.g., data that helps the computing device determine or predict a task that the user wishes to accomplish such as reserving a table), a user’s state of attention (such as a person or place that the current dialog is about), etc. The state data structure can also include information about the type of information for the dialog request. For example, a calendar dialog can require a date, a time, an event name, etc. The state data structure can track the type of value needed and whether the value was provided. The dialog state can also include an indication of previous responses that were accepted as system responses (e.g., responses provided to the user). The candidate list 150 is stored in memory and maintained by the dialog host 120. The candidate list 150 represents candidate responses received from the dialog mixer 130 and their corresponding states.

[0032] The candidate responses in the candidate list 150 can be system responses that provide an action to be taken and / or a response to be provided to the user. The candidate responses can also be backend requests to be run. The backend requests can be associated with a dialog mode, or in other words, a particular dialog manager 170. For example, there can be a dialog manager 170a for cooking, a dialog manager 170b for local directions, a dialog manager 170c for music, a dialog manager 170d for time, etc. Thus, the dialog host 120 can include any number of different dialog managers (e.g., 170a-170n). The dialog host 120 can use the ranking engine 122 and / or the triggering engine 124 to determine whether to run a backend request. For example, if the request is to search for "cry" in the music mode, then this can represent a search that is unlikely to provide a single response since the goal of the dialog host is to provide a single relevant response, thus representing a waste of resources. Alternatively, if the request is to search for "cry me a river" in the music, then the dialog host 120 can decide to run the request, which would result in a backend response being provided to the dialog host 120. The state data structure can track whether a candidate is a request or a response, which enables the dialog host 120 to determine whether a request is outstanding.

[0033] The dialog host 120 includes a ranking engine 122 and a triggering engine 124. The ranking engine 122 can rank the candidate responses provided by the dialog mixer 130. The ranking engine 122 can prune candidate responses that have a low posterior probability. In other words, the ranking engine 122 can determine whether a particular candidate response is unlikely to be selected as a good response and provided to the user. For example, in some embodiments, the dialog mixer 130 provides a failure candidate and a backend request candidate for the same dialog manager (e.g., dialog manager 170a), and the ranking engine 122 can rank the failure candidate low and prune the candidate since the backend request has not been run, and the failure candidate is premature. Pruning a candidate means removing the candidate from the list of candidates. In some embodiments, the ranking engine 122 keeps the failure candidate until the corresponding backend response is received, and until the corresponding backend response is received, the failure candidate is given a low order at each ranking event.

[0034] The ranking engine 122 can include a machine learning model that takes as input the candidate responses in the candidate list 150 and the annotations about the candidate responses, and that provides as output a ranking of each candidate response. In some embodiments, the ranking model can be a machine learning model. For example, the ranking model can be a long short-term memory (LSTM) neural network, a feed-forward neural network, a support vector machine (SVM) classifier, etc., that can take the form of annotations about a candidate to give a set of ranking signals to predict whether the candidate is likely to be selected for presentation to the user. In some embodiments, the ranking model can be trained at the server and provided to the computing device 105. In some embodiments, the dialog host 120 can be configured to further train the ranking model in dependence on user responses to candidates provided to the user. For example, if a candidate is selected and presented to the user, but the user indicates disagreement, that candidate (and its corresponding state, including annotations) can be labeled as a negative training example for the model. Likewise, the system can use responses in which the user indicates agreement as positive training examples. The ranking score can be thought of as a confidence score that indicates how confident the model is that the response candidate is a high-quality relevant response.

[0035] The annotations can include characteristics of the real-time stream chunks obtained through speech analysis. For example, the annotations can indicate whether a chunk includes an upward inflection. As another example, the annotations can indicate whether a speaker has finished speaking, and if so, how long it has been since the speaker finished speaking. As another example, the annotations can indicate that a chunk includes a filler or how much of the chunk is a filler. A filler is a sound that signals that the speaker is pausing. For example, [uhhh] is a verbal filler. As another example, the annotations can indicate the energy of the speech, e.g., whether the speaker is shouting or conveying an indication of frustration. The system can use conventional speech analysis of the chunks to provide the annotations.

[0036] The ranking engine 122 can also prune response candidates from the candidate list. The ranking engine 122 can prune candidates that are too old. The ranking engine 122 can also prune backend request candidates that are computationally expensive yet have little chance of success (e.g., because the search was too broad). The ranking engine 122 can prune candidates that do not match, e.g., a failure candidate. A failure candidate is a candidate response that is provided as a default response and indicates that a particular dialog manager could not understand the request and could not provide a better response. In general, the system can prune any response candidate for which the system is confident that its ranking will be superseded. The ranking engine 122 can also prune any candidate that is not likely to be correct based on new information (e.g., additional input). In other words, once the system is confident of one interpretation, the ranking engine 122 can prune candidate responses related to other interpretations.

[0037] The dialog host 120 can also include a trigger engine 124. The trigger engine 124 can decide whether to actually provide one of the top candidates as a response to the user via, for example, the input / output device 110. When the trigger engine 124 provides a response, it can update the base state of the dialog. The base state represents the state that the system is committed to, e.g., the state that the system provided a response. Thus, once the trigger engine 124 provides a response, it can change or promote the temporary state of the candidate that was provided as a response to the user to the base state. In some embodiments, the trigger engine 124 can be a machine-learned model. For example, the trigger engine 124 can be a long short-term memory (LSTM) neural network, a feed-forward neural network, a support vector machine (SVM) classifier, etc., that selects to take no action or to select a response from the candidate responses. The trigger engine 124 can select to take no action, or in other words, to not respond, as an effective response to the trigger event. Whether the trigger engine 124 selects to take no action depends on the context of the trigger event and the candidate responses in the candidate list. The trigger engine 124 can also select one of the system response candidates in the candidate list in response to the trigger event. If the model selects a candidate, the trigger engine 124 can provide the selected system response to the input / output device 110 for presentation to the user. Presenting to the user can involve actions performed by the computing device 105 such as playing an audio file, playing a video file, providing text on a display, and / or invoking an application. For example, providing a candidate with the system response of [play Cry Me a River] can cause the computing system 105 to provide an audio output of [play cry me a river], and open a media application and start playing a song titled “Cry Me a River.” Depending on the response, providing a candidate as a response can include other actions such as adding a calendar event, setting a timer, adding a contact, setting an alarm, playing a movie, playing an audio book, etc.

[0038] The real-time dialog management system 100 includes a dialog mixer 130. The dialog mixer 130 is configured to take as input the base state and information about the trigger event (e.g., a backend response, new input, time elapsed). The base state includes the context of the current session, which includes the dialog states of all recently accepted candidates in the path of the dialog bundle (e.g., from the state data structure). The information about the trigger event can include text from the user, e.g., from the input stream window or via a text box, etc. The information about the trigger event can also include a response to a backend request. The information about the trigger event can also include a timestamp of the event.

[0039] The dialog mixer 130 provides one or more candidate responses as output. A candidate response can be a system response. A system response is text to be provided as part of the conversation and any action that the system 100 should take. A system response is optional and is not always included in the candidates provided by the dialog mixer 130. A candidate response can also be a backend request that the dialog mixer 130 wants the host to run. A backend request identifies the schema or dialog manager that the request is directed to, and the query to run. In some embodiments, the query is processed as a bundle search. A backend request is also optional and is not always included in the candidates provided by the dialog mixer 130. However, the dialog mixer 130 provides at least one system response or one backend request for each trigger event. For each candidate response, the dialog mixer 130 also provides a temporary dialog state. The temporary state can use the state data structure discussed herein. If the candidate is accepted, the temporary state can be used as part of the base state provided to the dialog mixer 130 in subsequent calls to the dialog mixer 130. For example, the temporary state with a backend request is provided as the base state for a backend response to the backend request. Finally, the dialog mixer 130 also provides annotations about the candidate for each candidate response. The annotations are used as a signal for ordering and can also be used for logging.

[0040] When the dialog mixer 130 is invoked, it accepts the base dialog state provided with the input. When the triggering event is new input, the dialog mixer 130 determines whether the user is triggering a new dialog. A new dialog corresponds to a new dialog manager, e.g., a new mode or a new search in a dialog mode. If the user is triggering a new dialog, the dialog mixer 130 takes out the corresponding mode and initializes the dialog managers for that mode. Then, the dialog mixer 130 distributes the output of the natural language parser (also called the analyzer) to all the dialog managers. When the triggering event is a backend response, the dialog mixer 130 loads the dialog managers corresponding to the backend response and applies the backend response to the dialog managers that requested them, respectively. The dialog mixer 130 can solicit backend requests and new state tokens from the dialog managers. Each dialog manager that is solicited generates some response, even if it is an error or a failure response. In some embodiments, the dialog mixer 130 can also make backend requests. The dialog mixer 130 rolls up the output of each dialog manager (whether a system response or a backend request) into a response candidate. Each candidate has some combination of system responses and / or backend requests and a temporary dialog state. In some embodiments, the dialog mixer 130 can perform second stage candidate generation. In second stage candidate generation, the dialog mixer 130 can derive composite candidate responses from two or more separate modes. The dialog mixer 130 provides the candidate responses, the respective dialog state for each candidate response, and the annotations for each candidate response back to the dialog host 120, where the responses are ordered, pruned, and potentially, the responses are triggered and provided to the input / output device 110.

[0041] The real-time dialog management system 100 can also include a plurality of dialog managers 170a-170n. Each dialog manager is responsible for a single dialog thread and represents a searchable mode. For example, the dialog manager 170a can be a music dialog for searching a music digital library. The dialog manager 170b can be a local dialog for searching a local area of interest (e.g., "restaurants near me") and for providing directions to a particular area of interest. The dialog manager 170c can be a calendar dialog that can find appointments, set new appointments, set reminders for appointments, etc. Each dialog manager is configured to look at the provided input and determine whether the input matches the mode. For example, the input [take me to] can not be enough to trigger a search in the food dialog manager, but can be enough to trigger and make a backend request in the local dialog manager and the music dialog manager.

[0042] The real-time dialog management system 100 can include a backend system 190. The backend system 190 represents a searchable data repository that provides responses for a particular dialog manager. For example, the music dialog manager 170a can call a music server to search for a title, artist, album, etc., and can play music from the repository. In some embodiments, the repository is local to the computing device, as shown in Figure 1 In some embodiments, the repository is remote, for example at one or more servers, as shown in Figure 2

[0043] Figure 2 is a block diagram illustrating another example system 100 according to the disclosed subject matter. In Figure 2 the example, the real-time dialog management system 100 includes a server 210, which can be a computing device or a device in the form of multiple different devices, such as a standard server, a group of such servers, or a rack server system. For example, the server 210 can be implemented in a distributed manner across multiple computing devices. In addition, the server 210 can be implemented in a personal computer, such as a laptop computer. The server 210 can be an example of the computer device 500 as shown in Figure 5 or the system 600 as shown in Figure 6 The real-time dialog management system can include a client device 205. The client device 205 is similar to the client device 105 as shown in Figure 1 Accordingly, the client device 205 includes the dialog input / output device 110, the dialog host 120, the dialog mixer 130, and the candidate list 150. In Figure 2 the example, the server 210 includes the dialog manager 170 and the backend system 190. In Figure 2 the example, the client device 205 communicates with the server 210 and with other client devices 190 over a network 140. The network 140 can be, for example, the Internet, or the network 140 can be a wired or wireless local area network (LAN), a wide area network (WAN), etc., implemented using, for example, gateway devices, bridges, switches, etc. The network 140 can also represent a cellular communication network. Via the network 140, the server 210 can communicate with and send data to / from the client device 205.

[0044] Figure 1 The real-time dialog management system 100 and Figure 2 ​The real-time dialogue management system 200 represents an example configuration, but implementations can incorporate other configurations. For example, some implementations may have only backend systems 190 on server 210, or some backend systems 190 on server 210 and some backend systems 190 on client device 205. Some implementations may have dialogue managers 170 on client device 205 and some dialogue managers 170 on server 210. Some implementations may move dialogue mixer 130, or some functionality of dialogue mixer 130, to server 210. Some implementations may move dialogue host 120 to server 210. Some implementations may combine dialogue input / output device 110, dialogue host, dialogue mixer 130, and dialogue manager 170 into a single module or application.

[0045] Regarding the collection and storage of user-specific data or the potential use of personal information by the real-time dialogue management system 100, users can be provided with the opportunity to control whether a program or feature collects user information, or to control whether and / or how content that may be more relevant to the user is received. Furthermore, certain data may be processed in one or more ways before being stored or used, thereby removing personally identifiable information. For example, search records may be processed to make it impossible to determine personally identifiable information and / or to generalize the user's geographic location in places where location information (such as city, zip code, or state) is available, thus making it impossible to determine the user's specific location. Therefore, users can control how information about themselves is collected and how the real-time dialogue management system 100 uses that information.

[0046] Figure 3 A flowchart of an example process 300 for managing real-time conversations according to the disclosed implementation is shown. Process 300 can be provided by a real-time conversation management system (such as...) Figure 1 System 100 or Figure 2 The process 300 is executed by the dialog host (such as dialog host 120). In some implementations, process 300 is run by the dialog host. Process 300 can be used to invoke the dialog mixer in response to a triggering event, determine what input to provide to the dialog mixer, manage a list of candidates provided by the dialog mixer, and decide whether to provide a candidate response to the user or remain silent and continue waiting for further input.

[0047] Process 300 can represent a main loop for a real-time dialog management system. Thus, process 300 can run continuously while the dialog system is active. Process 300 can include a wait mode in which the system waits for a triggering event (305). The wait mode can be interrupted by a triggering event (310-320). One triggering event is receiving a backend response (310). A backend response is a system response generated by a backend request. The backend response includes the system response and identifies the dialog manager that processed the request. Another triggering event is receiving new input (315). The input can be speech captured from a user in a sliding window. The input can be text entered by a user. The input can be a selection made by a user. The system can provide new input periodically (e.g., every 100 milliseconds) while the user is speaking. The sliding window can include up to a predetermined number of previous inputs. Thus, for example, the initial input of the sliding window can be "play cry" and the next input of the sliding window can be "me a river" such that the input of the sliding window is "play cry me a river." Another triggering event is the passage of time (320). The system can trigger this event when no backend response and no new input is received for a certain predetermined period of time. This triggering event enables the system to advance the dialog without other triggering events.

[0048] In response to a triggering event, the system can determine a base state for the triggering event (330). The base state describes the context of the current session for the triggering event. The base state can be a single dialog state or multiple dialog states. The base state includes the dialog state of any accepted candidate in the candidate list for a particular dialog path. A system response candidate is accepted when the event is triggered, or in other words, is provided as a response to the user. A backend request candidate is accepted when a backend request is run. A dialog path begins at a root state and includes any candidates that are accepted or pending until the system backtracks. Once the system backtracks to an ancestor node in the path, the ancestor node represents the base state of a new path, the new dialog path diverges from the current path at the ancestor node. The ancestor node can be, but is not necessarily, the root node in the dialog bundle.

[0049] As part of determining the base state of the triggering event, the system must determine which dialog path corresponds to the triggering event. This can be the current path, or it can be a new path opened up as a result of the system deciding to backtrack. For example, when additional input changes the query provided to one or more dialog managers (e.g., updates the bundle search string), the system opens up a second dialog path. The dialog path branches, or diverges from the current path, at the ancestor node to which the system backtracks. Thus, the system can manage multiple paths diverging from any base state, and can make decisions (e.g., ordering and triggering decisions) between the paths. The system can also prune paths when the candidates in that path become outdated or are low in the ordering. The dialog state can include an indication of which path the state belongs to. Dialog paths can include competing candidates from different dialog managers, so the base state can include more than one dialog state, e.g., different dialog states for different dialog managers. The dialog state can be stored in the state data structure described above with respect to Figure 1

[0050] The dialog host can send the base state and triggering event information to the dialog mixer (335). The triggering event information depends on the type of triggering event. For example, if the triggering event is a backend response, the triggering event information includes the received backend response. If the triggering event is the receipt of new input, the triggering event information is the received input, the input in a sliding window (including the window of received input), the received text, or the received other input. If the triggering event is the passage of time, the input can be the current timestamp.

[0051] The system can then receive potential candidates from the dialog mixer. The potential candidates can be system responses. A system response is something the system says (e.g., provided via an output device) and / or does (e.g., plays a song, sets a timer, purchases an item, etc.). The potential candidates can be backend requests. A backend request can represent a query in a particular dialog mode. Thus, the backend request can be provided to the dialog manager for that mode. The dialog manager can submit the query to a backend system and form a response. The receipt of the response by the dialog host is a triggering event. Thus, the backend request candidate includes an identifier for matching the response to the corresponding candidate. Each potential candidate has a corresponding temporary dialog state. Each potential candidate can also have corresponding annotations or metadata that can be used by the system to order and prune the potential candidates. The annotations or metadata can also be used for logging.

[0052] ​The system ranks the potential candidates, pruning poor candidates (345). The ranking occurs on all branches, not just the branch selected in step 330. The ranking can include a machine learning model that takes as input the annotations and metadata about the potential candidates and returns a score for each potential candidate. The model can be trained to take as input the list of potential candidates, their state, and annotations in all branches. The ranking results in some candidates being pruned. The pruned candidates can be removed from the candidate list. The pruned candidates can also be marked as pruned or inactive. A candidate can be pruned because it is too old, because it is a duplicate of other candidates, because it is too expensive (e.g., the query is too broad while the user is still speaking). All of these can result in a poor ranking score, e.g., a score that fails to meet (e.g., meet or exceed) a ranking threshold. A pruned candidate is no longer considered in the candidate list, i.e., it is not considered a response candidate.

[0053] The system then decides whether to trigger any of the candidates in the candidate list (350). The triggering decision can also use a machine learning model that assigns a confidence score to each candidate in the list. In some embodiments, the confidence score can be the order assigned to the candidate. The confidence score can represent how confident the system is that "the candidate is appropriate at this time." In other words, the system is not at all sure whether to provide the candidate response. This is different from a round-robin dialog system, in which the system always provides one of the candidate responses for the triggering event. In a real-time dialog system, the system is constantly determining whether to respond, and the option to not respond at all is a valid determination. The system can use various input signals to compute the confidence score for each candidate. The input signals can include whether the last spoken input from the user has an upward inflection. An upward inflection is a factor used to indicate that the user is ending a question. The input signals can include how long the user has been silent. A brief silence can mean that the user is thinking. A longer silence can indicate that the user is waiting for some response or can use help. For example, if the input sliding window is [play the 1978th song named... of Boston], the system can have generated the candidate system response [play more than a feeling]. If the user's voice weakens, for example, as the user is trying to recall the song name, the system can trigger the candidate. The input signals can include the length of the sliding window, how long the user has been speaking without triggering a response. If the user has been speaking for a while without triggering a response, the system can trigger a backchannel candidate. In some embodiments, the candidate list can include a backchannel feedback candidate as a default candidate. The backchannel candidate represents some feedback from the system that indicates that the system is listening, but the dialog is primarily one-way, i.e., the user is speaking. For example, the backchannel feedback candidate can be [uh-huh], [hmm], or [right] or some other expression that indicates attention or understanding.

[0054] The system can trigger the system response candidate when the system response candidate has a confidence score that meets (meets or exceeds) a triggering threshold. The system can also trigger the system response candidate when the system response candidate has an order that meets the triggering threshold. If the system decides not to trigger any response candidate (350, No), the system can initiate, e.g., run, any backend requests that were candidates but have not been accepted (355). Any backend requests that are still in the candidate list at this point are accepted. In some embodiments, the system can keep track (e.g., via a flag in the candidate list) which backend requests are outstanding. The system can then return to the waiting state (305).

[0055] If the system decides to trigger a candidate (350, yes), the system can execute the system response (360). Only candidates that are system responses can be triggered, because only system responses have output provided to the user. Output can be something provided to an output device, such as text spoken or displayed. Output can be an action performed by the computing device, such as playing a media file. A triggered system response candidate is an accepted candidate. If the triggered candidate is a backchannel candidate (365, yes), the system can initiate any accepted backend requests (355) as described above, as explained above, so the system can wait for the user to keep talking and decide later whether to provide a more specific response. If the triggered candidate is not a backchannel candidate (365, no), the system can clear any non-triggered branches (370). This can include setting a new root state or new base state and clearing the candidate list. The system can then enter a wait state (505) for the next triggering event.

[0056] The following is an example real-time dialogue for illustrating the process 300. In this example, the input provided by the user (e.g., via a microphone or keyboard) is shown in brackets [] as audio output provided by the system. Actions taken by the system are shown in braces {}. This example is for illustrative purposes only. In this example, the dialogue host starts with an empty candidate list, so the root state is null or empty. At the beginning, the dialogue host receives the stream chunk [Take me to the church] as the current input, e.g., at 315. Because there are no candidates in the list, the base state is null. Thus, the dialogue host sends an empty or null base state and the input "Take me to the church" to the dialogue mixer. The dialogue mixer determines that this input applies to two dialogue managers; the media (media) dialogue manager and the local dialogue manager. The dialogue mixer provides four potential candidates, as shown in Table 1. All of the candidates in Table 1 are in path 1 because they stem from the same base state (e.g., null state) and search the same input (e.g., "Take me to the church").

[0057] Path Candidate Dialog state Dialog manager Identifier 1 LocalSearch ("Take me to the church") L1 Local Local1 1 MediaSearch ("Take me to the church") M1 Media Media1 1 [Sorry, I can't find the direction] L2 Local Local2 1 [Sorry, I can't find your media] M2 Media Media2

[0058] Table 1

[0059] The dialog host ranks the four potential candidates: Local 1, Local 2, Media 1, and Media 2. The ranking can occur via a machine learning model that looks at the four candidates and the attributes of each candidate. The model determines that the Local 2 and Media 2 candidates, which represent failed candidates for the dialog manager, are poor candidates because the other two candidates represent pending or running backend requests. These two candidates have a poor order, and the dialog host prunes the Local 2 and Media 2 candidates. Thus, the candidate list now includes only two backend request candidates, namely Local 1 and Media 1. The dialog host determines that neither candidate is eligible to trigger because they are backend requests and not system responses. If the backend requests had a high enough order, the dialog host would begin running the Local 1 backend request and the Media 1 backend request. Beginning to run a backend request is an accept candidate. Thus, the L1 dialog state and the M1 dialog state are accept states. The Local 1 backend request corresponds to a local dialog manager that provides directions and points of interest. The Local 1 candidate represents a search for input in the Local mode (e.g., take me to the church). Similarly, the Media 1 candidate corresponds to a media dialog manager that searches a media library. The Media 1 candidate represents a search for input in the Media mode. Once the dialog host begins running these two backend requests, the dialog host waits for another triggering event.

[0060] The next triggering event is a response for the Media 1 candidate. In other words, the media dialog manager returns a result corresponding to the Media 1 request. The dialog host determines that the response corresponds to the Media 1 candidate as part of Path 1, and determines that the base state includes the L1 dialog state and the M1 dialog state. The L1 state is included because the Local search is pending, so the L1 dialog state is still valid. Thus, the dialog host provides the backend response (corresponding to the Media 1 candidate) and the base state of L1, M1 to the dialog mixer. In response, the dialog mixer provides three potential candidates, as shown in Table 2:

[0061]

[0062] Table 2

[0063] Media3 candidate is a system response that provides output [Play take me to church] to the user and initiates an action that causes the media player to start playing the corresponding media file, audio or video, which is identified as a response. In some embodiments, the dialog host replaces the Media1 candidate in the candidate list with the Media3 candidate because the Media3 candidate is a response received by running the request represented by the Media1 candidate. In some embodiments, the Media1 candidate is marked as complete but remains active. The dialog host prunes the Local3 candidate because it is a duplicate of the still running Local1 candidate. In some embodiments, the dialog mixer can recognize that the Local3 candidate is a duplicate and can not provide Local3 as a potential candidate. The dialog host ranks the Local4 candidate poorly because the Local1 request is still running. Thus, the dialog host prunes the Local4 candidate. This leaves Local1 and Media3 in the candidate list. Media3 is an eligible trigger of a system response, but the Media3 candidate has a lower rank because the user is still speaking, the user does not have an explicit play intent, i.e., the input is not [Play take me to church], and there is an outstanding request. Thus, the dialog host decides not to respond and not to trigger the Media3 response. This means that the Media3 candidate is not accepted; rather, the Media3 candidate is pending. There is no backend request to run, so the dialog host waits for another triggering event.

[0064] The next triggering event is the arrival of another stream chunk. The next input is the stream chunk [Take me to church by bike]. This stream chunk represents a sliding window that contains the previous input. The dialog host determines that the new input should be a new beam search. In other words, the dialog host determines that the query is more specific and opens a second path in the dialog beam. The base state of the new path is empty, i.e., the system reverts to the root state, and starts a new path from the root with the new search condition of “Take me to church by bike.” Thus, the dialog host sends an empty or invalid base state and the input “Take me to church by bike” to the dialog mixer. The dialog mixer determines that the input is applicable to the local dialog manager. The dialog mixer does not trigger the media dialog manager because the input does not sound like a media request. Thus, the dialog mixer provides two potential candidates, as shown in Table 3. These candidates are included in the candidate list, which has the still valid but pending candidates from the first path:

[0065]

[0066] Table 3

[0067] The dialog host ranks the four candidates: Local1, Local B 1, Media3 and Local B2. Local B Candidate 2 is poorly ranked, so the dialogue host prunes it because of the Local... B The search has not yet provided a response or has timed out. The Media3 candidate does not trigger because it is not responsive to input; for example, it is for path 1, not path 2. Therefore, the dialog host has no system response to trigger and begins running for Local. B One candidate request. Therefore, L B 1. The dialog state is the accepted state in path 2, and the dialog manager is waiting for the next trigger event.

[0068] The next triggering event is the response to the Local1 backend request. The dialog host can determine that this response corresponds to the Local1 candidate and that the response is in path 1, not path 2. Therefore, the dialog host determines that the base state includes the L1 dialog state and the M1 dialog state, which is the most recently accepted state in path 1. The M3 dialog state is not an accepted state because the candidate was not triggered. This base state is provided to the dialog mixer along with the backend response. The dialog mixer provides three candidates in the response. The three candidates are added to the candidate list, as shown in Table 4:

[0069]

[0070]

[0071] Table 4

[0072] The dialogue host prioritizes the Media4 candidate low and prunes it because it is a copy. In some implementations, the dialogue mixer can recognize that the candidate is a copy of the accepted candidate Media1 and may not even offer Media4 as a candidate. The dialogue host also prioritizes the Media5 candidate low and prunes it. Local5 and Media3 candidates are system responses, but are not considered due to pending backend requests (e.g., Local...). B 1), therefore they may have a lower order. Therefore, the L5 dialog state has not yet become an accepted state. Therefore, the dialog host chooses to do nothing in response to the triggering event and wait for the next triggering event.

[0073] The next triggering event corresponds to Local. B 1. The response to the backend request. The session host can determine that the response corresponds to the Local request. B Candidate 1 is in path 2 but not in path 1. Therefore, the dialogue host determines the base state including L. B1 Dialog state, which is the most recently accepted state in path 2. L1 and M1 states are not associated with path 2, and thus are not included in the base state provided to the dialog mixer. The base state is provided to the dialog mixer along with the backend response. The dialog mixer provides one candidate in the response. The candidate is added to the candidate list, as shown in Table 5:

[0074]

[0075] Table 5

[0076] The dialog host can have metadata, etc., that indicate that the user has finished speaking for Local B 3 candidate is ranked higher because it responds to the entire query, and the system can have metadata, etc., that indicate that the user has finished speaking. The Local B 5 candidate is ranked lower because it does not consider the entire query and the Media B 3 candidate is ranked worse. The dialog host decides to trigger the Local B 3 candidate. Triggering the Local B 3 candidate causes the system to update the base state of the dialog bundle to L B 3 dialog state, e.g., making L B 3 dialog state the root state, and output the response and run its corresponding action.

[0077] Figure 4 is an example diagram showing the dialog bundle 400 of the above example. The tree starts with an empty root dialog state 405. In other words, there are no pending requests or responses, and the candidate list is empty. The first trigger event, DM Trigger 1, causes the four dialog states shown in Table 1. Two of the dialog states (L2 and M2) are pruned, and the other two (L1 and M1) are accepted. All four states are part of path 1, which is shown in Figure 4 by solid lines 410. The second trigger event, DM Trigger 2, causes three more dialog states, two of which (L3 and L4) are pruned, and one of which (M3) is retained but not accepted. Thus, M3 is a pending dialog state. The next trigger event, DM Trigger 3, causes the system to backtrack and open a new path, which is shown in Figure 4 by dotted and dashed lines 450. DM Trigger 3 causes two new dialog states, one of which is pruned (L B 2), and one of which is accepted (L B 1). The next trigger event, DM Trigger 4, applies to the first path and causes a new dialog state, L5, that is retained but not yet accepted. The L5 dialog state is pending. The next trigger event, DM Trigger 5, applies to the second path and causes a new dialog state, L B 3, that is accepted. The acceptance of the L B 3 dialog state causes the pending dialog states of the first path, i.e., L5 and M3, to be pruned.

[0078] Figure 5 This document demonstrates techniques that can be used in conjunction with the technologies described herein and can serve as... Figure 1 This is an example of a general-purpose computer device 500 operated by server 110 and / or client 150. The computing device 500 is intended to represent various exemplary forms of computing devices, such as laptop computers, desktop computers, workstations, personal digital assistants, cellular phones, smartphones, tablets, servers, and other computing devices (including wearable devices). The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the embodiments of the invention described and / or claimed in this document.

[0079] Computing device 500 includes a processor 502, a memory 504, a storage device 506, and an expansion port 510 connected via interface 508. In some embodiments, computing device 500 may include a transceiver 546, a communication interface 544, and a GPS (Global Positioning System) receiver module 548, as well as other components, connected via interface 508. Device 500 can communicate wirelessly via communication interface 544, which may include digital signal processing circuitry if necessary. Each of components 502, 504, 506, 508, 510, 540, 544, 546, and 548 may be mounted on a common motherboard or otherwise mounted as appropriate.

[0080] Processor 502 can process instructions for execution within computing device 500, including instructions stored in memory 504 or on storage device 506, to display graphical information for a GUI on an external input / output device such as display 516. Display 516 can be a monitor or a flat-panel touchscreen display. In some embodiments, multiple processors and / or multiple buses, as well as multiple memories and multiple types of memory, may be used as appropriate. Furthermore, multiple computing devices 500 and each device providing the necessary operation (e.g., a server group, a set of blade servers, or a multiprocessor system) can be connected.

[0081] Memory 504 stores information within computing device 500. In one embodiment, memory 504 is one or more volatile memory cells. In another embodiment, memory 504 is one or more non-volatile memory cells. Memory 504 may also be another form of computer-readable medium, such as a magnetic disk or optical disk. In some embodiments, memory 504 may include extended memory provided via an expansion interface.

[0082] Storage device 506 provides mass storage for computing device 500. In one embodiment, storage device 506 may be or include computer-readable media, such as floppy disk devices, hard disk devices, optical disk devices, magnetic tape devices, flash memory or other similar solid-state storage devices, or include devices in a storage area network or other configured arrays of devices. A computer program product may be tangibly embodied in such a computer-readable medium. The computer program product may also instruct, when executed, to perform one or more methods as described above. Computer or machine-readable media are storage devices such as memory 504, storage device 506, or memory on processor 502.

[0083] Interface 508 may be a high-speed controller managing bandwidth-intensive operations of computing device 500, a low-speed controller managing less bandwidth-intensive operations, or a combination of these controllers. An external interface 540 may be provided to enable near-field communication between device 500 and other devices. In some embodiments, controller 508 may be coupled to storage device 506 and expansion port 514. The expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or, for example, network devices such as switches or routers via network adapters.

[0084] The computing device 500 can be implemented in a variety of different forms, as shown in the figure. For example, it can be implemented as a standard server 530, or multiple times within such a server group. It can also be implemented as part of a rack-mount server system. Furthermore, it can be implemented in computing devices such as a laptop computer 532, a personal computer 534, or a tablet / smartphone 536. The entire system can consist of multiple computing devices 500 communicating with each other. Other configurations are possible.

[0085] Figure 6 This document illustrates techniques that can be used in conjunction with the technologies described herein, and can be... Figure 1 This is an example of a general-purpose computing device 600, specifically server 110. The computing device 600 is intended to represent various exemplary forms of large-scale data processing devices, such as servers, blade servers, data centers, mainframes, and other large-scale computing devices. The computing device 600 may be a distributed system with multiple processors, which may include network-attached storage nodes interconnected via one or more communication networks. The components shown herein, their connections and relationships, and their functionality are merely examples and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0086] The distributed computing system 600 can include any number of computing devices 680. The computing devices 680 can include servers or rack servers, mainframes, etc. that communicate over a local or wide area network, dedicated optical links, modems, bridges, routers, switches, wired or wireless networks, etc.

[0087] In some embodiments, each computing device can include multiple racks. For example, the computing device 680a includes multiple racks 658a-658n. Each rack can include one or more processors, such as processors 652a-652n and 662a-662n. The processors can include data processors, network attached storage, and other computer-controlled devices. In some embodiments, one processor can operate as a master processor and control scheduling and data allocation tasks. The processors can be interconnected by one or more rack switches 658, and the one or more racks can be connected by a switch 678. The switch 678 can handle communications between multiple connected computing devices 680.

[0088] Each rack can include memory, such as memory 654 and memory 664, and storage, such as 656 and 666. The storage 656 and 666 can provide mass storage and can include volatile or nonvolatile memory such as network attached disks, floppy disks, hard disks, optical disks, magnetic tape, flash memory, or other similar solid-state memory devices, or arrays of devices including devices in a storage area network or other configurations. The memory 656 or 666 can be shared among multiple processors, multiple racks, or multiple computing devices and can include computer-readable media that stores instructions executable by one or more processors. The memory 654 and 664 can include, for example, one or more volatile memory units, one or more nonvolatile memory units, and / or other forms of computer-readable media such as magnetic or optical disks, flash memory, cache, random access memory (RAM), read-only memory (ROM), and combinations thereof. The memory, such as memory 654, can also be shared among the processors 652a-652n. Data structures, such as indexes, can be stored in the storage 656 and the memory 654, for example. The computing devices 680 can include other components not shown, such as controllers, buses, input / output devices, communication modules, etc.

[0089] An entire system, such as the system 100, can be composed of multiple computing devices 680 that communicate with each other. For example, the device 680a can communicate with the devices 680b, 680c, and 680d, and these can collectively be referred to as the system 100. As another example, Figure 1The system 100 can include one or more computing devices 680. Some of the computing devices can be located in geographically close proximity to each other, and other computing devices can be located geographically far apart. The layout of the system 600 is merely an example, and the system can employ other layouts or configurations.

[0090] According to certain aspects of the disclosure, a mobile device includes at least one processor and memory storing instructions that, when executed by the at least one processor, cause the computing device to perform operations. The operations include generating a first candidate response to a trigger event. The trigger event can be receiving a live stream block for a conversation or receiving a backend response to a previous backend request for a conversation mode. The operations further include updating a list of accepted or pending candidate responses using at least one first candidate response, and determining, for the trigger event, whether the list of candidate responses includes a candidate response having a confidence score that satisfies a trigger threshold. When the list does not include a candidate response having a confidence score that satisfies the trigger threshold, the operations further include waiting for a next trigger event without providing a candidate response.

[0091] These and other aspects can include one or more of the following features. For example, at least one of the first candidate responses can have a highest order of the first candidate responses. As another example, each candidate in the candidate list can be a system response or a backend request, and each candidate in the candidate list has a respective conversation state and is associated with a path in a conversation bundle. As another example, the pending candidate response can be a system response that has not been provided in response to the trigger event, and the operations further include determining a path in the conversation bundle corresponding to the trigger event, determining a base state of the trigger event; the base state includes conversation states of accepted candidates in the candidate list for the path, and generating the first candidate response using information from the trigger event and the base state. As another example, one of the candidate responses in the candidate response list can represent a backward channel feedback. As another example, the accepted response can be a backend request that has already been initiated. As another example, the pending response is a system response that is not provided to a user. As another example, the trigger event is a first trigger event, and the candidates in the candidate list all correspond to a first path in a conversation bundle, and the operations further include receiving a second trigger event, determining that the second trigger event requires a second path in the conversation bundle, setting a base state for the second path, the base state for the second path being a base state of an ancestor node in the first path of a current base state of the first path, generating a second candidate response using the base state for the second path and information of the second trigger event, and updating the list of accepted or pending candidate responses using at least one second candidate response. As another example, updating the list can include pruning candidate responses that fail to satisfy an ordering threshold.

[0092] In another aspect, a method includes providing, in response to receiving a block from a live dialog flow, the block to a dialog mixer, receiving response candidates for the block from the dialog mixer, each response candidate being a system response for a dialog mode or a backend request for the dialog mode, and updating a rotating list of response candidates using at least one response candidate for the block. The method further includes ordering the response candidates in the list, each response candidate having a respective confidence score, determining whether the rotating list includes a response candidate having a confidence score that satisfies a trigger threshold, and initiating a backend request represented by a response candidate in the list having a confidence score that satisfies an ordering threshold but is not yet in an accepted dialog state when the rotating list does not include a response candidate having a confidence score that satisfies the trigger threshold.

[0093] These and other aspects can include one or more of the following features. For example, each response candidate in the list can have a respective annotation and a respective dialog state, and ordering the response candidates can include providing the annotations and the list of response candidates to a machine learning model that uses the annotations and the response candidates in the list to determine the respective confidence scores. In such an implementation, the annotations can include characteristics of the block obtained through speech analysis. As another example, each response candidate in the list of response candidates can have a respective dialog state. As another example, the method can further include updating the response candidates in the list includes pruning candidates having confidence scores that do not satisfy the ordering threshold. As another example, each response candidate in the list of response candidates can have a respective dialog state and be assigned to a path in a dialog bundle, the dialog bundle including at least two paths. In such an implementation, when the rotating list does include a response candidate having a confidence score that satisfies the trigger threshold, the method can further include determining a path associated with the response candidate having the confidence score that satisfies the trigger threshold, and pruning from the list response candidates that are not associated with the path.

[0094] In another aspect, a method includes receiving a trigger event for a live dialog, the live dialog having a dialog bundle with a first path associated therewith, the dialog bundle representing a dialog state for the live dialog with a user, determining that the trigger event opens a new path in the dialog bundle, and backtracking in the first path to an ancestor node in the dialog bundle. The method further includes opening the new path in the dialog bundle from the ancestor node by generating a response candidate using a base state represented by the ancestor node and information from the trigger event, wherein the paths in the dialog bundle include one or more accepted or pending response candidates, the response candidate being a system response generated by a dialog mode or a backend request for the dialog mode.

[0095] These and other aspects can include one or more of the following features. For example, the ancestor node can be a root node representing a blank base state. As another example, the response candidates can have respective dialog states and be assigned to one of the dialog paths. As another example, the method can further include determining, in response to a second trigger event, that the response candidate in the new path is a system response having a confidence score that satisfies a trigger threshold, providing the response candidate to the user, and pruning the first path in the dialog bundle.

[0096] Various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0097] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms “machine-readable medium” “computer-readable medium” refers to any non-transitory computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory (including read-only memory devices), programmable logic devices (PLDs)) that provides machine instructions and / or data to a programmable processor.

[0098] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), and the Internet.

[0099] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0100] A number of implementations have been described. Nevertheless, various modifications can be made without departing from the spirit and scope of the disclosure. Additionally, the logic flows described in the figures do not require the particular order shown, or sequential order, to achieve desirable results. Furthermore, other steps can be provided, or described steps can be eliminated, from the described flows, and other components can be added to, or removed from, the described systems. Accordingly, other implementations are within the scope of the following claims.

Claims

1. A computing device configured to manage a real-time conversation with a user, the computing device comprising: at least one processor; and memory storing instructions that, when executed by the at least one processor, cause the computing device to perform operations comprising: receiving an initial or next block of a stream of blocks representing a real-time speech stream from a user; generating first candidate responses to a triggering event in response to the triggering event, wherein the triggering event comprises receiving a block of the stream of blocks or receiving a system response generated from execution of a previous backend request for a conversation mode; updating a ranked list of candidate responses with at least one of the first candidate responses, the ranked list of candidate responses being accepted or pending at the time of receiving the block, the updated ranked list of candidate responses comprising one or more backend requests for one or more conversation modes and one or more system responses generated by one or more of the conversation modes, wherein a pending backend request is a backend request that has not been accepted by initiating the backend request, and wherein a pending system response is a system response that has not been accepted by triggering the system response; executing backend requests for two or more conversation modes based on the updated ranked list; and after executing the backend requests: generating backend responses based on information obtained for the two or more conversation modes in response to the backend requests; deriving a composite candidate response from the two or more conversation modes based on the generated backend responses; further updating the updated ranked list based on the composite candidate response; and pruning the further updated ranked list based on an updated ranking of the candidate responses of the further updated ranked list.

2. The computing device of claim 1, wherein each candidate response in the ranked list of candidate responses has a corresponding conversation state and is assigned to a path in a conversation bundle, the conversation bundle comprising at least two paths.

3. The computing device of claim 2, the operations further comprising: determining that the further updated ranked list includes a system response candidate having a confidence score that satisfies a triggering threshold; and triggering the system response candidate. each candidate response in the ranked list has a respective annotation and a respective conversation state, and ranking the response candidates comprises: processing the annotations and the candidate responses through a machine learning model to determine a confidence score for each candidate response in the ranked list. each backend request of the updated ranked list of candidate responses is associated with a temporary conversation state, and wherein at least one of the generated backend responses is generated based on the temporary conversation state.

4. The computing device of claim 3, wherein, 6. A method of managing a real-time conversation with a user, comprising: receiving an initial or next block of a stream of blocks representing a real-time speech stream from a user; 5. The computing device of claim 3, wherein, generating first candidate responses to a triggering event in response to the triggering event, wherein the triggering event comprises receiving a block of the stream of blocks or receiving a system response generated from execution of a previous backend request for a conversation mode; updating a ranked list of candidate responses with at least one of the first candidate responses, the ranked list of candidate responses being accepted or pending at the time of receiving the block, the updated ranked list of candidate responses comprising one or more backend requests for one or more conversation modes and one or more system responses generated by one or more of the conversation modes, wherein a pending backend request is a backend request that has not been accepted by initiating the backend request, and wherein a pending system response is a system response that has not been accepted by triggering the system response; executing backend requests for two or more conversation modes based on the updated ranked list; and after executing the backend requests: generating backend responses based on information obtained for the two or more conversation modes in response to the backend requests; deriving a composite candidate response from the two or more conversation modes based on the generated backend responses; further updating the updated ranked list based on the composite candidate response; and pruning the further updated ranked list based on an updated ranking of the candidate responses of the further updated ranked list. updating a ranked list of candidate responses with at least one of the first candidate responses, the candidate responses being accepted or pending at the time of receiving the block, the updated ranked list of candidate responses including one or more backend requests for one or more dialog modes and one or more system responses generated by one or more of the dialog modes, wherein a pending backend request is a backend request that has not yet been accepted by initiating the backend request, and wherein a pending system response is a system response that has not yet been accepted by triggering the system response; performing the backend requests to two or more of the dialog modes based on the updated ranked list; and after performing the backend requests: generating backend responses based on information obtained for the two or more dialog modes in response to the backend requests; deriving a composite candidate response from the two or more dialog modes based on the generated backend responses; further updating the updated ranked list based on the composite candidate response; and pruning the further updated ranked list based on the updated ranking of candidate responses of the further updated ranked list.

7. The method of claim 6, wherein, each candidate response in the ranked list of candidate responses has a corresponding dialog state and is assigned to a path in a dialog bundle, the dialog bundle including at least two paths.

8. The method of claim 7, further comprising: determining that the further updated ranked list includes a system response candidate having a confidence score that satisfies a triggering threshold; and triggering the system response candidate.

9. The method of claim 8, wherein, each candidate response in the ranked list has a respective annotation and a respective dialog state, and ranking the response candidates includes: processing the annotations and the candidate responses by a machine learning model to determine a confidence score for each candidate response in the ranked list.

10. The method of claim 8, wherein, each backend request of the updated ranked list of candidate responses is associated with a temporary dialog state, and wherein at least one of the generated backend responses is generated based on the temporary dialog state.

11. A non-transitory computer-readable storage medium comprising instructions that, when executed, cause one or more processors to perform operations comprising: receiving an initial block or a next block in a stream of blocks representing a real-time stream of speech from a user; generating a first candidate response to a triggering event in response to the triggering event, wherein the triggering event includes receiving a block in the stream of blocks or receiving a system response generated by an execution of a previous backend request to a dialog mode; updating a ranked list of candidate responses with at least one of the first candidate responses, the candidate responses being accepted or pending at the time of receiving the block, the updated ranked list of candidate responses including one or more backend requests for one or more dialog modes and one or more system responses generated by one or more of the dialog modes, wherein a pending backend request is a backend request that has not yet been accepted by initiating the backend request, and wherein a pending system response is a system response that has not yet been accepted by triggering the system response; performing the backend requests to two or more of the dialog modes based on the updated ranked list; and after performing the backend requests: generating a backend response based on information obtained for the two or more dialogue modes in response to the backend request; deriving a composite candidate response from the two or more dialogue modes based on the generated backend response; further updating the updated ranking list based on the composite candidate response; and pruning the further updated ranking list based on an updated ranking of the candidate responses of the further updated ranking list.

12. The non-transitory computer-readable storage medium of claim 11, wherein, Each candidate response in the ranking list of candidate responses has a corresponding dialogue state and is assigned to a path in a dialogue bundle, the dialogue bundle comprising at least two paths.

13. The non-transitory computer-readable storage medium of claim 12, the operations further comprising: determining that the further updated ranking list includes a system response candidate having a confidence score that satisfies a trigger threshold; and triggering the system response candidate.

14. The non-transitory computer-readable storage medium of claim 13, wherein, Each candidate response in the ranking list has a respective annotation and a respective dialogue state, and ranking the response candidates comprises: processing the annotations and the candidate responses through a machine learning model to determine a confidence score for each candidate response in the ranking list.

15. The non-transitory computer-readable storage medium of claim 13, wherein, Each backend request of the updated ranking list of candidate responses is associated with a temporary dialogue state, and wherein at least one of the generated backend responses is generated based on the temporary dialogue state.

Images