Managing conversational data providers

Abstract

The present disclosure relates to managing conversational data providers. Methods, systems, and apparatus, including computer programs encoded on computer storage media, for managing conversations. In one aspect, the method includes receiving a request associated with a task from a user device; submitting the request to each of a plurality of different data providers; receiving a plurality of suggested conversational responses from two or more of the data providers; scoring the one or more suggested conversational responses based on one or more scoring factors; determining a particular conversational response to provide to the user based on the scoring; and providing the determined conversational response to the user device.

Description

[0001] Divisional

[0002] This application is a divisional of Chinese Patent Application No. 201680034889.2, filed on July 29, 2016, which has a priority date of July 29, 2016. TECHNICAL FIELD

[0003] This specification relates to methods and systems for managing a conversation in response to a request issued by a user using an interface on a user device. BACKGROUND

[0004] Conventional mobile devices can include software for responding to speech of a user of the mobile device. The speech can typically include instructions to cause the mobile device to call a telephone number, send a text to a telephone number, or search for information on the mobile device or the Internet. The software can provide an audio output from the mobile device confirming the instructions. The mobile device can provide the received speech to a server system for processing and receive information identifying an operation to be performed. SUMMARY

[0005] This specification describes technology relating to managing a conversation including mediation between different data providers. In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include receiving, from a user device, a request associated with a task; submitting the request to each of a plurality of different data providers; receiving one or more suggested conversational responses from one or more of the data providers; scoring the one or more suggested conversational responses based on one or more scoring factors; determining, based on the scoring, a particular conversational response to provide to the user; and providing the selected conversational response to the user device. More specifically, aspects of the described embodiments include a method for conversational management for a user device, including receiving, at a conversational system, a request associated with a task from a user device; the user device communicably coupled to the conversational system via a communication network; submitting, by the conversational system, the request to each of a plurality of different data providers, wherein each data provider independently analyzes the request according to a respective data model; receiving a plurality of suggested conversational responses from two or more of the data providers; scoring, by a conversational management engine of the conversational system, the one or more suggested conversational responses based on one or more scoring factors; determining, by the conversational management engine, a particular conversational response to provide to the user based on the scoring criteria; and providing the determined conversational response to the user device.

[0006] Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation, causes the system to perform the actions either directly or indirectly. For one or more computer programs to be configured to perform particular actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions either directly or indirectly.

[0007] The above and other embodiments can each individually or in combination optionally include one or more of the following features. In particular, one embodiment includes all of the following features in combination. The method includes updating a state of the conversation based on the selected conversation response. The received request is a voice input, and wherein the method includes converting the voice input request to text prior to submitting the request to the plurality of data providers. The method further includes determining whether the conversation requires a further response from the user device, and in response to determining that no further response is required, completing the task requested by the user. Each data provider independently analyzes the request according to a corresponding data model. In response to determining that none of the respective scores of the plurality of suggested conversation responses meet the threshold amount, synthesizing the response provided to the user device to determine the user intent. Determining the particular conversation response to provide to the user based on the scores includes disqualifying suggested conversation responses having scores below the threshold amount, and further disqualifying all suggested conversation responses that reference the suggested conversation response.

[0008] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. Improving conversation management by mediating between different data providers that can respond to user input to respond to a task. In particular, different data providers with different strengths and weaknesses can be leveraged. This allows for specialized data providers while also providing flexibility in the types of user input that can be interpreted. Moreover, conversation management can allow for parallel responses to be received from different data providers and optionally aggregate the received responses. Additionally, the data providers can be diverse with respect to implementation, e.g., they can be generated by different vendors or built using different technologies, or can be accessed over different networks (e.g., local or remote). As long as each data provider provides data using a prescribed interface, the system can combine their data into a unified conversation response.

[0009] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is an example system for managing a conversation.

[0011] Figure 2 is a flowchart of an example process for managing a conversation.

[0012] Like reference numerals and characters in the various figures indicate like elements. DETAILED DESCRIPTION

[0013] A user can provide a voice instruction to a device to perform a particular task, such as generating a calendar item, placing a call or text, or searching for particular information. The performance of the task by the device or by a system in communication with the device can be driven by a conversational dialogue that asks the user questions for one or more values needed to complete the task, similar to filling in the fields of a form.

[0014] For example, an example dialogue for a calendar task can be:

[0015] User: [Create an alarm]

[0016] Device / System: What time do you want to set the alarm for?

[0017] User: [3:00pm]

[0018] Device / System: An alarm has been created for 3:00pm.

[0019] In response to the user inputting a command to create an alarm, the system interprets the command as a request for a task to set an alarm at a particular time. Specifically, a data provider for performing the task can interpret the received voice input and direct the particular dialogue to information necessary to complete the task, e.g., the time for the alarm.

[0020] In another example dialogue, a user can seek particular information:

[0021] User: [When does the Giants game start?]

[0022] Device / System: Do you mean the San Francisco Giants or the New York Giants?

[0023] Here, the system understands the question from the user as a game schedule query, but there is an ambiguity as to which "Giants" the user is referring to. The dialogue responds by requesting clarification as to which "Giants" team the user is referring to. Once clarified, the dialogue can continue to provide schedule information, e.g., as identified by a search system.

[0024] In some cases, depending on how a voice input is interpreted by a dialogue system, a user input can change from one type of task to potentially another type of task. For example:

[0025] User: [Set an alarm]

[0026] Device / System: What time do you want to set the alarm for?

[0027] User: [When does the Giants game start?]

[0028] In this example, the user's second response can be interpreted in more than one way depending on which data provider the voice input triggered. For example, a task data provider can interpret the voice input related to creating an alarm, but not treat the second input about the Giants game as noise or a meaningless response because it is not related to the expected input for the alarm task conversation. But another data provider that can interpret game-related voice inputs can handle the Giants game scheduling question. This specification describes techniques for moderating and combining conversation responses from multiple data providers.

[0029] Figure 1 is an example system 100 for managing a conversation. The system 100 includes a user 102, a user device 104, a conversation system 106, and a plurality of data providers 124a, 124b, through 124n. The user device 104 and the conversation system 106 communicate over a network 103, such as the Internet.

[0030] The user device 104 can be one of many suitable devices including a mobile device, a wearable computer, a tablet computer, a hybrid, a laptop computer, or a desktop computer. The user device 104 receives oral (e.g., voice input) and non-oral interactions from the user 102. In particular, the user device 104 includes a microphone 108 configured to receive voice input from the user 102. The user device 104 can also include one or more speakers configured to broadcast conversation questions in response to received user requests. For clarity, only a single user device is shown. However, there can be many user devices associated with corresponding different users. Each of these user devices can communicate with the conversation system 106 over the network 103.

[0031] The user device 104 also includes a user interface 110. The user interface 110 can present information to the user including some or all of the content associated with the conversation in response to user requests. The conversation 112 defines a plurality of responses, such as value questions, needed to perform a task requested by the user. Particular questions or other content of the conversation 112 can be presented to the user in a particular order, for example, through a sequence of audio questions broadcast by one or more speakers or displayed in the user interface 110.

[0032] In some other implementations, the user device includes a speech analysis engine to convert received voice input to the microphone 108 to text before transmission to the conversation system 106.

[0033] The dialog system 106 can be one or more computing resources, such as one or more computing systems, or virtual machines executing on one or more computing systems, that communicate with the user device 104 over the network 103. The dialog system 106 includes a dialog management engine 114 and a speech analysis engine 116.

[0034] The speech analysis engine 116 can use a suitable speech-to-text process to convert the received speech input 118 into a text string. In some implementations, the speech analysis engine 116 can also emit text that can be processed by one or more parsers to identify one or more meanings, such as by generating a parse tree. The meanings of the audio converted to text can be used by one or more data providers to identify a particular task to perform in response to the received request.

[0035] The dialog management engine 114 manages one or more dialogs associated with a requested task. This includes maintaining a state of the dialog, such as what question is being answered, so that the speech analysis engine 116 can correctly interpret received audio associated with the dialog based on the current context provided by the state. In addition, the dialog management engine 114 determines a response to the incoming speech input based on the state of the dialog and the interpreted speech input of one or more of the data providers 124a, 124b, and 124n representing data provider 1, data provider 2, and data provider N, where N is some integer representing a total number of data providers that can provide input to the dialog system 106.

[0036] Each of these data providers 124a-n represents a backend system configured to independently interpret speech input according to a particular data model for that data provider. For example, a first data provider can be a task provider having a data model configured to interpret speech queries related to a particular task, such as creating a calendar entry, setting an alarm, initiating a phone call, or generating a text message. The task data provider expects user input associated with the particular task and recognizes speech input associated with those expected user inputs. A second data provider can be a game provider having a data model configured to interpret speech queries related to a particular game theme that includes teams and scores. Other data providers can include search data providers that focus on particular types of information, such as local businesses, social network information, or commercial data such as shopping opportunities.

[0037] In response to the speech input received from the user device 104 and processed by the speech analysis engine 116, one or more of the data providers 124 can interpret the received input and provide a dialog response to the dialog management engine 114. The dialog management engine 114 then determines which dialog response to provide to the user device 104. Different response modalities will be described in greater detail below.

[0038] Each data provider 124a-n suggests a dialog response based on the interpretation of the received input according to the corresponding data provider's data model. The dialog response, and optional other information such as an expected response from the user 102, can be encoded in a data structure according to a particular data format. In some implementations, this information is encoded in a protocol buffer referred to in this specification as a DialogTurnIntent ("DTI"). For convenience, the DTI will be used throughout, but other suitable encoded data structures can be used.

[0039] For example, in a dialog for the task of composing an email message, one question of the dialog can be a request for a subject of the email. The task data provider can create a DTI in which the user's question is a prompt for a subject. The DTI is provided to the dialog management engine 114 and sent to the user device 104, where it can be presented to the user 102 as a visual question in a user interface, e.g., "What is the subject?", or as an audio question from a speaker, e.g., "What do you want the subject to be?"

[0040] The dialog management engine 114 can send 120 more than one DTI to the user device 102. In particular, the DTI can include not only the current prompt to the user, but also other DTIs related to other fields necessary to complete the requested task in the dialog. For example, when the dialog management engine 114 sends the "prompt for subject" DTI, it can also send a DTI for a subsequent question in the dialog, e.g., a "prompt for message body" DTI.

[0041] Figure 2 is a flowchart of an example process 200 for managing a dialog. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations and programmed appropriately in accordance with this specification. For example, a dialog system 106, such as the dialog system 106 of Figure 1 , appropriately programmed, can perform the process 200.

[0042] The system receives input including a request (202). The request can be as a speech input provided by a user to a user device or to a dialog system, e.g., the dialog system 106, of the user device (e.g., the user device 104), or as a text input provided by a user to a user device or to a dialog system, e.g., the dialog system 106, of the user device (e.g., the user device 104). Figure 1from a user of the user device 104) and are transmitted to the system. In some implementations, the user voice input can be converted to text by the user device before being sent to the system. The task can be, for example, generating a calendar item, setting an alarm, generating an email, placing a call or text, or searching for specific information.

[0043] The system provides the request to a plurality of data providers (204). Each data provider can be configured to interpret different types of requests using a particular data model. Thus, the received request can be interpreted differently by different data providers depending on how the request is processed according to the respective data model.

[0044] The system receives suggested dialog responses from one or more of the data providers (206). Each data provider independently analyzes the request according to the respective data model. The data model can include a model trained on a particular set of data associated with a particular type of information or action. The request can trigger a response from some but not all of the plurality of data providers. Thus, a voice input request for a schedule of baseball games can trigger a suggested dialog response from a data provider trained to provide sports-related responses, but not a data provider trained to provide weather information, as the voice input does not match an expected request associated with weather information. In some implementations, a data provider can provide a response to an unrecognised voice input, for example, indicating that the input was not understood, and asking the user to repeat the response.

[0045] One or more of the data providers can provide a suggested dialog response in the form of a DTI that includes the suggested response provided to the user device.

[0046] In some implementations, each data provider is able to view the suggested dialog responses provided by other data providers. In response, a designated data provider can modify or add a suggested dialog response based on the dialog responses of other data providers. For example, a task data provider can suggest a response based on the suggested response of a search data provider, for example, suggesting a time to set an alarm based on the response from the search data provider providing the time of a sports game. Additionally, in some implementations, each data provider can provide multiple suggested dialog responses. For example, each response can be based on the context of another data provider’s suggested response and ignore the suggested responses of other data providers.

[0047] The system determines which dialog response to select (208). Specifically, the system analyzes all of the suggested dialog responses and determines which suggested dialog response is the most suitable to select. The analysis is performed after all of the individual suggestions are received from the data providers, but before any dialog response is sent to the user device. The system can score each suggested response based on one or more factors. Various factors can be used to score the suggested dialog responses to select including one or more of: a log-based adjustment of the likely user intent, a user personalization model, a score indicating the likelihood of each data provider's semantic interpretation, or an overall dialog policy configuration that determines a balance between careful consistency and minimizing the number of questions asked as part of the dialog.

[0048] In some implementations, the data providers include a confidence score and its result. The confidence score can be used to determine which suggested response to select. In some other implementations, other selection factors can include the length of the dialog session and the history of the dialog session. For example, if the user asks the same question multiple times, the data provider can not provide the information the user is looking for and a different data provider can be selected.

[0049] In some implementations, a selection factor is the likelihood that the information being considered is something a normal user would be interested in (as opposed to esoteric). For example, if there is no sports team like the Giants, it should be considered that the user was misinterpreted before searching for, for example, "Giants game."

[0050] In some implementations, a selection factor is the quality of the information being considered. The quality can be based on, for example, whether the data provider obtained the information from a structured source, such as an official baseball schedule service, or from an unstructured source, such as a blog by a random person.

[0051] In some implementations, a selection factor is based on pragmatics and task planning. For example, if the system does not know when either of the Giants teams play, then the system should not ask the user which Giants they mean, because the answer would not have any impact on the broader task.

[0052] In some implementations, a selection factor is derived based on the relevance of the information being considered to the user's query. For example, if "Giants" is only a not well-known team in a far away country, then the system should consider them irrelevant.

[0053] In some implementations, the selection factor is based on a determination of whether the query is sufficiently ambiguous that the system should bother the user to clarify their intent rather than ask a dumb / obvious question. The determination can be made, for example, based on geographic information about where the user is located; personalization, e.g., a history of the user discussing the San Francisco Giants versus the New York Giants; or context, e.g., only one of the two teams has a game scheduled soon.

[0054] In some implementations, if a particular suggested dialog response scores low based on these factors, e.g., the suggested dialog response does not meet a particular threshold score, the dialog management engine can not only disqualify the suggested dialog response, but also disqualify any other suggested dialog responses that reference the suggested dialog response. In some implementations, if scores from two or more data providers cannot be distinguished by a specified threshold amount, the dialog management engine can generate an intent disambiguation question as a dialog response.

[0055] The system provides the selected dialog response to the user device (210) and updates the state of the dialog (212). The dialog response sent to the user device can include the DTI of the selected dialog response that indicates the response to be presented to the user on the user device. The response can be synthesized as a speech response or presented on the user interface of the user device. In some implementations, the response is sent with additional information, e.g., one or more search results associated with the dialog response. For example, if the dialog response is based on a data provider's search or relevant web pages, links to those web pages can be provided with the dialog response.

[0056] The updated state of the dialog can be used to determine the next part of the dialog to send in response to a received user response. In addition, the updated state of the dialog can provide context to the analysis of subsequent speech input received from the user device so that the speech input can be correctly interpreted.

[0057] Upon completion of a suitable dialog, the system completes the request (214).

[0058] The system can determine whether additional values are needed or whether the dialog is complete. If additional values are needed, one or more additional DTIs can be sent to the user device, or the system can wait for additional responses from the user device. If no additional values are needed, the system can complete the task, e.g., by generating a calendar entry or email message, or by providing specific requested information. The completed task can be sent to the user device for approval before execution, or it can be executed automatically.

[0059] There are a variety of ways that the system can interpret input based on the scores of the suggestions and applications of different data providers. Some example cases of interpreting user input are described below:

[0060] Case 1:

[0061] User: [Set alarm]

[0062] System / device: What time do you want to set the alarm for?

[0063] User: [When does the Giants game start?]

[0064] In this example, the data providers can include a task provider, a sports provider, and a web answer provider. If the system determines that the sports and web data provider's dialog suggestions are noisy, then these responses should be inappropriate, and the task provider that has already been triggered should manage the dialog. This results in a response such as the following:

[0065] System / device: I'm sorry, I didn't understand that. What time do you want to set the alarm for?

[0066] Case 2:

[0067] In this case, the user is asking a side question because they want to watch the Giants game. The system this time has higher confidence in the web answer, as reflected in the scores of the corresponding suggested dialog responses, so it is selected over the sports provider and the task provider to create a hybrid dialog:

[0068] User: [Set alarm]

[0069] System / device: What time do you want to set the alarm for?

[0070] User: [When does the Giants game start?]

[0071] System / device: Based on the following results, the Giants game starts at 3 PM today. Do you want to set an alarm for that time?

[0072] Case 3:

[0073] Similar to Case 2, except that the dialog management engine composes a new question to determine whether the user intends to change the topic or whether they are still interested in the original task:

[0074] User: [Set alarm]

[0075] System / device: What time do you want to set the alarm for?

[0076] User: [When does the Giants game start?]

[0077] System / device: Based on the following results, the Giants game starts at 3 PM today. Do you still want to set an alarm for that time?

[0078] Case 4:

[0079] In case 4, the user is asking a side question as the user wants to watch the Giants game. Although the web answer provider provides the time as the suggested answer, the system allows the higher scoring sports provider to disambiguate and confirm the correct entity. To maintain the intent of setting an alarm, the following mixed dialog occurs:

[0080] User: [set alarm]

[0081] System / device: What time do you want to set the alarm for?

[0082] User: [when does the giants game start?]

[0083] System / device: Do you mean the San Francisco Giants or the New York Giants?

[0084] User: [San Francisco Giants]

[0085] System / device: Based on the following results, the San Francisco Giants play at 3 PM.

[0086] Set an alarm for that time?

[0087] Case 5:

[0088] In this case, the user is changing the topic and no longer wants to set an alarm. At this point, the dialog management engine is confident that the web answer provider identified the time of the Giants game, which scored higher than other suggested dialog responses, and does not suggest to continue the previous task of setting an alarm.

[0089] User: [set alarm]

[0090] System / device: What time do you want to set the alarm for?

[0091] User: [when does the giants game start?]

[0092] System / device: Based on the following results, the Giants game starts at 3 PM today.

[0093] Case 6:

[0094] In this case, the user is changing the topic and no longer wants to set an alarm. The dialog management engine uses the highest scoring suggestion from the sports data provider to disambiguate the entity, and then the web answerer identifies the time. No alarm is suggested.

[0095] User: [set alarm]

[0096] System / device: What time do you want to set the alarm for?

[0097] User: [when does the giants game start?]

[0098] System / device: Are you referring to the San Francisco Giants or the New York Giants?

[0099] User: [San Francisco Giants]

[0100] System / device: Based on the following results, the Giants game starts at 3pm today.

[0101] Case 7:

[0102] In this case, the dialog management engine can synthesize a question to disambiguate whether the user is changing the topic or asking a side question. This can occur because two or more of the suggested responses scored too close to determine which one should be supported.

[0103] User: [Set alarm]

[0104] System / device: What time would you like to set the alarm for?

[0105] User: [When does the Giants game start?]

[0106] System / device: Wait, do you want to complete setting the alarm or are you trying to schedule the alarm around the game?

[0107] In each of these cases, various data providers suggest one or more dialog responses based on the user input. Additionally, data providers can suggest multiple dialog responses based on the suggested dialog responses of other data providers.

[0108] Referring to the example case above, the task provider can suggest the dialog response "I'm sorry, I didn't understand that. What time would you like to set the alarm for?" This corresponds to case 1 above.

[0109] In response to the task provider realizing that the suggested dialog response was sent by the web answer provider, the task provider can generate another suggested dialog response that references the web answer suggested dialog response and the additional content "Would you like to set an alarm for that time?" This corresponds to case 2 above.

[0110] In response to the task provider realizing that the suggested dialog response was sent by the web answer provider, the task provider can generate another suggested dialog response that references the web answer suggested dialog response and the additional content "Would you like to set an alarm for that time?" This corresponds to case 2 above.

[0111] In response to the task provider realizing that the suggested dialog response was sent by the sports data provider to disambiguate the entity, the task provider can generate another suggested dialog response that references the sports answer suggested dialog response and the additional content "Would you like to set an alarm for that time?" This corresponds to case 4 above.

[0112] The various suggested dialog responses from each data provider are scored according to one or more factors. Based on the scoring, the dialog management engine can then select an appropriate dialog response. Referring again to the example case above, based on the respective scores, the dialog management engine can:

[0113] A) select the suggested dialog response of the task provider (case 1)

[0114] B) select the alternative suggested dialog response of the task provider and also accept the suggested dialog response of the web answer provider (case 2)

[0115] C) select the alternative suggested dialog response of the task provider and also accept the alternative suggested dialog response of the web answer provider (case 3)

[0116] D) select the alternative suggested dialog response of the task provider and also accept the suggested dialog response of the sports data provider (case 4)

[0117] E) select the suggested dialog response of the web answer provider and abandon the alarm state altogether (case 5)

[0118] F) select the suggested dialog response of the sports data provider and abandon the alarm state altogether (case 6)

[0119] G) generate a brand new suggested dialog response to ask the user's intent (case 7). For example, if the scores from the data providers do not distinguish by a threshold amount.

[0120] Eliminating a suggested dialog response can cascade to other suggested dialog responses that reference it. Thus, if the suggested dialog response of the web answer provider has a failing score, then not only is case 5 eliminated, but cases 2 and 3 are also eliminated. In some implementations, geographic information can also be used for scoring. For example, using geographic information to indicate which Giant the user is referring to (San Francisco vs. New York) can suppress the sports data provider suggested response to not only eliminate case 6, but also case 4.

[0121] Embodiments of the subject matter and operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory array or a combination of one or more of them, or it can be included in one or more of the above. Furthermore, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or include, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0122] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0123] The term“data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones of the same. The apparatus can include special purpose logic, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

[0124] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or code portions). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and are interconnected by a communication network.

[0125] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit) and

[0126] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0127] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser.

[0128] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0129] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server.

[0130] Although the description contains many specifics, these should not be construed as limiting the scope of any invention or application in which such inventions can be employed, but merely as describing features that can be incorporated into a particular embodiment. Some features described in the context of separate embodiments can also be implemented together in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a sub-combination or a variation of a sub-combination.

[0131] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such order nor that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated in a single software product or packaged into multiple software products.

[0132] Accordingly, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, actions recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the process depicted in the accompanying figures can not require the particular order shown, or sequential execution in all cases, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

Claims

1. A method for managing a conversation, comprising: receiving, from a user device, a request associated with performance of a first task, wherein the request includes a first speech input of a user of the user device; submitting the request to each of a plurality of different data providers, wherein each data provider is associated with a different data model configured to interpret a particular type of speech input, and wherein the plurality of different data providers includes (i) a task data provider that directs a conversation toward information for completing a task and (ii) a search data provider; in response to the first speech input, receiving, from two or more of the data providers, a first plurality of suggested conversational responses to the first speech input; determining, from the first plurality of suggested conversational responses, a conversational intent of the first speech input and a corresponding first conversation of the first task that includes one or more first conversational responses to provide to the user device to complete the first task; receiving a second speech input in response to providing the one or more first conversational responses to the user and submitting the second speech input to each of the plurality of data providers; receiving, from two or more of the data providers, a second plurality of suggested conversational responses to the second speech input, the second plurality of suggested conversational responses including a first suggested conversational response from the search data provider and a first suggested conversational response from the task data provider, the first suggested conversational response from the task data provider being based on the first suggested conversational response from the search data provider; determining, for each of the second plurality of suggested conversational responses, a respective score associated with that conversational response; determining a second conversational response based on the respective scores associated with the second plurality of suggested conversational responses, the second conversational response being based on both the first suggested conversational response from the search data provider and the first suggested conversational response from the task data provider; and providing the second conversational response to the user device.

2. The method of claim 1, wherein, Determining the respective scores associated with the conversational responses is based on a user personalization model.

3. The method of claim 1, wherein, Determining the respective scores associated with the conversational responses is based on respective confidence scores included with the conversational responses received from one of the data providers.

4. The method of claim 1, comprising: updating a state of the first conversation in response to the one or more first conversational responses to the first speech input; and providing the updated state to the plurality of data providers as a context for analyzing the second speech input.

5. A system for managing a conversation, comprising: a user device; and one or more computers configured to interact with the user device and perform operations, the operations comprising: receiving, from the user device, a request associated with performance of a first task, wherein the request includes a first speech input of a user of the user device; submitting the request to each of a plurality of different data providers, wherein each data provider is associated with a different data model configured to interpret a particular type of voice input, and wherein the plurality of different data providers includes (i) a task data provider that directs a conversation toward information for completing a task and (ii) a search data provider; receiving, from two or more of the data providers, a first plurality of suggested conversational responses to the first voice input in response to the first voice input; determining, from the first plurality of suggested conversational responses, a conversational intent of the first voice input and a corresponding first conversation of the first task that includes one or more first conversational responses to provide to the user device to complete the first task; receiving a second voice input in response to providing the one or more first conversational responses to the user and submitting the second voice input to each of the plurality of data providers; receiving, from two or more of the data providers, a second plurality of suggested conversational responses to the second voice input, the second plurality of suggested conversational responses including a first suggested conversational response from the search data provider and a first suggested conversational response from the task data provider, the first suggested conversational response from the task data provider being based on the first suggested conversational response from the search data provider; determining, for each of the second plurality of suggested conversational responses, a respective score associated with the conversational response; determining a second conversational response based on the respective scores associated with the second plurality of suggested conversational responses, the second conversational response being based on both the first suggested conversational response from the search data provider and the first suggested conversational response from the task data provider; and providing the second conversational response to the user device.

6. The system of claim 5, wherein, Determining the respective scores associated with the conversational responses is based on a user personalization model.

7. The system of claim 5, wherein, Determining the respective scores associated with the conversational responses is based on respective confidence scores included with the conversational responses received from one of the data providers.

8. The system of claim 5, wherein, the one or more computers are further configured to perform operations comprising: updating a state of the first conversation in response to the one or more first conversational responses to the first voice input; and providing the updated state to the plurality of data providers as context for analyzing the second voice input.

9. One or more non-transitory computer storage media encoded with computer program instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: receiving, from a user device, a request associated with performance of a first task, wherein the request includes a first voice input of a user of the user device; submitting the request to each of a plurality of different data providers, wherein each data provider is associated with a different data model configured to interpret a particular type of voice input, and wherein the plurality of different data providers includes (i) a task data provider that directs a conversation toward information for completing a task and (ii) a search data provider; receiving, from two or more of the data providers, a first plurality of suggested conversational responses to the first voice input in response to the first voice input; determining, from the first plurality of suggested conversational responses, a conversational intent of the first voice input and a corresponding first conversation of the first task that includes one or more first conversational responses to provide to the user device to complete the first task; receiving a second voice input in response to providing the one or more first conversational responses to the user and submitting the second voice input to each of the plurality of data providers; receiving, from two or more of the data providers, a second plurality of suggested conversational responses to the second voice input, the second plurality of suggested conversational responses including a first suggested conversational response from the search data provider and a first suggested conversational response from the task data provider, the first suggested conversational response from the task data provider being based on the first suggested conversational response from the search data provider; determining, for each of the second plurality of suggested conversational responses, a respective score associated with the conversational response; determining a second conversational response based on the respective scores associated with the second plurality of suggested conversational responses, the second conversational response being based on both the first suggested conversational response from the search data provider and the first suggested conversational response from the task data provider; and providing the second conversational response to the user device.

10. The one or more non-transitory computer storage media of claim 9, wherein, Determining the respective scores associated with the conversational responses is based on a user personalization model.

11. The one or more non-transitory computer storage media of claim 9, wherein, Determining the respective scores associated with the conversational responses is based on respective confidence scores included with the conversational responses received from one of the data providers.

12. The one or more non-transitory computer storage media of claim 9, further comprising computer program instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: updating a state of the first conversation in response to the one or more first conversational responses to the first voice input; and providing the updated state to the plurality of data providers as context for analyzing the second voice input.

13. A method for managing a conversation, comprising: receiving, at a conversational system from a user device, a request associated with performance of a first task, wherein the request includes a first voice input of a user of the user device; submitting the request to each of a plurality of different data providers, wherein each data provider is associated with a different data model configured to interpret a particular type of voice input; receiving, from two or more of the data providers, a first plurality of suggested dialog responses to the first voice input in response to the first voice input; determining, from the first plurality of suggested dialog responses, a dialog intent of the first voice input and a corresponding first dialog of the first task that includes one or more first dialog responses to provide to the user device to complete the first task; receiving a second voice input at the dialog system in response to providing the one or more first dialog responses to the user and submitting the second voice input to each of the plurality of data providers; receiving, from two or more of the data providers, a second plurality of suggested dialog responses to the second voice input; receiving, from a first data provider, an enhanced response in response to one of the second plurality of suggested dialog responses of a second data provider, the enhanced response including a modification or addition based on disambiguation of an entity included in the suggested dialog response provided by the second data provider; and determining a second dialog response includes determining a combined response that includes the suggested dialog response of the second data provider and the enhanced response of the first data provider and providing the combined response to the user device.

14. The method of claim 13, comprising: updating a state of the first dialog in response to the one or more first dialog responses to the first voice input; and providing the updated state to the plurality of data providers as context for analyzing the second voice input.

15. The method of claim 13, wherein, the first dialog is generated by a first data provider and the second dialog is generated by a second data provider of the plurality of data providers.

16. The method of claim 13, further comprising: combining dialog responses from a first data provider and a second data provider to provide to the user device, wherein the combined dialog responses are associated with the first task and a second task determined from the second voice input.

17. The method of claim 16, further comprising: updating a state of both the first dialog and the second dialog in response to a third voice input, the third voice input received from the user device in response to the combined dialog responses.

18. The method of claim 13, wherein, scoring each of the first plurality of suggested dialog responses, and wherein the particular dialog response corresponding to the first task is determined based on the scoring.

19. The method of claim 13, further comprising receiving a non-enhanced response from the first data provider, the non-enhanced response ignoring the suggested dialog response of the second data provider.

20. The method of claim 13, wherein: each of the first plurality of suggested dialog responses includes a respective confidence score provided by a respective data provider of the plurality of data providers; and the one or more first dialog responses in the corresponding first dialog are selected based on the respective confidence scores.

21. The method of claim 13, wherein: the first dialog is generated by a first data provider and the second dialog is generated by a second data provider of the plurality of data providers.

22. The method of claim 13, further comprising: combining dialog responses from a first data provider and a second data provider to provide to the user device, wherein the combined dialog responses are associated with the first task and a second task determined from the second voice input.

23. The method of claim 22, further comprising: updating a state of both the first dialog and the second dialog in response to a third voice input, the third voice input received from the user device in response to the combined dialog responses.

24. The method of claim 13, wherein: each of the first plurality of suggested dialog responses is scored, and wherein the particular dialog response corresponding to the first task is determined based on the scoring.

25. The method of claim 13, further comprising receiving a non-enhanced response from the first data provider, the non-enhanced response ignoring the suggested dialog response of the second data provider.

26. The method of claim 13, wherein: each of the first plurality of suggested dialog responses includes a respective confidence score provided by a respective data provider of the plurality of data providers; and the one or more first dialog responses in the corresponding first dialog are selected based on the respective confidence scores.

21. A dialog system comprising: a user device; and one or more computers configured to interact with the user device and perform operations comprising: receiving, at the dialog system from the user device, a request associated with performance of a first task, wherein the request comprises a first speech input of a user of the user device; submitting the request to each of a plurality of different data providers, wherein each data provider is associated with a different data model configured to interpret a particular type of speech input; receiving, from two or more of the data providers, a first plurality of suggested dialog responses to the first speech input in response to the first speech input; determining, from the first plurality of suggested dialog responses, a dialog intent of the first speech input and a corresponding first dialog of the first task comprising one or more first dialog responses to provide to the user device to complete the first dialog of the first task; receiving, at the dialog system, a second speech input in response to providing the one or more first dialog responses to the user and submitting the second speech input to each of the plurality of data providers; receiving, from two or more of the data providers, a second plurality of suggested dialog responses to the second speech input; receiving, from a first data provider, an enhanced response in response to one of the second plurality of suggested dialog responses of a second data provider, the enhanced response comprising a modification or addition based on disambiguation of an entity included in the suggested dialog response provided by the second data provider; and determining a second dialog response comprises determining a combined response comprising the suggested dialog response of the second data provider and the enhanced response of the first data provider and providing the combined response to the user device.

22. One or more non-transitory computer storage media encoded with computer program instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: receiving, at a dialog system from a user device, a request associated with performance of a first task, wherein the request comprises a first speech input of a user of the user device; submitting the request to each of a plurality of different data providers, wherein each data provider is associated with a different data model configured to interpret a particular type of speech input; receiving, from two or more of the data providers, a first plurality of suggested dialog responses to the first speech input in response to the first speech input; determining, from the first plurality of suggested dialog responses, a dialog intent of the first speech input and a corresponding first dialog of the first task comprising one or more first dialog responses to provide to the user device to complete the first dialog of the first task; receiving a second speech input at the dialog system in response to providing the one or more first dialog responses to the user and submitting the second speech input to each of the plurality of data providers; receiving a second plurality of suggested dialog responses to the second speech input from two or more of the data providers; receiving an enhanced response from a first data provider in response to one of the second plurality of suggested dialog responses of a second data provider, the enhanced response including a modification or addition based on disambiguation of an entity included in the suggested dialog response provided by the second data provider; and determining a second dialog response includes determining a combined response including the suggested dialog response of the second data provider and the enhanced response of the first data provider and providing the combined response to the user device.

Images