Language model prediction for API calls and voice responses

Abstract

The method includes obtaining an utterance from a user directed to a digital assistant. The method includes generating a first predicted string using a language model based on the utterance, and determining via an API whether the first predicted string includes an application programming interface (API) call for invoking a program. If the first predicted string includes an API call, the method includes calling the program via the API to obtain a program result, receiving the program result, updating a conversation context with the program result including the utterance, and using the language model to generate a second predicted string based on the updated conversation context. If the first predicted string does not include an API call, the method includes providing a speech response to the utterance based on the first predicted string.

Description

[Technical field]

[0001] This disclosure relates to language model prediction of API call invocations and verbal responses. [Background technology]

[0002] Task-based dialogue systems (e.g., ticketing, shopping, reservations, food ordering, etc.) aim to solve the complex problem of responding naturally and relevantly to any user input based on the current conversation context, while steering the conversation towards task accomplishment (e.g., purchasing, confirming a reservation, etc.). Often, a response to the user (e.g., "Which movie tickets do you need?") is required to move the conversation further. However, in some scenarios, the conversation context requires the task-based dialogue system to perform information lookup, for example from an external data source. The system can compile the results in a response to the user. Summary of the Invention

[0003] One aspect of the disclosure provides a computer-implemented method for a language prediction model that, when executed on data processing hardware, causes the data processing hardware to perform an operation. The operation includes obtaining an utterance from a user, the utterance including a user query directed to a digital assistant. The operation also includes using the language model to generate a first predicted string based on the utterance. The operation also includes determining whether the first predicted string includes an API call to invoke a program through an application programming interface (API). When the first predicted string includes an API call to invoke the program, the operation includes calling the program through the API using the API call to obtain a program result responsive to the utterance, receiving the program result through the API, updating a conversation context with the program result, the conversation context including the utterance, and using the language model to generate a second predicted string based on the updated conversation context. When the first predicted string does not include the API call, the operation includes providing a speech response to the user based on the first predicted string for output from the user device.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operations further include, if the first predicted string includes an API call to invoke the program, determining whether the second predicted string includes an API call to invoke the program via the API, and, if the second predicted string does not include an API call, providing a second speech response to the user based on the second predicted string for output from the user device. In these implementations, the operations may further include, after providing the second speech response to the user, obtaining additional utterances from the user directed to the digital assistant, further updating the updated conversation context with the additional utterances from the user, and using the language model to generate a third predicted string based on the further updated conversation context.

[0005] In some examples, obtaining the utterance includes obtaining a transcription of a verbal utterance spoken by a user and captured by a user device in the streaming audio. Optionally, obtaining the transcription may include receiving a transcription of the verbal utterance from a digital assistant, where the digital assistant generates the transcription by performing speech recognition on audio data characterizing the verbal utterance. In these examples, obtaining the transcription may include receiving audio data characterizing the verbal utterance and performing speech recognition on the audio data characterizing the verbal utterance to generate the transcription.

[0006] In some implementations, obtaining the utterance includes receiving a textual representation of the utterance entered by the user via the user device. The speech response may include an audible representation including a synthesized voice that is audibly output from the digital assistant. Optionally, the speech response may include a textual representation that is displayed on a graphical user interface (GUI) executed on the digital assistant.

[0007] In some examples, the language model includes a pre-trained language model that is fine-tuned using labeled training samples. In these examples, the labeled training samples may include annotated API call invocations. Optionally, the operations further include, if the first predicted string includes an API call for invoking a program via an API, determining from the first predicted string one or more parameters for the API call and at least one value for each of the one or more parameters. In some implementations, the language model includes a single end-to-end machine learning model. The language model may include a text-to-text transfer transformer language model.

[0008] Another aspect of the disclosure provides a system for language model prediction. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include obtaining an utterance from a user, the utterance including a user query directed to a digital assistant. The operations also include using the language model to generate a first predicted string based on the utterance. The operations include determining whether the first predicted string includes an API call for invoking a program through an application programming interface (API). If the first predicted string includes an API call for invoking the program, the operations include calling the program through the API using the API call to obtain a program result responsive to the utterance, receiving the program result through the API, updating a conversation context with the program result, the conversation context including the utterance, and using the language model to generate a second predicted string based on the updated conversation context. If the first predicted string does not include the API call, the operations include providing a speech response to the user based on the first predicted string for output from the user device.

[0009] This aspect may include one or more of the following optional features. In some implementations, the operations further include, if the first predicted string includes an API call to invoke the program, determining whether the second predicted string includes an API call to invoke the program via the API, and, if the second predicted string does not include an API call, providing a second speech response to the user based on the second predicted string for output from the user device. In these implementations, the operations may further include, after providing the second speech response to the user, obtaining additional utterances from the user directed to the digital assistant, further updating the updated conversation context with the additional utterances from the user, and using the language model to generate a third predicted string based on the further updated conversation context.

[0010] In some examples, obtaining the utterance includes obtaining a transcription of a voice utterance spoken by a user and captured by a user device in the streaming audio. Optionally, obtaining the transcription may include receiving a transcription of the voice utterance from a digital assistant, where the digital assistant generates the transcription by performing speech recognition on audio data characterizing the voice utterance. In these examples, obtaining the transcription may include receiving audio data characterizing the voice utterance and performing speech recognition on the audio data characterizing the voice utterance to generate the transcription.

[0011] In some implementations, obtaining the utterance includes receiving a textual representation of the utterance entered by the user via the user device. The speech response may include an audible representation including a synthesized voice that is audibly output from the digital assistant. Optionally, the speech response includes a textual representation that is displayed on a graphical user interface (GUI) executed on the digital assistant.

[0012] In some examples, the language model includes a pre-trained language model that is fine-tuned using labeled training samples. In these examples, the labeled training samples may include annotated API call invocations. Optionally, the operations further include, if the first predicted string includes an API call for invoking a program via an API, determining from the first predicted string one or more parameters for the API call and at least one value for each of the one or more parameters. In some implementations, the language model includes a single end-to-end machine learning model. The language model may include a text-to-text transfer transformer language model.

[0013] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic diagram of an example system for language model prediction using both application programming interface (API) call invocation and voice response. [Diagram 2] FIG. 2 is a schematic diagram of exemplary components of the system of FIG. 1. [Diagram 3] 1 is an exemplary flow chart of a conversational turn in a conversational context. [Figure 4] FIG. 2 is a schematic diagram of the training process for the language prediction model of FIG. [Diagram 5] 1 is a flowchart of an exemplary configuration of operation of a method of language model prediction using API call invocation and voice response. [Figure 6] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Like reference symbols in the various drawings refer to like elements.

[0016] Voice-based interfaces, such as digital assistants, are becoming more prevalent in a variety of devices, including but not limited to mobile phones and smart speakers / displays that include microphones for capturing voice. These assistants may utilize task-based dialogue systems that prompt a user to accomplish a task (e.g., buying movie tickets) in a conversational manner. Traditionally, these systems implement a large number of handwritten rules. The handwritten rules are used to determine whether the system should provide a response to the user or should first obtain data from an external source (e.g., via an application programming interface (API)). For example, the system may include a handwritten rule that instructs the system to access the user's location when the user asks for nearby restaurant suggestions. Not only are these handwritten rules time-consuming to implement manually, but they also significantly limit the scalability and reusability of the system.

[0017] Modern machine learning models are used to train to recognize patterns in data. General-purpose language models are commonly used in applications such as automatic speech recognition (ASR). These models are capable of recognizing and transforming a broad portion of human speech. These models are typically trained on large datasets to enable widespread use. However, tasks common to task-based dialogue systems (e.g., issuing tickets, shopping, making reservations, etc.) involve specific vocabulary and canonical ordering of events that are not commonly found in the datasets used to train general-purpose language models.

[0018] Embodiments herein are directed to a language model prediction system that provides language model predictions of, for example, API call invocations and responses in voice. The system includes a language prediction model that is fine-tuned from a pre-trained large-scale language model using a custom labeled dataset. The language prediction model, based on a conversational context, predicts whether the system should perform an information lookup (e.g., via calling an external program via an API) or whether the system should provide a response to the user (e.g., voice, text, etc.), which may or may not be based on the results of the information lookup via the API call. The language prediction system is capable of predicting both program invocations and responses to the user with high accuracy, thereby simplifying the system architecture and eliminating the need for hand-written rules.

[0019] 1 is an example of a system 100 operating in an audio environment 101. In the audio environment 101, a way for a user 10 to interact with a computing device, such as a user device 110 (e.g., a digital assistant), may be through voice input. The user device 110 is configured to capture sounds (e.g., streaming audio data) from one or more users 10 within the audio environment 101. Here, the streaming audio data may refer to utterances 106 spoken by the user 10 that function as audible queries, commands to the user device 110, or audible communications captured by the user device 110. The voice-enabled system of the user device 110 may act on the queries or commands by responding to the queries and / or causing the commands to be executed / fulfilled by one or more downstream applications.

[0020] The user device 110 may correspond to any computing device associated with the user 10 and capable of receiving audio data or other user input. Some examples of the user device 110 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, Internet of Things (IoT) devices, in-vehicle infotainment systems, smart displays, smart speakers, etc. The user device 110 includes data processing hardware 112 and memory hardware 114 in communication with the data processing hardware 112, storing instructions that, when executed by the data processing hardware 112, cause the data processing hardware 112 to perform one or more operations. The user device 110 further includes an audio system 116 with audio capture devices (e.g., microphones) 116, 116a for capturing and converting spoken speech 106 into electrical signals within the audio environment 101, and audio output devices (e.g., speakers) 116, 116b for communicating audible audio signals (e.g., as output audio data from the user device 110). In the illustrated example, the user device 110 implements a single audio capture device 116a, but the user device 110 may implement an array of audio capture devices 116a without departing from the scope of this disclosure, where one or more capture devices 116a of the array may communicate with the audio system 116 without being physically resident at the user device 110.

[0021] In the speech environment 101, a language prediction system 150 implementing the language prediction model 210 resides on the user device 110 of the user 10 and / or on a remote computing device 140 (e.g., one or more remote servers of a distributed system executing in a cloud computing environment) that communicates with the user device 110 via a network 130. In some implementations, the user device 110 communicates with the remote system 140 via the network 130. The remote system 140 may include remote resources 142, such as remote data processing hardware 144 (e.g., a remote server or CPU) and / or remote memory hardware 146 (e.g., a remote database or other storage hardware). The user device 110 may utilize the remote resources 142 to perform various functions related to language prediction. For example, some or all of the language prediction system 150 may reside on the remote system 140. In one example, the language prediction system 150 resides on the user device 110, while in another example, the language prediction system 150 resides on a remote system and provides server-side language prediction. In yet another example, the functionality of the language prediction system 150 is divided across the user device 110 and the server 140 .

[0022] The language prediction system 150 converts the utterance 106 into a corresponding digital representation 212 (e.g., a textual representation such as a string of characters), for example via an ASR system (not shown), and provides the digital representation 212 to the language prediction model 210. In some examples, the digital representation 212 is a transcription of the voice utterance 106 provided by the user 10 and directed to the user device 50 and / or the digital assistant application 50 executing on the remote system 140. That is, the ASR system can generate the transcription by performing speech recognition on audio data characterizing the voice utterance 106 spoken by the user 10 and captured by the user device 110. For example, the ASR system receives audio data characterizing the voice utterance 106 from the user device 110 and performs speech recognition on the audio data characterizing the voice utterance to generate the transcription (i.e., the digital representation 212).

[0023] The language prediction model 210 may include an end-to-end machine learning model (e.g., a text-to-text transfer transformer (T5) language model) that predicts or generates the predicted strings 214, 214a-n. In the illustrated example, the user 10 speaks a respective utterance 106, and the language prediction system 150 converts the utterance 106 into a digital representation 212 for input to the language prediction model 210. The language prediction model 210 then uses the digital representation 212 to generate / predict a first predicted string 214a based on the utterance 106 as an output.

[0024] In some examples, such as FIG. 1, the user 10 interacts with a program or application 50 (e.g., a digital assistant application 50) executing on the user device 110 that uses and / or communicates with a language prediction system 150. For example, the assistant application 50 obtains or receives a digital representation 212 of an utterance 106 that the user directed to the digital assistant application 50 and provides the digital representation 212 to the language prediction system 150 (e.g., executing on the user device 110 and / or the remote system 140). The digital assistant application 50 may display a transcription 120 corresponding to the user utterance 106. Similarly, the digital assistant application 50 may execute on the remote system 140 and the user device 110 may interact with the digital assistant application 50 via a web-based interface.

[0025] In some implementations, the language prediction system 150 includes a prediction controller 220. The prediction controller 220 receives each predicted string 214 generated by the language prediction model 210. In some examples, the prediction controller 220 determines, for each respective predicted string 214 generated by the language prediction model 210, whether the respective predicted string 214 includes an API call 222 for invoking a program 60, 60a-n via an API 62, 62a-n (or any other means for requesting / obtaining data external to the language prediction system 150 and / or the language prediction model 210, e.g., function call, process call, etc.). Each API 62 defines a software-to-software interface that allows the prediction controller 220 to request information from a corresponding program 60 via the API call 222 in response to the utterance 106. The predicted string 214, in some examples, includes one or more parameters and at least one value for each parameter of the API call 222. A value is provided for the parameter of the API call 222 when the corresponding API 62 is invoked. Thus, for each API call 222, the predictive controller 220 may also determine the parameters and values ​​necessary to use the API call 222. For example, an API call 222 for a restaurant reservation system may include a parameter for the number of guests and a value (e.g., 4) that satisfies that parameter.

[0026] When each prediction string 214 includes an API call 222, the prediction controller 220 uses the API call 222 to invoke a corresponding program 60 via a corresponding API 62. For example, the prediction controller 220 determines whether a predetermined sequence of characters or symbols is present within the prediction string 214 that specifies the API call 222. In some examples, the prediction controller 220 performs one or more transformations or modifications on the API call 222 to put the API call 222 in an appropriate format for the corresponding API 62. That is, the prediction string 214 may include a generic, abstract, or other non-conforming format of the API call 222, and the prediction controller 220 transforms the abstract API call 222 to conform to the specific requirements of the API 62 to invoke the program 60. For example, the prediction string 214 may convey an API call 222 that queries local movie theater show times in the natural language of the user 10. While the prediction string 214 may include abstract (natural language) API calls 222 to query for showtimes, the prediction controller 220 translates, updates, or modifies the API calls 222 to match the API calls 222 for the particular API 62 being invoked, for example, based on the user's location (e.g., which theaters are queried based on the user's location). This allows the language prediction model 210 to be scalable and reusable across a wide variety of applications, users 10, and user devices 110. For example, the same language prediction model 210 may be used for different task-related schemes by simply modifying or updating the prediction controller 220 (e.g., by replacing and / or customizing the APIs based on real-world constraints and requirements).

[0027] After the prediction controller 220 uses the API call 222 to call the corresponding program 60 via the corresponding API 62 (i.e., when the prediction string 214 includes the API call 222), the prediction controller 220 receives the program result 224. The program result 224 represents data received from the program 60 in response to the program 60 responding to the API call 222. Using the received program result 224, the prediction controller 220 updates the conversation context 250. As described in more detail below, the conversation context 250 provides a context for all interactions (user utterances 106 and assistant responses 226) between the user 10 and the digital assistant application 50 during the dialogue session, along with all API calls 222 and / or program results 224 sent or received by the prediction controller 220 since the initial utterance 106 from the user 10. In some examples, the conversation context 250 includes a string that the prediction controller updates by appending the program result 224 and / or the digital representation 212 of the utterance 106 received from the user 10.

[0028] After updating the conversation context 250 with the program results 224, the prediction controller 220 provides the updated conversation context 250 to the language prediction model 210. The language prediction model 210 uses the updated conversation context 250 as an input to generate a new predicted string 214. The new predicted string 214, in some examples, includes another API call 222. In these examples, the prediction controller 220 again calls the respective API 62 using the API call 222 of the new predicted string 214. In other examples, described in more detail below, the new predicted string 214 does not include an API call 222.

[0029] If the predicted string 214 does not include the API call 222 (i.e., the language prediction model 210 generates an output that does not include a reference to the API call 222), the prediction controller 220 provides a speech response 226 based on the predicted string 214 for output from the user device 110. That is, the digital assistant application 50 delivers the speech response 226 as output from the user device 110 to the user 10. For example, the digital assistant application 50 can cause the user device 110 to visually display a textual representation of the speech response 226 in a graphical user interface (GUI) 118 displayed on a screen of the user device 110 and / or the digital assistant application 50 to audibly provide the speech response 226 to the user 10 (e.g., via one or more speakers of the user device 110). Thus, the prediction controller 220 determines whether the output from the language prediction model 210 (i.e., the predicted string 214) includes the API call 222 (or a similar data request) or whether the output does not include the API call 222. If the prediction string 214 includes an API call 222, the prediction controller 220 executes steps to call the appropriate API 62 via the API call 222 and to call the respective program 60 to obtain data. If the prediction string 214 does not include an API call 222, the prediction controller 220 generates and provides a speech response 226 for the digital assistant application 50 to the user 10. The speech response 226 further accomplishes a task specified by the user 10 (e.g., via the utterance 106), as described in more detail below.

[0030] In the illustrated example, a user 10 interacts with a digital assistant application 50. For example, FIG. 1 shows a user 10 communicating with a digital assistant application 50 and a GUI 118 displayed on the screen of a user device 110 to show a conversation between the user 10 and the digital assistant application 50. In this example, the user 10 asks the digital assistant application 50, "Hello, I need two tickets to Alien Invasion 3. Is it playing near me?" This question from the user 10 is a spoken utterance 106 that is captured by an audio capture device 116a and processed by an audio system 116 of the user device 110.

[0031] Continuing with this example, the language prediction model 210 receives a digital representation 212 of the user utterance 106 via the language prediction system 150. The language prediction model 210 generates one or more predicted strings 214 based on the digital representation 212 and a conversation context 250 provided by the prediction controller 220. Here, the language prediction model 210 generates a predicted string 214 that does not include an API call 222, which causes the prediction controller 220 to send a spoken response 226 to the digital assistant application 50. In this example, the digital assistant application 50 displays a textual representation of the spoken response 226 in the interface 118. Here, the response includes, "Yes, you can. Just to confirm, would you like two tickets to see Alien Invasion 3 at the Surfside Theatre tonight at 6:05 p.m." Additionally or alternatively, the digital assistant application 50 provides an audible representation of the spoken response 226 for the user 10 to hear.

[0032] 2, in some implementations, the language prediction model 210 generates a predicted string 214 for each of one or more conversational turns 310. Each conversational turn 310 defines a language prediction model 210 that receives a conversational context 250 and generates a corresponding predicted string 214 based on the received conversational context 250. The prediction controller 220 receives the predicted string 214 and, if the predicted string 214 includes an API call 222 that responds to the utterance 106, retrieves the program result 224 and updates the conversational context 250 with the program result 224. The prediction controller 220 provides the updated conversational context 250 back to the language prediction model 210 to initiate the next conversational turn 310 and generate the next predicted string 214. If the predicted string 214 does not include an API call 222, the prediction controller 220 instead provides a speech response 226 to the user 10. In response to the speech response 226, the user 10 can provide another utterance 106, which the language prediction system 150 (e.g., via the prediction controller 220 or any other module) uses to update the conversation context 250 and initiate the next conversation turn 310 (e.g., by providing the updated conversation context 250 to the language prediction model 210).

[0033] As shown in the schematic diagram 200 of FIG. 2, the conversational turn 310 provides a "loop" that continuously updates the conversational context 250 provided to the language prediction model 210 to generate the predicted string 214. A first loop or a first type of conversational turn 310, 310A occurs when the prediction controller 220 updates the conversational context 250 with the program result 224 from the program 60. A second loop or a second type of conversational turn 310, 310B occurs when the prediction controller 220 and / or the language prediction system 150 updates the conversational context 250 with an utterance 106 from the user 10. Of note, the first utterance 106 received from the user 10 (i.e., an initial query by the user 10 directed to the digital assistant application 50) may initiate or establish a first instance of the conversational context 250. As the language prediction system 150 moves toward accomplishing any task specified by the user 10 in the utterance 106, the language prediction model 210 receives, for each conversation turn 310, a conversation context 250 that includes all of the context available since the initial query by the user 10 (e.g., the utterance 106 and / or program results 224).

[0034] 3, a flow diagram 300 of conversational turns 310, 310a-f illustrates exemplary responses from the language prediction system 150 as it progresses toward task accomplishment (i.e., buying movie tickets) during an interaction with the user 10. In some implementations, the digital assistant application 50 initiates an interaction with the user 10 via an introductory message 320 that is predefined and not defined by the language prediction model 210. Here, the introductory message 320 presents the user 10 with the message "Hello. To buy movie tickets, first ask about a specific title, movies playing near me, or something else." In this example, in response to the introductory message 320, the user 10 provides an utterance 106, 106a including "Hello, I need two tickets for Alien Invasion 3. Is it playing near me?" Because this utterance 106a is the first substantial input for the language prediction model 210, the first utterance 106a establishes the conversational context 250 and the first conversational turn 310, 310a.

[0035] In response to the first utterance 106a, the language prediction model 210 generates a first predicted string 214 (not shown) including a first API call 222, 222a, where the first API call 222a retrieves data related to the user's current location to a location service. Thus, the prediction controller 220 uses the first API call 222a to call the respective API 62 to invoke the respective program 60. The program 60 responds with a first program result 224, 224a (i.e., an API response). In this example, the first program result 224 includes data related to the user's 10's current location. As described above with respect to Figures 1 and 2, the prediction controller 220 updates the conversation context 250 with the first program result 224a and provides the updated conversation context 250 to the language prediction model 210 to establish the second conversation turn 310, 310b.

[0036] In response to input from the second conversational turn 310, 310b (i.e., the conversational context 250 including the first utterance 106a and the first program result 224a), the language prediction model 210 generates a second predicted string 214 (not shown). Here, the prediction controller 220 determines that the second predicted string 214 includes a second API call 222, 222b. The second API call 222b is for a movie service (e.g., to obtain data regarding the closest location to the user 10's current location where "Alien Invasion 3" is playing). Thus, the prediction controller 220 uses the second API call 222b to call the API 62. The second API call 222b may call the same API 62 as the first API call 222a, or a different API 62. The prediction controller 220 receives the second program result 224, 224b. In this example, the second program outcome 224b includes data communicating that the nearest theater showing the movie "Alien Invasion 3" is the Surfside Theater. The second program outcome 224b establishes a third conversational turn 310, 310c. The prediction controller 220 updates the conversational context 250 with the program outcome 224b and provides the updated conversational context 250 back to the language prediction model 210.

[0037] Continuing with the example of FIG. 3, in response to the updated conversational context 250, the language prediction model 210 generates a third predicted string 214 (not shown) that the prediction controller 220 determines includes a third API call 222, 222c to a theater service. The theater service can provide showtimes at a particular movie theater for a particular movie. The prediction controller 220 calls the theater service using the third API call 222c and receives a third program outcome 224, 224c indicating showtimes for the movie "Alien Invasion 3" at the Surfside Theater. The third program outcome 224c ends a fourth conversational turn 310, 310d.

[0038] The prediction controller 220 again updates the conversation context 250 and provides the updated conversation context 250 to the language prediction model 210. At this time, the language prediction model 210 generates a fourth predicted string 214 (not shown) that does not include the API call 222. Thus, the prediction controller 220 generates a first speech response 226, 226a based on the fourth predicted string 214. In this case, the first speech response 226a includes "Yes, you can. Just to confirm, would you like two tickets to Alien Invasion 3 at the Surfside Theatre today at 6:05 PM?" The first response utterance 226a is based on the fourth predicted string 214 generated by the language prediction model 210, which generates the fourth predicted string 214 based on the conversation context 250 including the first utterance 106a and the three program results 224a-c.

[0039] The user 10 responds to the first speech response 226a by providing a second utterance 106, 106b directed to the digital assistant application 50. In this example, the second utterance 106b includes "Yes, please." The prediction controller 220 updates the conversation context 250 with the second utterance 106b to establish a fifth conversation turn 310, 310e. The updated conversation context 250, in this example, causes the language prediction model 210 to generate a fifth predicted string 214 (not shown) including a fourth API call 222, 222d to the reservation service. The reservation service returns a fourth program result 224, 224d. The prediction controller 220 updates the conversation context 250 to complete a sixth conversation turn 310, 31f. The prediction controller 220 provides the updated conversational context 250 to the language prediction model 210, which results in a second response utterance 225, 226b that includes "Okay, the ticket has been purchased."

[0040] Referring now to FIG. 4, a training process 400 for the language prediction model 210 includes a first training using a generic training sample 410, 410A, a generic language model 420 in some implementations. The generic training sample 410A may include a very large number of samples for training natural language processing (NLP). In some examples, the generic language model 420 is a text-to-text transfer transformer (T5) or similar model (i.e., a model that uses a unified framework to transform text-based language problems into a text-to-text format). By using the text-to-text framework, the generic language model 420, in addition to its loss function and hyperparameters, may be suitable for many (if not all) NLP tasks, such as machine translation, document summarization, question answering, classification tasks, etc. The generic language model 420 may be trained in multiple languages.

[0041] In some examples, to generate the language prediction model 210, the generic language model 420 is "fine-tuned" by further training the generic language model on a smaller set of API training samples 410, 410B (i.e., smaller than the set of generic training samples 410A). In this context, fine-tuning refers to retraining the model using a custom dataset and adjusting the weights accordingly. The API training samples 410B include annotations of API calls that call programs to obtain data external to the language prediction system 150 or the language prediction model 210. For example, one or more of the API training samples 410B include annotated utterances 106 and / or speech responses 226 and corresponding well-formed labels 430 that indicate to the language prediction model 210 the appropriate API calls 222 to generate for each API training sample 410B.

[0042] Thus, the language prediction system 150 and the language prediction model 210 provide an “end-to-end” solution in which the language prediction system 150 adds each conversational turn 310 to the conversational context 250 to generate a predicted string 214 that predicts whether the language prediction system 150 should next execute an API call 222 or provide a spoken response 226 back to the user 10. The language prediction model 210 may generate multiple predicted strings 214 for each conversational turn 310. Each generated predicted string 214 is associated with a respective weight, score, probability, or the like in some implementations. The prediction controller 220 may select the best predicted string 214 based on the corresponding weight, score, or probability. In some examples, the prediction controller 220 selects different API calls 222 for two or more conversational turns 310 in a row to determine a logical and efficient path to accomplishing a task for the user 10. That is, two or more consecutive conversational turns 310 may each include an API call 222 before a spoken response 226 is generated.

[0043] The examples herein are applied to a task-based dialogue system that provides text and / or voice responses to a user to facilitate the accomplishment of a task, such as ordering movie tickets. However, it is understood that these are non-limiting examples and that the language prediction system 150 is applicable to a wide range of tasks (e.g., multimodal user interfaces). For example, the language prediction model may be trained to accurately display results in a visual format or to interact with a web page (e.g., clicking buttons and / or links based on voice input from the user 10). That is, the language prediction system 150 may leverage the language prediction model 210 for any task that requires predicting either an API call invocation or a response to a user.

[0044] 5 is a flowchart of an exemplary configuration of operations of a method 500 of language model prediction of API call invocation and speech response. The computer-implemented method 500, when executed by the data processing hardware 112, 144, causes the data processing hardware 112, 144 to perform operations. The method 500 includes, at operation 502, obtaining an utterance 106 from the user 10, including a user query directed to the digital assistant 50. The method 500 includes, at operation 504, generating a first predicted string 214 based on the utterance 106 using the language prediction model 210. At operation 506, the method 500 includes determining whether the first predicted string 214 includes an API call 222 that invokes the program 60 via the API 62. At operation 508, the method 500 includes calling the program 60 via the API 62 using the API call 222 to obtain a program result 224 in response to the utterance 106 when the first predicted string 214 includes an API call 222 for invoking the program 60. At operation 510, the method 500 includes receiving the program result 224 via the API 62. At operation 512, the method 500 includes updating the conversation context 250 with the program result 224. The conversation context 250 includes the utterance 106. At operation 514, the method 500 includes generating the second predicted string 214 based on the updated conversation context 250 using the language prediction model 210. At operation 516, the method 500 includes providing a speech response 226 to the utterance 106 based on the first predicted string 214 for output from the digital assistant 50 when the first predicted string 214 does not include the API call 222.

[0045] 6 is a schematic diagram of an exemplary computing device 600 that can be used to implement the systems and methods described herein. Computing device 600 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0046] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 that connects to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 that connects to a low-speed bus 670 and storage device 630. Each component 610, 620, 630, 640, 650, and 660 are interconnected using various buses and may be mounted on a common motherboard or otherwise implemented as desired. Processor 610 may process instructions for execution within computing device 600, including instructions stored in memory 620 or storage device 630, to display graphical information for a graphical user interface (GUI) on an external input / output device such as a display 680 connected to high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used as needed, along with multiple memories and memory types. Also, multiple computing devices 600 may be connected, with each device performing some of the required operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0047] The memory 620 stores non-transient information within the computing device 600. The memory 620 may be a computer readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 620 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0048] The storage device 630 can provide mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 can be a computer-readable medium such as a floppy disk device, a hard disk device, an optical disk device, or a number of devices including a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product also includes instructions that, when executed, perform one or more methods as described above. The information carrier is a computer-readable medium or a machine-readable medium such as the memory 620, the storage device 630, or a memory on the processor 610.

[0049] The high-speed controller 640 manages the more bandwidth-intensive operations of the computing device 600, and the low-speed controller 660 manages the less bandwidth-intensive operations. Such allocation of functionality is merely exemplary. In some implementations, the high-speed controller 640 is coupled to a memory 620, a display 680 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to a storage device 630 and a low-speed expansion port 690. The low-speed expansion port 690 may include various communication ports (USB, Bluetooth, Ethernet, wireless Ethernet, etc.) and can connect, via a network adapter or the like, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router.

[0050] The computing device 600 can be implemented in many different forms, as shown in the figure: for example, it can be implemented as a standard server 600a, or multiple times in a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0051] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, which may be specialized or general purpose, coupled to receive data and instructions from the storage system, and to transmit data and instructions to the storage system.

[0052] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0053] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic circuit (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0054] The processes and logic flows described herein can be executed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be executed by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, for example, both general purpose and special purpose processors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operatively coupled to them for receiving and / or transmitting data. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0055] To provide for interaction with a user, one or more aspects of the present disclosure may be implemented in a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or a touch screen for displaying information to the user, and optionally a keyboard and a pointing device (such as a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide for interaction with a user, for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, spoken language, or tactile input. Additionally, the computer may interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request being received from the web browser.

[0056] Although several embodiments have been described, it is understood, of course, that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (500) that, when executed by data processing hardware (112, 144), causes the data processing hardware (112, 144) to perform an operation, comprising: The operation includes: Obtaining an utterance (106) from a user (10) including a user query directed to a digital assistant (50); generating a first predicted string of characters (214) based on the utterance (106) using a language model (210); determining whether the first predicted string (214) includes an application programming interface (API) call (222) for invoking a program (60) via an API (62); If the first predicted string (214) includes the API call (222) for invoking the program (60), calling the program (60) via the API (62) using the API call (222) to obtain a program result (224) responsive to the utterance (106); receiving said program results (224) via said API (62); updating a conversation context (250) with the program results (224), the conversation context (250) including the utterance (106); and generating a second predicted string (214) based on the updated conversation context (250) using the language model (210); and providing a spoken response (226) to the user (10) based on the first predicted string (214) for output from the digital assistant (50) if the first predicted string (214) does not include the API call (222); The method (500).

2. The operation includes, if the first predicted string (214) includes the API call (222) for invoking the program (60), determining whether the second predicted string (214) includes the API call (222) for invoking the program via the API (62); and providing a second spoken response (226) to the user (10) based on the second predicted string (214) for output from the digital assistant (50) if the second predicted string (214) does not include the API call (222); The method of claim 1 , further comprising:

3. The operation includes, after providing the second spoken response (226) to the user (10), Obtaining additional utterances (106b) from the user (10) directed to the digital assistant (50); further updating the updated conversation context (250) with the additional utterances (106b) from the user (10); and generating a third predicted string (214) based on the further updated conversation context (250) using the language model (210); The method (500) of claim 1 or 2, further comprising:

4. The method (500) of any one of claims 1 to 3, wherein obtaining the utterance (106) includes obtaining a transcription (120) of a voice utterance (106) spoken by the user (10) and captured by the digital assistant (50) in streaming audio.

5. Obtaining the transcription (120) includes receiving the transcription (120) of the voice utterance (106) from the digital assistant (50); The digital assistant (50) generates the transcription (120) by performing speech recognition on audio data characterizing the voice utterance (106). The method (500) of any of claims 1 to 4.

6. Obtaining the transcription (120) comprises: receiving audio data characterizing the voice utterance (106); and performing speech recognition on the audio data characterizing the speech utterance (106) to generate the transcription (120); The method (500) of any of claims 1 to 5, comprising:

7. The method (500) of any one of claims 1 to 6, wherein obtaining the utterance (106) includes receiving a textual representation of the utterance from the digital assistant (50), the textual representation of the utterance (106) being input by the user (10) via a user device (110).

8. 8. The method (500) of claim 1, wherein the speech response (226) includes an audible representation including a synthesized voice that is audibly output from the digital assistant (50).

9. 9. The method of claim 1, wherein the speech response includes a textual representation displayed on a graphical user interface executed on the digital assistant.

10. The method (500) of any of claims 1 to 9, wherein the language model (210) comprises a pre-trained language model that has been fine-tuned using labeled training samples.

11. The method (500) of any of claims 1 to 10, wherein the labeled training samples include annotated API call invocations.

12. The operation includes, if the first predicted string (214) includes the API call (222) for invoking the program (60) via the API (62), extracting from the first predicted string (214): One or more parameters for the API call (222); and at least one value for each of the one or more parameters; The method (500) of any of claims 1 to 11, further comprising determining:

13. The method (500) of any of claims 1 to 12, wherein the language model (210) comprises a single end-to-end machine learning model.

14. The method (500) of any of claims 1 to 13, wherein the language model (210) comprises a text-to-text transfer transformer language model.

15. A system (100), comprising: Data processing hardware (112, 144); and memory hardware (114, 146) in communication with said data processing hardware (112, 144), said memory hardware (114, 146) storing instructions that, when executed by said data processing hardware (112, 144), cause said data processing hardware (112, 144) to perform operations; The operation includes: Obtaining an utterance (106) from a user (10) including a user query directed to a digital assistant (50); generating a first predicted string of characters (214) based on the utterance (106) using a language model (210); determining whether the first predicted string (214) includes an application programming interface (API) call (222) for invoking a program (60) via an API (62); If the first predicted string (214) includes the API call (222) for invoking the program (60), calling the program (60) via the API (62) using the API call (222) to obtain a program result (224) responsive to the utterance (106); receiving said program results (224) via said API (62); updating a conversation context (250) with the program results (224), the conversation context (250) including the utterance (106); and generating a second predicted string (214) based on the updated conversation context (250) using the language model (210); and providing a speech response (226) to the utterance (106) based on the first predicted string (214) for output from the user device (110) if the first predicted string (214) does not include the API call (222); A system (100).

16. The operation includes, if the first predicted string (214) includes the API call (222) for invoking the program (60), determining whether the second predicted string (214) includes an API call (222) for invoking the program (60) via the API (62); and providing a second spoken response (226) to the user (10) based on the second predicted string (214) for output from the user device (110) if the second predicted string (214) does not include the API call (222); The system (100) of claim 15, further comprising:

17. The operation includes, after providing the second spoken response (226) to the user (10), Obtaining additional utterances (106b) from the user (10) directed to the digital assistant (50); further updating the updated conversation context (250) with the additional utterances (106b) from the user (10); and generating a third predicted string (214) based on the further updated conversation context (250) using the language model (210); The system (100) of claim 15 or 16, further comprising:

18. 18. The system (100) of claim 15, wherein obtaining the utterance (106) comprises obtaining a transcription (120) of an audio utterance (106) spoken by the user (10) and captured by the user device (110) in streaming audio.

19. Obtaining the transcription (120) includes receiving the transcription (120) of the voice utterance (106) from the digital assistant (50); 19. The system (100) of any of claims 15 to 18, wherein the digital assistant (50) generates the transcription (120) by performing speech recognition on audio data characterizing the voice utterance (106).

20. Obtaining the transcription (120) comprises: receiving audio data characterizing the voice utterance (106); and performing speech recognition on the audio data characterizing the speech utterance (106) to generate the transcription (120); The system (100) of any of claims 15 to 19, comprising:

21. 21. The system (100) of claim 15, wherein obtaining the utterance (106) comprises receiving a textual representation of the utterance (106), the textual representation of the utterance (106) being input by the user (10) via the user device (110).

22. 22. The system (100) of any of claims 15 to 21, wherein the speech response (226) comprises an audible representation comprising a synthesized voice audibly output from the user device (110).

23. 23. The system (100) of any of claims 15 to 22, wherein the speech response (226) comprises a textual representation displayed on a graphical user interface (GUI) (118) executing on the user device (110).

24. 24. The system (100) of any of claims 15 to 23, wherein the language model (210) comprises a pre-trained language model that is fine-tuned using labeled training samples.

25. The system (100) of any of claims 15 to 24, wherein the labeled training samples include annotated API call invocations.

26. The operation includes, if the first predicted string (214) includes the API call (222) for invoking the program (60) via the API (62), extracting from the first predicted string (214): One or more parameters for the API call (222); and at least one value for each of the one or more parameters; The system (100) of any of claims 15 to 25, further comprising determining:

27. 27. The system (100) of any of claims 15 to 26, wherein the language model (210) comprises a single end-to-end machine learning model.

28. 28. The system (100) of any of claims 15 to 27, wherein the language model (210) comprises a text-to-text transfer transformer language model.

Images