Unstructured description-based chatbot deployment technology

Abstract

Embodiments are directed to receiving unstructured, free-form natural language input, generating a chatbot based on the unstructured, free-form natural language input in response to receiving the unstructured, free-form natural language input, and having the chatbot perform a task associated with an entity on behalf of a user. In various embodiments, the unstructured, free-form natural language input conveys details of the task to be performed but does not define any corresponding dialogue state map (e.g., does not define any dialogue states or dialogue state transitions). Nevertheless, the unstructured, free-form natural language input may be used to fine-tune and / or prime machine learning models that can already be utilized in conducting generalized conversations. As a result, chatbots can be generated and deployed in a fast and efficient manner to perform tasks on behalf of a user.

Description

TECHNICAL FIELD

[0001] Humans can participate in human-computer interactions via various computing devices using interactive software applications referred to as "chatbots", "voicebots", "automated assistants", "interactive personal assistants", "intelligent personal assistants", "conversational agents", and the like. As an example, these chatbots may correspond to a machine learning model or a combination of different machine learning models, and may be utilized to perform various tasks on behalf of users. For example, some of these chatbots can conduct conversations with various humans and perform action(s) on behalf of other humans or on behalf of entities. In some of these examples, the conversations may include voice-based conversations, such as conversations conducted locally at a computing device, conversations conducted remotely on multiple computing devices via a telephone network, or other voice-based scenarios. In other examples, the conversations may include text-based conversations, such as text or SMS messaging, conversations conducted via electronic mail, and / or other text-based scenarios.

BACKGROUND ART

[0002] However, some of the functionality of these chatbots may be limited in various ways. For example, some of the functionality of these chatbots may be limited by the predefined intent schema that the chatbot uses to perform an action(s). In other words, if a person participating in a given conversation with a given chatbot provides a spoken utterance that is determined to contain an intent not defined by the predefined intent schema, the given chatbot may fail. Furthermore, existing intent schemas may be modified or new intent schemas may be added to update these chatbots. As another example, some of the functionality of these chatbots may be limited by the example corpus used to train the chatbot. In other words, if a person participating in a given conversation with a given chatbot provides a spoken utterance that was not included in the given corpus of examples, the given chatbot may fail. Furthermore, existing examples in the corpus may be modified or new examples may be added to update these chatbots. However, in both of these examples, there may be instances where an infinite number of intent schemas, and / or various subtle differences in human speech, need to be predefined to make the bot robust and mitigate examples of failure.

[0003] In particular, manually defining and / or refining such intent schemas and / or examples requires a significant use of computational resources. Furthermore, even if a large number of intent schemas and / or examples are defined, a large amount of memory is required to store and / or utilize these large intent schemas for these chatbots, and / or to train these chatbots based on a large number of examples in the corpus. Therefore, intent schemas for rule-based chatbots and examples for example-based chatbots are not practically scalable to the extent that they can learn the subtle differences in human speech. [Overview of the project]

[0004] The embodiments relate to receiving unstructured free-form natural language input, generating a chatbot based on and in response to the unstructured free-form natural language input, and having the chatbot perform tasks associated with an entity on behalf of the user. In some versions of these embodiments, the unstructured free-form natural language input conveys details of the task(s) to be performed but does not define any corresponding dialogue state maps (e.g., it does not define any dialogue states or dialogue state transitions). Furthermore, the unstructured free-form natural language input is not provided according to any schema. Nevertheless, the unstructured free-form natural language input may be used to fine-tune a machine learning model (ML) that is already available for more generalized conversations, and / or the ML model may be used as input throughout the entire machine learning model without fine-tuning. As a result, the chatbot can be generated and deployed in a rapid and efficient manner to perform tasks(s) on behalf of the user.

[0005] For example, suppose unstructured free-form natural language input corresponds to the spoken utterance, "Please check if the restaurant where I had dinner yesterday has my red leather jacket," received on the user's client device. A processor(s) could use various automatic speech recognition (ASR), natural language understanding (NLU), and / or fulfillment technologies to determine that the unstructured free-form natural language input contains a task(s) to be performed by a chatbot. In this example, the task(s) could include, for example, submitting the query, "Did the restaurant where [user] had dinner yesterday have [user]'s red leather jacket?" Furthermore, the processor(s) could generate a chatbot to engage in a corresponding conversation with a representative of the "restaurant" submitting the query, and determine responsive content to be presented to the user on the client device based on the corresponding conversation with the "restaurant" representative. The above example illustrates a relatively simple query (e.g., submitting a query), but it should be understood that this is merely an example, and the same or similar techniques may be used in executing relatively complex tasks that may involve multiple other tasks or subtasks that may be performed during the corresponding conversation.

[0006] In various embodiments, the corresponding conversation may be a voice-based conversation in which the chatbot participates in the corresponding conversation via a phone call or locally on the client device. In these embodiments, the chatbot may be additionally or alternatively referred to as a voicebot. In other embodiments, the corresponding conversation may be a text-based conversation in which the chatbot participates in the corresponding conversation with a text messaging or SMS service, an email service, or other text-based service. Thus, chatbots can be deployed in various environments to participate in corresponding conversations with various entities and perform various tasks. In some implementations, the chatbots generated to perform various tasks can be fine-tuned in an "on-the-fly" manner so that they are generated in response to the chatbot receiving unstructured free-form natural language input. Some chatbots may be used when participating in corresponding conversations with multiple different entities (e.g., in parallel or sequentially), but the corresponding chatbot may be generated based on each instance of unstructured free-form natural language input provided by the user, which may include tasks(s) to be performed on behalf of the user. In other embodiments, the generated chatbot may not be fine-tuned in an "on-the-fly" manner, but may be identified and utilized in response to receiving unstructured, free-form natural language input.

[0007] In various embodiments, the processor(s) may be implemented locally on the user's client device, receiving unstructured, free-form natural language input that conveys the details of the task(s) to be performed. In these embodiments, the processor(s) may retrieve a pre-trained Large-Scale Language Model (LLM) from the client device's on-device storage as an ML model already available for more generalized conversations. Furthermore, the processor(s) may fine-tune the pre-trained LLM based on the unstructured, free-form natural language input to generate a fine-tuned LLM. The processor(s) may then use the fine-tuned LLM as a chatbot that participates in the corresponding conversation on behalf of the user. In other versions of these embodiments, the processor(s) may retrieve a pre-trained LLM from the client device's on-device storage as an ML model already available for more generalized conversations, but may refrain from fine-tuning the pre-trained LLM based on the unstructured, free-form natural language input.

[0008] In other embodiments, the processor(s) may be implemented remotely from the user's client device (for example, in a remote system such as a high-performance server or a cluster of high-performance servers). In these embodiments, the processor(s) may retrieve pre-trained large-scale language models (LLMs) from the remote storage of the remote system as ML models already available for more generalized conversations. Furthermore, the processor(s) may generate fine-tuned LLMs and use these fine-tuned LLMs as chatbots that participate in corresponding conversations on behalf of the user. Continuing the above example, the processor(s) may have the chatbot process the respective data so that queries are submitted during the corresponding conversation, determining whether the user left their jacket at the restaurant where they ate the previous night. In some versions of these embodiments, the processor(s) may retrieve pre-trained LLMs from the remote storage of the remote system as ML models already available for more generalized conversations, but may refrain from fine-tuning the pre-trained LLMs based on unstructured free-form natural language input.

[0009] In particular, pre-trained LLMs can correspond to existing LLMs such as LaMDA, BERT, Meena, GPT-3, and / or any other pre-trained LLMs. These pre-trained LLMs are pre-trained on a vast amount of diverse data and can participate in conversations with users in a natural and intuitive way. However, these LLMs have multiple ML layers and hundreds of millions to tens of billions of ML parameters. Therefore, in embodiments where the fine-tuned chatbot is generated locally on the client device, the pre-trained LLM that is acquired and fine-tuned may be a sparsified version of the pre-trained LLM. In contrast, in embodiments where the fine-tuned chatbot is generated remotely from the client device, the pre-trained LLM that is acquired and fine-tuned may be a non-sparsed version of the pre-trained LLM. A sparsified version of a pre-trained LLM may have fewer ML layers, fewer ML parameters, masked weights, and / or other sparsified aspects to reduce the size of the pre-trained LLM due to various hardware and / or software constraints on the client device compared to the virtually unlimited resources of the remote system.

[0010] In some embodiments, when a chatbot participates in a corresponding conversation, the processor(s) may use a finely tuned LLM (Language-Linguistic Model) corresponding to the chatbot to process task data (e.g., output generated based on processing unstructured free-form natural language input using ASR models(s), NLU models(s), and / or fulfillment models(s) or rules(s)), audio data capturing spoken input from any of the entity's representatives, text data predicted to correspond to audio data capturing any spoken input from any of the entity's representatives, arbitrary conversational context data, and / or any other data described herein to generate output. In other embodiments, when a chatbot participates in a corresponding conversation, the processor(s) may use a pre-trained LLM(s) (e.g., untuned) to process natural language descriptions contained in unstructured free-form natural language input, task data, audio data capturing spoken input from any of the entity's representatives, text data predicted to correspond to audio data capturing any spoken input from any of the entity's representatives, arbitrary conversational context data, and / or any other data described herein to generate output. The output may be, for example, a probability distribution across a vocabulary or sequence of terms and / or phrases. Based on the probability distribution across the vocabulary or sequence of terms and / or phrases, the processor(s) may select instances of text data corresponding to the text and / or speech provided by the chatbot.

[0011] In embodiments where the corresponding conversation is text-based, the processor(s) may cause instances of the text data to be visually rendered for presentation to representatives of the entity on the client device and / or additional client devices of the entity. However, in embodiments where the corresponding conversation is voice-based, the processor(s) may use text-to-speech (TTS) models(s) to cause the chatbot to process instances of the text data to generate instances of synthesized speech audio data that capture synthesized speech corresponding to the text data. Furthermore, the processor(s) may cause instances of synthesized speech audio data to be visually rendered for presentation to representatives of the entity on the client device and / or additional client devices of the entity. In particular, in embodiments where the chatbot corresponds to a pre-trained LLM fine-tuned on unstructured free-form natural language input, the chatbot may generate conversational output that directs attention to tasks(s) performed on unstructured free-form natural language input. Furthermore, in embodiments where the chatbot corresponds to an untuned, pre-trained LLM, the unstructured, free-form natural language input is still applied as input to the entire untuned, pre-trained LLM, so the chatbot can still generate conversational output that directs attention to the task(s) specified by the unstructured, free-form natural language input.

[0012] In various embodiments, the processor(s) may be configured to provide responsive content for presentation to the user of a client device that has provided unstructured, free-form natural language input. The responsive content may be determined based on one or more responses provided by a representative of an entity during a corresponding conversation. Furthermore, the responsive content may include, for example, the corresponding results of one or more tasks determined during the corresponding conversation, a corresponding summary of the corresponding conversation, and / or other content. Continuing the above example, the chatbot may, in response to the responses provided by a representative of an entity, and during the corresponding conversation, determine whether the user left a red leather jacket at the restaurant.

[0013] In various embodiments, during a corresponding conversation, the chatbot may utilize one or more peripheral actions when engaging in the corresponding conversation with a representative of an entity. These peripheral actions may include, for example, a greeting action that allows the chatbot to identify the user and / or identify itself as a chatbot, a hold action that allows the chatbot to pause and resume the corresponding conversation, a exit action that allows the chatbot to end the corresponding conversation with a representative of an entity, and / or other peripheral actions. These peripheral actions are some non-limiting examples of why a pre-trained LLM enables the chatbot to perform generalized conversational patterns, and why unstructured free-form natural language input does not need to specify that the chatbot is capable of performing these generalized conversational patterns. However, a finely tuned chatbot, which is fine-tuned based on unstructured free-form natural language input, can perform tasks on behalf of the user while still being able to perform these generalized conversational patterns.

[0014] In various embodiments, the processor(s) may discard a chatbot generated based on unstructured free-form natural language input in response to a determination that one or more conditions are met. The processor(s) may discard the chatbot based, for example, whether the chatbot successfully performs tasks associated with an entity, whether the chatbot is used to perform tasks associated with additional entities, whether a threshold duration has elapsed since the chatbot was generated, whether a threshold amount of on-device storage on the client device is consumed by the chatbot in embodiments where the chatbot is generated locally on the client device, whether a threshold amount of on-device storage on the client device is available while the chatbot is stored in on-device storage in embodiments where the chatbot is generated locally on the client device, and / or based on other conditions. In other words, the system can balance the performance of the chatbot with how the chatbot affects the client device when deciding whether to discard the chatbot in embodiments where the chatbot is generated locally on the client device.

[0015] By using the technologies described herein, various technical advantages can be achieved. In a non-limiting example, the technologies described herein enable a processor(s) of a client device and / or a remote system to generate a chatbot based on unstructured, free-form natural language input to perform user-specified tasks(s), and / or to utilize an existing chatbot based on unstructured, free-form natural language input to perform user-specified tasks(s). These tasks(s) may be specified by user-provided natural language descriptions(s). This allows the process(s) to generate and deploy the chatbot in a rapid and efficient manner to perform tasks(s) on behalf of the user. Furthermore, in some examples, the chatbot may be discarded after performing tasks(s) on behalf of the user. Thus, in embodiments where the chatbot is implemented locally on the client device, the technologies described herein balance the current and / or future performance of the chatbot with respect to how the chatbot may affect the performance of the client device. Furthermore, in embodiments where the chatbot is implemented remotely from a client device (for example, by a remote system communicatively coupled to the client device), the task(s) may still be performed on behalf of the user while the computing resources are stored on the client device.

[0016] The above description is provided as an overview of only some of the embodiments disclosed herein. These embodiments and other embodiments are described in further detail herein. [Brief explanation of the drawing]

[0017] [Figure 1] Various aspects of this disclosure are described, and block diagrams of exemplary environments in which embodiments disclosed herein may be implemented are shown. [Figure 2]This document presents exemplary process flows for generating chatbots and enabling them to participate in conversations with entities, using various embodiments. [Figure 3] A flowchart is provided illustrating exemplary methods for generating a chatbot locally on a client device and having the chatbot participate in a conversation with an entity, using various embodiments. [Figure 4] A flowchart is shown illustrating exemplary methods for remotely generating a chatbot in a remote system and having the chatbot participate in a conversation with an entity, using various embodiments. [Figure 5A] This document presents a variety of non-limiting, exemplary interactions between a corresponding unstructured, free-form natural language input used to generate a corresponding chatbot, and a corresponding chatbot that performs a corresponding task(s) based on the corresponding unstructured, free-form natural language input. [Figure 5B] This document presents a variety of non-limiting, exemplary interactions between a corresponding unstructured, free-form natural language input used to generate a corresponding chatbot, and a corresponding chatbot that performs a corresponding task(s) based on the corresponding unstructured, free-form natural language input. [Figure 6A] This document presents additional, non-limiting, exemplary interactions between a corresponding unstructured, free-form natural language input used to generate a corresponding chatbot, and a corresponding chatbot that performs a corresponding task(s) based on the corresponding unstructured, free-form natural language input. [Figure 6B] This document presents additional, non-limiting, exemplary interactions between a corresponding unstructured, free-form natural language input used to generate a corresponding chatbot, and a corresponding chatbot that performs a corresponding task(s) based on the corresponding unstructured, free-form natural language input. [Figure 7] This document illustrates exemplary architectures of computing devices in various embodiments. [Modes for carrying out the invention]

[0018] Referring here to Figure 1, a block diagram illustrating various aspects of the present disclosure and an exemplary environment in which embodiments disclosed herein may be implemented is shown. A client device 110 is shown in Figure 1, which in various embodiments includes a user input engine 120, a rendering engine 130, an on-device machine learning (ML) model engine 140, and a chatbot deployment engine client 150. The client device 110 may be, for example, a standalone device (e.g., having a microphone(s), visual components(s), speakers(s), displays(s), and / or other user interface components), a laptop, a desktop computer, a tablet, a wearable computing device, a vehicle computing device, and / or any other client device that can implement the chatbot deployment engine client 150.

[0019] The user input engine 120 can detect various types of user input in the client device 110. In some examples, the user input detected in the client device 110 may include spoken input detected via the client device 110's microphone(s). In these examples, the client device 110's microphone(s) can generate audio data that captures the spoken utterance(s) contained in the spoken input. In other examples, the user input detected in the client device 110 may include touch input detected via the client device 110's user interface input devices(s) (e.g., touch-sensitive displays(s)) and / or typed input detected via the client device 110's user interface input devices(s) (e.g., touch-sensitive displays(s) and / or keyboards(s)). In these examples, the client device 110's user interface input devices(s) can generate text data that captures the touch input and / or typed input. In particular, the unstructured, free-form natural language input described herein may be provided by the user of the client device 110 as any combination of spoken input, touch input, and / or typed input.

[0020] The rendering engine 130 can cause responsive content and / or other outputs to be visually rendered (e.g., via a touch-sensitive display or other user interface output device(s)) for presentation to a user at the client device 110, and / or to be audibly rendered (e.g., via speaker(s) or other user interface output device(s)) for presentation to a user at the client device 110. The responsive content and / or other outputs can include, for example, various types of user interfaces associated with the chatbot deployment engine client 150 that can be visually rendered via the user interface of the client device 110, such as unstructured free-form natural language input provided by the user of the client device 110 that conveys details of the task(s) performed on behalf of the user of the client device by the chatbot, transcript(s) of the corresponding conversation(s) performed by the chatbot on behalf of the user of the client device 110, various prompts associated with the corresponding conversation(s) performed by the chatbot on behalf of the client device 110, result(s) and / or summary of the corresponding conversation(s) performed by the chatbot on behalf of the client device 110, and / or any other responsive content or output that can be visually and / or audibly rendered for presentation to a user at the client device 110.

[0021] The on-device ML model engine 140 may, in various embodiments, include an automatic speech recognition (ASR) engine 141, a natural language understanding (NLU) engine 142, a fulfillment engine 143, and a text-to-speech (TTS) engine 144. As will be described in greater detail below, these on-device ML model engines of the on-device ML model engine 140 utilize various on-device ML models (e.g., stored in the on-device ML model(s) database 140A) to process various user inputs (e.g., received via the user input engine 120) and generate various outputs (that are rendered visually and / or audibly for presentation to the user via the rendering engine 130). This in turn enables the chatbot deployment engine client 150 to leverage the on-device ML model engine 140 in processing various user inputs received at the client device 110 and generating various outputs provided for presentation to the user at the client device 110.

[0022] Further, the client device 110 is shown in FIG. 1 as being communicatively coupled to a remote system 160 via one or more networks 199 (e.g., any combination of Wi-Fi, Bluetooth, or other local area networks (LAN), Ethernet, the Internet, or other wide area networks (WAN) and / or other networks). The remote system 160, in various embodiments, includes a remote system ML model engine 170 and a chatbot deployment engine 180. The remote system 160 may be, for example, a high-performance server, a cluster of high-performance servers, and / or any other computing device that is remote from the client device 110.

[0023] The remote ML model engine 170 may, in various embodiments, include an ASR engine 171, an NLU engine 172, a fulfillment engine 173, and a TTS engine 174. As will be described in more detail below, these remote ML model engines of the remote engine 170 can utilize various remote ML models (for example, stored in a remote ML model database 170A) to process various user inputs (e.g., received via the user input engine 120) based on data provided to the remote system 160 by a client device 110 via one or more networks 199, in the same or similar manner as the on-device ML model engine 140, and generate various outputs (e.g., rendered visually and / or audibly for presentation to the user via the rendering engine 130). This then enables the chatbot deployment engine 180 to leverage the remote ML model engine 170 when processing various user inputs received by the client device 110 and when generating various outputs provided for presentation to the user in the client device 110. In an embodiment in which the remote ML model engine 170 processes various user inputs received by the client device 110 and generates various outputs provided for presentation to the user on the client device 110, the various user inputs received by the client device 110 may be transmitted to the remote system 160 (e.g., via one or more networks 199), and the various user outputs may be sent back to the client device 110 (e.g., via one or more networks 199).

[0024] In particular, the chatbot deployment engine client 150 of the client device can communicate with the chatbot deployment engine 180 via one or more networks 199. The chatbot deployment engine client 150 and the chatbot deployment engine 180 form a logical instance of the chatbot deployment platform from the perspective of a user interacting with the client device 110. In Figure 1, the chatbot deployment platform is shown to be implemented in a distributed manner via one or more networks 199 (for example, using the chatbot deployment engine client 150 and the chatbot deployment engine 180), but this is for illustrative purposes only and should be understood as not being limiting. For example, the chatbot deployment platform could alternatively be implemented on the client device 110 alone. As another example, the chatbot deployment platform could alternatively be implemented on the remote system 160 alone, but the client device 110 would still be used to enable the user to interact with the chatbot deployment platform.

[0025] The chatbot deployment platform can be used by a user (e.g., a user of client device 110) to train the chatbot described herein and deploy it to handle conversations on behalf of the user and / or on behalf of a third party associated with the user (e.g., via a third-party system 192). In particular, the chatbot deployment platform can be provided by a first party, and a user can use the chatbot deployment platform to generate chatbots for themselves or for third parties associated with the user. As used herein, the term first party refers to an entity that exposes the chatbot deployment platform, and the term third party refers to an entity that, unlike entities associated with the first party, does not expose the chatbot deployment system. Thus, a user of client device 110 interacting with the chatbot deployment platform may also be called a third-party deployer.

[0026] The corresponding conversations described herein as being conducted by the chatbot and on behalf of the user of client device 110 may include various types of conversations, such as voice-based conversations and text-based conversations. Voice-based conversations may include, for example, corresponding conversations that take place during automated telephone calls (e.g., Voice over Internet Protocol (VoIP), Public Switched Telephone Network (PSTN), and / or other telephone communication protocols) and between the client device and additional client device(s) 191, and corresponding conversations in which the chatbot participates with other entities and / or users locally on a given client device (e.g., in a scenario where client device 110 is a shared client accessible by multiple users), and / or in any other voice-based scenario in which the chatbot is deployed to conduct corresponding conversations with users. Text-based conversations may include, for example, corresponding conversations that take place during text or SMS messaging, email, and / or in any other text-based scenario in which the chatbot is deployed to conduct corresponding conversations with users.

[0027] As described above, the chatbot deployment platform may leverage the on-device ML model engine 140 and / or the remote system ML model engine 170 when processing various user inputs received on the client device 110 and when generating various outputs to be presented to the user on the client device 110. Each of the sub-engines of the on-device ML model engine 140 and / or the remote system ML model engine 170 may be configured to perform one or more functions. In particular, the remote system ML model engine 170 includes a remote-based counterpart of the sub-engines of the on-device ML model engine 140. In various embodiments, the use of the on-device ML model engine 140 may be prioritized, at least in part, by latency considerations, network bandwidth privacy considerations, and / or other considerations. In these embodiments, the remote system ML model engine 170 may be utilized when one or more of the sub-engines of the on-device ML model engine 140 fail. In other embodiments, the use of the remote ML model engine 170 may be prioritized, at least in part, by computational considerations on the client device 110, hardware considerations on the client device 110, software considerations on the client device 110, and / or other considerations. In yet another embodiment, the on-device ML model engine 140 and the remote system ML model engine 170 may be used in relation to each other.

[0028] For example, the ASR engines 141 and / or 171 may use the ASR models (or multiple models) stored in their respective ML model databases (e.g., recurrent neural network (RNN) models, transformer models, and / or any other type of ML model capable of performing ASR) to process audio data captured from spoken utterances and generated by the microphone(or multiple models) of the client device 110 to generate an ASR output. Furthermore, the NLU engines 142 and / or 172 may use the NLU models (or multiple models) and / or NLU rules (or multiple models) stored in their respective ML model databases (e.g., long short-term memory (LSTM), gated recurrent units (GRU), and / or any other type of RNN or other ML model capable of performing NLU) to process the ASR output (or other type of input or touch input received via the user input engine 120 of the client device 110) to generate an NLU output. Furthermore, the fulfillment engines 143 and / or 173 can process NLU data using the fulfillment models and / or fulfillment rules stored in their respective ML model databases to generate fulfillment output. Finally, the TTS engines 144 and / or 174 can process text data (e.g., text formulated by a chatbot) using the TTS models stored in the ML model databases to generate synthesized speech audio data, including computer-generated synthesized speech.

[0029] In various embodiments, the ASR output may include multiple speech hypotheses (e.g., terminology hypotheses and / or transcription hypotheses) about a spoken input based on, for example, processing of audio data, and optionally, a particular speech hypothesis may be selected as the recognized text of the spoken input text based on corresponding values ​​(e.g., probability values, log-likelihood values, and / or other values) associated with each of the multiple speech hypotheses. In various embodiments, the ASR model(s) stored in each ML model database are end-to-end speech recognition models(s), thereby allowing the ASR engines 141 and / or 171 to directly generate multiple speech hypotheses using the models. For example, the ASR model(s) may be an end-to-end model(s) used to generate each of the multiple speech hypotheses for each character (or other token). One non-restrictive example of such an end-to-end model(s) used to generate recognized text string by string is a recurrent neural network transducer (RNN-T) model. An RNN-T model is a form of inter-sequence model that does not use an attention mechanism. In other embodiments, the ASR model(s) is not an end-to-end speech recognition model(s) such that the ASR engines 141 and / or 171 can instead generate predicted phonemes(s)(and / or other representations). For example, the predicted phonemes(s)(and / or other representations) can then be used by the ASR engines 141 and / or 171 to determine multiple speech hypotheses that fit the predicted phonemes(s). In doing so, the ASR engines 141 and / or 171 can optionally use a decode graph, lexicon, and / or other resources(s). In various embodiments, the corresponding transcript can be rendered on the client device 110 (e.g., in relation to training instance inputs, training instance outputs, corresponding feature enhancement inputs(s), illustrative conversations, and / or other aspects of the chatbot deployment platform).

[0030] In various embodiments, the NLU output may include annotated recognized text, for example, one or more annotations of the recognized text for one or more (e.g., all) of the terms in the recognized text. For example, NLU engines 142 and / or 172 may include a portion of a speech tagger (not shown) configured to annotate terms in their grammatical role. Additionally or alternatively, NLU engines 142 and / or 172 may include an entity tagger (not shown) configured to annotate entity references within one or more segments of the recognized text, such as references to people (e.g., characters in literary works, celebrities, public figures, etc.), organizations, places (real and fictional), etc. In some embodiments, data about entities may be stored in one or more databases, such as a knowledge graph (not shown). In some embodiments, the knowledge graph may include nodes representing known entities (and, optionally, entity attributes), and edges connecting the nodes and representing relationships between entities. The entity tagger may annotate references to entities at a high level of granularity (for example, to enable identification of all references to an entity class such as a person) and / or at a lower level of granularity (for example, to enable identification of all references to a specific entity such as a particular person). The entity tagger may resolve a particular entity depending on the content of the natural language input and / or optionally, to resolve a particular entity by communicating with a knowledge graph or other entity database. Additionally or alternatively, the NLU engines 142 and / or 172 may include a coreference resolver (not shown) configured to group or "cluster" references to the same entity based on one or more context queues.For example, a coreference resolver may be used to resolve the term "they" in the natural language input "buy them" to "buy theater tickets" based on the fact that "theater tickets" is mentioned in a client device notification rendered immediately before the input "buy them" is received. In some embodiments, one or more components of the NLU engine 142 and / or 172 may depend on annotations from one or more other components of the NLU engine 142 and / or 172. For example, in some embodiments, an entity tagger may depend on annotations from a coreference resolver when annotating all references to a particular entity. Also, for example, in some embodiments, a coreference resolver may depend on annotations from an entity tagger when clustering references to the same entity. Also, for example, in some embodiments, a coreference resolver may depend on the user data of the client device 110 (e.g., stored in the user data database 110A) at coreference resolution and / or entity resolution. User data may include, for example, historical location data, historical time data, user preference data, user account data, calendar information, email data, and / or any other user data accessible on the client device 110.

[0031] In various embodiments, the fulfillment output may include one or more tasks performed by the chatbot and on behalf of the user of the client device 110. As described in more detail herein (for example with respect to Figures 2, 3, 4, 5A, 5B, 6A, and 6B), the user can provide unstructured free-form natural language input that includes one or more tasks performed by the chatbot on behalf of the user of the client device 110. One or more tasks may require the chatbot to participate in a corresponding conversation with an entity (or a representative of an entity). In particular, the unstructured free-form natural language input can convey details of one or more tasks performed by the chatbot without a corresponding dialogue state map that the chatbot would use when conducting the corresponding conversation. Nevertheless, by utilizing the chatbot deployment engine client 150 and / or the chatbot deployment engine 180, the chatbot can be generated and deployed to perform one or more tasks in response to receiving unstructured free-form natural language input. Therefore, it should be understood that the fulfillment output may be based on one or more tasks performed by the chatbot, or on responsive content determined during the corresponding conversation with the entity (or a representative of the entity).

[0032] In various embodiments, the TTS engines 144 and / or 174 can generate synthesized speech audio data that captures computer-generated synthesized speech. The synthesized speech audio data can be rendered on client device 110 via the speaker(s) of client device 110 and / or on additional client devices(s) 191 (e.g., client devices associated with entities and / or representatives of entities) via the respective speakers(s) of additional client devices(s). The synthesized speech may include any output generated by the chatbot described herein, for example, synthesized speech generated as part of a conversation between a user of client device 110 and the chatbot, and synthesized speech generated as part of a conversation between an entity (or representative of an entity) and the chatbot, and / or other synthesized speech.

[0033] Figure 1 illustrates a single client device with a single user; however, this is for illustrative purposes only and should not be considered limiting. For example, one or more additional client devices of a user may also implement the techniques described herein. For instance, client device 110, one or more additional client devices, and / or any other computing devices of the user may form an ecosystem of devices that can employ the techniques described herein. These additional client devices and / or computing devices may communicate with client device 110 and / or remote system 160 (e.g., via one or more networks 199). As another example, a given client device may be utilized by multiple users of a shared configuration (e.g., a group of users, a household, etc.).

[0034] The chatbot deployment engine 180 may, in various embodiments, include a chatbot identification engine 181, a chatbot fine-tuning engine 182, a task / entity identification engine 183, a conversation engine 184, a conversation context engine 185, a responsive content engine 186, and a peripheral behavior engine 187, as shown in Figure 1. While the chatbot deployment engine 180 is shown as having certain sub-engines, this is illustrative and not intended to be limiting. For example, one or more of the sub-engines shown in Figure 1 may be combined, and one or more other sub-engines shown in Figure 1 may be omitted. Furthermore, while the chatbot deployment engine client 150 is not shown as including any sub-engines, this is for brevity and not intended to be limiting. For example, the chatbot deployment engine client 150 may include the same sub-engines described with respect to the chatbot deployment engine 180 or a subset thereof. Further descriptions of the chatbot deployment engine 180 and its various sub-engines are provided with respect to Figure 2.

[0035] Referring here to Figure 2, an exemplary process flow 200 is shown for generating a chatbot and getting the chatbot to participate in a conversation with an entity. For illustrative purposes, assume that a user of client device 110 in Figure 1 provides unstructured free-form natural language input 201 as input to client device 110. Client device 110 may receive the unstructured free-form natural language input 201 via the client device 110's user input engine 120. Client device 110 can use various sub-engines of the on-device ML model engine 140 and / or various sub-engines of the remote ML model engine 170 to process the unstructured free-form natural language input 201.

[0036] In particular, when processing unstructured free-form natural language input 201, the client device 110 can identify one or more features 202 based on the output generated by one or more of the various sub-engines of the on-device ML model engine 140 and / or one or more of the various sub-engines of the remote ML model engine 170. One or more features 202 may include, for example, an ASR output if the unstructured free-form natural language input 201 is a spoken input, an NLU output if the unstructured free-form natural language input 201 is a spoken or typed input, and / or a fulfillment output if the unstructured free-form natural language input 201 is a spoken or typed input, such as entities, intents, slot values, tasks associated with entities, and / or other features executed by the chatbot.

[0037] For illustrative purposes, we further assume that the unstructured free-form natural language input 201 is the spoken utterance, “Call the example hotel and ask if there is a small dog pet fee.” In this example, the ASR output may also be the recognized text corresponding to the spoken utterance (e.g., the recognized text “Call the example hotel and ask if there is a small dog pet fee”), and the NLU output may have a slot value “Example Hotel,” which may be the intent “Phone Call” associated with the task of submitting the query “Do you have a small dog pet fee?” to a representative of the “Example Hotel.” Thus, in some of these examples, in response to receiving the unstructured free-form natural language input 201, the client device 110 may generate a chatbot, which is configured to dial the phone number associated with the “Example Hotel” and submit the query “Do you have a small dog pet fee?” to a representative of the “Example Hotel” during the training phase (e.g., enclosed by box 280A in Figure 2).

[0038] During the training phase, the chatbot identification engine 181 can identify the chatbot 203 (for example, stored in the chatbot database 180A). The chatbot 203 may be a pre-trained ML model, or a combination of various pre-trained ML models that can be fine-tuned based on unstructured free-form natural language input 201 and / or one or more features 202 extracted from the unstructured free-form natural language input 201. For example, the chatbot 203 may correspond to a pre-trained large-scale language model (LLM), such as LaMDA, BERT, Meena, GPT-3, and / or other pre-trained LLMs. In particular, these pre-trained LLMs are generative ML models that have been pre-trained on a vast amount of diverse data and can typically engage in conversations with users in a more natural and intuitive way. These LLMs have multiple ML layers and hundreds of millions to tens of billions of ML parameters and can generalize conversations with users. For example, as will be described in more detail herein (with respect to, for example, Figures 5A, 5B, and 6B), text data may be provided as input across these pre-trained LLMs to generate LLM outputs, such as probability distributions across the vocabulary, and responses to the text data may be generated based on these probability distributions across the vocabulary. Due to the multiple ML layers and hundreds of millions to tens of billions of ML parameters, it should be noted that LLMs do not typically contribute to local implementation on the client device 110, such as when the chatbot database 180A is located locally on the client device 110 (e.g., stored in the on-device storage of the client device 110). Nevertheless, by utilizing various sparsification techniques, the number of ML layers and / or ML parameters used by these LLMs can be reduced, thereby allowing sparsified versions of the pre-trained LLMs to be implemented locally on the client device 110 while mitigating the loss of accuracy and / or the retraction of the pre-trained LLMs due to sparsification.These sparsification techniques may include, but are not limited to, aggregating and / or combining multiple layers of multiple ML layers of a pre-trained LLM, pruning multiple layers of multiple ML layers of a pre-trained LLM, masking the weights of a pre-trained LLM, pruning the weights of a pre-trained LLM, and / or other sparsification techniques. However, if the chatbot(s) database 180A is remote from the client device 110 (e.g., stored in remote storage of a remote system 160), a non-sparsed version of the pre-trained LLM may be implemented remotely in the remote system 160. Thus, the chatbot 203 may be identified locally in the client device 110 and / or remotely in the remote system 160 (e.g., remotely from the client device 110 that received the unstructured free-form natural language input 201).

[0039] Furthermore, during the training phase, the chatbot fine-tuning engine 182 may utilize various fine-tuning techniques to generate a fine-tuned chatbot 204 based on the unstructured free-form natural language input 201 and / or one or more features 202 extracted from the unstructured free-form natural language input 201 by fine-tuning the chatbot 203 (and the fine-tuned chatbot 204 may optionally be stored in the chatbot database 180A). These fine-tuning techniques may include, but are not limited to, instruction tuning, fusion learning, and / or other fine-tuning techniques, and the fine-tuning performed may vary based on the unstructured free-form natural language input 201 provided by the user. In other words, the pre-trained LLM corresponding to the chatbot 203 may be further trained based on the unstructured free-form natural language input 201 and / or one or more features 202 extracted from the unstructured free-form natural language input 201. This allows the finely tuned and pre-trained LLM corresponding to the finely tuned chatbot 204 to be adapted to perform tasks(s) on behalf of the user. By fine-tuning chatbot 203, the resulting finely tuned chatbot 204 is also adapted to perform tasks(s) associated with entities on behalf of the user, leveraging the generalization capabilities of the pre-trained LLM. Thus, chatbot 203 can be fine-tuned locally on client device 110 and / or remotely on remote system 160 (e.g., remotely from client device 110 that has received unstructured free-form natural language input 201) to generate the finely tuned chatbot 204. The finely tuned chatbot 204 may then be used during the inference phase (e.g., enclosed by box 280B in Figure 2).

[0040] Figure 2 illustrates how to fine-tune the chatbot 203 based on an unstructured free-form natural language input 201 and / or one or more features 202 of the unstructured free-form natural language input 201 to produce a fine-tuned chatbot 204, but it should be understood that this is only one embodiment contemplated herein. For example, in other embodiments, the chatbot 203 may not need to be fine-tuned during the inference phase (e.g., enclosed by box 280B in Figure 2) so that the chatbot 203 can be subsequently utilized by the client device 110 and / or the remote system 160.

[0041] During the inference phase, the task / entity identification engine 183 may determine that the task data 205 and entity data 206 will be used by the chatbot to perform the task(s) in the corresponding conversation. Continuing the above example where the unstructured free-form natural language input 201 is the spoken utterance "Call the example hotel and ask if there is a small dog pet fee," the task data 205 could include a task to submit, for example, the query "Do you have a small dog pet fee?" to a representative of the "example hotel." Furthermore, the entity data 206 could include a corresponding identifier for the "example hotel," such as a phone number to initiate a call with a representative of the "example hotel." In various embodiments, information about tasks and / or entities may be stored in the task / entity database 180B and / or other data sources accessible by the client device 110. It should be understood that while the task data 205 and entity data 206 are described to include specific data, this is for illustrative purposes only and not intended to be limiting. For example, task data 205 may include any data relating to any task that may be specified by the user of the client device 110 in unstructured free-form natural language input 201. Furthermore, entity data 106 may include references to any entity and / or any corresponding identifiers of entities. Moreover, it should be understood that in the above example, task data 205 includes only a single task and entity data 206 identifies only a single entity, but this is for illustrative purposes only and is not meant to be limiting. For example, task data 205 may include any data for multiple tasks that may be specified by the user of the client device 110 in unstructured free-form natural language input 201.Furthermore, entity data 206 may identify multiple entities belonging to a particular type of entity (for example, if unstructured free-form natural language input 201 instead corresponds to a spoken utterance, "Call a nearby hotel and ask if they have a pet fee for small dogs," where "hotel" is a particular type of entity and may identify a specific hotel geographically close to the user).

[0042] In particular, the task / entity identification engine 183 can use the entity data 206 to initiate a corresponding conversation with a representative of an entity (for example, by making a phone call that dials the phone number associated with “Example Hotel” and makes an outgoing call to a given additional client device 191A via chatbot 203 or a fine-tuned chatbot 204), and can provide task data 205 to the conversation engine 184, which may enable chatbot 203 or a fine-tuned chatbot 204 to participate in a corresponding conversation with a representative of an entity. In various embodiments, the conversation context engine 185 may provide conversation context data 207 to the conversation engine 184, further enabling chatbot 203 or a fine-tuned chatbot 204 to participate in a more contextualized corresponding conversation with a representative of the entity task data 205. In these embodiments, the conversation context data 207 may represent initial context information for the corresponding conversation, or subsequent context information determined during the corresponding conversation (for example, determined based on data stored in the chatbot activity database 180C) (e.g., as a vector or other data structure).

[0043] Continuing the above example where the unstructured free-form natural language input 201 is the spoken utterance "I'm going to call the example hotel and ask if there's a pet fee for a small dog," the conversation context engine 185 can generate conversation context data 207 indicating that "[User] intends to stay at the example hotel," along with information associated with the user's intended stay on the client device 110 (e.g., check-in or check-out date and time from the user's email account on the client device 110, the "Example Hotel" points membership number from the user's email account on the client device 110, or the "Example Hotel" software application accessible on the client device 110), along with "[User] wants to bring their small dog," and other contextual information that can be inferred based on the unstructured free-form natural language input 201 and / or other data accessible on the client device 110 (e.g., via the user data database 110A).

[0044] Furthermore, during the inference phase and after the initiation of correspondence (e.g., using entity data 206), and in embodiments where the chatbot corresponds to a fine-tuned chatbot 204, the conversational engine 184 may first process task data 205 (and optionally conversation context data 207) using the fine-tuned chatbot 204 to generate output such as a probability distribution over a sequence of words or phrases. The conversational engine 184 may generate conversational data 208 based on the output generated using the fine-tuned chatbot 204. The conversational data 208 may include, for example, one or more instances of synthesized speech audio data in embodiments where the corresponding conversation is a voice-based conversation, or one or more instances of text data in embodiments where the corresponding conversation is a text-based conversation. In various embodiments, as shown in Figure 2, the conversational data 208 may be sent to a given additional client device 191A so that the conversational data 208 is rendered audibly and / or visually on the given additional client device 191A. However, in other embodiments, such as when the fine-tuned chatbot 204 participates in a locally corresponding conversation on the client device 110 (for example, when the client device 110 is deployed in a shared configuration), the conversation data 208 may be rendered audibly and / or visually on the client device 110.

[0045] Alternatively, during the inference phase, and in embodiments where the chatbot corresponds to chatbot 203 (and, for example, a fine-tuned chatbot 204), the conversational engine 184 may use chatbot 203 to first process an unstructured free-form natural language input 201, one or more features 202 determined based on the unstructured free-form natural language input 201, task data 205 (and any other data, optionally described herein) to generate an output such as a probability distribution over a sequence of words or phrases. The conversational engine 184 may generate conversational data 208 based on the output generated using chatbot 203. The conversational data 208 may include, for example, instances of synthesized speech audio data in embodiments where the corresponding conversation is a speech-based conversation, or instances of text data in embodiments where the corresponding conversation is a text-based conversation. In various embodiments, as shown in Figure 2, the conversational data 208 may be sent to a given additional client device 191A so that the conversational data 208 can be rendered audibly and / or visually on the given additional client device 191A. However, in other embodiments, such as when the chatbot 203 participates in a corresponding conversation locally on the client device 110 (for example, when the client device 110 is deployed in a shared configuration), the conversation data 208 may be rendered audibly and / or visually on the client device 110. In other words, rather than fine-tuning the chatbot 203 during the training phase, the chatbot 203 may be primed during the inference phase based on one or more features 202 determined based on the unstructured free-form natural language input 201 and / or processing the unstructured free-form natural language input 201. This allows the client device 110 and / or the remote system 160 to conserve computational resources while effectively deploying the chatbot to participate in a corresponding conversation.

[0046] Continuing the above example where the unstructured free-form natural language input 201 is the spoken utterance "I'm going to call an example hotel and ask if they have a small dog pet fee," the conversation data 208 may also include synthesized speech audio data, which will be audibly rendered on a given additional client device 191A, and this will include synthesized speech saying, "Hi, this is the chatbot calling on behalf of [User], do you have a small dog pet fee?" In particular, the synthesized speech includes the query, "Do you have a small dog pet fee?", and seeks a response from a representative of the entity answering the phone call initiated by the chatbot 203 or a fine-tuned chatbot 204. Thus, the response data 209 may include a response to the query contained in the synthesized speech. The response data 209 may include audio data provided by the representative of the entity. In these embodiments, the responsive content engine 186 can use the ML model engine 140 and / or 170 to process various ML models and determine whether the response data 209 contains a response indicating that the task was successfully completed.

[0047] For example, suppose the response data 209 also includes audio data capturing a spoken utterance from a representative of the entity, "Yes, the pet fee for a small dog is $25 per night." In this instance, the responsive content engine 186 can process the audio data provided by the entity's representative (e.g., using an ASR model, NLU model, and / or fulfillment rule for input parsing) and determine that the entity's representative has provided a response to the query. In other words, the responsive content engine 186 can determine the responsive content 210 provided by the entity's representative for the query corresponding to the task in this example (e.g., "The pet fee for a small dog is $25 per night"). Furthermore, the responsive content engine 186 can provide the responsive content 210 to the rendering engine 130 so that the client device can audibly and / or visually present the rendered responsive content 211 to the user. Accordingly, the embodiments described herein enable a user to provide unstructured free-form natural language input 201 to generate a finely tuned bot 204 which can be used to perform the task(s) contained in the unstructured free-form natural language input 201.

[0048] As will be described in more detail herein (for example with respect to Figures 5A, 5B, and 6B), chatbot 203 and the fine-tuned chatbot 204 may have various peripheral operations that can be implemented by chatbot 203 and the fine-tuned chatbot 204 using the peripheral operation engine 187. These peripheral operations may include, but are not limited to, greeting operations that enable chatbot 203 and the fine-tuned chatbot 204 to identify the user of client device 110 and / or identify themselves as chatbots; remote procedure call (RPC) operations that enable chatbot 203 and the fine-tuned chatbot 204 to look up one or more databases during a corresponding conversation; hold operations that enable chatbot 203 and the fine-tuned chatbot 204 to pause and resume a corresponding conversation; exit operations that enable chatbot 203 and the fine-tuned chatbot 204 to prompt the user of client device 110 to join a corresponding conversation and / or to otherwise terminate a corresponding conversation when requested by a representative of an entity; clarification operations that enable chatbot 203 and the fine-tuned chatbot 204 to clarify and / or repeat information previously provided during a corresponding conversation; and / or other peripheral operations that may be invoked by chatbot 203 and the fine-tuned chatbot 204 when conditions for invoking those other peripheral operations are met.

[0049] Figure 2 illustrates a corresponding conversation, which is a telephone call between a chatbot 203 or a fine-tuned chatbot 204 (for example, implemented locally on client device 110 and / or remotely on remote system 160) and a representative of an entity (for example, accessible on a given additional client device 191A), but it should be understood that this is not meant to be limiting. Rather, it should be understood that the techniques described herein can be used to fine-tune chatbots that can be deployed to participate in voice-based and text-based conversations conducted across multiple computing devices and / or across a single computing device.

[0050] Referring here to Figure 3, a flowchart is shown illustrating an exemplary method 300 for generating a chatbot locally on a client device and having the chatbot participate in a conversation with an entity. For convenience, the operation of method 300 is described with reference to the system that performs the operation. This system of method 300 includes at least one processor, memory, and / or other components of a client device (or more) (e.g., client device 110 in Figure 1, computing device 710 in Figure 7, and / or other client devices). Furthermore, the operation of method 300 is shown in a particular order, but this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0051] In block 352, the system receives unstructured free-form natural language input from a user of a client device, the unstructured free-form natural language input including one or more tasks related to an entity. Unstructured free-form natural language input received from a user of a client device may include, for example, spoken input received via the client device's microphone(s), typed input received via the client device's touch-sensitive display, and / or touch input received via the client device's touch-sensitive display. Furthermore, the unstructured free-form natural language input may convey details of one or more tasks performed by the chatbot and on behalf of the user of the client device. In particular, unstructured free-form natural language input is unstructured in the sense that the user is not required to provide free-form natural language input according to any schema or particular method.

[0052] In block 354, the system generates a chatbot that performs one or more tasks related to an entity on behalf of the user, locally on the client device, based on at least unstructured, free-form natural language input. In some embodiments, as shown in block 354A, the system obtains a pre-trained large-scale language model (LLM) that is stored locally on the client device. Furthermore, in these embodiments, as shown in block 354B, the system generates a fine-tuned LLM such that the pre-trained LLM stored locally on the client device is fine-tuned based on unstructured, free-form natural language input. Furthermore, in these embodiments, as shown in block 354C, the system utilizes the fine-tuned LLM as a chatbot. The system can generate a chatbot that performs one or more tasks related to an entity on behalf of the user in the same or similar manner as described above with respect to Figure 2 (for example, in the embodiment described with respect to box 280A, where the training phase is performed locally on the client device). In particular, in these embodiments, the system is implemented locally on the client device, and as a result, the pre-trained LLM may be a sparsed version of the pre-trained LLM, which would otherwise be available (for example, otherwise available on a remote system) due to various hardware and / or software constraints of the client device.

[0053] In block 356, the system causes the chatbot to perform one or more tasks related to an entity on behalf of the user. In some embodiments, as shown in block 356A, the system causes the chatbot to participate in a corresponding conversation with the entity by rendering multiple instances of synthesized speech audio data to be presented to a representative associated with the entity and / or rendering multiple instances of text data to be presented to a representative of the entity. Furthermore, in these embodiments, as shown in block 356B, the system determines responsive content to respond to one or more instances of synthesized speech audio data and / or one or more instances of text data. The system causes the chatbot to perform one or more tasks related to an entity on behalf of the user in the same or similar manner as described above with respect to Figure 2 (for example, in the embodiment described with respect to box 280B) and with respect to Figures 5A, 5C and 6B.

[0054] In block 358, the system determines whether the chatbot has successfully performed one or more tasks associated with the entity. The system may determine whether the chatbot has successfully performed one or more tasks associated with the entity based, for example, on responsive content that responds to one or more instances of synthesized speech audio data and / or one or more instances of text data. Various non-exclusive examples of how the system determines whether the chatbot has successfully performed one or more tasks associated with the entity are described herein (for example, with respect to Figures 5A, 5B, and 5C).

[0055] In an iteration of block 358, if the system determines that the chatbot has successfully performed one or more tasks related to the entity, the system may proceed to block 360. In block 360, the system provides responsive content for presentation to the user of the client device. The responsive content may be determined, for example, based on one or more responses provided by the entity's representative during the corresponding conversation. The system proceeds to block 364, which is described in more detail below.

[0056] If, in an iteration of block 358, the system determines that the chatbot has failed to successfully perform one or more tasks related to the entity, the system may proceed to block 362. In block 362, the system prompts the user to join a corresponding conversation with a representative of the entity. In other words, if the chatbot has failed to successfully perform one or more tasks related to the entity, the chatbot may prompt the user to join a corresponding conversation to ensure that one or more tasks are still being performed. The system then proceeds to block 364.

[0057] In block 364, the system decides whether to discard the chatbot. The system may decide whether to discard the chatbot based on, for example, whether the chatbot successfully performs one or more tasks associated with an entity, whether the chatbot is used to perform one or more tasks associated with an additional entity, whether a threshold duration has elapsed since the chatbot was created, whether the chatbot consumes a threshold amount of the client device's on-device storage, whether the threshold amount of the client device's on-device storage is available while the chatbot is stored in on-device storage, and / or other conditions. In other words, when deciding whether to discard the chatbot, the system can balance the chatbot's performance with how the chatbot affects the client device.

[0058] If, in an iteration of block 364, the system decides not to discard the chatbot, the system may proceed to block 366. In block 366, the system continues to utilize the chatbot. For example, the system may continue to utilize the chatbot if one or more tasks associated with an entity failed to perform successfully, but one or more tasks related to an additional entity could perform successfully. For example, if unstructured free-form natural language input corresponds to the spoken utterance "Find a local plumber available as soon as possible," but the responsive content determined based on calling a first local plumber indicates that the first local plumber is unavailable, the system may continue to utilize the chatbot and call a second local plumber. In this case, the system may call the first and second local plumbers sequentially or in parallel.

[0059] If, in an iteration of block 364, the system decides to discard the chatbot, the system may proceed to block 368. In block 368, the system discards the chatbot. For example, the system may discard the chatbot in response to determining that one or more tasks associated with an entity have been successfully completed, and optionally in response to determining that there are no additional entities to participate in the corresponding conversation. As another example, the system may discard the chatbot if the chatbot consumes more memory resources than a threshold amount on the client device, or if the chatbot makes less memory resources available on the client device than a threshold amount.

[0060] The system can perform other iterations of method 300 based on additional unstructured, free-form natural language input received by the system. Figure 3 illustrates a system implemented locally on a user's client device; however, this is for illustrative purposes only and should not be considered limiting. For example, as shown below with respect to Figure 4, the system may be implemented by a remote system located away from the user's client device that provided the unstructured, free-form natural language input.

[0061] Furthermore, while Method 300 in Figure 3 illustrates a chatbot being a pre-trained LLM fine-tuned based on unstructured free-form natural language input (e.g., block 354), it should be understood that this is illustrative and not limiting. In additional or alternative embodiments, a pre-trained LLM may be used as a chatbot without fine-tuning. In these embodiments, when the chatbot is to participate in a given corresponding conversation with a given additional user (e.g., block 356), the system can prime the pre-trained LLM based on unstructured free-form natural language input. This enables the chatbot to participate in the corresponding conversation without explicit fine-tuning during the training phase.

[0062] Referring here to Figure 4, a flowchart is shown illustrating an exemplary method 400 for remotely generating a chatbot in a remote system and having the chatbot participate in a conversation with an entity. For convenience, the operation of method 400 is described with reference to the system that performs the operation. This system of method 400 includes at least one processor, memory, and / or other components of the remote system(s) (e.g., remote system 160 in Figure 1, computing device 710 in Figure 7, and / or other remote systems). Furthermore, the operation of method 400 is shown in a particular order, but this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0063] In block 452, the system receives instructions for unstructured free-form natural language input provided by the user of a client device, the instructions for unstructured free-form natural language input including one or more tasks related to an entity. Unstructured free-form natural language input received from the user of a client device may include, for example, spoken input received via the microphone(s) of the client device, typed input received via the touch-sensitive display of the client device, and / or touch input received via the touch-sensitive display of the client device. Furthermore, the unstructured free-form natural language input may convey details of one or more tasks to be performed by the chatbot and on behalf of the user of the client device. Furthermore, instructions for unstructured free-form natural language input may be transmitted to the system via one or more networks. In particular, unstructured free-form natural language input is unstructured in the sense that the user does not need to provide free-form natural language input according to any schema or particular method.

[0064] In block 454, the system generates a chatbot that performs one or more tasks related to an entity on behalf of the user, remotely in a remote system, based on instructions from at least unstructured free-form natural language input. In some embodiments, as shown in block 454A, the system retrieves a pre-trained large-scale language model (LLM) that is stored remotely in a remote system. Furthermore, in these embodiments, as shown in block 454B, the system generates a fine-tuned LLM such that the pre-trained LLM stored remotely in a remote system is fine-tuned based on unstructured free-form natural language input. Furthermore, in these embodiments, as shown in block 454C, the system utilizes the fine-tuned LLM as a chatbot. The system can generate a chatbot that performs one or more tasks related to an entity on behalf of the user in the same or similar manner as described above with respect to Figure 2 (for example, in the embodiment described with respect to box 280A, where the training phase is performed remotely in a remote system). In particular, in these embodiments, the system is implemented remotely on a remote system from a client device, and as a result, the pre-trained LLM may be a non-sparsed version of the pre-trained LLM that is more robust than the sparsified version of the pre-trained LLM, due to the virtually absence of hardware and / or software constraints on the remote system.

[0065] In block 456, the system causes the chatbot to perform one or more tasks related to an entity on behalf of the user. In some embodiments, as shown in block 456A, the system causes the chatbot to participate in a corresponding conversation with the entity by rendering multiple instances of synthesized speech audio data to be presented to a representative associated with the entity, and / or rendering multiple instances of text data to be presented to a representative of the entity. Furthermore, in these embodiments, as shown in block 456B, the system determines responsive content to respond to one or more instances of synthesized speech audio data and / or one or more instances of text data. The system causes the chatbot to perform one or more tasks related to an entity on behalf of the user in the same or similar manner as described above with respect to Figure 2 (for example, in the embodiment described with respect to box 280B), and with respect to Figures 5A, 5C, and 6B. In particular, in these embodiments, when participating in a corresponding conversation, the system can communicate directly with additional client devices of the entity's representatives, and in some examples, without interacting with the client devices during the corresponding conversation.

[0066] In block 458, the system determines whether the chatbot has successfully performed one or more tasks associated with the entity. The system may determine whether the chatbot has successfully performed one or more tasks associated with the entity based, for example, on responsive content that responds to one or more instances of synthesized speech audio data and / or one or more instances of text data. Various non-exclusive examples of how the system determines whether the chatbot has successfully performed one or more tasks associated with the entity are described herein (for example, with respect to Figures 5A, 5B, and 5C).

[0067] In an iteration of block 458, if the system determines that the chatbot has successfully performed one or more tasks related to the entity, the system may proceed to block 460. The responsive content may be determined, for example, based on one or more responses provided by the entity's representative during the corresponding conversation. In block 460, the system sends the responsive content to the user's client device. By sending the responsive content to the user's client device, the responsive content is made available for presentation to the user on the client device. The system proceeds to block 464, which is described in more detail below.

[0068] In an iteration of block 458, if the system determines that the chatbot has failed to successfully perform one or more tasks related to the entity, the system may proceed to block 462. In other words, if the chatbot has failed to successfully perform one or more tasks related to the entity, the chatbot may prompt the user to join the corresponding conversation in order to ensure that one or more tasks are still being performed. In block 462, the system sends a prompt to the user's client device for the user to join the corresponding conversation. By sending the prompt to the user's client device, the system ensures that the prompt is available for presentation to the user on the client device. The system proceeds to block 464.

[0069] In block 464, the system decides whether to discard the chatbot. The system may decide whether to discard the chatbot based, for example, on whether the chatbot successfully performs one or more tasks associated with an entity, whether the chatbot is used to perform one or more tasks associated with an additional entity, whether a threshold duration has elapsed since the chatbot was created, and / or on other conditions. In particular, in these embodiments, the chatbot may not be created and / or implemented locally on the user's client device, so the system may not need to balance the chatbot's performance with how the chatbot affects the client device when deciding whether to discard the chatbot.

[0070] If, in an iteration of block 464, the system decides not to discard the chatbot, the system may proceed to block 466. In block 466, the system continues to use the chatbot. For example, the system may continue to use the chatbot if one or more tasks associated with an entity failed to complete successfully, but one or more tasks related to an additional entity could be completed successfully (as illustrated with respect to block 364 in Figure 3). If, in an iteration of block 464, the system decides to discard the chatbot, the system may proceed to block 468. In block 468, the system discards the chatbot. For example, the system may discard the chatbot in response to determining that one or more tasks associated with an entity have completed successfully, and optionally in response to determining that there are no additional entities to participate in the corresponding conversation.

[0071] The system can perform other iterations of Method 400 based on additional instructions from additional unstructured, free-form natural language input received by the system. Figure 4 illustrates a system implemented remotely on a remote system from a user's client device, but it should be understood that this is for illustrative purposes only and not intended to be limiting. For example, as shown above with respect to Figure 3, the system may be implemented on the user's client device that provided the unstructured, free-form natural language input. As another example, the system may be implemented in a distributed manner on both the client device and the remote system. For example, a chatbot may be generated locally on the client device but implemented on a remote system such that the computing resources of the client device are not used to participate in the conversation corresponding to the chatbot. Alternatively, for example, a chatbot may be generated on a remote system but implemented on the client device such that the computing resources of the client device are only used to participate in the conversation corresponding to the chatbot.

[0072] Furthermore, Method 400 in Figure 4 also describes a chatbot that is a pre-trained LLM fine-tuned based on unstructured free-form natural language input (e.g., block 454), but it should be understood that this is for illustrative purposes only and not intended to be limiting. In additional or alternative embodiments, a pre-trained LLM may be used as a chatbot without fine-tuning. In these embodiments, when the chatbot is to participate in a given corresponding conversation with a given additional user (e.g., block 456), the system can prime the pre-trained LLM based on unstructured free-form natural language input. This enables the chatbot to participate in the corresponding conversation without explicit fine-tuning during the training phase.

[0073] Referring here to Figures 5A and 5B, various non-limiting illustrative interactions are shown between a corresponding unstructured free-form natural language input used to generate a corresponding chatbot and a corresponding chatbot that performs a corresponding task(s) based on the corresponding unstructured free-form natural language input. In particular, interactions 500A and 500B described with respect to Figures 5A and 5B may be implemented across multiple computing devices, respectively, to cause the chatbot to perform a corresponding task(s) to be executed. For example, the corresponding unstructured free-form natural language input, as described in the examples of Figures 5A and 5B, may be received on the user's client device (e.g., client device 110 in Figure 1); the chatbot, as described in the examples of Figures 5A and 5B, may be generated on the user's client device (e.g., client device 110 in Figure 1) and / or on a remote system (e.g., remote system 160 in Figure 1); and the chatbot, as described in the examples of Figures 5A and 5B, may be implemented on the user's client device (e.g., client device 110 in Figure 1) and / or on a remote system (e.g., remote system 160 in Figure 1) and communicate with a representative of an entity via an additional computing device of the representative. Each of these computing devices may include its own components, such as user interface input components (e.g., microphones, visual components, presence sensors, touch-sensitive displays, keyboards, hardware buttons, software buttons, etc.), user interface output components (e.g., touch-sensitive displays, speakers, monitors, projectors, etc.), network interfaces, and / or other components. Therefore, it should be understood that interactions 500A and 500B in Figures 5A and 5B are shown within a single interface, respectively, but this is to illustrate, and not to limit, the various technologies described herein.

[0074] Specifically, referring to Figure 5A, assume that an on-device conversation with a user of a client device (e.g., Jane Doe) is initiated as shown by interaction 500A1. In various embodiments, the on-device conversation with Jane Doe may be initiated as part of a dialogue between Jane Doe and an automated assistant running at least partially on the client device. In these embodiments, interaction 500A1 may be a voice-based interaction or a text-based interaction. For example, Jane Doe may invoke an automated assistant running at least partially on the client device (e.g., by activating a software or hardware button, by speaking specific terms or phrases such as “assistant,” “hey assistant,” and similar, and / or by other means) to provide unstructured free-form natural language input 552A1 as spoken input. In another example, Jane Doe may access an automated assistant application that is accessible on the client device and associated with the automated assistant to provide unstructured free-form natural language input 552A1 as typed input.

[0075] For the example in Figure 5A, assume that Jane Doe provides the spoken input 552A1 as unstructured free-form natural language input 552A1: "Please check if I found my red leather jacket at the restaurant where I had dinner yesterday." In this example, the automated assistant can use an ASR model(s) to process the audio data capturing the spoken input to generate an ASR output, such as the recognized text corresponding to the spoken input (e.g., the recognized text: "Please check if I found my red leather jacket at the restaurant where I had dinner yesterday"). Furthermore, the automated assistant can use an NLU model(s) to process the ASR output to generate an NLU output, such as intents(s), slot values(s) of parameters associated with the intents(s), and / or other NLU outputs. In particular, the spoken input explicitly includes a [Query Submission](or similar) intent with a slot value of the [Query Content] parameter (e.g., task data) for "[Jane Doe's] red leather jacket found" (or similar). However, spoken input only implicitly identifies entities (for example, "the restaurant where I had dinner yesterday").

[0076] Therefore, in this example, the automated assistant can use data from various sources to resolve entities implicitly included in the spoken input (e.g., Jane Doe's user data and / or other information). For example, the automated assistant can leverage Jane Doe's historical location information to identify the restaurant where she had dinner yesterday, calendar information including reservations to identify the restaurant where she had dinner yesterday, software application information including reservations to identify the restaurant where she had dinner yesterday, email information including confirmation emails to identify the restaurant where she had dinner yesterday, and / or other information. As a result, let's further assume that "the restaurant where I had dinner yesterday" corresponds to the restaurant entity (e.g., entity data) of "Virtual Cafe". Therefore, the automated assistant can provide the response 554A1, "Okay, call Virtual Cafe and let me know if you need anything or if they've found it," for auditory and / or visual presentation to Jane Doe, indicating that the automated assistant will perform the task of querying the entity of "Virtual Cafe".

[0077] In this example, based on unstructured free-form natural language input 552A1, the automated assistant can cause a chatbot to generate a chatbot that performs the task of submitting queries to a representative of the "virtual cafe" on a client device and / or a remote system. In some embodiments, the chatbot may correspond to a pre-trained LLM that has been fine-tuned based on the unstructured free-form natural language input 552A1 using, for example, various fine-tuning techniques (as described, for example, with respect to Figure 2). In other embodiments, the chatbot may correspond to a pre-trained LLM that has not been fine-tuned based on the unstructured free-form natural language input 552A1, but has been primed based on one or more of several unstructured free-form natural language inputs. Furthermore, assume that the chatbot participates in a corresponding conversation with a representative of the "virtual cafe" as part of a phone call that performs the task of submitting queries to a representative of the "virtual cafe". In this example, the automated assistant may determine a corresponding identifier (e.g., entity data) associated with the "virtual cafe," such as a phone number that can be used to initiate a corresponding conversation with a representative of the "virtual cafe".

[0078] Therefore, as shown in interaction 500A2, the automated assistant can be implemented on a computing device, such as a client device in an embodiment where the chatbot is generated locally on the client device, or a remote system in an embodiment where the chatbot is generated remotely from the client device. This allows the chatbot to facilitate the execution of a task of calling a phone number associated with the "virtual cafe" and submitting a query to a representative of the "virtual cafe". Furthermore, assume that after the call is made, the chatbot and the representative of the "virtual cafe" engage in a corresponding conversation. For example, further assume that the representative of the "virtual cafe" answers the incoming call and provides the spoken input 552A2, "Hello, this is John Smith from the virtual cafe. How can I help you?" In this example, the automated assistant can have the chatbot process a [Submit Query] (or similar) intent with a slot value of "[Have you found Jane Doe's] red leather jacket?" for the [Query Content] parameter (e.g., task data), audio data to capture spoken input 552A2, text data corresponding to the audio data capturing spoken input 552A2 (e.g., determined using an ASR model(s)), and / or any contextual conversation data to generate an instance of synthesized speech audio data. The instance of synthesized speech audio data may be audibly rendered on the client device of a representative of the "virtual cafe" and may capture synthesized speech 554A2 saying, "Hi, I'm a virtual assistant calling on Jane Doe to see if anyone found her red leather jacket yesterday."

[0079] In this example, when generating an instance of synthesized speech audio data, the automated assistant can ensure that this data is applied as input to a pre-trained LLM that has been fine-tuned and / or primed based on the unstructured free-form natural language input 552A1, generating an output such as a probability distribution across a vocabulary of terms and / or phrases. Based on the probability distribution across the vocabulary of terms and / or phrases, the automated assistant can cause the chatbot to select text data corresponding to the synthesized speech 554A2. Furthermore, the automated assistant can cause the chatbot to process the text data corresponding to the synthesized speech 554A2 using a TTS model(s), thereby generating an instance of synthesized speech audio data that is audibly rendered on the client device of the “virtual cafe” representative. Furthermore, the automated assistant can cause the chatbot to generate an output and / or select text data corresponding to the synthesized speech 554A2, because the pre-trained LLM has been fine-tuned and / or primed based on the unstructured free-form natural language input 552A1 provided by Jane Doe during the interaction 500A1. Therefore, the automated assistant can have the chatbot perform the task of submitting queries to representatives of the "virtual cafe" entities on behalf of Jane Doe.

[0080] Furthermore, assume that the "virtual cafe" representative responds to the synthesized voice 554A2 with spoken input 556A2, "Please wait while I check for lost items." In this example, the automated assistant can have the chatbot process audio data capturing the spoken input 556A2, text data corresponding to the audio data capturing the spoken input 556A2 (determined, for example, using an ASR model), and / or arbitrary contextual conversation data to generate additional instances of the synthesized voice audio data. These additional instances of the synthesized voice audio data can be audibly rendered on the "virtual cafe" representative's client device and can capture the synthesized voice 558A2 of "OK." Additionally, the automated assistant can have the chatbot monitor for additional spoken input from the "virtual cafe" representative to indicate that the representative has rejoined the phone call after putting the chatbot on hold.

[0081] Furthermore, assume that the representative of the "virtual cafe" returns from hold by providing the spoken input 560A2, "I found her red leather jacket. I'll keep it for her on the counter." In this example, the automated assistant can have the chatbot process audio data capturing the spoken input 560A2, text data corresponding to the audio data capturing the spoken input 560A2 (determined, for example, using an ASR model), and / or any contextual conversation data to generate additional instances of the synthesized speech audio data. These additional instances of the synthesized speech audio data can be audibly rendered on the client device of the representative of the "virtual cafe" and can capture the synthesized speech 562A2, "Thank you, I'll let her know." Furthermore, the automated assistant can have the chatbot terminate the phone call with the representative of the "virtual cafe."

[0082] In this example, the automated assistant may, in response to determining that the task has been successfully completed, instruct the chatbot to terminate the corresponding conversation with the representative of the "virtual cafe." The automated assistant or chatbot may determine that the task has been successfully completed, for example, based on the representative of the "virtual cafe" responding to the query by confirming that Jane Doe actually left her red leather jacket at the virtual cafe last night and that it can be picked up at the reception desk. Thus, based on the response to the query provided by the representative of the "virtual cafe," the automated assistant may determine responsive content 552A3 that can be provided to present to Jane Doe in interaction 500A3. Responsive content 552A3 may include the result of the chatbot's task execution, such as, "Your red leather jacket is at the virtual cafe. They will leave it at the reception desk for you." Thus, interaction 500A3 could be a notification that is generated to be presented to the user, or provided to be presented to the user during a subsequent dialogue session between Jane Doe and the automated assistant, which is at least partially executed on her client device.

[0083] In particular, in interaction 500A2, the chatbot performs various peripheral actions. For example, the chatbot introduces itself in synthesized voice 554A2 as "a virtual assistant calling on behalf of Jane Doe" by utilizing a greeting action that enables the chatbot to identify Jane Doe and itself as a chatbot; the chatbot puts itself on hold after synthesized voice 558A2 by utilizing a hold action that enables the chatbot to pause and resume the conversation it is responding to; and the voice ends the call by utilizing a exit action after providing synthesized voice 562A2 to end the conversation it is responding to with the representative of "the virtual cafe". These peripheral actions are just a few examples of how the chatbot can perform generalized forms of conversation without the need for unstructured free-form natural language input 552A1 to specify that the chatbot can perform these generalized forms of conversation by using a pre-trained LLM before the chatbot is fine-tuned and / or primed. Furthermore, please note that other peripheral operations may be implemented by the chatbot, and that the explanation with respect to Figure 5A is for illustrative purposes only and not intended to be limiting.

[0084] Furthermore, in various embodiments, a chatbot generated to perform the task of submitting queries to a representative of the "virtual cafe" may be discarded. For example, the chatbot may be discarded in response to determining that the task has been successfully completed. In another example, the chatbot may be discarded in response to determining that the task has been successfully completed and therefore the chatbot can no longer be used to participate in any additional corresponding conversation. However, in various embodiments, the chatbot may not always successfully perform the task during the corresponding conversation.

[0085] Specifically, referring to Figure 5B, assume that an on-device conversation with a client device user (e.g., Jane Doe) is initiated in the same or similar manner as described with respect to Figure 5A, as indicated by interaction 500B1. For the example in Figure 5B, assume that Jane Doe provides the spoken input 552B1 as unstructured free-form natural language input, "Tell Bobby Jones the locksmith that I have transferred money to his quickcash account." In this example, the automated assistant could use an ASR model(s) to process the audio data capturing the spoken input to produce an ASR output. Furthermore, the automated assistant could use an NLU model(s) to process the ASR output to produce an NLU output. In particular, the spoken input explicitly includes an intent of [Notify Entity] (or similar) with a slot value of “I have transferred money to your QuickCash account” (or similar) for the [Notification Content] parameter (e.g., Task Data), and the spoken input explicitly includes an [Entity] (e.g., Entity Data) of [Bobby Jones the Locksmith]. Thus, the automated assistant may provide the response “Okay, I will notify Bobby Jones” 554A2 for the auditory and / or visual presentation to Jane Doe to indicate that the automated assistant will perform the task of notifying the entity “Bobby Jones”.

[0086] In this example, based on unstructured free-form natural language input 552B1, the automated assistant can cause the client device and / or remote system to generate a chatbot that performs the task of notifying a representative of "Bobby Jones" (e.g., "Bobby Jones" is an entity, and "Bobby Jones" is its representative). In this example, the automated assistant may determine a corresponding identifier associated with "Bobby Jones," e.g., a phone number or contact entry available to initiate a corresponding conversation with a representative of "Bobby Jones." Thus, as shown in interaction 500B2, the automated assistant can ensure that the chatbot is implemented on the computing device in the same or similar manner as described above with respect to Figure 5A. This allows the chatbot to call the phone number associated with "Bobby Jones" and facilitate the performance of the task of notifying "Bobby Jones."

[0087] Furthermore, assume that after making a call, you are participating in a conversation with a chatbot and a representative of "Bobby Jones." For example, further assume that a representative of "Bobby Jones" answers an incoming call, and the chatbot generates an instance of synthesized speech audio data in the same or similar manner as described with respect to Figure 5A, and that the synthesized speech audio data is audibly rendered on the client device of the representative of "Bobby Jones." In this example, further assume that the instance of synthesized speech audio data captures synthesized speech 552B2 saying, "Hi, I'm a virtual assistant calling on behalf of Jane Doe. I just wanted to let you know that Jane Doe has transferred the money she owes you to your QuickCash account."

[0088] Furthermore, suppose a representative of "Bobby Jones" responds to the synthesized voice 552B2 with spoken input 554B2, "I don't understand what you're talking about. I don't know who Jane Doe is?". In this example, the automated assistant can have the chatbot process the audio data to capture at least the spoken input 554B2, thereby generating an additional instance of the synthesized voice audio data. This additional instance of the synthesized voice audio data can be audibly rendered on the "Bobby Jones" representative's client device and can capture the synthesized voice 556B2, "Okay, I'll contact Jane." In other words, based on the failure to successfully complete the task (for example, based on the determination that "Bobby Jones" is not waiting for a transfer to his QuickCash account and / or based on the determination that "Bobby Jones" does not know who Jane Doe is), the automated assistant can have the chatbot utilize an exit action that allows Jane Doe to engage in the corresponding conversation.

[0089] Therefore, as shown in interaction 500B3, the automated assistant can generate and provide the chatbot with the prompt 552B3, “Hi Jane, I need to talk to Bobby Jones the locksmith about a payment,” to present to Jane Doe. The prompt may include a specific reason why the chatbot failed to successfully complete the task (e.g., “Bobby Jones doesn’t know who you are”). Consequently, as shown in interaction 500B4, Jane Doe may respond to prompt 500B3 to join the corresponding conversation and provide the spoken input 552B1, “Hi Bobby, sorry to confuse you, I’m Jane Doe. The payment I sent is on behalf of my father, Jane Doe, for the work you did for him last week.” Thus, the automated assistant can have the chatbot join the corresponding conversation with a representative of “Bobby Jones,” but if the chatbot fails to successfully complete the task, it can prompt the chatbot to join the corresponding conversation with Jane Doe.

[0090] In particular, in the examples of Figures 5A and 5B, Jane Doe is not an active participant in the corresponding conversation between the chatbot and the representative for the initial interaction with the corresponding representative (e.g., interaction 500A2 in Figure 5A and interaction 500B2 in Figure 5B). However, the automated assistant can prompt the chatbot to prompt Jane Doe to join the corresponding conversation as needed (e.g., as shown in the example of Figure 5B). Furthermore, in various embodiments, a transcript of the corresponding conversation may be provided to Jane Doe as the corresponding conversation progresses and / or after the corresponding conversation has concluded. In these embodiments, the user may monitor the corresponding conversations and actively participate in one or more of the corresponding conversations without receiving prompts.

[0091] Figures 5A and 5B illustrate corresponding conversations that are telephone calls, but please understand that this is for illustrative purposes only and not intended to be limiting. For example, the corresponding conversation may be a text-based conversation conducted through any text-based platform or service that a chatbot can participate in a corresponding conversation with an entity or its representative (e.g., text or SMS messaging, email, and / or other text-based platforms). Furthermore, while the corresponding representative in the examples in Figures 5A and 5B is a human, please understand that this is for illustrative purposes only and not intended to be limiting. For example, the corresponding representative may be an additional corresponding chatbot deployed on behalf of the corresponding entity. In these examples, the chatbot may participate in a corresponding conversation with the additional corresponding chatbot. Furthermore, while the unstructured free-form natural language input in Figures 5A and 5B is illustrated as being a single sentence, please understand that this is for illustrative purposes only and not intended to be limiting. In particular, the chatbots generated and implemented in Figures 5A and 5B may be different chatbots by providing different unstructured free-form natural language inputs for different tasks(s) provided by the user.

[0092] Referring here to Figures 6A and 6B, additional non-limiting exemplary interactions are shown between a corresponding unstructured free-form natural language input used to generate a corresponding chatbot and a corresponding chatbot that performs a corresponding task(s) based on the corresponding unstructured free-form natural language input. In particular, interactions 600A and 600B described with respect to Figures 6A and 6B may be implemented across multiple computing devices, respectively, to cause the chatbot to perform a corresponding task(s) to be executed. For example, the corresponding unstructured free-form natural language input, as described in the examples of Figures 6A and 6B, may be received on the user's client device (e.g., client device 110 in Figure 1); the chatbot, as described in the examples of Figures 6A and 6B, may be generated on the user's client device (e.g., client device 110 in Figure 1) and / or on a remote system (e.g., remote system 160 in Figure 1); and the chatbot, as described in the examples of Figures 6A and 6B, may be implemented on the user's client device (e.g., client device 110 in Figure 1) and / or on a remote system (e.g., remote system 160 in Figure 1) and communicate with a representative of an entity via an additional computing device of the representative. Each of these computing devices may include its own components, such as user interface input components (e.g., microphones, visual components, presence sensors, touch-sensitive displays, keyboards, hardware buttons, software buttons, etc.), user interface output components (e.g., touch-sensitive displays, speakers, monitors, projectors, etc.), network interfaces, and / or other components. Therefore, it should be understood that interactions 600A and 600B in Figures 6A and 6B are shown within a single interface, respectively, but this is to illustrate, and not to limit, the various technologies described herein.

[0093] Specifically, referring to Figure 6A, assume that an on-device conversation with a client device user (e.g., Jane Doe) is initiated in the same or similar manner as described with respect to Figures 5A and 5B, as shown by interaction 600A1. However, in contrast to the examples in Figures 5A and 5B, assume that Jane Doe provides multiple spoken inputs 652A1, 654A1, 656A1, 658A1, and 660A1, as shown in interaction 600A. The multiple spoken inputs 652A1, 654A1, 656A1, 658A1, and 660A1 convey details of a more complex task (e.g., booking lunch catering) than those described with respect to Figures 5A and 5B (e.g., submitting a query in Figure 5A and providing a notification in Figure 5B), but the automated assistant can further generate a chatbot to book lunch catering and have the chatbot participate in a corresponding conversation with a representative of the “virtual cafe”.

[0094] In this example, the task may be considered more complex than the task from either Figure 5A or 5B, based on the task in Figure 6A which includes multiple subtasks. These subtasks may include, for example, determining (based on spoken inputs 652A1 and 654A1) whether the “virtual cafe” is available for lunch catering for a specified number of people at a specified date / time; determining (based on spoken input 654A1) whether the “virtual cafe” is available for lunch catering with specified dietary restrictions or menu requests; determining (based on spoken input 658A1) whether the “virtual cafe” is available for lunch catering at a specified price; and actively paying for lunch (based on spoken input 658A1) if the “virtual cafe” is available for lunch catering at a specified price. In particular, each of these subtasks includes a different intent and different slot values ​​for parameters associated with a different intent. As a result, in the example of Figure 6A, the chatbot can be fine-tuned and / or primed with more task data than the chatbots in Figures 5A and 5B, since the task in Figure 6A is more complex. Thus, the automated assistant may provide the response 662A1 to Jane Doe in response to an auditory and / or visual presentation indicating that the automated assistant will perform the task of making a lunch reservation with the entity of “Virtual Cafe”.

[0095] Specifically, referring to Figure 6B, as shown in interaction 600B1, the automated assistant can be implemented on a computing device in the same or similar manner as described with respect to Figures 5A and 5B. Furthermore, suppose after making a call, the chatbot and a representative of the "Virtual Cafe" engage in a corresponding conversation. For example, suppose the representative of the "Virtual Cafe" answers the incoming call and provides spoken input 652B1, "Hello, this is John Smith from the Virtual Cafe. How can I help you?". In this example, the automated assistant can process task data determined based on interaction 600A1, audio data capturing the spoken input 652B1, text data corresponding to the audio data capturing the spoken input 652B1 (e.g., determined using an ASR model(s)), and / or task data determined based on any contextual conversation data to generate an instance of synthesized speech audio data in the same or similar manner as described with respect to Figures 5A and 5B. An instance of synthesized speech audio data can be audibly rendered on a client device of a representative of the "virtual cafe," and can capture synthesized speech 654B1 saying, "Hi, I'm a virtual assistant calling on behalf of Jane Doe to check if catering for her lunch is possible at noon on December 12, 2022."

[0096] Furthermore, assume that the representative of the "virtual cafe" responds to the synthesized voice 654B1 with spoken input 656B1, "Okay, could you tell me a little more about lunch?" In this example, the automated assistant can have the chatbot process task data, audio data capturing the spoken input 656B1, text data corresponding to the audio data capturing the spoken input 656B1 (determined, for example, using an ASR model), and / or arbitrary contextual conversation data to generate additional instances of the synthesized voice audio data. These additional instances of the synthesized voice audio data can be audibly rendered on the client device of the representative of the "virtual cafe" and capture the synthesized voice 658B1, "There are 50 people, some are gluten intolerant and others are..." but the chatbot can then have the representative of the "virtual cafe" interrupt and provide spoken input 660B1, "Excuse me, did you say 50 or 15?" In this example, the automated assistant can cause the chatbot to repeat "50 people" in synthesized speech 662B1 by utilizing clarification behavior, which allows the chatbot to clarify and / or repeat information provided in advance during the conversation it is responding to.

[0097] Furthermore, let's assume the "virtual cafe" representative responds to the synthesized voice 662B1 with spoken input 664B1, "I get 50 people, please continue." In this example, the automated assistant can resume the conversation with the chatbot by generating additional instances of the synthesized voice that include information not provided to the "virtual cafe" representative. For example, additional instances of the synthesized voice audio data can be audibly rendered on the "virtual cafe" representative's client device and capture the synthesized voice 666B1, "Some of the people are gluten intolerant, others are vegan, and Jane would prefer a sandwich platter with several different options, a salad bar, and other side options." Let's assume the "virtual cafe" representative responds to the synthesized voice 666B1 with spoken input 668B1, "Okay, we can cater for $225." As a result, the automated assistant can complete the task of booking lunch with the chatbot by generating additional instances of the synthesized voice audio data, based on having successfully completed all other subtasks of the task. Further additional instances of the synthesized speech audio data can be audibly rendered on the client device of the representative of the "virtual cafe," saying, "That's great, here is Jane Doe's credit card information [ * Provide credit card information * It is possible to capture synthesized speech 670B1 saying, "[There is a problem, please forward the details to janedoe@exampleurl.com]".

[0098] In this example, the automated assistant may, in response to determining that the task (or all of its subtasks) has been successfully completed, have the chatbot terminate the corresponding conversation with the "Virtual Cafe" representative. The automated assistant or chatbot may determine that the task has been successfully completed, for example, based on confirmation of lunch catering by the "Virtual Cafe" representative. Thus, based on the successful reservation of lunch with the "Virtual Cafe" representative, the automated assistant may determine responsive content 652B2 that may be provided to present to Jane Doe in interaction 600B2. Responsive content 652B2 may include the result of the chatbot's task execution, such as "The Virtual Cafe has scheduled lunch catering. John Smith will forward the details to your email." Thus, interaction 600B2 may be a summary of interaction 600B1, which is generated for presentation to the user, or provided for presentation to the user during a subsequent dialogue session between Jane Doe and the automated assistant, which is at least partially executed on her client device.

[0099] Referring now to Figure 7, a block diagram of an exemplary computing device 710 that may be optionally used to perform one or more embodiments of the techniques described herein is shown. In some embodiments, one or more client devices, remote system components and / or other components may include one or more components of the exemplary computing device 710.

[0100] The computing device 710 typically includes at least one processor 714 that communicates with several peripheral devices via a bus subsystem 712. These peripheral devices may include, for example, a storage subsystem 724 including a memory subsystem 725 and a file storage subsystem 726, a user interface output device 720, a user interface input device 722, and a network interface subsystem 716. The input and output devices enable user interaction with the computing device 710. The network interface subsystem 716 provides an interface to an external network and is connected to a corresponding interface device in another computing device.

[0101] User interface input devices 722 may include pointing devices such as keyboards, mice, trackballs, touchpads, or graphics tablets, scanners, touchscreens integrated into displays (e.g., touch-sensitive displays), voice input devices such as voice recognition systems and microphones, and / or other types of input devices. Generally, the use of the term “input device” is intended to include all possible types of devices and methods for inputting information into the computing device 710 or communication network.

[0102] The user interface output device 720 may include non-visual displays such as a display subsystem, printer, facsimile, or audio output device. The display subsystem may include flat panel devices such as cathode ray tubes (CRTs) and liquid crystal displays (LCDs), projection devices, or other mechanisms for creating visible images. The display subsystem may also provide non-visual displays via audio output devices, etc. In general, the use of the term “output device” is intended to include all kinds of devices and methods for outputting information from the computing device 710 to a user or other machine or computing device.

[0103] The storage subsystem 724 stores programming and data structures that provide some or all of the functionality of the modules described herein. For example, the storage subsystem 724 may include logic that performs selected embodiments of the methods disclosed herein, and logic that implements the various components shown in Figures 1 and 2.

[0104] These software modules typically run on processor 714 alone or in combination with other processors. The memory 725 used by the storage subsystem 724 may include a number of memories, such as main random access memory (RAM) 730 for storing instructions and data during program execution, and read-only memory (ROM) 732 for storing fixed instructions. The file storage subsystem 726 can provide persistent storage of program files and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules performing the functions of a particular embodiment may be stored by the file storage subsystem 726 within the storage subsystem 724, or on other machines accessible by the processor 714(or more).

[0105] The bus subsystem 712 provides a mechanism that enables various components and subsystems of the computing device 710 to communicate with each other as intended. Although the bus subsystem 712 is schematically shown as a single bus, alternative embodiments of the bus subsystem 712 may use multiple buses.

[0106] The computing device 710 can be of various types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing systems or computing devices. Because the nature of computers and networks is constantly changing, the description of the computing device 710 shown in Figure 7 is intended only as a specific example to illustrate several embodiments. Many other configurations of the computing device 710 may have more or fewer components than the computing device shown in Figure 7.

[0107] Wherever the systems described herein may collect or monitor personal information relating to a user, or may use personal and / or monitored information, the user may be given the opportunity to control whether the program or function collects user information (e.g., information about the user's social networks, social behavior or activities, occupation, user preferences, or current geographical location), or whether and / or how it receives content that may be more relevant to the user from a content server. Furthermore, certain data may be processed in one or more ways before being stored or used, such that personally identifiable information is removed. For example, a user's identity may be processed in such a way that personally identifiable information cannot be determined, or if geographical location information (such as city, zip code, or state level) is obtained, the user's geographical location may be generalized in such a way that the user's specific geographical location cannot be determined. Thus, the user may control how information relating to them is collected and / or how that information is used.

[0108] In some embodiments, a method is provided which is implemented by one or more processors of a client device, the method comprising: the client device receiving unstructured free-form natural language input from a user of the client device; and, in response to receiving unstructured free-form natural language input including one or more tasks related to an entity, generating a chatbot that performs one or more tasks related to an entity locally on the client device on behalf of the user, based on at least the unstructured free-form natural language input. The unstructured free-form natural language input includes one or more tasks related to an entity. The method further comprises causing the chatbot to perform one or more tasks related to an entity on behalf of the user. Causing the chatbot to perform one or more tasks related to an entity on behalf of the user includes causing the chatbot to participate in a corresponding conversation with the entity; causing the chatbot to render multiple instances of synthesized speech audio data for presentation to a representative of the entity during the corresponding conversation with the entity; and receiving responsive content in response to at least a given instance of the synthesized speech audio data. At least a given instance of the synthesized speech audio data among the multiple instances of the synthesized speech audio data conveys details of one or more tasks related to the entity. The method further includes ensuring that responsive content is provided for presentation to the user of the client device.

[0109] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features:

[0110] In some embodiments, generating a chatbot that performs one or more tasks related to an entity on behalf of a user may include obtaining a pre-trained large-scale language model (LLM), fine-tuning the pre-trained LLM based on unstructured free-form natural language input to generate a fine-tuned LLM, and utilizing the fine-tuned LLM as a chatbot.

[0111] In some versions of these embodiments, having a chatbot render a given instance of synthesized speech audio data for presentation to a representative may include processing one or more features of unstructured free-form natural language input to generate a given instance of text data conveying details of one or more tasks related to an entity using a finely tuned LLM; processing a given instance of text data conveying details of one or more tasks related to an entity using a text-to-speech (TTS) model; and transmitting the given instance of synthesized speech audio data from the client device to an additional client device of the representative. By transmitting the given instance of synthesized speech audio data to the additional client device, the additional client device can be caused to audibly render the given instance of synthesized speech audio data for presentation to the representative via one or more speakers of the additional client device.

[0112] In some further versions of these embodiments, the method may further include processing the corresponding context of a corresponding conversation, along with one or more features of unstructured free-form natural language input, to generate a given instance of text data that conveys details of one or more tasks related to an entity, using a finely tuned LLM.

[0113] In additional or alternative further versions of those embodiments, the method may further include, in a client device, receiving from an additional client device a given instance of response audio data containing responsive content in response to at least a given instance of the synthesized speech audio data, in response to a given instance of synthesized speech audio data being audibly rendered for presentation to a representative through one or more speakers of an additional client device; processing the given instance of the response audio data to generate a given instance of response text data using an automatic speech recognition (ASR) model; and determining, based on the given instance of the response text data, whether one or more of the tasks performed during the corresponding conversation were successfully completed.

[0114] In some, and further versions of, these embodiments, the responsive content may be provided for presentation to the user of the client device in response to determining that one or more tasks have been successfully completed during the corresponding conversation.

[0115] Further additional or alternative versions of those embodiments may further include causing a chatbot to render instructions that, in response to determining that one or more tasks during a corresponding conversation have not been successfully completed, the user is being prompted to join a corresponding conversation for presentation to a representative of the entity; generating a prompt asking the user to join a corresponding conversation; and ensuring that a prompt asking the user to join a corresponding conversation is provided for presentation to the user on the client device.

[0116] In further versions of these embodiments, the prompt may further include specific reasons why one or more of the tasks failed to complete successfully during the corresponding conversation.

[0117] Further additional or alternative versions of those embodiments may further include extracting one or more features from the unstructured free-form natural language input before processing one or more of the features of the unstructured free-form natural language input using the finely tuned LLM.

[0118] In some further versions of these embodiments, one or more of the features may be explicitly included in the unstructured free-form natural language input, and extracting one or more of the features from the unstructured free-form natural language input that is explicitly included in the unstructured free-form natural language input may involve using an input parser to extract one or more of the features explicitly included in the unstructured free-form natural language input.

[0119] In additional or alternative versions of those embodiments, one or more features may be implicitly included in the unstructured free-form natural language input, and extracting one or more features from the unstructured free-form natural language input that is implicitly included in the unstructured free-form natural language input may involve using an input parser to identify one or more features implicitly included in the unstructured free-form natural language input, and using a coreference resolver to extract one or more features implicitly included in the unstructured free-form natural language input.

[0120] In further versions of those embodiments, the coreference resolver may also access locally generated user data on the client device to extract one or more features implicitly included in the unstructured free-form natural language input, which may include one or more of the following: historical location data, historical time data, user preference data, user account data, calendar information, or email data.

[0121] In additional or alternative versions of those embodiments, the pre-trained LLM may be stored in the on-device storage of the client device, and the pre-trained LLM that may be stored in the on-device storage of the client device may be a sparsed version of the global pre-trained LLM available to a remote system communicatively coupled to the client device.

[0122] In some further versions of these embodiments, the finely tuned LLM may be stored in the on-device storage of the client device.

[0123] In yet another version of these embodiments, the method may further include having the chatbot perform one or more tasks related to an entity on behalf of the user, then discarding the finely tuned LLM from the client device's on-device storage, and refraining from discarding the pre-trained LLM from the client device's on-device storage.

[0124] In some embodiments, the method may further include, on a client device, receiving additional unstructured free-form natural language input from a user of the client device, and, in response to receiving additional unstructured free-form natural language input including one or more additional tasks relating to an entity or additional entity, generating an additional chatbot that performs one or more additional tasks relating to an entity or additional entity locally on the client device on behalf of the user, based on at least the additional natural language input. The additional unstructured free-form natural language input may include one or more additional tasks relating to an entity or additional entity. The method may further include having the additional chatbot perform one or more additional tasks relating to an entity or additional entity on behalf of the user. Having an additional chatbot perform one or more additional tasks related to an entity or an additional entity on behalf of a user may include having the additional chatbot participate in an additional corresponding conversation with the entity or the additional entity; having the chatbot render multiple additional instances of synthesized speech audio data for presentation to a representative of the entity or an additional representative of the additional entity during the additional corresponding conversation with the entity or the additional entity; receiving additional responsive content in response to at least a given additional instance of the synthesized speech audio data; and making the additional responsive content available for presentation to a user on a client device. At least one of the multiple additional instances of the synthesized speech audio data conveys additional details of one or more additional tasks related to the entity or the additional entity.

[0125] In some embodiments, the method may further include identifying entities associated with one or more tasks based on unstructured, free-form natural language input, determining corresponding identifiers for the entities associated with one or more tasks, and utilizing the corresponding identifiers for the entities associated with one or more tasks to have a chatbot engage in corresponding conversations with the entities.

[0126] In some versions of these embodiments, the corresponding identifier of an entity associated with one or more tasks may also be the corresponding telephone number of the entity, and using the corresponding identifier of an entity associated with one or more tasks to get a chatbot to engage in a corresponding conversation with the entity may include having the chatbot use the corresponding telephone number of the entity to initiate an automated telephone call on behalf of the user and perform one or more tasks related to the entity.

[0127] In some embodiments, involving a chatbot in a corresponding conversation with an entity may include having the chatbot respond to a phone call received on a client device and from a representative of the entity, and having the chatbot participate in a corresponding conversation with the entity as part of a phone call.

[0128] In some embodiments, the user may not actively participate in the corresponding conversation between the chatbot and the representative.

[0129] In some embodiments, during a corresponding conversation with an entity, the method may further include receiving a request from a representative for a user to join the corresponding conversation, generating a prompt in response to receiving the request for a user to join the corresponding conversation, and ensuring that a prompt instructing the user to join the corresponding conversation is provided for presentation to the user on a client device.

[0130] In some embodiments, entities may be explicitly identified in unstructured, free-form natural language input.

[0131] In some embodiments, entities may not be explicitly identified in unstructured free-form natural language input, entities may be of a specific type, entities of a specific type may be explicitly identified in unstructured free-form natural language input, and one or more tasks may be associated with a specific type of entity.

[0132] In some versions of these embodiments, one or more tasks may also be associated with additional entities, which are entities of a particular type, in addition to the entity. Having the chatbot perform one or more tasks related to the entity on behalf of the user may further include having the chatbot participate in an additional corresponding conversation with the additional entity, having the chatbot render multiple additional instances of additional synthesized speech audio data for presentation to additional representatives of the additional entity during the additional corresponding conversation with the additional entity, and receiving additional responsive content in response to at least a given additional instance of the synthesized speech audio data. At least a given additional instance of the synthesized speech audio data among the multiple additional instances of the synthesized speech audio data may convey details of one or more tasks that are also related to the additional entity. The method may further include ensuring that additional responsive content is provided for presentation to the user of the client device.

[0133] In some further versions of these embodiments, the chatbot can conduct corresponding conversations with entities and additional corresponding conversations with additional entities in parallel.

[0134] In additional or alternative further versions of those embodiments, the chatbot may engage in additional corresponding conversations with additional entities after a corresponding conversation with an entity, and in response to determining that one or more of the tasks were not successfully completed during the corresponding conversation.

[0135] In some embodiments, the entity's representative may be a human representative. In other embodiments, the entity's representative may be an additional chatbot trained to conduct the corresponding conversation on behalf of the entity.

[0136] In some embodiments, the unstructured free-form natural language input may be a typed or spoken input that conveys details of one or more tasks related to an entity, and it is not necessary to define a corresponding dialogue state map, a dialogue state of the corresponding dialogue state map, or a dialogue state transition of the corresponding dialogue state map that is used to perform one or more tasks related to the entity.

[0137] In some versions of these embodiments, unstructured free-form natural language input may be typed input that conveys details of one or more tasks related to an entity in one or more sentences, and responsive content provided for presentation to the user of a client device may be the result of performing one or more tasks associated with the entity.

[0138] In additional or alternative versions of these embodiments, the unstructured free-form natural language input may be a typed input that conveys details of one or more tasks related to an entity in one or more paragraphs, and the responsive content provided for presentation to the user of the client device may be a summary of the corresponding conversation with the entity.

[0139] In some embodiments, a method is provided which is implemented by one or more processors in a remote system and includes receiving instructions in the remote system and from a client device from a user of the client device, and in response to receiving instructions in the unstructured free-form natural language input which includes one or more tasks related to an entity, generating a chatbot remotely in the remote system that is associated with an entity and performs one or more tasks on behalf of the user, based on at least the instructions in the natural language input. The unstructured free-form natural language input includes one or more tasks related to the entity. The method further includes causing the chatbot to perform one or more tasks related to the entity on behalf of the user. Causing the chatbot to perform one or more tasks related to the entity on behalf of the user includes causing the chatbot to participate in a corresponding conversation with the entity, causing the chatbot to render multiple instances of synthesized speech audio data for presentation to a representative of the entity during the corresponding conversation with the entity, and receiving responsive content in response to at least a given instance of the synthesized speech audio data. At least a given instance of the synthesized speech audio data among the multiple instances of the synthesized speech audio data conveys details of one or more tasks related to the entity. This method further includes sending instructions for responsive content from a remote system to a client device. By sending instructions for responsive content to the client device, the client device is prompted to provide responsive content to be presented to the user of the client device.

[0140] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features:

[0141] In some embodiments, generating a chatbot that performs one or more tasks related to an entity on behalf of a user may include obtaining a pre-trained large-scale language model (LLM), fine-tuning the pre-trained LLM based on unstructured free-form natural language input to generate a fine-tuned LLM, and utilizing the fine-tuned LLM as a chatbot.

[0142] In some versions of these embodiments, the pre-trained LLM may be stored in the remote storage of the remote system, and the pre-trained LLM that may be stored in the remote storage of the remote system may be a non-sparse version of the global pre-trained LLM available on the remote system, which is communicably coupled to the client device.

[0143] In some further versions of these embodiments, the finely tuned LLM may be stored in remote storage of a remote system.

[0144] In yet another version of these embodiments, the method may further include having the chatbot perform one or more tasks related to an entity on behalf of the user, then discarding the finely tuned LLM from the remote storage of the remote system, and refraining from discarding the pre-trained LLM from the remote storage of the remote system.

[0145] In some embodiments, a method is provided which is implemented by one or more processors in a client device, and which, in response to the client device receiving unstructured free-form natural language input from a user of the client device and receiving instructions in the unstructured free-form natural language input that include one or more tasks related to an entity, generates a chatbot remotely in a remote system that is associated with an entity and performs one or more tasks on behalf of the user, based on at least the instructions in the natural language input. The unstructured free-form natural language input includes one or more tasks related to an entity. The method further includes causing the chatbot to perform one or more tasks related to an entity on behalf of the user. Causing the chatbot to perform one or more tasks related to an entity on behalf of the user includes causing the chatbot to participate in a corresponding conversation with the entity, causing the chatbot to render multiple instances of text data for presentation to a representative of the entity during the corresponding conversation with the entity, and receiving responsive content that responds to at least a given instance of the text data. At least a given instance of the text data among the multiple instances of text data conveys details of one or more tasks related to the entity. The method further includes ensuring that the responsive content is provided for presentation to the user of the client device.

[0146] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features:

[0147] In some embodiments, the method may further include identifying entities associated with one or more tasks based on unstructured, free-form natural language input, determining corresponding identifiers for the entities associated with one or more tasks, and utilizing the corresponding identifiers for the entities associated with one or more tasks to have a chatbot engage in corresponding conversations with the entities.

[0148] In versions of those embodiments, the corresponding identifier of an entity associated with one or more tasks may be one or more of the entity's corresponding phone number, the entity's corresponding email address, or the entity's corresponding username. Using the corresponding identifier of an entity associated with one or more tasks to get a chatbot to engage in a corresponding conversation with one or more entities may include having the chatbot use the corresponding identifier of the entity to initiate an automated text-based messaging session on behalf of the user and perform one or more tasks related to the entity.

[0149] In some embodiments, involving a chatbot in a corresponding conversation with an entity may include having the chatbot respond to text messages, emails, or other text-based messages received on a client device and from a representative of the entity, and having the chatbot in a corresponding conversation with the entity to facilitate text messages, emails, or other text-based messages.

[0150] In some embodiments, a method is provided which is implemented by one or more processors in a remote system and includes receiving instructions in the remote system and from a client device from a user of the client device, and in response to receiving instructions in the unstructured free-form natural language input which includes one or more tasks related to an entity, generating a chatbot remotely in the remote system that is associated with an entity and performs one or more tasks on behalf of the user, based on at least the instructions in the natural language input. The unstructured free-form natural language input includes one or more tasks related to the entity. The method further includes causing the chatbot to perform one or more tasks related to the entity on behalf of the user. Causing the chatbot to perform one or more tasks related to the entity on behalf of the user includes causing the chatbot to participate in a corresponding conversation with the entity, causing the chatbot to render multiple instances of synthesized speech audio data for presentation to a representative of the entity during the corresponding conversation with the entity, and receiving responsive content in response to at least a given instance of the synthesized speech audio data. At least a given instance of the synthesized speech audio data among the multiple instances of the synthesized speech audio data conveys details of one or more tasks related to the entity. This method further includes sending instructions for responsive content from a remote system to a client device. By sending instructions for responsive content to the client device, the client device is prompted to provide responsive content to be presented to the user of the client device.

[0151] In some embodiments, a method is provided which is implemented by one or more processors of a client device, the method comprising: receiving from a user of the client device an unstructured free-form natural language input, which includes one or more tasks associated with an entity; identifying a chatbot that performs one or more tasks related to an entity on behalf of the user in response to receiving an unstructured free-form natural language input, which includes a natural language description of a corresponding dialogue state map; and causing the chatbot to perform one or more tasks related to the entity on behalf of the user. Causing the chatbot to perform one or more tasks related to an entity on behalf of the user comprises: causing the chatbot to participate in a corresponding conversation with the entity; causing the chatbot to render a plurality of instances of synthesized speech audio data for presentation to a representative of the entity during the corresponding conversation with the entity, wherein at least a given instance of the synthesized speech audio data conveys details of one or more tasks related to the entity; and receiving responsive content in response to at least a given instance of the synthesized speech audio data. The method further comprises providing the responsive content for presentation to the user of the client device.

[0152] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features:

[0153] In some embodiments, identifying a chatbot that performs one or more tasks related to an entity on behalf of a user may include obtaining a pre-trained large-scale language model (LLM) and making the pre-trained LLM available as a chatbot.

[0154] In some versions of these embodiments, the pre-trained LLM may be stored in the on-device storage of the client device, and the pre-trained LLM that may be stored in the on-device storage of the client device may be a sparsed version of the global pre-trained LLM available to a remote system communicatively coupled to the client device.

[0155] In some versions of these embodiments, the method may further include refraining from having a pre-trained LLM fine-tuned based on unstructured, free-form natural language input.

[0156] In some versions of these embodiments, having a chatbot render a given instance of synthesized speech from multiple instances of synthesized speech for presentation to a representative of an entity may include processing free-form natural language input and task data reflecting details of one or more tasks related to an entity to generate an instance of text data reflecting a given behavior of a given implicit dialogue state using a pre-trained LLM; processing a given instance of text data reflecting a given behavior of a given implicit dialogue state to generate a given instance of synthesized speech using a Text-to-Speech (TTS) model; and transmitting the given instance of synthesized speech from a client device to an additional client device of the entity's representative. By transmitting the given instance of synthesized speech to the additional client device, the additional client device can audibly render the given instance of synthesized speech for presentation to the entity's representative via one or more speakers of the additional client device.

[0157] Furthermore, some embodiments include one or more processors in one or more computing devices (e.g., a central processing unit (CPU(or more)), a graphics processing unit (GPU(or more)), and / or a tensor processing unit (or more) (TPU(or more)), where one or more processors are operable to execute instructions stored in associated memory, and the instructions are configured to perform one of the methods described above. Some embodiments also include one or more non-temporary computer-readable storage media that store computer instructions executable by one or more processors for performing one of the methods described above. Some embodiments also include a computer program product that includes instructions executable by one or more processors for performing one of the methods described above.

[0158] It will be understood that all combinations of the above concepts and additional concepts described in more detail herein are intended to be part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are intended to be part of the subject matter disclosed herein.

Claims

1. A method implemented by one or more processors in a client device, The client device receives unstructured free-form natural language input from the user of the client device, wherein the unstructured free-form natural language input includes one or more tasks related to an entity. In response to receiving the unstructured, free-form natural language input, which includes one or more tasks related to the entity, To generate a chatbot that performs one or more tasks related to the entity on behalf of the user, based at least on the unstructured, free-form natural language input and locally on the client device, The chatbot is configured to perform one or more tasks related to the entity on behalf of the user, and the chatbot is configured to perform one or more tasks related to the entity on behalf of the user, The chatbot is to participate in the corresponding conversation with the entity, During the corresponding conversation with the entity, The chatbot is to render multiple instances of synthesized speech audio data for presentation to a representative of the entity, wherein at least one instance of the synthesized speech audio data among the multiple instances of the synthesized speech audio data is to render details of one or more tasks related to the entity. Receiving responsive content in response to at least one given instance of synthesized speech audio data, To ensure that the responsive content is provided to the user of the client device, Methods that include...

2. Generating the chatbot that performs one or more tasks related to the entity on behalf of the user is, Obtaining a pre-trained Large-Scale Language Model (LLM), To generate a finely tuned LLM, the pre-trained LLM is fine-tuned based on the unstructured free-form natural language input, The aforementioned finely tuned LLM is used as the chatbot, The method according to claim 1, including the method described in claim 1.

3. Having the chatbot render the given instance of the synthesized speech audio data for presentation to the representative is: Using the finely tuned LLM, one or more features of the unstructured free-form natural language input are processed to generate a given instance of text data that conveys the details of one or more tasks related to the entity. Using a text-to-speech (TTS) model, the process of a given instance of text data conveying the details of one or more tasks related to the entity in order to generate the given instance of synthesized speech audio data, Transmitting the given instance of synthesized speech audio data from the client device to an additional client device of the representative, thereby causing the additional client device to audibly render the given instance of synthesized speech audio data for presentation to the representative via one or more speakers of the additional client device. The method according to claim 2, including the method described in claim 2.

4. Using the finely tuned LLM, processing the corresponding context of the corresponding conversation, along with one or more of the features of the unstructured free-form natural language input, to generate a given instance of text data that conveys the details of one or more tasks related to the entity. The method according to claim 3, further comprising:

5. In response to the given instance of synthesized speech audio data being audibly rendered for presentation to the representative via one or more speakers of the additional client device, The client device receives from the additional client device the given instance of responsive audio data, which includes responsive content that responds to at least a given instance of synthesized speech audio data. Using an automatic speech recognition (ASR) model, a given instance of response audio data is processed to generate a given instance of response text data, Based on the given instance of the response text data, determine whether one or more of the tasks were successfully completed during the corresponding conversation. The method according to claim 3, further comprising:

6. The method according to claim 5, wherein providing the responsive content for presentation to the user of the client device is in response to determining that one or more of the tasks have been successfully completed during the corresponding conversation.

7. In response to determining that one or more of the tasks have not been successfully completed during the corresponding conversation, The chatbot renders an instruction that the user is being prompted to participate in the corresponding conversation in order to present it to the representative of the entity, To generate a prompt requesting the user to participate in the corresponding conversation, The client device is provided with the prompt requesting the user to participate in the corresponding conversation, The method according to claim 5, further comprising:

8. The method according to claim 7, wherein the prompt further includes specific reasons for why one or more of the tasks did not complete successfully during the corresponding conversation.

9. Before processing one or more of the features of the unstructured free-form natural language input using the finely tuned LLM, Extracting one or more of the above features from the aforementioned unstructured free-form natural language input. The method according to claim 3, further comprising:

10. If one or more of the aforementioned features are explicitly included in the unstructured free-form natural language input, then extracting one or more of the aforementioned features explicitly included in the unstructured free-form natural language input from the unstructured free-form natural language input is: An input parser is used to extract one or more of the features explicitly included in the unstructured free-form natural language input. The method according to claim 9, including the method described in claim 9.

11. If one or more of the aforementioned features are implicitly included in the unstructured free-form natural language input, then extracting one or more of the aforementioned features implicitly included in the unstructured free-form natural language input from the unstructured free-form natural language input is To identify one or more of the features implicitly included in the unstructured free-form natural language input, an input parser is used. To extract one or more of the features implicitly included in the unstructured free-form natural language input, a coreference resolver is used, The method according to claim 9, including the method described in claim 9.

12. The method according to claim 11, wherein the coreference resolver accesses locally generated user data on the client device to extract one or more of the features implicitly included in the unstructured free-form natural language input, the user data includes one or more of historical location data, historical time data, user preference data, user account data, calendar information, or email data.

13. The method according to claim 2, wherein the pre-trained LLM is stored in the on-device storage of the client device, and the pre-trained LLM stored in the on-device storage of the client device is a sparsed version of a global pre-trained LLM available to a remote system communicably coupled to the client device.

14. The method according to claim 13, wherein the finely tuned LLM is stored in the on-device storage of the client device.

15. After having the chatbot perform one or more tasks related to the entity on behalf of the user, Discard the finely adjusted LLM from the on-device storage of the client device, To refrain from discarding the pre-trained LLM from the on-device storage of the client device, The method according to claim 14, further comprising:

16. The client device receives additional unstructured free-form natural language input from the user of the client device, wherein the additional unstructured free-form natural language input includes one or more additional tasks relating to the entity or the additional entity. In response to receiving the additional unstructured free-form natural language input, which includes the one or more additional tasks relating to the entity or the additional entity, To generate an additional chatbot that performs, on behalf of the user, one or more additional tasks related to the entity or the additional entity, based at least on the additional natural language input and locally on the client device, The additional chatbot is to perform one or more additional tasks relating to the entity or the additional entity on behalf of the user, and the additional chatbot is to perform one or more additional tasks relating to the entity or the additional entity on behalf of the user. Allowing the additional chatbot to participate in the entity or an additional corresponding conversation with the additional entity, During the additional corresponding conversation with the entity or the additional entity, The chatbot is to render a plurality of additional instances of synthesized speech audio data for presentation to the representative of the entity or an additional representative of an additional entity, wherein at least one of the plurality of additional instances of synthesized speech audio data is rendered to convey additional details of the one or more additional tasks relating to the entity or the additional entity. Receiving additional responsive content in response to at least the given additional instance of synthesized speech audio data, To ensure that the additional responsive content is provided for presentation to the user of the client device, The method according to claim 1, including the method described in claim 1.

17. Based on the unstructured, free-form natural language input, identify the entities associated with one or more tasks, Determining the corresponding identifier of the entity associated with one or more of the tasks, To cause the chatbot to participate in the corresponding conversation with the entity, the corresponding identifier of the entity associated with one or more tasks is used, The method according to claim 1, further comprising:

18. The corresponding identifier of the entity associated with one or more tasks is the corresponding telephone number of the entity, and using the corresponding identifier of the entity associated with one or more tasks in order to have the chatbot participate in the corresponding conversation with the entity is, To cause the chatbot to use the corresponding telephone number of the entity in order to initiate an automated telephone call on behalf of the user and perform one or more tasks related to the entity, the chatbot is made to use the corresponding telephone number of the entity. The method according to claim 17, including the method described in claim 17.

19. Allowing the chatbot to participate in the corresponding conversation with the entity means The chatbot is to respond to a telephone call received on the client device from the representative of the entity, The chatbot is to participate in the corresponding conversation with the entity as part of the telephone call, The method according to claim 1, including the method described in claim 1.

20. The method according to claim 1, wherein the user does not actively participate in the corresponding conversation between the chatbot and the representative.

21. During the corresponding conversation with the entity, The representative receives a request from the user to participate in the corresponding conversation, In response to receiving the request from the user to participate in the corresponding conversation, To generate a prompt requesting the user to participate in the corresponding conversation, The client device is provided with the prompt requesting the user to participate in the corresponding conversation, The method according to claim 1, further comprising:

22. The method according to claim 1, wherein the entity is explicitly identified in the unstructured free-form natural language input.

23. The method according to claim 1, wherein the entity is not explicitly identified in the unstructured free-form natural language input, the entity is of a particular type, the entity of a particular type is explicitly identified in the unstructured free-form natural language input, and the one or more tasks are associated with the entity of a particular type.

24. The one or more tasks are also associated with additional entities, which are entities of a specific type added to the entity, and the chatbot is to perform the one or more tasks related to the entity on behalf of the user. The chatbot is to participate in additional corresponding conversations with the additional entities, During the additional corresponding conversation with the additional entity, The chatbot is to render a plurality of additional instances of synthesized speech audio data for presentation to additional representatives of the additional entities, wherein at least one of the plurality of additional instances of synthesized speech audio data is rendered to convey the details of the one or more tasks that are also related to the additional entities. Receiving additional responsive content in response to at least the given additional instance of synthesized speech audio data, To ensure that the additional responsive content is provided for presentation to the user of the client device, The method according to claim 23, further comprising:

25. The method according to claim 24, wherein the chatbot conducts the corresponding conversation with the entity and the additional corresponding conversation with the additional entity in parallel.

26. The method according to claim 24, wherein the chatbot, after the corresponding conversation with the entity and in response to determining during the corresponding conversation that one or more of the tasks have not been successfully completed, engages in the additional corresponding conversation with the additional entity.

27. The method according to claim 1, wherein the representative of the entity is a human representative.

28. The method according to claim 1, wherein the representative of the entity is an additional chatbot trained to conduct the corresponding conversation on behalf of the entity.

29. The method according to claim 1, wherein the unstructured free-form natural language input is a typed or spoken input that conveys the details of one or more tasks relating to the entity, and does not define a corresponding dialogue state map, a dialogue state of the corresponding dialogue state map, or a dialogue state transition of the corresponding dialogue state map used to perform the one or more tasks relating to the entity.

30. The method according to claim 29, wherein the unstructured free-form natural language input is a typed input that conveys the details of one or more tasks relating to the entity in one or more sentences, and the responsive content provided for presentation to the user of the client device is the result of performing one or more tasks associated with the entity.

31. The method according to claim 1, wherein the unstructured free-form natural language input is a typed input that conveys the details of one or more tasks relating to the entity in one or more paragraphs, and the responsive content provided for presentation to the user of the client device is a summary of the corresponding conversation with the entity.

32. A method implemented by one or more processors of a remote system, In the remote system, receiving instructions for unstructured free-form natural language input from a client device, wherein the unstructured free-form natural language input includes one or more tasks related to an entity. In response to receiving instructions for the unstructured, free-form natural language input, which includes one or more tasks related to an entity, To generate a chatbot that performs one or more tasks related to the entity on behalf of the user, based at least on the instructions of the natural language input and remotely in the remote system, The chatbot is configured to perform one or more tasks related to the entity on behalf of the user, and the chatbot is configured to perform one or more tasks related to the entity on behalf of the user, The chatbot is to participate in the corresponding conversation with the entity, During the corresponding conversation with the entity, The chatbot is to render multiple instances of synthesized speech audio data for presentation to a representative of the entity, wherein at least one instance of the synthesized speech audio data among the multiple instances of the synthesized speech audio data is to render details of one or more tasks related to the entity. Receiving responsive content in response to at least one given instance of synthesized speech audio data, Transmitting instructions for the responsive content from the remote system to the client device, wherein the client device provides and transmits the responsive content for presentation to the user of the client device by transmitting the instructions for the responsive content to the client device. Methods that include...

33. Generating the chatbot that performs one or more tasks related to the entity on behalf of the user is, Obtaining a pre-trained Large-Scale Language Model (LLM), To generate a finely tuned LLM, the pre-trained LLM is fine-tuned based on the unstructured free-form natural language input, The aforementioned finely tuned LLM is used as the chatbot, The method according to claim 32, including the method described in claim 32.

34. The method according to claim 33, wherein the pre-trained LLM is stored in the remote storage of the remote system, and the pre-trained LLM stored in the remote storage of the remote system is a non-sparse version of a global pre-trained LLM available in the remote system, which is communicably coupled to the client device.

35. The method according to claim 34, wherein the finely tuned LLM is stored in the remote storage of the remote system.

36. After having the chatbot perform one or more tasks related to the entity on behalf of the user, Discard the finely adjusted LLM from the remote storage of the remote system, The aforementioned pre-trained LLMs are not discarded from the remote storage of the remote system. The method according to claim 35, further comprising:

37. A method implemented by one or more processors in a client device, The client device receives unstructured free-form natural language input from the user of the client device, wherein the unstructured free-form natural language input includes one or more tasks related to an entity. In response to receiving instructions for the unstructured, free-form natural language input, which includes one or more tasks related to an entity, To generate a chatbot that performs one or more tasks related to the entity on behalf of the user, based at least on the instructions of the natural language input and remotely in a remote system, The chatbot is configured to perform one or more tasks related to the entity on behalf of the user, and the chatbot is configured to perform one or more tasks related to the entity on behalf of the user, The chatbot is to participate in the corresponding conversation with the entity, During the corresponding conversation with the entity, The chatbot is to render multiple instances of text data for presentation to a representative of the entity, wherein at least one instance of the text data conveys details of one or more tasks related to the entity. Receiving responsive content in response to at least one given instance of the text data, To ensure that the responsive content is provided to the user of the client device, Methods that include...

38. Based on the unstructured, free-form natural language input, identify the entities associated with one or more tasks, Determining the corresponding identifier of the entity associated with one or more of the tasks, To cause the chatbot to participate in the corresponding conversation with the entity, the corresponding identifier of the entity associated with one or more tasks is used, The method according to claim 37, further comprising:

39. The corresponding identifier of the entity associated with one or more tasks is one or more of the corresponding telephone number of the entity, the corresponding email address of the entity, or the corresponding username of the entity, and using the corresponding identifier of the entity associated with one or more tasks in order to have the chatbot participate in the corresponding conversation with the entity is, To enable the chatbot to use the corresponding identifier of the entity in order to initiate an automated text-based messaging session on behalf of the user and perform one or more tasks related to the entity, The method according to claim 38, including the method described in claim 38.

40. Allowing the chatbot to participate in the corresponding conversation with the entity means The chatbot is to respond to text messages, emails, or other text-based messages received on the client device from the representative of the entity. To facilitate the text message, the email, or other text-based message, the chatbot participates in the corresponding conversation with the entity, The method according to claim 37, including the method described in claim 37.

41. A method implemented by one or more processors of a remote system, In the remote system, receiving instructions for unstructured free-form natural language input from a client device, wherein the unstructured free-form natural language input includes one or more tasks related to an entity. In response to receiving instructions for the unstructured, free-form natural language input, which includes one or more tasks related to an entity, To generate a chatbot that performs one or more tasks related to the entity on behalf of the user, based at least on the instructions of the natural language input and remotely in the remote system, The chatbot is configured to perform one or more tasks related to the entity on behalf of the user, and the chatbot is configured to perform one or more tasks related to the entity on behalf of the user, The chatbot is to participate in the corresponding conversation with the entity, During the corresponding conversation with the entity, The chatbot is to render multiple instances of text data for presentation to a representative of the entity, wherein at least one instance of the text data conveys details of one or more tasks related to the entity. Receiving responsive content in response to at least the given instance of text data, Transmitting instructions for the responsive content from the remote system to the client device, wherein the client device provides and transmits the responsive content for presentation to the user of the client device by transmitting the instructions for the responsive content to the client device. Methods that include...

42. A method implemented by one or more processors in a client device, The client device receives unstructured free-form natural language input from the user of the client device, wherein the unstructured free-form natural language input includes one or more tasks related to an entity. Identifying a chatbot that performs one or more tasks related to the entity on behalf of the user in response to receiving the unstructured, free-form natural language input, which includes a natural language description of the corresponding dialogue state map; The chatbot is configured to perform one or more tasks related to the entity on behalf of the user, and the chatbot is configured to perform one or more tasks related to the entity on behalf of the user, The chatbot is to participate in the corresponding conversation with the entity, During the corresponding conversation with the entity, The chatbot is to render multiple instances of synthesized speech audio data for presentation to a representative of the entity, wherein at least one instance of the synthesized speech audio data among the multiple instances of the synthesized speech audio data is to render details of one or more tasks related to the entity. Receiving responsive content in response to at least one given instance of synthesized speech audio data, To ensure that the responsive content is provided to the user of the client device, Methods that include...

43. Identifying the chatbot that performs one or more tasks related to the entity on behalf of the user is, Obtaining a pre-trained Large-Scale Language Model (LLM), The aforementioned pre-trained LLM is to be used as the chatbot, The method according to claim 42, including the method described in claim 42.

44. The method according to claim 43, wherein the pre-trained LLM is stored in the on-device storage of the client device, and the pre-trained LLM stored in the on-device storage of the client device is a sparsed version of a global pre-trained LLM available to a remote system communicably coupled to the client device.

45. Refrain from fine-tuning the pre-trained LLM based on the aforementioned unstructured free-form natural language input. The method according to claim 43, further comprising:

46. To cause the chatbot to render a given instance of the synthesized speech audio data from among the multiple instances of the synthesized speech audio data for presentation to the representative of the entity is: Using the aforementioned pre-trained LLM, the process involves processing the unstructured free-form natural language input and task data reflecting the details of one or more tasks associated with the entity in order to generate an instance of text data that reflects a given behavior of a given implicit dialogue state. Using a text-to-speech (TTS) model, the process involves processing a given instance of text data that reflects a given behavior of a given implicit dialogue state in order to generate a given instance of synthesized speech audio data, Transmitting the given instance of synthesized speech audio data from the client device to an additional client device of the representative of the entity, by transmitting the given instance of synthesized speech audio data to the additional client device, thereby causing the additional client device to audibly render the given instance of synthesized speech audio data for presentation to the representative of the entity via one or more speakers of the additional client device. The method according to claim 43, including the method described in claim 43.

47. It is a system, At least one processor, A memory that, when executed by the at least one processor, stores instructions that cause the method according to any one of claims 1 to 46 to be executed, A system that includes this.

48. A computer-readable storage medium that, when executed by at least one processor, stores instructions causing the at least one processor to perform an operation according to any one of claims 1 to 46.

Images