Unstructured description-based chatbot development technology

Abstract

Embodiments are directed to receiving unstructured, free-form natural language input, generating a chatbot based on the unstructured, free-form natural language input in response to receiving the unstructured, free-form natural language input, and having the chatbot perform a task associated with an entity on behalf of a user. In various embodiments, the unstructured, free-form natural language input conveys details of the task to be performed but does not define any corresponding dialogue state map (e.g., does not define any dialogue states or dialogue state transitions). Nevertheless, the unstructured, free-form natural language input may be used to fine-tune and / or prime machine learning models that can already be utilized in conducting generalized conversations. As a result, chatbots can be generated and deployed in a fast and efficient manner to perform tasks on behalf of a user.

Description

[Technical Field]

[0001] Humans can participate in human-computer interactions through various computing devices using interactive software applications known as “chatbots,” “voicebots,” “automated assistants,” “interactive personal assistants,” “intelligent personal assistants,” “conversational agents,” and the like. As an example, these chatbots may correspond to a machine learning model or a combination of different machine learning models and may be utilized to perform various tasks on behalf of a user. For example, some of these chatbots may engage in conversations with various humans and perform action(s) on behalf of other humans or entities. In some of these examples, the conversation may include a voice-based conversation, such as a conversation conducted locally at a computing device, a conversation conducted remotely across multiple computing devices over a telephone network, or other voice-based scenarios. In other examples, the conversation may include a text-based conversation, such as a conversation conducted via text or SMS messaging, email, and / or other text-based scenarios. [Background technology]

[0002] However, the functionality of some of these chatbots may be limited in various ways. For example, the functionality of some of these chatbots may be limited by the predefined intent schema that the chatbot utilizes to perform an action(s). In other words, if a human participating in a given conversation with a given chatbot provides a spoken utterance that is determined to include an intent that is not defined by the predefined intent schema, the given chatbot may fail. Furthermore, to update these chatbots, existing intent schemas may be modified or new intent schemas may be added. As another example, the functionality of some of these chatbots may be limited by the corpus of examples utilized to train the chatbot. In other words, if a human participating in a given conversation with a given chatbot provides a spoken utterance that was not included in the given corpus of examples, the given chatbot may fail. Furthermore, to update these chatbots, existing examples in the corpus may be modified or new examples may be added. However, in both of these examples, there are instances where intents may need to be defined upfront to make the bot robust to the virtually infinite number of intent schemas and / or various nuances of human speech and mitigate instances of failure.

[0003] In particular, manually defining and / or refining such intent schemata and / or examples requires significant computational resource utilization. Moreover, even if a large number of intent schemata and / or examples are defined, a large amount of memory is required to store and / or utilize the large number of intent schemata for these chatbots and / or to train these chatbots based on the large number of examples in a corpus. Thus, intent schemata for rule-based chatbots and examples for example-based chatbots are not practically scalable to learning the nuances of human speech. Summary of the Invention

[0004] Embodiments are directed to receiving unstructured, free-form natural language input, generating a chatbot based on the unstructured, free-form natural language input and in response to receiving the unstructured, free-form natural language input, and having the chatbot perform task(s) associated with the entity on behalf of a user. In some versions of these embodiments, the unstructured, free-form natural language input conveys details of the task(s) to be performed but does not define any corresponding dialogue state map (e.g., does not define any dialogue states or any dialogue state transitions). Furthermore, the unstructured, free-form natural language input is not provided according to any schema. Nevertheless, the unstructured, free-form natural language input may be utilized to fine-tune machine learning models (ML) already available for conducting more generalized conversations and / or may be utilized as input throughout the machine learning model without the ML model being fine-tuned. As a result, chatbots can be generated and deployed in a fast and efficient manner to perform task(s) on behalf of a user.

[0005] For example, assume that unstructured, free-form natural language input corresponds to the spoken utterance, "Please check with the restaurant where I had dinner yesterday to see if you found my red leather jacket," received at a user's client device. The processor(s) can use various automatic speech recognition (ASR), natural language understanding (NLU), and / or fulfillment techniques to determine that the unstructured, free-form natural language input includes task(s) to be performed by a chatbot. In this example, the task(s) may include, for example, submitting a query, "Did you find [the user's] red leather jacket," to the "restaurant where [the user] had dinner yesterday." Further, the processor(s) can generate the chatbot to engage in a corresponding conversation with a representative of the "restaurant" submitting the query and determine responsive content to be provided for presentation to the user of the client device based on the corresponding conversation with the representative of the "restaurant." While the above examples are described with respect to relatively simple queries (e.g., submitting a query), it should be understood that this is by way of example only and that the same or similar techniques may be utilized in performing relatively complex tasks that may include multiple other tasks or subtasks that may be performed during a corresponding conversation.

[0006] In various implementations, the corresponding conversation may be a voice-based conversation in which the chatbot participates in the corresponding conversation via a telephone call or locally at the client device. In these implementations, the chatbot may additionally or alternatively be referred to as a voicebot. In other implementations, the corresponding conversation may be a text-based conversation in which the chatbot participates in the corresponding conversation with a text messaging or SMS service, an email service, or other text-based service. Thus, chatbots can be deployed in various environments to participate in corresponding conversations with various entities and perform various tasks. In some implementations, chatbots generated to perform various tasks can be fine-tuned in an “on-the-fly” manner, such that the chatbots are generated in response to receiving unstructured, free-form, natural language input. While some chatbots may be utilized to participate in corresponding conversations with multiple different entities (e.g., in parallel or serially), a corresponding chatbot may be generated based on each instance of unstructured, free-form, natural language input provided by a user, including task(s) to be performed on behalf of the user. In other implementations, the generated chatbot may not be fine-tuned in an "on-the-fly" manner, but may be identified and utilized in response to receiving unstructured, free-form, natural language input.

[0007] In various embodiments, the processor(s) may be implemented locally on a user's client device, where unstructured, free-form natural language input conveying details of the task(s) to be performed is received. In these embodiments, the processor(s) may retrieve a pre-trained large-scale language model (LLM) from the client device's on-device storage as an ML model ready for use in conducting more generalized conversations. Additionally, the processor(s) may fine-tune the pre-trained LLM based on the unstructured, free-form natural language input to generate a fine-tuned LLM. Additionally, the processor(s) utilize the fine-tuned LLM as a chatbot that participates in the corresponding conversation on behalf of the user. In other versions of these embodiments, the processor(s) may retrieve the pre-trained LLM from the client device's on-device storage as an ML model ready for use in conducting more generalized conversations, but may refrain from fine-tuning the pre-trained LLM based on the unstructured, free-form natural language input.

[0008] In other embodiments, the processor(s) may be implemented remotely from the user's client device (e.g., in a remote system such as a high-performance server or cluster of high-performance servers). In these embodiments, the processor(s) may retrieve pre-trained large-scale language models (LLMs) from remote storage at the remote system as ML models already available for use in conducting more generalized conversations. Additionally, the processor(s) may generate fine-tuned LLMs and utilize the fine-tuned LLMs as chatbots that participate in corresponding conversations on behalf of the user. Continuing with the above example, the processor(s) may cause the chatbot to process respective data to be posed during the corresponding conversation to determine whether the user left their jacket at the restaurant where they ate the previous night. In some versions of these embodiments, the processor(s) may retrieve pre-trained LLMs from remote storage at the remote system as ML models already available for use in conducting more generalized conversations, but may refrain from fine-tuning the pre-trained LLMs based on unstructured, free-form natural language input.

[0009] In particular, the pre-trained LLM may correspond to an existing LLM, such as LaMDA, BERT, Meena, GPT-3, or any other pre-trained LLM. These pre-trained LLMs are pre-trained with vast amounts of diverse data and can engage in corresponding conversations with users in a natural and intuitive manner. However, these LLMs have multiple ML layers and hundreds of millions to hundreds of billions of ML parameters. Thus, in embodiments in which the fine-tuned chatbot is generated locally on the client device, the pre-trained LLM obtained and fine-tuned may be a sparsified version of the pre-trained LLM. In contrast, in embodiments in which the fine-tuned chatbot is generated remotely from the client device, the pre-trained LLM obtained and fine-tuned may be a non-sparsified version of the pre-trained LLM. The sparsified version of the pre-trained LLM may have fewer ML layers, fewer ML parameters, masked weights, and / or other sparsified aspects to reduce the size of the pre-trained LLM due to various hardware and / or software constraints on the client device compared to the virtually infinite resources of the remote system.

[0010] In some implementations, when engaging a chatbot in a corresponding conversation, the processor(s) may use a fine-tuned LLM corresponding to the chatbot to process the task data (e.g., output generated using the ASR model(s), NLU model(s), and / or fulfillment model(s) or rule(s) and based on processing the unstructured free-form natural language input), audio data capturing the spoken input of any of the entity representatives, text data predicted to correspond to the audio data capturing the spoken input of any of the entity representatives, any conversation context data, and / or any other data described herein to generate the output. In other implementations, when engaging a chatbot in a corresponding conversation, the processor(s) may use a pre-trained LLM (e.g., not fine-tuned) to process the natural language description included in the unstructured free-form natural language input, the task data, audio data capturing the spoken input of any of the entity representatives, text data predicted to correspond to the audio data capturing the spoken input of any of the entity representatives, any conversation context data, and / or any other data described herein to generate the output. The output may be, for example, a probability distribution over a vocabulary or sequence of terms and / or phrases. Based on the probability distribution over the vocabulary or sequence of terms and / or phrases, the processor(s) may select instances of text data that correspond to text and / or speech provided by the chatbot.

[0011] In embodiments where the corresponding conversation is a text-based conversation, the processor(s) may cause instances of the text data to be visually rendered at the client device and / or additional client devices of the entity for presentation to a representative of the entity. However, in embodiments where the corresponding conversation is a voice-based conversation, the processor(s) may cause the chatbot to process the instances of the text data corresponding to the text data using text-to-speech (TTS) model(s) to generate instances of synthetic voice audio data that capture synthetic speech corresponding to the text data. Furthermore, the processor(s) may cause instances of the synthetic voice audio data to be visually rendered at the client device and / or additional client devices of the entity for presentation to a representative of the entity. In particular, in embodiments where the chatbot corresponds to a pre-trained LLM that is fine-tuned based on unstructured free-form natural language input, the chatbot may generate conversational output that directs attention to task(s) to be performed based on the unstructured free-form natural language input. Furthermore, in embodiments in which the chatbot corresponds to an unfine-tuned pre-trained LLM, the unstructured, free-form natural language input continues to be applied as input to the entire unfine-tuned pre-trained LLM, so that the chatbot can still generate conversational output that attends to the task(s) specified by the unstructured, free-form natural language input.

[0012] In various implementations, the processor(s) may cause responsive content to be provided for presentation to a user of a client device that provided the unstructured, free-form, natural language input. The responsive content may be determined based on one or more responses provided by a representative of the entity during the corresponding conversation. Further, the responsive content may include, for example, corresponding results of one or more tasks determined during the corresponding conversation, a corresponding summary of the corresponding conversation, and / or other content. Continuing with the above example, the chatbot may determine, in response to the responses provided by the representative of the entity and during the corresponding conversation, whether the user left a red leather jacket at the restaurant.

[0013] In various implementations, during the corresponding conversation, the chatbot may utilize one or more peripheral actions when participating in the corresponding conversation with the entity's representative. These peripheral actions may include, for example, a greeting action that enables the chatbot to identify the user and / or identify itself as a chatbot, a hold action that enables the chatbot to pause and resume the corresponding conversation, a leave action that enables the chatbot to end the corresponding conversation with the entity's representative, and / or other peripheral actions. These peripheral actions are some non-limiting examples of why a pre-trained LLM enables a chatbot to perform generalized conversational aspects, and unstructured, free-form, natural language input need not specify that the chatbot can perform these generalized conversational aspects. However, a fine-tuned chatbot that is fine-tuned based on unstructured, free-form, natural language input may enable the chatbot to perform a task(s) on behalf of a user while still being able to perform these generalized conversational aspects.

[0014] In various implementations, the processor(s) can discard a chatbot generated based on unstructured free-form natural language input in response to determining that one or more conditions are met. The processor(s) can discard the chatbot based on, for example, whether the chatbot successfully performs the task(s) associated with the entity, whether the chatbot is utilized in performing the task(s) associated with additional entities, whether a threshold duration has elapsed since the chatbot was generated, whether a threshold amount of on-device storage on the client device is consumed by the chatbot in implementations in which the chatbot is generated locally on the client device, whether a threshold amount of on-device storage on the client device is available while the chatbot is stored in the on-device storage in implementations in which the chatbot is generated locally on the client device, and / or other conditions. In other words, the system can balance the performance of the chatbot and how the chatbot impacts the client device when determining whether to discard the chatbot in implementations in which the chatbot is generated locally on the client device.

[0015] Various technical advantages can be achieved by using the techniques described herein. As one non-limiting example, the techniques described herein enable a processor(s) of a client device and / or a remote system to generate a chatbot based on unstructured, free-form natural language input to perform a task(s) specified by a user, and / or to utilize an existing chatbot based on unstructured, free-form natural language input to perform a task(s) specified by a user. These task(s) may be specified in a natural language description(s) provided by the user. This enables process(es) to generate and deploy chatbots to perform task(s) on behalf of a user in a fast and efficient manner. Furthermore, in some examples, a chatbot may be destroyed after performing a task(s) on behalf of a user. Thus, in implementations in which a chatbot is implemented locally on a client device, the techniques described herein balance the current and / or future performance of the chatbot with how the chatbot may affect the performance of the client device. Additionally, in embodiments in which the chatbot is implemented remotely from the client device (e.g., implemented by a remote system communicatively coupled to the client device), the task(s) may still be performed on behalf of the user while computational resources are stored on the client device.

[0016] The above description is provided as a summary of only some of the embodiments disclosed herein. These and other embodiments are described in further detail herein. [Brief explanation of the drawings]

[0017] [Figure 1] 1 illustrates various aspects of the present disclosure and shows a block diagram of an exemplary environment in which the embodiments disclosed herein may be implemented. [Figure 2]1 illustrates an exemplary process flow for creating a chatbot and having the chatbot participate in a conversation with an entity, according to various embodiments. [Figure 3] 1 depicts a flowchart illustrating an exemplary method for creating a chatbot locally on a client device and engaging the chatbot in a conversation with an entity, according to various embodiments. [Figure 4] 1 depicts a flowchart illustrating an exemplary method for remotely creating a chatbot in a remote system and having the chatbot participate in a conversation with an entity, according to various embodiments. [Figure 5A] 1 illustrates various non-limiting example interactions between corresponding unstructured free-form natural language inputs utilized to generate corresponding chatbots and corresponding chatbots that perform corresponding task(s) based on the corresponding unstructured free-form natural language inputs, according to various embodiments. [Figure 5B] 1 illustrates various non-limiting example interactions between corresponding unstructured free-form natural language inputs utilized to generate corresponding chatbots and corresponding chatbots that perform corresponding task(s) based on the corresponding unstructured free-form natural language inputs, according to various embodiments. [Figure 6A] 1 illustrates additional non-limiting example interactions between corresponding unstructured free-form natural language inputs utilized to generate corresponding chatbots and corresponding chatbots that perform corresponding task(s) based on the corresponding unstructured free-form natural language inputs, according to various embodiments. [Figure 6B] 1 illustrates additional non-limiting example interactions between corresponding unstructured free-form natural language inputs utilized to generate corresponding chatbots and corresponding chatbots that perform corresponding task(s) based on the corresponding unstructured free-form natural language inputs, according to various embodiments. [Figure 7] 1 illustrates an exemplary architecture of a computing device, according to various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0018] 1, a block diagram of an exemplary environment is shown that illustrates various aspects of the present disclosure and in which implementations disclosed herein may be implemented. A client device 110 is shown in FIG. 1 and, in various implementations, includes a user input engine 120, a rendering engine 130, an on-device machine learning (ML) model engine 140, and a chatbot deployment engine client 150. Client device 110 may be, for example, a standalone device (e.g., having microphone(s), visual component(s), speaker(s), display(s), and / or other user interface components), a laptop, a desktop computer, a tablet, a wearable computing device, a vehicle computing device, and / or any other client device capable of implementing chatbot deployment engine client 150.

[0019] The user input engine 120 can detect various types of user input at the client device 110. In some examples, the user input detected at the client device 110 can include spoken input detected via microphone(s) of the client device 110. In these examples, the microphone(s) of the client device 110 can generate audio data capturing the spoken utterance(s) included in the spoken input. In other examples, the user input detected at the client device 110 can include touch input detected via input device(s) of a user interface of the client device 110 (e.g., touch-sensitive display(s)) and / or typed input detected via user interface input device(s) of the client device 110 (e.g., touch-sensitive display(s) and / or keyboard(s)). In these examples, the user interface input device(s) of the client device 110 can generate text data capturing the touch input and / or typed input. In particular, the unstructured free-form natural language input described herein may be provided by a user of client device 110 as any combination of spoken input, touch input, and / or typed input.

[0020] The rendering engine 130 can cause the responsive content and / or other output to be rendered visually (e.g., via a touch-sensitive display or other user interface output device(s)) for presentation to a user at the client device 110 and / or audibly (e.g., via a speaker(s) or other user interface output device(s)) for presentation to a user at the client device 110. The responsive content and / or other output may include various types of user interfaces associated with the chatbot deployment engine client 150 that may be visually rendered via the user interface of the client device 110, such as, for example, unstructured, free-form, natural language input provided by the user of the client device 110 conveying details of the task(s) to be performed by the chatbot on behalf of the user of the client device, transcription(s) of the corresponding conversation(s) performed by the chatbot on behalf of the user of the client device 110, various prompts related to the corresponding conversation(s) performed by the chatbot on behalf of the client device 110, result(s) and / or summary(s) of the corresponding conversation(s) performed by the chatbot on behalf of the client device 110, and / or any other responsive content or output that may be visually and / or audibly rendered for presentation to a user at the client device 110.

[0021] On-device ML model engine 140, in various implementations, may include an automatic speech recognition (ASR) engine 141, a natural language understanding (NLU) engine 142, a fulfillment engine 143, and a text-to-speech (TTS) engine 144. As described in more detail below, these on-device ML model engines of on-device ML model engine 140 may utilize various on-device ML models (e.g., stored in on-device ML model(s) database 140A) to process various user inputs (e.g., received via user input engine 120) and generate various outputs (rendered visually and / or audibly for presentation to a user via rendering engine 130). This, in turn, enables chatbot deployment engine client 150 to utilize on-device ML model engine 140 in processing various user inputs received at client device 110 and in generating various outputs that are provided for presentation to a user at client device 110.

[0022] 1 as being communicatively coupled to a remote system 160 via one or more networks 199 (e.g., any combination of Wi-Fi, Bluetooth, or other local area networks (LANs), Ethernet, the Internet, or other wide area networks (WANs), and / or other networks). Remote system 160, in various implementations, includes a remote system ML model engine 170 and a chatbot deployment engine 180. Remote system 160 may be, for example, a high-performance server, a cluster of high-performance servers, and / or any other computing device that is remote from client device 110.

[0023] Remote ML model engine 170, in various implementations, may include ASR engine 171, NLU engine 172, fulfillment engine 173, and TTS engine 174. As described in more detail below, these remote ML model engines of remote engine 170 can utilize various remote ML models (e.g., stored in remote ML model(s) database 170A) to process various user inputs (e.g., received via user input engine 120) and generate various outputs (e.g., visually and / or audibly rendered for presentation to a user via rendering engine 130) based on data provided by client device 110 to remote system 160 over one or more networks 199 in the same or similar manner as on-device ML model engine 140. This, in turn, enables chatbot deployment engine 180 to leverage remote ML model engine 170 in processing various user inputs received at client device 110 and in generating various outputs that are provided for presentation to a user at client device 110. In implementations in which remote ML model engine 170 processes various user inputs received at client device 110 and generates various outputs that are provided for presentation to a user at client device 110, the various user inputs received at client device 110 may be transmitted to remote system 160 (e.g., via one or more networks 199), and the various user outputs may be transmitted back to client device 110 (e.g., via one or more networks 199).

[0024] In particular, chatbot deployment engine client 150 of a client device can communicate with chatbot deployment engine 180 over one or more networks 199. Chatbot deployment engine client 150 and chatbot deployment engine 180 form a logical instance of a chatbot deployment platform from the perspective of a user interacting with client device 110. It should be understood that while chatbot deployment platform is illustrated in FIG. 1 as being implemented in a distributed manner over one or more networks 199 (e.g., utilizing chatbot deployment engine client 150 and chatbot deployment engine 180), this is for illustrative purposes and is not meant to be limiting. For example, chatbot deployment platform could alternatively be implemented solely on client device 110. As another example, chatbot deployment platform could alternatively be implemented solely on remote system 160, while client device 110 would still be utilized to allow a user to interact with the chatbot deployment platform.

[0025] The chatbot deployment platform can be utilized by a user (e.g., a user of client device 110) to train and deploy a chatbot described herein to conduct corresponding conversations on behalf of the user and / or on behalf of a third party associated with the user (e.g., via third-party system(s) 192). In particular, the chatbot deployment platform can be provided by a first party, and a user can utilize the chatbot deployment platform to generate a chatbot for themselves or for a third party associated with the user. As used herein, the term first party refers to an entity that publishes the chatbot deployment platform, and the term third party refers to an entity that, unlike an entity associated with the first party, did not publish the chatbot deployment system. Thus, a user of client device 110 that interacts with the chatbot deployment platform may also be referred to as a third-party deployer.

[0026] The corresponding conversations described herein as conducted by the chatbot and on behalf of the user of the client device 110 may include various types of conversations, such as voice-based conversations and text-based conversations. Voice-based conversations may include, for example, corresponding conversations conducted during automated telephone calls (e.g., Voice over Internet Protocol (VoIP), Public Switched Telephone Network (PSTN), and / or other telephony protocols) and between the client device and additional client device(s) 191, as well as corresponding conversations in which the chatbot participates with other entities and / or users, locally at a given client device (e.g., in a scenario in which the client device 110 is a shared client accessible by multiple users), and / or in any other voice-based scenario in which the chatbot is deployed to conduct corresponding conversations with users. Text-based conversations may include, for example, corresponding conversations conducted during text or SMS messaging, email, and / or any other text-based scenario in which the chatbot is deployed to conduct corresponding conversations with users.

[0027] As described above, the chatbot deployment platform may utilize on-device ML model engine 140 and / or remote system ML model engine 170 in processing various user inputs received at client device 110 and in generating various outputs provided for presentation to the user at client device 110. Each of the sub-engines of on-device ML model engine 140 and / or remote system ML model engine 170 may be configured to perform one or more functions. In particular, remote system ML model engine 170 includes remote-based counterparts of the sub-engines of on-device ML model engine 140. In various implementations, utilization of on-device ML model engine 140 may be prioritized, at least in part, by latency considerations, network bandwidth privacy considerations, and / or other considerations. In these implementations, remote system ML model engine 170 may be utilized when one or more of the sub-engines of on-device ML model engine 140 fail. In other embodiments, utilization of the remote ML model engine 170 may be prioritized, at least in part, by computational considerations at the client device 110, hardware considerations at the client device 110, software considerations at the client device 110, and / or other considerations. In still other embodiments, the on-device ML model engine 140 and the remote system ML model engine 170 may be utilized in conjunction with one another.

[0028] For example, ASR engines 141 and / or 171 may use ASR model(s) stored in their respective ML model(s) databases (e.g., recurrent neural network (RNN) models, Transformer models, and / or any other type of ML model capable of performing ASR) to process audio data captured from spoken speech and generated by microphone(s) of client device 110 to generate ASR output. Additionally, NLU engines 142 and / or 172 may use NLU model(s) and / or NLU rule(s) stored in their respective ML model(s) databases (e.g., long short-term memory (LSTM), gated recurrent unit (GRU), and / or any other type of RNN or other ML model capable of performing NLU) to process the ASR output (or other typed or touch input received via user input engine 120 of client device 110) to generate NLU output. Additionally, fulfillment engines 143 and / or 173 may process NLU data using fulfillment model(s) and / or fulfillment rules stored in their respective ML model(s) databases to generate fulfillment output. Finally, TTS engines 144 and / or 174 may process text data (e.g., text formulated by a chatbot) using TTS model(s) stored in their ML model(s) databases to generate synthetic speech audio data, including computer-generated synthetic speech.

[0029] In various implementations, the ASR output may include, for example, multiple phonetic hypotheses (e.g., term hypotheses and / or transcription hypotheses) for the spoken input based on processing of the audio data, and optionally, a particular phonetic hypothesis may be selected as the recognized text for the spoken input text based on corresponding values ​​(e.g., probability values, log-likelihood values, and / or other values) associated with each of the multiple phonetic hypotheses. In various implementations, the ASR model(s) stored in each ML model database are end-to-end speech recognition model(s), such that the ASR engines 141 and / or 171 can use the models to directly generate multiple phonetic hypotheses. For example, the ASR model(s) may be end-to-end model(s) used to generate each of multiple phonetic hypotheses for each character (or other token). One non-limiting example of such end-to-end model(s) used to generate recognized text for each character string is a recurrent neural network transducer (RNN-T) model. An RNN-T model is a form of sequence-to-sequence model that does not use an attention mechanism. In other implementations, the ASR model(s) are not end-to-end speech recognition model(s), such that the ASR engines 141 and / or 171 can instead generate predicted phoneme(s) (and / or other representations). For example, the predicted phoneme(s) (and / or other representations) can then be utilized by the ASR engines 141 and / or 171 to determine multiple phonetic hypotheses that match the predicted phoneme(s). In doing so, the ASR engines 141 and / or 171 can optionally use a decoding graph, a lexicon, and / or other resource(s). In various implementations, a corresponding transcription can be rendered at the client device 110 (e.g., in conjunction with the training instance input, the training instance output, the corresponding feature highlight input(s), the illustrative dialogue, and / or other aspects of the chatbot deployment platform).

[0030] In various implementations, the NLU output may include annotated recognized text, including, for example, one or more annotations of the recognized text for one or more (e.g., all) of the terms in the recognized text. For example, NLU engine 142 and / or 172 may include a portion of a speech tagger (not shown) configured to annotate terms with their grammatical roles. Additionally or alternatively, NLU engine 142 and / or 172 may include an entity tagger (not shown) configured to annotate entity references in one or more segments of the recognized text, such as references to people (e.g., including literary characters, famous people, public figures, etc.), organizations, places (real and fictional), etc. In some implementations, data about entities may be stored in one or more databases, such as a knowledge graph (not shown). In some implementations, the knowledge graph may include nodes representing known entities (and, in some cases, entity attributes) and edges connecting the nodes and representing relationships between the entities. The entity tagger may annotate references to entities at a high level of granularity (e.g., to enable identification of all references to an entity class, such as people) and / or at a lower level of granularity (e.g., to enable identification of all references to a particular entity, such as a particular person). The entity tagger may rely on the content of the natural language input to resolve specific entities and / or may optionally communicate with a knowledge graph or other entity database to resolve specific entities. Additionally or alternatively, NLU engine 142 and / or 172 may include a coreference resolver (not shown) configured to group or "cluster" references to the same entity based on one or more contextual cues.For example, the coreference resolver may be utilized to resolve the term “them” in the natural language input “buy those” to “buy theater tickets” based on the fact that “theater tickets” was mentioned in a client device notification rendered immediately prior to receiving the input “buy those.” In some implementations, one or more components of NLU engine 142 and / or 172 may rely on annotations from one or more other components of NLU engine 142 and / or 172. For example, in some implementations, the entity tagger may rely on annotations from the coreference resolver to annotate all references to a particular entity. Also, for example, in some implementations, the coreference resolver may rely on annotations from the entity tagger to cluster references to the same entity. Also, for example, in some implementations, the coreference resolver may rely on user data (e.g., stored in user data database 110A) of a user of client device 110 for coreference resolution and / or entity resolution. User data may include, for example, historical location data, historical time data, user preference data, user account data, calendar information, email data, and / or any other user data accessible at client device 110 .

[0031] In various implementations, the fulfillment output may include, for example, one or more tasks to be performed by the chatbot and on behalf of the user of client device 110. As described in more detail herein (e.g., with respect to FIGS. 2, 3, 4, 5A, 5B, 6A, and 6B), a user may provide unstructured, free-form, natural language input including one or more tasks to be performed by the chatbot on behalf of the user of client device 110. The one or more tasks may require the chatbot to participate in a corresponding conversation with an entity (or a representative of the entity). In particular, the unstructured, free-form, natural language input may convey details of one or more tasks to be performed by the chatbot without a corresponding dialogue state map utilized by the chatbot in conducting the corresponding conversation. Nevertheless, by utilizing chatbot deployment engine client 150 and / or chatbot deployment engine 180, a chatbot may be generated and deployed to perform one or more tasks in response to receiving unstructured, free-form, natural language input. It should therefore be understood that the fulfillment output may be based on one or more tasks performed by the chatbot, or may be based on responsive content determined during a corresponding conversation with the entity (or a representative of the entity).

[0032] In various implementations, TTS engine 144 and / or 174 can generate synthetic speech audio data that captures computer-generated synthetic speech. The synthetic speech audio data can be rendered at client device 110 via speaker(s) of client device 110 and / or at additional client device(s) 191 (e.g., client devices associated with entities and / or entity representatives) via respective speaker(s) of the additional client device(s). The synthetic speech can include any output generated by a chatbot described herein, including, for example, synthesized speech generated as part of an interaction between a user of client device 110 and the chatbot, and synthesized speech generated as part of an interaction between an entity (or entity representative) and the chatbot, and / or other synthesized speech.

[0033] 1 is described with respect to a single client device with a single user, it should be understood that this is for purposes of illustration and not meant to be limiting. For example, one or more additional client devices of the user may also implement the techniques described herein. For example, client device 110, one or more additional client devices, and / or any other computing devices of the user may form an ecosystem of devices that may employ the techniques described herein. These additional client devices and / or computing devices may communicate with client device 110 and / or remote system 160 (e.g., via one or more networks 199). As another example, a given client device may be utilized by multiple users in a shared setting (e.g., a group of users, a household, etc.).

[0034] Chatbot deployment engine 180, in various embodiments, may include a chatbot identification engine 181, a chatbot fine-tuning engine 182, a task / entity identification engine 183, a conversation engine 184, a conversation context engine 185, a responsive content engine 186, and a peripheral action engine 187, as shown in FIG. 1 . While chatbot deployment engine 180 is shown as having certain sub-engines, it should be understood that this is for purposes of illustration and is not meant to be limiting. For example, one or more of the sub-engines shown in FIG. 1 may be combined, or one or more other sub-engines shown in FIG. 1 may be omitted. Furthermore, while chatbot deployment engine client 150 is not shown as including any sub-engines, it should be understood that this is for purposes of simplicity and is not meant to be limiting. For example, chatbot deployment engine client 150 may include the same sub-engines described with respect to chatbot deployment engine 180, or a subset thereof. Additional description of chatbot deployment engine 180 and its various sub-engines is provided with respect to FIG. 2.

[0035] Referring now to Figure 2, an example process flow 200 for creating a chatbot and engaging the chatbot in a conversation with an entity is shown. For illustrative purposes, assume that a user of client device 110 of Figure 1 provides unstructured, free-form, natural language input 201 as input at client device 110. Client device 110 may receive unstructured, free-form, natural language input 201 via user input engine 120 of client device 110. Client device 110 can cause unstructured, free-form, natural language input 201 to be processed using various sub-engines of on-device ML model engine 140 and / or various sub-engines of remote ML model engine 170.

[0036] In particular, when processing the unstructured free-form natural language input 201, the client device 110 can identify one or more features 202 based on output generated by one or more of the various sub-engines of the on-device ML model engine 140 and / or one or more of the various sub-engines of the remote ML model engine 170. The one or more features 202 can include, for example, an ASR output when the unstructured free-form natural language input 201 is spoken input, an NLU output when the unstructured free-form natural language input 201 is spoken input or typed input, and / or a fulfillment output when the unstructured free-form natural language input 201 is spoken input or typed input, such as an entity, intent, slot value, task(s) associated with the entity, and / or other features to be performed by the chatbot.

[0037] For illustrative purposes, further assume that the unstructured, free-form natural language input 201 is the spoken utterance of "Call the example hotel and ask if they have a pet fee for small dogs." In this example, the ASR output may be recognized text corresponding to the spoken utterance (e.g., the recognized text of "Call the example hotel and ask if they have a pet fee for small dogs"), and the NLU output may have a slot value of "example hotel," which may be an intent of "phone call" associated with the task of submitting the query "does the example hotel have a pet fee for small dogs?" to a representative of the "example hotel." Thus, in some of these examples, in response to receiving the unstructured, free-form natural language input 201, the client device 110 may generate a chatbot that is configured to call a phone number associated with the "example hotel" to submit the query "does the example hotel have a pet fee for small dogs?" to a representative of the "example hotel" during a training phase (e.g., as enclosed by box 280A in FIG. 2 ).

[0038] During the training phase, the chatbot identification engine 181 can identify a chatbot 203 (e.g., stored in the chatbot(s) database 180A). The chatbot 203 may be a pre-trained ML model or a combination of various pre-trained ML models that can be fine-tuned based on unstructured free-form natural language input 201 and / or one or more features 202 extracted from the unstructured free-form natural language input 201. For example, the chatbot 203 may correspond to a pre-trained large-scale language model (LLM), such as LaMDA, BERT, Meena, GPT-3, and / or other pre-trained LLMs. In particular, these pre-trained LLMs are pre-trained with vast amounts of diverse data and are typically generative ML models that can engage in corresponding conversations with users in a more natural and intuitive way. These LLMs have multiple ML layers and hundreds of millions to hundreds of billions of ML parameters, allowing them to generalize corresponding conversations with users. For example, as described in more detail herein (e.g., with respect to FIGS. 5A, 5B, and 6B), text data can be provided as input across these pre-trained LLMs to generate LLM outputs, such as probability distributions over vocabularies, and responses to the text data can be generated based on the probability distributions over the vocabulary. Note that, due to the multiple ML layers and hundreds of millions to hundreds of billions of ML parameters, LLMs typically do not lend themselves to being implemented locally at client device 110, such as when chatbot(s) database 180A is located locally at client device 110 (e.g., stored in on-device storage of client device 110). Nevertheless, various sparsification techniques can be utilized to reduce the amount of ML layers and / or ML parameters utilized by these LLMs, such that sparsified versions of the pre-trained LLMs can be implemented locally at client device 110 while mitigating accuracy loss and / or retraction of the pre-trained LLMs due to sparsification.These sparsification techniques may include, but are not limited to, aggregating and / or combining layers of multiple ML layers of pre-trained LLMs, pruning layers of multiple ML layers of pre-trained LLMs, masking weights of pre-trained LLMs, pruning weights of pre-trained LLMs, and / or other sparsification techniques. However, if chatbot(s) database 180A is remote from client device 110 (e.g., stored in remote storage at remote system 160), the non-sparsed version of the pre-trained LLMs may be implemented remotely at remote system 160. Thus, chatbot 203 may be identified locally at client device 110 and / or remotely at remote system 160 (e.g., remotely from client device 110 that received unstructured free-form natural language input 201).

[0039] Additionally, during the training phase, the chatbot fine-tuning engine 182 may utilize various fine-tuning techniques to fine-tune the chatbot 203 to generate a fine-tuned chatbot 204 based on the unstructured free-form natural language input 201 and / or one or more features 202 extracted from the unstructured free-form natural language input 201 (and the fine-tuned chatbot 204 may optionally be stored in the chatbot(s) database 180A). These fine-tuning techniques may include, but are not limited to, command adjustment, short-shot learning, and / or other fine-tuning techniques, and the fine-tuning performed may vary based on the unstructured free-form natural language input 201 provided by the user. In other words, the pre-trained LLM corresponding to the chatbot 203 may be further trained based on the unstructured free-form natural language input 201 and / or one or more features 202 extracted from the unstructured free-form natural language input 201. This fine-tunes the pre-trained LLM corresponding to the fine-tuned chatbot 204 so that it is adapted to perform task(s) on behalf of the user. By fine-tuning the chatbot 203, the resulting fine-tuned chatbot 204 is also adapted to perform task(s) associated with the entity on behalf of the user while leveraging the generalization capabilities of the pre-trained LLM. Thus, the chatbot 203 can be fine-tuned locally at the client device 110 and / or remotely at the remote system 160 (e.g., remotely from the client device 110 that received the unstructured free-form natural language input 201) to generate the fine-tuned chatbot 204. The fine-tuned chatbot 204 may then be utilized during an inference phase (e.g., enclosed by box 280B in FIG. 2).

[0040] 2 is described with respect to fine-tuning a chatbot 203 based on unstructured free-form natural language input 201 and / or one or more features 202 of the unstructured free-form natural language input 201 to generate a fine-tuned chatbot 204, it should be understood that this is but one implementation contemplated herein. For example, in other implementations, the chatbot 203 may not be fine-tuned such that the chatbot 203 can be subsequently utilized by the client device 110 and / or the remote system 160 during the inference phase (e.g., enclosed by box 280B in FIG. 2).

[0041] During the inference phase, the task / entity identification engine 183 may determine that the task data 205 and entity data 206 are to be utilized by the chatbot in performing the task(s) during the corresponding conversation. Continuing with the example above, where the unstructured, free-form natural language input 201 is the spoken utterance of "Call example hotel and inquire if they have a pet fee for small dogs," the task data 205 may include, for example, a task to submit the query "Do they have a pet fee for small dogs" to a representative of "example hotel." Further, the entity data 206 may include a corresponding identifier for "example hotel," such as a phone number for initiating a call with a representative of "example hotel." In various implementations, information regarding the tasks and / or entities may be stored in the task / entity database 180B and / or other data sources accessible by the client device 110. While the task data 205 and entity data 206 are described as including particular data, it should be understood that this is for purposes of illustration and not meant to be limiting. For example, task data 205 may include any data related to any task that may be specified by a user of client device 110 in unstructured, free-form natural language input 201. Additionally, entity data 106 may include an indication of any entity and / or any corresponding identifier of the entity. Furthermore, while task data 205 in the above examples includes only a single task and entity data 206 identifies only a single entity, it should be understood that this is for purposes of illustration and not meant to be limiting. For example, task data 205 may include any data for multiple tasks that may be specified by a user of client device 110 in unstructured, free-form natural language input 201.Additionally, entity data 206 may identify multiple entities that belong to a particular type of entity (e.g., if unstructured free-form natural language input 201 instead responded to the spoken utterance "call a nearby hotel and ask if they have a pet fee for small dogs," where "hotel" is a particular type of entity, a particular hotel in geographic proximity to the user may be identified).

[0042] In particular, the task / entity identification engine 183 can utilize the entity data 206 to initiate a corresponding conversation with a representative of the entity (e.g., by calling a phone number associated with “example hotel” and placing an outgoing call to a given additional client device 191A via the chatbot 203 or fine-tuned chatbot 204) and can provide the task data 205 to the conversation engine 184, which can enable the chatbot 203 or fine-tuned chatbot 204 to participate in the corresponding conversation with the representative of the entity. In various implementations, the conversation context engine 185 can provide conversation context data 207 to the conversation engine 184, which can further enable the chatbot 203 or fine-tuned chatbot 204 to participate in a more contextualized corresponding conversation with a representative of the entity task data 205. In these implementations, the conversation context data 207 may represent (e.g., as a vector or other data structure) initial context information for the corresponding conversation or subsequent context information determined during the corresponding conversation (e.g., determined based on data stored in the chatbot activity database 180C).

[0043] Continuing with the above example where the unstructured free-form natural language input 201 is the spoken utterance of “Call the example hotel and inquire about whether there is a pet fee for small dogs,” the conversation context engine 185 can generate conversation context data 207 indicating that “[the user] intends to stay at the example hotel,” information associated with the intended stay of the user of the client device 110 (e.g., check-in or check-out date and time from the user's email account of the client device 110, an “example hotel” loyalty program number from the user's email account of the client device 110, or an “example hotel” software application accessible at the client device 110), that “[the user] wants to bring his or her small dog,” and / or other context information that can be inferred based on the unstructured free-form natural language input 201 and / or other data accessible at the client device 110 (e.g., via the user data database 110A).

[0044] Further, during the inference phase and after initiation of correspondence (e.g., using entity data 206), and in embodiments where the chatbot corresponds to a fine-tuned chatbot 204, the conversation engine 184 can first process the task data 205 (and optionally the conversation context data 207) using the fine-tuned chatbot 204 to generate output, such as a probability distribution over sequences of words or phrases. The conversation engine 184 can generate conversation data 208 based on the output generated using the fine-tuned chatbot 204. The conversation data 208 can include, for example, one or more instances of synthesized voice audio data in embodiments where the corresponding conversation is a voice-based conversation, or one or more instances of text data in embodiments where the corresponding conversation is a text-based conversation. In various embodiments, as shown in FIG. 2, the conversation data 208 can be transmitted to a given additional client device 191A so that the conversation data 208 can be audibly and / or visually rendered at the given additional client device 191A. However, in other implementations, such as when the fine-tuned chatbot 204 participates in a conversation that corresponds locally on the client device 110 (e.g., when the client device 110 is deployed in a shared setting), the conversation data 208 may be rendered audibly and / or visually on the client device 110.

[0045] Alternatively, during the inference phase, and in embodiments where the chatbot corresponds to a chatbot 203 (e.g., not a fine-tuned chatbot 204), the conversation engine 184 can first process the unstructured, free-form, natural language input 201, one or more features 202 determined based on the unstructured, free-form, natural language input 201, the task data 205 (and optionally any other data described herein) using the chatbot 203 to generate output, such as a probability distribution over sequences of words or phrases. The conversation engine 184 can generate conversation data 208 based on the output generated using the chatbot 203. The conversation data 208 can include, for example, instance(s) of synthesized speech audio data in embodiments where the corresponding conversation is a voice-based conversation, or instances of text data in embodiments where the corresponding conversation is a text-based conversation. In various embodiments, the conversation data 208 can be transmitted to a given additional client device 191A, as shown in FIG. 2, so that the conversation data 208 can be audibly and / or visually rendered at the given additional client device 191A. However, in other implementations, such as when the chatbot 203 participates in a corresponding conversation locally at the client device 110 (e.g., when the client device 110 is deployed in a shared setting), the conversation data 208 may be audibly and / or visually rendered at the client device 110. In other words, rather than fine-tuning the chatbot 203 during a training phase, the chatbot 203 may be primed during an inference phase based on the unstructured free-form natural language input 201 and / or one or more features 202 determined based on processing the unstructured free-form natural language input 201. This allows the client device 110 and / or the remote system 160 to conserve computational resources while effectively deploying the chatbot to participate in a corresponding conversation.

[0046] Continuing with the above example, where the unstructured free-form natural language input 201 is the spoken utterance of "Call example hotel and ask if they have a pet fee for small dogs," the conversation data 208 may include synthesized speech audio data, which is to be audibly rendered at a given additional client device 191A, including the synthesized speech of "Hi, this is a chatbot calling on behalf of [user], do you have a pet fee for small dogs?" In particular, the synthesized speech includes the query, "Do you have a pet fee for small dogs?" and solicits a response from a representative of the entity answering the phone call initiated by the chatbot 203 or fine-tuned chatbot 204. Thus, the response data 209 may include a response to the query included in the synthesized speech. The response data 209 may include audio data provided by the representative of the entity. In these implementations, the responsive content engine 186 may utilize the ML model engine(s) 140 and / or 170 to process using various ML model(s) to determine whether the response data 209 includes a response indicating that the task was successfully performed.

[0047] For example, further assume that the response data 209 includes audio data capturing a spoken utterance from an entity representative: "Yes, the pet fee for a small dog is $25 per night." In this instance, the responsive content engine 186 can cause the audio data provided by the entity representative to be processed (e.g., input parsing using ASR model(s), NLU model(s), and / or fulfillment rules) and determine that the entity representative has provided a response responsive to the query. In other words, the responsive content engine 186 can determine responsive content 210 for the query provided by the entity representative that corresponds to the task in this example (e.g., the pet fee for a small dog is $25 per night). Further, the responsive content engine 186 can provide the responsive content 210 to the rendering engine 130, causing the client device to audibly and / or visually provide rendered responsive content 211 for presentation to the user. Thus, the implementations described herein enable a user to provide unstructured free-form natural language input 201 such that a fine-tuned bot 204 is generated and utilized in performing the task(s) contained in the unstructured free-form natural language input 201.

[0048] As described in more detail herein (e.g., with respect to Figures 5A, 5B, and 6B), the chatbot 203 and the fine-tuned chatbot 204 may have various peripheral behaviors that may be implemented by the chatbot 203 and the fine-tuned chatbot 204 using the peripheral behavior engine 187. These peripheral actions may include, but are not limited to, a greet action that enables the chatbot 203 and the fine-tuned chatbot 204 to identify the user of the client device 110 and / or identify itself as a chatbot; a remote procedure call (RPC) action that enables the chatbot 203 and the fine-tuned chatbot 204 to search one or more databases during the corresponding conversation; a hold action that enables the chatbot 203 and the fine-tuned chatbot 204 to pause and resume the corresponding conversation; a leave action that enables the chatbot 203 and the fine-tuned chatbot 204 to prompt the user of the client device 110 to join the corresponding conversation if requested by a representative of the entity and / or to otherwise terminate the corresponding conversation; a clarification action that enables the chatbot 203 and the fine-tuned chatbot 204 to clarify and / or repeat information previously provided during the corresponding conversation; and / or other peripheral actions that may be invoked by the chatbot 203 and the fine-tuned chatbot 204 when conditions for invoking those other peripheral actions are met.

[0049] 2 is described with respect to a corresponding conversation being a telephone call between chatbot 203 or fine-tuned chatbot 204 (e.g., implemented locally at client device 110 and / or remotely at remote system 160) and a representative of an entity (e.g., accessible at a given additional client device 191A), it should be understood that this is not meant to be limiting. Rather, it should be understood that the techniques described herein can be utilized to fine-tune chatbots that can be deployed to participate in voice-based and text-based conversations that take place between multiple computing devices and / or across a single computing device.

[0050] Referring now to FIG. 3 , a flowchart illustrating an example method 300 of generating a chatbot locally on a client device and engaging the chatbot in a conversation with an entity is shown. For convenience, the operations of method 300 are described with reference to a system that performs the operations. This system of method 300 includes at least one processor, memory, and / or other component(s) of client device(s) (e.g., client device 110 of FIG. 1 , computing device 710 of FIG. 7 , and / or other client devices). Furthermore, although the operations of method 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0051] At block 352, the system receives unstructured free-form natural language input from a user of the client device, the unstructured free-form natural language input including one or more tasks associated with the entity. The unstructured free-form natural language input received from the user of the client device may include, for example, spoken input received via microphone(s) of the client device, typed input received via a touch-sensitive display of the client device, and / or touch input received via a touch-sensitive display of the client device. Additionally, the unstructured free-form natural language input may convey details of one or more tasks to be performed by the chatbot and on behalf of the user of the client device. Notably, the unstructured free-form natural language input is unstructured in the sense that the user is not required to provide the free-form natural language input according to any schema or particular manner.

[0052] At block 354, the system generates a chatbot locally at the client device to perform one or more tasks related to the entity on behalf of the user based on at least the unstructured, free-form natural language input. In some implementations, the system retrieves a pre-trained large-scale language model (LLM) stored locally at the client device, as shown in block 354A. Further in these implementations, the system fine-tunes the pre-trained LLM stored locally at the client device based on the unstructured, free-form natural language input to generate a fine-tuned LLM, as shown in block 354B. Further in these implementations, the system utilizes the fine-tuned LLM as the chatbot, as shown in block 354C. The system can generate the chatbot to perform one or more tasks related to the entity on behalf of the user in the same or similar manner as described above with respect to FIG. 2 (e.g., in implementations where the training phase is performed locally at the client device and is described with respect to box 280A). Notably, in these embodiments, the system is implemented locally on the client device, and as a result, the pre-trained LLM may be a sparsified version of a pre-trained LLM that may otherwise be available (e.g., available to a remote system) due to various hardware and / or software constraints of the client device.

[0053] At block 356, the system causes the chatbot to perform one or more tasks related to the entity on behalf of the user. In some implementations, as shown in block 356A, the system causes the chatbot to engage in a corresponding conversation with the entity by rendering multiple instances of synthesized voice audio data for presentation to a representative associated with the entity and / or rendering multiple instances of text data for presentation to a representative of the entity. Further, in these implementations, as shown in block 356B, the system determines responsive content responsive to one or more of the instances of synthesized voice audio data and / or one or more of the instances of text data. The system may cause the chatbot to perform one or more tasks related to the entity on behalf of the user in the same or similar manner as described above with respect to FIG. 2 (e.g., in implementations where the inference phase is performed locally on the client device and is described with respect to box 280B) and with respect to FIGS. 5A, 5C, and 6B.

[0054] At block 358, the system determines whether the chatbot successfully performed one or more tasks associated with the entity. The system can determine whether the chatbot successfully performed one or more tasks associated with the entity based on, for example, responsive content responsive to one or more of the instances of synthesized voice audio data and / or one or more of the instances of text data. Various non-limiting examples of determining whether the chatbot successfully performed one or more tasks related to an entity are described herein (e.g., with respect to FIGS. 5A, 5B, and 5C).

[0055] If, in the iteration of block 358, the system determines that the chatbot successfully performed one or more tasks associated with the entity, the system may proceed to block 360. At block 360, the system causes responsive content to be provided for presentation to the user of the client device. The responsive content may be determined based on, for example, one or more responses provided by a representative of the entity during the corresponding conversation. The system proceeds to block 364. Block 364 is described in more detail below.

[0056] If, in the iteration of block 358, the system determines that the chatbot did not successfully perform one or more tasks related to the entity, the system may proceed to block 362. In block 362, the system prompts the user to join a corresponding conversation with a representative of the entity. In other words, if the chatbot does not successfully perform one or more tasks related to the entity, the chatbot may prompt the user to join a corresponding conversation to ensure that the one or more tasks are still performed. The system proceeds to block 364.

[0057] At block 364, the system determines whether to discard the chatbot. The system may determine whether to discard the chatbot based on, for example, whether the chatbot successfully performs one or more tasks associated with the entity, whether the chatbot is utilized in performing one or more task(s) associated with additional entities, whether a threshold duration has elapsed since the chatbot was created, whether a threshold amount of on-device storage on the client device is consumed by the chatbot, whether a threshold amount of on-device storage on the client device is available while the chatbot is stored in the on-device storage, and / or other conditions. In other words, the system may balance the performance of the chatbot and how the chatbot impacts the client device when determining whether to discard the chatbot.

[0058] If, in the iteration of block 364, the system determines not to discard the chatbot, the system may proceed to block 366. In block 366, the system continues to utilize the chatbot. For example, the system may continue to utilize the chatbot if one or more tasks associated with an entity were not successfully performed, but one or more tasks can be successfully performed for additional entities. For example, if the unstructured free-form natural language input corresponds to the spoken utterance "Find a local plumber available as soon as possible," but the responsive content determined based on calling a first local plumber indicates that the first local plumber is unavailable, the system may continue to utilize the chatbot to call a second local plumber. In this case, the system may call the first local plumber and the second local plumber serially or in parallel.

[0059] If, in an iteration of block 364, the system determines to destroy the chatbot, the system may proceed to block 368. At block 368, the system destroys the chatbot. For example, the system may destroy the chatbot in response to determining that one or more tasks associated with the entity have been successfully performed, and optionally, in response to determining that there are no additional entities to participate in the corresponding conversation. As another example, the system may destroy the chatbot if the chatbot consumes more than a threshold amount of memory resources on the client device or if the chatbot makes less than a threshold amount of memory resources available on the client device.

[0060] The system can perform other iterations of method 300 based on additional unstructured, free-form, natural language input received by the system. While Figure 3 is described with respect to a system implemented locally on a user's client device, it should be understood that this is for purposes of illustration and not meant to be limiting. For example, as shown below with respect to Figure 4, the system may be implemented by a remote system separate from the user's client device that provided the unstructured, free-form, natural language input.

[0061] Additionally, while the method 300 of FIG. 3 is described with respect to the chatbot being a pre-trained LLM that is fine-tuned based on unstructured, free-form natural language input (e.g., block 354), it should be understood that this is for purposes of illustration and not meant to be limiting. In additional or alternative implementations, the pre-trained LLM may be utilized as the chatbot without fine-tuning. In these implementations, when engaging the chatbot in a given corresponding conversation with a given additional user (e.g., block 356), the system may prime the pre-trained LLM based on the unstructured, free-form natural language input. This allows the chatbot to participate in the corresponding conversation without explicit fine-tuning during the training phase.

[0062] 4, a flowchart illustrating an example method 400 for remotely generating a chatbot at a remote system and engaging the chatbot in a conversation with an entity is shown. For convenience, the operations of method 400 are described with reference to a system that performs the operations. This system of method 400 includes at least one processor, memory, and / or other component(s) of a remote system(s) (e.g., remote system 160 of FIG. 1 , computing device 710 of FIG. 7 , and / or other remote systems). Furthermore, although the operations of method 400 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0063] At block 452, the system receives an indication of unstructured free-form natural language input provided by a user of the client device, the indication of the unstructured free-form natural language input including one or more tasks related to the entity. The unstructured free-form natural language input received from the user of the client device may include, for example, spoken input received via microphone(s) of the client device, typed input received via a touch-sensitive display of the client device, and / or touch input received via a touch-sensitive display of the client device. Further, the unstructured free-form natural language input may convey details of one or more tasks to be performed by the chatbot and on behalf of the user of the client device. Furthermore, the indication of the unstructured free-form natural language input may be transmitted to the system via one or more networks. Notably, the unstructured free-form natural language input is unstructured in the sense that the user is not required to provide the free-form natural language input according to any schema or particular manner.

[0064] At block 454, the system generates a chatbot remotely at the remote system to perform one or more tasks related to the entity on behalf of the user based on instructions from at least the unstructured, free-form natural language input. In some embodiments, the system retrieves a pre-trained large-scale language model (LLM) stored remotely at the remote system, as shown in block 454A. Further in these embodiments, the system fine-tunes the pre-trained LLM stored remotely at the remote system based on the unstructured, free-form natural language input to generate a fine-tuned LLM, as shown in block 454B. Further in these embodiments, the system utilizes the fine-tuned LLM as the chatbot, as shown in block 454C. The system can generate the chatbot to perform one or more tasks related to the entity on behalf of the user in the same or similar manner as described above with respect to FIG. 2 (e.g., in embodiments where the training phase is performed remotely at the remote system and described with respect to box 280A). In particular, in these embodiments, the system is implemented remotely at a remote system from the client device, and as a result, the pre-trained LLM may be an unsparsified version of the pre-trained LLM that is more robust than a sparsified version of the pre-trained LLM due to the virtual absence of hardware and / or software constraints at the remote system.

[0065] At block 456, the system causes the chatbot to perform one or more tasks related to the entity on behalf of the user. In some implementations, as shown in block 456A, the system causes the chatbot to participate in a corresponding conversation with the entity by rendering multiple instances of synthesized voice audio data for presentation to a representative associated with the entity and / or rendering multiple instances of text data for presentation to a representative of the entity. Further, in these implementations, as shown in block 456B, the system determines responsive content responsive to one or more of the instances of synthesized voice audio data and / or one or more of the instances of text data. The system can cause the chatbot to perform one or more tasks related to the entity on behalf of the user in the same or similar manner as described above with respect to FIG. 2 (e.g., in implementations in which the inference phase is performed remotely at a remote system and is described with respect to box 280B) and with respect to FIGS. 5A, 5C, and 6B. Notably, in these implementations, when participating in the corresponding conversation, the system can communicate directly with additional client devices of the entity's representative, in some examples, without interacting with the client device during the corresponding conversation.

[0066] At block 458, the system determines whether the chatbot successfully performed one or more tasks associated with the entity. The system can determine whether the chatbot successfully performed one or more tasks associated with the entity based on, for example, responsive content responsive to one or more of the instances of synthesized voice audio data and / or one or more of the instances of text data. Various non-limiting examples of determining whether the chatbot successfully performed one or more tasks related to an entity are described herein (e.g., with respect to FIGS. 5A, 5B, and 5C).

[0067] If, in the iteration of block 458, the system determines that the chatbot successfully performed one or more tasks associated with the entity, the system may proceed to block 460. The responsive content may be determined based on, for example, one or more responses provided by a representative of the entity during the corresponding conversation. At block 460, the system sends the responsive content to the user's client device. Sending the responsive content to the user's client device causes the responsive content to be provided for presentation to the user of the client device. The system proceeds to block 464. Block 464 is described in more detail below.

[0068] If, in the iteration of block 458, the system determines that the chatbot did not successfully perform one or more tasks related to the entity, the system may proceed to block 462. In other words, if the chatbot does not successfully perform one or more tasks related to the entity, the chatbot may prompt the user to join the corresponding conversation to ensure that the one or more tasks are still being performed. At block 462, the system sends a prompt to the user's client device for the user to join the corresponding conversation. Sending the prompt to the user's client device causes the prompt to be provided for presentation to the user of the client device. The system proceeds to block 464.

[0069] At block 464, the system determines whether to discard the chatbot. The system may determine whether to discard the chatbot based on, for example, whether the chatbot successfully performs one or more tasks associated with the entity, whether the chatbot is utilized in performing one or more task(s) associated with additional entities, whether a threshold duration has elapsed since the chatbot was created, and / or based on other conditions. Notably, in these implementations, because the chatbot may not be created and / or implemented locally on the user's client device, the system may not need to balance the performance of the chatbot and how the chatbot impacts the client device when determining whether to discard the chatbot.

[0070] If, in the iteration of block 464, the system determines not to discard the chatbot, the system may proceed to block 466. At block 466, the system continues to utilize the chatbot. For example, the system may continue to utilize the chatbot if one or more tasks associated with the entity were not successfully performed, but one or more tasks can be successfully performed with additional entities (as described with respect to block 364 of FIG. 3 ). If, in the iteration of block 464, the system determines to discard the chatbot, the system may proceed to block 468. At block 468, the system discards the chatbot. For example, the system may discard the chatbot in response to determining that one or more tasks associated with the entity were successfully performed, and optionally, in response to determining that there are no additional entities to participate in the corresponding conversation.

[0071] The system can perform other iterations of method 400 based on additional instructions from additional unstructured, free-form, natural language input received by the system. While FIG. 4 is described with respect to a system implemented remotely at a system remote from a user's client device, it should be understood that this is for illustrative purposes only and is not meant to be limiting. For example, as shown above with respect to FIG. 3, the system may be implemented at the user's client device that provided the unstructured, free-form, natural language input. As another example, the system may be implemented at both the client device and the remote system in a distributed manner. For example, a chatbot may be created locally at the client device but implemented at a remote system such that the client device's computing resources are not utilized in participating in a conversation corresponding to the chatbot. Also, for example, a chatbot may be created at a remote system but implemented at the client device such that the client device's computing resources are utilized only in participating in a conversation corresponding to the chatbot.

[0072] Additionally, while method 400 of FIG. 4 is also described with respect to the chatbot being a pre-trained LLM that has been fine-tuned based on unstructured, free-form natural language input (e.g., block 454), it should be understood that this is for purposes of illustration and not meant to be limiting. In additional or alternative embodiments, the pre-trained LLM may be utilized as the chatbot without fine-tuning. In these embodiments, when engaging the chatbot in a given corresponding conversation with a given additional user (e.g., block 456), the system may prime the pre-trained LLM based on the unstructured, free-form natural language input. This allows the chatbot to participate in the corresponding conversation without explicit fine-tuning during the training phase.

[0073] 5A and 5B, various non-limiting example interactions are shown between corresponding unstructured, free-form, natural language inputs that are utilized to generate corresponding chatbots, and the corresponding chatbots that perform corresponding task(s) based on the corresponding unstructured, free-form, natural language inputs. In particular, the interactions 500A and 500B described with respect to FIGS. 5A and 5B, respectively, may be implemented across multiple computing devices to cause the chatbots to perform the corresponding task(s) to be performed. For example, the corresponding unstructured free-form natural language input described with respect to the examples of Figures 5A and 5B may be received at a user's client device (e.g., client device 110 of Figure 1), the chatbot described with respect to the examples of Figures 5A and 5B may be generated at the user's client device (e.g., client device 110 of Figure 1) and / or at a remote system (e.g., remote system 160 of Figure 1), and the chatbot described with respect to the examples of Figures 5A and 5B may be implemented at the user's client device (e.g., client device 110 of Figure 1) and / or at a remote system (e.g., remote system 160 of Figure 1) and communicate with a representative of the entity via the representative's additional computing device. Each of these computing devices may include respective components, such as user interface input components (e.g., microphone(s), vision component(s), presence sensor(s), touch-sensitive display(s), keyboard(s), hardware button(s), software button(s), etc.), user interface output components (e.g., touch-sensitive display(s), speaker(s), monitor(s), projector(s), etc.), network interface(s), and / or other components. Thus, while interactions 500A and 500B in FIGS. 5A and 5B are each shown within a single interface, it should be understood that this is for purposes of illustrating the various techniques described herein and is not meant to be limiting.

[0074] 5A , assume that an on-device conversation with a user of a client device (e.g., Jane Doe) is initiated as shown by interaction 500A1. In various implementations, the on-device conversation with Jane Doe may be initiated as part of a dialogue between Jane Doe and an automated assistant executing at least partially on the client device. In these implementations, interaction 500A1 may be a voice-based interaction or a text-based interaction. For example, Jane Doe may invoke the automated assistant executing at least partially on the client device (e.g., by activating a software or hardware button, by speaking specific terms or phrases such as “assistant,” “hey assistant,” and the like, and / or by other means) to provide unstructured free-form natural language input 552A1 as spoken input. As another example, Jane Doe may access an automated assistant application accessible on the client device and associated with the automated assistant to provide unstructured free-form natural language input 552A1 as typed input.

[0075] For the example of FIG. 5A , assume that Jane Doe provides unstructured, free-form natural language input 552A1 as spoken input: “Please check with the restaurant where I had dinner yesterday to see if you found my red leather jacket.” In this example, the automated assistant can use ASR model(s) to process audio data capturing the spoken input to generate ASR output, such as recognized text corresponding to the spoken input (e.g., the recognized text “Please check with the restaurant where I had dinner yesterday to see if you found my red leather jacket”). Additionally, the automated assistant can use NLU model(s) to process the ASR output to generate NLU output, such as intent(s), slot value(s) for parameters associated with the intent(s), and / or other NLU output. In particular, the spoken input explicitly includes an intent of “Submit Query” (or the like) with a slot value of “Did You Find Jane Doe's Red Leather Jacket” (or the like) in the “Query Content” parameter (e.g., task data). However, spoken input only implicitly identifies entities (e.g., "the restaurant where I had dinner yesterday").

[0076] Thus, in this example, the automated assistant can utilize data from various sources to resolve entities implicit in the spoken input (e.g., Jane Doe's user data and / or other information). For example, the automated assistant can leverage Jane Doe's historical location information to identify the restaurant where Jane Doe had dinner yesterday, calendar information including reservations to identify the restaurant where she had dinner yesterday, software application information including reservations to identify the restaurant where she had dinner yesterday, email information including confirmation emails to identify the restaurant where she had dinner yesterday, and / or other information. As a result, further assume that "the restaurant where I had dinner yesterday" corresponds to the restaurant entity (e.g., entity data) of "Virtual Cafe." Thus, the automated assistant can provide response 554A1, "Okay, I'll call Virtual Cafe and let you know if you need anything or if they found it," for audio and / or visual presentation to Jane Doe to indicate that the automated assistant will perform the task of submitting a query to the "Virtual Cafe" entity.

[0077] In this example, based on the unstructured free-form natural language input 552A1, the automated assistant can cause the client device and / or remote system to generate a chatbot that performs the task of submitting a query to a representative of the “virtual cafe.” In some implementations, the chatbot may correspond to a pre-trained LLM that has been fine-tuned based on the unstructured free-form natural language input 552A1 using various fine-tuning techniques (e.g., as described with respect to FIG. 2 ). In other implementations, the chatbot may correspond to a pre-trained LLM that has not been fine-tuned based on the unstructured free-form natural language input 552A1, but that has been primed based on one or more of the multiple unstructured free-form natural language inputs. Further, assume that the chatbot participates in a corresponding conversation with a representative of the “virtual cafe” as part of a telephone call that performs the task of submitting a query to a representative of the “virtual cafe.” In this example, the automated assistant may determine a corresponding identifier (e.g., entity data) associated with the “virtual cafe,” such as a telephone number that can be utilized to initiate the corresponding conversation with the representative of the “virtual cafe.”

[0078] Thus, as shown in interaction 500A2, the automated assistant can have a chatbot implemented on a computing device, such as a client device in an embodiment where the chatbot is generated locally at the client device, or a remote system in an embodiment where the chatbot is generated remotely from the client device. This can facilitate the chatbot performing the task of calling a phone number associated with a "virtual cafe" to submit a query to a representative of the "virtual cafe." Assume further that after the call is placed, the chatbot and the representative of the "virtual cafe" engage in a corresponding conversation. For example, assume further that the representative of the "virtual cafe" answers the incoming call and provides spoken input 552A2 of, "Hello, this is John Smith from the virtual cafe. How can I help you?" In this example, the automated assistant may have a chatbot process an intent to "Submit a Query" (or the like) having a slot value of "Did you find [Jane Doe's] red leather jacket" for a "Query Content" parameter (e.g., task data), audio data capturing spoken input 552A2, text data corresponding to the audio data capturing spoken input 552A2 (e.g., determined using ASR model(s)), and / or any contextual conversation data to generate an instance of synthetic speech audio data. The instance of synthetic speech audio data may be audibly rendered at a client device of a representative of the "virtual cafe" and may capture synthetic speech 554A2 saying, "Hi, I'm a virtual assistant calling on behalf of Jane Doe to see if anyone found her red leather jacket yesterday."

[0079] In this example, when generating an instance of synthetic speech audio data, the automated assistant can apply this data as input across a pre-trained LLM that has been fine-tuned and / or primed based on unstructured, free-form natural language input 552A1 to generate an output, such as a probability distribution over a vocabulary of terms and / or phrases. Based on the probability distribution over the vocabulary of terms and / or phrases, the automated assistant can cause the chatbot to select text data corresponding to synthetic speech 554A2. Further, the automated assistant can cause the chatbot to process the text data corresponding to synthetic speech 554A2 using a TTS model(s) to generate an instance of synthetic speech audio data that is audibly rendered on the client device of the “virtual cafe” representative. Further, the automated assistant can cause the chatbot to generate an output and / or select text data corresponding to synthetic speech 554A2, at least in part, because the pre-trained LLM has been fine-tuned and / or primed based on the unstructured, free-form natural language input 552A1 provided by Jane Doe during interaction 500A1. Thus, the automated assistant can have the chatbot perform the task of submitting a query on Jane Doe's behalf to a representative of the "virtual cafe" entity.

[0080] Further assume that the “virtual cafe” representative responds to the synthesized speech 554A2 with spoken input 556A2 of, “Please wait while I check for lost property.” In this example, the automated assistant can have the chatbot process the audio data capturing the spoken input 556A2, text data corresponding to the audio data capturing the spoken input 556A2 (e.g., determined using ASR model(s)), and / or any contextual conversation data to generate an additional instance of synthesized speech audio data. The additional instance of synthesized speech audio data can be audibly rendered at the client device of the “virtual cafe” representative, and can capture synthesized speech 558A2 of, “Okay.” Further, the automated assistant can have the chatbot monitor for additional spoken input from the “virtual cafe” representative to indicate that the representative has rejoined the phone call after the representative placed the chatbot on hold.

[0081] Further assume that the "virtual cafe" representative returns from hold by providing spoken input 560A2 that states, "I found her red leather jacket. I'll leave it for her at the reception desk." In this example, the automated assistant can have the chatbot process the audio data capturing the spoken input 560A2, the text data corresponding to the audio data capturing the spoken input 560A2 (e.g., determined using ASR model(s)), and / or any contextual conversation data to generate a further instance of synthetic speech audio data. The further instance of synthetic speech audio data can be audibly rendered at the client device of the "virtual cafe" representative, and can capture synthetic speech 562A2 that states, "Thanks, I'll let her know." Further, the automated assistant can have the chatbot end the telephone call with the "virtual cafe" representative.

[0082] In this example, in response to determining that the task was successfully completed, the automated assistant may cause the chatbot to end the corresponding conversation with the representative of the “virtual cafe.” The automated assistant or chatbot may determine that the task was successfully completed based, for example, on the “virtual cafe” representative responding to a query by confirming that Jane Doe did indeed leave her red leather jacket at the virtual cafe last night and that it was available for pickup at the reception desk. Thus, based on the response to the query provided by the “virtual cafe” representative, the automated assistant may determine responsive content 552A3 that can be provided for presentation to Jane Doe in interaction 500A3. Response content 552A3 may include the results of the chatbot's performance of the task, such as, “Your red leather jacket is at the virtual cafe. They will leave it for you at the reception desk.” Thus, interaction 500A3 may be a notification generated for presentation to a user or provided for presentation to a user during a subsequent interaction session between Jane Doe and the automated assistant that is at least partially executing on her client device.

[0083] In particular, in interaction 500A2, the chatbot performs various peripheral actions. For example, the chatbot introduces itself as "a virtual assistant calling on behalf of Jane Doe" in synthesized voice 554A2 by utilizing a greet action that allows the chatbot to identify Jane Doe and identify itself as a chatbot; the chatbot places itself on hold in synthesized voice 558A2 by utilizing a hold action that allows the chatbot to pause and resume the corresponding conversation; and the voice ends the call by providing synthesized voice 562A2 and then using a leave action to end the corresponding conversation with the "virtual cafe" representative. These peripheral actions are non-limiting examples of how using a pre-trained LLM prior to fine-tuning and / or priming the chatbot allows the chatbot to perform generalized aspects of a conversation without the need for unstructured, free-form natural language input 552A1 to specify that the chatbot can perform these generalized aspects of a conversation. Additionally, it should be noted that other peripheral actions may be implemented by the chatbot, and that those described with respect to FIG. 5A are for illustrative purposes and are not meant to be limiting.

[0084] Additionally, in various implementations, a chatbot generated to perform the task of submitting a query to a representative of the "virtual cafe" may be destroyed. For example, the chatbot may be destroyed in response to determining that the task has been completed successfully. As another example, the chatbot may be destroyed in response to determining that the task has been completed successfully and therefore the chatbot will not be available to participate in any additional corresponding conversations. However, in various implementations, the chatbot may not always successfully perform the task during the corresponding conversation.

[0085] Referring specifically to FIG. 5B , assume that an on-device conversation with a user of a client device (e.g., Jane Doe) begins as shown by interaction 500B1 in the same or similar manner as described with respect to FIG. 5A . For the example of FIG. 5B , assume that Jane Doe provides unstructured, free-form, natural language input 552B1 as spoken input: "Tell Bobby Jones the locksmith that I transferred the money to his QuickCash account." In this example, the automated assistant can process the audio data capturing the spoken input using ASR model(s) to generate ASR output. Additionally, the automated assistant can process the ASR output using NLU model(s) to generate NLU output. In particular, the spoken input explicitly includes an intent to [notify an entity] (or the like) with a slot value of “Money transferred to [your] QuickCash account” (or the like) for the [notification content] parameter (e.g., task data), and the spoken input explicitly includes an [entity] (e.g., entity data) of [Bobby Jones the locksmith]. Thus, the automated assistant may provide response 554A2, “Okay, I'll notify Bobby Jones,” for auditory and / or visual presentation to Jane Doe to indicate that the automated assistant will perform the task of notifying the entity “Bobby Jones.”

[0086] In this example, based on unstructured free-form natural language input 552B1, the automated assistant can cause the client device and / or a remote system to generate a chatbot that performs the task of notifying a representative for "Bobby Jones" (e.g., "Bobby Jones" is an entity and "Bobby Jones" is its representative). In this example, the automated assistant can determine a corresponding identifier associated with "Bobby Jones," such as a phone number or contact entry that can be utilized to initiate a corresponding conversation with a representative for "Bobby Jones." Thus, as shown in interaction 500B2, the automated assistant can cause a chatbot to be implemented on a computing device in the same or similar manner as described above with respect to FIG. 5A. This allows the chatbot to call the phone number associated with "Bobby Jones" to facilitate the performance of the task of notifying "Bobby Jones."

[0087] Assume further that, after the call is placed, the chatbot and a representative of "Bobby Jones" are engaged in a corresponding conversation. For example, assume further that the representative of "Bobby Jones" answers the incoming call, the chatbot generates an instance of synthesized speech audio data in the same or similar manner as described with respect to FIG. 5A, and the synthesized speech audio data is audibly rendered at the client device of the representative of "Bobby Jones." In this example, assume further that the instance of synthesized speech audio data captures synthesized speech 552B2 saying, "Hi, I'm your virtual assistant calling on behalf of Jane Doe. I just wanted to let you know that Jane Doe transferred the money you owe to your QuickCash account."

[0088] Further assume that the representative for "Bobby Jones" responds to synthesized speech 552B2 with spoken input 554B2 that reads, "I don't know what you're talking about. I don't know who Jane Doe is." In this example, the automated assistant can have the chatbot process audio data capturing at least spoken input 554B2 to generate an additional instance of synthesized speech audio data. The additional instance of synthesized speech audio data can be audibly rendered at the client device of the representative for "Bobby Jones" and can capture synthesized speech 556B2 that reads, "I'll contact Jane." In other words, based on a failure to successfully perform a task (e.g., based on determining that "Bobby Jones" is not waiting for a transfer to his QuickCash account and / or based on determining that "Bobby Jones" does not know who Jane Doe is), the automated assistant can have the chatbot utilize a leave behavior that enables it to encourage Jane Doe to participate in the corresponding conversation.

[0089] Thus, as shown in interaction 500B3, the automated assistant can generate and provide prompt 552B3 to the chatbot to present to Jane Doe: "Hey Jane, I need to talk to Bobby Jones the locksmith about payment." The prompt may include a specific reason as to why the chatbot failed to successfully complete the task (e.g., "Bobby Jones doesn't know who you are"). As a result, as shown in interaction 500B4, Jane Doe may respond to prompt 500B3 and participate in the corresponding conversation, providing spoken input 552B1: "Hey Bobby, sorry for the confusion, this is Jane Doe. The payment I sent is on behalf of my father, Jane Doe, for the work you did for him last week." Thus, the automated assistant can have the chatbot participate in the corresponding conversation with a representative of "Bobby Jones," but if the chatbot does not successfully complete the task, it can prompt the chatbot to participate in the corresponding conversation with Jane Doe.

[0090] Notably, in the examples of Figures 5A and 5B, Jane Doe is not an active participant in the corresponding conversation between the chatbot and the representative for her initial interactions with the corresponding representative (e.g., interaction 500A2 in Figure 5A and interaction 500B2 in Figure 5B). However, the automated assistant can prompt Jane Doe to join the corresponding conversation as needed (e.g., as shown in the example of Figure 5B). Furthermore, in various embodiments, a transcript of the corresponding conversation may be provided for presentation to Jane Doe as the corresponding conversation progresses and / or after the corresponding conversation has ended. In these embodiments, the user may monitor the corresponding conversation and actively participate in one or more of the corresponding conversations without receiving a prompt.

[0091] While FIGS. 5A and 5B are described with respect to the corresponding conversation being a telephone call, it should be understood that this is for illustrative purposes only and is not meant to be limiting. For example, the corresponding conversation may be a text-based conversation conducted through any text-based platform or service that allows a chatbot to participate in a corresponding conversation with an entity or its representative (e.g., text or SMS messaging, email, and / or other text-based platform). Furthermore, while the corresponding representative in the examples of FIGS. 5A and 5B is a human, it should be understood that this is for illustrative purposes only and is not meant to be limiting. For example, the corresponding representative may be a corresponding additional chatbot deployed on behalf of the corresponding entity. In these examples, the chatbot may participate in a corresponding conversation with a corresponding additional chatbot. Furthermore, while the unstructured free-form natural language input in FIGS. 5A and 5B is described with respect to being a single sentence, it should be understood that this is for illustrative purposes only and is not meant to be limiting. In particular, the chatbot generated and implemented in FIGS. 5A and 5B may be different chatbots for different task(s) as the user provides different unstructured free-form natural language input.

[0092] 6A and 6B, additional non-limiting example interactions are shown between corresponding unstructured, free-form, natural language inputs utilized to generate corresponding chatbots and the corresponding chatbots performing corresponding task(s) based on the corresponding unstructured, free-form, natural language inputs. In particular, the interactions 600A and 600B described with respect to FIGS. 6A and 6B, respectively, may be implemented across multiple computing devices to cause the chatbots to perform the corresponding task(s) to be performed. For example, the corresponding unstructured free-form natural language input described with respect to the examples of Figures 6A and 6B may be received at a user's client device (e.g., client device 110 of Figure 1), the chatbot described with respect to the examples of Figures 6A and 6B may be generated at the user's client device (e.g., client device 110 of Figure 1) and / or at a remote system (e.g., remote system 160 of Figure 1), and the chatbot described with respect to the examples of Figures 6A and 6B may be implemented at the user's client device (e.g., client device 110 of Figure 1) and / or at a remote system (e.g., remote system 160 of Figure 1) and communicate with a representative of the entity via the representative's additional computing device. Each of these computing devices may include respective components, such as user interface input components (e.g., microphone(s), vision component(s), presence sensor(s), touch-sensitive display(s), keyboard(s), hardware button(s), software button(s), etc.), user interface output components (e.g., touch-sensitive display(s), speaker(s), monitor(s), projector(s), etc.), network interface(s), and / or other components. Thus, while interactions 600A and 600B in FIGS. 6A and 6B are each shown within a single interface, it should be understood that this is for purposes of illustrating the various techniques described herein and is not meant to be limiting.

[0093] 6A , assume that an on-device conversation with a client device user (e.g., Jane Doe) is initiated as shown by interaction 600A1 in the same or similar manner as described with respect to FIGS. 5A and 5B. However, in contrast to the example of FIGS. 5A and 5B, assume that Jane Doe provides multiple spoken inputs 652A1, 654A1, 656A1, 658A1, and 660A1, as shown in interaction 600A. Although the multiple spoken inputs 652A1, 654A1, 656A1, 658A1, and 660A1 convey details of a more complex task (e.g., booking catering for lunch) than those described with respect to FIGS. 5A and 5B (e.g., submitting a query in FIG. 5A and providing a notification in FIG. 5B), the automated assistant can further spawn a chatbot that books the catering for lunch and engage the chatbot in a corresponding conversation with a representative of a “virtual cafe.”

[0094] In this example, the task may be considered more complex than the task from either of Figures 5A and 5B based on the task of Figure 6A, which includes multiple subtasks. These subtasks may include, for example, determining whether the "Virtual Cafe" is available for lunch catering for a specified number of people at a specified date / time (based on spoken input 652A1 and spoken input 654A1), determining whether the "Virtual Cafe" is available for lunch catering with specified dietary restrictions or menu requirements (based on spoken input 654A1), determining whether the "Virtual Cafe" is available for lunch catering at a specified price (based on spoken input 658A1), and proactively paying for lunch if the "Virtual Cafe" is available for lunch catering at the specified price (based on spoken input 658A1). Notably, each of these subtasks includes a different intent and different slot values ​​for parameters associated with the different intents. As a result, in the example of Figure 6A, the chatbot may be fine-tuned and / or primed with more task data than the chatbots of Figures 5A and 5B because the task in Figure 6A is more complex. Thus, the automated assistant may provide response 662A1 of "Okay, I'll call the virtual cafe and let you know the rest" in response to an auditory and / or visual presentation to Jane Doe to indicate that the automated assistant will perform the task of making a lunch reservation with an entity at "Virtual Cafe."

[0095] With specific reference to FIG. 6B , as shown in interaction 600B1, the automated assistant can have a chatbot implemented on a computing device in the same or similar manner as described with respect to FIGS. 5A and 5B . Further assume that after placing the phone call, the chatbot and a representative from the “virtual cafe” engage in a corresponding conversation. For example, further assume that the representative from the “virtual cafe” answers the incoming call and provides spoken input 652B1: “Hello, this is John Smith from the virtual cafe. How can I help you?” In this example, the automated assistant can process the task data determined based on interaction 600A1, audio data capturing the spoken input 652B1, text data corresponding to the audio data capturing the spoken input 652B1 (e.g., determined using ASR model(s)), and / or any contextual conversation data to generate an instance of synthesized speech audio data in the same or similar manner as described with respect to FIGS. 5A and 5B . An instance of synthesized speech audio data may be audibly rendered at a client device of a representative of the "virtual cafe" to capture synthesized speech 654B1 saying, "Hi, I'm a virtual assistant calling on behalf of Jane Doe to see if catering is available for her lunch on December 12, 2022 at noon."

[0096] Further assume that the “virtual cafe” representative responds to the synthesized speech 654B1 with spoken input 656B1 of, “Okay, tell me a bit more about lunch.” In this example, the automated assistant can have the chatbot process the task data, the audio data capturing the spoken input 656B1, the text data corresponding to the audio data capturing the spoken input 656B1 (e.g., determined using ASR model(s)), and / or any contextual conversation data to generate an additional instance of synthesized speech audio data. The additional instance of synthesized speech audio data can be audibly rendered at the client device of the “virtual cafe” representative, and can capture synthesized speech 658B1 saying, “There are 50 people, some are gluten intolerant, and others…” while the chatbot interrupts with the “virtual cafe” representative providing spoken input 660B1 of, “Excuse me, did you say 50 or 15?” In this example, the automated assistant can have the chatbot repeat "50 people" in synthesized voice 662B1 by utilizing a clarification operation that allows the chatbot to clarify and / or repeat information previously provided during the corresponding conversation.

[0097] Further assume that the "Virtual Cafe" representative responds to the synthesized speech 662B1 with spoken input 664B1 of "50 people, please continue." In this example, the automated assistant can resume the corresponding conversation with the chatbot by generating a further instance of synthesized speech that includes information not provided to the "Virtual Cafe" representative. For example, a further instance of synthesized speech audio data can be audibly rendered on the "Virtual Cafe" representative's client device, capturing synthesized speech 666B1 that states, "Some of our people are gluten intolerant, others are vegan, and Jane would like a sandwich platter with several different options, a salad bar, and other side options." Further assume that the "Virtual Cafe" representative responds to the synthesized speech 666B1 with spoken input 668B1 of "Okay, we can cater for $225." As a result, the automated assistant can have the chatbot complete the task of booking lunch based on successfully completing all of the task's other subtasks by generating a further instance of synthesized speech audio data. Further additional instances of the synthesized speech audio data may be audibly rendered at the client device of the representative of the "virtual cafe" such as "Great, here's Jane Doe's credit card information [ * Provide your credit card information * ], please forward the details to janedoe@exampleurl.com."

[0098] In this example, in response to determining that the task (or all of the subtasks) were successfully completed, the automated assistant may cause the chatbot to terminate the corresponding conversation with the representative of the “Virtual Cafe.” The automated assistant or chatbot may determine that the task was successfully completed based, for example, on confirmation of lunch catering by the representative of the “Virtual Cafe.” Thus, based on the successful booking of lunch with the representative of the “Virtual Cafe,” the automated assistant may determine responsive content 652B2 that may be provided for presentation to Jane Doe in interaction 600B2. Response content 652B2 may include the results of the chatbot's performance of the task, such as “Virtual Cafe has scheduled lunch catering. John Smith will forward the details to your email.” Thus, interaction 600B2 may be a summary of interaction 600B1 that is generated for presentation to a user or provided for presentation to a user during a subsequent interaction session between Jane Doe and the automated assistant that is at least partially executing on her client device.

[0099] 7, there is shown a block diagram of an exemplary computing device 710 that may optionally be utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client device, remote system component(s), and / or other component(s) may include one or more components of the exemplary computing device 710.

[0100] Computing device 710 typically includes at least one processor 714 that communicates with several peripheral devices via a bus subsystem 712. These peripheral devices may include, for example, a storage subsystem 724, including a memory subsystem 725 and a file storage subsystem 726, a user interface output device 720, a user interface input device 722, and a network interface subsystem 716. The input and output devices enable user interaction with computing device 710. The network interface subsystem 716 provides an interface to external networks and is connected to corresponding interface devices in other computing devices.

[0101] The user interface input devices 722 may include pointing devices such as a keyboard, a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touchscreen integrated into a display (e.g., a touch-sensitive display), an audio input device such as a voice recognition system or a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 710 or a communications network.

[0102] The user interface output devices 720 may include a display subsystem, a printer, a facsimile, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include any type of device or method for outputting information from the computing device 710 to a user or other machine or computing device.

[0103] Storage subsystem 724 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 724 may include logic to perform selected aspects of the methods disclosed herein, as well as logic to implement the various components shown in Figures 1 and 2.

[0104] These software modules typically execute on the processor 714 alone or in combination with other processors. The memory 725 used by the storage subsystem 724 can include multiple memories, such as a main random access memory (RAM) 730 for storing instructions and data during program execution and a read-only memory (ROM) 732 in which fixed instructions are stored. The file storage subsystem 726 can provide persistent storage of program files and data files and can include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of an embodiment may be stored by the file storage subsystem 726 in the storage subsystem 724 or on another machine accessible by the processor(s) 714.

[0105] Bus subsystem 712 provides a mechanism that allows the various components and subsystems of computing device 710 to communicate with each other as intended. Although bus subsystem 712 is shown schematically as a single bus, alternative implementations of bus subsystem 712 may use multiple buses.

[0106] Computing device 710 can be of various types, such as a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 710 shown in Figure 7 is intended only as a specific example to illustrate some implementations. Many other configurations of computing device 710 can have more or fewer components than the computing device shown in Figure 7.

[0107] In situations where the systems described herein may collect or monitor personal information about users or utilize personal and / or monitored information, users may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social behavior or activities, occupation, user preferences, or the user's current geographic location) or whether and / or how content that may be more relevant to the user is received from a content server. Also, certain data may be processed in one or more ways to remove personally identifiable information before it is stored or used. For example, a user's identity may be processed so that information that can personally identify the user cannot be determined, or if geographic location information (such as to the city, zip code, or state level) is obtained, the user's geographic location may be generalized so that the user's specific geographic location cannot be determined. Thus, users may control how information about them is collected and / or how the information is used.

[0108] In some implementations, a method is provided that is implemented by one or more processors of a client device, the method including: receiving, at the client device, unstructured free-form natural language input from a user of the client device; and, in response to receiving the unstructured free-form natural language input including one or more tasks related to an entity, generating locally at the client device a chatbot based at least on the unstructured free-form natural language input to perform one or more tasks related to the entity on behalf of the user. The unstructured free-form natural language input includes one or more tasks related to the entity. The method further includes causing the chatbot to perform the one or more tasks related to the entity on behalf of the user. Having the chatbot perform the one or more tasks related to the entity on behalf of the user includes causing the chatbot to participate in a corresponding conversation with the entity; and, during the corresponding conversation with the entity, causing the chatbot to render multiple instances of synthesized speech audio data for presentation to a representative of the entity; and receiving responsive content responsive to at least a given instance of the synthesized speech audio data. At least a given instance of the synthesized speech audio data of the multiple instances of synthesized speech audio data conveys details of the one or more tasks related to the entity. The method further includes causing the responsive content to be provided for presentation to a user of the client device.

[0109] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0110] In some implementations, generating a chatbot to perform one or more tasks related to an entity on behalf of a user may include obtaining a pre-trained large-scale language model (LLM), fine-tuning the pre-trained LLM based on unstructured free-form natural language input to generate a fine-tuned LLM, and utilizing the fine-tuned LLM as the chatbot.

[0111] In some versions of those embodiments, causing the chatbot to render a given instance of synthetic speech audio data for presentation to the representative may include processing one or more features of the unstructured free-form natural language input using the fine-tuned LLM to generate a given instance of text data conveying details of one or more tasks associated with the entity, processing the given instance of text data conveying details of one or more tasks associated with the entity using a text-to-speech (TTS) model to generate a given instance of synthetic speech audio data, and transmitting the given instance of synthetic speech audio data from the client device to an additional client device of the representative. Sending the given instance of synthetic speech audio data to the additional client device causes the additional client device to audibly render the given instance of synthetic speech audio data for presentation to the representative via one or more speakers of the additional client device.

[0112] In some further versions of those embodiments, the method may further include processing, using the fine-tuned LLM, a corresponding context of a corresponding conversation along with one or more of the features of the unstructured free-form natural language input to generate a given instance of text data that conveys details of one or more tasks associated with the entity.

[0113] In additional or alternative further versions of those embodiments, the method may further include, in response to the given instance of the synthesized voice audio data being audibly rendered for presentation to the representative via one or more speakers of the additional client device, receiving, at the client device, from the additional client device, a given instance of response audio data including responsive content responsive to at least the given instance of the synthesized voice audio data; processing the given instance of the response audio data using an automatic speech recognition (ASR) model to generate a given instance of response text data; and determining, based on the given instance of the response text data, whether performance of one or more of the tasks during the corresponding conversation was successfully completed.

[0114] In some yet further versions of those embodiments, causing the responsive content to be provided for presentation to a user of the client device may be in response to determining that one or more of the tasks have been successfully completed during the corresponding conversation.

[0115] In additional or alternative further versions of those embodiments, the method may further include, in response to determining that one or more of the tasks are not successfully completed in the corresponding conversation, causing the chatbot to render an indication that the user is prompted to join the corresponding conversation for presentation to a representative of the entity, generating a prompt requesting the user to join the corresponding conversation, and causing the prompt requesting the user to join the corresponding conversation to be provided for presentation to the user at the client device.

[0116] In further versions of these embodiments, the prompt may further include specific reasons as to why one or more of the tasks are not successfully completed during the corresponding conversation.

[0117] In additional or alternative further versions of those embodiments, the method may further include extracting one or more of the features from the unstructured free-form natural language input before processing the one or more features of the unstructured free-form natural language input using the fine-tuned LLM.

[0118] In some further versions of those embodiments, one or more of the features may be explicitly included in the unstructured free-form natural language input, and extracting one or more of the features from the unstructured free-form natural language input that are explicitly included in the unstructured free-form natural language input may include utilizing an input parser to extract one or more of the features that are explicitly included in the unstructured free-form natural language input.

[0119] In further additional or alternative versions of these embodiments, one or more of the features may be implicitly included in the unstructured free-form natural language input, and extracting one or more of the features from the unstructured free-form natural language input that are implicitly included in the unstructured free-form natural language input may include utilizing an input parser to identify one or more of the features that are implicitly included in the unstructured free-form natural language input, and utilizing a coreference resolver to extract one or more of the features that are implicitly included in the unstructured free-form natural language input.

[0120] In further versions of these embodiments, the coreference resolver may also access user data generated locally on the client device to extract one or more of the features implicit in the unstructured free-form natural language input, where the user data may include one or more of historical location data, historical time data, user preference data, user account data, calendar information, or email data.

[0121] In additional or alternative versions of these embodiments, the pre-trained LLM may be stored in on-device storage of the client device, and the pre-trained LLM that may be stored in on-device storage of the client device may be a sparsified version of a global pre-trained LLM available at a remote system communicatively coupled to the client device.

[0122] In some further versions of these embodiments, the fine-tuned LLM may be stored in on-device storage of the client device.

[0123] In yet another version of those embodiments, the method may further include discarding the fine-tuned LLM from on-device storage of the client device after causing the chatbot to perform one or more tasks related to the entity on behalf of the user, and refraining from discarding the pre-trained LLM from on-device storage of the client device.

[0124] In some implementations, the method may further include receiving, at the client device, additional unstructured free-form natural language input from a user of the client device, and in response to receiving the additional unstructured free-form natural language input including one or more additional tasks related to the entity or the additional entity, generating locally at the client device an additional chatbot based at least on the additional natural language input to perform one or more additional tasks related to the entity or the additional entity on behalf of the user. The additional unstructured free-form natural language input may include one or more additional tasks related to the entity or the additional entity. The method may further include causing the additional chatbot to perform the one or more additional tasks related to the entity or the additional entity on behalf of the user. Having the additional chatbot perform one or more additional tasks related to the entity or the additional entity on behalf of the user may include: causing the additional chatbot to participate in an additional corresponding conversation with the entity or the additional entity; during the additional corresponding conversation with the entity or the additional entity, causing the chatbot to render multiple additional instances of synthesized voice audio data for presentation to a representative of the entity or an additional representative of the additional entity; receiving additional responsive content responsive to at least the given additional instance of the synthesized voice audio data; and causing the additional responsive content to be provided for presentation to the user of the client device, wherein at least the given additional instance of the synthesized voice audio data of the multiple additional instances of synthesized voice audio data conveys additional details of the one or more additional tasks related to the entity or the additional entity.

[0125] In some implementations, the method may further include identifying entities associated with the one or more tasks based on the unstructured free-form natural language input, determining corresponding identifiers of the entities associated with the one or more tasks, and utilizing the corresponding identifiers of the entities associated with the one or more tasks to engage the chatbot in a corresponding conversation with the entities.

[0126] In some versions of those embodiments, the corresponding identifier of the entity associated with the one or more tasks may be a corresponding telephone number of the entity, and utilizing the corresponding identifier of the entity associated with the one or more tasks to have the chatbot engage in a corresponding conversation with the entity may include having the chatbot utilize the corresponding telephone number of the entity to initiate an automated telephone call on behalf of the user to perform one or more tasks related to the entity.

[0127] In some embodiments, having the chatbot participate in a corresponding conversation with the entity may include having the chatbot respond to a telephone call received at the client device and from a representative of the entity, and having the chatbot participate in the corresponding conversation with the entity as part of the telephone call.

[0128] In some implementations, the user may not actively participate in the corresponding conversation between the chatbot and the representative.

[0129] In some implementations, during the corresponding conversation with the entity, the method may further include receiving a request from the representative for the user to join the corresponding conversation; and in response to receiving the request for the user to join the corresponding conversation, generating a prompt requesting the user to join the corresponding conversation; and causing the prompt requesting the user to join the corresponding conversation to be provided for presentation to the user at the client device.

[0130] In some implementations, entities may be explicitly identified in unstructured free-form natural language input.

[0131] In some implementations, an entity may not be explicitly identified in the unstructured free-form natural language input, an entity may be a particular type of entity, and a particular type of entity may be explicitly identified in the unstructured free-form natural language input, and one or more tasks may be associated with a particular type of entity.

[0132] In some versions of those embodiments, the one or more tasks may also be associated with additional entities, in addition to the entity, that are entities of a particular type. Having the chatbot perform one or more tasks related to the entity on behalf of the user may further include: engaging the chatbot in an additional corresponding conversation with the additional entity; causing the chatbot to render multiple additional instances of additional synthesized voice audio data for presentation to additional representatives of the additional entity during the additional corresponding conversation with the additional entity; and receiving additional responsive content responsive to at least the given additional instance of the synthesized voice audio data. At least the given additional instance of the synthesized voice audio data of the multiple additional instances of synthesized voice audio data may convey details of one or more tasks also related to the additional entity. The method may further include causing the additional responsive content to be provided for presentation to the user of the client device.

[0133] In some further versions of these implementations, the chatbot can have a corresponding conversation with the entity and additional corresponding conversations with additional entities in parallel.

[0134] In additional or alternative further versions of these embodiments, the chatbot may conduct additional corresponding conversations with additional entities after the corresponding conversation with the entity and in response to determining that one or more of the tasks were not successfully completed during the corresponding conversation.

[0135] In some implementations, the entity's representative may be a human representative, while in other implementations, the entity's representative may be an additional chatbot trained to conduct corresponding conversations on behalf of the entity.

[0136] In some implementations, the unstructured free-form natural language input may be typed or spoken input that conveys details of one or more tasks associated with the entity and may not define a corresponding dialogue state map, a dialogue state of the corresponding dialogue state map, or a dialogue state transition of the corresponding dialogue state map that is utilized to perform one or more tasks associated with the entity.

[0137] In some versions of those implementations, the unstructured free-form natural language input may be typed input that conveys details of one or more tasks associated with the entity in one or more sentences, and the responsive content provided for presentation to a user of the client device may be the results of performing the one or more tasks associated with the entity.

[0138] In additional or alternative versions of these embodiments, the unstructured free-form natural language input may be typed input conveying details of one or more tasks related to the entity in one or more paragraphs, and the responsive content provided for presentation to the user of the client device may be a summary of a corresponding conversation with the entity.

[0139] In some embodiments, a method implemented by one or more processors of a remote system is provided, including receiving, at the remote system and from a client device, an indication of unstructured free-form natural language input from a user of the client device; and, in response to receiving the indication of the unstructured free-form natural language input including one or more tasks related to the entity, remotely generating, at the remote system, a chatbot associated with the entity and performing the one or more tasks on behalf of the user based at least on the indication of the natural language input. The unstructured free-form natural language input includes one or more tasks related to the entity. The method further includes causing the chatbot to perform the one or more tasks related to the entity on behalf of the user. Having the chatbot perform the one or more tasks related to the entity on behalf of the user includes causing the chatbot to participate in a corresponding conversation with the entity; and, during the corresponding conversation with the entity, causing the chatbot to render multiple instances of synthesized speech audio data for presentation to a representative of the entity; and receiving responsive content responsive to at least a given instance of the synthesized speech audio data. At least a given instance of the synthesized speech audio data of the multiple instances of synthesized speech audio data conveys details of the one or more tasks related to the entity. The method further includes transmitting an indication of the responsive content from the remote system to the client device. Sending the indication of the responsive content to the client device causes the client device to provide the responsive content for presentation to a user of the client device.

[0140] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0141] In some implementations, generating a chatbot to perform one or more tasks related to an entity on behalf of a user may include obtaining a pre-trained large-scale language model (LLM), fine-tuning the pre-trained LLM based on unstructured free-form natural language input to generate a fine-tuned LLM, and utilizing the fine-tuned LLM as the chatbot.

[0142] In some versions of these embodiments, the pre-trained LLM may be stored in remote storage on a remote system, and the pre-trained LLM that may be stored in remote storage on a remote system may be an unsparsed version of a global pre-trained LLM available on a remote system communicatively coupled to the client device.

[0143] In some further versions of these embodiments, the fine-tuned LLM may be stored in remote storage on a remote system.

[0144] In yet another version of those embodiments, the method may further include discarding the fine-tuned LLM from remote storage at the remote system after causing the chatbot to perform one or more tasks related to the entity on behalf of the user, and refraining from discarding the pre-trained LLM from remote storage at the remote system.

[0145] In some embodiments, a method implemented by one or more processors of a client device is provided, comprising: receiving, at the client device, unstructured free-form natural language input from a user of the client device; and, in response to receiving an indication of the unstructured free-form natural language input including one or more tasks related to the entity, remotely generating, at a remote system, a chatbot associated with the entity and configured to perform the one or more tasks on behalf of the user based at least on the indication of the natural language input. The unstructured free-form natural language input includes one or more tasks related to the entity. The method further includes causing the chatbot to perform the one or more tasks related to the entity on behalf of the user. Having the chatbot perform the one or more tasks related to the entity on behalf of the user includes causing the chatbot to participate in a corresponding conversation with the entity; and, during the corresponding conversation with the entity, causing the chatbot to render multiple instances of text data for presentation to a representative of the entity; and receiving responsive content responsive to at least the given instance of the text data. At least the given instance of the text data of the multiple instances of text data conveys details of the one or more tasks related to the entity. The method further includes causing the responsive content to be provided for presentation to the user of the client device.

[0146] These and other implementations of the technology disclosed herein may optionally include one or more of the following features.

[0147] In some implementations, the method may further include identifying entities associated with the one or more tasks based on the unstructured free-form natural language input, determining corresponding identifiers of the entities associated with the one or more tasks, and utilizing the corresponding identifiers of the entities associated with the one or more tasks to engage the chatbot in a corresponding conversation with the entities.

[0148] In versions of these embodiments, the corresponding identifier of the entity associated with the one or more tasks may be one or more of a corresponding phone number of the entity, a corresponding email address of the entity, or a corresponding username of the entity. Utilizing the corresponding identifier of the entity associated with the one or more tasks to have the chatbot participate in the corresponding conversation with the one or more entities may include having the chatbot utilize the corresponding identifier of the entity to initiate an automated text-based messaging session on behalf of the user to perform one or more tasks related to the entity.

[0149] In some implementations, engaging the chatbot in a corresponding conversation with the entity may include having the chatbot respond to a text message, email, or other text-based message received at the client device from a representative of the entity, and having the chatbot engage in a corresponding conversation with the entity to further the text message, email, or other text-based message.

[0150] In some embodiments, a method implemented by one or more processors of a remote system is provided, including receiving, at the remote system and from a client device, an indication of unstructured free-form natural language input from a user of the client device; and, in response to receiving the indication of the unstructured free-form natural language input including one or more tasks related to the entity, remotely generating, at the remote system, a chatbot associated with the entity and performing the one or more tasks on behalf of the user based at least on the indication of the natural language input. The unstructured free-form natural language input includes one or more tasks related to the entity. The method further includes causing the chatbot to perform the one or more tasks related to the entity on behalf of the user. Having the chatbot perform the one or more tasks related to the entity on behalf of the user includes causing the chatbot to participate in a corresponding conversation with the entity; and, during the corresponding conversation with the entity, causing the chatbot to render multiple instances of synthesized speech audio data for presentation to a representative of the entity; and receiving responsive content responsive to at least a given instance of the synthesized speech audio data. At least a given instance of the synthesized speech audio data of the multiple instances of synthesized speech audio data conveys details of the one or more tasks related to the entity. The method further includes transmitting an indication of the responsive content from the remote system to the client device. Sending the indication of the responsive content to the client device causes the client device to provide the responsive content for presentation to a user of the client device.

[0151] In some implementations, a method is provided that is implemented by one or more processors of a client device, the method including: receiving, at the client device, unstructured, free-form natural language input from a user of the client device, the unstructured, free-form natural language input including one or more tasks associated with an entity; identifying a chatbot to perform one or more tasks related to the entity on behalf of the user in response to receiving the unstructured, free-form natural language input including a natural language description of a corresponding dialogue state map; and causing the chatbot to perform the one or more tasks related to the entity on behalf of the user. Having the chatbot perform the one or more tasks related to the entity on behalf of the user includes causing the chatbot to participate in a corresponding conversation with the entity; and during the corresponding conversation with the entity, causing the chatbot to render multiple instances of synthesized speech audio data for presentation to a representative of the entity, wherein at least a given instance of the synthesized speech audio data among the multiple instances of synthesized speech audio data conveys details of the one or more tasks related to the entity; and receiving responsive content responsive to the at least a given instance of the synthesized speech audio data. The method further includes causing the responsive content to be provided for presentation to the user of the client device.

[0152] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0153] In some implementations, identifying a chatbot to perform one or more tasks related to the entity on behalf of the user may include obtaining a pre-trained large-scale language model (LLM) and utilizing the pre-trained LLM as the chatbot.

[0154] In some versions of those embodiments, the pre-trained LLM may be stored in on-device storage of the client device, and the pre-trained LLM that may be stored in on-device storage of the client device may be a sparsified version of a global pre-trained LLM available at a remote system communicatively coupled to the client device.

[0155] In some versions of those embodiments, the methods may further include refraining from allowing the pre-trained LLM to be fine-tuned based on unstructured, free-form natural language input.

[0156] In some versions of these embodiments, causing the chatbot to render a given instance of synthetic speech among the multiple instances of synthetic speech for presentation to a representative of the entity may include processing the free-form natural language input and task data reflecting details of one or more tasks associated with the entity using a pre-trained LLM to generate an instance of text data that reflects a given behavior of a given implicit dialog state, processing the given instance of text data that reflects a given behavior of a given implicit dialog state using a text-to-speech (TTS) model to generate the given instance of synthetic speech, and transmitting the given instance of synthetic speech from the client device to an additional client device of the representative of the entity. Sending the given instance of synthetic speech to the additional client device causes the additional client device to audibly render the given instance of synthetic speech for presentation to the representative of the entity via one or more speakers of the additional client device.

[0157] Further, some embodiments include one or more processors (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s)), and / or tensor processing unit(s) (TPU(s)) of one or more computing devices, the one or more processors operable to execute instructions stored in associated memory, the instructions configured to cause any of the aforementioned methods to be performed. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions executable by the one or more processors to perform any of the aforementioned methods. Some embodiments also include a computer program product including instructions executable by the one or more processors to perform any of the aforementioned methods.

[0158] It will be understood that all combinations of the above concepts and additional concepts described in more detail herein are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.

Claims

1. 1. A method implemented by one or more processors of a client device, comprising: receiving, at the client device, unstructured free-form natural language input from a user of the client device, the unstructured free-form natural language input including one or more tasks associated with an entity; in response to receiving the unstructured free-form natural language input including the one or more tasks associated with the entity; generating a chatbot based at least on the unstructured free-form natural language input and locally on the client device to perform the one or more tasks related to the entity on behalf of the user; causing the chatbot to perform the one or more tasks related to the entity on behalf of the user, wherein causing the chatbot to perform the one or more tasks related to the entity on behalf of the user includes: engaging the chatbot in a corresponding conversation with the entity; During the corresponding conversation with the entity, causing the chatbot to render a plurality of instances of synthesized speech audio data for presentation to a representative of the entity, at least a given instance of synthesized speech audio data among the plurality of instances of synthesized speech audio data conveying details of the one or more tasks associated with the entity; receiving responsive content responsive to at least the given instance of synthesized voice audio data; causing the responsive content to be provided for presentation to the user of the client device; A method comprising:

2. generating the chatbot to perform the one or more tasks related to the entity on behalf of the user, Obtaining a pre-trained large-scale language model (LLM); fine-tuning the pre-trained LLM based on the unstructured free-form natural language input to generate a fine-tuned LLM; Utilizing the fine-tuned LLM as the chatbot; The method of claim 1 , comprising:

3. causing the chatbot to render the given instance of synthesized speech audio data for presentation to the representative includes: using the fine-tuned LLM to process one or more features of the unstructured free-form natural language input to generate a given instance of text data conveying the details of the one or more tasks associated with the entity; processing the given instance of text data conveying the details of the one or more tasks associated with the entity to generate the given instance of synthetic speech audio data using a text-to-speech (TTS) model; transmitting, from the client device, the given instance of synthesized speech audio data to an additional client device of the representative, wherein transmitting the given instance of synthesized speech audio data to the additional client device causes the additional client device to audibly render the given instance of synthesized speech audio data for presentation to the representative via one or more speakers of the additional client device; The method of claim 2 , comprising:

4. using the fine-tuned LLM to process one or more of the features of the unstructured free-form natural language input along with the corresponding context of the corresponding conversation to generate a given instance of text data conveying the details of the one or more tasks associated with the entity. The method of claim 3 further comprising:

5. in response to the given instance of synthesized speech audio data being audibly rendered for presentation to the representative via the one or more speakers of the additional client device; receiving, at the client device, from the additional client device, a given instance of responsive audio data including the responsive content responsive to at least the given instance of synthesized speech audio data; processing the given instance of response audio data using an automatic speech recognition (ASR) model to generate the given instance of response text data; determining whether performance of one or more of the tasks was successfully completed during the corresponding conversation based on the given instance of response text data; The method of claim 3 further comprising:

6. 6. The method of claim 5, wherein causing the responsive content to be provided for presentation to the user of the client device is in response to determining that one or more of the tasks are successfully completed during the corresponding conversation.

7. in response to determining that one or more of the tasks are not successfully completed during the corresponding conversation; causing the chatbot to render an indication that the user is prompted to participate in the corresponding conversation for presentation to the representative of the entity; generating a prompt requesting the user to join the corresponding conversation; causing the prompt requesting the user to join the corresponding conversation to be provided for presentation to the user at the client device; The method of claim 5 further comprising:

8. The method of claim 7 , wherein the prompt further includes a specific reason as to why one or more of the tasks are not successfully completed during the corresponding conversation.

9. prior to using the fine-tuned LLM to process one or more of the features of the unstructured free-form natural language input; extracting one or more of said features from said unstructured free-form natural language input. The method of claim 3 further comprising:

10. one or more of the features are explicitly included in the unstructured free-form natural language input, and extracting from the unstructured free-form natural language input one or more of the features that are explicitly included in the unstructured free-form natural language input comprises: utilizing an input parser to extract one or more of the features explicitly included in the unstructured free-form natural language input.

10. The method of claim 9, comprising:

11. One or more of the features are implicitly included in the unstructured free-form natural language input, and extracting from the unstructured free-form natural language input one or more of the features implicitly included in the unstructured free-form natural language input comprises: utilizing an input parser to identify one or more of the features implicit in the unstructured free-form natural language input; utilizing a coreference resolver to extract one or more of the features implicit in the unstructured free-form natural language input; 10. The method of claim 9, comprising:

12. 12. The method of claim 11, wherein the coreference resolver accesses user data generated locally on the client device to extract one or more of the features implicit in the unstructured free-form natural language input, the user data including one or more of historical location data, historical time data, user preference data, user account data, calendar information, or email data.

13. 3. The method of claim 2, wherein the pre-trained LLM is stored in on-device storage of the client device, and the pre-trained LLM stored in the on-device storage of the client device is a sparsified version of a global pre-trained LLM available at a remote system communicatively coupled to the client device.

14. The method of claim 13 , wherein the fine-tuned LLM is stored in the on-device storage of the client device.

15. after causing the chatbot to perform the one or more tasks related to the entity on behalf of the user; discarding the fine-tuned LLM from the on-device storage of the client device; and refraining from discarding the pre-trained LLM from the on-device storage of the client device; and 15. The method of claim 14, further comprising:

16. receiving, at the client device, additional unstructured free-form natural language input from the user of the client device, the additional unstructured free-form natural language input including one or more additional tasks related to the entity or additional entities; in response to receiving the additional unstructured free-form natural language input including the one or more additional tasks associated with the entity or the additional entity; generating, based on at least the additional natural language input and locally on the client device, an additional chatbot that performs the one or more additional tasks related to the entity or the additional entity on behalf of the user; and causing the additional chatbot to perform the one or more additional tasks related to the entity or the additional entity on behalf of the user, wherein causing the additional chatbot to perform the one or more additional tasks related to the entity or the additional entity on behalf of the user includes: causing the additional chatbot to engage in an additional corresponding conversation with the entity or the additional entity; During said additional corresponding conversation with said entity or said additional entity, causing the chatbot to render a plurality of additional instances of synthesized speech audio data for presentation to the representative of the entity or additional representatives of the additional entities, wherein at least a given additional instance of synthesized speech audio data from the plurality of additional instances of synthesized speech audio data conveys additional details of the one or more additional tasks associated with the entity or the additional entities; receiving additional responsive content responsive to at least the given additional instance of synthesized voice audio data; causing the additional responsive content to be provided for presentation to the user of the client device; The method according to any one of claims 1 to 15, comprising:

17. identifying the entities associated with the one or more tasks based on the unstructured free-form natural language input; determining corresponding identifiers of the entities associated with the one or more tasks; utilizing the corresponding identifier of the entity associated with the one or more tasks to have the chatbot participate in the corresponding conversation with the entity; The method of any one of claims 1 to 16, further comprising:

18. the corresponding identifier of the entity associated with the one or more tasks is a corresponding phone number of the entity, and utilizing the corresponding identifier of the entity associated with the one or more tasks to engage the chatbot in the corresponding conversation with the entity, having the chatbot utilize the corresponding telephone number of the entity to initiate an automated telephone call on behalf of the user to perform one or more tasks related to the entity.

18. The method of claim 17, comprising:

19. Engaging the chatbot in the corresponding conversation with the entity includes: causing the chatbot to answer a telephone call received at the client device from the representative of the entity; engaging the chatbot in the corresponding conversation with the entity as part of the telephone call; The method of any one of claims 1 to 18, comprising:

20. The method of any one of claims 1 to 19, wherein the user does not actively participate in the corresponding conversation between the chatbot and the representative.

21. During the corresponding conversation with the entity, receiving a request from the representative for the user to join the corresponding conversation; In response to receiving the request for the user to join the corresponding conversation, generating a prompt requesting the user to join the corresponding conversation; causing the prompt requesting the user to join the corresponding conversation to be provided for presentation to the user at the client device; The method of any one of claims 1 to 20, further comprising:

22. A method according to any preceding claim, wherein the entities are explicitly identified in the unstructured free-form natural language input.

23. 22. The method of any one of claims 1 to 21, wherein the entities are not explicitly identified in the unstructured, free-form, natural language input, the entities are of a particular type, the particular type of entities are explicitly identified in the unstructured, free-form, natural language input, and the one or more tasks are associated with the particular type of entities.

24. the one or more tasks are also associated with an additional entity of the particular type that is added to the entity, and causing the chatbot to perform the one or more tasks related to the entity on behalf of the user; causing the chatbot to engage in additional corresponding conversations with the additional entities; During said additional corresponding conversation with said additional entity, causing the chatbot to render a plurality of additional instances of synthesized speech audio data for presentation to additional representatives of the additional entities, wherein at least a given additional instance of synthesized speech audio data among the plurality of additional instances of synthesized speech audio data conveys the details of the one or more tasks also associated with the additional entities; receiving additional responsive content responsive to at least the given additional instance of synthesized voice audio data; causing the additional responsive content to be provided for presentation to the user of the client device; 24. The method of claim 23, further comprising:

25. 25. The method of claim 24, wherein the chatbot conducts the corresponding conversation with the entity and the additional corresponding conversation with the additional entity in parallel.

26. 25. The method of claim 24, wherein the chatbot conducts the additional corresponding conversation with the additional entity after the corresponding conversation with the entity and in response to determining that one or more of the tasks are not successfully completed during the corresponding conversation.

27. The method of any one of claims 1 to 26, wherein the representative of the entity is a human representative.

28. The method of any one of claims 1 to 26, wherein the representative of the entity is an additional chatbot trained to conduct the corresponding conversation on behalf of the entity.

29. 29. The method of any one of claims 1 to 28, wherein the unstructured, free-form natural language input is typed or spoken input that conveys the details of the one or more tasks associated with the entity, and does not define a corresponding dialogue state map, a dialogue state of the corresponding dialogue state map, or a dialogue state transition of the corresponding dialogue state map that is utilized in performing the one or more tasks associated with the entity.

30. 30. The method of claim 29, wherein the unstructured, free-form natural language input is typed input that conveys the details of the one or more tasks associated with the entity in one or more sentences, and the responsive content provided for presentation to the user of the client device is a result of performing the one or more tasks associated with the entity.

31. 30. The method of any one of claims 1 to 29, wherein the unstructured free-form natural language input is typed input conveying the details of the one or more tasks associated with the entity in one or more paragraphs, and the responsive content provided for presentation to the user of the client device is a summary of the corresponding conversation with the entity.

32. 1. A method implemented by one or more processors of a remote system, comprising: receiving, at the remote system, from a client device, an indication of unstructured free-form natural language input from a user of the client device, the unstructured free-form natural language input including one or more tasks associated with an entity; in response to receiving an indication of the unstructured free-form natural language input including the one or more tasks associated with an entity; generating a chatbot that performs the one or more tasks related to the entity on behalf of the user based on at least the indication of the natural language input and remotely at the remote system; causing the chatbot to perform the one or more tasks related to the entity on behalf of the user, wherein causing the chatbot to perform the one or more tasks related to the entity on behalf of the user includes: engaging the chatbot in a corresponding conversation with the entity; During the corresponding conversation with the entity, causing the chatbot to render a plurality of instances of synthesized speech audio data for presentation to a representative of the entity, at least a given instance of synthesized speech audio data among the plurality of instances of synthesized speech audio data conveying details of the one or more tasks associated with the entity; receiving responsive content responsive to at least the given instance of synthesized voice audio data; transmitting an indication of the responsive content from the remote system to the client device, the indication of the responsive content being transmitted to the client device, causing the client device to provide the responsive content for presentation to the user of the client device; A method comprising:

33. generating the chatbot to perform the one or more tasks related to the entity on behalf of the user, Obtaining a pre-trained large-scale language model (LLM); fine-tuning the pre-trained LLM based on the unstructured free-form natural language input to generate a fine-tuned LLM; Utilizing the fine-tuned LLM as the chatbot; 33. The method of claim 32, comprising:

34. 34. The method of claim 33, wherein the pre-trained LLM is stored in a remote storage of the remote system, and the pre-trained LLM stored in the remote storage of the remote system is a non-sparsed version of a global pre-trained LLM available at the remote system communicatively coupled to the client device.

35. 35. The method of claim 34, wherein the fine-tuned LLM is stored in the remote storage of the remote system.

36. after causing the chatbot to perform the one or more tasks related to the entity on behalf of the user; discarding the fine-tuned LLM from the remote storage on the remote system; and refraining from discarding the pre-trained LLM from the remote storage at the remote system; and 36. The method of claim 35, further comprising:

37. 1. A method implemented by one or more processors of a client device, comprising: receiving, at the client device, unstructured free-form natural language input from a user of the client device, the unstructured free-form natural language input including one or more tasks associated with an entity; in response to receiving an indication of the unstructured free-form natural language input including the one or more tasks associated with an entity; generating a chatbot based at least on the indication of the natural language input and remotely at a remote system to perform the one or more tasks related to the entity on behalf of the user; causing the chatbot to perform the one or more tasks related to the entity on behalf of the user, wherein causing the chatbot to perform the one or more tasks related to the entity on behalf of the user includes: engaging the chatbot in a corresponding conversation with the entity; During the corresponding conversation with the entity, causing the chatbot to render a plurality of instances of text data for presentation to a representative of the entity, at least a given instance of text data of the plurality of instances of text data conveying details of the one or more tasks associated with the entity; receiving responsive content responsive to at least the given instance of the text data; causing the responsive content to be provided for presentation to the user of the client device; A method comprising:

38. identifying the entities associated with the one or more tasks based on the unstructured free-form natural language input; determining corresponding identifiers of the entities associated with the one or more tasks; utilizing the corresponding identifier of the entity associated with the one or more tasks to have the chatbot participate in the corresponding conversation with the entity; 38. The method of claim 37, further comprising:

39. the corresponding identifier of the entity associated with the one or more tasks is one or more of a corresponding phone number of the entity, a corresponding email address of the entity, or a corresponding username of the entity, and utilizing the corresponding identifier of the entity associated with the one or more tasks to engage the chatbot in the corresponding conversation with the entity includes: having the chatbot utilize the corresponding identifier of the entity to initiate an automated text-based messaging session on behalf of the user to perform the one or more tasks related to the entity.

39. The method of claim 38, comprising:

40. Engaging the chatbot in the corresponding conversation with the entity includes: causing the chatbot to respond to text messages, emails, or other text-based messages received at the client device from the representative of the entity; engaging the chatbot in the corresponding conversation with the entity to facilitate the text message, the email, or other text-based message; 40. The method of any one of claims 37 to 39, comprising:

41. 1. A method implemented by one or more processors of a remote system, comprising: receiving, at the remote system, from a client device, an indication of unstructured free-form natural language input from a user of the client device, the unstructured free-form natural language input including one or more tasks associated with an entity; in response to receiving an indication of the unstructured free-form natural language input including the one or more tasks associated with an entity; generating a chatbot that performs the one or more tasks related to the entity on behalf of the user based on at least the indication of the natural language input and remotely at the remote system; causing the chatbot to perform the one or more tasks related to the entity on behalf of the user, wherein causing the chatbot to perform the one or more tasks related to the entity on behalf of the user includes: engaging the chatbot in a corresponding conversation with the entity; During the corresponding conversation with the entity, causing the chatbot to render a plurality of instances of synthesized speech audio data for presentation to a representative of the entity, at least a given instance of synthesized speech audio data among the plurality of instances of synthesized speech audio data conveying details of the one or more tasks associated with the entity; receiving responsive content responsive to at least the given instance of synthesized voice audio data; transmitting an indication of the responsive content from the remote system to the client device, the indication of the responsive content being transmitted to the client device, causing the client device to provide the responsive content for presentation to the user of the client device; A method comprising:

42. 1. A method implemented by one or more processors of a client device, comprising: receiving, at the client device, unstructured free-form natural language input from a user of the client device, the unstructured free-form natural language input including one or more tasks associated with an entity; identifying a chatbot to perform one or more tasks associated with the entity on behalf of the user in response to receiving the unstructured free-form natural language input including a natural language description of a corresponding dialogue state map; causing the chatbot to perform the one or more tasks related to the entity on behalf of the user, wherein causing the chatbot to perform the one or more tasks related to the entity on behalf of the user includes: engaging the chatbot in a corresponding conversation with the entity; During the corresponding conversation with the entity, causing the chatbot to render a plurality of instances of synthesized speech audio data for presentation to a representative of the entity, at least a given instance of synthesized speech audio data among the plurality of instances of synthesized speech audio data conveying details of the one or more tasks associated with the entity; receiving responsive content responsive to at least the given instance of synthesized voice audio data; causing the responsive content to be provided for presentation to the user of the client device; A method comprising:

43. Identifying the chatbot to perform the one or more tasks related to the entity on behalf of the user includes: Obtaining a pre-trained large-scale language model (LLM); causing the pre-trained LLM to be utilized as the chatbot; and 43. The method of claim 42, comprising:

44. 44. The method of claim 43, wherein the pre-trained LLM is stored in on-device storage of the client device, and wherein the pre-trained LLM stored in the on-device storage of the client device is a sparsified version of a global pre-trained LLM available at a remote system communicatively coupled to the client device.

45. refraining from fine-tuning the pre-trained LLM based on the unstructured free-form natural language input.

44. The method of claim 43, further comprising:

46. causing the chatbot to render the given instance of the synthetic speech among the plurality of instances of the synthetic speech for presentation to the representative of the entity includes: using the pre-trained LLM to process the unstructured free-form natural language input and task data reflecting details of the one or more tasks associated with the entity to generate instances of text data reflecting given behavior of the given implicit dialogue state; processing the given instance of text data reflecting the given operation of the given implicit dialogue state to generate the given instance of synthetic speech using a text-to-speech (TTS) model; transmitting the given instance of synthesized speech from the client device to an additional client device of the representative of the entity, wherein transmitting the given instance of synthesized speech to the additional client device causes the additional client device to audibly render the given instance of synthesized speech audio data for presentation to the representative of the entity via one or more speakers of the additional client device; 44. The method of claim 43, comprising:

47. 1. A system comprising: at least one processor; a memory storing instructions which, when executed by said at least one processor, cause the method of any one of claims 1 to 46 to be performed; Including, the system.

48. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations of the method of any one of claims 1 to 46.

Images