Chat robot development technology based on structured description

Abstract

Implementations involve: receiving an unstructured free form natural language input; generating a chat robot based on the unstructured free form natural language input and in response to receiving the unstructured free form natural language input; and causing the chat robot to perform participation in a corresponding session with the additional user. In various implementations, an unstructured free-form natural language input implicitly defines a corresponding dialog state graph (e.g., defines a corresponding dialog state and / or a corresponding dialog state transition) without defining any explicit dialog state and / or explicit dialog state transition. In other implementations, unstructured free-form natural language inputs are assigned to explicit dialog states and / or explicit dialog state transitions. Although, unstructured free-form natural language inputs may be used to trim and / or launch machine learning models that have already been able to be used to conduct generalized sessions. Thus, chat robots may be generated and deployed in a fast and efficient manner.

Description

BACKGROUND OF THE INVENTION

[0001] Humans can participate in human - machine conversations via various computing devices using interactive software applications known as "chatbots", "voicebots", "automated assistants", "interactive personal assistants", "intelligent personal assistants", "conversational agents", etc. As an example, these chatbots can correspond to machine - learning models or combinations of different machine - learning models and can be used to perform various tasks on behalf of a user. For example, some of these chatbots can engage in conversations with various people to perform actions on behalf of another person or entity. In some of these cases, the conversation can include a voice - based conversation, such as a conversation conducted locally at a computing device, a conversation conducted remotely on multiple computing devices via a telephone network, or a conversation in other voice - based scenarios. In other cases, the conversation can include a text - based conversation, such as a conversation conducted via text or SMS messaging, email, and / or other text - based scenarios.

[0002] However, the functionality of some of these chatbots can be limited in various ways. For example, the functionality of some of these chatbots can be restricted by predefined intent patterns that the chatbot uses to perform actions. In other words, if a person participating in a given conversation with a given chatbot provides an utterance that is determined to include an intent not defined by the predefined intent pattern, the given chatbot can fail. Further, to update these chatbots, existing intent patterns can be modified or new intent patterns can be added. As another example, the functionality of some of these chatbots can be restricted by the corpus of examples used to train the chatbot. In other words, if a person participating in a given conversation with a given chatbot provides an utterance that is not included in the given corpus of examples, the given chatbot can fail. Further, to update these chatbots, existing examples in the corpus can be modified or new examples can be added. However, in both of these examples, there are almost an infinite number of intent patterns and / or examples of situations that may need to be previously defined to make the robot robust to the various nuances of human speech and reduce failures.

[0003] Note that a large amount of computing resources are required to manually define and / or manually refine such intent patterns and / or examples. Further, even if a large number of intent patterns and / or examples are defined, a large amount of memory is required to store and / or utilize a large number of intent patterns for these chatbots, and / or to train these chatbots based on a large number of examples in a corpus. Thus, the intent patterns for rule-based chatbots and the examples for example-based chatbots are not actually scalable to the extent of learning the nuances of human speech. Summary of the Invention

[0004] Implementations involve: receiving unstructured free-form natural language input from a user; generating a chatbot based on the unstructured free-form natural language input and in response to receiving the unstructured free-form natural language input; and causing the chatbot to engage in a corresponding conversation with other users besides the user. In some versions of those implementations, the unstructured free-form natural language input implicitly defines a corresponding dialogue state diagram (e.g., defines corresponding dialogue states and / or corresponding dialogue state transitions) without explicitly defining any dialogue states and / or explicit dialogue state transitions. In other versions of those implementations, the unstructured free-form natural language input is assigned to explicit dialogue states and / or explicit dialogue state transitions. Nevertheless, in various implementations, the unstructured free-form natural language input can be used to fine-tune a machine learning (ML) model that can already be used to conduct general conversations, and / or can be used as an input to the ML model without fine-tuning the ML model. Thus, chatbots can be generated and deployed in a fast and efficient manner and used to engage in corresponding conversations on behalf of the user or an entity associated with the user.

[0005] For example, assume that the unstructured free-form natural language input corresponds to a natural language description of how the chatbot should engage in a corresponding conversation with an additional user. The processor can use various automatic speech recognition (ASR), natural language understanding (NLU), and / or fulfillment techniques to determine that the unstructured free-form natural language input implicitly defines a corresponding dialogue state diagram to be utilized by the chatbot. In this example, even if the user only provides a natural language description to implicitly define the corresponding dialogue state diagram without explicitly defining any explicit dialogue states or any explicit dialogue state transitions, the processor can generate a chatbot for engaging in a corresponding conversation with the additional user based on the corresponding dialogue state diagram. Additionally or alternatively, the processor can utilize the natural language description at one or more turns of the corresponding conversation without explicitly fine-tuning the chatbot. Based on the corresponding conversation, the processor can determine responsive content for the user to be provided for presentation to the client device.

[0006] In various implementations, the corresponding session can be a voice-based session, where the chatbot participates in the corresponding session via a corresponding phone call or locally at the client device. In these implementations, the chatbot can additionally or alternatively be referred to as a voicebot. In other implementations, the corresponding session can be a text-based session, where the chatbot participates in the corresponding session using text messaging or SMS services, email services, or other text-based services. Thus, the chatbot can be deployed in various environments to participate in corresponding sessions with various additional users. In some implementations, the generated chatbot can be fine-tuned in an "instant" manner such that the chatbot is generated in response to receiving an unstructured free-form natural language input. In other implementations, the generated chatbots may not be fine-tuned in an "instant" manner, but they can be recognized and utilized in response to receiving an unstructured free-form natural language input.

[0007] In various implementations, the processor can be implemented locally at the client device of the user's unstructured free-form natural language input. In some versions of these implementations, the processor can obtain a previously trained large language model (LLM) from the on-device storage of the client device as an ML model that is already capable of being used for more generalized sessions. Further, the processor can fine-tune the previously trained LLM based on the unstructured free-form natural language input to generate a fine-tuned LLM. Additionally, the processor uses the fine-tuned LLM as a chatbot to represent the user in participating in the corresponding session. In other versions of these implementations, the processor can obtain a previously trained LLM from the on-device storage of the client device as an ML model that is already capable of being used for more generalized sessions, but avoids fine-tuning the previously trained LLM based on the unstructured free-form natural language input.

[0008] In other implementations, the processor can be implemented away from the user's client device (e.g., at a remote system such as a high-performance server or a high-performance server cluster). In some versions of these implementations, the processor can obtain a previously trained large language model (LLM) from the remote storage of the remote system as an ML model that is already capable of being used for more generalized sessions. Further, the processor can generate a fine-tuned LLM and use the fine-tuned LLM as a chatbot to represent the user in participating in the corresponding session. In other versions of these implementations, the processor can obtain a previously trained LLM from the remote storage of the remote system as an ML model that is already capable of being used for more generalized sessions, but avoids fine-tuning the previously trained LLM based on the unstructured free-form natural language input.

[0009] Notably, the previously trained LLM can correspond to an existing LLM such as LaMDA, BERT, Meena, GPT-3, and / or any other previously trained LLM. These previously trained LLMs have been previously trained on a large amount of diverse data and are capable of participating in a corresponding conversation with a user in a natural and intuitive manner. However, these LLMs have multiple ML layers and hundreds of millions to hundreds of billions of ML parameters. Thus, in an implementation where the chatbot fine-tuned therein is locally generated at the client device, the previously trained LLM obtained and fine-tuned can be a sparsified version of the previously trained LLM. In contrast, in an implementation where the chatbot fine-tuned therein is generated away from the client device, the previously trained LLM obtained and fine-tuned can be an unsparsified version of the previously trained LLM. Compared with the almost infinite resources of the remote system, due to various hardware constraints and / or software constraints at the client device, the sparsified version of the previously trained LLM can have fewer ML layers, fewer ML parameters, masked weights, and / or other sparsification aspects to reduce the size of the previously trained LLM.

[0010] In some implementations, and when causing the chatbot to participate in a corresponding conversation, the processor can use the fine-tuned LLM to process state / transition data that implicitly or explicitly defines the corresponding conversation state / transition, additional user data (e.g., an output generated by using an ASR model, an NLU model, and / or a fulfillment model or rule and based on processing responses provided by the additional user during the corresponding conversation), any session context data for the corresponding conversation, and / or any other data described herein to generate an output. In other implementations, and when causing the chatbot to participate in a corresponding conversation, the processor can use a previously trained (e.g., not fine-tuned) LLM to process a natural language description included in an unstructured free-form natural language input, state / transition data that implicitly or explicitly defines the corresponding conversation state / transition, additional user data (e.g., an output generated by using an ASR model, an NLU model, and / or a fulfillment model or rule and based on processing responses provided by the additional user during the corresponding conversation), any session context data for the corresponding conversation, and / or any other data described herein to generate an output. For example, the output can be a probability distribution over a vocabulary or a sequence of terms and / or phrases. Based on the probability distribution over the vocabulary or the sequence of terms and / or phrases, the processor can select an instance of text data corresponding to the text and / or speech to be provided by the chatbot.

[0011] In an implementation where the corresponding session is a text-based session, the processor may cause an instance of text data to be visually rendered for presentation to an additional user at a client device and / or at an additional client device of the additional user. However, in an implementation where the corresponding session is a voice-based session, the processor may cause a chatbot to use a text-to-speech (TTS) model to process an instance of the corresponding text data to generate an instance of synthetic speech audio data that captures the synthetic speech corresponding to the text data. Further, the processor may cause the instance of the synthetic speech audio data to be visually rendered for presentation to an additional user at a client device of the entity and / or at an additional client device. Notably, in an implementation where the chatbot corresponds to a previously trained LLM that is fine-tuned based on unstructured free-form natural language input, the chatbot is capable of generating a conversation output that focuses on the state / transition implicitly and / or explicitly defined by the unstructured free-form natural language input. Further, in an implementation where the chatbot corresponds to a previously trained LLM that is not fine-tuned, the chatbot is still capable of generating a conversation output that focuses on the state / transition implicitly and / or explicitly defined by the unstructured free-form natural language input, since the unstructured free-form natural language input is still applied as input to the previously trained LLM that is not fine-tuned.

[0012] In various implementations, the processor may cause corresponding responsive content to be provided for presentation to a user of a client device that provided unstructured free-form natural language input. The responsive content may be determined based on one or more responses provided by an additional user during the corresponding session. Further, the responsive content may include, for example, the corresponding result of one or more tasks performed during the corresponding session, the corresponding summary of the corresponding session, and / or other content.

[0013] In various implementations and during the corresponding session, the chatbot may engage in the corresponding session with an additional user by utilizing one or more peripheral behaviors. These peripheral behaviors may include, for example, a greeting behavior that enables the chatbot to identify the user and / or identify itself as a chatbot, a hold behavior that enables the chatbot to pause and resume the corresponding session, an exit behavior that enables the chatbot to terminate the corresponding session with the additional user, and / or other peripheral behaviors. These peripheral behaviors are some non-limiting examples of why a previously trained LLM enables the chatbot to perform the general aspects of a conversation and without the unstructured free-form natural language input having to specify these general aspects of the conversation that the chatbot is capable of performing. However, a fine-tuned chatbot that is fine-tuned based on unstructured free-form natural language input enables the chatbot to perform aspects of the conversation specified by the user while still being capable of performing these general aspects of the conversation.

[0014] By using the techniques described herein, various technical advantages can be achieved. As a non-limiting example, the techniques described herein enable a processor of a client device and / or a remote system to generate a chatbot based on unstructured free-form natural language input to participate in a corresponding conversation with an additional user and / or to use an existing chatbot based on unstructured free-form natural language input to participate in a corresponding conversation with an additional user. These corresponding conversations can be conducted according to a conversation state diagram implicitly and / or explicitly defined in the natural language description provided by the user. This enables the processor to generate and deploy a chatbot in a fast and efficient manner to participate in the corresponding conversation.

[0015] The above description is provided as an overview of only some implementations disclosed herein. Those implementations and other implementations are described in more detail herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A block diagram depicting an example environment that illustrates various aspects of the present disclosure and in which the implementations disclosed herein can be implemented.

[0017] Figure 2 A flow diagram depicting an example process flow for generating a chatbot and causing the chatbot to participate in a corresponding conversation with an additional user according to various implementations.

[0018] Figure 3 A flow diagram depicting an example method of locally generating a chatbot at a client device and causing the chatbot to participate in a corresponding conversation with an additional user according to various implementations.

[0019] Figure 4 A flow diagram depicting an example method of remotely generating a chatbot at a remote system and causing the chatbot to participate in a corresponding conversation with an additional user according to various implementations.

[0020] Figure 5A , Figure 5B and Figure 5C Depicts various non-limiting example interactions in which corresponding unstructured free-form natural language input is used to generate a corresponding chatbot and the corresponding chatbot participates in a corresponding conversation with an additional user based on the corresponding unstructured free-form natural language input according to various implementations.

[0021] Figure 6 Depicts an example architecture of a computing device according to various implementations. DETAILED DESCRIPTION

[0022] Now turning to Figure 1, which depicts a block diagram showing an example environment that illustrates various aspects of the present disclosure and in which the implementations disclosed herein may be implemented. The client device 110 is shown in Figure 1 and, in various implementations, includes a user input engine 120, a rendering engine 130, an on-device machine learning (ML) model engine 140, and a chatbot development engine client 150. The client device 110 can be, for example, a stand-alone device (e.g., having a microphone, vision components, speakers, a display, and / or other user interface components), a laptop computer, a desktop computer, a tablet, a wearable computing device, an in-vehicle computing device, and / or any other client device capable of implementing the chatbot development engine client 150.

[0023] The user input engine 120 can detect various types of user input at the client device 110. In some examples, the user input detected at the client device 110 can include verbal input detected via a microphone of the client device 110. In these examples, the microphone of the client device 110 can generate audio data that captures the verbal utterances included in the verbal input. In other examples, the user input detected at the client device 110 can include touch input detected via a user interface input device (e.g., a touch-sensitive display) of the client device 110, and / or keystroke input detected via a user interface input device (e.g., a touch-sensitive display and / or a keyboard) of the client device 110. In these examples, the user interface input device of the client device 110 can generate text data that captures the touch input and / or the keystroke input. It is noted that the unstructured free-form natural language input described herein can be provided by a user of the client device 110 as any combination of verbal input, touch input, and / or keystroke input.

[0024] The rendering engine 130 can cause responsive content and / or other output to be visually rendered for presentation to a user at the client device 110 (e.g., via a touch-sensitive display or other user interface output device), and / or to be aurally rendered for presentation to a user at the client device 110 (e.g., via a speaker or other user interface output device). The responsive content and / or other output can include, for example, various types of user interfaces associated with the chatbot development engine client 150 that can be visually rendered via the user interface of the client device 110, such as unstructured free-form natural language inputs provided by a user of the client device 110 that implicitly define corresponding dialogue state diagrams to be utilized by the chatbot when participating in a corresponding session with an additional user (e.g., a user other than the user of the client device 110), unstructured free-form natural language inputs provided by a user of the client device 110 that explicitly define corresponding dialogue state diagrams to be utilized by the chatbot when participating in a corresponding session with an additional user (e.g., a user other than the user of the client device 110), various prompts associated with the corresponding session executed by the chatbot, the results and / or summaries of the corresponding session executed by the chatbot, and / or any other responsive content or output that can be visually and / or aurally rendered for presentation to a user at the client device 110.

[0025] In various implementations, the on-device ML model engine 140 can include an automatic speech recognition (ASR) engine 141, a natural language understanding (NLU) engine 142, an execution engine 143, and a text-to-speech (TTS) engine 144. As described in more detail below, these on-device ML models of the on-device ML model engine 140 can utilize various on-device ML models (e.g., stored in the on-device ML model database 140A) to process various user inputs (e.g., received via the user input engine 120) and generate various outputs (e.g., to be visually and / or aurally rendered via the rendering engine 130 for presentation to a user). This, in turn, enables the chatbot development engine client 150 to use the on-device ML model engine 140 to process various user inputs received at the client device 110 and generate various outputs to be provided for presentation to a user at the client device 110.

[0026] Further, the client device 110 is in Figure 1is shown as communicatively coupled to remote system 160 via one or more networks 199 (e.g., any combination of Wi-Fi, Bluetooth, or other local area networks (LANs); Ethernet, the Internet, or other wide area networks (WANs); and / or other networks). In various implementations, remote system 160 includes remote system ML model engine 170 and chatbot development engine 180. Remote system 160 can be, for example, a high-performance server, a high-performance server cluster, and / or any other computing device remote from client device 110.

[0027] In various implementations, remote ML model engine 170 can include ASR engine 171, NLU engine 172, fulfillment engine 173, and TTS engine 174. As described in more detail below, these remote ML model engines of remote engine 170 can utilize various remote ML models (e.g., stored in remote ML model database 170A) to process various user inputs (e.g., received from client device 110 and via user input engine 120) and generate various outputs (e.g., to be transmitted to client device 110 via rendering engine 130 for visual and / or audible presentation to the user) in the same or similar manner as the on-device ML model engine 140. This, in turn, enables chatbot development engine 180 to use remote ML model engine 170 to process various user inputs received at client device 110 and generate various outputs to be provided for presentation to the user at client device 110. In implementations where remote ML model engine 170 is used to process various user inputs received at client device 110 and generate various outputs to be provided for presentation to the user at client device 110, the various user inputs received at client device 110 can be transmitted from client device (or additional client device 191) to remote system 160, and the various user outputs can be transmitted from remote system to client device 110 (or additional client device 191).

[0028] Notably, chatbot development engine client 150 of the client device can communicate with chatbot development engine 180 via one or more networks 199. From the perspective of a user interacting with client device 110, chatbot development engine client 150 and chatbot development engine 180 form a logical instance of a chatbot development platform. Although the chatbot development platform is in Figure 1is depicted as being implemented in a distributed manner over one or more networks 199 (e.g., via the use of a chatbot development engine client 150 and a chatbot development engine 180), but it should be understood that this is for purposes of example and is not intended to be limiting. For example, the chatbot development platform may alternatively be implemented specifically at the client device 110. As another example, the chatbot development platform may alternatively be implemented specifically at the remote system 160, but the client device 110 may still enable a user to interact with the chatbot development platform.

[0029] The chatbot development platform may be used by a user (e.g., a user of the client device 110) to generate a chatbot as described herein to be deployed to conduct a corresponding session on behalf of an entity (e.g., via a third-party system 192) such as a third party associated with the user of the client device 110. It is noted that the chatbot development platform may be provided by a first party, and a user may utilize the chatbot development platform to generate a chatbot for himself or herself or for a third party associated with the user. As used herein, the term first party refers to the entity that publishes the chatbot development platform, and the term third party refers to an entity that is different from the entity associated with the first party and has not published a chatbot development system. Thus, a user of the client device 110 who interacts with the chatbot development platform may also be referred to as a third-party developer.

[0030] The corresponding sessions described herein as being conducted by the chatbot and on behalf of an entity may include various types of sessions, such as voice-based sessions and text-based sessions. Voice-based sessions may include, for example, corresponding sessions conducted during an automated telephone call (e.g., Voice over Internet Protocol (VoIP), Public Switched Telephone Network (PSTN), and / or other telephone communication protocols) and between the client device and an additional client device 191, corresponding sessions in which the chatbot and other entities and / or users participate locally at a given client device (e.g., in a scenario where the client device 110 is a shared client accessible by multiple users) and / or any other voice-based scenario in which the chatbot is deployed to conduct a corresponding session with a user. Text-based sessions may include, for example, corresponding sessions conducted during text or SMS messaging, email, and / or any other text-based scenario in which the chatbot is deployed to conduct a corresponding session with a user.

[0031] As described above, the chatbot development platform can utilize the on-device ML model engine 140 and / or the remote system ML model engine 170 to process various user inputs received at the client device 110 and generate various outputs to be provided for presentation to the user at the client device 110 and / or for presentation to additional users at the corresponding additional client devices 191. Each sub-engine in the on-device ML model engine 140 and / or the remote system ML model engine 170 can be configured to perform one or more functions. Notably, the remote system ML model engine 170 includes a remote-based counterpart of the sub-engines of the on-device ML model engine 140. In various implementations, the utilization of the on-device ML model engine 140 can be prioritized, at least in part, due to latency considerations, network bandwidth considerations, privacy considerations, and / or other considerations. In these implementations, when one or more sub-engines in the on-device ML model engine 140 fail, the remote system ML model engine 170 can be utilized. In other implementations, the utilization of the remote ML model engine 170 can be prioritized, at least in part, due to computational considerations at the client device 110, hardware considerations at the client device 110, software considerations at the client device 110, and / or other considerations. In still other implementations, the on-device ML model engine 140 and the remote system ML model engine 170 can be utilized in combination with each other.

[0032] For example, the ASR engine 141 and / or 171 can use an ASR model (e.g., a recurrent neural network (RNN) model, a transformer model, and / or any other type of ML model capable of performing ASR) stored in the corresponding ML model database to process audio data captured from spoken utterances and generated by the microphone of the client device 110 to generate an ASR output. Further, the NLU engine 142 and / or 172 can use an NLU model (e.g., a long short-term memory (LSTM), a gated recurrent unit (GRU), and / or any other type of RNN or other ML model capable of performing NLU) and / or NLU rules stored in the corresponding ML model database to process the ASR output (or other typed input or touch input received via the user input engine 120 of the client device 110) to generate an NLU output. Additionally, the fulfillment engine 143 and / or 173 can use a fulfillment model and / or fulfillment rules stored in the corresponding ML model database to process the NLU data to generate a fulfillment output. Additionally, the TTS engine 144 and / or 174 can use a TTS model stored in the corresponding ML model database to process text data (e.g., text formulated by the chatbot) to generate synthetic speech audio data including computer-generated synthetic speech corresponding to the text data.

[0033] In various implementations, the ASR output can include, for example, multiple speech hypotheses (e.g., word hypotheses and / or transcription hypotheses) for the verbal input based on the processing of the audio data, and a particular speech hypothesis can be optionally selected as the recognized text for the verbal input based on a corresponding value (e.g., a probability value, a log-likelihood value, and / or other value) associated with each of the multiple speech hypotheses. In various implementations, the ASR model stored in the corresponding ML model database is an end-to-end speech recognition model such that the ASR engine 141 and / or 171 can directly use the model to generate multiple speech hypotheses. For example, the ASR model can be an end-to-end model for generating each of the multiple speech hypotheses on a character-by-character basis (or other sub-word basis). A non-limiting example of such an end-to-end model for generating recognized text on a character-by-character basis is the recurrent neural network transducer (RNN-T) model. The RNN-T model is in the form of a sequence-to-sequence model that does not employ an attention mechanism. In other implementations, the ASR model is not an end-to-end speech recognition model such that the ASR engine 141 and / or 171 can alternatively generate the predicted phonemes (and / or other representations). For example, the predicted phonemes (and / or other representations) can then be used by the ASR engine 141 and / or 171 to determine multiple speech hypotheses that match the predicted phonemes. In doing so, the ASR engine 141 and / or 171 can optionally employ a decoding graph, a dictionary, and / or other resources. In various implementations, the corresponding transcription can be rendered at the client device 110 (e.g., rendered in association with the training instance input, the training instance output, the corresponding feature emphasis input, the demonstration session, and / or other aspects of the chatbot development platform).

[0034] In various implementations, the NLU output can include, for example, the annotated identified text, which includes one or more (e.g., all) annotations for one or more of the terms of the identified text for the identified text. For example, the NLU engines 142 and / or 172 can include a part-of-speech tagger (not depicted) configured to annotate terms using the syntactic role of the terms. Additionally or alternatively, the NLU engines 142 and / or 172 can include an entity tagger (not depicted) configured to annotate entity references in one or more segments of the identified text, such entity references being references to persons (including, for example, literary figures, celebrities, public figures, etc.), organizations, locations (real and imaginary), etc. In some implementations, data about entities can be stored in one or more databases, such as stored in a knowledge graph (not depicted). In some implementations, the knowledge graph can include nodes representing known entities (and in some cases, entity attributes), and edges connecting the nodes and representing relationships between the entities. The entity tagger can annotate references to entities at a high granularity level (e.g., such that all references to an entity category, such as persons, can be identified) and / or at a lower granularity level (e.g., such that all references to a specific entity, such as a specific person, a specific location, etc., can be identified). The entity tagger can rely on the content of the unstructured free-form natural language input to resolve specific entities and / or can optionally communicate with a knowledge graph or other entity database to resolve specific entities. Additionally or alternatively, the NLU engines 142 and / or 172 can include a coreference resolver (not depicted) configured to group or "cluster" references to the same entity based on one or more context clues. For example, the coreference resolver can be used to resolve the term "them" in the input "buy them" to "buy theater tickets" based on the mention of "theater tickets" in the client device notification rendered immediately before receiving the natural language input "buy them". In some implementations, one or more components of the NLU engines 142 and / or 172 can rely on annotations from one or more other components of the NLU engines 142 and / or 172. For example, in some implementations, the entity tagger can rely on annotations from the coreference resolver to annotate all mentions of a specific entity. Additionally, for example, in some implementations, the coreference resolver can rely on annotations from the entity tagger to cluster references to the same entity.Additionally, for example, in some implementations, the coreference resolver may depend on user data of the user of the client device 110 (e.g., stored in the user data database 110A) in coreference resolution and / or entity resolution. The user data may include, for example, historical location data, historical time data, user preference data, user account data, calendar data, email data, and / or any other user data available at the client device 110.

[0035] In various implementations, the fulfillment output may include, for example, one or more tasks to be performed by the chatbot on behalf of the user of the client device 110, on behalf of an entity associated with the user of the client device 110, and / or on behalf of an additional user of a corresponding additional client device 191. As described in more detail herein (e.g., with respect to Figure 2 , Figure 3 , Figure 4 , Figure 5A , Figure 5B and Figure 5C ), the user of the client device 110 may provide unstructured free-form natural language input that implicitly and / or explicitly defines a corresponding dialogue state diagram to be utilized by the chatbot when participating in a corresponding session with an additional user (e.g., a user other than the user of the client device 110). The corresponding session with the additional user may require the chatbot to perform tasks on behalf of the additional user and / or on behalf of an entity. Notably, the unstructured free-form natural language input may implicitly define (e.g., as described with reference to Figure 5A ) the corresponding dialogue state diagram or explicitly define (e.g., as described with reference to Figure 5B ) the corresponding dialogue state diagram for utilization by the chatbot when participating in the corresponding session. Nevertheless, and by utilizing the chatbot development engine client 150 and / or the chatbot development engine 180, a chatbot may be generated and deployed to participate in the corresponding session in response to receiving the unstructured free-form natural language input. Thus, it should be understood that the fulfillment output may depend on the corresponding session.

[0036] In various implementations, the TTS engine 144 and / or 174 can generate synthetic speech audio data that captures computer-generated synthetic speech. The synthetic speech audio data can be rendered at the client device 110 via the speakers of the client device 110, and / or at an additional client device (e.g., a client device associated with an entity) via the corresponding speakers of the additional client device 191. The synthetic speech can include any output generated by the chatbots described herein, and can include, for example, synthetic speech generated as part of a conversation between a user of the client device 110 and the chatbot, synthetic speech generated as part of a conversation between an entity (or an additional user associated with the entity) and the chatbot, and / or other synthetic speech.

[0037] Although Figure 1 described with respect to a single client device with a single user, it should be understood that this is for illustrative purposes only and is not meant to be limiting. For example, one or more additional client devices of the user can also implement the techniques described herein. For example, the client device 110, one or more additional client devices, and / or any other computing device of the user can form a device ecosystem that can employ the techniques described herein. These additional client devices and / or computing devices can communicate with the client device 110 and / or the remote system 160 (e.g., via one or more networks 199). As another example, a given client device can be utilized by multiple users in a shared setting (e.g., a group of users, a family, etc.).

[0038] In various implementations, the chatbot development engine 180 can include a chatbot identification engine 181, a chatbot fine-tuning engine 182, a dialogue state / transition engine 183, a session engine 184, a session context engine 185, a responsive content engine 186, and a peripheral behavior engine 187, as Figure 1 depicted. Although the chatbot development engine 180 is depicted as having specific sub-engines, it should be understood that this is for illustrative purposes only and is not intended to be limiting. For example, one or more of the sub-engines depicted in Figure 1 can be combined, while one or more of the other sub-engines depicted in Figure 1 can be omitted. Further, although the chatbot development engine client 150 is not depicted as including any sub-engines, it should be understood that this is for simplicity and is not intended to be limiting. For example, the chatbot development engine client 150 can include the same sub-engines or a subset thereof described with respect to the chatbot development engine 180. Additional descriptions of the chatbot development engine 180 and its various sub-engines are provided with respect to Figure 2

[0039] Now refer to Figure 2, depicts an example process flow 200 for generating a chatbot and causing the chatbot to participate in a corresponding conversation with an additional user. For example purposes, it is assumed that a user of the client device 110 from Figure 1 provides an unstructured free-form natural language input 201 as an input at the client device 110. The client device 110 can receive the unstructured free-form natural language input 201 via the user input engine 120 of the client device 110. The client device 110 can cause the unstructured free-form natural language input 201 to be processed using various sub-engines of the on-device ML model engine 140 and / or using various sub-engines of the remote ML model engine 170.

[0040] Notably, when processing the unstructured free-form natural language input 201, the client device 110 can identify one or more features 202 based on the output generated by one or more of the various sub-engines of the on-device ML model engine 140 and / or one or more of the various sub-engines of the remote ML model engine 170. The one or more features 202 can include, for example, ASR output in cases where the unstructured free-form natural language input 201 is a verbal input, NLU output in cases where the unstructured free-form natural language input 201 is a verbal input or a typed input, and / or fulfillment output in cases where the unstructured free-form natural language input 201 is a verbal input or a typed input, such as entities, intents, slot values, tasks associated with the entities to be performed by the chatbot, and / or other features. The one or more features 202 can include features for a given portion of the unstructured free-form natural language input 201 or for multiple portions (e.g., two or more portions, or all portions) of the unstructured free-form natural language input 201.

[0041] For purposes of illustration, further assume that the unstructured free-form natural language input 201 is received from a user associated with a "Ma & Pa Store" (a fictional merchandise store) and corresponds to the spoken utterance "Tell everyone who calls our operating hours and our physical address, but that we are only accepting online orders at this time". In this example, the ASR output can be the recognized text corresponding to the spoken utterance (e.g., the recognized text of "Tell everyone who calls our operating hours and our physical address, but that we are only accepting online orders at this time"), and the NLU output can be a "Provide Content" intent with a slot value of [operating hours] associated with the "operating hours" dialogue state, a slot value of [physical address] associated with the "physical address" dialogue state, and a slot value of "we are only accepting online orders at this time" associated with the "additional information" dialogue state. It is noted that one or more features 202 in this example include corresponding dialogue states, and corresponding dialogue state transitions can be inferred based on the unstructured free-form natural language input 201. Thus, in some of these examples, and in response to receiving the unstructured free-form natural language input 201, a chatbot can be generated during a training phase (e.g., enclosed by box 280A in Figure 2 configured to answer any incoming phone calls directed to the "Ma & Pa Store".

[0042] During the training phase, the chatbot recognition engine 181 may recognize the chatbot 203 (e.g., stored in the chatbot database 180A). The chatbot 203 may be a previously trained ML model or a combination of various previously trained ML models, and may fine-tune the previously trained ML model or the combination of various previously trained ML models based on the unstructured free-form natural language input 201 and / or one or more features 202 extracted from the unstructured free-form natural language input 201. For example, the chatbot 203 may correspond to a previously trained LLM such as LaMDA, BERT, Meena, GPT-3, and / or another previously trained large language model (LLM). Notably, these previously trained LLMs have been previously trained on a large amount of different data (e.g., thousands of hours) and are generally generative ML models capable of participating in corresponding conversations with users in a more natural and intuitive manner. These LLMs have multiple ML layers and hundreds of millions to hundreds of billions of ML parameters and are capable of generalizing corresponding conversations with users.

[0043] For example, and as described herein (e.g., with respect to Figure 5A , Figure 5B and Figure 5C)More specifically, text data can be provided as input to these previously trained LLMs to generate an LLM output, such as a probability distribution over a vocabulary, and a response to the text data can be generated based on the probability distribution over the vocabulary. Due to the multiple ML layers and hundreds of millions to hundreds of billions of ML parameters, it should be noted that LLMs are generally not conducive to being locally implemented at the client device 110, such as when the chatbot database 180A is local to the client device 110 (e.g., stored in the device storage of the client device 110). Nevertheless, various sparsification techniques can be utilized to reduce the amount of ML layers and / or the amount of ML parameters utilized by these LLMs, such that a sparsified version of the previously trained LLM can be locally implemented at the client device 110 while mitigating a reduction in the accuracy and / or recall of the previously trained LLM due to sparsification. These sparsification techniques can include, but are not limited to, folding and / or combining multiple layers among the multiple ML layers of the previously trained LLM, pruning multiple layers among the multiple ML layers of the previously trained LLM, masking the weights of the previously trained LLM, pruning the weights of the previously trained LLM, and / or other sparsification techniques. However, when the chatbot database 180A is remote from the client device 110 (e.g., stored in the remote storage of the remote system 160), an unsparsified version of the previously trained LLM can be remotely implemented at the remote system 160. Thus, the chatbot 203 can be locally identified at the client device 110 and / or remotely identified at the remote system 160 (e.g., remote from the client device 110 that received the unstructured free-form natural language input 201).

[0044] Further, during the training phase, the chatbot fine-tuning engine 182 can utilize various fine-tuning techniques to generate a fine-tuned chatbot 204 by fine-tuning the chatbot 203 and based on the unstructured free-form natural language input 201 and / or one or more features 202 extracted from the unstructured free-form natural language input 201 (and the fine-tuned chatbot 204 can optionally be stored in the chatbot database 180A). These fine-tuning techniques can include, but are not limited to, instruction tuning, few-shot learning, and / or other fine-tuning techniques, and the fine-tuning performed can vary based on the unstructured free-form natural language input 201 provided by the user. In other words, the previously trained LLM corresponding to the chatbot 203 can be further trained based on the unstructured free-form natural language input 201 and / or one or more features 202 extracted from the unstructured free-form natural language input 201 such that the previously trained LLM that is fine-tuned and corresponds to the fine-tuned chatbot 204 is adapted to represent the user in the corresponding conversation. By fine-tuning the chatbot 203, the resulting fine-tuned chatbot 204 utilizes the generalization ability of the previously trained LLM while also being adapted to represent the entity in the corresponding conversation. Thus, the chatbot 203 can be fine-tuned to locally generate at the client device 110 and / or remotely generate at the remote system 160 (e.g., away from the client device 110 that received the unstructured free-form natural language input 201) the fine-tuned chatbot 204. The fine-tuned chatbot 204 can then be utilized by the client device 110 and / or the remote system 160 during the inference phase (e.g., enclosed by the box 280B in Figure 2 ).

[0045] Although described with respect to fine-tuning the chatbot 203 based on the unstructured free-form natural language input 201 and / or one or more features 202 of the unstructured free-form natural language input 201 to generate the fine-tuned chatbot 204 Figure 2 , it should be understood that this is just one implementation contemplated herein. For example, in other implementations, the chatbot 203 may not be fine-tuned such that the chatbot 203 can then be utilized by the client device 110 and / or the remote system 160 during the inference phase (e.g., enclosed by the box 280B in Figure 2 ).

[0046] During the inference phase, the dialogue state / transition recognition engine 183 may determine state / transition data 205 to be utilized by the chatbot when participating in a given corresponding session with a given additional user. Continuing the above example where the unstructured free-form natural language input 201 is the spoken utterance “Tell everyone who calls our operating hours and our physical address, but that we are only accepting online orders at this time”, the state / transition data 205 may include, for example, instructions for generating and providing an instance of synthetic speech audio data for presentation to the given user based on the “operating hours” dialogue state, the “physical address” dialogue state, the “additional information” dialogue state, and / or other dialogue states. In various implementations, information about dialogue states / transitions may be stored in the dialogue state / transition database 180B and / or other data sources. For example, the operating hours of the “Ma & Pa Store”, the physical address of the “Ma & Pa Store”, and the additional information “we are only accepting online orders at this time” may be stored in the dialogue state / transition database 180B in association with the respective dialogue states. Although the state / transition data 205 is described as including specific data, it should be understood that this is for illustrative purposes and is not intended to be limiting. For example, the state / transition data 205 may include any data related to any dialogue state and / or transitions therebetween that is implicitly and / or explicitly provided by the user of the client device 110 in the unstructured free-form natural language input 201.

[0047] Notably, when participating in a corresponding session, the chatbot may answer incoming electronic communications (including incoming voice-based communications such as incoming phone calls and incoming text-based communications such as incoming emails and incoming text messages) on behalf of the entity, and initiate outgoing electronic communications (including outgoing voice-based communications such as outgoing phone calls and outgoing text-based communications such as outgoing emails and outgoing text messages) on behalf of the entity. Further, the dialogue state / transition recognition engine 183 may provide the state / transition data 205 to the session engine 184 to enable the fine-tuned chatbot 204 (or simply the chatbot 203) to participate in the corresponding session. For example, the session engine 184 may at least process the state / transition data 205 to generate session data 206, such as an instance of synthetic speech audio data and / or an instance of text data that may be rendered at a given additional client device 191A of a given additional user.

[0048] In various implementations, the response data 207 received during a corresponding session can also be used to generate the session data 206. The response data 207 can include responses or other inputs initially provided by a given additional user when initiating a given corresponding session, or other responses provided by the given additional user in response to instances of synthesized voice audio data and / or instances of text data. In various implementations, the session context engine 185 can provide session context data 208 to the session engine 184 and enable the fine-tuned chatbot 204 (or simply the chatbot 203) to participate in a more context-aware corresponding session with a given additional user. In these implementations, the session context data 208 can represent (e.g., as a vector or other data structure) the initial context information for the corresponding session or subsequent context information determined during the corresponding session (e.g., determined based on data stored in the chatbot activity database 180C).

[0049] Continuing the above example where the unstructured free-form natural language input 201 is the verbal expression “Tell everyone who calls our operating hours and our physical address, but that we are only accepting online orders at this time”, the session context engine 185 can generate session context data 208 indicating “Hello, thanks for calling Ma & Pa Store”, information associated with the operating hours of “Ma & Pa Store”, information associated with the physical address of “Ma & Pa Store”, information indicating that “Ma & Pa Store” only accepts online orders, and / or other context information that can be inferred based on the unstructured free-form natural language input 201.

[0050] Further, during the inference phase and in an implementation where the chatbot corresponds to the fine-tuned chatbot 204, the session engine 184 can initially use the fine-tuned chatbot 204 (e.g., instead of the chatbot 203) to process the state / transition data 205 (and optionally any response data 207 and / or session context data 208) to generate an output, such as a probability distribution over a sequence of words or phrases. The session engine 184 can generate session data 206 based on the output generated using the fine-tuned chatbot 204. The session data 206 can include, for example, an instance of synthesized speech audio data in an implementation where the corresponding session is a voice-based session, and an instance of text data in an implementation where the corresponding session is a text-based session. In various implementations, and as Figure 2 depicted, the session data 206 can be transmitted to a given additional client device 191A such that the session data 206 is rendered audibly and / or visually at the given additional client device 191A. However, in other implementations, such as when the fine-tuned chatbot 204 participates in the corresponding session locally at the client device 110 (e.g., when the client device 110 is deployed in a shared setup), the session data 206 can be rendered audibly and / or visually at the client device 110.

[0051] Alternatively, during the inference phase and in an implementation where the chatbot corresponds to the chatbot 203 (e.g., instead of the fine-tuned chatbot 204), the session engine 184 can initially use the chatbot 203 to process the unstructured free-form natural language input 201, one or more features 202 determined based on processing the unstructured free-form natural language input 201, the state / transition data 205 (and optionally any response data 207 and / or session context data 208) to generate an output, such as a probability distribution over a sequence of words or phrases. The session engine 184 can generate session data 206 based on the output generated using the chatbot 203. The session data 206 can include, for example, an instance of synthesized speech audio data in an implementation where the corresponding session is a voice-based session, and an instance of text data in an implementation where the corresponding session is a text-based session. In various implementations, and as Figure 2As depicted, the conversation data 206 can be transmitted to a given additional client device 191A such that the conversation data 206 is rendered audibly and / or visually at the given additional client device 191A. However, in other implementations, such as when the chatbot 203 participates in the corresponding conversation locally at the client device 110 (e.g., when the client device 110 is deployed in a shared setting), the conversation data 206 can be rendered audibly and / or visually at the client device 110. In other words, the chatbot 203 can be initiated during the inference phase based on the unstructured free-form natural language input 201 and / or one or more features 202 determined based on processing the unstructured free-form natural language input 201, rather than fine-tuning the chatbot 203 during the training phase. This enables the client device 110 and / or the remote system 160 to save computing resources while still effectively deploying the chatbot to participate in the corresponding conversation.

[0052] In addition, during the inference phase, the responsive content engine 186 can cause the audio data and / or text data captured in the response data 207 to be processed (e.g., input parsing using an ASR model, an NLU model, and / or fulfillment rules) to determine that a given additional user has provided a response to an instance of the synthesized speech audio data and / or an instance of the text data. In other words, the responsive content engine 186 can determine responsive content 209 based on the corresponding conversation, such as the result of the corresponding conversation and / or a summary of the corresponding conversation. For example, the responsive content engine 186 can provide the responsive content 209 to the rendering engine 130 such that the client device provides the rendered responsive content 210 audibly and / or visually for presentation to the user. Thus, the implementations described herein enable a user to provide an unstructured free-form natural language input 201 such that a chatbot 203 or a fine-tuned chatbot 204 is generated based on the unstructured free-form natural language input 201 and the chatbot or fine-tuned chatbot is used to participate in the corresponding conversation.

[0053] As described in more detail herein (e.g., with respect to Figure 5C) The chatbot 203 and the fine-tuned chatbot 204 can have various peripheral behaviors, which can be implemented by the chatbot 203 or the fine-tuned chatbot 204 by utilizing the peripheral behavior engine 187. These peripheral behaviors can include, but are not limited to: a greeting behavior that enables the fine-tuned chatbot 204 to identify the user of the client device 110 and / or identify itself as a chatbot; a remote procedure call (RPC) behavior that enables the chatbot 203 or the fine-tuned chatbot 204 to search one or more databases during the corresponding session; a hold behavior that enables the chatbot 203 or the fine-tuned chatbot 204 to pause and resume the corresponding session; an exit behavior that enables the chatbot 203 or the fine-tuned chatbot 204 to prompt the user of the client device 110 to join the corresponding session and / or otherwise terminate the corresponding session when requested by a given additional user; a clarification behavior that enables the chatbot 203 or the fine-tuned chatbot 204 to clarify and / or repeat information previously provided during the corresponding session; and / or those other peripheral behaviors that can be invoked by the chatbot 203 or the fine-tuned chatbot 204 under corresponding conditions for invoking other peripheral behaviors.

[0054] Although described with respect to a telephone call between the chatbot 203 or the fine-tuned chatbot 204 (e.g., implemented locally at the client device 110 and / or remotely at the remote system 160) and a given additional user (e.g., accessible at a given additional client device 191A) during a corresponding session Figure 2 , it should be understood that this is not intended to be limiting. Instead, it should be understood that the techniques described herein can be used to fine-tune a chatbot that can be deployed to participate in voice-based and / or text-based sessions across multiple computing devices and / or at a single computing device.

[0055] Now turning to Figure 3 , a flowchart of an example method 300 is depicted that shows generating a chatbot locally at a client device and enabling the chatbot to participate in a corresponding session with an additional user. For convenience, the operations of method 300 are described with reference to the system performing the operations. This system of method 300 includes at least one processor, memory, and / or other components of a client device (e.g., Figure 1 's client device 110, Figure 6 's computing device 610, and / or other client devices). Additionally, although the operations of method 300 are shown in a specific order, this is not intended to be limiting. One or more operations can be reordered, omitted, and / or added.

[0056] At block 352, the system receives, on behalf of an entity, unstructured free-form natural language input from a user of a client device, the unstructured free-form natural language input including a natural language description of a corresponding dialogue state diagram. The unstructured free-form natural language input received from the user of the client device can include, for example, oral input received via a microphone of the client device, typed input received via a touch-sensitive display of the client device, and / or touch input received via the touch-sensitive display of the client device. Notably, the unstructured free-form natural language input is unstructured in the sense that the user is not required to provide the free-form natural language input according to any pattern or specific manner. However, the subsequent generation of the chatbot can be considered structured because the unstructured free-form natural language input implicitly and / or explicitly includes a natural language description of a corresponding dialogue state diagram. As described herein (e.g., with respect to Figure 5A and Figure 5B ), the corresponding dialogue state diagram can implicitly and / or explicitly include corresponding dialogue states and corresponding dialogue state transitions between the corresponding dialogue states as the corresponding dialogue state diagram.

[0057] At block 354, the system generates, at least based on the unstructured free-form natural language input and locally at the client device, a chatbot for the entity to participate in multiple corresponding conversations with additional users. In some implementations, and as indicated at block 354A, the system obtains a previously trained large language model (LLM) locally stored at the client device. Further, in these implementations, and as indicated at block 354B, the system causes the previously trained LLM locally stored at the client device to be fine-tuned based on the unstructured free-form natural language input to generate a fine-tuned LLM. Additionally, in these implementations, and as indicated at block 354C, the system utilizes the fine-tuned LLM as the chatbot. The system can generate the chatbot for the entity to participate in multiple corresponding conversations with multiple additional users in the same or similar manner as described above with respect to Figure 2 (e.g., in the implementation where the training phase is implemented locally at the client device and described with respect to block 280A). Notably, in these implementations, the system is being implemented locally at the client device, and thus, due to various hardware and / or software constraints of the client device, the previously trained LLM can be a sparsified version of the previously trained LLM that may otherwise be available (e.g., otherwise available to a remote system).

[0058] At block 356, the system causes the chatbot to participate on behalf of the entity in a given corresponding session with a given additional user among multiple corresponding sessions. In some implementations, and as indicated at block 356A, the system causes the chatbot to participate in the given corresponding session with the given additional user by: rendering multiple instances of synthesized speech audio data for presentation to the given additional user, and / or rendering multiple instances of text data for presentation to the given additional user. Further, in these implementations, and as indicated at block 356B, the system determines responsive content in response to one or more instances of the synthesized speech audio data and / or one or more instances of the text data. The system may cause the chatbot to participate in the given corresponding session with the given additional user on behalf of the user in the same or a similar manner as described above with respect to Figure 2 (e.g., in an implementation where the inference phase is implemented locally at the client device and described with respect to block 280B) and with respect to Figure 5A , Figure 5B and Figure 5C . Notably, in an implementation where the system implements the chatbot locally at the client device, a given corresponding session can occur between the chatbot via the client device and the given additional user via a given additional client device.

[0059] At block 358, the system causes the responsive content to be provided for presentation to the user of the client device. The responsive content can be determined based on, for example, one or more responses provided by the given additional user during the given corresponding session (e.g., as described with reference to Figure 5C ). The responsive content can include, for example, a summary of the given corresponding session, the result of the given corresponding session, and / or any other content that can be determined based on the chatbot's participation in the given corresponding session with the given additional user. However, it should be noted that the responsive content can depend on the verbal input provided by the given additional user during the given corresponding session.

[0060] At block 360, the system determines whether to cause the chatbot to participate in a given additional corresponding session with a given additional user among multiple corresponding sessions on behalf of the entity. The system can determine whether to cause the chatbot to participate in the given additional corresponding session with the given additional user based on, for example, whether there is an incoming electronic communication directed to the entity (e.g., an incoming voice-based communication such as an incoming phone call directed to the entity, an incoming text-based communication such as an incoming email message, etc.), whether the system is responsible for initiating an outgoing electronic communication on behalf of the entity (e.g., an outgoing voice-based communication such as an outgoing phone call, an outgoing text-based communication such as an outgoing email message or text message, etc.), whether the system is returning a previously missed or ignored incoming electronic communication, and / or based on other considerations.

[0061] At an iteration of block 360, if the system determines to cause the chatbot to participate in the given additional corresponding session with the given additional user on behalf of the entity, the system can return to block 356 and continue additional iterations of method 300 regarding the given additional corresponding session and the given additional user. If, at an iteration of block 360, the system determines not to cause the chatbot to participate in the given additional corresponding session with the given additional user on behalf of the entity, the system can continue to monitor whether to cause the chatbot to participate in the given additional corresponding session with the given additional user.

[0062] Although the previous trained LLM for which the chatbot is fine-tuned based on unstructured free-form natural language input (e.g., at block 354) is described Figure 3 in method 300, it should be understood that this is for illustrative purposes and not intended to be limiting. In additional or alternative implementations, the previous trained LLM can be used as the chatbot without any fine-tuning. In these implementations, and (e.g., at block 356) when causing the chatbot to participate in a given corresponding session with a given additional user, the system can initiate the previous trained LLM based on unstructured free-form natural language input. This enables the chatbot to participate in corresponding sessions without any explicit fine-tuning during the training phase.

[0063] Further, although the operation of block 360 is depicted as occurring after the system causes the chatbot to participate in a given corresponding session with a given additional entity (e.g., in a sequential manner), it should be understood that this is for illustrative purposes and not intended to be limiting. For example, as Figure 3As part of method 300, one or more iterations of the operations of block 360 can be executed continuously. This enables the system to enable the chatbot to participate in multiple conversations (e.g., in parallel) at any given time. In these implementations, the system can implement multiple instances of the chatbot or utilize a single instance of the chatbot. Further, although the system has been described as being implemented locally at the user's client device, it should be understood that this is for illustrative purposes and is not intended to be limiting. For example, and as described with respect to Figure 3 , it should be understood that this is for illustrative purposes and is not intended to be limiting. For example, and as described with respect to Figure 4 , the system can be implemented by a remote system that is remote from the user's client device that provides the unstructured free-form natural language input.

[0064] Turning now to Figure 4 , a flowchart of an example method 400 is depicted that shows remotely generating a chatbot at a remote system and enabling the chatbot to participate in a corresponding conversation with an additional user. For convenience, the operations of method 400 are described with reference to the system that performs the operations. The system of method 400 includes at least one processor, memory, and / or other components of a remote system (e.g., the remote system 160 of Figure 1 , the computing device 610 of Figure 6 , and / or other remote systems). Additionally, although the operations of method 400 are shown in a particular order, this is not intended to be limiting. One or more operations can be reordered, omitted, and / or added.

[0065] At block 452, the system receives an indication of unstructured free-form natural language input from a user of a client device on behalf of an entity, the indication of the unstructured free-form natural language input including a natural language description of a corresponding dialogue state diagram. The indication of the unstructured free-form natural language input received from the user of the client device can include, for example, an indication of oral input received via a microphone of the client device, an indication of typed input received via a touch-sensitive display of the client device, and / or an indication of touch input received via a touch-sensitive display of the client device. Also, the unstructured free-form natural language input is unstructured in the sense that the user does not need to provide the free-form natural language input according to any pattern or particular manner. However, the subsequent generation of the chatbot can be considered structured because the unstructured free-form natural language input implicitly and / or explicitly includes a natural language description of a corresponding dialogue state diagram.

[0066] At box 454, the system remotely generates a chatbot for representing the entity in multiple corresponding conversations with additional users based at least on the indication of the unstructured free-form natural language input and at a remote system (e.g., remote from the client device). In some implementations, and as indicated at box 454A, the system obtains a previously trained large language model (LLM) stored remotely at the remote system. Further, in these implementations, and as indicated at box 454B, the system causes the previously trained LLM stored remotely at the remote system to be fine-tuned based on the indication of the unstructured free-form natural language input to generate a fine-tuned LLM. Furthermore, in these implementations, and as indicated at box 454C, the system utilizes the fine-tuned LLM as the chatbot. The system may be used in the same manner as described above regarding Figure 2 The chatbot for representing an entity in a plurality of corresponding conversations with a plurality of additional users is generated in the same or similar manner as described (e.g., in implementations in which the training phase is remotely implemented at a remote system and described with respect to block 280A). Notably, in these implementations, the system is being implemented remotely at a remote system, and therefore, because there are few hardware and / or software constraints at the remote system, the previously trained LLM may be an unsparsed version of the previously trained LLM that is more robust than a sparsed version of the previously trained LLM.

[0067] At box 456, the system causes the chatbot to participate on behalf of the entity in a given corresponding session with a given additional user in the multiple corresponding sessions. In some implementations, and as indicated at box 456A, the system causes the chatbot to participate in the given corresponding session with the given additional user by rendering multiple instances of synthesized speech audio data for presentation to the given additional user, and / or rendering multiple instances of text data for presentation to the given additional user. Further, in these implementations, and as indicated at box 456B, the system determines responsive content in response to one or more instances of the instances of synthesized speech audio data and / or one or more instances of the instances of text data. The system may cause the chatbot to participate in the given corresponding session with the given additional user in a manner similar to that described above with respect to Figure 2 (e.g., in implementations where the inference phase is implemented locally at the client device and described with respect to block 280B) and with respect to Figure 5A , Figure 5B and Figure 5C The same or similar manner described represents the user's participation in a given corresponding conversation with a given additional user. It is worth noting that in an implementation in which the system remotely implements a chatbot at a remote system, a given corresponding conversation can be conducted between the chatbot via the remote system and a given additional user via a given additional client device.

[0068] At block 458, the system causes responsive content to be provided for presentation to a user of the client device. The responsive content can be determined based on, for example, one or more responses provided by a given additional user during a given corresponding session (e.g., as described with respect to Figure 5C ). The responsive content can include, for example, a summary of a given corresponding session, the results of a given corresponding session, and / or any other content that can be determined based on the chatbot's participation in a given corresponding session with a given additional user. However, it should be noted that the responsive content can depend on the verbal input provided by the given additional user during the given corresponding session.

[0069] At block 460, the system determines whether to cause the chatbot to represent the entity in a given additional corresponding session with a given additional one of the additional users in a plurality of corresponding sessions. The system can determine whether to cause the chatbot to represent the entity in a given additional corresponding session with a given additional one of the additional users based on, for example, whether there is an incoming electronic communication directed to the entity (e.g., an incoming voice-based communication such as an incoming phone call directed to the entity, an incoming text-based communication such as an incoming email message), whether the system is responsible for initiating an outgoing electronic communication on behalf of the entity (e.g., an outgoing voice-based communication such as an outgoing phone call, an outgoing text-based communication such as an outgoing email message or text message), whether the system is returning a previously missed or ignored incoming electronic communication, and / or based on other considerations.

[0070] At an iteration of block 460, if the system determines to cause the chatbot to represent the entity in a given additional corresponding session with a given additional one of the additional users, the system can return to block 456 and continue additional iterations of method 400 with respect to the given additional corresponding session and the given additional one of the additional users. At an iteration of block 460, if the system determines not to cause the chatbot to represent the entity in a given additional corresponding session with a given additional one of the additional users, the system can continue to monitor whether to cause the chatbot to represent the entity in a given additional corresponding session with a given additional one of the additional users.

[0071] Although also described with respect to a previously trained LLM where the chatbot is fine-tuned based on unstructured free-form natural language input (e.g., at block 454) Figure 4Method 400, but it should be understood that this is for illustrative purposes and is not intended to be limiting. In additional or alternative implementations, a previously trained LLM can be used as a chatbot without any fine-tuning. In these implementations, and (e.g., at block 456) when causing the chatbot to participate in a given corresponding session with a given additional user, the system can initiate the previously trained LLM based on an unstructured free-form natural language input. This enables the chatbot to participate in the corresponding session without any explicit fine-tuning during the training phase.

[0072] Further, although the operation of block 460 is depicted as occurring after the system causes the chatbot to participate in a given corresponding session with a given additional entity (e.g., in a sequential manner), it should be understood that this is for illustrative purposes and is not intended to be limiting. For example, as Figure 4 part of method 400, one or more iterations of the operation of block 460 can be performed continuously. This enables the system to cause the chatbot to participate in multiple sessions at any given moment (e.g., in a parallel manner). In these implementations, the system can implement multiple instances of the chatbot or utilize a single instance of the chatbot. Further, although described as being implemented locally at the user's client device Figure 4 , it should be understood that this is for illustrative purposes and is not intended to be limiting.

[0073] Now turning to Figure 5A , Figure 5B and Figure 5C , various non-limiting example interactions are depicted where corresponding unstructured free-form natural language inputs are used to generate corresponding chatbots, and where the corresponding chatbots participate in corresponding sessions with additional users based on the corresponding unstructured free-form natural language inputs. Notably, the interactions 500A, 500B, and 500C described with respect to Figure 5A , Figure 5B and Figure 5C can be implemented across multiple computing devices to enable the chatbot to participate in the corresponding session. For example, the corresponding unstructured free-form natural language inputs described in the examples with respect to Figure 5A and Figure 5B can be received at the user's client device (e.g., the client device 110 of Figure 1 ), the chatbots described in the examples with respect to Figure 5A , Figure 5B and Figure 5C can be generated at the user's client device (e.g., the client device 110 of Figure 1 ) and / or at a remote system (e.g., the remote system 160 from Figure 1 ), and the corresponding session described in the example with respect to Figure 5A, Figure 5B and Figure 5C The chatbots described by way of example can be implemented at the user's client device (e.g., Figure 1 client device 110) and / or at a remote system (e.g., from Figure 1 remote system 160) and communicate with additional users. Each of these computing devices can include corresponding components such as user interface input components (e.g., microphones, visual components, presence sensors, touch-sensitive displays, keyboards, hardware buttons, software buttons, etc.), user interface output components (e.g., touch-sensitive displays, speakers, monitors, projectors, etc.), network interfaces, and / or other components. Thus, although Figure 5A , Figure 5B and Figure 5C interactions 500A, 500B, and 500C are depicted as single interfaces respectively, it should be understood that this is for the purpose of illustrating the various techniques described herein and is not intended to be limiting.

[0074] Specifically referring to Figure 5A , assume that the user of the client device begins to provide a natural language description of the chatbot, as indicated by interaction 500A1. In various implementations, the corresponding session in which the natural language description of the chatbot is provided can be initiated as part of a conversation between the user of the client device and an automated assistant that is at least partially executed at the client device. The natural language description of the chatbot can be provided by the user of the client device as one or more inputs. As Figure 5AAs shown, natural language descriptions of a chatbot are provided across multiple unstructured free-form natural language inputs 552A1, 554A1, 556A1, 558A1, 560A1, and 562A1. The multiple unstructured free-form natural language inputs 552A1, 554A1, 556A1, 558A1, 560A1, and 562A1 can be provided as oral inputs, typed inputs, or imported from another software application (e.g., as a text-based document). For example, a user of a client device can invoke an automated assistant that is at least partially executed at the client device (e.g., by actuating a software or hardware button, by saying a specific term or phrase such as "Assistant", "Hey Assistant", etc., and / or by other means), and provide the multiple unstructured free-form natural language inputs 552A1, 554A1, 556A1, 558A1, 560A1, and 562A1 as oral inputs. As another example, a user of a client device can access an automated assistant application that is accessible at the client device and associated with the automated assistant, and provide the multiple unstructured free-form natural language inputs 552A1, 554A1, 556A1, 558A1, 560A1, and 562A1 as typed inputs. As yet another example, a user of a client device can use a separate text processing software application to generate a natural language description of the chatbot described by the multiple unstructured free-form natural language inputs 552A1, 554A1, 556A1, 558A1, 560A1, and 562A1, and can import a text-based document from the separate text processing software application. Although the natural language description is shown in Figure 5A as being provided across the multiple unstructured free-form natural language inputs 552A1, 554A1, 556A1, 558A1, 560A1, and 562A1, it should be understood that this is for illustrative purposes only and is not intended to be limiting. For example, in other examples, the natural language description of the chatbot can be provided as a single instance of an unstructured free-form natural language input.

[0075] For Figure 5AFor the purposes of the example in, assume that the user of the client device is associated with the entity "John’s Hardware Store" (a fictional hardware store), and assume that the user of the client device provides a natural language description of the chatbot by implicitly defining the corresponding dialogue state diagram to be utilized by the chatbot when participating in the corresponding conversation on behalf of "John’s Hardware Store". Further assume that the user of the client device provides the unstructured free-form natural language input 552A1 of "Start off by greeting the customer, thanking them for calling, and letting them know you are a chatbot capable of handling conversations such that thy can you any question at any time and don’t have to wait for you to provide any particular options" as an oral input. In this example, the automated assistant can use an ASR model to process the audio data capturing the oral input to generate an ASR output, such as the recognized text corresponding to the oral input. Further, the automated assistant can use an NLU model to process the ASR output to generate an NLU output, such as an intent, slot values of parameters associated with the intent, and / or other NLU outputs. Based on the ASR output and / or NLU output generated by processing the unstructured free-form natural language input 552A1, the automated assistant can determine that the user of the client device is implicitly defining a "greeting" dialogue state that enables the chatbot to introduce itself when an additional user calls "John's Hardware Store", and differentiates itself from an IVR system by indicating to the additional user that "they can ask any question at any time" and that they "don’t have to wait for [the chatbot] to provide any particular options" (like a regular interactive voice response (IVR) system).Notably, the user of the client device is implicitly defining a “hello” dialog state, where the natural language description included in the unstructured free-form natural language input 552A1 defines the “hello” dialog state (e.g., as about. Figure 5B described).

[0076] Assume further that the user of the client device provides an unstructured free-form natural language input 554A1 of “If the customer asks any particular question, go ahead and answer the question to assist the customer.” However, if the customer does not ask any particular question, go ahead and provide the physical location of the store and the store's business hours, and then ask “whether they would like to place an order” as verbal input. Based on the ASR output and / or NLU output generated based on processing the unstructured free-form natural language input 554A1, the automated assistant can determine that the user of the client device is implicitly defining a "store location" dialog state, which enables the chatbot to provide the actual location of the store (e.g., where the actual location of the store is determined from one or more databases accessible to the automated assistant or provided by the user); a "store hours" dialog state, which enables the chatbot to provide the store's operating hours (e.g., where the store's operating hours are determined from one or more databases accessible to the automated assistant or provided by the user); and a "new order" dialog state, which enables the chatbot to place an order over the phone and on behalf of additional users participating in the corresponding session with the chatbot. Notably, in providing unstructured free-form natural language input 554A1, the automated assistant can determine that the user of the client device is implicitly defining dialog state transitions that enable the chatbot to transition from a "Greetings" dialog state to a "Store Location" dialog state, to a "Store Hours" dialog state, and to a "New Hours" dialog state without the additional user otherwise asking any questions.

[0077] Further assume that the user of the client device provides the unstructured free-form natural language input 556A1, "The physical location of the store is 123 Main St, we are open 9 AM to 5 PM Monday through Friday, and you have access to our inventory to know what the customer can and cannot order" as an oral input. Based on the ASR output and / or NLU output generated from processing the unstructured free-form natural language input 556A1, the automated assistant can determine that the user of the client device has provided the physical location of the store, the business hours of the store, and has indicated that the chatbot can utilize an RPC peripheral behavior indicating that the chatbot can make a remote procedure call (RPC) to the inventory system of "John’s Hardware Store". Although the slot values for the parameters associated with the various conversation states are described in terms of the user explicitly providing Figure 5A , it should be understood that this is for illustrative purposes and is not intended to be limiting. In other examples, and as described above, the physical location of the store, the business hours of the store, and / or other slot values for the parameters associated with the various conversation states can be determined from one or more databases accessible to the automated assistant.

[0078] Further assume that the user of the client device provides the unstructured free-form natural language input 558A1 of "If the customer wants to order something, but we do not have inventory, apologize, and tell them that ‘we replenish our inventory every Tuesday morning, please call back Tuesday afternoon’" as an oral input, and also provides the unstructured free-form natural language input 560A1 of "If the customer successfully places an order, confirm the order, get the payment information for the order, and send the user a confirmation email, or let them pay at the store" as an oral input. Based on the ASR output and / or NLU output generated from processing the unstructured free-form natural language input 558A1, the automated assistant can determine that the user of the client device has provided information on how to handle any unsuccessful orders in the "new order" conversation state, where the user provides the explicit output to be provided by the chatbot in these scenarios (e.g., by providing "we replenish our inventory every Tuesday morning, please call back Tuesday afternoon"). Further, based on the ASR output and / or NLU output generated from processing the unstructured free-form natural language input 560A1, the automated assistant can determine that the user of the client device has implicitly defined an "order confirmation" conversation state that enables the chatbot to accept orders for available inventory and optionally enables additional users participating in the corresponding conversation with the chatbot to pay for the order or wait until they arrive at the store to pick up the order and then pay for the order.

[0079] Further assume that a user of the client device provides an unstructured free-form natural language input 562A1 of "End by thanking the customer for their time and business" as an oral input. Based on the ASR output and / or NLU output generated from processing the unstructured free-form natural language input 562A1, the automated assistant can determine that the user of the client device is implicitly defining an "end" conversation state that enables the chatbot to end the corresponding session with an additional user. Accordingly, the automated assistant can provide a response 564A1 of "Okay, I’ll use this information in handling future conversations" to be presented audibly and / or visually to the user of the client device to indicate that the automated assistant will generate the chatbot described by the plurality of unstructured free-form natural language inputs 552A1, 554A1, 556A1, 558A1, 560A1, and 562A1 and use the chatbot to participate in the corresponding session with the additional user.

[0080] For example, the automated assistant can cause the client device and / or the remote system to generate a chatbot for participating in the corresponding session with the additional user. In some implementations, the chatbot can correspond to a previously trained LLM that is fine-tuned based on the plurality of unstructured free-form natural language inputs 552A1, 554A1, 556A1, 558A1, 560A1, and 562A1 using various fine-tuning techniques (e.g., as described with respect to Figure 2 ). In other implementations, the chatbot can correspond to a previously trained LLM that is not fine-tuned based on the plurality of unstructured free-form natural language inputs 552A1, 554A1, 556A1, 558A1, 560A1, and 562A1 but is initiated based on one or more of the unstructured free-form natural language inputs 552A1, 554A1, 556A1, 558A1, 560A1, and 562A1. Accordingly, the automated assistant can cause the chatbot to be implemented at a computing device, such as the client device in an implementation where the chatbot is locally generated at the client device or the remote system in an implementation where the chatbot is generated away from the client device. Described with respect to Figure 5C is how the automated assistant causes based on Figure 5AA non - limiting example of a chatbot that generates natural language descriptions. Although the corresponding dialogue state diagram (e.g., corresponding dialogue states and corresponding dialogue state transitions) for the chatbot is implicitly defined by the user of the client device in the description of interaction 500A, it should be understood that this is for illustrative purposes.

[0081] For example, and referring to Figure 5B interaction 500B, the user of the client device can alternatively define the same corresponding dialogue state diagram in an explicit manner, but still utilize unstructured free - form natural language input. For example, interaction 500B can be implemented at an interface that enables the user of the client device to explicitly define the corresponding dialogue states and corresponding dialogue state transitions. As Figure 5B shown, dialogue state 552B1 corresponds to the "greeting" dialogue state implicitly defined by the unstructured free - form natural language input 552A1 from Figure 5A and includes the same unstructured free - form natural language input as the unstructured free - form natural language input 552A1 from Figure 5A . Further, dialogue state 554B1 corresponds to the "store location" dialogue state, dialogue state 556B1 corresponds to the "store business hours" dialogue state, and dialogue state 558B1 corresponds to the "new order" dialogue state (e.g., and the ability to perform RPC in this dialogue state, as indicated by [*CHECKINVENTORY*]), and these dialogue states are implicitly defined by the unstructured free - form natural language input 554A1 from Figure 5A . It is worth noting that dialogue state 554B1 and dialogue state 556B1 include the same unstructured free - form natural language input as the unstructured free - form natural language input 556A1 from Figure 5A , and dialogue state 558B1 includes the same unstructured free - form natural language input as the unstructured free - form natural language inputs 556A1 and 558A1 from Figure 5A . In addition, dialogue state 560B1 corresponds to the "order confirmation" dialogue state implicitly defined by the unstructured free - form natural language input 560A1 from Figure 5A and includes the same unstructured free - form natural language input as the unstructured free - form natural language input 560A1 from Figure 5A . Finally, dialogue state 562B1 corresponds to the "end" dialogue state implicitly defined by the unstructured free - form natural language input 562A1 from Figure 5A and includes the same unstructured free - form natural language input as the unstructured free - form natural language input 562A1 from Figure 5AThe unstructured free-form natural language input 562A1 is the same unstructured free-form natural language input. It is worth noting that the arrows between different dialogue states indicate the corresponding dialogue state transitions, and can also be explicitly defined by the user in the example of Figure 5B .

[0082] Therefore, the techniques described herein not only enable users to implicitly define the corresponding dialogue states and describe the dialogue state transitions, but also enable users to explicitly define the corresponding dialogue states and describe the dialogue state transitions. In various implementations, Figure 5A The explicit dialogue graph of can be generated (e.g., by utilizing Figure 5B and Figure 5B 's dialogue state / transition recognition engine 183) based on multiple unstructured free-form natural language inputs 552A1, 554A1, 556A1, 558A1, 560A1, and 562A1 from Figure 1 and Figure 2 . Although chatbots are described as being generated in different ways (e.g., implicitly generated in Figure 5A and explicitly generated in Figure 5A ), chatbots can participate in the corresponding conversations with additional users in the same or similar ways. Figure 5B

[0083] For example, and with specific reference to Figure 5C , assume that a user of an additional client device (e.g., "Jane Doe") makes a phone call to "John’s Hardware Store", and the chatbot answers the phone call on behalf of "John’s Hardware Store", as shown in interaction 500C1. In this example, the automated assistant can cause the chatbot to generate an instance of synthetic speech audio data that captures the synthetic speech 552C1 of "Hi, thanks for calling John’s Hardware Store, I’m a virtual assistant and can handle any of your questions or other needs". Further, the automated assistant can cause an instance of the synthetic speech audio data that captures the synthetic speech 552C1 to be transmitted to the additional client device of the additional user, so that the instance of the synthetic speech audio data can be audibly rendered for presentation to the additional user.

[0084] In this example, and when generating the synthetic speech 552C1, the automated assistant can process theFigure 5A and Figure 5B data associated with the "greeting" conversation state to generate an instance of synthetic speech audio data. The data can include, for example, state / transition data associated with the "greeting" conversation state, session context data, and / or other data. Further, when generating an instance of synthetic speech audio data, the automated assistant can cause the data associated with the "greeting" conversation state to be applied as input to a previously trained LLM, which is fine-tuned and / or initialized based on interactions 500A1 from Figure 5A and / or interactions 500B1 from Figure 5B to generate an output such as a probability distribution over a vocabulary of terms and / or phrases. Based on the probability distribution over the vocabulary of terms and / or phrases, the automated assistant can cause the chatbot to select text data corresponding to the synthetic speech 552C1. Further, the automated assistant can cause the chatbot to use a TTS model to process the text data corresponding to the synthetic speech 552C1 to generate an instance of synthetic speech audio data that is audibly rendered at an additional client device of an additional user. Additionally, the automated assistant can cause the chatbot to generate an output and / or select text data corresponding to the synthetic speech 552C1 at least in part because the previously trained LLM is fine-tuned and / or initialized based on unstructured free-form natural language input provided by the user during interactions 500A1 from Figure 5A and / or interactions 500B1 from Figure 5B

[0085] Further assume that Jane Doe responds to the synthetic speech 552C1 with the verbal input 554C1 of "Hi, this is Jane Doe, I was wondering if you had any 50 Watt lightbulbs in stock?". In this example, the automated assistant can cause the chatbot to use an ASR model to process the audio data capturing the verbal input 554C1 to generate an ASR output. Further, the automated assistant can cause the chatbot to use an NLU model to process the ASR output to generate an NLU output. Additionally, the automated assistant can cause the chatbot to use a fulfillment model and / or fulfillment rules to process the NLU output to generate a fulfillment output. In this way, the automated assistant can cause the chatbot to determine that Jane Doe is requesting an inventory check for 50 Watt lightbulbs and thus cause the chatbot to execute an RPC of an inventory system to determine the available inventory of 50 Watt lightbulbs.

[0086] ​Accordingly, and assuming that 50-watt light bulbs are available, the automated assistant can cause the chatbot to generate additional instances of synthetic speech audio data based on the results of an RPC to an inventory system, where the additional instances capture the synthetic speech 556C1 of "Let me check", followed by an RPC to the inventory system, followed by "Yes, we have plenty in stock". Further, the automated assistant can cause the additional instances of synthetic speech audio data that capture the synthetic speech 556C1 to be transmitted to an additional client device of an additional user, such that the additional instances of synthetic speech audio data can be audibly rendered for presentation to the additional user. In this example, and when generating the synthetic speech 556C1, the automated assistant can process data associated with the "new order" conversation state from Figure 5A and Figure 5B to generate additional instances of synthetic speech audio data. The data can include, for example, state / transition data, session context data, inventory availability data from the RPC to the inventory system, and / or other data associated with the "new order" conversation state in the same or similar manner as described for the synthetic speech 552C1.

[0087] Further assume that Jane Doe responds to the synthetic speech 556C1 with the verbal input 558C1 of "Great, I would like to purchase 10 of them". In this example, the automated assistant can cause the chatbot to use an ASR model to process the audio data that captures the verbal input 558C1 to generate an ASR output. Further, the automated assistant can cause the chatbot to use an NLU model to process the ASR output to generate an NLU output. Additionally, the automated assistant can cause the chatbot to use a fulfillment model and / or fulfillment rules to process the NLU output to generate a fulfillment output. In this way, the automated assistant can cause the chatbot to determine that Jane Doe wants to purchase 10 50-watt light bulbs.

[0088] Accordingly, the automated assistant can cause the chatbot to generate further additional instances of synthetic speech audio data that capture the synthetic speech 560C1 of "Excellent, they are $3.99 each for a total of $42.29 after tax, would you like to pay now or when you get to the store?". Further, the automated assistant can cause further additional instances of the synthetic speech audio data that capture the synthetic speech 560C1 to be transmitted to an additional client device of an additional user such that the further additional instances of the synthetic speech audio data can be aurally rendered for presentation to the additional user. In this example, and when generating the synthetic speech 560C1, the automated assistant can process additional data associated with the "new order" conversation state from Figure 5A and Figure 5B to generate further additional instances of the synthetic speech audio data. The data can include, for example, state / transition data, session context data, inventory price data from an RPC to an inventory system, and / or other data associated with the "new order" conversation state in the same or a similar manner as described for the synthetic speech 552C1.

[0089] Further assume that Jane Doe responds to the synthetic speech 560C1 with the verbal input 562C1 of "When I get there". In this example, the automated assistant can cause the chatbot to use an ASR model to process the audio data that captures the verbal input 562C1 to generate an ASR output. Further, the automated assistant can cause the chatbot to use an NLU model to process the ASR output to generate an NLU output. Additionally, the automated assistant can cause the chatbot to use a fulfillment model and / or fulfillment rules to process the NLU output to generate a fulfillment output. In this way, the automated assistant can cause the chatbot to determine that Jane Doe wants to pay for the light bulbs after arriving at the store rather than during the phone call.

[0090] Accordingly, the automated assistant can enable yet further additional instances of the chatbot to generate synthetic speech audio data, the yet further additional instances capturing the synthetic speech 564C1 of "That works for us, you can pick them up at 123 Main St any Monday - Friday between 9 AM and 5 PM". Further, the automated assistant can enable yet further additional instances of the synthetic speech audio data capturing the synthetic speech 564C1 to be transmitted to an additional client device of an additional user, such that yet further additional instances of the synthetic speech audio data can be aurally rendered for presentation to the additional user. In this example, and when generating the synthetic speech 564C1, the automated assistant can process additional data associated with the "store location" and "store business hours" dialogue states from Figure 5A and Figure 5B to generate yet further additional instances of the synthetic speech audio data. The data can include, for example, status / transition data, session context data, and / or other data associated with the "store location" dialogue state and the "store business hours" dialogue state in the same or a similar manner as described for the synthetic speech 552C1.

[0091] Further assume that Jane Doe uses the verbal input 566C1 of "Thanks, see you this afternoon" to respond to the synthetic speech 564C1. In this example, the automated assistant can enable the chatbot to use an ASR model to process the audio data capturing the verbal input 566C1 to generate an ASR output. Further, the automated assistant can enable the chatbot to use an NLU model to process the ASR output to generate an NLU output. Additionally, the automated assistant can enable the chatbot to use a fulfillment model and / or fulfillment rules to process the NLU output to generate a fulfillment output. In this way, the automated assistant can enable the chatbot to determine that Jane Doe plans to pick up 10 50 - watt light bulbs at the store on the same afternoon.

[0092] Accordingly, the automated assistant can enable yet a further additional instance of the chatbot to generate synthetic speech audio data that captures "We appreciate your business, see you shortly". Further, the automated assistant can enable yet a further additional instance of the synthetic speech audio data that captures synthetic speech 564C1 to be transmitted to an additional client device of an additional user such that yet a further additional instance of the synthetic speech audio data can be aurally rendered for presentation to the additional user. In this example, and when generating synthetic speech 568C1, the automated assistant can process additional data associated with an "end" conversation state from Figure 5A and Figure 5B to generate yet a further additional instance of the synthetic speech audio data. The data can include, for example, state / transition data, session context data, and / or other data associated with the "end" conversation state in the same or similar manner as described with respect to synthetic speech 552C1.

[0093] Notably, in the interaction 500C1 depicted in Figure 5C the additional user is able to drive the corresponding conversation by providing various verbal inputs, and the chatbot is able to respond to the additional user and participate in the corresponding conversation by leveraging natural language descriptions provided by the user in Figure 5A and / or Figure 5B . Further, and as shown in interaction 500C2, the automated assistant can determine responsive content 552C2 that can be provided for presentation to the user. The responsive content 552C2 can include the result of the corresponding conversation between Jane Doe and the chatbot, such as "Jane Doe will be here this afternoon to pick up 10, 50 Watt lightbulbs – she will pay when she gets here". Accordingly, interaction 500C2 can be a notification that is generated for presentation to the user, or provided for presentation to the user during a subsequent conversation session between the user and the automated assistant that is at least partially executed at the client device.

[0094] Although the corresponding conversation has been described as a phone call in Figure 5C, but it should be understood that this is for illustrative purposes and is not intended to be limiting. For example, the corresponding session can be a text-based session via any text-based platform or service (e.g., text or SMS messaging, email, and / or other text-based platforms), and the chatbot can participate in the corresponding session via the text-based platform or service.

[0095] Turning now to Figure 6 , Figure 6 FIG. 610 is a block diagram of an example computing device 610 that can optionally be utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client device, remote system components, and / or other components can include one or more components of the example computing device 610.

[0096] The computing device 610 generally includes at least one processor 614 that communicates with a plurality of peripheral devices via a bus subsystem 612. These peripheral devices can include a storage subsystem 624 (which includes, for example, a memory subsystem 625 and a file storage subsystem 626), a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices allow user interaction with the computing device 610. The network interface subsystem 616 provides an interface to an external network and is coupled to a corresponding interface device in other computing devices.

[0097] The user interface input device 622 can include a keyboard, a pointing device (such as a mouse, trackball, touchpad, or graphics tablet), a scanner, a touchscreen incorporated into a display (e.g., a touch-sensitive display), an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computing device 610 or onto a communication network.

[0098] The user interface output device 620 can include a display subsystem, a printer, a fax machine, or a non-visual display, such as an audio output device. The display subsystem can include a cathode ray tube (CRT), a flat panel device (such as a liquid crystal display (LCD)), a projection device, or some other mechanism for creating a visible image. The display subsystem can also provide a non-visual display, such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways of outputting information from the computing device 610 to a user or to another machine or computing device.

[0099] The storage subsystem 624 stores the programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 may include aspects for performing selected aspects of the methods disclosed herein and logic for implementing Figure 1 and Figure 2 the various components depicted.

[0100] These software modules are typically executed by the processor 614 alone or in conjunction with other processors. The memory 625 used in the storage subsystem 624 may include multiple memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical disk drive, or a removable media cartridge. Modules implementing the functionality of certain implementations may be stored by the file storage subsystem 626 in the storage subsystem 624 or in other machines accessible to the processor 614.

[0101] The bus subsystem 612 provides the mechanism for enabling the various components and subsystems of the computing device 610 to communicate with each other as intended. Although the bus subsystem 612 is schematically shown as a single bus, alternative implementations of the bus subsystem 612 may use multiple buses.

[0102] The computing device 610 may have different types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 610 depicted in Figure 6 is only intended as a specific example for illustrating some implementations. Many other configurations of the computing device 610 are possible, which have more or fewer components compared to the computing device depicted in Figure 6

[0103] In cases where the systems described herein collect or otherwise monitor personal information about a user or information that can be used with personal information and / or monitored information, the user may be given the opportunity to control whether the program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographical location), or to control whether and / or how to receive content from a content server that may be more relevant to the user. Additionally, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, a user's identity may be processed so that the user's personally identifiable information cannot be determined, or in the case of obtaining geographical location information, the user's geographical location may be generalized (such as to the city, zip code, or state level) so that the user's specific geographical location cannot be determined. Thus, the user can control how information about the user is collected and / or used.

[0104] In some implementations, a method implemented by one or more processors of a client device is provided, and the method includes: receiving, at the client device on behalf of an entity, unstructured free-form natural language input from a user of the client device, the unstructured free-form natural language input including a natural language description of a corresponding dialogue state diagram, the natural language description of the corresponding dialogue state diagram defining implicit dialogue states and implicit dialogue state transitions of the corresponding dialogue state diagram without defining any explicit dialogue states or any explicit dialogue state transitions of the corresponding dialogue state diagram; in response to receiving the unstructured free-form natural language input including the natural language description of the corresponding dialogue state diagram: generating, based on the unstructured free-form natural language input, a chatbot for participating on behalf of the entity in a plurality of corresponding conversations with additional users; and causing the chatbot to participate on behalf of the entity in the plurality of corresponding conversations with the additional users. Causing the chatbot to participate on behalf of the entity in a given corresponding conversation with a given additional user among the additional users in the plurality of corresponding conversations includes: causing the chatbot to render a plurality of instances of synthetic speech audio data for presentation to the given additional user; and causing responsive content determined in response to rendering one or more of the plurality of instances of synthetic speech for presentation to the given additional user to be provided for presentation to the user of the client device. One or more of the plurality of instances of synthetic speech are generated based on the implicit dialogue states and / or the implicit dialogue state transitions.

[0105] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0106] In some implementations, generating a chatbot for representing an entity to participate in multiple corresponding sessions with additional users may include: obtaining a previously trained large language model (LLM); causing the previously trained LLM to be fine-tuned based on unstructured free-form natural language input to generate a fine-tuned LLM; and using the fine-tuned LLM as the chatbot.

[0107] In some versions of those implementations, causing the chatbot to render a given instance of synthetic speech from among multiple instances of synthetic speech for presentation to a given additional user may include: using the fine-tuned LLM to process one or more features associated with a given implicit dialogue state in an implicit dialogue state to generate an instance of text data reflecting a given behavior of the given implicit dialogue state; using a text-to-speech (TTS) model to process the given instance of text data reflecting the given behavior of the given implicit dialogue state to generate the given instance of synthetic speech; and transmitting the given instance of synthetic speech from a client device and to an additional client device of the given additional user. Transmitting the given instance of synthetic speech to the additional client device may cause the additional client device to audibly render the given instance of synthetic speech for presentation to the given additional user via one or more speakers of the additional client device.

[0108] In some further versions of those implementations, the method may further include using the fine-tuned LLM and one or more of the features associated with the given implicit dialogue state to process the corresponding context of the corresponding session to generate a given instance of text data reflecting a given behavior of the given implicit dialogue state.

[0109] In additional or alternative further versions of those implementations, the method may further include in response to the given instance of synthetic speech being audibly rendered for presentation to the given additional user via one or more speakers of the additional client device: receiving at the client device and from the additional client device a given instance of response audio data including responsive content in response to at least the given instance of synthetic speech; using an automatic speech recognition (ASR) model to process the given instance of response audio data to generate a given instance of response text data; and determining based on the given instance of response text data whether to: use the fine-tuned LLM to process one or more features associated with the given implicit dialogue state and the response text data to generate an additional instance of text data reflecting a given behavior of the given implicit dialogue state; or use the fine-tuned LLM to process one or more additional features associated with a given additional implicit dialogue state in the implicit dialogue state and the response text data to generate an additional instance of text data reflecting a given additional behavior of the given additional implicit dialogue state.

[0110] In further versions of those implementations, making responsive content available for presentation to a user of a client device may include the results of a corresponding session.

[0111] In additional or alternative further versions of those implementations, the method may further include, before using the fine-tuned LLM to process one or more of the features associated with a given implicit dialogue state: extracting one or more of the features from an unstructured free-form natural language input.

[0112] In yet further versions of those implementations, one or more of the features may be explicitly included in the unstructured free-form natural language input, and extracting one or more of the features that are explicitly included in the unstructured free-form natural language input from the unstructured free-form natural language input may include using an input parser to extract one or more of the features that are explicitly included in the unstructured free-form natural language input.

[0113] In yet further versions of those implementations, the method may further include causing the fine-tuned LLM to use one or more of the features to generate an instance of text data representing a given behavior reflecting a given implicit dialogue state.

[0114] In additional or alternative yet further versions of those implementations, one or more of the features may be implicitly included in the unstructured free-form natural language input, and extracting one or more of the features that are implicitly included in the unstructured free-form natural language input from the unstructured free-form natural language input may include using an input parser to identify one or more of the features that are implicitly included in the unstructured free-form natural language input; and using a coreference resolver to extract one or more of the features that are implicitly included in the unstructured free-form natural language input.

[0115] In yet further versions of those implementations, the method may further include causing the fine-tuned LLM to use one or more of the features to generate an instance of text data representing a given behavior reflecting a given implicit dialogue state.

[0116] In additional or alternative versions of those implementations, a previously trained LLM may be stored in the device storage of the client device, and the previously trained LLM stored in the device storage of the client device may be a sparsified version of a global previously trained LLM available at a remote system communicatively coupled to the client device.

[0117] In some further versions of those implementations, the fine-tuned LLM can be stored in the device storage of the client device.

[0118] In some implementations, causing the chatbot to participate in a given corresponding session with a given additional user can include causing the chatbot to answer a phone call received at the client device and from the given additional entity; and causing the chatbot to participate in a corresponding session with the entity as part of the phone call.

[0119] In some implementations, the user may not be an active participant in the corresponding session between the chatbot and the given additional user.

[0120] In some implementations, a method implemented by one or more processors of a remote system is provided, and the method includes: receiving, at the remote system and from the client device on behalf of an entity, an indication of an unstructured free-form natural language input of a user from the client device, the indication of the unstructured free-form natural language input including a natural language description of a corresponding dialogue state diagram, the indication of the natural language description of the corresponding dialogue state diagram defining an implicit dialogue state and implicit dialogue state transitions of the corresponding dialogue state diagram and not defining any explicit dialogue state or any explicit dialogue state transitions of the corresponding dialogue state diagram; in response to receiving the indication of the unstructured free-form natural language input including the natural language description of the corresponding dialogue state diagram: generating, based on the indication of the unstructured free-form natural language input, a chatbot for representing the entity to participate in multiple corresponding sessions with an additional user; and causing the chatbot to represent the entity to participate in multiple corresponding sessions with the additional user. Causing the chatbot to represent the entity to participate in a given corresponding session with a given additional user among the multiple corresponding sessions includes: causing the chatbot to render multiple instances of synthesized speech for presentation to the given additional user; and transmitting, from the remote system and to the client device, an indication of responsive content. Generating one or more of the multiple instances of synthesized speech based on the implicit dialogue state and / or the implicit dialogue state transitions. Further, transmitting the indication of the responsive content to the client device causes the client device to provide the responsive content for presentation to the user of the client device.

[0121] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0122] In some implementations, generating a chatbot for representing an entity to participate in multiple corresponding sessions with an additional user can include: obtaining a previously trained large language model (LLM); causing the previously trained LLM to be fine-tuned based on the unstructured free-form natural language input to generate a fine-tuned LLM; and using the fine-tuned LLM as the chatbot.

[0123] In some versions of those implementations, a previously trained LLM can be stored in a remote storage of a remote system, and the previously trained LLM stored in the remote storage of the remote system is an unsparsified version of a globally previously trained LLM that is available at a remote system communicatively coupled to the client device.

[0124] In some further versions of those implementations, a fine-tuned LLM can be stored in a remote storage of a remote system.

[0125] In some implementations, a method implemented by one or more processors of a client device is provided, and the method includes: receiving, at the client device on behalf of an entity, unstructured free-form natural language input from a user of the client device, the unstructured free-form natural language input including a natural language description of a corresponding dialogue state graph, the natural language description of the corresponding dialogue state graph defining an implicit dialogue state and implicit dialogue state transitions of the corresponding dialogue state graph without defining any explicit dialogue state or any explicit dialogue state transitions of the corresponding dialogue state graph; in response to receiving the unstructured free-form natural language input including the natural language description of the corresponding dialogue state graph: generating, based on the unstructured free-form natural language input, a chatbot for participating on behalf of the entity in a plurality of corresponding conversations with additional users; and causing the chatbot to participate on behalf of the entity in the plurality of corresponding conversations with the additional users. Causing the chatbot to participate on behalf of the entity in a given corresponding conversation with a given additional user among the additional users in the plurality of corresponding conversations includes: causing the chatbot to render a plurality of instances of text data for presentation to the given additional user; and causing responsive content determined in response to rendering one or more of the plurality of instances of text data for presentation to the given additional user to be provided for presentation to the user of the client device. Generating one or more of the plurality of instances of text data based on the implicit dialogue state and / or the implicit dialogue state transitions.

[0126] In some implementations, a method implemented by one or more processors of a client device is provided, and the method includes: receiving, at a remote system and on behalf of an entity from the client device, an indication of an unstructured free-form natural language input from a user of the client device, the indication of the unstructured free-form natural language input including an indication of a natural language description of a corresponding dialogue state graph, the indication of the natural language description of the corresponding dialogue state graph defining an implicit dialogue state and implicit dialogue state transitions of the corresponding dialogue state graph without defining any explicit dialogue state or any explicit dialogue state transitions of the corresponding dialogue state graph; in response to receiving the indication of the unstructured free-form natural language input including the indication of the natural language description of the corresponding dialogue state graph: generating, based on the indication of the unstructured free-form natural language input, a chatbot for the entity to participate in multiple corresponding conversations with additional users; and causing the chatbot to participate in multiple corresponding conversations with additional users on behalf of the entity. Causing the chatbot to participate in a given corresponding conversation with a given additional user among the multiple corresponding conversations includes: causing the chatbot to render multiple instances of text data for presentation to the given additional user; and transmitting, from the remote system and to the client device, an indication of responsive content. One or more instances of the multiple instances of text data are generated based on the implicit dialogue state and / or the implicit dialogue state transitions. Further, transmitting the indication of the responsive content to the client device causes the client device to provide the responsive content for presentation to a user of the client device.

[0127] In some implementations, a method implemented by one or more processors of a client device is provided, and the method includes: receiving, at the client device, a corresponding dialogue state diagram from a user of the client device, the corresponding dialogue state diagram including an explicit dialogue state and an explicit dialogue state transition; at the client device and for each explicit dialogue state in the explicit dialogue states, a representative entity receiving a corresponding unstructured free-form natural language input from the user, the corresponding unstructured free-form natural language input including a natural language description for each explicit dialogue state and explicit dialogue state transition in the explicit dialogue states; in response to receiving the unstructured free-form natural language input including the natural language description for each explicit dialogue state and explicit dialogue state transition in the explicit dialogue states: generating, based on the unstructured free-form natural language input, a chatbot for the representative entity to participate in a plurality of corresponding conversations with additional users; and causing the chatbot to represent the entity to participate in the plurality of corresponding conversations with the additional users. Causing the chatbot to represent the entity to participate in a given corresponding conversation with a given additional user among the plurality of corresponding conversations includes: causing the chatbot to render a plurality of instances of synthesized speech for presentation to the given additional user; and causing responsive content determined in response to one or more of the plurality of instances of synthesized speech for presentation to the given additional user to be provided for presentation to the user of the client device. One or more of the plurality of instances of synthesized speech are generated based on the natural language description for each explicit dialogue state and explicit dialogue state transition in the explicit dialogue states.

[0128] In some implementations, a method implemented by one or more processors of a remote system is provided, and the method includes: receiving, at the remote system and from a client device, an indication of a corresponding dialogue state diagram of a user of the client device, the indication of the corresponding dialogue state diagram including an explicit dialogue state and an explicit dialogue state transition; receiving, at the remote system and from the client device and for each explicit dialogue state of the explicit dialogue states, an indication of a corresponding unstructured free-form natural language input from the user, the indication of the corresponding unstructured free-form natural language input including a natural language description for each explicit dialogue state and explicit dialogue state transition of the explicit dialogue states; in response to receiving the indication of the unstructured free-form natural language input including the natural language description for each explicit dialogue state and explicit dialogue state transition: generating, based on the indication of the unstructured free-form natural language input, a chatbot for a representative entity to participate in a plurality of corresponding conversations with additional users; and causing the chatbot to represent the entity to participate in the plurality of corresponding conversations with the additional users. Causing the chatbot to represent the entity to participate in a given corresponding conversation with a given additional user among the plurality of corresponding conversations includes: causing the chatbot to render a plurality of instances of synthetic speech for presentation to the given additional user; and transmitting, from the remote system and to the client device, an indication of responsive content. One or more instances of the plurality of instances of synthetic speech are generated based on the natural language description for each explicit dialogue state and explicit dialogue state transition. Transmitting the indication of the responsive content to the client device causes the client device to provide the responsive content for presentation to the user of the client device.

[0129] In some implementations, a method implemented by one or more processors of a client device is provided, and the method includes: receiving, at the client device on behalf of an entity, unstructured free-form natural language input from a user of the client device, the unstructured free-form natural language input including a natural language description of a corresponding dialogue state diagram, the natural language description of the corresponding dialogue state diagram defining an implicit dialogue state and implicit dialogue state transitions of the corresponding dialogue state diagram without defining any explicit dialogue state or any explicit dialogue state transitions of the corresponding dialogue state diagram; in response to receiving the unstructured free-form natural language input including the natural language description of the corresponding dialogue state diagram: identifying a chatbot for participating on behalf of the entity in multiple corresponding conversations with additional users; and causing the chatbot to participate on behalf of the entity in multiple corresponding conversations with additional users. Causing the chatbot to participate in a given corresponding conversation with a given additional user among the multiple corresponding conversations includes: causing the chatbot to render multiple instances of synthetic speech audio data for presentation to the given additional user; and causing responsive content determined in response to rendering one or more of the multiple instances of synthetic speech for presentation to the given additional user to be provided for presentation to the user of the client device. One or more of the multiple instances of synthetic speech are generated based on the implicit dialogue state and / or the implicit dialogue state transitions.

[0130] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0131] In some implementations, identifying a chatbot for participating on behalf of an entity in multiple corresponding conversations with additional users includes obtaining a previously trained large language model (LLM); and causing the previously trained LLM to be used as the chatbot.

[0132] In some versions of those implementations, the previously trained LLM may be stored in the device storage of the client device, and the previously trained LLM that may be stored in the device storage of the client device may be a sparsified version of a global previously trained LLM available at a remote system communicatively coupled to the client device.

[0133] In additional or alternative versions of those implementations, the method may further include avoiding causing the previously trained LLM to be fine-tuned based on the unstructured free-form natural language input.

[0134] In additional or alternative versions of those implementations, causing the chatbot to render a given instance of synthetic speech from among multiple instances of synthetic speech for presentation to a given additional user may include: using a previously trained LLM to process unstructured free-form natural language input and one or more features associated with a given implicit dialogue state in an implicit dialogue state to generate an instance of text data reflecting a given behavior of the given implicit dialogue state; using a text-to-speech (TTS) model to process the given instance of text data reflecting a given behavior of the given implicit dialogue state to generate the given instance of synthetic speech; and transmitting the given instance of synthetic speech from a client device and to an additional client device of the given additional user. Transmitting the given instance of synthetic speech to the additional client device causes the additional client device to audibly render the given instance of synthetic speech for presentation to the given additional user via one or more speakers of the additional client device.

[0135] Additionally, some implementations include one or more processors of one or more computing devices (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)), where the one or more processors are operable to execute instructions stored in an associated memory, and where the instructions are configured to cause the execution of any of the methods described above. Some implementations also include one or more non-transitory computer-readable storage media that store computer instructions that are executable by the one or more processors to execute any of the methods described above. Some implementations also include a computer program product that includes instructions that are executable by the one or more processors to execute any of the methods described above.

[0136] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein.

Claims

1. A method implemented by one or more processors of a client device, the method comprising: Receiving, at the client device on behalf of an entity, unstructured free-form natural language input from a user of the client device, the unstructured free-form natural language input including a natural language description of a corresponding dialogue state diagram, the natural language description of the corresponding dialogue state diagram defining implicit dialogue states and implicit dialogue state transitions of the corresponding dialogue state diagram without defining any explicit dialogue states or any explicit dialogue state transitions of the corresponding dialogue state diagram; In response to receiving the unstructured free-form natural language input including the natural language description of the corresponding dialogue state diagram: Generating, based on the unstructured free-form natural language input, a chatbot for representing the entity in participating in a plurality of corresponding sessions with additional users; and Causing the chatbot to represent the entity in participating in the plurality of corresponding sessions with the additional users, wherein causing the chatbot to represent the entity in participating in a given corresponding session with a given additional user among the additional users in the plurality of corresponding sessions includes: Causing the chatbot to render a plurality of instances of synthetic speech audio data for presentation to the given additional user, wherein one or more instances of the synthetic speech are generated based on the implicit dialogue state and / or the implicit dialogue state transitions; and Causing responsive content determined in response to rendering one or more of the plurality of instances of synthetic speech for presentation to the given additional user to be provided for presentation to the user of the client device.

2. The method according to claim 1, wherein generating the chatbot for representing the entity in participating in the plurality of corresponding sessions with the additional users includes: Obtaining a previously trained large language model (LLM); Causing the previously trained LLM to be fine-tuned based on the unstructured free-form natural language input to generate a fine-tuned LLM; and And Utilizing the fine-tuned LLM as the chatbot.

3. The method according to claim 2, wherein causing the chatbot to render a given instance of synthetic speech from the plurality of instances of synthetic speech for presentation to the given additional user includes: Using the fine-tuned LLM to process one or more features associated with a given implicit dialogue state in the implicit dialogue state to generate an instance of text data reflecting a given behavior of the given implicit dialogue state; Using a text-to-speech (TTS) model to process the given instance of text data reflecting the given behavior of the given implicit dialogue state to generate the given instance of synthetic speech; and Transmit the given instance of the synthesized speech from the client device and to an additional client device of the given additional user, wherein transmitting the given instance of the synthesized speech to the additional client device enables the additional client device to audibly render the given instance of the synthesized speech for presentation to the given additional user via one or more speakers of the additional client device.

4. The method of claim 3, further comprising: Processing the corresponding context of the corresponding session using the fine-tuned LLM and one or more of the features associated with the given implicit dialogue state to generate the given instance of the text data reflecting the given behavior of the given implicit dialogue state.

5. The method of claim 3 or claim 4, further comprising: In response to the given instance of the synthesized speech being audibly rendered for presentation to the given additional user via the one or more speakers of the additional client device: Receiving, at the client device and from the additional client device, a given instance of response audio data including the responsive content in response to at least the given instance of the synthesized speech; Processing the given instance of the response audio data using an automatic speech recognition ASR model to generate a given instance of response text data; and Determining, based on the given instance of the response text data, whether to: Process one or more features associated with the given implicit dialogue state and the response text data using the fine-tuned LLM to generate an additional instance of the text data reflecting the given behavior of the given implicit dialogue state; Or Process one or more additional features associated with a given additional implicit dialogue state in the implicit dialogue state and the response text data using the fine-tuned LLM to generate an additional instance of the text data reflecting the given additional behavior of the given additional implicit dialogue state.

6. The method of claim 5, wherein causing the responsive content to be provided for presentation to the user of the client device includes the result of the corresponding session.

7. The method of any one of claims 3 to 6, further comprising: Before processing one or more of the features associated with the given implicit dialogue state using the fine-tuned LLM: Extracting one or more of the features from the unstructured free-form natural language input.

8. The method of claim 7, wherein one or more of the features are explicitly included in the unstructured free-form natural language input, and wherein extracting one or more of the features explicitly included in the unstructured free-form natural language input from the unstructured free-form natural language input includes: Utilizing an input parser to extract one or more of the features explicitly included in the unstructured free-form natural language input.

9. The method of claim 8, further comprising: Cause the fine-tuned LLM to use one or more of the features to generate an instance of text data for the given behavior that reflects the given implicit conversation state.

10. The method of claim 7, wherein one or more of the features are implicitly included in the unstructured free-form natural language input, and wherein extracting one or more of the features that are implicitly included in the unstructured free-form natural language input from the unstructured free-form natural language input comprises: Using an input parser to identify one or more of the features that are implicitly included in the unstructured free-form natural language input; And Using a coreference resolver to extract one or more of the features that are implicitly included in the unstructured free-form natural language input.

11. The method of claim 10, further comprising: Cause the fine-tuned LLM to use one or more of the features to generate an instance of text data for the given behavior that reflects the given implicit conversation state.

12. The method of any one of claims 2 to 11, wherein the previously trained LLM is stored in the device storage of the client device, and wherein the previously trained LLM stored in the device storage of the client device is a sparsified version of a global previously trained LLM available at a remote system communicatively coupled to the client device.

13. The method of claim 12, wherein the fine-tuned LLM is stored in the device storage of the client device.

14. The method of any of the preceding claims, wherein causing the chatbot to participate in the given corresponding session with the given additional user comprises: Causing the chatbot to answer a phone call received at the client device and from the given additional entity; And Causing the chatbot to participate in the corresponding session with the entity as part of the phone call.

15. The method of any of the preceding claims, wherein the user is not an active participant in the corresponding session between the chatbot and the given additional user.

16. A method implemented by one or more processors of a remote system, the method comprising: Receiving, at the remote system and on behalf of an entity from a client device, an indication of an unstructured free-form natural language input of a user of the client device, the indication of the unstructured free-form natural language input including a natural language description of a corresponding dialogue state diagram, the indication of the natural language description of the corresponding dialogue state diagram defining an implicit conversation state and an implicit conversation state transition of the corresponding dialogue state diagram and not defining any explicit conversation state or any explicit conversation state transition of the corresponding dialogue state diagram; In response to receiving the indication of the unstructured free-form natural language input including the natural language description of the corresponding dialogue state diagram: Generating a chatbot for representing the entity to participate in multiple corresponding conversations with additional users based on the indication of the unstructured free-form natural language input; and Causing the chatbot to represent the entity to participate in the multiple corresponding conversations with the additional users, wherein causing the chatbot to represent the entity to participate in a given corresponding conversation with a given additional user among the multiple corresponding conversations includes: Causing the chatbot to render multiple instances of synthesized speech for presentation to the given additional user, wherein one or more instances of the multiple instances of synthesized speech are generated based on the implicit conversation state and / or the implicit conversation state transition; and Transmitting an indication of the responsive content from the remote system and to the client device, wherein transmitting the indication of the responsive content to the client device causes the client device to provide the responsive content for presentation to the user of the client device.

17. The method according to claim 16, wherein generating the chatbot for representing the entity to participate in the multiple corresponding conversations with the additional users includes: Obtaining a previously trained large language model (LLM); Causing the previously trained LLM to be fine-tuned based on the unstructured free-form natural language input to generate a fine-tuned LLM; And Utilizing the fine-tuned LLM as the chatbot.

18. The method according to claim 17, wherein the previously trained LLM is stored in a remote storage of the remote system, and wherein the previously trained LLM stored in the remote storage of the remote system is an unsparsified version of a global previously trained LLM available at the remote system communicatively coupled to the client device.

19. The method according to claim 18, wherein the fine-tuned LLM is stored in the remote storage of the remote system.

20. A method implemented by one or more processors of a client device, the method including: Receiving, at the client device on behalf of an entity, unstructured free-form natural language input from a user of the client device, the unstructured free-form natural language input including a natural language description of a corresponding dialogue state diagram, the natural language description of the corresponding dialogue state diagram defining an implicit conversation state and an implicit conversation state transition of the corresponding dialogue state diagram without defining any explicit conversation state or any explicit conversation state transition of the corresponding dialogue state diagram; In response to receiving the unstructured free-form natural language input including the natural language description of the corresponding dialogue state diagram: Generating a chatbot for representing the entity to participate in multiple corresponding conversations with additional users based on the unstructured free-form natural language input; and Causing the chatbot to participate on behalf of the entity in the plurality of corresponding conversations with additional users, wherein causing the chatbot to participate on behalf of the entity in a given corresponding conversation with a given additional user among the additional users in the plurality of corresponding conversations includes: Causing the chatbot to render multiple instances of text data for presentation to the given additional user, wherein one or more of the multiple instances of text data are generated based on the implicit conversation state and / or the implicit conversation state transition; and Causing responsive content determined in response to rendering one or more of the multiple instances of text data for presentation to the given additional user to be provided for presentation to the user of the client device.

21. A method implemented by one or more processors of a remote system, the method comprising: Receiving, at the remote system and on behalf of an entity from a client device, an indication of an unstructured free-form natural language input from a user of the client device, the indication of the unstructured free-form natural language input including an indication of a natural language description of a corresponding conversation state diagram, the natural language description of the corresponding conversation state diagram defining an implicit conversation state and an implicit conversation state transition of the corresponding conversation state diagram and not defining any explicit conversation state or any explicit conversation state transition of the corresponding conversation state diagram; In response to receiving the indication of the unstructured free-form natural language input including the indication of the natural language description of the corresponding conversation state diagram: Generating, based on the indication of the unstructured free-form natural language input, a chatbot for participating on behalf of the entity in a plurality of corresponding conversations with additional users; and Causing the chatbot to participate on behalf of the entity in the plurality of corresponding conversations with additional users, wherein causing the chatbot to participate on behalf of the entity in a given corresponding conversation with a given additional user among the additional users in the plurality of corresponding conversations includes: Causing the chatbot to render multiple instances of text data for presentation to the given additional user, wherein one or more of the multiple instances of text data are generated based on the implicit conversation state and / or the implicit conversation state transition; and Transmitting, from the remote system and to the client device, an indication of the responsive content, wherein transmitting the indication of the responsive content to the client device causes the client device to provide the responsive content for presentation to the user of the client device.

22. A method implemented by one or more processors of a client device, the method comprising: Receiving, at the client device, a corresponding conversation state diagram from a user of the client device, the corresponding conversation state diagram including explicit conversation states and explicit conversation state transitions; At the client device and for each explicit dialogue state of the explicit dialogue states, receive, on behalf of the entity, a corresponding unstructured free-form natural language input from the user, the corresponding unstructured free-form natural language input including a natural language description for each explicit dialogue state of the explicit dialogue states and for the explicit dialogue state transitions; In response to receiving the unstructured free-form natural language input including the natural language description for each explicit dialogue state of the explicit dialogue states and for the explicit dialogue state transitions: Generate, based on the unstructured free-form natural language input, a chatbot for participating, on behalf of the entity, in a plurality of corresponding conversations with additional users; and Cause the chatbot to participate, on behalf of the entity, in the plurality of corresponding conversations with the additional users, wherein causing the chatbot to participate, on behalf of the entity, in a given corresponding conversation with a given additional user of the additional users in the plurality of corresponding conversations includes: Cause the chatbot to render a plurality of instances of synthetic speech for presentation to the given additional user, wherein one or more of the plurality of instances of synthetic speech are generated based on the natural language description for each explicit dialogue state of the explicit dialogue states and for the explicit dialogue state transitions; and Cause responsive content determined in response to rendering one or more of the plurality of instances of synthetic speech for presentation to the given additional user to be provided for presentation to the user of the client device.

23. A method implemented by one or more processors of a remote system, the method comprising: At the remote system and receive an indication of a corresponding dialogue state diagram from a user of the client device at the client device, the indication of the corresponding dialogue state diagram including explicit dialogue states and explicit dialogue state transitions; At the remote system and from the client device and for each explicit dialogue state of the explicit dialogue states, receive an indication of a corresponding unstructured free-form natural language input from the user on behalf of the entity, the indication of the corresponding unstructured free-form natural language input including a natural language description for each explicit dialogue state of the explicit dialogue states and for the explicit dialogue state transitions; In response to receiving the indication of the unstructured free-form natural language input including the natural language description for each explicit dialogue state of the explicit dialogue states and for the explicit dialogue state transitions: Generate, based on the indication of the unstructured free-form natural language input, a chatbot for participating, on behalf of the entity, in a plurality of corresponding conversations with additional users; and Cause the chatbot to participate, on behalf of the entity, in the plurality of corresponding conversations with the additional users, wherein causing the chatbot to participate, on behalf of the entity, in a given corresponding conversation with a given additional user of the additional users in the plurality of corresponding conversations includes: Cause the chatbot to render multiple instances of synthesized speech for presentation to the given additional user, where one or more of the multiple instances of synthesized speech are generated based on the natural language description for each explicit dialogue state and the explicit dialogue state transitions in the explicit dialogue state; and Transmit an indication of the responsive content from the remote system and to the client device, where transmitting the indication of the responsive content to the client device causes the client device to provide the responsive content for presentation to the user of the client device.

24. A method implemented by one or more processors of a client device, the method comprising: Receiving, at the client device on behalf of an entity, unstructured free-form natural language input from a user of the client device, the unstructured free-form natural language input including a natural language description of a corresponding dialogue state diagram, the natural language description of the corresponding dialogue state diagram defining implicit dialogue states and implicit dialogue state transitions of the corresponding dialogue state diagram, without defining any explicit dialogue states or any explicit dialogue state transitions of the corresponding dialogue state diagram; In response to receiving the unstructured free-form natural language input including the natural language description of the corresponding dialogue state diagram: identifying a chatbot for representing the entity in participating in multiple corresponding sessions with an additional user; and Causing the chatbot to represent the entity in participating in the multiple corresponding sessions with the additional user, where causing the chatbot to represent the entity in participating in a given corresponding session with a given additional user among the multiple corresponding sessions includes: Causing the chatbot to render multiple instances of synthesized speech audio data for presentation to the given additional user, where one or more of the multiple instances of synthesized speech are generated based on the implicit dialogue state and / or the implicit dialogue state transitions; and Causing responsive content determined in response to rendering one or more of the multiple instances of synthesized speech for presentation to the given additional user to be provided for presentation to the user of the client device.

25. The method of claim 24, wherein identifying the chatbot for representing the entity in participating in the multiple corresponding sessions with the additional user includes: Obtaining a previously trained large language model (LLM); And Causing the previously trained LLM to be used as the chatbot.

26. The method of claim 25, wherein the previously trained LLM is stored in the on-device storage of the client device, and the previously trained LLM stored in the on-device storage of the client device is a sparsified version of a global previously trained LLM available at a remote system communicatively coupled to the client device.

27. The method of claim 25 or claim 26, further comprising: Avoid fine-tuning the previously trained LLM based on the unstructured free-form natural language input.

28. The method according to any one of claims 25 to 27, wherein causing the chatbot to render a given instance of the synthetic speech from among the plurality of instances of synthetic speech for presentation to the given additional user comprises: Using the previously trained LLM to process the unstructured free-form natural language input and one or more features associated with a given implicit conversation state in the implicit conversation state to generate an instance of text data reflecting a given behavior of the given implicit conversation state; Using a text-to-speech (TTS) model to process the given instance of the text data reflecting the given behavior of the given implicit conversation state to generate the given instance of the synthetic speech; and Transmitting the given instance of the synthetic speech from the client device and to an additional client device of the given additional user, wherein transmitting the given instance of the synthetic speech to the additional client device causes the additional client device to audibly render the given instance of the synthetic speech for presentation to the given additional user via one or more speakers of the additional client device.

29. A system comprising: At least one processor; And A memory that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform the method according to any one of claims 1 to 28.

30. A non-transitory computer-readable storage medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the operations of the method according to any one of claims 1 to 28.

Images