Language model assisted human-computer interaction

Abstract

Implementations provide a method comprising: receiving a user input from a particular user; generating an attribute embedding based on the attribute information provided by the specific user, the attribute embedding representing the attribute information of the specific user in a numerical form but not revealing the attribute information of the specific user; processing both the attribute embedding and the user input using the language model to generate a language model output; generating a response to the user input based on the language model output; and causing the generated response to be rendered at the client device in response to a user input from the particular user.

Description

Background Art

[0001] Humans can participate in and have a human-machine conversation with an interactive software application herein referred to as an “automation assistant” (also known as a “chatbot”, “interactive personal assistant”, “intelligent personal assistant”, “personal voice assistant”, “conversational agent” or simply “assistant” etc.). For example, a human (sometimes referred to as a “user” when they interact with the automation assistant) can use user inputs such as oral natural language inputs (e.g., oral utterances that can be converted into text and then processed) or text (e.g., typed) natural language inputs to provide commands or requests to the automation assistant. The automation assistant typically responds to the command or request by providing user interface outputs (e.g., audible and / or graphical user interface outputs), controlling smart devices, and / or performing other actions in response to the command or request from the user.

[0002] However, during a human-machine conversation, the automation assistant may not be able to robustly adapt the user interface output or actions based on the user's attributes, such as the temporary and / or persistent attributes the user has provided in their profile and / or the attributes inferred from the user's commands, requests, and / or other interactions with the automation assistant.

[0003] In other words, the automation assistant may not be able to adapt the user interface output it provides based on the attributes of the user participating in the human-machine conversation session with the automation assistant. This results in the automation assistant generating user interface outputs that do not resonate with the user, which may inhibit the user's ability to understand such outputs. This can additionally or alternatively prolong the human-machine conversation session because additional user input may be required to confirm the user's intent. The prolonged human-machine conversation session between the user and the client device (through which the conversation session occurs) can lead to overutilization of the battery, processor, and / or other resources of the client device. Summary of the Invention

[0004] The implementations disclosed herein relate to using a language model (e.g., a large language model (LLM)) to facilitate a human-machine conversation between a user and an interactive software application (e.g., an “automation assistant”) installed on or accessible via a client device. In those implementations, the user can provide natural language user inputs (oral or text) to the automation assistant during the human-machine conversation. The automation assistant can generate responsive user interface outputs for the automation assistant to render (e.g., audible and / or visual) and / or responsive actions to be performed or initiated by the automation assistant based on the text of the natural language user input.

[0005] In addition, the implementations disclosed herein seek to ensure that responsive user interface outputs and / or responsive actions resonate with the user. To do so, the automated assistant further generates responsive user interface outputs and / or responsive actions based on the attribute information of the user participating in the human-machine conversation. The attribute information is utilized with the permission from the user, and may include attribute information based on attributes explicitly specified by the user prior to the human-machine conversation (e.g., in the user profile) and / or based on attributes inferred from the human-machine conversation and / or from previous human-machine conversations involving the user. In some of those implementations, an attribute embedding may be generated based on the attribute information, and the automated assistant uses the attribute embedding when generating responsive user interface outputs and / or responsive actions. The attribute embedding can be used to generate responsive user interface outputs and / or responsive actions, and can be used independently of any use of the underlying attribute information utilized when generating the attribute embedding. In addition, the attribute embedding may be represented in numerical form without disclosing the underlying attribute information it is generated based on. In these and other ways, the utilization of the attribute embedding enables the generation of responses that resonate with a given user and / or enables the more efficient (e.g., faster) resolution of interactions with a given user while maintaining user privacy and / or the security of user data. For example, the attribute embedding can be effectively utilized in the processing performed by the LLM without disclosing the underlying attribute information it is generated based on.

[0006] In various implementations, the automated assistant can generate a language model output based on processing both (a) the current attribute embedding of the user involved in the human-machine conversation and (b) the latest instance of the natural language user interface input from the user. In addition, the automated assistant can generate a responsive user interface output and / or a responsive action based on the language model output. Optionally, when generating the language model output, the LLM can be used and additional data can be processed along with (a) the current attribute embedding and (b) the latest instance of the natural language user input. For example, the additional data processed can be based on the conversation history from the human-machine conversation. For example, it can include previous responses from the automated assistant and / or previous natural language user inputs from the user.

[0007] In some implementations, when generating a language model output based on both (a) a current attribute embedding and (b) a most recent instance of a natural language user input from a user, an LLM is used to process (a) the current attribute embedding (optionally together with additional data) to prime the LLM, and then the LLM is used to process (b) the most recent instance of the natural language user input. For example, (a) the current attribute embedding and (b) the most recent instance of the natural language user input can be concatenated into a continuous string, and the LLM is used to process the continuous string. In some of those implementations, the language model output generated after processing (b) the most recent instance of the natural language user input can be the language model output on which a responsive user interface output and / or a responsive action is based.

[0008] As described above, using the LLM to process (a) the current attribute embedding can ensure that the responsive user interface output and / or the responsive action resonates with the user. As a non-limiting example, assume that a natural language user input of "I’m bored" is provided to an automated assistant. When the natural language user input is processed using the LLM together with a first attribute embedding, a first language model output can be generated. In this non-limiting example, the first attribute embedding can be, for example, an age embedding that is generated based on a previous user input (e.g., in the same human-machine conversation as the natural language user input 1 minute ago, or from a different conversation) indicating that the user is in their early 20s, based on voice features of the spoken utterance from which the natural language user input is identified, or based on one or more terms in the natural language user input indicating age information (e.g., youth language or old-fashioned words). It should be noted that the first attribute embedding does not necessarily need to be an age embedding, but can include embeddings of additional or alternative types of attributes, such as a hobby embedding generated based on a user profile indicating that the user is a music fan. Additionally, the automated assistant can generate and implement a first response based on the first language model output, such as a first text or audible recommendation (e.g., "wanna hear the song X? I believe it’s one you might like", where song X is a popular song among people in their early 20s and is thus recommended to a user in their early 20s) and / or a first action (e.g., an action that causes "song X" to be played). The music-focused first response can be partially based on the processing of the first attribute embedding and on the first attribute embedding that indirectly reflects an interest in music.

[0009] Continuing with the above non-limiting example, when alternatively using an LLM and processing natural language user input along with a distinct second attribute embedding (e.g., a weekly routine embedding generated based on calendar data shared by the user indicating that the user plays trivia with a group of friends every Saturday night), a distinct second language model output can be alternatively generated. Additionally, the automated assistant can generate and implement a second response based on the second language model output, such as a second text or audible recommendation (e.g., "want to play some trivia?") and / or a second action (e.g., an action that causes the trivia application to be launched). The trivia-focused second response can be partially based on the processing of the second attribute embedding and on the second attribute embedding that indirectly reflects an interest in trivia.

[0010] As described above, the attribute information utilized in generating the attribute embedding can include attribute information based on attributes explicitly specified by the user prior to the human-machine conversation (e.g., in the user profile) and / or based on attributes inferred from the human-machine conversation and / or from previous human-machine conversations involving the user.

[0011] As a specific example, an initial attribute embedding can be generated for the user based on attribute information from the user's profile for which the user has provided permission to utilize the user profile. For example, the attribute information can include the user's specific age or age range, the user's geographical region, the user's gender, the user's explicitly indicated preferences, and / or other attribute information of the user. For example, a neural network encoder can be used to process such attribute information, and the final or intermediate output of the encoder generated based on the processing can be used as the initial attribute embedding. The initial attribute embedding can be used for one or more iterations of generating the automated assistant response described herein and / or can be iteratively updated over time (e.g., as described below), and the corresponding updated attribute embedding is used for the iterations of generating the automated assistant response described herein.

[0012] In some implementations, the initial attribute embedding for a user can be updated over time based on attributes inferred from past or current human-machine conversations the user has participated in. The updates over time can occur iteratively during a given human-machine conversation session, and / or can occur across multiple human-machine conversation sessions (e.g., updated iteratively during a first session and then continued to be updated iteratively during a second session). In some of those implementations, the initial attribute embedding is updated by determining a conversation attribute embedding associated with the conversation the user has participated in and adapting the initial attribute embedding such that it moves closer in terms of distance in the embedding space to the conversation attribute embedding. For example, suppose the conversation the user has participated in is about music. The conversation attribute embedding can be determined based on processing attribute information reflecting an interest in music using a neural network encoder, and the final or intermediate output of the encoder is used as the conversation attribute. Additionally, the adapted attribute embedding can be generated as the (weighted or unweighted) average of the initial attribute embedding and the conversation embedding. Additionally or alternatively, the conversation attribute embedding can be determined based on the attribute embeddings for a group of users who have participated in the same or similar conversations about music. In those additional or alternative scenarios, the adapted attribute embedding can likewise be generated as the average of the initial attribute embedding and the conversation embedding. In these and other ways, the initial attribute embedding of the user can be updated or further updated by moving the initial attribute embedding of the user closer to the conversation embedding derived from the human-machine conversation involving the user. Such updates are performed without reprocessing the attribute information using an encoder model or other model used to generate the attribute embedding. Such updates are not only computationally efficient, but they can further ensure that the updated embedding does not directly reveal the underlying attribute information reflected by such updated embedding.

[0013] As another specific example, the initial attribute embedding for a user can be generated not based on the user's attribute information, but rather as a default attribute embedding or a randomly selected attribute embedding, such as an attribute embedding randomly selected from a distribution around the default attribute embedding. Additionally, such default or randomly selected initial attribute embeddings can be updated over time for the user based on attributes inferred from past or current human-machine conversations the user has participated in. In some implementations, in response to determining that the user has not provided any attribute information and / or the user has not shared any attribute information with the automated assistant, a default or randomly selected attribute embedding can be used as the initial attribute embedding.

[0014] As yet another specific example, the initial attribute embedding for a user can be generated based on attributes inferred from the user's input during the current human-machine conversation the user has participated in.

[0015] The language model (e.g., LLM) utilized in the implementations disclosed herein can be trained to generate a language model output that depends at least on an attribute embedding and a natural language input instance. For example, the LLM can be trained at least in part on training instances, each training instance including: a corresponding training instance input having at least a corresponding attribute embedding and a corresponding natural language input instance, and a corresponding ground truth training instance response.

[0016] As a specific example, training instances can be generated based on at least a chat conversation, email communication, or other communication exchange between two human users. For example, the training instance input of a training instance can include a natural language input provided by a first user among the users in the communication exchange, and can include an attribute embedding generated based on the attribute information of the first user among the users. The training instance output of the training instance can include a natural language input provided by a second user among the users in the communication exchange and responsive to the training instance input. Utilizing such training instances (and a large number of additional similar training instances) takes advantage of the fact that the response of the second user will be adapted to the attribute information of the first user among the users, such that the language model can be trained such that the language model output is also generated according to the attribute information of the user participating in the human-machine conversation. For example, assume that the natural language input provided by the first user is "anysuggestions for a fun activity around town?". The response of the second user to this input will vary significantly depending on the attribute information of the first user. For example, if the first user is a young professional in a metropolitan area, a first response will be provided, while if the first user is an elderly person in a remote rural area, the response will be different.

[0017] As another specific example, training instances can be generated based on all or part of a web page attributable to a specific author. For example, the training instance input of a training instance can include a natural language input generated based on a first part attributable to a given author in the natural language in the web page, and can include an attribute embedding generated based on the attribute information of the given author. The training instance output can include a natural language input that conforms to a second part of the natural language, such as a second part that follows immediately after the first part. For example, the first part can be the first sentence, and the natural language input of the training instance input can conform to the first part or can be a paraphrase of the first part. Additionally, the second part can conform to the second sentence that follows immediately after the first sentence. Utilizing such training instances (and a large number of additional similar training instances) takes advantage of the fact that different parts made by an author will each be adapted to the author's attribute information, such that a language model can be trained such that the language model output is also generated according to the attribute information of the user participating in the human-machine conversation.

[0018] In some implementations, the LLM can include at least hundreds of millions of parameters. In some of those implementations, the LLM includes at least billions of parameters, such as two billion or more parameters, or one hundred billion or more parameters. In some additional or alternative implementations, the LLM is a sequence-to-sequence model, is based on Transformer, includes an attention mechanism, and / or can include an encoder and / or a decoder (e.g., a decoder-only model). A non-limiting example of the LLM is Google's Pathways Language Model (PaLM). Another non-limiting example of the LLM is Google's Language Model for Dialog Applications (LaMDA).

[0019] The above is provided only as an overview of some implementations. Those and / or other implementations are disclosed in more detail herein.

[0020] Various implementations can include a non-transitory computer-readable storage medium storing instructions that can be executed by a processor to perform a method such as one or more of the methods described herein. Yet other various implementations can include a system that includes a memory and one or more hardware processors that are operable to execute instructions stored in the memory to perform a method such as one or more of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other aspects, features, and advantages of certain implementations of the present disclosure will become more apparent from the following description taken in conjunction with the drawings. In the drawings:

[0022] Figure 1A block diagram depicting an example environment that shows various aspects of the present disclosure and in which implementations disclosed herein can be realized.

[0023] Figure 2A Depicts an example process for using a language model to assist in human - machine conversations according to various implementations.

[0024] Figure 2B Depicts another example process for using a language model to assist in human - machine conversations according to various implementations.

[0025] Figure 2C Depicts yet another example process for using a language model to assist in human - machine conversations according to various implementations.

[0026] Figure 3A Depicts another example process for using a language model to assist in human - machine conversations according to various implementations.

[0027] Figure 3B Depicts according to various implementations Figure 3A An enlarged view of the user interface in

[0028] Figure 4A Illustrates a flowchart of an example method of using a language model to assist in human - machine conversations according to various implementations.

[0029] Figure 4B Illustrates according to various implementations Figure 4A A flowchart of an example method of generating attribute embeddings in

[0030] Figure 5 Is a flowchart of another example method of using a language model to assist in human - machine conversations according to various implementations.

[0031] Figure 6 Illustrates an example architecture of a computing device according to various implementations. Detailed Description

[0032] Figure 1 Is a block diagram of an example environment 100 that shows various aspects of the present disclosure and in which implementations disclosed herein can be realized. As Figure 1As shown, the environment 100 may include a client computing device 11 (also referred to herein as a "client device"), which includes a client automation assistant 110, additional applications 116, and / or a data store 115. The client computing device 11 may communicate with one or more servers via one or more networks 15. For example, the servers may include servers that implement a cloud-based automation assistant application 13 (or some components thereof), and the client automation assistant application 110 may communicate with the cloud-based automation assistant application 13 via one or more networks 15. The client automation assistant application 110 and / or the cloud-based automation assistant application 13 may be referred to herein as an "automation assistant".

[0033] The client computing device 11 may be, for example, a cellular phone, a laptop computer, a desktop computer, a notebook computer, a tablet computer, a smart TV, a messaging device, or a personal digital assistant (PDA), and the present disclosure is not limited thereto. The one or more servers may include, for example, a cluster of high-performance computing devices. The one or more networks 15 may include, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, and / or any other suitable network. The additional applications 116 may include social media applications, music applications, messaging applications, and / or other applications that are different from the client automation assistant 110 but are accessible or installed at the client computing device 11.

[0034] In various implementations, the client automation assistant application 110 may have multiple components, including: an automatic speech recognition (ASR) engine 111, a text-to-speech (TTS) engine 113, a natural language understanding (NLU) engine 115, and / or an execution engine 117. The multiple components may further include, for example, an attribute determination engine 112 and / or a language model engine 114.

[0035] In various implementations, the cloud-based automation assistant application 13 may have multiple cloud-based components, including: a cloud-based automatic speech recognition (ASR) engine 131, a cloud-based text-to-speech (TTS) engine 133, a cloud-based natural language understanding (NLU) engine 135, a cloud-based execution engine 137, a cloud-based attribute determination engine 132, a cloud-based attribute embedding generation engine 134, and / or a cloud-based language model engine 136. Each of the multiple cloud-based components may have the same or similar functionality as their counterparts at the client computing device 11. For example, a cloud-based component among the multiple cloud-based components (e.g., the cloud-based ASR engine 131) may be more widely trained or have stronger processing capabilities, but has the same functionality as the corresponding local component (e.g., the ASR engine 111) at the client computing device 11. Although not shown for simplicity in Figure 1is illustrated, but the client automation assistant application 110 may also include an attribute embedding generation engine.

[0036] The ASR engine 111 may process audio data capturing an oral utterance to generate a speech recognition of the oral utterance. The NLU engine 115 may determine the semantic meaning of the audio (e.g., the aforementioned audio data capturing the oral utterance) and / or text (e.g., the natural language content from a message or the aforementioned speech recognition converted from the audio data by the ASR engine 111), and decompose the determined semantic meaning to determine the intent and / or parameters of the assistant action. For example, the NLU engine 115 may process the natural language content “Weather today in Louisville?”, to determine the intent (e.g., an Internet search) and / or parameters (e.g., including: “weather”, “today” and “Louisville”, or the search parameters of “Weather today in Louisville?”) of the assistant action (e.g., searching the Internet for the weather in Louisville today).

[0037] In some implementations, the NLU engine 115 may parse the intent and / or parameters based on a single utterance of the user, and in other cases, may generate a prompt based on the unparsed intent and / or parameters. In the latter case, the generated prompt may be rendered to the user to receive a user response, where the NLU engine 115 may utilize the user response to the rendered prompt to parse the intent and / or parameters. Optionally, the NLU engine 115 may work in cooperation with a dialogue manager engine (not illustrated) that determines the unparsed intent and / or parameters. For example, the aforementioned prompt may alternatively or additionally be generated using the dialogue manager engine. In some implementations, the NLU engine 115 may utilize one or more NLU machine learning models to determine the intent and / or parameters.

[0038] In some implementations, the NLU engine 115 may be completely omitted and the language model engine 114 may be used instead of the NLU engine 115. In some other implementations, both the NLU engine 115 and the language model engine 114 may be provided. In some of those other implementations, both the NLU engine 115 and the language model engine 114 may optionally process at least some user inputs in parallel and utilize a responsive output from one of the two when fulfilling the user input. For example, the output from the NLU engine 115 may be used to parse some inputs, and the output from the language model engine 114 may be used to parse other inputs.

[0039] In various implementations, the fulfillment engine 117 of the client automation assistant application 110 can receive an intent and / or parameters of the intent to fulfill the intent by performing a corresponding assistant action. The intent and / or parameters of the intent can be received from the NLU engine 115 or the language model engine 114. As a non-limiting example, the fulfillment engine 117 can receive the intent of the foregoing Internet search and the foregoing search parameter “Weather today in Louisville?” to cause the search engine of the client computing device 11 to search the Internet for “Weather today in Louisville?”. In this example, the fulfillment engine 117 can fulfill the intent by: (1) causing the search engine to search the Internet for the user query (i.e., “Weather today in Louisville?”), (2) generating fulfillment information (e.g., “it’s cloudy outside, with a temperature of 26℃”) based on the search results of the search (e.g., “Louisville, KY, Monday 11:00 am, cloudy, 26℃”), and / or (3) rendering the fulfillment information to the user of the client computing device 11. As another non-limiting example, the fulfillment engine 117 can receive the intent and / or parameters of an assistant action that causes the thermostat in the living room to set the room temperature to 72 degrees Fahrenheit. In this example, the fulfillment engine 117 can fulfill the intent by generating and forwarding a control signal to the thermostat in the living room, where the control signal causes the thermostat to set the room temperature to 72 degrees Fahrenheit.

[0040] In some implementations, the TTS engine 113 can convert text (e.g., the foregoing fulfillment information “it’s cloudy outside, with a temperature of 26℃”) into synthesized speech. For example, the synthesized speech can be generated by processing the text (e.g., processing phonemes determined from the text) using one or more trained speech synthesis neural network models. The synthesized speech can be audibly rendered via the hardware speaker (e.g., a stand-alone speaker) of the client computing device 11 or via another device (e.g., a cellular phone). Although the above uses one or more components of the client automation assistant 110 (e.g., the ASR engine 111) for illustration, the same or similar functions, processes, or features can be implemented using the corresponding components of the cloud-based automation assistant 13.

[0041] In various implementations, the attribute determination engine 112 (or the cloud-based attribute determination engine 132) can retrieve or determine attribute information from one or more sources (e.g., user input, user profile, user account, publicly accessible databases, etc.). In some implementations, the attribute determination engine 112 can determine some or all of the attribute information based on user input. Alternatively or additionally, the attribute determination engine 112 can determine some or all of the attribute information from a user profile (or other data authorized by the user) to which the automated assistant has access.

[0042] As a non-limiting example, the user input can be an oral utterance from a particular user, and based on the voice of the particular use reflected by such oral utterance, the attribute determination engine 112 can estimate the age of the particular user, and / or can estimate the gender of the particular user. In this case, the attribute determination engine 112 can include the estimated age and / or gender of the particular user in the attribute information for use in generating an attribute embedding that represents in numerical form but does not disclose the attribute information of the particular user. For example, the attribute embedding can be in the form of an N-dimensional vector represented by N numerical components. In this case, the attribute embedding generated for the attribute information "age 46, female" can be closer to the attribute embedding generated for the attribute information "age 47, female" than the attribute embedding generated for the attribute information "age 27, male". It is noted that the attribute information determined from the voice of the oral utterance can additionally or alternatively include other information, such as dialect.

[0043] As another non-limiting example, the user input can be an oral input or a typed input from the user, such as the input "I was born in the 1980s". Based on such user input (e.g., "I was born in the 1980s"), the attribute determination engine 112 can determine that the attribute information of the user includes: the age determined or estimated for the user (e.g., late 30s to early 40s). The attribute embedding generation engine 134 (or the corresponding entity implemented locally at the client device 11) can generate an attribute embedding that represents in numerical form but does not disclose the attribute information of the user based on the determined attribute information of the user (e.g., the determined or estimated age) and / or based on the initial attribute embedding.

[0044] The initial attribute embedding may, but need not necessarily, be user-specific. For example, the initial attribute embedding may be generated as the final output (or intermediate output) of an attribute embedding generation model 14 that processes as input the attribute information of a user extracted from a user account of a specific user. As another example, the initial attribute embedding may be generated as the final output (or intermediate output) of an attribute embedding generation model 14 that processes as input attribute information characterizing a group of users, where the group of users may include but does not necessarily include a specific user. In this case, the attribute information characterizing the group of users may come from, for example, a database 16 that stores or indexes publicly accessible posts, articles, or other data related to attribute information of public users. The attribute embedding generation model 14 may, for example, be a neural network model, such as a neural network encoder.

[0045] Continuing with the above non-limiting example, client-side automated assistant 110 may receive additional typed input (e.g., “I started to wear corrective lenses to treat nearsightedness about 10 years ago”) after the typed input (e.g., “I was born in the 1980s”). In this case, attribute determination engine 112 may determine updated attribute information for the user (e.g., late 30s to early 40s, nearsighted). Correspondingly, attribute embedding generation engine (or its counterpart 134) may generate additional / updated attribute embeddings that represent the updated attribute information of the specific user in numerical form but do not reveal the updated attribute information of the specific user based on the user's updated attribute information (e.g., late 30s to early 40s, nearsighted) and / or attribute embeddings.

[0046] Alternatively or additionally, attribute information may come from sources other than user input. For example, attribute information for a particular user may come from account information 115B for client automated assistant 110 (or other application) stored in data store 115, user profile 115A for client computing device 11 stored in data store 115, or Figure 1 Other sources not illustrated in (eg, email, text, or other information authorized by a particular user to be accessible to the client computing device 11 or an application installed at the client computing device 11).

[0047] In various implementations, the language model engine 114 can access and use a language model 12 (e.g., an LLM) to process both the attribute embeddings (or the aforementioned additional attribute embeddings) and the user input to generate a corresponding language model output. Alternatively, in various implementations, the user input (e.g., an oral utterance) can be processed to generate a natural language representation of the user input, and the language model engine 114 can access and use a language model 12 (e.g., an LLM) to process both the attribute embeddings and the natural language representation of the user input to generate a corresponding language model output.

[0048] Based on the corresponding language model output, the automated assistant can generate a response to the user input and cause the generated response to be rendered at the client device in response to the user input.

[0049] In some implementations, the language model engine 114 can process both the attribute embeddings and the user input (textual or audible) by: using the language model to process the attribute embeddings to prime the language model; and using the primed language model to process the user input to generate the aforementioned language model output. Based on the language model output, the client automated assistant 110 can, for example, use the fulfillment engine 117 to generate a response / utterance in response to the user input, or suggest content (or an action) in response to the user input to the user.

[0050] Figure 2A Illustrated is an example process for leveraging a language model to assist in a human-machine conversation according to various implementations. Figure 2B Illustrated is according to various implementations Figure 2A Another example process for leveraging a language model to assist in a human-machine conversation in Figure 2C Illustrated is according to various implementations Figure 2A Yet another example process for leveraging a language model to assist in a human-machine conversation in

[0051] As a non-limiting example, referring to Figure 2A, the user 200A of the client device 20 can type a user input 21 into the client device 20 via a user interface 200 of an application installed at the client device 20 (e.g., an automated assistant application 110 graphically represented by a symbol or avatar 200B at the user interface 200), where such user input 21 can be displayed at the user interface 200. The user input 21 can be processed such that the attribute information 201 of the user 200A (if any) can be determined from the user input 21. The attribute information 201 of the user 200A can be processed to generate an attribute embedding 23 that numerically represents the attribute information 201 of the user 200A. Alternatively or additionally, the attribute information 201 of the user 200A and an initial attribute embedding 22 can be processed to generate an attribute embedding 23 that numerically represents the attribute information 201 of the user 200A. For example, the initial attribute embedding 22 can be updated with the attribute information 201 to generate the attribute embedding 23. For example, an input embedding can be generated based on processing only the attribute information 201 using an attribute embedding generation model. Additionally, the attribute embedding 23 can be generated based on the input embedding and the initial attribute embedding 22. For example, the attribute embedding 23 can be generated as a weighted average of the input embedding and the initial attribute embedding, with the initial attribute embedding 22 being more heavily weighted.

[0052] The language model 24 can be used to process both the user input 21 and the attribute embedding 23 as inputs to generate a language model output 25. Based on the language model output 25, the client device 20 (e.g., via an application such as the automated assistant 110) can generate a response 26 in response to the user input 21 of the user 200A, where the response 26 can be displayed at the user interface 200 of the client device 20 as a statement of the automated assistant 200B. For example, the fulfillment engine 117 can utilize the language model output 25 to generate the response 26. Notably, instead of or in addition to being displayed at the user interface 200 of the client device 20, the response 26 can also be audibly rendered to the user 200A via one or more hardware speakers of the client device 20.

[0053] Now refer to Figure 2B , instead of or in addition to determining the attribute information 201 from the user input 21 as in Figure 2A , the attribute information 201 can also be determined from a user account 202. The user account 202 can be an account of the client device 20, an account of the automated assistant, or an account of another application accessible by the client device 20 (or the automated assistant). The user account 202 can include the attribute information of the user 200A.

[0054] Refer to Figure 2C, in addition to user input 21 and attribute embedding 23, the language model 24 can also process a customized assistant embedding 27 to generate a language model output 25. Based on such a language model output 25, the client device 20 can generate and display a response 26 at the user interface 200 of the client device 20. The customized assistant embedding 27 can be in the same embedding space as the attribute embedding 23, where the customized assistant embedding 27 can numerically represent one or more features or characteristics of the client device 20 (or an automated assistant visually represented by the symbol 200B). Alternatively or additionally, the customized assistant embedding 27 can numerically represent the relationship between the user 200A and the client device 20 (or an automated assistant visually represented by the symbol 200B).

[0055] Figure 3A Depicts another example process of using a language model to assist in human-machine conversation according to various implementations. Figure 3B Depicts according to various implementations Figure 3A An enlarged view of the user interface in. As Figure 3A shown, in various implementations, the client device 20 can receive an oral utterance of the user 300A as user input 31. The user input 31 can be processed (e.g., using the Figure 1 ASR engine 111 in) to generate a natural language representation / recognition 32 of the user input 31. Optionally, the natural language representation / recognition 32 of the user input 31 can be displayed at the user interface 300 of the automated assistant, which is visually represented by a symbol (e.g., "AA" or an avatar) 300B.

[0056] In response to receiving user input 31, attribute information 301 of user 300A can be determined or retrieved. For example, the attribute information 301 can be determined from the user input 31 or the natural language representation 32. Alternatively or additionally, the attribute information 301 can be determined based on the account information of user 300A or authorized user data. The attribute information 301 and / or the initial attribute embedding 39 can be processed to generate an attribute embedding 33. In addition, the language model 34 can process the natural language representation 32 of the user input 31 and the attribute embedding 33 to generate a language model output 35. Based on the language model output 35, a response 36 in response to the user input can be generated and displayed at the user interface 300 of the client device 30. It should be noted that the initial attribute embedding 39 can be generated as an output of the attribute embedding generation model 37, which is generated based on processing additional attribute information 303 different from the attribute information 301. In addition, in some implementations, the attribute embedding generation model 37 is also used to process the attribute information 301 without processing the additional attribute information 303 to generate an additional embedding. In some of those implementations, the attribute embedding 33 is determined based on averaging or otherwise combining the additional embedding and the initial attribute embedding 39.

[0057] Reference Figure 3B , as Figure 3A As a practical example, user 300A can provide the spoken utterance "I miss the old days and the old songs" as the user input 31 to an application visually represented by symbol 300B. In this example, in response to receiving the user input 31, a natural language recognition 32 of the spoken utterance "I miss the old days and the old songs" can be displayed, and the attribute information 301 can be determined. The attribute information 301 can be determined, for example, from the user profile and / or the historical chat history shared by the application (visually represented by symbol 300B) to include or indicate that user 300A is a woman in her 30s. The attribute information 301 can be processed to generate an attribute embedding 33. The language model 34 can be utilized to process the natural language representation 32 of the user input 31 (e.g., "I miss the old days and the old songs") and the attribute embedding 33 to generate a language model output 35. Based on such a language model output 35, a response 36 (e.g., "Do you want to hear song XX") can be generated and displayed at Figure 3Bat the user interface 300 illustrated in. Alternatively or additionally, based on the language model output 35 and / or the response 36, actionable suggestions 300C can be generated and displayed at the user interface 300. For example, the actionable suggestion 300C can be displayed as a selectable element that shows the natural language content "Click to hear song XX", where when the selectable element 300C is selected, song XX can be played via the client device 30 for the user 300A to enjoy.

[0058] Figure 4A illustrates a flowchart of an example method 400 that illustrates an example method of using a language model to assist in human-machine conversations according to various implementations. Figure 4B illustrates according to various implementations Figure 4A a flowchart of an example method of block 403. For convenience, the operations of method 400 are described with reference to the system performing the operations. The system of method 400 includes one or more processors and / or other components of a client device and / or a server device. Additionally, although the operations of method 400 are shown in a particular order, this is not intended to be limiting. One or more operations can be reordered, omitted, or added.

[0059] Refer to Figure 4A , in various implementations, at block 401, the system can receive user input from a specific user via a client device. As a non-limiting example, the client device can be a cellular phone, laptop computer, desktop computer, notebook computer, tablet computer, smart TV, messaging device, or personal digital assistant (PDA), and the present disclosure is not limited thereto. As a non-limiting example, the user input can be or include spoken words and / or typed or touch-controlled input. The user input can be an input that initiates a human-machine conversation or can be a user input that continues to be provided in an ongoing human-machine conversation.

[0060] In various implementations, at block 403, the system can generate an attribute embedding based on the attribute information of a specific user, which is represented in numerical form but does not disclose the attribute information of the specific user. In some implementations or iterations of block 403, block 403 can include Figure 4B sub-blocks 4031, 4033, and / or 4035 of. At sub-block 4031, the system determines the attribute information from the user input and / or from other sources such as the user account of the specific user. At the optional sub-block 4033, the system retrieves the initial attribute embedding of the specific user, such as when performing Figure 4AThe attribute embeddings generated in the latest iteration. At sub - box 4035, the system generates attribute embeddings based on the attribute information of box 4031 and optionally based on the initial attribute embeddings of optional box 4033. For example, the system can generate attribute embeddings by updating the initial attribute embeddings of box 4033 based on the attribute information of box 4031. For example, the system can determine additional embeddings based on the attribute information of box 4031 and then update the initial attribute embeddings of box 4031 to make the initial attribute embeddings closer to the additional embeddings in the embedding space.

[0061] In various implementations, at box 405, the system can use a language model to process both the attribute embeddings and the user input to generate a language model output. For example, the language model can be an LLM. For example, the language model can be an LLM trained based on example conversations and the corresponding attribute embeddings of those example conversations. In some implementations, the system can use the language model to process both the attribute embeddings and the user input in the following way: using the language model to process the attribute embeddings to guide the language model; and using the language model to process the user input to generate a language model output after guiding the language model with the attribute embeddings.

[0062] In various implementations, at box 407, the system can generate a response to the user input based on the language model output. The response can be in natural language and can be rendered audibly and / or visually at box 409. In various implementations, at box 409, the system can cause the generated response to be rendered at the client device in response to the user input from a particular user. In response to receiving further user input from a particular user, the system can return to box 401.

[0063] Figure 5 is a flowchart illustrating an additional example method 500 for using a language model to assist in human - machine conversations according to various implementations. For convenience, the operations of method 500 are described with reference to the system performing the operations. The system of method 500 includes one or more processors and / or other components of a client device and / or a server device. Additionally, although the operations of method 500 are shown in a particular order, this is not intended to be limiting. One or more operations can be reordered, omitted, or added.

[0064] In various implementations, at block 501, the system can receive user input from a particular user via a client device. In various implementations, at block 503, the system can determine a natural language representation of the user input from a particular user. In various implementations, at block 505, the system can generate an attribute embedding based on the user input from a particular user, where the attribute embedding is represented in numerical form but does not disclose the attribute information of the particular user. In various implementations, at block 507, the system can use a language model to process both the attribute embedding and the natural language representation to generate a language model output. In various implementations, at block 501, the system can generate a response to the user input based on the language model output. In various implementations, at block 501, the system can cause the generated response to be presented to the particular user via the client device.

[0065] In some implementations, the system can generate an attribute embedding at least based on the user input from a particular user by: retrieving an initial attribute embedding; and generating an attribute embedding by updating the initial attribute embedding based on the attribute information extracted from the user input.

[0066] Optionally, the initial attribute embedding can be generated based on the attribute information of the particular user extracted from the user account of the particular user. Optionally, the user account of the particular user is associated with the client device or an application of the client device. Optionally, the initial attribute embedding is a default embedding or a randomly selected embedding.

[0067] Optionally, the initial attribute embedding can be generated by an attribute embedding generation model using multiple instances from multiple users. The attribute embedding generation model can be, for example, a neural network, and the initial attribute embedding can be an intermediate output or a final output of the attribute embedding generation model.

[0068] In various implementations, the system can further receive additional user input from a particular user via the client device or an application accessible at the client device. In response to receiving the additional user input, the system can determine a natural language representation of the additional user input. In response to receiving the additional user input and based on the natural language representation of the additional user input and the attribute embedding, the system can generate an additional attribute embedding that represents the updated attribute information of the particular user in numerical form.

[0069] In various implementations, the system can use a language model to process both the natural language representation of the additional user input and the additional attribute embedding to generate an additional language model output. In various implementations, in response to the additional user input and based on the additional language model output, the system can generate an additional response to the additional user input and cause the generated additional response to be presented to the particular user via the client device.

[0070] Figure 6 FIG. 610 is a block diagram of an example computing device 610 that can optionally be utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more of a client computing device, a cloud-based automated assistant component, and / or other components can include one or more components of the example computing device 610.

[0071] The computing device 610 generally includes at least one processor 614 that communicates with a plurality of peripheral devices via a bus subsystem 612. These peripheral devices can include a storage subsystem 624 that includes, for example, a memory subsystem 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices allow interaction with a user of the computing device 610. The network interface subsystem 616 provides an interface to an external network and is coupled to a corresponding interface device in other computing devices.

[0072] The user interface input device 622 can include a keyboard, a pointing device (such as a mouse, trackball, touchpad, or graphics tablet), a scanner, a touchscreen incorporated into a display, an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways for inputting information into the computing device 610 or onto a communication network.

[0073] The user interface output device 620 can include a display subsystem, a printer, a fax machine, or a non-visual display, such as an audio output device. The display subsystem can include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for generating a visible image. The display subsystem can also provide a non-visual display, such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways for outputting information from the computing device 610 to a user or another machine or computing device.

[0074] The storage subsystem 624 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 can include selected aspects for performing the methods disclosed herein and for implementing Figure 1 the various components depicted in FIG. 2.

[0075] These software modules are typically executed by the processor 614 alone or in combination with other processors. The memory 625 used in the storage subsystem 624 may include multiple memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution and a read-only memory (ROM) 632 for storing fixed instructions therein. The file storage subsystem 626 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive together with an associated removable medium, a CD-ROM drive, an optical disk drive, or a removable media cartridge. Modules implementing the functionality of certain implementations may be stored by the file storage subsystem 626 in the storage subsystem 624 or stored in other machines accessible by the processor 614.

[0076] The bus subsystem 612 provides a mechanism for enabling the various components and subsystems of the computing device 610 to communicate with each other as intended. Although the bus subsystem 612 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0077] The computing device 610 can be of different types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 610 depicted in Figure 6 is only intended to be a specific example for illustrating some implementations. Many other configurations of the computing device 610 are possible, which have more or fewer components compared to the computing device depicted in Figure 6 .

[0078] Although several implementations have been described and illustrated herein, many other means and / or structures may be utilized for performing the functions and / or obtaining the results and / or one or more of the advantages described herein, and each such variation and / or modification is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which this teaching is used. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation many equivalents to the specific implementations described herein. Accordingly, it is to be understood that the foregoing implementations are presented by way of example only, and it is to be understood that within the scope of the appended claims and their equivalents, implementations may be practiced in ways other than as specifically described and claimed. Implementations of the present disclosure relate to each individual feature, system, and / or method described herein. Moreover, any combination of two or more such features, systems, and / or methods, where such features, systems, and / or methods are not mutually inconsistent, is included within the scope of the present disclosure.

[0079] In various implementations, a computer-implemented method is provided, and the method includes: receiving, via a client device, user input from a specific user, and generating an attribute embedding based on attribute information provided by the specific user, where the attribute embedding is represented numerically without disclosing the attribute information of the specific user. In various implementations, the method may further include: using a language model to process both the attribute embedding and the user input to generate a language model output; generating a response to the user input based on the language model output; and causing the generated response to be rendered at the client device in response to the user input from the specific user.

[0080] In various implementations, using a language model to process both the attribute embedding and the user input to generate a language model output may include: using the language model to process the attribute embedding to guide the language model; and using the language model to process the user input to generate a language model output after guiding the language model with the attribute embedding.

[0081] In various implementations, generating an attribute embedding based on attribute information may include: extracting attribute information from the user input; retrieving an initial attribute embedding associated with the client device; and generating an attribute embedding by updating the initial attribute embedding based on the attribute information of the specific user extracted from the user input. In these and other implementations, the initial attribute embedding may be generated based on additional attribute information of the specific user identified from the user account of the specific user. The user account of the specific user may be associated with the client device or an application accessible via the client device.

[0082] In various implementations, the initial attribute embedding may be generated based on processing additional attribute information using an attribute embedding generation model. In various implementations, the attribute embedding generation model may be a neural network, and the initial attribute embedding may be an intermediate output of the attribute embedding generation model. Alternatively, in some implementations, the initial attribute embedding may be the final output of the attribute embedding generation model.

[0083] In various implementations, generating an attribute embedding by updating the initial attribute embedding based on attribute information may include: determining an additional embedding based on the attribute information; and updating the initial attribute embedding to make the initial attribute embedding closer to the additional embedding in the embedding space.

[0084] In various implementations, the method may further include: receiving, via a client device, additional user input from a particular user; generating, based on the additional user input from the particular user and an attribute embedding, an additional attribute embedding that is represented numerically but does not disclose updated attribute information of the particular user; using a language model to process both the additional user input and the additional attribute embedding to generate an additional language model output; generating, based on the additional language model output, an additional response to the additional user input; and causing the generated additional response to be presented to the particular user via the client device.

[0085] In various implementations, an additional computer-implemented method is provided, and the method includes: receiving, via a client device, user input from a particular user; determining a natural language representation of the user input from the particular user; generating, based on the user input from the particular user, an attribute embedding that is represented numerically but does not disclose the attribute information of the particular user; using a language model to process both the attribute embedding and the natural language representation to generate a language model output; generating, based on the language model output, a response to the user input; and causing the generated response to be presented to the particular user via the client device.

[0086] In these implementations, generating an attribute embedding based at least on user input from a particular user may include: retrieving an initial attribute embedding; and generating the attribute embedding by updating the initial attribute embedding based on attribute information extracted from the user input. The initial attribute embedding may be generated, for example, based on attribute information of the particular user extracted from the user account of the particular user, where the user account of the particular user may be associated with the client device or an application of the client device.

[0087] In some implementations, the initial attribute embedding is a default embedding or a randomly selected embedding. In some implementations, the initial attribute embedding may be generated by an attribute embedding generation model using multiple instances collected from multiple users.

[0088] In some implementations, the attribute embedding generation model is a neural network, and the initial attribute embedding is the final output or an intermediate output of the attribute embedding generation model.

[0089] In some implementations, the additional method may further include: receiving additional user input from a specific user via a client device; determining a natural language representation of the additional user input; generating an additional attribute embedding based on the natural language representation of the additional user input and an attribute embedding, the additional attribute embedding numerically representing updated attribute information of the specific user; using a language model to process both the natural language representation of the additional user input and the additional attribute embedding to generate an additional language model output; generating an additional response in response to the additional user input based on the additional language model output; and causing the generated additional response to be presented to the specific user via the client device.

[0090] In various implementations, a system is provided, and the system includes: one or more processors and a memory storing instructions that, when executed, cause the one or more processors to perform the following operations: receiving user input from a specific user via a client device; generating an attribute embedding based on the user input from the specific user, the attribute embedding numerically representing but not disclosing the attribute information of the specific user; using a language model to process both the user input and the attribute embedding to generate a language model output; generating a response to the user input based on the language model output; and causing the generated response to be presented to the specific user via the client device.

[0091] In various implementations of the system, the one or more processors are further configured to perform the operation of generating an attribute embedding by: extracting attribute information from the user input; retrieving an initial attribute embedding; and generating an attribute embedding by updating the initial attribute embedding based on the attribute information of the specific user extracted from the user input. In various implementations of the system, the initial attribute embedding may be generated based on the attribute information of the specific user extracted from the user account of the specific user.

[0092] In various implementations of the system, the one or more processors may be or may include: a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU) of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause the execution of any of the above methods. Some implementations further include one or more non-transitory computer-readable storage media storing computer instructions executable by the one or more processors to execute any of the above methods. Some implementations further include a computer program product including instructions executable by the one or more processors to execute any of the above methods.

Claims

1. A computer-implemented method, the method comprising: Receiving user input from a specific user, the user input being formulated via a client device; Generating an attribute embedding based on attribute information provided by the specific user, the attribute embedding being represented numerically without disclosing the attribute information of the specific user; Using a language model to process both the attribute embedding and the user input to generate a language model output; Generating a response to the user input based on the language model output; And Causing the generated response to be rendered at the client device in response to the user input from the specific user.

2. The method according to claim 1, wherein, Using the language model to process both the attribute embedding and the user input to generate the language model output includes: Using the language model to process the attribute embedding to guide the language model; and After using the attribute embedding to guide the language model, using the language model to process the user input to generate the language model output.

3. The method according to any one of the preceding claims, wherein, Generating the attribute embedding based on the attribute information includes: Extracting the attribute information from the user input; Retrieving an initial attribute embedding associated with the client device; and Generating the attribute embedding by updating the initial attribute embedding based on the attribute information of the specific user extracted from the user input.

4. The method according to claim 3, wherein, The initial attribute embedding is generated based on additional attribute information of the specific user identified from the user account of the specific user.

5. The method according to claim 4, wherein, The user account of the specific user is associated with the client device or an application accessible via the client device.

6. The method according to claim 4 or claim 5, wherein The initial attribute embedding is generated based on processing the additional attribute information using an attribute embedding generation model.

7. The method according to claim 6, wherein: The attribute embedding generation model is a neural network model, and The initial attribute embedding is the final output or an intermediate output of the attribute embedding generation model.

8. The method according to any one of claims 3 to 7, wherein, Generating the attribute embedding by updating the initial attribute embedding based on the attribute information includes: Determining an additional embedding based on the attribute information; and Updating the initial attribute embedding to make the initial attribute embedding closer to the additional embedding in the embedding space.

9. The method according to any one of the preceding claims, further comprising: Receiving additional user input from the specific user, the additional user input being formulated via the client device; Generating an additional attribute embedding based on the additional user input from the specific user and the attribute embedding, the additional attribute embedding being represented numerically without disclosing the updated attribute information of the specific user; Using the language model to process both the additional user input and the additional attribute embedding to generate an additional language model output; Generating an additional response to the additional user input based on the additional language model output; And Causing the generated additional response to be presented to the specific user via the client device.

10. A computer-implemented method, comprising: Receiving user input from a specific user, the user input being formulated via a client device; Determine a natural language representation of the user input from the specific user; Generate an attribute embedding based on the user input from the specific user, the attribute embedding being represented numerically without revealing the attribute information of the specific user; Use a language model to process both the attribute embedding and the natural language representation to generate a language model output; Generate a response to the user input based on the language model output; And Cause the generated response to be presented to the specific user via the client device.

11. The method according to claim 10, wherein, Generating the attribute embedding based at least on the user input from the specific user includes: Retrieve an initial attribute embedding; and Generate the attribute embedding by updating the initial attribute embedding based on attribute information extracted from the user input.

12. The method according to claim 11, wherein, The initial attribute embedding is generated based on the attribute information of the specific user extracted from the user account of the specific user.

13. The method according to claim 12, wherein, The user account of the specific user is associated with the client device or an application of the client device.

14. The method according to any one of claims 11 to 13, wherein The initial attribute embedding is a default embedding or a randomly selected embedding.

15. The method according to any one of claims 11 to 13, wherein The initial attribute embedding is generated by an attribute embedding generation model using multiple instances collected from multiple users.

16. The method according to claim 15, wherein: The attribute embedding generation model is a neural network, and The initial attribute embedding is the final output or an intermediate output of the attribute embedding generation model.

17. The method according to any one of claims 10 to 15, further comprising: Receive additional user input from the specific user via the client device; Determine a natural language representation of the additional user input; Generate an additional attribute embedding based on the natural language representation of the additional user input and the attribute embedding, the additional attribute embedding numerically representing the updated attribute information of the specific user; Use the language model to process both the natural language representation of the additional user input and the additional attribute embedding to generate an additional language model output; Generate an additional response in response to the additional user input based on the additional language model output; And Cause the generated additional response to be presented to the specific user via the client device.

18. A system comprising one or more processors and a memory storing instructions which, when executed, cause the one or more processors to perform the method according to any one of the preceding claims.

19. One or more computer-readable media comprising instructions which, when executed, cause the performance of any one of the preceding claims.

Images