Dynamic adaptation to fulfillment of a given voice utterance based on the user who provided the given voice utterance

Abstract

and determining how to execute the voice utterance based on a user who provided the voice utterance. For example, an embodiment may receive a voice utterance from a user, determine a set of fulfillment actions for the voice utterance, and determine whether the user who provided the voice utterance corresponds to a first user or a second user. Further, in response to determining that the user corresponds to the first user, an embodiment may select a subset of first fulfillment actions from the set and cause the first subset of fulfillment actions to be performed to satisfy the voice utterance. Further, in response to determining that the user corresponds to a second user, an embodiment may select a different subset of second fulfillment actions from the set and cause the second subset of fulfillment actions to be performed to satisfy the voice utterance.

Description

[Background technology]

[0001] Humans may engage in human-computer interactions using interactive software applications referred to herein as "automated assistants" (also referred to as "chatbots," "interactive personal assistants," "intelligent personal assistants," "personal voice assistants," "conversational agents," etc.). For example, humans (who may be referred to as "users" when interacting with an automated assistant) may provide speech natural language input (i.e., speech utterances) directed to the automated assistance, in some cases by providing textual (e.g., typed) natural language that may be converted to text and then processed, and / or textual directed to the automated assistance. These speech utterances and / or typed inputs often include commands directed to the automated assistant. The automated assistant generally responds to these commands by providing responsive user interface output(s) (e.g., audible user interface output and / or visual user interface output), controlling smart device(s), and / or performing other action(s).

[0002] These automated assistants typically rely on a pipeline of components in interpreting and responding to these spoken utterances and / or typed input. For example, an automatic speech recognition (ASR) engine can process audio data corresponding to a user's spoken utterance to generate an ASR output, such as a transcription of the spoken utterance (i.e., a sequence of term(s) and / or a sequence of other token(s)). Additionally, a natural language understanding (NLU) engine can process the ASR output (or typed input) to generate an NLU output, such as a user intent, upon providing the spoken utterance and optionally slot value(s) for parameter(s) associated with the intent. Additionally, a fulfillment engine can process the NLU output and obtain fulfillment outputs, such as responsive user interface output(s), responsive smart device control command(s), and / or other responsive action(s).

[0003] In some cases, these automated assistants may achieve a particular voice utterance in a particular way. For example, assume that the automated assistant is running at least partially on an in-vehicle computing device of a given user's vehicle, and assume that the user directs the voice utterance "It's cold" to the automated assistant. In this example, the given automated assistant may utilize the above-described pipeline to respond to the voice utterance, resulting in a fulfillment action of turning on the blower in the vehicle to satisfy the voice utterance. However, the given user who provided the voice utterance may only have wanted the heated seats in the vehicle to be turned on, not the blower, which may differ from the preferences of other users. As a result, since the automated assistant performed the wrong fulfillment action, the given user may need to manually turn off the blower, turn on the heated seats, and / or initiate additional interactions with the automated assistant to have the desired fulfillment action performed. As a result, computer resources are wasted on these interactions based on the incorrect fulfillment action being performed (and optionally, any additional interactions that result from the incorrect fulfillment action being performed), and natural resources are wasted on this interaction based on turning on a blower when it was not the desired fulfillment action. Summary of the Invention

[0004] Implementations described herein relate to dynamically adapting fulfillment of a given voice utterance based on a user who provided the given voice utterance and, optionally, based on a context scenario in which the given voice utterance is received. In some implementations, a system (e.g., including a processor(s) and a memory) can receive a given voice utterance from a user, determine a set of fulfillment actions for the given voice utterance, and determine whether the user who provided the given voice utterance corresponds to a first user or a second user. Further, in response to determining that the user corresponds to a first user, the system can select a subset of first fulfillment action(s) from the set and cause the subset of first fulfillment action(s) to be performed to fulfill the given voice utterance. Further, in response to determining that the user corresponds to a second user, the system can select a different subset of second fulfillment action(s) from the set and cause the subset of second fulfillment action(s) to be performed to fulfill the given voice utterance. In particular, the second subset of fulfillment actions(s) may include at least one unique fulfillment action compared to the first subset of fulfillment actions(s). Thus, fulfillment of the same voice utterance provided by different users can be dynamically adapted based on whether the voice utterance is provided by a first user or a second user.

[0005] For example, assume that a first user is driving a vehicle and provides a voice utterance of "it's hot," and assume that a microphone(s) of an on-board computing device of the vehicle generates audio data capturing the voice utterance. In this example, the audio data capturing the voice utterance can be processed using an automatic speech recognition (ASR) model of the voice utterance to generate ASR data of the voice utterance (e.g., recognized text of "hot"). Further, the ASR data of the voice utterance can be processed using a natural language understanding (NLU) model to generate NLU data of the voice utterance (e.g., recognized intent(s) of "hot," "turn down the temperature," etc.). Using various audio-based techniques (e.g., speaker identification (SID) and / or other audio-based techniques) and / or non-audio-based techniques (e.g., facial recognition (FID), user account identification, device identifier identification, and / or other non-audio-based techniques), among others, described herein, it can be determined that the user who provided the voice utterance corresponds to the first user.

[0006] In some implementations, the NLU data of the voice utterance can be processed to determine a set of fulfillment actions that, when performed, facilitate reducing the temperature inside the vehicle. The set of fulfillment actions can include, for example, lowering the vehicle window(s) (if the vehicle is in a state indicating that the window is up) to reduce the temperature inside the vehicle, turning on the vehicle blower to circulate cool air throughout the vehicle to reduce the temperature inside the vehicle, and / or other fulfillment actions that can be performed to reduce the temperature inside the vehicle. In this example, the set of fulfillment actions can be processed using a fulfillment action model and / or fulfillment rule(s) (e.g., personalized mapping(s)) that is specific to the first user to select the first fulfillment action(s) to be performed to satisfy the voice utterance. For example, the first fulfillment action(s) can include lowering the vehicle window(s) but not turning on the vehicle blower to circulate cool air throughout the vehicle.

[0007] In contrast, assume that a second user is driving the same vehicle at a subsequent time instance and provides the same voice utterance, "It's hot." The voice utterance can be processed in the same or similar manner to determine a set of fulfillment actions. However, in this example, the set of fulfillment actions can be processed using a fulfillment action model and / or fulfillment rule(s) that are specific to the second user (e.g., not the first user) to select the second fulfillment action(s) to be performed to satisfy the voice utterance. For example, the second fulfillment action(s) can include lowering a window(s) of the vehicle and turning on a blower in the vehicle to circulate cool air throughout the vehicle.

[0008] In various implementations, the system may generate corresponding training instances that can be subsequently utilized in training the above-mentioned fulfillment action model(s) and / or the above-mentioned fulfillment rules. These corresponding training instances may be specific to different users and may be generated based on the different users' corresponding user interactions with various smart devices performed in different context scenarios. For example, assume that a first user causes a window(s) of the vehicle to be lowered at a given time instance of the plurality of time instances during a vehicle ride. Further assume that an ambient temperature sensor of the vehicle indicates that the temperature inside the vehicle is 80 degrees Fahrenheit (e.g., a context signal characterizing a state of the vehicle's environment), which represents a "hot" context scenario at the given time instance. In this example, the given training instance may be generated based on the fulfillment action(s) manually performed by the user in this "hot" context scenario. The given training instance may include a training instance output and a training instance input. In this example, the training instance input may include a set of fulfillment actions that can be performed to reduce the temperature inside the vehicle, and a context scenario in which the fulfillment actions were performed (e.g., a "hot" context scenario) (or an indication thereof). Additionally, the training instance output may include a fulfillment action of a user interaction performed by the user (e.g., rolling down a window(s) of the vehicle) (or an indication thereof). This process may be repeated for multiple users in multiple different context scenarios to generate corresponding training instances. In some of these implementations, the system may assign one or more voice commands (e.g., corresponding to semantic terms or phrases of the context scenario, such as "hot," "burning," "burning," etc.) to each of the different context scenarios.

[0009] In some versions of those implementations, the system may train the fulfillment action model(s) based on the corresponding training instances. For example, the system may use the fulfillment action model of the first user to process a set of fulfillment actions that can be performed to reduce the temperature inside the vehicle and training instance inputs corresponding to the context scenario in which the fulfillment actions were performed to generate predicted outputs. The predicted outputs may correspond to a subset of predicted first fulfillment action(s) that the first user is predicted to perform given the fulfillment actions available in the "hot" context scenario. Furthermore, the predicted outputs may be compared to the fulfillment actions of the user interactions performed by the user (e.g., over a distribution of action(s)) to generate one or more losses. Furthermore, the fulfillment action model of the first user may be updated (e.g., via backpropagation) based on the one or more losses. In these implementations, the system may only train the fulfillment action model(s) in response to determining that one or more training conditions described herein are met.

[0010] In various implementations, the system may generate personalized mapping(s) (e.g., fulfillment rule(s)) to be utilized in addition to or instead of the fulfillment action model(s) described above. For example, the system can generate the personalized mapping based on one or more past instances of a user causing the performance of one or more fulfillment actions corresponding to a user interaction in a context scenario and independent of providing a voice utterance. Furthermore, the system can use various techniques to determine that a context scenario correlates to one or more voice commands. For example, the system can map different context scenarios represented by different context signals that characterize a state of the user and / or characterize a state of the environment when one or more fulfillment actions are performed to different portions of a lower dimensional space (e.g., an embedding space, a semantic space, etc.). Furthermore, one or more commands can be assigned to different context scenarios in the lower dimensional space. Thus, when a given voice command is subsequently included in a given voice utterance received from a user, the system can determine a given context scenario associated with the given voice command in the lower dimensional space and cause one or more given fulfillment actions previously performed in the given context scenario to be performed to fulfill the given voice utterance. Also, for example, the system can map different context scenarios represented by different context signals characterizing a state of the user and / or characterizing a state of the environment when the one or more fulfillment actions are performed to different portions of the lower dimensional space (e.g., an embedding space, a semantic space, etc.). Additionally, one or more voice commands can be assigned to different intents.Thus, when the given voice command is later included in a given voice utterance received from a user, the system can determine a given context scenario associated with the given voice command determined based on the given intent, and cause one or more given fulfillment actions previously performed in the given context scenario to be performed to satisfy the given voice utterance.

[0011] In some versions of those implementations, the system may only generate the personalized mapping in response to determining that one or more fulfillment actions have been performed by the user a threshold quantity of instances (e.g., 1 time, 3 times, 5 times, and / or 1 or any other positive integer greater than 1) in the context scenario. For example, assume that there were three past instances of the user rolling down the vehicle's window(s) without turning on the vehicle's blower when the ambient temperature inside the vehicle was between 70° F. and 80° F., and assume that the three past instances meet the threshold quantity of instances. In this example, the system may map the context scenario (e.g., the ambient temperature inside the vehicle being between 70° F. and 80° F.) to a first portion of the lower dimensional space and / or a first portion associated with a "hot" intent, and assign voice commands such as "hot" and "burning" to the first portion of the lower dimensional space and / or the first portion associated with a "hot" intent. Thus, even if the user did not provide any voice utterances during the three previous instances, if the user later provides a voice utterance saying "it's hot" when the ambient temperature inside the vehicle is between 70° F. and 80° F., the system may utilize the personalized mapping to automatically lower the window(s). The system may also take into account the ambient temperature outside the vehicle in these instances.

[0012] Notably, in these implementations, the same voice command may be mapped to different fulfillment actions. Continuing with the above example, assume further that there were three past instances where the user rolled down the vehicle's window(s) and turned on the vehicle's blower when the ambient temperature inside the vehicle exceeded 80 degrees Fahrenheit, and further assume that the three past instances met the threshold quantity of instances. In this example, the system may map the context scenario (e.g., the ambient temperature inside the vehicle being above 80 degrees Fahrenheit) to a completely different second portion of the lower dimensional space and / or a different second portion associated with the "hot" intent, assigning voice commands such as "hot" and "burning" to a completely different second portion of the lower dimensional space and / or a different second portion associated with the "hot" intent. However, based on the different fulfillment actions performed by the user in these three instances (e.g., lowering the vehicle's window(s) and turning on the vehicle's blower), these past instances may be mapped to an entirely different second portion of the lower dimensional space, but may be associated with the same "hot" intent due to the different fulfillment actions being performed in a slightly different, yet similar context scenario (e.g., when the ambient temperature inside the vehicle is above 80 degrees Fahrenheit, rather than between 70 and 80 degrees Fahrenheit). Thus, even if the user did not provide any voice utterances during the three past instances, when the user later provides a voice utterance of "hot" when the ambient temperature inside the vehicle is above 80 degrees Fahrenheit, the system may utilize the personalized mapping to automatically lower the window(s) and automatically turn on the blower.

[0013] While the above examples are described with respect to voice utterances received in a vehicular environment, it should be understood that this is for purposes of example and not intended to be limiting. For example, the techniques described herein may be utilized in any environment in which a given computing device is used by multiple users to perform fulfillment of voice utterances. For example, the techniques described herein may be utilized in a home environment, a work environment, a school environment, a commercial environment (e.g., hotels and / or other businesses), and / or any other environment. Additionally, it should be noted that the fulfillment actions performed to fulfill the voice utterances may vary from environment to environment. For example, fulfillment actions that may be performed in a home environment may include controlling various Internet of Things (IoT) devices and / or causing other devices to change state, while fulfillment actions that may be performed in the above-mentioned vehicular environment may include controlling various components and / or systems of the vehicle.

[0014] Additionally, while the above examples are described with respect to determining whether the user who provided the voice utterance is a first user or a second user, it should be understood that this is for purposes of example and not intended to be limiting. For example, the number of users considered in determining the user who provided the voice utterance may be based on the number of users associated with the computing device that receives the voice utterance (e.g., the on-vehicle computing device in the above examples). Furthermore, if it is determined that the user who provided the voice utterance does not correspond to any of the users associated with the computing device, the user may be considered to be a default user or a guest user, and default fulfillment model(s) and / or default fulfillment rule(s) (e.g., default mapping(s)) may be utilized to determine how to fulfill the voice utterance.

[0015] Furthermore, while the above example is described with respect to a first user and a second user providing the same voice utterance at different time instances and while each is driving a vehicle, it should be understood that this is for purposes of example and not intended to be limiting. For example, various environments may be partitioned into multiple zones and multiple users may be co-located within the environment but in different zones. For example, a vehicle environment may be partitioned into a "driver" zone, a "passenger" zone, etc., while a home environment may be partitioned into a "living room" zone, a "kitchen" zone, etc. In these instances, fulfillment actions may be limited to the particular zone occupied by the user who provided the voice utterance. For example, assume that a first user is driving a vehicle while a second user is a passenger in the vehicle. In this instance, when the first user provides a voice utterance of "hot", only the windows in the "driver" zone may be lowered, and when the second user provides a voice utterance of "hot", only the blower associated with the "passenger" zone may be turned on. In other instances, the windows may not be lowered and the blower may be considered a more conservative fulfillment action (e.g., the second user does not want the windows lowered, while the first user does not care about the blower), so only the blowers in the "driver" and "passenger" zones may be turned on. Additionally or alternatively, joint fulfillment action model(s) and / or joint fulfillment rule(s) may be utilized in this type of scenario.

[0016] By using the techniques described herein, various technical advantages can be achieved. As one non-limiting example, the techniques described herein enable a system to train fulfillment action model(s) and / or generate fulfillment rule(s) (e.g., personalized mapping(s)) of voice command(s) to fulfillment action(s) that are specific to each user based on respective user interaction(s) performed in respective context scenario(s). These fulfillment action model(s) and / or fulfillment rule(s) can later be utilized to personalize the fulfillment of voice utterances, such that different fulfillment action(s) can be performed for different users based on receiving the same voice utterance from different users. As a result, the amount of interactions and / or user inputs when an incorrect fulfillment is performed can be reduced, thereby saving computational resources at the computing device to undo the incorrect fulfillment and / or perform the correct fulfillment based on the reduced amount of subsequent interactions. Additionally, it may reduce the amount of natural resources that are wasted when incorrect fulfillment is performed.

[0017] The above description is provided as a summary of only some of the embodiments disclosed herein. These and other embodiments are described in more detail herein. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 illustrates various aspects of the present disclosure and shows a block diagram of an exemplary hardware and software environment in which embodiments disclosed herein may be implemented. [Diagram 2] 2 illustrates an example process flow for the example hardware and software environment of FIG. 1, in accordance with various embodiments. [Diagram 3]1 depicts a flowchart illustrating an example method for dynamically adapting fulfillment of a given voice utterance based on a user providing the given voice utterance, in accordance with various embodiments. [Figure 4] 1 shows a flowchart illustrating an example method for generating training instance(s) for training a fulfillment action model that is specific to a user, and training the fulfillment action model for subsequent use in dynamically adapting the fulfillment of a given voice utterance provided by a user, according to various embodiments. [Figure 5A] 13 illustrates various non-limiting examples of dynamically adapting the fulfillment of a given voice utterance in a vehicular environment based on the user who provided the given voice utterance, according to various implementations. [Figure 5B] 13 illustrates various non-limiting examples of dynamically adapting the fulfillment of a given voice utterance in a vehicular environment based on the user who provided the given voice utterance, according to various implementations. [Figure 6A] 13 illustrates various non-limiting examples of dynamically adapting the fulfillment of a given voice utterance in a primary residential environment based on the user who provided the given voice utterance, according to various implementations. [Figure 6B] 13 illustrates various non-limiting examples of dynamically adapting the fulfillment of a given voice utterance in a primary residential environment based on the user who provided the given voice utterance, according to various implementations. [Figure 7] 1 illustrates an exemplary architecture of a computing device in accordance with various embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] 1, an environment in which one or more selected aspects of the present disclosure may be implemented is shown. An exemplary environment includes a number of computing devices 110 1~NThese components 110 include a fulfillment system 120, a vehicle 100A, one or more original equipment manufacturer (OEM) applications 181, one or more first party applications 182, and one or more third party applications 183. 1~N , 120, 181, 182, and 183 may communicate, for example, over one or more networks, indicated generally at 195. The one or more networks may include a wired or wireless network, such as a local area network (LAN), including Wi-Fi, Bluetooth, near field communication, and / or other LANs, a wide area network (WAN), including the Internet, and / or any other network for facilitating communication between the components illustrated in FIG.

[0020] In various embodiments, a user may 1~N The computing device 110 may operate one or more of the following components to interact with the other components shown in FIG. 1~N may be, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a standalone speaker device optionally having a display, a vehicle-mounted computing device of the vehicle 100A (e.g., 110 N The vehicle may be equipped with a vehicle-mounted communication system, a vehicle-mounted entertainment system, and / or a vehicle-mounted navigation system as illustrated in FIG. 1, or a head mounted display ("HMD") providing an augmented reality ("AR") or virtual reality ("VR") immersive computing experience, or a wearable device including a computing device such as a "smart" watch. Additional and / or alternative computing devices may be provided.

[0021] Computing device 110 1~Nand fulfillment system 120 may each include one or more memories for storing data and software applications (e.g., one or more of OEM applications 181, one or more of first party applications 182, and / or one or more of third party applications 183), one or more processors for accessing the data and executing the software applications, and other components that facilitate communication over one or more of networks 195. 1~N And / or operations performed by one or more of fulfillment systems 120 may be distributed across multiple computer systems. For example, fulfillment system 120 may be implemented as a computer program executing solely on one or more computers or as a computer program distributed across one or more computers, e.g., at one or more locations communicatively coupled to each other via one or more of networks 195. Additionally or alternatively, computing device 110 1~N And / or the operations performed by one or more of fulfillment systems 120 may be performed locally on a computing system. For example, fulfillment system 120 (or an instance thereof) may be implemented by computing device 110. 1~N The above-mentioned configuration may be implemented locally in each of the above-mentioned systems.

[0022] Component 110 1~N One or more of the voice input devices 120, 181, 182, and 183 may include a variety of different components that may be used, for example, to dynamically adapt the fulfillment of a given voice utterance based on the user who provided the given voice utterance, as described herein. For example, the computing device 1101 may include a user interface engine 111 to detect and process user input (e.g., voice utterances, typed input, and / or touch input) directed to the computing device 1101. 1をAs another example, computing device 1101 may include one or more sensors 1121 to generate corresponding sensor data. The one or more sensors may include, for example, a global positioning system ("GPS") sensor for generating GPS data, a vision component for generating vision data within a field of view of the vision component, a microphone for generating audio data based on voice utterances captured in an environment of computing device 1101, and / or other sensors for generating corresponding sensor data.

[0023] As yet another example, the computing device 1101 may operate an input processing engine 1131 (which may be standalone or may be part of another application, such as part of an automated assistant application, for example) to process various user inputs received at the computing device 1101. For example, the input processing engine 1131 may capture voice utterances and process audio data generated by the microphone(s) of the client device 1101 using automatic speech recognition (ASR) model(s) (e.g., a recurrent neural network (RNN) model, a transformer model, and / or other ML models capable of performing ASR) to generate an ASR output. Additionally, the input processing engine 1131 may process the ASR output (or typed input) using natural language understanding (NLU) model(s) (e.g., a long short-term memory (LSTM), a gated recurrent unit (GRU), and / or any other type of RNN or other ML model capable of performing NLU) and grammar-based NLU rule(s) to generate an NLU output. Further, the input processing engine 1131 can process at least the NLU output using the fulfillment action model(s) and / or fulfillment rule(s) (e.g., personalized mapping) to obtain one or more candidate responses responsive to the user input, such as action(s) to be performed by the automated assistant based on the user input, content item(s) to be provided for presentation to the user based on the user input, etc. These fulfillment action model(s) and / or fulfillment rule(s) can be stored in the fulfillment action model(s) and / or rule(s) database 127A and can be trained and later utilized in the manner described herein.In implementations in which textual content is audibly rendered in response to spoken utterances or typed input, the user interface engine 1111 may process the textual content using text-to-speech model(s) to generate synthetic speech audio data including computer-generated synthetic speech capturing the content. The synthetic speech audio data may be audibly rendered for presentation to a user via speaker(s) of the computing device 1101. In implementations in which visual content is visually rendered in response to spoken utterances or typed input, the user interface engine 1111 may visually render the visual content for presentation to a user via a display of 1101.

[0024] In various implementations, the ASR output may include, for example, one or more speech hypotheses (e.g., term hypotheses and / or transcript hypotheses) predicted to correspond to the user's speech activity and / or speech utterances captured in the audio data, one or more corresponding predicted values ​​(e.g., probabilities, log-likelihoods, and / or other values) for each of the one or more speech hypotheses, a number of phonemes predicted to correspond to the user's speech activity and / or speech utterances captured in the audio data, and / or other ASR outputs. In some versions of these implementations, the input processing engine 1131 may cause one or more of the speech hypotheses to be selected as recognized text corresponding to the speech utterance (e.g., based on the corresponding predicted values).

[0025] In various implementations, the NLU output may include annotated recognized text, for example, including one or more annotations of the recognized text for one or more (e.g., all) of the terms of the recognized text. For example, the input processing engine 1131 may employ a speech tagger (not shown) configured to annotate terms with their grammatical roles. Additionally or alternatively, the input processing engine 1131 may employ an entity tagger (not shown) configured to annotate entity references in one or more segments of the recognized text. The entity references may include references to people (including, for example, literary characters, celebrities, public figures, etc.), organizations, places (real and virtual), etc. In some implementations, data about the entities may be stored in one or more databases, such as a knowledge graph (not shown). In some implementations, the knowledge graph may include nodes representing known entities (and, in some cases, entity attributes), and edges connecting the nodes and representing relationships between the entities. An entity tagger may annotate references to entities at a high level of granularity (e.g., to enable identification of all references to an entity class, such as person) and / or at a lower level of granularity (e.g., to enable identification of all references to a particular entity, such as a particular person). The entity tagger may rely on the content of the natural language input to resolve particular entities and / or may optionally communicate with knowledge graphs and / or other entity databases to resolve particular entities.

[0026] Additionally or alternatively, the input processing engine 1131 may employ a coreference resolver (not shown) configured to group, or "cluster," references to the same entity based on one or more context clues. As one non-limiting example, the coreference resolver may be utilized to resolve the term "that" in the natural language input "What's that light?" to a particular light or indicator associated with the operation of the vehicle 100A based on the corresponding sensor data instance generated by the vehicle sensor that caused the particular light or indicator associated with the operation of the vehicle 100A to be generated. In some implementations, one or more components utilized by the input processing engine 1131 may rely on annotations from one or more other components utilized by the input processing engine 1131. For example, in some implementations, an entity tagger may rely on annotations from a coreference resolver in annotating all mentions to a particular entity. Also, for example, in some implementations, a coreference resolver may rely on annotations from an entity tagger in clustering references to the same entity.

[0027] As yet another example, the computing device 1101 may operate a fulfillment system client 1141 (e.g., which may be standalone or part of another application, such as part of an automated assistant application) to interact with the fulfillment system 120. Additionally, additional computing devices 110 N may take the form of an on-board computing device of the vehicle 100A. Although not shown, additional computing devices 110 N may include the same or similar components as the computing device 1101. For example, the additional computing device 110 Nmay include a user interface engine for detecting and processing user input, one or more sensors for generating corresponding vehicle sensor data instances of the vehicle sensor data, an input processing engine, and / or respective instances of fulfillment system clients for interacting with fulfillment system 120. In this example, the one or more sensors may include tire pressure sensors for generating pressure data for tires of vehicle 100A, air flow sensors for generating air flow data for an air conditioning system of vehicle 100A, vehicle speed sensors for generating vehicle speed data for vehicle 100A, energy sensors for generating energy source data for an energy source of vehicle 100A, a transmission sensor for generating transmission data for a transmission of vehicle 100A, and / or vehicle on-board computing device 110 of vehicle 100A. N 1 also includes a computing device 1101 and a vehicle-mounted computing device 1102. N While only one computing device is shown, it should be understood that this is for the sake of example, and that additional or alternative computing devices may be provided (eg, as described with respect to FIGS. 6A and 6B).

[0028] In various embodiments, fulfillment system 120 may include interface engine 121, input processing engine 122, user interaction engine 123, context engine 124, user identification engine 125, training instance engine 126, training engine 127, and fulfillment engine 128, as shown in FIG. 1 . In some embodiments, one or more of engines 121-128 of fulfillment system 120 may be omitted. In some embodiments, all or aspects of one or more of engines 121-128 of fulfillment system 120 may be combined. In some embodiments, one or more of engines 121-128 of fulfillment system 120 may be integrated into computing device 110. 1~N In some embodiments, one or more of the engines 121-128 of the fulfillment system 120, or any operational portion thereof, may be implemented in components that execute partially or exclusively remotely from one or more of the computing device 110. 1~N The present invention may be implemented in a component that is executed partially or exclusively locally by one or more of the following:

[0029] Referring to Figure 2, an example process flow of the example hardware and software environment of Figure 1 is shown. For the sake of example, assume that audio data 201A capturing a user's voice utterance is received at the fulfillment system 120. The input processing engine 113 1~N The input processing engine 113 / 122 can process the audio data to generate processed input data 202. For example, the input processing engine 113 1~N The input processing engine 113 may process the audio data 201A using the ASR model to generate ASR data for the speech utterances captured in the audio data 201A. 1~NThe input processing engine 113 may process the ASR data of the speech utterance captured in the audio data 201A using the NLU model to generate NLU data of the speech utterance captured in the audio data 201A. 1~N / 122 can provide audio data 201A, ASR data of the voice utterances captured in the audio data 201A, and / or NLU data of the voice utterances captured in the audio data 201A to the fulfillment engine as processed input data 202.

[0030] Additionally, the user identification engine 125 can process the audio data 201A and / or non-audio data 201B (e.g., visual data capturing the user who provided the voice utterance, user profile data of a user profile of the user who provided the voice utterance, device identifier data of the user's computing device that received the voice utterance, etc.) to determine a user identity 225 of the user who provided the voice utterance. As described in more detail with respect to FIG. 3, the user identification engine can utilize audio-based identification techniques to determine the user identity 225 (e.g., text-dependent (TD) speaker identification (SID), text-independent (TI) SID) and / or can utilize non-audio-based identification techniques to determine the user identity 225 (e.g., facial recognition (FID), user profile identification, device identifier identification, etc.). Notably, in determining the user identity 225, the user identification engine 125 can utilize various user embeddings (e.g., stored in the user embedding(s) database 125A). These various embeddings can include TD speaker embeddings, TI speaker embeddings, face embeddings, and / or other embeddings. The generation and utilization of these embeddings in determining the user identity 225 is described in more detail herein (e.g., with respect to FIG. 3). Additionally, the user identification engine 125 can provide the user identity to the fulfillment engine 128.

[0031] Further, fulfillment engine 128 can process at least processed input data 202 and user identity 225 to determine one or more fulfillment actions to be performed in response to receiving the voice utterance. In some implementations, fulfillment engine 128 can process at least processed input data 202 and user identity 225 to determine a set of fulfillment actions. For example, fulfillment engine 128 can generate one or more structured requests and send one or more of the structured requests to various applications to obtain the fulfillment actions included in the set. For example, fulfillment engine 128 can generate one or more structured requests and send one or more of the structured requests to one or more OEM applications 181 (e.g., in an implementation in which the user is located within vehicle 100A), one or more first party applications 182, and / or one or more third party applications 183. As used herein, the term "first party application" may refer to a software application developed and / or maintained by the same entity that develops and / or maintains the fulfillment system 120 described herein. Additionally, as used herein, the term "third party application" may refer to a software application or system developed and / or maintained by an entity different from the entity that develops and / or maintains the fulfillment system 120 described herein. In response to one or more structured requests, one or more OEM applications 181 can send OEM data 281 to the fulfillment engine 128, one or more first party applications 182 can send first party data 282 to the fulfillment engine 128, and / or one or more third party applications 183 can send third party data 283 to the fulfillment engine 128.Based on the OEM data 281, the first party data 282, and / or the third party data 283, the fulfillment engine 128 can determine a set of fulfillment actions.

[0032] In these implementations, the fulfillment engine 128 can further process the set of fulfillment actions to determine one or more fulfillment actions to be performed in response to receiving the voice utterance. The fulfillment engine 128 can optionally further process context signals obtained via the context engine 124 (e.g., user signal(s) characterizing the state of the user and / or environmental signal(s) characterizing the state of the user's environment). For example, the fulfillment engine 128 can process one or more fulfillment actions to be performed in response to receiving the voice utterance and, optionally, the context signal from the set of fulfillment actions using a fulfillment action model and fulfillment rule(s) (e.g., stored in the fulfillment action model(s) and / or rule(s) database 127A) that are specific to the user who provided the voice utterance, as indicated by the user identity 225. Furthermore, the fulfillment engine 128 can cause one or more fulfillment actions to be performed to fulfill the voice utterance in response to receiving the voice utterance. Stated another way, in these implementations, fulfillment system 120 may initially determine a set of fulfillment actions using conventional techniques. However, fulfillment system 120 may later process the set of fulfillment actions to select one or more fulfillment actions using a fulfillment action model and / or fulfillment rule(s) that are specific to the user who provided the voice utterance, as indicated by user identity 225. This allows fulfillment system 120 to personalize the fulfillment of the voice utterance to the user who provided the voice utterance, as indicated by user identity 225.

[0033] In particular, a fulfillment action model specific to a user who provided the voice utterance as indicated by user identity 225 may be pre-trained and / or a fulfillment rule(s) specific to a user who provided the voice utterance as indicated by user identity 225 may be pre-generated. For example, prior to receiving the voice utterance captured in audio data 201A, user interaction engine 123 may identify a user interaction of a user with one or more smart devices at a given time instance of the multiple time instances. The user interaction may correspond to one or more fulfillment actions 223. In various implementations, the one or more fulfillment actions 223 of a user interaction may be limited to those that change the state of one or more of the smart devices in some way. User interaction engine 123 may provide one or more fulfillment actions 223 to training instance engine 126.

[0034] Additionally, the context engine 124 can obtain one or more context signals characterizing a state of the user, as indicated by user signal(s) 224A, and / or a state of the user's environment, as indicated by environmental signal(s) 224B, at a given time instance of the user interaction. The user signal(s) 224A and / or environmental signal(s) 224B can be utilized to define a context scenario, which can then be utilized in biasing the interpretation of subsequent speech utterances received in the context scenario. The context engine 124 can provide the user signal(s) 224A and the environmental signal(s) 224B to the training instance engine 126. These context signal(s) and context scenarios are described in more detail herein (e.g., with respect to FIG. 4). As described above, during inference, the context engine 124 can provide the context signals to the fulfillment engine 128.

[0035] The training instance engine 126 can process one or more fulfillment actions 223 corresponding to the user interaction, as well as the user signal(s) 224A and / or the environmental signal(s) 224B to generate the training instance 226. Furthermore, the training instance engine 126 can store the training instance 226 in a training instance(s) database 126A. Furthermore, the training engine 127 can train a fulfillment action model specific to the user who provided the voice utterance and / or generate one or more fulfillment action models specific to the user who provided the voice utterance based on the training instance 226 and, optionally, one or more additional training instances generated in the same or similar manner. Training a fulfillment action model specific to the user who provided the voice utterance and / or generating one or more fulfillment rules specific to the user who provided the voice utterance is described in more detail herein (e.g., with respect to FIG. 4).

[0036] 3, a flow chart is shown illustrating an example method 300 for dynamically adapting fulfillment of a given voice utterance based on a user providing the given voice utterance. For convenience, the method 300 operations are described with reference to a system that performs the operations. The system of method 300 may include at least one processor, at least one memory, and / or computing device(s) (e.g., computing device(s) 110 of FIGS. 1, 5A, 5B, 6A, and 6B). 1~N , fulfillment system 120 of FIG. 1, computing device 710 of FIG. 7, remote server(s), and / or other components of other computing device(s). Although the operations of method 300 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0037] At block 352, the system receives a given voice utterance from a user and via a computing device. For example, the user can provide the given voice utterance to an on-board computing device while the user is located within the user's vehicle, and the given voice utterance can be captured in the generated audio data via the microphone(s) of the on-board computing device (e.g., as described with respect to FIGS. 5A and 5B). Also, for example, the user can provide the given voice utterance to a mobile computing device or a standalone computing device while the user is located within the user's primary residence, and the given voice utterance can be captured in the generated audio data via the microphone(s) of the mobile computing device or the standalone computing device (e.g., as described with respect to FIGS. 6A and 6B).

[0038] At block 354, the system determines a set of fulfillment actions based on processing the given speech utterance. Each fulfillment action included in the set of fulfillment actions, when performed, facilitates the given speech utterance. For example, as shown at block 354A, the system may process audio data capturing the given speech utterance using an ASR model to generate ASR data for the given speech utterance. Further, as shown at block 354B, the system may process the ASR data for the given speech utterance using an NLU model to generate NLU data for the given speech utterance. For example, assume that a given speech utterance provided by a user corresponds to "cold". In this instance, the system may process audio data capturing the speech utterance to generate a recognized text of "cold" as the ASR data for the given speech utterance. Further, the system may process the recognized text of "cold" to identify one or more intents for the given speech utterance, such as intents of "cold", "increase heat", etc. Further, the system may determine a set of fulfillment actions based on at least one or more intents. In this instance, assuming the given voice utterance is received in a vehicle environment, the set of fulfillment actions may include, for example, raising a window(s) of the vehicle, turning on a heated seat of the vehicle, turning on a blower of the vehicle, and / or any other action that may be performed to increase the temperature inside the vehicle. In contrast, assuming the given voice utterance is received in a home environment, the set of fulfillment actions may include, for example, raising a thermostat temperature of a primary residence to a first temperature, raising a thermostat temperature of the primary residence to a second temperature, and / or any other action that may be performed to increase the temperature inside the primary residence, corresponding to the home environment.

[0039] At block 356, the system determines whether the user who provided the given speech utterance corresponds to a first user or a second user. The system may utilize various techniques for determining whether the user who provided the given speech utterance corresponds to a first user or a second user. In some implementations, the system may perform speaker identification (SID) to determine whether the user who provided the given speech utterance corresponds to a first user or a second user. For example, the system may use the SID model(s) to process audio data capturing the given speech utterance to generate a speaker embedding for the user who provided the given speech utterance. The generated speaker embedding can be compared in the embedding space to one or more previously generated speaker embeddings for one or more users of the computing device (e.g., embeddings associated with user profiles of one or more users). For example, if a distance criterion determined based on a comparison in an embedding space between the generated speaker embedding and a given one of the one or more previously generated speaker embeddings for one or more users meets a threshold, then the user may be determined to correspond to a user associated with the given one of the one or more previously generated speaker embeddings. Otherwise, the user may be identified as a guest user.

[0040] In these implementations, the one or more previously generated speaker embeddings for one or more users can be generated based on output(s) generated based on processing one or more instances of audio data including speech utterances, respectively, from one or more users. For example, the one or more previously generated speaker embeddings may be based on an average (or other combination) of multiple different embeddings, each generated based on processing corresponding instances of audio data including corresponding speech utterances, respectively, from one or more users. Further, in these implementations, the SID model may be a text-independent (TI) SID model, a text-dependent (TD) SID model, and / or any other suitable SID model.

[0041] For example, in TD SID, one or more previously generated speaker embeddings of one or more users are generated based on a speech utterance that includes only one or more specific words or phrases. Furthermore, in use, one or more users may be required to speak one or more specific words or phrases for one or more TD SID speaker embeddings generated using the TD SID model that can be effectively compared to one or more previously generated TD speaker embeddings of one or more users to determine whether the user providing the given speech utterance corresponds to a first user or a second user. For example, one or more words or phrases in TD speaker recognition may be constrained to one or more invocation phrases (e.g., hot words and / or trigger words such as "hey, assistant", "okay, assistant", and / or "assistant") configured to invoke an automated assistant or one or more other words or phrases. In contrast, in TI SID, speech utterances processed using the TI SID model are not constrained to one or more specific words or phrases. In other words, audio data based on any hypothetical spoken utterance can be processed using the TI SID model to generate a TI SID speaker embedding, which can then be effectively compared to one or more previously generated TI speaker embeddings of one or more users to determine whether the user providing the given speech utterance corresponds to a first user or a second user. Further, in various embodiments, the one or more previously generated TI speaker embeddings of one or more users utilized in the TI SID are generated based on speech utterances that include entirely different words and / or phrases, words and / or phrases of a call, and / or speech utterances that are not limited to any other particular speech utterance.

[0042] In additional or alternative implementations, the system may perform other forms of identification to determine whether the user who provided the given voice utterance corresponds to a first user or a second user. For example, the system may perform face identification, fingerprint identification, and / or other types of biometric authentication. For example, a vision component of the client device may capture visual data including the face of the user who provided the voice utterance. In this instance, the system may use a face identification (FID) model to process the visual data to generate a face embedding of the user who provided the voice utterance. The face embedding of the user who provided the voice utterance may correspond to an embedding that can be compared in the embedding space to one or more previously generated embeddings that correspond to face embeddings of one or more users of the computing device to determine whether the user who provided the given voice utterance corresponds to the first user or the second user. Also, for example, a fingerprint sensor may receive touch input of the user who provided the given voice utterance and compare a detected fingerprint to known fingerprints of one or more users of the computing device to determine whether the user who provided the given voice utterance corresponds to the first user or the second user. Also, for example, a display interface of the client device can receive a user touch input corresponding to a particular password of the client device to determine whether a user who provided a given voice utterance corresponds to a first user or a second user. As another example, the system can utilize device identifiers and / or user accounts associated with the computing device and / or additional computing devices in communication with the computing device to determine whether a user who provided a given voice utterance corresponds to a first user or a second user based on inferring the user from the device identifier and / or user account.

[0043] At block 358, the system determines how to dynamically adapt the fulfillment of the given voice utterance based on determining from block 356 whether the user providing the given voice utterance corresponds to the first user or the second user. For example, during an iteration of block 358, if the system determines that the user providing the given voice utterance corresponds to the first user, the system may proceed to block 360. At block 360, the system selects a subset of one or more first fulfillment actions from the set of fulfillment actions from block 354 based on the user providing the given voice utterance corresponding to the first user. In some implementations, the system may select the subset of one or more first fulfillment actions based on context signals characterizing a state of the user and / or a state of the user's environment. At block 362, the system causes the subset of one or more first fulfillment actions to be performed to fulfill the given voice utterance. The system can select a subset of one or more first fulfillment actions from the set of fulfillment actions using a fulfillment action model and / or fulfillment rule(s) (e.g., personalized mapping(s)) that is specific to the first user. For example, the system can process NLU data for a given voice utterance and / or a set of fulfillment actions from block 354, and optionally a context signal characterizing a state of the user who provided the given voice utterance and / or a state of the environment in which the user is located when the given voice utterance is provided, using a fulfillment action model and / or fulfillment rule(s) that is specific to the first user to select one or more first fulfillment actions for inclusion in the subset. Generating and utilizing a fulfillment action model(s) and / or fulfillment rule(s) (e.g., personalized mapping) that is specific to a given user is described in more detail with respect to FIG. 4.

[0044] However, upon iteration of block 358, if the system determines that the user who provided the given voice utterance corresponds to a second user, the system may proceed to block 364. At block 364, the system selects a subset of one or more second fulfillment actions from the set of fulfillment actions from block 354 based on the user who provided the given voice utterance corresponding to a second user. At block 366, the system causes the subset of one or more second fulfillment actions to be performed to fulfill the given voice utterance. Similarly, the system may select a subset of one or more second fulfillment actions from the set of fulfillment actions using a fulfillment action model and / or fulfillment rule(s) that are specific to the second user. For example, the system may process NLU data for a given voice utterance and / or a set of fulfillment actions from block 354, and optionally a context signal characterizing a state of the user who provided the given voice utterance and / or a state of the environment in which the user is located when the given voice utterance is provided, using a fulfillment action model and / or fulfillment rule(s) specific to a second user to select one or more second fulfillment actions for inclusion in the subset. As mentioned above, generating and utilizing a fulfillment action model(s) and / or fulfillment rule(s) specific to a given user is described in more detail with respect to FIG. 4. Notably, the subset of one or more second fulfillment actions may include at least one unique fulfillment action from the set of fulfillment actions compared to the subset of one or more first fulfillment actions. In other words, the fulfillment of the same voice utterance may vary depending on the user who provided the voice utterance.

[0045] Continuing with the above example where a given user provided voice utterance corresponds to "cold", further assume that the given voice utterance was provided by a first user. In this example, assuming that the given voice utterance was provided in a vehicle environment, then a fulfillment action associated with turning on the heated seats for the first user may be selected and implemented since the fulfillment action model and / or fulfillment rule(s) specific to the first user may have learned that the first user typically turns on only the heated seats of the vehicle when it is cold, and not the blower. Further assuming that the system has access to various environmental signals characterizing the state of the vehicle, and that the environmental signals indicate that the windows of the vehicle are lowered when the voice utterance is received, then a fulfillment action associated with raising the windows may be selected and implemented since the fulfillment action model and / or fulfillment rule(s) specific to the first user may have learned that the first user typically raises the windows when it is cold if the windows are lowered. Also in this example, assuming that a given voice utterance was provided in a home environment, then a fulfillment action associated with raising the thermostat temperature by N degrees (e.g., N is a positive integer) or raising the thermostat temperature to a first particular temperature (e.g., 74 degrees) may be selected and implemented because the fulfillment action model and / or fulfillment rule(s) specific to the first user may have learned that this is how the first user typically utilizes the thermostat when it is cold.

[0046] In contrast, further assume that the given voice utterance was provided by a second user, in this example, in a vehicle environment, then a fulfillment action model and / or fulfillment rule(s) specific to the second user may have learned that this second user typically only turns on both the heated seats and the blower in the vehicle when it is cold, and thus a fulfillment action associated with turning on the heated seats and turning on the blower for the second user may be selected and implemented. Also in this example, assuming that a given voice utterance was provided in a home environment, then a fulfillment action associated with raising the thermostat temperature by M degrees (e.g., M is a positive integer different from N) or raising the thermostat temperature to a second particular temperature (e.g., 75 degrees) may be selected and implemented because the fulfillment action model and / or fulfillment rule(s) specific to the second user may have learned that this is how the second user typically utilizes the thermostat when it is cold.

[0047] From blocks 362 and 366, the system returns to block 352 to perform additional iterations of the method 300 of FIG. 3 based on additional given voice utterances received via the computing device. Thus, when a different user provides the same voice utterance (e.g., "cold" in the above example), the system can dynamically adapt the fulfillment of that same voice utterance based on how different users would typically perform different actions in the same or similar context scenarios. As a result, the fulfillment of the given voice utterance is improved, thereby reducing the amount of user interactions and / or the amount of user input received to perform a desired fulfillment action in instances where an incorrect fulfillment action was initially performed, since the fulfillment of the given voice utterance may not be specific to the user who provided the given voice utterance.

[0048] In various implementations, the system may cause an indication of why one or more fulfillment actions were selected and performed to satisfy a given voice utterance of the corresponding user to be provided for audible and / or visual display to the corresponding user via the computing device. The indication of why one or more fulfillment actions were selected and performed to satisfy the given voice utterance may be generated using the fulfillment action model(s) and / or the fulfillment rule(s). Continuing with the above example, assuming the given voice utterance was provided by a first user in a vehicular environment, the indication of why one or more fulfillment actions were selected and performed to satisfy the given voice utterance may correspond to "I turned on your heated seats but I didn't turn on the blower because that's what you normally do when it's cold in a car." Synthetic speech audio data including the synthesized speech capturing the indication may be audibly rendered for presentation to the first user via the speaker(s) of the computing device, and / or text data corresponding to the indication may be visually rendered for presentation to the first user via a display of the computing device. In particular, an indication of why one or more fulfillment actions were selected and performed to satisfy a given voice utterance may only be provided for presentation a threshold number of times (e.g., at least once) to educate the corresponding user, but may avoid flooding and confusing the user with the same information.

[0049] In various implementations, the system may generate a prompt requesting the user to provide feedback regarding the selected and performed one or more fulfillment actions, and cause the prompt to be provided for presentation to the user. Based on user input received from the user in response to the prompt, the system may cause the fulfillment action model(s) and / or fulfillment rule(s) specific to the user to be updated. For example, the system may generate the above indication (e.g., "You turned on your heated seats, but not the blower, because that's what you usually do when it's cold in the car") and may also or instead generate a prompt saying, "Is that right?" In this example, if a positive user input is received, then the system may generate a positive training instance to bias subsequent fulfillment of the same voice utterance toward the selection and performance of the one or more first fulfillment actions. However, if a negative user input is received, then the system may generate a negative training instance to bias subsequent fulfillment of the same voice utterance away from the selection and performance of the one or more first fulfillment actions.

[0050] In additional or alternative embodiments, the system may utilize one or more subsequent user interactions as feedback regarding the selected and implemented one or more fulfillment actions. For example, if the system selects and implements a fulfillment action associated with turning on the heated seats for a first user based on a given voice utterance "It's cold" provided by a first user, but the first user turns off the heated seats following implementation of the fulfillment action, then the system can generate negative training instances to bias subsequent fulfillment of the same voice utterance away from selecting and implementing one or more first fulfillment actions (e.g., the fulfillment action associated with turning on the heated seats for the first user). In contrast, if the system selects and performs a fulfillment action associated with turning on the heated seats for a first user based on a given voice utterance "it's cold" provided by the first user, and the first user does not turn off the heated seats following performance of the fulfillment action, then the system can generate positive training instances to bias subsequent fulfillment of the same voice utterance toward the selection and implementation of one or more first fulfillment actions (e.g., fulfillment actions associated with turning on the heated seats for the first user) based on inferring that the fulfillment action was correctly performed in response to the given voice utterance.

[0051] It should be understood that while FIG. 3 is described herein with respect to a particular utterance and a particular fulfillment action, that is by way of example and not by way of limitation. Additionally, it should be understood that while FIG. 3 is described herein with respect to only a first user and a second user, that is by way of example and not by way of limitation. For example, the number of users taken into consideration when determining which user provided a given voice utterance may be based on the number of users associated with the computing device. If the user provided a given voice utterance is not determined to correspond to any known user, then the user may be considered a default user, and fulfillment action model(s) and / or fulfillment rule(s) that are not specific to any user may be utilized in dynamically adapting the fulfillment of the given voice utterance.

[0052] 4, a flow chart is shown illustrating an example method 400 of generating training instance(s) for training a fulfillment action model that is specific to a user, and training the fulfillment action model for subsequent use in dynamically adapting the fulfillment of a given voice utterance provided by a user. For convenience, the method 400 operations are described with reference to a system that performs the operations. The system of method 400 includes at least one processor, at least one memory, and / or computing device(s) (e.g., computing device(s) 110 of FIGS. 1, 5A, 5B, 6A, and 6B). 1~N , fulfillment system 120 of FIG. 1, computing device 710 of FIG. 7, remote server(s), and / or other components of other computing device(s). Although the operations of method 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0053] At block 452, the system identifies an occurrence of a user interaction of the user with one or more smart devices at a given time instance of the plurality of time instances, the user interaction corresponding to one or more fulfillment actions. The system is communicatively coupled to the one or more smart devices and may consider any user interaction that changes a state of the one or more smart devices as an occurrence of a user interaction. Furthermore, the one or more fulfillment actions may correspond to any particular action performed by the user that changes a state of the one or more smart devices. The one or more smart devices may include, for example, the user's computing device (e.g., a mobile computing device, a vehicle-mounted computing device, etc.), vehicle components of the user's vehicle (e.g., a window defroster, power windows and doors, a radio, a blower, heated seats, etc.), the user's Internet of Things (IoT) device (e.g., a smart TV, a smart thermostat, a smart alarm system, a smart appliance, etc.), and / or any other smart device with which the user may interact and thereby change a state of the smart device.

[0054] At block 454, the system obtains one or more context signals characterizing the state of the user at a given time instance and / or the state of the user's environment at the given time instance. In other words, the system obtains one or more context signals at the same given time instance at which an occurrence of a user interaction is identified to determine a context scenario that caused the user to perform one or more fulfillment actions of the user interaction. The one or more context signals characterizing the user's state at a given time instance may be based on user profile data of the user's user profile(s), application data of application(s) accessible to the system, sensor data instances of sensor data generated by one or more sensors in the user's environment (e.g., vehicle sensor(s), computing device sensor(s), home sensor(s), etc.), and / or any other context signals characterizing the user's state. Additionally, the state of the user determined based on the one or more context signals may include, for example, the user's location or activity (e.g., whether the user is located at the user's primary residence, whether the user is driving a vehicle, etc.), the time of day at the user's location, the day of the week at the user's location, and / or any other state of the user determined based on the one or more context signals. Additionally, the one or more context signals characterizing the state of the user's environment at a given time instance may be based on sensor data instances of sensor data generated by one or more sensors in the user's environment (e.g., vehicle sensor(s), computing device sensor(s), home sensor(s), etc.) and / or any other context signals characterizing the state of the user's environment.Additionally, the conditions of the user's environment determined based on the one or more context signals may include, for example, ambient noise detected in the user's environment, ambient indoor and / or outdoor temperature in the user's environment, whether other users are co-located in the user's environment (e.g., within a primary residence of the user, a passenger in a vehicle, etc.), and / or any other conditions of the user's environment determined based on the one or more context signals.

[0055] At block 456, the system generates a given user training instance based on the user interaction at the given time instance and based on the one or more context signals at the given time instance. Furthermore, the system can store the given user training instance in a training instance(s) database (e.g., training instance(s) database 126A of FIG. 1). The given training instance can include training instance inputs and training instance outputs. The training instance inputs can include, for example, (i) one or more context signals that characterize a state at the user at the given time instance and / or characterize a state of the user's environment at the given time instance, and (ii) a set of fulfillment actions associated with the one or more context signals. Among other things, the set of fulfillment actions includes at least one or more fulfillment actions corresponding to the user interaction and at least one additional fulfillment action. In some implementations, the training instance inputs can include one or more terms or phrases that describe the one or more context signals and that may be subsequently provided by the user to describe a low ambient temperature context scenario (e.g., "cold," "freezing," "chilly," etc.). Stated another way, the training instance input may include a given context scenario, as indicated by one or more context signals, and a set of fulfillment actions that are predicted to be relevant for the given context scenario. The set of fulfillment actions that are predicted to be relevant for the given context scenario may be defined by a developer, a user, and / or another party associated with the system. For example, if the ambient temperature of the user's environment, as indicated by one or more context signals, is less than X degrees (e.g., X is a positive integer), then the set of fulfillment actions may include an action that, when performed, promotes raising the ambient temperature of the user's environment.Further, the training instance output may include one or more fulfillment actions of the user interaction. In particular, the one or more fulfillment actions of the user interaction may include each of the fulfillment actions of the set of fulfillment actions or the subset thereof. In other words, the training instance output may include, in effect, the fulfillment actions performed by the user in a given context scenario.

[0056] In some implementations, as shown in block 456A, before generating a given user training instance, the system prompts the user to verify whether one or more fulfillment actions have been performed based on the state of the user at the given time instance and / or based on the state of the user's environment at the given time instance. For example, assume that a user manually turns on a heated seat in a vehicle. The system can identify the occurrence of the user manually turning on the heated seat as a user interaction that includes a fulfillment action to increase the ambient temperature of the user's environment. Additionally, the system can identify the ambient temperature of the user's environment (e.g., inside the vehicle and / or outside the vehicle) as one or more context signals at the given time instance at which the user manually turned on the heated seat. In these implementations, before generating a given training instance based on this user interaction, the system can prompt the user to verify whether the user manually turned on the heated seat because he was cold. In additional or alternative implementations, the system can simply infer this without prompting the user.

[0057] At block 458, the system determines whether one or more conditions for training a fulfillment action model that is specific to the user are met. The one or more conditions for training a fulfillment action model that is specific to the user may include, for example, whether there is a threshold quantity of training instances available for the fulfillment action model, time of day, day of week, and / or other conditions. For example, in an embodiment in which the system is implemented locally on the user's computing device, the one or more conditions may also or instead include whether the computing device is charging, whether the computing device has at least a threshold state of charge, whether the temperature of the computing device is below a threshold, whether the computing device is being held by the user. If, upon an iteration of block 458, the system determines that the one or more conditions are not met, the system proceeds to block 460.

[0058] At block 460, the system determines whether there is a subsequent occurrence of the user interaction (e.g., including the same one or more fulfillment actions) or whether there is an additional user interaction in addition to the user interaction (e.g., including at least one unique fulfillment action compared to the one or more fulfillment actions of the user interaction). If, during an iteration of block 460, the system determines that there is a subsequent occurrence of the user interaction or an additional user interaction, the system may return to block 452 and perform additional iterations of the operations of blocks 452-458 with respect to the subsequent occurrence of the user interaction or the additional user interaction to generate additional given user training instances and determine whether one or more conditions are met to train a fulfillment action model that is specific to the user. If, during an iteration of block 460 and / or subsequent iterations, the system determines that there is no subsequent occurrence of the user interaction or an additional user interaction, the system may return to block 458. Further, upon any iteration of block 458, if the system determines that one or more conditions are satisfied, the system proceeds to block 462. In other words, the system may continuously monitor for occurrences of user interactions during the training phase and generate training instances based on those user interactions and / or may continuously monitor for one or more conditions being satisfied until the system determines to train fulfillment actions based on the training instances.

[0059] At block 462, the system trains a fulfillment action model that is specific to the user based on at least the given training instance. Additionally, the system can store the trained fulfillment action model in one or more databases (e.g., the fulfillment action model(s) and / or the rule(s) database 127A) accessible to the system. At block 464, the system utilizes the fulfillment action model that is specific to the user (e.g., as described with respect to FIG. 3) in responding to the voice utterance received from the user. The fulfillment action model can be any suitable machine learning (ML) model that can be trained in any suitable manner, including, for example, a classification model (e.g., a support vector machine (SVM) model, a K-nearest neighbor (KNN) model, a Naive Bayes model, etc.), a regression model, and / or other types of ML models. For example, when training a fulfillment action based on the given training instance generated at block 456, the system can use the fulfillment action model to process the set of one or more context signals and fulfillment actions included in the training instance input to generate a predicted output. In this example, the prediction output may correspond to one or more fulfillment actions from a set of fulfillment actions that the user is predicted to perform based on a given context scenario defined by one or more context signals. Furthermore, the one or more predicted fulfillment actions (or representations thereof) of the prediction output can be compared to one or more fulfillment actions (or representations thereof) of the user interactions included in the training instance output to generate one or more losses. Furthermore, the fulfillment action model can be updated based on the one or more losses. For example, the one or more losses can be backpropagated through the fulfillment action model to update its weight(s).Notably, this process can be repeated for multiple additional training instances generated by the methods described herein to further train and personalize fulfillment action models that are specific to the user. Additionally, this process can be repeated for multiple different users to train and personalize corresponding fulfillment action models that are specific to those users.

[0060] In additional or alternative implementations, the system can present various context scenarios for presentation to the user and request that the user identify one or more fulfillment actions that the user would perform given the various context scenarios. In these implementations, the system can generate training instances for training a fulfillment action model and / or generating fulfillment rule(s) based on how the user responds to the various context scenarios. For example, the system can present that the ambient temperature of the vehicle is low (e.g., a context scenario in which the user would be cold) and list various actions that the user can perform in response to the ambient temperature of the vehicle being low (e.g., training instance inputs for a given training instance as described above). Additionally, the system can utilize one or more user selections of actions as training instance outputs.

[0061] In various implementations, the fulfillment action model and / or the fulfillment rule(s) can be updated over time. For example, the system can continue to generate training instances based on occurrences of user interactions identified at various time instances and / or one or more context signals acquired at those various time instances. The system may update the fulfillment action model and / or the fulfillment rule(s) whenever one or more conditions are determined to be subsequently met. Also, for example, the system can update the fulfillment action model and / or the fulfillment rule(s) based on changing context scenarios. For example, during the fall season, a user may raise one or more windows of a vehicle when it is cold, but may not turn on any heated seats. However, as the fall season turns into the winter season, the user may never lower the windows, but may turn on the heated seats. Thus, the fulfillment action model and / or the fulfillment rule(s) are not only specific to a user at a given time instance, but are updated over time to ensure better fulfillment of voice utterances for the user.

[0062] In various implementations, the system can assign one or more voice commands to the training instance and / or fulfillment rule(s) so that the system can later determine that voice utterances received in the same or similar context scenarios map to different fulfillment actions for different users. For example, the one or more voice commands can include one or more terms or phrases that describe one or more context signals and that may later be provided by the user. For example, terms such as "cold," "freezing," and "chilly" can be assigned to the training instance and / or fulfillment rule(s) associated with a context signal that indicates a low ambient temperature of the user's environment when one or more fulfillment actions of the user interaction are performed to increase the low ambient temperature in the environment. Also, for example, terms such as "hot," "scorching," and "scorching" can be assigned to the training instance and / or fulfillment rule(s) associated with a context signal that indicates a high ambient temperature of the user's environment when one or more fulfillment actions of the user interaction are performed to decrease the high ambient temperature in the environment. Also, for example, terms such as "dark," "dim," "complete darkness," etc. may be assigned to training instances and / or fulfillment rule(s) associated with a context signal indicating that an environment has low ambient light when one or more fulfillment actions of a user interaction are performed to increase the ambient light of the environment. Thus, it should be appreciated that the techniques described herein can be utilized not only to train fulfillment action model(s) and / or train fulfillment rule(s) as described herein, but also to assign one or more voice commands to particular context scenarios that a user may encounter.

[0063] In additional or alternative embodiments, the system can generate a personalized mapping (e.g., also referred to herein as fulfillment rule(s)) for the user that maps one or more voice commands to one or more fulfillment actions. For example, the system can generate the personalized mapping based on one or more past instances of the user causing the execution of one or more fulfillment actions corresponding to a user interaction in a context scenario and independent of providing a voice utterance. Furthermore, the system can use various techniques to determine that a context scenario correlates to one or more voice commands. For example, the system can map different context scenarios represented by different context signals that characterize the state of the user and / or the state of the environment when the one or more fulfillment actions are performed to different portions of a lower dimensional space (e.g., an embedding space, a semantic space, etc.). Furthermore, one or more commands can be assigned to different context scenarios in the lower dimensional space. Thus, when a given voice command is subsequently included in a given voice utterance received from a user, the system can determine a given context scenario associated with the given voice command in the lower dimensional space and cause one or more given fulfillment actions previously performed in the given context scenario to be performed to fulfill the given voice utterance. Also, for example, the system can map different context scenarios represented by different context signals characterizing a state of the user and / or characterizing a state of the environment when the one or more fulfillment actions are performed to different portions of the lower dimensional space (e.g., an embedding space, a semantic space, etc.). Additionally, one or more voice commands can be assigned to different intents.Thus, when the given voice command is later included in a given voice utterance received from a user, the system can determine a given context scenario associated with the given voice command determined based on the given intent, and cause one or more given fulfillment actions previously performed in the given context scenario to be performed to satisfy the given voice utterance.

[0064] In some versions of those implementations, the system may only generate the personalized mapping in response to determining that one or more fulfillment actions have been performed by the user a threshold quantity of instances (e.g., 1 time, 3 times, 5 times, and / or 1 or any other positive integer greater than 1) in the context scenario. For example, assume that there were three past instances in which the user turned on the heated seats of the vehicle but did not turn on the blower of the vehicle when the ambient temperature inside the vehicle was between 45 degrees Fahrenheit and 60 degrees Fahrenheit, and assume that the three past instances meet the threshold quantity of instances. In this example, the system may map the context scenario (e.g., the ambient temperature inside the vehicle is between 45 degrees Fahrenheit and 60 degrees Fahrenheit) to a first portion of the lower dimensional space and / or a first portion associated with the “cold” intent, and assign voice commands such as “cold” and “freezing” to the first portion of the lower dimensional space and / or the first portion associated with the “cold” intent. Thus, even if the user did not provide any voice utterances during the three previous instances, if the user later provides the voice utterance "it's cold" when the ambient temperature inside the vehicle is between 45 degrees Fahrenheit and 60 degrees Fahrenheit, the system can utilize the personalized mapping to automatically turn on the heated seats.

[0065] Notably, in these embodiments, the same voice command may be mapped to different fulfillment actions. Continuing with the above example, assume further that there were three past instances where the user turned on the heated vehicle seats and the vehicle blower when the ambient temperature inside the vehicle was 45 degrees Fahrenheit or less, and further assume that the three past instances met the threshold quantity of instances. In this example, the system may map the context scenario (e.g., the ambient temperature inside the vehicle is 45 degrees Fahrenheit or less) to a completely different second portion of the lower dimensional space and / or a different second portion associated with the "cold" intent, and assign voice commands such as "cold," "freezing," etc. to a completely different second portion of the lower dimensional space and / or a different second portion associated with the "cold" intent. However, based on the different fulfillment actions the user performs in these three instances (e.g., turning on both the heated vehicle seats and the vehicle blower), these past instances may be mapped to completely different second portions of the lower dimensional space, but associated with the same "cold" intent. Thus, even if the user did not provide any voice utterances during the three previous instances, if the user later provides the voice utterance "it's cold" when the ambient temperature inside the vehicle is below 45 degrees Fahrenheit, the system can utilize the personalized mapping to automatically turn on the heated seats and automatically turn on the blower.

[0066] It should be understood that while Figure 4 is described herein with respect to a particular user interaction having a particular fulfillment action, that is by way of example and not by way of limitation. Additionally, it should be understood that while Figure 4 is described herein with respect to training a fulfillment action model and / or generating fulfillment rule(s) for a single user, that is by way of example and not by way of limitation. For example, it should be understood that corresponding iterations of method 400 of Figure 4 can be performed to train corresponding fulfillment action models and / or generate corresponding fulfillment rule(s) for multiple other users.

[0067] 5A and 5B, various non-limiting examples are shown for dynamically adapting the fulfillment of a given voice utterance in a vehicular environment based on the user who provided the given voice utterance. For purposes of example, assume that a vehicle 100A is configured to dynamically adapt the fulfillment of a given voice utterance in a vehicular environment based on the user who provided the given voice utterance. N Assume that the vehicle includes an on-board computing device 110A, and that the vehicle is at least partially shared by multiple users (a first user 102 shown in FIG. 5A and a second user 104 shown in FIG. 5B). Assume further that action fulfillment models have been previously trained for both the first user 102 and the second user 104, and / or fulfillment rules have been previously generated for both the first user 102 and the second user 104. Thus, the vehicle 100A includes at least a portion of the on-board computing device 110A. N A fulfillment system running on the voice utterance server (e.g., fulfillment system 120 of FIG. 1) can utilize these fulfillment models and / or fulfillment rules that are specific to first user 102 and second user 104 when determining how to dynamically adapt the fulfillment of a voice utterance.

[0068] 5A, and further assuming that the first user 102 is driving the vehicle 100A and provides a voice utterance 552A of “it’s cold,” the on-board computing device 110 of the vehicle 100A N Assume further that the microphone(s) generate audio data capturing the speech utterance 552A. In this example, the audio data capturing the speech utterance 552A can be processed using an ASR model to generate ASR data for the speech utterance 552A (e.g., the recognized text of "cold"). Additionally, the ASR data for the speech utterance 552A can be processed using an NLU model to generate NLU data for the speech utterance 552A (e.g., the recognized intent(s) of "cold", "turn up the temperature", etc.). Notably, the user who provided the speech utterance 552A can be determined to correspond to the first user 102 using any of the techniques described herein and / or any other suitable techniques.

[0069] In some implementations, the NLU data for the voice utterance 552A can be processed to determine a set of fulfillment actions that, when performed, facilitate increasing the temperature inside the vehicle 100A. The set of fulfillment actions can include, for example, raising a window of the vehicle 100A (if the window of the vehicle 100A is in a state indicating that the window is lowered) to increase the temperature inside the vehicle 100A, turning on a blower of the vehicle 100A to circulate warm air throughout the vehicle to increase the temperature inside the vehicle 100A, turning on a heated seat for a seat occupied by the first user 100A to increase the temperature inside the vehicle 100A and / or the temperature of the first user 102 located within the vehicle 100A, and / or other fulfillment actions that can be performed to increase the temperature inside the vehicle 100A and / or the temperature of the first user 102 located within the vehicle 100A.

[0070] In this implementation, the set of fulfillment actions can be processed using a fulfillment action model and / or fulfillment rule(s) (e.g., personalized mapping(s)) that is specific to the first user 102 to select one or more first fulfillment action(s) to be performed to satisfy the voice utterance 552A. The voice utterance and / or one or more contextual signals obtained at the time instance at which the voice utterance was provided by the first user 102 can additionally or alternatively be processed using a fulfillment action model and / or fulfillment rule(s) that is specific to the first user 102 in selecting one or more first fulfillment actions to be performed to satisfy the voice utterance 552A. For example, the one or more first fulfillment actions may include turning on a heated seat occupied by the first user 102 to increase the temperature inside the vehicle 100A and / or the body temperature of the first user 102 located in the vehicle 100A, but may not include turning on a blower in the vehicle 100A to circulate warm air throughout the vehicle to increase the temperature inside the vehicle 100A based on the output generated using a fulfillment action model and / or fulfillment rule(s) that are specific to the first user 102 (e.g., by the vehicle-onboard computing device 110 N (as indicated by synthesized voice 554A saying, "I turned on your heated seats, but not the blower," which is audibly presented to the user via speaker(s) of the first user 102. In other words, based on the output generated using the fulfillment action model and / or fulfillment rule(s) that are specific to the first user 102, the fulfillment of the voice utterance 552A is dynamically adapted to the fulfillment action(s) that the first user 102 would manually perform in the same or a similar context scenario.

[0071] In further or alternative implementations, the one or more first fulfillment actions to be performed to satisfy the voice utterance 552A can be selected directly without considering a set of fulfillment actions (e.g., using a fulfillment rule(s)). For example, the one or more first fulfillment actions can be mapped to various voice commands that, when detected in a particular context scenario, cause the one or more first fulfillment actions to be performed. For example, in generating one or more fulfillment rules specific to the first user 102, a context scenario matching the scenario shown in FIG. 5A (e.g., the first user 102 is cold in the vehicle 100A) may have been encountered. Thus, a voice command that includes the terms “cold,” “freezing,” etc., can be pre-assigned to one or more first fulfillment actions for the first user 102.

[0072] In contrast, with reference to FIG. 5B, further assume that a second user 104 is driving the vehicle 100A and provides a voice utterance 552B of "it's cold." Notably, the voice utterance 552B provided by the second user 104 in FIG. 5B is the same as the voice utterance 552A provided by the first user 102 in FIG. 5A. The voice utterance 552B of "it's cold" provided by the second user 104 can be processed in the same or similar manner as described above with respect to the voice utterance 552A of FIG. 5A. However, it should be noted that the one or more second fulfillment actions performed to satisfy the voice utterance 552B in FIG. 5B are different from the one or more first fulfillment actions performed to satisfy the voice utterance 552A in FIG. 5A. For example, the one or more second fulfillment actions may include turning on a heated seat occupied by the second user 104 to increase the temperature inside the vehicle 100A and / or the body temperature of the first user 102 located inside the vehicle 100A, and may also include turning on a blower in the vehicle 100A to circulate warm air throughout the vehicle to increase the temperature inside the vehicle 100A based on the output generated using the fulfillment action model and / or fulfillment rule(s) specific to the second user 104 (e.g., by the on-vehicle computing device 110 N (as indicated by synthesized voice 554B saying, "I've turned on your heated seats and blower," which is audibly presented to the user via speaker(s) of the second user 102. In other words, based on the output generated using the fulfillment action model and / or fulfillment rule(s) that are specific to the second user 102, the fulfillment of the voice utterance 552B is dynamically adapted to the fulfillment action(s) that the second user 104 would manually perform in the same or a similar context scenario.

[0073] As Figures 5A and 5B are described with respect to a first user 102 and a second user 104 both driving a vehicle 100A at separate time instances, it should be understood that it is for purposes of example and not intended to be limiting. For example, the techniques described herein can be utilized when both the first user 102 and the second user 104 are co-located within a vehicle environment. As one non-limiting example, assume that the first user 102 is driving the vehicle 100A and the second user 104 is a passenger in the vehicle 100A. In other words, the first user 102 may be associated with a "driver" zone of the vehicle 100A and the second user 104 may be associated with a "passenger" zone of the vehicle 100A. These zones may be determined based on occupancy sensor(s) of the vehicle 100A and / or any technique described herein and / or any other suitable technique for identifying users of the vehicle 100A. Assume further that the first user 102 provides the same voice utterances from Figures 5A and 5B. In this example, the heated seats of the first user 102 in the "driver" zone may be turned on, but other heated seats of the vehicle may not be turned on, and the blower may not be turned on as described with respect to the one or more first fulfillment actions of Figure 5A. In some of these examples, the heated seats of the second user 104 and the blower associated with the "passenger" zone may be turned on as described with respect to the one or more second fulfillment actions of Figure 5B. However, any blowers associated with the "driver" zone may remain off. Assume further that the second user 102 provides the same voice utterances from Figures 5A and 5B (e.g., before or after the first user 102 provides the same voice utterances from Figures 5A and 5B). In this example, the heated seat for the second user 102 in the "passenger" zone may be turned on and the blower associated with the "passenger" zone may be turned on, as described with respect to FIG. 5B.Thus, in various implementations, one or more fulfillment actions performed on different users co-located within the environment may be restricted to a zone determined to be occupied by the user who provided the voice utterance.

[0074] 6A and 6B, various non-limiting examples are shown for dynamically adapting the fulfillment of a given voice utterance in a primary residential environment based on the user who provided the given voice utterance. The floor plan shown in FIGS. 6A and 6B includes a number of rooms 660-672. A number of computing devices 110 2~6 are deployed in at least some of the rooms. 2~6 Each of the rooms may implement an instance of a fulfillment system client (e.g., fulfillment system client 114 of FIG. 1 ) configured in selected aspects of the disclosure and may include one or more input devices, such as microphone(s) capable of capturing speech spoken by nearby users. For example, computing devices in the form of interactive standalone speakers are deployed in room 660, which is a kitchen in FIGS. 6A and 6B (e.g., kitchen computing device 1102), room 664, which is a bedroom in FIGS. 6A and 6B (e.g., bedroom computing device 1104), and room 666, which is a living room in FIGS. 6A and 6B (e.g., living room computing device 1104). Additionally, computing devices in the form of so-called "smart" televisions are deployed in room 662, which is a study in FIGS. 6A and 6B (e.g., study TV 1103), and room 660, which is a kitchen in FIGS. 6A and 6B (e.g., kitchen TV 1106). Various users of the primary residence may use multiple computing devices 110 2~6 , to control other smart device(s) in the primary residence (e.g., to cause the smart device(s) to change state(s)).

[0075] The smart devices in the primary residence shown in FIGS. 6A and 6B may include, for example, smart light(s) in one or more of rooms 660-672 of the primary residence, smart thermostat(s) in one or more of rooms 660-672 of the primary residence, smart appliance(s) (e.g., a smart oven, a smart toaster, a smart refrigerator, etc.) in the kitchen 660 of FIGS. 6A and 6B, smart appliance(s) (e.g., a smart washer, a smart dryer, etc.) in the laundry room 672 of FIGS. 6A and 6B, smart lock(s) on the door of the primary residence, a smart alarm system in the primary residence (e.g., smart contact sensor(s), a smart 5A and 5B, the vehicle 100A may also include a number of smart devices. These smart devices may include smart windows of the vehicle 100A, a smart lock of the vehicle 100A, a smart defroster of the vehicle 100A, a smart heated seats of the vehicle 100A, an on-vehicle computing device 110 of the vehicle, and / or any other smart device. Although certain smart devices are described herein with respect to FIGS. 6A and 6B, it should be understood that this is for purposes of example and not intended to be limiting. For example, and referring back briefly to FIGS. 5A and 5B, the vehicle 100A may also include a number of smart devices. These smart devices may include smart windows of the vehicle 100A, a smart lock of the vehicle 100A, a smart defroster of the vehicle 100A, a smart heated seats of the vehicle 100A, a smart on-vehicle computing device 110 of the vehicle, and / or any other smart device. N The smart devices may include, but are not limited to, a smart infotainment system operating in a vehicle, a smart blower system in the vehicle, and / or other smart devices that may be integrated with the vehicle 100A. In these examples, a user interaction corresponding to one or more fulfillment actions may include causing any of these smart devices to change state.

[0076] Although not shown in FIGS. 6A and 6B, multiple computing devices 106 2~6may be communicatively coupled to one another, to smart devices in the primary residence, and / or to other resources (e.g., the Internet) via one or more wired or wireless WANs and / or LANs (e.g., via network(s) 195 of FIG. 1). Additionally, other computing devices (e.g., mobile devices such as smartphones, tablets, laptops, wearable devices, etc.) may also be present, e.g., carried by one or more people in the home, and may or may not be connected to the same WANs and / or LANs. It should be understood that the configuration of computing devices and smart devices shown in FIGS. 6A and 6B is only an example, and that more or fewer and / or different computing devices and / or smart devices may be deployed across any number of other rooms and / or areas of the primary residence and / or in locations outside the residence (e.g., businesses, hotels, public places, airports, vehicles, and / or other locations or spaces).

[0077] For purposes of Figures 6A and 6B, it is assumed that a home environment is at least partially co-resident with multiple users (a first user 102 and a second user 104 as shown in Figures 6A and 6B). It is further assumed that fulfillment action models have been previously trained for both the first user 102 and the second user 104 and / or fulfillment rules have been previously generated for both the first user 102 and the second user 104. Thus, it is assumed that a computing device 110 at least partially resides in a primary residence. 2~6 A fulfillment system executing on one or more of the above (e.g., fulfillment system 120 of FIG. 1 ) can utilize these fulfillment models and / or fulfillment rules that are specific to first user 102 and second user 104 when determining how to dynamically adapt the fulfillment of a voice utterance.

[0078] 6A specifically, assume that the first user 102 is located within the room 666, which is the living room in FIG. 6A and FIG. 6B, and further assume that the first user 102 provides the voice utterance 652A of “dark” and that the microphone(s) of the living room computing device 1105 generates audio data capturing the voice utterance 652A. In this example, the audio data capturing the voice utterance 652A can be processed in the same manner or similarly as described with respect to FIG. 5A and FIG. 5B to determine one or more fulfillment actions to be performed in response to receiving the voice utterance 652A. In particular, the user who provided the voice utterance 652A can be determined to correspond to the first user 102 using any technique described herein and / or any other suitable technique. As a result, the one or more first fulfillment actions can be performed to make the living room darker, such as turning on the smart lights in the living room to 100% brightness (e.g., as indicated by the synthesized voice 654A of “I turned the lights on to 100% brightness”).

[0079] 6B , assume further that the second user 104 is located within the room 662, which is the study in FIGS. 6A and 6B, and provides the voice utterance 652B of “dark,” and further assume that the microphone(s) of the study TV 1103 generates audio data capturing the voice utterance 652B. In this example, the audio data capturing the voice utterance 652B can be processed in the same manner or similarly as described with respect to FIGS. 5A and 5B to determine one or more fulfillment actions to be performed in response to receiving the voice utterance 652B. In particular, the user who provided the voice utterance 652A can be determined to correspond to the second user 104 using any technique described herein and / or any other suitable technique. As a result, one or more second fulfillment actions can be performed to make the study darker, such as turning on the study smart lights, but only to 50% brightness (e.g., as indicated by the synthesized voice 654B of “I turned the lights on to 50% brightness”).

[0080] Notably, the voice utterances provided by the first user 102 and the second user 104 in Figures 6A and 6B are the same. However, the fulfillment actions performed in response to the voice utterances are different in that the living room light is turned on to 100% brightness based on the first user 102 providing the voice utterance 652a, and the study light is turned on to 50% brightness based on the second user 104 providing the voice utterance 652b. In some implementations, this difference in fulfillment actions may be based on different fulfillment action models previously trained for both the first user 102 and the second user 104, as described with respect to Figures 5A and 5B, and / or fulfillment rules previously generated for both the first user 102 and the second user 104. In other implementations, this difference in fulfillment action may be based on the different zones in which the voice utterance is received (e.g., a "living room" zone for the first user 102 and a "presence" zone for the second user 104). Nevertheless, the techniques described herein are utilized to dynamically adapt the fulfillment of the same voice utterance.

[0081] 7, shown is a block diagram of an exemplary computing device 710 that may optionally be utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more computing devices, one or more vehicles, and / or other component(s) may include one or more components of the exemplary computing device 710.

[0082] The computing device 710 typically includes at least one processor 714 that communicates with a number of peripheral devices via a bus subsystem 712. These peripheral devices may include, for example, a storage subsystem 724 including a memory subsystem 725 and a file storage subsystem 726, user interface output devices 720, user interface input devices 722, and a network interface subsystem 716. The input and output devices enable user interaction with the computing device 710. The network interface subsystem 716 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0083] The user interface input devices 722 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, a graphics tablet, a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 710 or a communications network.

[0084] The user interface output devices 720 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 710 to a user or to another machine or computing device.

[0085] Storage subsystem 724 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 724 may include logic to perform selected aspects of the methods disclosed herein and to implement the various components illustrated in FIGS. 1 and 2.

[0086] These software modules are typically executed by the processor 714 alone or in combination with other processors. The memory 725 used by the storage subsystem 724 may include several memories including a main random access memory (RAM) 730 for storing instructions and data during program execution, and a read only memory (ROM) 732 in which fixed instructions are stored. The file storage subsystem 726 may provide persistent storage of program files and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of an embodiment may be stored by the file storage subsystem 726 in the storage subsystem 724 or on other machines accessible by the processor(s) 714.

[0087] Bus subsystem 712 provides a mechanism that allows the various components and subsystems of computing device 710 to communicate with each other as intended. Although bus subsystem 712 is illustrated generally as a single bus, alternative implementations of bus subsystem 712 may use multiple buses.

[0088] Computing device 710 may be of various types including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 710 shown in Figure 7 is intended only as a specific example to illustrate some implementations. Many other configurations of computing device 710 are possible having more or fewer components than the computing device shown in Figure 7.

[0089] In situations where the systems described herein may collect or monitor personal information about a user or utilize personal information and / or monitored information, the user may be given the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social behavior or activities, occupation, user preferences, or the user's current geographic location) or whether and / or how it receives content from a content server that may be more relevant to the user. Certain data may also be processed in one or more ways such that personally identifiable information is removed before it is stored or used. For example, the user's identity may be processed such that personally identifiable information cannot be determined, or if location information is obtained (e.g., to the city, zip code, or state level), the user's geographic location may be generalized such that the user's specific location cannot be determined. Thus, the user may control how information is collected and / or used about the user.

[0090] In some implementations, a method implemented by one or more processors is provided, the method including receiving, from a user and via a computing device, a given voice utterance, the given voice utterance being provided while the user is located within the user's vehicle, and determining a set of fulfillment actions based on processing the given voice utterance. Each fulfillment action included in the set of fulfillment actions, when performed, facilitates the given voice utterance. The method further includes determining whether a user who provided the given voice utterance corresponds to a first user or a second user, and in response to determining that the user who provided the given voice utterance corresponds to the first user, selecting a subset of one or more first fulfillment actions from the set of fulfillment actions based on the user who provided the given voice utterance corresponding to the first user, and causing the subset of one or more first fulfillment actions to be performed to fulfill the given voice utterance.

[0091] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0092] In some implementations, the method may further include, in response to determining that the user who provided the given voice utterance corresponds to the second user, selecting a subset of one or more second fulfillment actions from the set of fulfillment actions based on the user who provided the given voice utterance corresponding to the second user. The second user may be an additional user added to the first user, and the subset of one or more second fulfillment actions may include at least one fulfillment action not included in the subset of one or more fulfillment actions. The method may further include causing the one or more second fulfillment actions to be performed to fulfill the given voice utterance.

[0093] In some versions of those implementations, the first user and the second user may be located at the same location within the vehicle when the given voice utterance is received. Further, causing the subset of one or more first fulfillment actions to be performed to satisfy the given voice utterance may include performing the one or more first fulfillment actions with respect to a first zone of the vehicle's multiple zones occupied by the first user, and causing the subset of one or more second fulfillment actions to satisfy the given voice utterance may include performing the one or more second fulfillment actions with respect to a second zone of the vehicle's multiple zones occupied by the second user.

[0094] In additional or alternative versions of those embodiments, selecting a subset of one or more first fulfillment actions from the set of fulfillment actions based on the user who provided the given voice utterance corresponding to the first user may include processing the set of fulfillment actions to select the subset of one or more first fulfillment actions using a first user fulfillment action model that is specific to the first user and / or one or more first user fulfillment rules that are specific to the first user. In some additional versions of those embodiments, selecting a subset of one or more second fulfillment actions from the set of fulfillment actions based on the user who provided the given voice utterance corresponding to a second user may include processing the set of fulfillment actions to select the subset of one or more second fulfillment actions using a second user fulfillment action model that is specific to the second user and / or one or more second user fulfillment rules that are specific to the second user.

[0095] In some implementations, the method may further include, following causing the performance of the subset of the one or more first fulfillment actions to satisfy the given voice utterance, causing an indication of why the one or more first fulfillment actions were performed to satisfy the given voice utterance to be provided for presentation to the first user.

[0096] In some implementations, processing the given speech utterance may include processing audio data capturing the given speech utterance with an automatic speech recognition (ASR) model to generate ASR data for the given speech utterance, and processing the ASR data for the given speech utterance with a natural language understanding (NLU) model to generate NLU data for the given speech utterance. In some versions of those implementations, determining a set of fulfillment actions based on processing the given speech utterance may include determining a set of fulfillment actions based on the NLU data for the given speech utterance.

[0097] In some implementations, determining whether a user who provided the given voice utterance corresponds to a first user or a second user may include processing audio data capturing the given voice utterance using a speaker identification (SID) model to determine whether a user who provided the given voice utterance corresponds to a first user or a second user. In some versions of those embodiments, processing audio data capturing a given speech utterance using a SID model to determine whether a user who provided the given speech utterance corresponds to a first user or a second user may include, in response to determining that the speech utterance is not long enough to perform text-independent (TI) SID, processing the audio data using a text-dependent (TD) SID model as the SID model to generate a TD speaker embedding, comparing the TD speaker embedding to one or more stored TD speaker embeddings in the TD SID embedding space, and determining whether the user who provided the given speech utterance corresponds to the first user or the second user based on the comparing the TD speaker embedding to the one or more stored TD speaker embeddings. In some further versions of those embodiments, processing audio data capturing a given voice utterance using the SID model to determine the identity of a user who provided the given voice utterance may include, in response to determining that the voice utterance is of sufficient length to perform a TI SID, processing the audio data using the TI SID model as the SID model to generate a TI speaker embedding, comparing the TI speaker embedding to one or more stored TI speaker embeddings within the TI SID embedding space, and determining whether the user who provided the given voice utterance corresponds to a first user or a second user based on comparing the TI speaker embedding to the one or more stored TI speaker embeddings.In some versions of these implementations, audio data capturing a given voice utterance may be generated via one or more microphones of a computing device.

[0098] In some implementations, determining whether a user who provided the given voice utterance corresponds to a first user or a second user may include processing visual data capturing the user who provided the given voice utterance using a face identification (FID) model to generate a face embedding, comparing the face embedding to one or more stored face embeddings in the face embedding space, and determining whether the user who provided the given voice utterance corresponds to the first user or the second user based on comparing the face embedding to the one or more stored face embeddings. In some versions of those implementations, the visual data capturing the user who provided the given voice utterance may be generated via one or more vision components of the computing device or an additional computing device communicatively coupled to the computing device.

[0099] In some implementations, determining whether the user who provided the given voice utterance corresponds to the first user or the second user may include utilizing a corresponding device identifier of the computing device or an additional computing device communicatively coupled to the computing device to determine whether the user who provided the given voice utterance corresponds to the first user or the second user.

[0100] In some implementations, determining whether the user who provided the given voice utterance corresponds to the first user or the second user may include utilizing a corresponding user account on the computing device or an additional computing device communicatively coupled to the computing device to determine whether the user who provided the given voice utterance corresponds to the first user or the second user.

[0101] In some implementations, a method implemented by one or more processors is provided, the method including receiving, from a user and via a computing device, a given voice utterance, the given voice utterance being provided while the user is located within the user's primary residence, and determining a set of fulfillment actions based on processing the given voice utterance. Each fulfillment action included in the set of fulfillment actions, when performed, facilitates the given voice utterance. The method further includes determining whether a user who provided the given voice utterance corresponds to a first user or a second user, and in response to determining that the user who provided the given voice utterance corresponds to the first user, selecting a subset of one or more first fulfillment actions from the set of fulfillment actions based on the user who provided the given voice utterance corresponding to the first user, and causing the subset of one or more first fulfillment actions to be performed to fulfill the given voice utterance.

[0102] In some embodiments, a method implemented by one or more processors is provided, the method including: identifying, at a given time instance of a plurality of time instances, an occurrence of a user interaction of a user with one or more smart devices, the user interaction corresponding to one or more fulfillment actions; obtaining one or more context signals characterizing a state of the user at the given time instance and / or characterizing a state of an environment of the user at the given time instance; generating a given training instance based on the user interaction and based on the one or more context signals; in response to determining that one or more training conditions are met, training a fulfillment action model specific to the user based on at least the given time instance; and utilizing the user-specific fulfillment action model in responding to a voice utterance received from the user.

[0103] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0104] In some implementations, the method may further include, prior to generating a given training instance based on user interactions and based on the one or more context signals, generating a prompt requesting that the user verify whether one or more fulfillment actions have been performed based on the user's state at the given time instance and / or based on a state of the user's environment at the given time instance, causing the prompt to be provided for presentation to the user, and receiving user input in response to the prompt verifying that one or more fulfillment actions have been performed based on the user's state at the given time instance and / or based on a state of the user's environment at the given time instance. In some versions of those implementations, generating a given training instance based on user interactions and based on the one or more context signals may be in response to receiving the verification that one or more fulfillment actions have been performed based on the user's state at the given time instance and / or based on a state of the user's environment at the given time instance.

[0105] In some implementations, generating a given training instance based on user interactions and based on one or more context signals may include determining, for the given training instance, training instance inputs including (i) one or more context signals characterizing a state of the user at the given time instance and / or a state of the user's environment at the given time instance and (ii) a set of fulfillment actions associated with the one or more context signals, and determining training instance outputs, the training instance outputs including one or more fulfillment actions of the user interactions. In some versions of those implementations, training a fulfillment action model that is user-specific based on at least the given training instance may include processing the training instance inputs using the fulfillment action model to generate predicted outputs, comparing the predicted outputs to the training instance outputs to generate one or more losses, and updating the fulfillment action model based on the one or more losses.

[0106] In some implementations, the one or more training conditions may include a quantity, time of day, and / or day of the week of training instances available for training the fulfillment action model.

[0107] In some implementations, the method may further include identifying, at an additional given time instance of the plurality of time instances, an occurrence of an additional user interaction of an additional user with the one or more smart devices, the additional user interaction corresponding to one or more additional fulfillment actions, the one or more additional fulfillment actions including at least one fulfillment action different from the one or more fulfillment actions; obtaining one or more additional context signals characterizing an additional state of the additional user at the additional given time instance and / or characterizing an additional state of an environment of the additional user at the additional given time instance; generating an additional given training instance based on the additional user interaction and based on the one or more additional context signals; and, in response to determining that the one or more training conditions are met, training an additional fulfillment action model specific to the additional user based on at least the additional given training instance; and utilizing the additional fulfillment action model specific to the additional user in responding to voice utterances received from the additional user.

[0108] In some embodiments, a method implemented by one or more processors is provided, the method including: identifying, at a given time instance among a plurality of time instances, an occurrence of a user interaction of a user with one or more smart devices, the user interaction corresponding to one or more fulfillment actions; obtaining one or more context signals characterizing a state of the user at the given time instance and / or characterizing a state of an environment of the user at the given time instance; and generating a prompt requesting the user to verify whether the one or more fulfillment actions have been performed based on the state of the user at the given time instance and / or based on the state of the user's environment at the given time instance. providing a prompt for presentation to the user; receiving user input in response to the prompt verifying that one or more fulfillment actions have been performed based on a state of the user at the given time instance and / or based on a state of the user's environment at the given time instance; generating one or more fulfillment action rules that are user specific in response to receiving the user input verifying that one or more fulfillment actions have been performed based on the state of the user at the given time instance and / or based on the state of the user's environment at the given time instance; and utilizing the one or more user specific fulfillment action rules in responding to voice utterances received from the user.

[0109] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0110] In some implementations, the method includes identifying, at an additional given time instance of the plurality of time instances, an occurrence of an additional user interaction of an additional user with one or more smart devices of an additional user, the additional user interaction corresponding to one or more additional fulfillment actions, the one or more additional fulfillment actions including at least one fulfillment action different from the one or more fulfillment actions; acquiring one or more additional context signals characterizing an additional state of the additional user at the additional given time instance and / or characterizing an additional state of an environment of the additional user at the additional given time instance; and verifying whether the one or more additional fulfillment actions were performed by the additional user based on the additional state of the additional user at the additional given time instance and / or based on the additional state of the environment of the additional user at the additional given time instance. generating an additional prompt requesting that the user provide the additional prompt for presentation to the additional user; receiving an additional user input in response to the additional prompt verifying that the one or more additional fulfillment actions have been performed based on an additional state of the additional user at the additional given time instance and / or based on an additional state of the user's environment at the additional given time instance; generating one or more additional fulfillment action rules specific to the additional user in response to receiving the additional user input verifying that the one or more additional fulfillment actions have been performed based on an additional state of the additional user at the additional given time instance and / or based on an additional state of the user's environment at the additional given time instance; and utilizing the one or more additional fulfillment action rules specific to the additional user in responding to the voice utterance received from the additional user.

[0111] In some implementations, a method implemented by one or more processors is provided, the method including generating, for a user, a personalized mapping of one or more voice commands to one or more fulfillment actions. Generating the personalized mapping is responsive to determining one or more past instances of the user causing performance of the one or more fulfillment actions independent of providing the voice utterance and in a context scenario determined to correlate with the one or more voice commands. The method further includes receiving a given voice utterance from the user and subsequent to generating the personalized mapping, determining based on processing the given voice utterance that the given voice utterance is one of the one or more voice commands, and automatically performing the one or more fulfillment actions using the personalized mapping in response to determining that the given utterance is one of the one or more voice commands.

[0112] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0113] In some implementations, generating a personalized mapping of one or more voice commands to one or more fulfillment actions may include generating a mapping between a voice command semantic representation of the one or more voice commands and a fulfillment action semantic representation of the one or more fulfillment actions within a semantic space and based on one or more past instances, and storing the mapping in one or more databases.

[0114] In some implementations, generating a personalized mapping of one or more voice commands to one or more fulfillment actions may include generating a mapping between the intent of the one or more voice commands and the fulfillment actions of the one or more fulfillment actions based on one or more past instances and storing the mapping in one or more databases.

[0115] In some embodiments, a method implemented by one or more processors is provided, the method including identifying, at a given time instance of a plurality of time instances, an occurrence of a user interaction of a user with one or more smart devices, the user interaction corresponding to one or more fulfillment actions, and training one or more context signals characterizing a context scenario of the user. The one or more context signals characterizing the context scenario of the user characterize a state of the user at the given time instance and / or characterize a state of an environment of the user at the given time instance. The method further includes identifying one or more voice commands for assigning to the context scenario of the user. The one or more voice commands, when later provided by the user, cause one or more fulfillment actions of the user interaction to be performed. The method further includes receiving an instance of a given voice utterance from a user and at a subsequent time instance of the multiple time instances, the subsequent time instance following the given time instance, determining that the given instance of voice utterance includes one or more of the voice commands based on processing the given instance of voice utterance, determining to cause one or more fulfillment actions to be performed to satisfy the given instance of voice utterance, and causing the one or more fulfillment actions to be performed to satisfy the given instance of voice utterance.

[0116] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0117] In some embodiments, the method may further include identifying, at a further subsequent time instance of the plurality of time instances, an additional occurrence of an additional user interaction with the one or more smart devices of an additional user, the additional user interaction corresponding to one or more additional fulfillment actions, the one or more additional fulfillment actions including at least one fulfillment action different from the one or more fulfillment actions, the further subsequent time instance following the given time instance, and obtaining one or more additional context signals characterizing an additional context scenario of the additional user. The one or more additional context signals characterizing the additional context scenario of the additional user may characterize an additional state of the additional user at the further subsequent time instance and / or may characterize an additional state of the environment of the additional user at the further subsequent time instance, the additional context scenario of the additional user may match the context scenario of the user. The method may further include identifying one or more voice commands for assigning to the additional context scenario of the additional user. The one or more voice commands, when later provided by the additional user, may cause one or more fulfillment actions of the user interaction below the value to be performed. The method may further include receiving an additional instance of the given voice utterance from an additional user and at a still further subsequent time instance of the plurality of time instances, where the still further subsequent time instance follows the still further subsequent time instance, and based on processing the additional instance of the given voice utterance, determining that the additional instance of the given voice utterance includes one or more of the voice commands, determining to cause one or more additional fulfillment actions to be performed to satisfy the additional instance of the given voice utterance, and causing the one or more additional further fulfillment actions to be performed to satisfy the additional instance of the given voice utterance.

[0118] Further, some embodiments include one or more processors (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s)), and / or tensor processing unit(s) (TPU(s))) of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, where the instructions are configured to cause performance of any of the methods described above. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions executable by the one or more processors to perform any of the methods described above. Some embodiments also include a computer program product including instructions executable by the one or more processors to perform any of the methods described above.

[0119] It should be understood that all combinations of the above concepts, and additional concepts described in more detail herein, are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.

Claims

1. 1. A method implemented by one or more processors, comprising: receiving, from a user and via a computing device, a given voice utterance, the given voice utterance being provided while the user is located within the user's vehicle; determining a set of fulfillment actions based on processing the given voice utterance, each fulfillment action included in the set of fulfillment actions, when performed, to facilitate satisfying the given voice utterance; determining whether the user providing the given voice utterance corresponds to a first user or a second user; in response to determining that the user who provided the given voice utterance corresponds to the first user; selecting a subset of one or more first fulfillment actions from the set of fulfillment actions based on the user providing the given voice utterance corresponding to the first user; performing a subset of the one or more first fulfillment actions to satisfy the given voice utterance; and A method comprising:

2. in response to determining that the user who provided the given voice utterance corresponds to the second user; selecting a subset of one or more second fulfillment actions from the set of fulfillment actions based on the user providing the given voice utterance corresponding to the second user; the second user is an additional user added to the first user; selecting, wherein the subset of the one or more second fulfillment actions includes at least one fulfillment action that is not included in the subset of the one or more first fulfillment actions; performing the one or more second fulfillment actions to satisfy the given voice utterance; and The method of claim 1 further comprising:

3. the first user and the second user are co-located within the vehicle when the given voice utterance is received; causing the subset of one or more first fulfillment actions to be performed to satisfy the given voice utterance includes performing the one or more first fulfillment actions with respect to a first zone of the vehicle occupied by the first user among a plurality of zones of the vehicle; causing the subset of one or more second fulfillment actions to be performed to satisfy the given voice utterance includes performing the one or more second fulfillment actions with respect to a second zone of the vehicle occupied by the second user among the plurality of zones of the vehicle. The method of claim 2.

4. selecting a subset of the one or more first fulfillment actions from the set of fulfillment actions based on the user providing the given voice utterance corresponding to the first user; processing the set of fulfillment actions to select a subset of the one or more first fulfillment actions using a first user fulfillment action model specific to the first user and / or one or more first user fulfillment rules specific to the first user; The method of claim 2 or claim 3, comprising:

5. selecting a subset of the one or more second fulfillment actions from the set of fulfillment actions based on the user providing the given voice utterance corresponding to the second user; processing the set of fulfillment actions using a second user fulfillment action model specific to the second user and / or one or more second user fulfillment rules specific to the second user to select a subset of the one or more second fulfillment actions; The method of claim 4 , comprising:

6. subsequent to causing the one or more first subset of fulfillment actions to be performed to satisfy the given voice utterance; providing, for presentation to the first user, an indication of why the one or more first fulfillment actions were performed to satisfy the given voice utterance.

13. The method of any preceding claim, further comprising:

7. Processing the given speech utterance comprises: processing audio data capturing the given speech utterance using an automatic speech recognition (ASR) model to generate ASR data for the given speech utterance; processing the ASR data for the given speech utterance using a natural language understanding (NLU) model to generate NLU data for the given speech utterance; 2. A method according to any preceding claim, comprising:

8. determining the set of fulfillment actions based on processing the given voice utterance; determining the set of fulfillment actions based on the NLU data of the given voice utterance; The method of claim 7, comprising:

9. Determining whether the user providing the given voice utterance corresponds to the first user or the second user. processing audio data capturing the given voice utterance using a speaker identification (SID) model to determine whether the user providing the given voice utterance corresponds to the first user or the second user; 2. A method according to any preceding claim, comprising:

10. processing the audio data capturing the given voice utterance using the SID model to determine whether the user providing the given voice utterance corresponds to the first user or the second user; in response to determining that the speech utterance is not of sufficient length to perform a text-independent (TI) SID, processing the audio data to generate TD speaker embeddings using a text-dependent (TD) SID model as the SID model; and comparing said TD speaker embedding to one or more stored TD speaker embeddings in a TD SID embedding space; determining whether the user who provided the given speech utterance corresponds to the first user or the second user based on comparing the TD speaker embedding to the one or more stored TD speaker embeddings; 10. The method of claim 9, comprising:

11. processing the audio data capturing the given voice utterance using the SID model to determine an identity of the user who provided the given voice utterance; in response to determining that the voice utterance is of sufficient length to perform a TI SID; processing the audio data to generate TI speaker embeddings using a TI SID model as the SID model; comparing said TI speaker embedding to one or more stored TI speaker embeddings in a TI SID embedding space; determining whether the user who provided the given speech utterance corresponds to the first user or the second user based on comparing the TI speaker embedding to the one or more stored TI speaker embeddings; The method of claim 10, comprising:

12. The method of claim 9 , wherein the audio data capturing the given voice utterance is generated via one or more microphones of the computing device.

13. Determining whether the user providing the given voice utterance corresponds to the first user or the second user. processing visual data capturing the user providing the speech utterance using a facial identification (FID) model to generate a facial embedding; comparing said face embedding to one or more stored face embeddings in a face embedding space; determining whether the user providing the given voice utterance corresponds to the first user or the second user based on comparing the facial embedding to the one or more stored facial embeddings; The method according to any one of claims 1 to 8, comprising:

14. 14. The method of claim 13, wherein the visual data capturing the user providing the given voice utterance is generated via one or more visual components of the computing device or an additional computing device communicatively coupled to the computing device.

15. Determining whether the user providing the given voice utterance corresponds to the first user or the second user. utilizing a corresponding device identifier of the computing device or an additional computing device communicatively coupled to the computing device to determine whether the user providing the given voice utterance corresponds to the first user or the second user. The method according to any one of claims 1 to 8, comprising:

16. Determining whether the user providing the given voice utterance corresponds to the first user or the second user. utilizing a corresponding user count of the computing device or an additional computing device communicatively coupled to the computing device to determine whether the user providing the given voice utterance corresponds to the first user or the second user. The method according to any one of claims 1 to 8, comprising:

17. 1. A method implemented by one or more processors, comprising: receiving from a user, via a computing device, a given voice utterance, the given voice utterance being provided while the user is located within the user's primary residence; determining a set of fulfillment actions based on processing the given voice utterance, each fulfillment action included in the set of fulfillment actions, when performed, to facilitate satisfying the given voice utterance; determining whether the user providing the given voice utterance corresponds to a first user or a second user; in response to determining that the user who provided the given voice utterance corresponds to a first user; selecting a subset of one or more first fulfillment actions from the set of fulfillment actions based on the user providing the given voice utterance corresponding to the first user; performing a subset of the one or more first fulfillment actions to satisfy the given voice utterance; and A method comprising:

18. 1. A method implemented by one or more processors, comprising: identifying, at a given time instance of a plurality of time instances, an occurrence of a user interaction with one or more smart devices of a user, the user interaction corresponding to one or more fulfillment actions; obtaining one or more context signals characterizing a state of the user at the given time instance and / or characterizing a state of an environment of the user at the given time instance; generating a given training instance based on the user interactions and based on the one or more context signals; in response to determining that one or more training conditions are satisfied, training a Fulfillment Action Model specific to the user based on at least the given training instance; utilizing said fulfillment action model specific to said user in responding to voice utterances received from said user. A method comprising:

19. prior to generating the given training instance based on the user interactions and based on the one or more context signals; generating a prompt requesting the user to verify whether the one or more fulfillment actions were performed based on the state of the user at the given time instance and / or based on the state of the environment of the user at the given time instance; providing the prompt for presentation to the user; receiving user input in response to the prompt verifying that the one or more fulfillment actions were performed based on the state of the user at the given time instance and / or based on the state of the environment of the user at the given time instance; 20. The method of claim 18, further comprising:

20. 20. The method of claim 19, wherein generating the given training instance based on the user interactions and based on the one or more context signals is responsive to receiving verification that the one or more fulfillment actions have been performed based on the state of the user at the given time instance and / or based on the state of the environment of the user at the given time instance.

21. generating the given training instances based on the user interactions and based on the one or more context signals; For the given training instance, determining training instance inputs, the training instance inputs including: (i) the one or more context signals characterizing the state of the user at the given time instance and / or the state of the environment of the user at the given time instance; and (ii) a set of fulfillment actions associated with the one or more context signals; determining a training instance output, the training instance output including the one or more fulfillment actions of the user interaction; The method according to any one of claims 18 to 20, comprising:

22. training the Fulfillment Action Model specific to the user based at least on the given training instances; processing the training instance inputs using the fulfillment action model to generate predicted outputs; comparing the predicted outputs to the training instance outputs to generate one or more losses; updating the fulfillment action model based on the one or more losses; 22. The method of claim 21 , comprising:

23. The method according to any of claims 18 to 22, wherein the one or more training conditions include a quantity of training instances, a time of day, and / or a day of the week available for training the fulfillment action model.

24. identifying, at an additional given time instance of the plurality of time instances, an occurrence of an additional user interaction of an additional user with the one or more smart devices, the additional user interaction corresponding to one or more additional fulfillment actions, the one or more additional fulfillment actions including at least one fulfillment action different from the one or more fulfillment actions; obtaining one or more additional context signals characterizing an additional state of the additional user at the additional given time instance and / or characterizing an additional state of the environment of the additional user at the additional given time instance; generating additional given training instances based on the additional user interactions and based on the one or more additional context signals; in response to determining that the one or more training conditions are satisfied, training an additional Fulfillment Action Model specific to the additional user based on at least the additional given training instance; utilizing the additional fulfillment action model specific to the additional user in responding to voice utterances received from the additional user. The method according to any one of claims 18 to 23, comprising:

25. 1. A method implemented by one or more processors, comprising: identifying, at a given time instance of the plurality of time instances, an occurrence of a user interaction with one or more smart devices of a user, the user interaction corresponding to one or more fulfillment actions; obtaining one or more context signals characterizing a state of the user at the given time instance and / or characterizing a state of an environment of the user at the given time instance; generating a prompt requesting the user to verify whether the one or more fulfillment actions were performed based on the state of the user at the given time instance and / or based on the state of the environment of the user at the given time instance; providing the prompt for presentation to the user; receiving user input in response to the prompt verifying that the one or more fulfillment actions were performed based on the state of the user at the given time instance and / or based on the state of the environment of the user at the given time instance; generating one or more fulfillment action rules specific to the user in response to receiving the user input verifying that the one or more fulfillment actions were performed based on the state of the user at the given time instance and / or based on the state of the environment of the user at the given time instance; utilizing the one or more fulfillment action rules specific to the user in responding to voice utterances received from the user. A method comprising:

26. identifying, at an additional given time instance of the plurality of time instances, an occurrence of an additional user interaction of an additional user with the one or more smart devices, the additional user interaction corresponding to one or more additional fulfillment actions, the one or more additional fulfillment actions including at least one fulfillment action different from the one or more fulfillment actions; obtaining one or more additional context signals characterizing an additional state of the additional user at the additional given time instance and / or characterizing an additional state of the environment of the additional user at the additional given time instance; generating an additional prompt requesting the additional user to verify whether the one or more additional fulfillment actions have been performed based on the state of the additional user at the additional given instance of time and / or based on the additional state of the environment of the additional user at the additional given instance of time; providing the additional prompts for presentation to the additional users; and receiving additional user input in response to the additional prompt verifying that the one or more additional fulfillment actions were performed based on the additional state of the additional user at the additional given instance of time and / or based on the additional state of the environment of the user at the additional given instance of time; generating one or more additional fulfillment action rules specific to the additional user in response to receiving the additional user input verifying that the one or more additional fulfillment actions were performed based on the additional state of the additional user at the additional given instance of time and / or based on the additional state of the environment of the additional user at the additional given instance of time; utilizing the one or more additional fulfillment action rules specific to the additional user in responding to voice utterances received from the additional user.

26. The method of claim 25, comprising:

27. 1. A method implemented by one or more processors, comprising: generating, for a user, a personalized mapping of one or more voice commands to one or more fulfillment actions, the personalized mapping being responsive to determining one or more past instances of the user that caused performance of the one or more fulfillment actions independent of providing a voice utterance and in a context scenario determined to be correlated with the one or more voice commands; receiving a given voice utterance from the user and subsequent to generating the personalized mapping; determining, based on processing the given voice utterance, that the given voice utterance is one of the one or more voice commands; in response to determining that the given utterance is one of the one or more voice commands, using the personalized mapping to automatically perform the one or more fulfillment actions; and A method comprising:

28. generating the personalized mapping of the one or more voice commands to the one or more fulfillment actions; generating a mapping between voice command semantic representations of the one or more voice commands and fulfillment action semantic representations of the one or more fulfillment actions within a semantic space and based on the one or more past instances; storing the mapping in one or more databases; 28. The method of claim 27, comprising:

29. generating the personalized mapping of the one or more voice commands to the one or more fulfillment actions; generating a mapping between an intent of the one or more voice commands and the one or more fulfillment actions based on the one or more past instances; storing the mapping in one or more databases; 29. The method of claim 27 or claim 28, comprising:

30. 1. A method implemented by one or more processors, comprising: identifying, at a given time instance of the plurality of time instances, an occurrence of a user interaction with one or more smart devices of a user, the user interaction corresponding to one or more fulfillment actions; obtaining one or more context signals characterizing a context scenario of the user, the one or more context signals characterizing the context scenario of the user characterizing a state of the user at the given time instance and / or characterizing a state of an environment of the user at the given time instance; identifying one or more voice commands for assignment to the context scenario of the user, the one or more voice commands causing the one or more fulfillment actions of the user interaction to be performed when subsequently provided by the user; receiving an instance of a given voice utterance from the user and at a subsequent time instance of the plurality of time instances, the subsequent time instance following the given time instance; determining, based on processing the instance of the given voice utterance, that the instance of the given voice utterance includes one or more of the voice commands; determining to cause the one or more fulfillment actions to be performed to satisfy the instance of the given voice utterance; performing the one or more fulfillment actions to fulfill the instance of the given voice utterance; and A method comprising:

31. identifying, at a further subsequent given time instance of the plurality of time instances, an additional occurrence of an additional user interaction with the one or more smart devices of an additional user, the additional user interaction corresponding to one or more additional fulfillment actions, the one or more additional fulfillment actions including at least one fulfillment action different from the one or more fulfillment actions, the further subsequent time instance following the given time instance; obtaining one or more additional context signals characterizing an additional context scenario of the additional user, wherein the one or more additional context signals characterizing the additional context scenario of the additional user characterize an additional state of the additional user at the further subsequent time instance and / or characterize an additional state of the environment of the additional user at the further subsequent time instance, wherein the additional context scenario of the additional user is consistent with the context scenario of the user; identifying the one or more voice commands for assignment to the additional context scenario of the additional user, the one or more voice commands causing the one or more additional fulfillment actions of the additional user interaction to be performed when subsequently provided by the additional user; receiving an additional instance of the given voice utterance from the additional user and at a still further subsequent time instance of the plurality of time instances, the still further subsequent time instance following the still further subsequent time instance; determining, based on processing the additional instances of the given voice utterance, that the additional instances of the given voice utterance include the one or more voice commands; determining to cause the one or more additional fulfillment actions to be performed to satisfy the additional instances of the given voice utterance; performing the one or more additional fulfillment actions to satisfy the additional instances of the given voice utterance; and 31. The method of claim 30, further comprising:

32. 1. A system comprising: one or more processors; a memory storing instructions, which when executed cause said one or more processors to perform the operations according to any one of claims 1 to 31; A system comprising:

33. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to perform the operations recited in any of claims 1 to 31.

34. 1. An on-board computing device for a vehicle, comprising: one or more processors; a memory storing instructions, which when executed cause said one or more processors to perform the operations according to any one of claims 1 to 31; The vehicle-mounted computing system comprises:

Images