Dynamic adaptation of a given voice utterance fulfillment based on the user who provided the given voice utterance

Abstract

The implementation described herein relates to determining a method for executing a voice utterance based on a user who has provided the voice utterance. For example, the implementation may receive a voice utterance from a user, determine a set of execution actions for the voice utterance, and determine whether the user who has provided the voice utterance corresponds to a first user or a second user. Additionally, in response to the determination that the user corresponds to a first user, the implementation may select a subset of first execution actions(s) from the set and enable the subset of first execution actions(s) to satisfy the voice utterance. Additionally, in response to the determination that the user corresponds to a second user, the implementation may select a distinct subset of second execution actions(s) from the set and enable the subset of second execution actions(s) to satisfy the voice utterance.

Description

Background Technology

[0001] Humans may engage in human-to-computer conversations with an interactive software application referred herein as an "automated assistant" (also referred to as a "chatbot," "interaction assistant," "intelligent personal assistant," "voice assistant," "conversational agent," etc.). For example, a human (who may be referred to as a "user" when interacting with the automated assistant) may provide text (e.g., typed) natural language input directed to the automated assistant and / or provide voice natural language input (i.e., voice utterances) directed to the automated assistant, and in some cases, the voice natural language input may be converted into text and then processed. Such voice utterances and / or typed inputs often include assistant commands directed to the automated assistant. Automated assistants generally respond to these assistant commands by providing response user interface output(s) (e.g., audible and / or visual user interface output), controlling smart device(s), and / or performing other action(s).

[0002] These automated assistants typically rely on a pipeline of components to interpret and respond to these voice utterances and / or typed inputs. For example, an automatic speech recognition (ASR) engine can process audio data corresponding to a user's voice utterance to generate ASR outputs, such as a transcription of the voice utterance (i.e., a sequence of term(s) and / or other token(s)). Additionally, a natural language understanding (NLU) engine can process the ASR outputs (or typed inputs) to generate NLU outputs, such as the user's intent when providing the voice utterance and optionally slot value(s) for the parameter(s) associated with the intent. Furthermore, a fulfillment engine can be used to process the NLU outputs and obtain fulfillment outputs, such as response user interface output(s), response smart device control command(s), and / or other response action(s).

[0003] In some cases, such automated assistants may fulfill specific voice utterances in a specific manner. For example, assume that an automated assistant is running at least partially on an in-vehicle computing device in a given user's vehicle, and assume that the user instructs the automated assistant with the voice utterance "I'm cold." In this example, the given automated assistant may utilize the aforementioned pipeline to respond to the voice utterance, resulting in the implementation of a fulfillment action to turn on the blower in the vehicle to satisfy the voice utterance. However, the given user who provided the voice utterance may want only the vehicle's heated seats to be turned on, but not the blower, which may differ from the preferences of other users. Consequently, the given user may have to manually turn off the blower and turn on the heated seats because the automated assistant implemented an incorrect fulfillment action and / or initiated additional interaction with the automated assistant to implement the desired fulfillment action. As a result, computational resources are wasted in these interactions due to incorrect implementation actions being implemented (and optionally any additional interactions arising from incorrect implementation actions being implemented), and natural resources are wasted in these interactions, such as turning on a blower when it is not a desired implementation action.

[0004] The implementation described herein relates to dynamically adapting the execution of a given voice utterance based on the user who provided the given voice utterance, and optionally based on the context scenario in which the given voice utterance is received. In some embodiments, a system (e.g., including processor(s) and memory) may receive a given voice utterance from a user, determine a set of execution actions for the given voice utterance, and determine whether the user who provided the given voice utterance corresponds to a first user or a second user. Additionally, in response to the determination that the user corresponds to a first user, the system may select a subset of first execution actions(s) from the set and enable the subset of first execution actions(s) to satisfy the given voice utterance. Additionally, in response to the determination that the user corresponds to a second user, the system may select a distinct subset of second execution actions(s) from the set and enable the subset of second execution actions(s) to satisfy the given voice utterance. In particular, a subset of the second implementation action(s) may include at least one unique implementation action compared to a subset of the first implementation action(s). Accordingly, the implementation of the same voice utterance provided by different users can be dynamically adapted based on whether the voice utterance is provided by the first user or by the second user.

[0005] For example, assume that a first user is driving a vehicle and provides a voice utterance of "I'm hot," and assume that the microphone(s) of the vehicle's in-vehicle computing device generate audio data that captures the voice utterance. In this example, the audio data capturing the voice utterance may be processed using an Automatic Speech Recognition (ASR) model to generate ASR data for the voice utterance (e.g., recognized text of "I'm hot"). Additionally, the ASR data for the voice utterance may be processed using a Natural Language Understanding (NLU) model to generate NLU data for the voice utterance (e.g., recognized intent(s) such as "hot," "temperature decrease," etc.). In particular, the user who provided the voice utterance may be determined to correspond to the first user using various audio-based (e.g., Speaker Identification (SID) and / or other audio-based technologies) and / or non-audio-based technologies (e.g., Face Identification (FID), User Account Identification, Device Identifier Identification, and / or other non-audio-based technologies) described herein.

[0006] In some embodiments, NLU data for a voice utterance may be processed to determine a set of implementation actions that, when implemented, help to reduce the temperature inside the vehicle. The set of implementation actions may include, for example, rolling down the vehicle's window(s) to reduce the temperature inside the vehicle (when the vehicle's windows are in a state indicating they are rolled up), turning on the vehicle's blower to circulate cold air throughout the vehicle to reduce the temperature inside the vehicle, and / or other implementation actions that may be performed to reduce the temperature inside the vehicle. In this example, the set of implementation actions may be processed using a first user-specific implementation action model and / or implementation rule(s) (e.g., personalized mapping) to select a first implementation action(s) to be implemented to satisfy the voice utterance. For example, the first implementation action(s) may include rolling down the vehicle's window(s) but not turning on the vehicle's blower to circulate cold air throughout the vehicle.

[0007] In contrast, assume that a second user is driving the same vehicle in a subsequent time instance and provides the same voice utterance, "I am hot." The voice utterance may be processed in the same or similar manner to determine a set of fulfillment actions. However, in this example, the set of fulfillment actions may be processed using a fulfillment action model and / or fulfillment rule(s) specific to the second user (e.g., not the first user) to select the second fulfillment action(s) to be implemented to satisfy the voice utterance. For example, the second fulfillment action(s) may include rolling down the vehicle's window(s) and turning on the vehicle's blower to circulate cold air throughout the vehicle.

[0008] In various embodiments, the system may generate corresponding training instances that can subsequently be utilized to train the implementation action model(s) and / or implementation rule(s) described above. These corresponding training instances may be specific to different users and may be generated based on corresponding user interactions of different users with various smart devices performed in different context scenarios. For example, assume that while a first user is in a vehicle, the vehicle's window(s) are made to roll down in a given time instance among a plurality of time instances. Also, assume that the vehicle's ambient temperature sensor indicates that the temperature inside the vehicle is 80 degrees Fahrenheit (e.g., a context signal characterizing the state of the vehicle environment), which indicates a "hot" context scenario in a given time instance. In this example, a given training instance may be generated based on implementation action(s) manually performed by the user in this "hot" context scenario. A given training instance may include a training instance output and a training instance input. In this example, the training instance input may include a set of implementation actions that can be performed to reduce the temperature inside the vehicle, and a context scenario (or indication thereof) in which the implementation action is performed (e.g., a "hot" context scenario). Additionally, the training instance output may include an implementation action of a user interaction (or indication thereof) performed by the user (e.g., rolling down the vehicle's window(s)). This process may be repeated for multiple users in multiple different context scenarios to generate corresponding training instances.In some of these embodiments, the system may assign one or more voice commands to each of different context scenarios (corresponding, for example, to semantic terms or phrases of context scenarios such as “hot,” “scorching,” “burning,” etc. in the above examples).

[0009] In some versions of these embodiments, the system may train implementation action model(s) based on corresponding training instances. For example, the system may use an implementation action model for a first user to process training instance inputs corresponding to context scenarios in which implementation actions are performed to generate a set of implementation actions that may be performed to reduce the temperature inside the vehicle and a predicted output. The predicted output may correspond to a subset of predicted first implementation actions(s) that the first user is predicted to perform given implementation actions available in a "hot" context scenario. Additionally, the predicted output may be compared with implementation actions of user interactions performed by the user to generate one or more losses (e.g., across a distribution of action(s)). Additionally, the implementation action model for the first user may be updated based on one or more losses (e.g., via backpropagation). In these embodiments, the system may train only the implementation action model(s) in response to determining that one or more training conditions described herein are satisfied.

[0010] In various embodiments, the system may generate corresponding personalization mapping(s) (e.g., implementation rule(s)) that are utilized in addition to or instead of the implementation action model(s) described above. For example, the system may generate personalization mappings based on one or more past instances of the user that trigger the performance of one or more implementation actions corresponding to user interactions in a context scenario, and independently of providing arbitrary voice utterances. Additionally, the system may determine that a context scenario corresponds to one or more voice commands using various techniques. For example, the system may map different context scenarios represented by different context signals that characterize the user's state and / or the state of the environment when one or more implementation actions are performed in different parts of a low-dimensional space (e.g., embedding space, semantic space, etc.). Additionally, one or more voice commands may be assigned to various context scenarios in the low-dimensional space. Accordingly, when a given voice command is subsequently included in a given voice utterance received from a user, the system can determine a given context scenario associated with the given voice command in a low-dimensional space and enable one or more given fulfillment actions previously performed in the given context scenario to be implemented to satisfy the given voice utterance. Additionally, for example, the system can map different context scenarios represented by different context signals that characterize the user's state and / or the state of the environment when one or more fulfillment actions are performed with different intents. Furthermore, one or more voice commands can be assigned to different intents.Accordingly, when a given voice command is subsequently included in a given voice utterance received from a user, the system determines a given context scenario associated with the given voice command determined based on a given intent, and can enable one or more given fulfillment actions previously performed in the given context scenario to be implemented to satisfy the given voice utterance.

[0011] In some versions of these implementation examples, the system may generate only a personalized mapping in response to determining that one or more fulfillment actions have been performed by the user in a context scenario for a threshold quantity of instances (e.g., 1, 3, 5, and / or one or more other positive integers). For example, assume that when the ambient temperature inside the vehicle is 70 to 80 degrees Fahrenheit, there were 3 past instances where the user rolled down the vehicle window(s) without turning on the vehicle's blower, and that the 3 past instances satisfy a threshold quantity of instances. In this example, the system maps the context scenario (e.g., the ambient temperature inside the vehicle is 70 to 80 degrees Fahrenheit) to a first part of the lower-dimensional space and / or associates it with a "hot" intent, and assigns voice commands such as "hot" or "very hot" to the first part of the lower-dimensional space and / or associates them with a "hot" intent. Therefore, even if the user has not provided any voice utterances during three past instances, when the user subsequently provides the voice utterance "I am hot" when the ambient temperature inside the vehicle is between 70 and 80 degrees Fahrenheit, the system can utilize the personalization mapping to automatically roll down the window(s). The system can also consider the ambient temperature outside the vehicle in these examples.

[0012] In particular, in these implementations, the same voice command can be mapped to different fulfillment actions. Continuing with the example above, it is further assumed that when the ambient temperature inside the vehicle is over 80 degrees Fahrenheit, there were three past instances where the user rolled down the vehicle's window(s) and turned on the vehicle's blower, and that the three past instances satisfy a threshold quantity of instances. In this example, the system maps the context scenario (e.g., the ambient temperature inside the vehicle is over 80 degrees Fahrenheit) to another second part of the lower-dimensional space and / or associates it with a "hot" intent, and assigns voice commands such as "hot", "very hot", etc., to another second part of the lower-dimensional space and / or associates them with a "hot" intent. However, based on the different fulfillment actions performed by the user in these three instances (e.g., rolling down the vehicle's window(s) and turning on the vehicle's blower), these past instances may be mapped to a different second part of a lower-dimensional space, but may be associated with the same "hot" intent because the different fulfillment actions are performed in slightly different but similar contextual scenarios (e.g., when the ambient temperature inside the vehicle is above 80 degrees Fahrenheit rather than between 70 and 80 degrees Fahrenheit). Therefore, even if the user did not provide any voice utterances during the three past instances, when the user subsequently provides the voice utterance "I am hot" when the ambient temperature inside the vehicle is above 80 degrees Fahrenheit, the system can utilize the personalized mapping to automatically roll down the window(s) and automatically turn on the blower.

[0013] While the above examples describe voice utterances received in a vehicle environment, it should be understood that this is for illustrative purposes only and is not intended to be limiting. For example, the techniques described herein may be utilized in any environment where a given computing device is utilized by multiple users to enable the fulfillment of voice utterances. For example, the techniques described herein may be utilized in a home environment, a workplace environment, a school environment, a commercial environment (e.g., a hotel and / or other business), and / or any other environment. Additionally, it should be noted that the fulfillment actions that may be implemented to satisfy voice utterances may vary depending on the environment. For example, fulfillment actions that may be implemented in a home environment may include controlling various Internet of Things (IoT) devices and / or causing other devices to change their states, whereas fulfillment actions that may be implemented in the vehicle environment described above may include controlling various components and / or systems of the vehicle.

[0014] Additionally, the above examples are described in relation to determining whether the user providing the voice utterance is the first user or the second user, but it should be understood that this is for illustrative purposes only and is not intended to be limiting. For example, the number of users considered in determining the user providing the voice utterance may be based on the number of users associated with the computing device receiving the voice utterance (e.g., the in-vehicle computing device in the above examples). Furthermore, if it is determined that the user providing the voice utterance does not correspond to any of the users associated with the computing device, the user may be considered a default user or a guest user, and default fulfillment model(s) and / or default fulfillment rule(s) (e.g., default mapping(s)) may be utilized to determine how to satisfy the voice utterance.

[0015] Furthermore, it should be understood that while the above examples describe a first user and a second user providing the same voice utterance at different time instances and during each vehicle operation, this is for illustrative purposes only and is not intended to be limiting. For example, various environments may be partitioned into multiple zones, and multiple users may be located together in the environment but in different zones. For instance, a vehicle environment may be divided into a "driver" zone, a "front passenger" zone, etc., whereas a home environment may be divided into a "living room" zone, a "kitchen" zone, etc. In such cases, the execution action may be limited to a specific zone occupied by the user who provided the voice utterance. For example, assume that the first user is driving the vehicle while the second user is a passenger in the vehicle. In this case, if the first user provides the voice utterance "I am hot," only the "driver" zone window rolls down, and if the second user provides the voice utterance "I am hot," only the fan in the "front passenger" zone can be turned on. In other cases, no windows may be rolled down, and since the blower may be considered a more conservative implementation action, only the blowers in both the "driver" zone and the "front passenger" zone may be turned on (for example, the second user may not like the windows rolling down, while the first user does not mind the blower). Additionally or alternatively, joint implementation action model(s) and / or joint implementation rule(s) may be utilized in this type of scenario.

[0016] Various technical advantages can be achieved by using the technology described herein. As one non-limiting example, the technology described herein enables a system to train implementation action model(s) and / or generate implementation rule(s) (e.g., personalized mapping(s)) for voice command(s) for implementation action(s) specific to an individual user based on individual user interaction(s) performed in individual context scenario(s). Such implementation action model(s) and / or implementation rule(s) can subsequently be utilized to personalize the implementation of voice utterances, so that different implementation action(s) can be implemented for different users based on receiving the same voice utterance from different users. As a result, the amount of interactions and / or user inputs in cases where incorrect implementation is implemented can be reduced, thereby conserving computational resources on a computing device based on a reduced number of subsequent interactions to cancel the incorrect implementation and / or enable the implementation of the correct implementation. Additionally, the amount of natural resources wasted in cases where incorrect implementation is implemented can be reduced.

[0017] The above description is provided as an overview of only some embodiments disclosed herein. These embodiments and other embodiments are described in further detail herein. Brief explanation of the drawing

[0018] FIG. 1 illustrates various aspects of the present disclosure and shows a block diagram of an exemplary hardware and software environment in which the embodiments disclosed herein may be implemented. FIG. 2 illustrates an exemplary process flow of the exemplary hardware and software environment of FIG. 1 according to various implementations. FIG. 3 illustrates a flowchart illustrating an exemplary method for dynamically adapting the execution of a given voice utterance based on a user who provided the given voice utterance according to various implementations. FIG. 4 illustrates a flowchart illustrating an exemplary method for training a user-specific execution action model according to various embodiments and creating training instance(s) for training the execution action model for subsequent use in dynamically adapting the execution of a given voice utterance provided by the user. FIGS. 5A and 5B illustrate various non-limiting examples of dynamically adapting the execution of a given voice utterance in a vehicle environment based on a user who has provided a given voice utterance according to various embodiments. FIGS. 6a and 6b illustrate various non-limiting examples of dynamically adapting the execution of a given voice utterance in a primary residential environment based on a user who provided a given voice utterance according to various embodiments. Figure 7 illustrates an exemplary architecture of a computing device according to various implementations. Specific details for implementing the invention

[0019] Now, referring to FIG. 1, an environment in which one or more selected embodiments of the present disclosure may be implemented is illustrated. An exemplary environment comprises a plurality of computing devices ( 110 1-N It includes an implementation system (120), a vehicle (100A), one or more OEM (original equipment manufacturer) applications (181), one or more first-party applications (182), and one or more third-party applications (183). These components (110 1-NEach of , 120, 181, 182, and 183) may communicate through one or more networks, for example, generally denoted as 195. One or more networks may include wired or wireless networks, such as a local area network (LAN) including Wi-Fi, Bluetooth, short-range communication and / or other LANs, a wide area network (WAN) including the Internet, and / or any other network that facilitates communication between the components shown in FIG. 1.

[0020] In various embodiments, the user interacts with computing devices (110) and other components illustrated in FIG. 1. 1-N It can operate one or more of the computing devices (110 1-N ) is, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a standalone speaker device having optionally a display, and an in-vehicle computing device of a vehicle (100A) (e.g., 110 N An in-vehicle communication system, in-vehicle entertainment system, and / or in-vehicle navigation system as described above), or a computing device, such as a wearable device including a head-mounted display ("HMD") that provides an augmented reality ("AR") or virtual reality ("VR") immersive computing experience, a "smart" watch, etc. Additional and / or alternative computing devices may be provided.

[0021] computing devices ( 110 1-NEach of the computing devices (110) and the implementation system (120) may include one or more memories for storing data and software applications (e.g., one or more OEM applications (181), one or more first-party applications (182), and / or one or more third-party applications (183)), one or more processors for accessing data and executing software applications, and other components for facilitating communication through one or more of the networks (195). 1-N Operations performed by one or more of the ) and / or implementation systems (120) may be distributed across multiple computer systems. For example, the implementation system (120) may be implemented as a computer program that is executed exclusively on one or more computers at one or more locations coupled to communicate with each other via one or more networks (195), for example, or distributed across one or more computers. Additionally or alternatively, a computing device (110 1-N The operation performed by one or more of the ) and / or implementation system (120) may be executed locally on the computing system. For example, the implementation system (120) (or an instance thereof) may be on computing devices (110 1-N ) can be implemented locally in each.

[0022] Components ( 110 1-NOne or more of , 120, 181, 182, and 183) may include various and different components that can be used to dynamically adapt the execution of a given voice utterance based on the user who provided the given voice utterance, for example, as described in this specification. For example, the computing device (1101) may include a user interface engine (1111) for detecting and processing user input (e.g., voice utterance, typed input, and / or touch input) directed toward the computing device (1101). As another example, the computing device ( 1101) may include one or more sensors (1121) to generate corresponding sensor data. One or more sensors may include, for example, a GPS sensor for generating GPS (global positioning system) data, a vision component for generating vision data in the field of view of the vision component, a microphone for generating audio data based on voice utterances captured in the environment of the computing device (1101), and / or other sensors for generating corresponding sensor data.

[0023] As another example, the computing device (1101) may operate an input processing engine (1131) (e.g., may be part of another application, such as a standalone or automated assistant application) to process various user inputs received from the computing device (1101). For example, the input processing engine (1131) may capture speech utterances and allow audio data generated by the microphone(s) of the client device (1101) to be processed using automatic speech recognition (ASR) model(s) (e.g., recurrent neural network (RNN) models, transformer models, and / or any other ML model capable of performing ASR) to produce an ASR output. Additionally, the input processing engine (1131) may use natural language understanding (NLU) model(s) (e.g., long short-term memory (LSTM), gated recurrent unit (GRU), and / or any other type of RNN or other ML model capable of performing NLU) and / or grammar-based NLU rule(s) to process the ASR output (or typed input) to generate an NLU output. Additionally, the input processing engine (1131) may have at least the NLU output processed using implementation action model(s) and / or implementation rule(s) (e.g., personalization mapping(s)) to obtain one or more candidate responses to user input, such as action(s) to be performed by an automated assistant based on user input, or content item(s) to be provided to be presented to the user based on user input. These implementation action model(s) and / or implementation rule(s) may be stored in the implementation action model(s) and / or rule(s) database 127A, trained in the manner described herein, and subsequently utilized.In an embodiment where text content is audibly rendered in response to spoken words or typed input, the user interface engine (1111) may cause the text content to be processed using text-to-speech model(s) to generate synthesized speech audio data including computer-generated synthesized speech that captures the content. The synthesized speech audio data may be audibly rendered to be presented to the user through the speaker(s) of the computing device (1101). In embodiments where visual content is visually rendered in response to spoken words or typed input, the user interface engine (1111) may cause the visual content to be visually rendered to be presented to the user through the display of the 1101.

[0024] In various embodiments, the ASR output may include, for example, one or more speech hypotheses (e.g., term hypotheses and / or transcription hypotheses) predicted to correspond to a user’s speech activity and / or speech utterance captured in audio data, one or more corresponding predicted values ​​for each of the one or more speech hypotheses (e.g., probability, log-likelihood and / or other values), a plurality of phonemes predicted to correspond to a user’s speech activity and / or speech utterance captured in audio data, and / or other ASR outputs. In some versions of these embodiments, the input processing engine (1131) may allow one or more of the speech hypotheses to be selected as recognized text corresponding to the speech utterance (e.g., based on the corresponding predicted values).

[0025] In various embodiments, the NLU output may include annotated recognized text containing one or more annotations of the recognized text for one or more (e.g., all) of the terms of the recognized text. For example, the input processing engine (1131) may employ a portion of a speech tagger (not shown) configured to annotate the terms by their grammatical roles. Additionally or alternatively, the input processing engine (1131) may employ an entity tagger (not shown) configured to annotate entity references within one or more segments of the recognized text. Entity references may include references to people (e.g., literary figures, celebrities, public figures, etc.), organizations, locations (real and imagined), etc. In some embodiments, data regarding entities may be stored in one or more databases, such as a knowledge graph (not shown). In some implementations, the knowledge graph may include nodes representing known entities (and in some cases, entity attributes), as well as edges connecting the nodes and representing relationships between entities. The entity tagger may annotate references to entities at a high granularity level (e.g., to enable identification of all references to an entity class such as people) and / or at a lower granularity level (e.g., to enable identification of all references to a specific entity such as a specific person). The entity tagger may rely on the content of natural language input to resolve a specific entity and / or optionally communicate with the knowledge graph or another entity database to resolve a specific entity.

[0026] Additionally or alternatively, the input processing engine (1131) may employ a core reference resolver (not shown) configured to group or “cluster” references to the same entity based on one or more context queues. As one non-limiting example, the core reference resolver may be utilized to interpret the term “that” as a specific light or indicator associated with the operation of the vehicle (100A) in the natural language input “What is that light?” based on a corresponding sensor data instance generated by the vehicle sensor(s) that caused the generation of a specific light or indicator associated with the operation of the vehicle (100A). In some embodiments, one or more components utilized by the input processing engine (1131) may rely on annotations from one or more other components utilized by the input processing engine (1131). For example, in some implementations, an entity tagger may rely on annotations from a correlation resolver when annotating all mentions of a specific entity. Additionally, for example, in some implementations, a correlation resolver may rely on annotations from an entity tagger when clustering references to the same entity.

[0027] As another example, the computing device (1101) may operate a fulfillment system client (1141) to interact with the fulfillment system (120) (e.g., may be part of another application, such as a standalone or automated assistant application). Additionally, an additional computing device (110 N ) can take the form of an in-vehicle computing device of the vehicle (100A). Although not illustrated, an additional computing device (110 N) may include components identical or similar to the computing device (1101). For example, the additional computing device (110N) may include a user interface engine for detecting and processing user input, one or more sensors for generating corresponding instances of vehicle sensor data, an input processing engine, and / or individual instances of a transition system client for interacting with the transition system (120). In this example, the one or more sensors are vehicle sensors, e.g., tire pressure sensors for generating tire pressure data for the tires of the vehicle (100A), airflow sensors for generating airflow data for the air conditioning system of the vehicle (100A), vehicle speed sensors for generating vehicle speed data for the vehicle (100A), energy sensors for generating energy source data for the energy source of the vehicle (100A), transmission sensors for generating transmission data for transmission of the vehicle (100A), and / or the vehicle (100A) and / or the in-vehicle computing device (110) of the vehicle (100A). N It may include any other sensors integrated into ). Also, FIG. 1 includes a computing device (1101) and an in-vehicle computing device (110 N Although only ) is shown, it should be understood that this is merely an example and additional or alternative computing devices may be provided (e.g., as described in relation to Figures 6a and 6b).

[0028] In various embodiments, the implementation system (120) may include an interface engine (121), an input processing engine (122), a user interaction engine (123), a context engine (124), a user identification engine (125), a training instance engine (126), a training engine (127), and an implementation engine (128) as illustrated in FIG. 1. In some embodiments, one or more of the engines (121-128) of the implementation system (120) may be omitted. In some embodiments, all or aspects of one or more of the engines (121-128) of the implementation system (120) may be combined. In some embodiments, one or more of the engines (121-128) of the implementation system (120) are computing devices (110 1-N It may be implemented as a component executed remotely, either partially or exclusively, from one or more of the following. In some embodiments, one or more of the engines (121-128) of the implementation system (120) or any part of the operation thereof may be implemented as a computing device (110 1-N It can be implemented in a component running locally, either partially or exclusively by one or more of the following.

[0029] Referring to FIG. 2, an exemplary process flow of the exemplary hardware and software environment of FIG. 1 is illustrated. For example, assume that audio data (201A) capturing a user's voice utterance is received by an execution system (120). Input processing engine (113 1-N / 122) can process audio data to generate processed input data (202). For example, an input processing engine (113 1-N / 122) can use an ASR model to process audio data (201A) and generate ASR data for speech utterances captured in audio data (201A). Additionally, an input processing engine (113 1-N / 122) can generate NLU data for speech utterances captured in audio data (201A) by processing ASR data for speech utterances captured in audio data (201A) using an NLU model. Additionally, the input processing engine (113 -N / 122 ) can provide audio data (201A), ASR data for speech utterances captured in audio data (201A), and / or NLU data for speech utterances captured in audio data (201A) to the implementation engine as processed input data (202).

[0030] Additionally, the user identification engine (125) may process audio data (201A) and / or non-audio data (201B) (e.g., vision data capturing the user who provided the voice utterance, user profile data of the user profile of the user who provided the voice utterance, device identifier data of the user’s computing device that received the voice utterance, etc.) to determine the user identity (225) of the user who provided the voice utterance. As described in more detail in relation to FIG. 3, the user identification engine may utilize audio-based identification techniques (e.g., text-dependent (TD), speaker identification (SID), text-independent (TI) SID) and / or non-audio-based identification techniques (e.g., face identification (FID), user profile identification, device identifier identification, etc.) to determine the user identity (225). In particular, when determining the user identity (225), the user identification engine (125) may utilize various user embeddings (e.g., stored in the user embedding(s) database (125A)). These various user embeddings may include TD speaker embeddings, TI speaker embeddings, face embeddings, and / or other embeddings. The generation and utilization of these embeddings to determine the user identity (225) is described in more detail herein (e.g., in relation to FIG. 3). Additionally, the user identification engine (125) may provide the user identity to the implementation engine (128).

[0031] Additionally, the fulfillment engine (128) may process at least processed input data (202) and user identity (225) to determine one or more fulfillment actions to be implemented in response to receiving a voice utterance. In some embodiments, the fulfillment engine (128) may process at least processed input data (202) and user identity (225) to determine a set of fulfillment actions. For example, the fulfillment engine (128) may generate one or more structured requests and transmit one or more of the structured requests to various applications to obtain the fulfillment actions included in the set. For example, the fulfillment engine (128) may generate one or more structured requests and transmit one or more of the structured requests to one or more OEM applications (181) (e.g., in an embodiment where the user is located in a vehicle (100A)), one or more first-party applications (182), and / or one or more third-party applications (183). As used herein, the term “first-party application” may refer to a software application developed and / or maintained by the same entity that develops and / or maintains the implementation system (120) described herein. Additionally, as used herein, the term “third-party application” may refer to a software application or system developed and / or maintained by an entity different from the entity that develops and / or maintains the implementation system (120) described herein. In response to one or more structured requests, one or more OEM applications (181) may transmit OEM data (281) to the implementation engine (128), one or more first-party applications (182) may transmit first-party data (282) to the implementation engine (128), and / or one or more third-party applications (183) may transmit third-party data (283) to the implementation engine (128).

[0032] In this embodiment, the implementation engine (128) may further process a set of implementation actions to determine one or more implementation actions to be implemented in response to receiving a voice utterance. The implementation engine (128) may optionally further process context signals obtained through the context engine (124) (e.g., user signal(s) characterizing the state of the user and / or environment signal(s) characterizing the state of the user's environment). For example, the implementation engine (128) may process one or more implementation actions to be implemented in response to receiving a voice utterance from a set of implementation actions, and optionally context signals, using implementation action models and / or implementation rule(s) specific to the user who provided the voice utterance as indicated by the user identity (225) (e.g., stored in the implementation action model(s) and / or rule(s) database (127A)). Additionally, the implementation engine (128) may cause one or more implementation actions to be implemented in response to receiving a voice utterance to satisfy the voice utterance. In other words, in this embodiment, the fulfillment system (120) may initially determine a set of fulfillment actions using conventional techniques. However, the fulfillment system (120) may subsequently process the set of fulfillment actions to select one or more fulfillment actions using a fulfillment action model and / or fulfillment rule(s) specific to the user who provided the voice utterance as indicated by the user identity (225). This enables the fulfillment system (120) to personalize the fulfillment of the voice utterance to the user who provided the voice utterance as indicated by the user identity (225).

[0033] In particular, a specific implementation action model for a user who has provided a voice utterance indicated by a user identity (225) may be previously trained, and / or implementation rule(s) specific for a user who has provided a voice utterance indicated by a user identity (225) may be previously generated. For example, before receiving a voice utterance captured in audio data (201A), the user interaction engine (123) may identify user interactions between one or more smart devices and a user in a given time instance among a plurality of time instances. User interactions may correspond to one or more implementation actions (223). In various embodiments, one or more implementation actions (223) of user interactions may be limited to those that cause one or more of the smart devices to change their states in some way. The user interaction engine (123) may provide one or more implementation actions (223) to the training instance engine (126).

[0034] Additionally, the context engine (124) may acquire one or more context signals that characterize the state of the user as indicated by user signal(s) (224A) and / or the state of the user's environment as indicated by environment signal(s) (224B) at a given time instance of user interaction. The user signal(s) (224A) and / or environment signal(s) (224B) may be utilized to define a context scenario, and the context scenario may subsequently be utilized to bias the interpretation of subsequent speech utterances received in the context scenario. The context engine (124) may provide the user signal(s) (224A) and environment signal(s) (224B) to the training instance engine (126). These context signal(s) and context scenarios are described in more detail herein (e.g., in relation to FIG. 4). As mentioned above, in inference, the context engine (124) may provide the context signals to the execution engine (128).

[0035] The training instance engine (126) may process one or more fulfillment actions (223) corresponding to user interactions, and user signal(s) (224A) and / or environment signal(s) (224B) to generate a training instance (226). Additionally, the training instance engine (126) may store the training instance (226) in a training instance(s) database (126A). Additionally, the training engine (127) may train a fulfillment action model specific to the user who provided the voice utterance and / or generate one or more fulfillment rules specific to the user who provided the voice utterance based on the training instance (226) and optionally one or more additional training instances generated in the same or similar manner. Training a fulfillment action model specific to the user who provided the voice utterance and / or generating one or more fulfillment rules specific to the user who provided the voice utterance is described in more detail herein (e.g., in relation to FIG. 4).

[0036] Now, referring to FIG. 3, a flowchart illustrating an exemplary method (300) for dynamically adapting the execution of a given voice utterance based on a user who provided the given voice utterance is shown. For convenience, the operations of the method (300) are described with reference to a system that performs the operations. This system of the method (300) is a computing device(s) (e.g., the computing device(s) of FIG. 1, FIG. 5a, FIG. 5b, FIG. 6a and FIG. 6b) (110 1-N It includes at least one processor, at least one memory, and / or other component(s) of the implementation system (120) of FIG. 1, the computing device (710) of FIG. 7, the remote server(s) and / or other computing devices). The operations of the method (300) are shown in a specific order, but this is not intended to be limiting. One or more operations may be rearranged, omitted, and / or added.

[0037] In block (352), the system receives a given voice utterance from a user and through a computing device. For example, the user may provide the given voice utterance to a computing device in the vehicle while the user is located in the user's vehicle, and the given voice utterance may be captured in audio data generated through the microphone(s) of the computing device in the vehicle (e.g., as described in relation to FIGS. 5a and 5b). Additionally, for example, the user may provide the given voice utterance to a mobile computing device or a standalone computing device while the user is located in the user's primary residence, and the given voice utterance may be captured in audio data generated through the microphone(s) of the mobile computing device or the standalone computing device (e.g., as described in relation to FIGS. 6a and 6b).

[0038] In block (354), the system determines a set of implementation actions based on processing a given voice utterance. Each implementation action is included in the set of implementation actions and facilitates satisfying the given voice utterance upon implementation. For example, as shown in block (354A), the system may use an ASR model to process audio data capturing the given voice utterance to generate ASR data for the given voice utterance. Additionally, as shown in block (354B), the system may use an NLU model to process the ASR data for the given voice utterance to generate NLU data for the given voice utterance. For example, assume that a given voice utterance provided by a user corresponds to "I am cold." In this case, the system may process audio data capturing the voice utterance to generate the recognized text of "I am cold" as ASR data for the given voice utterance. Additionally, the system may process the recognized text of "I am cold" to identify one or more intents for the given voice utterance, such as intents like "cold" or "increase heat." Additionally, the system may determine a set of fulfilled actions based on at least one intent. In this example, and assuming a given voice utterance is received in a vehicle environment, the set of fulfilled actions may include, for example, rolling up the vehicle's window(s), turning on the vehicle's heated seats, turning on the vehicle's blower, and / or any other actions that can be implemented to increase the temperature inside the vehicle.In contrast, assuming that a given voice utterance is received in a home environment, the set of implementation actions may include, for example, increasing the thermostat temperature of the primary residence corresponding to the home environment to a first temperature, increasing the thermostat temperature of the primary residence to a second temperature, and / or any other actions that can be implemented to increase the temperature inside the residence.

[0039] In block (356), the system determines whether the user who provided the given voice utterance corresponds to the first user or the second user. The system may utilize various techniques to determine whether the user who provided the given voice utterance corresponds to the first user or the second user. In some embodiments, the system may allow speaker identification (SID) to be performed to determine whether the user who provided the given voice utterance corresponds to the first user or the second user. For example, the system may process audio data capturing the given voice utterance using SID model(s) to generate a speaker embedding for the user who provided the given voice utterance. The generated speaker embedding may be compared in the embedding space with one or more previously generated speaker embeddings for one or more users of the computing device (e.g., embeddings associated with the user profiles of one or more users). For example, if the distance metric between a generated speaker embedding, determined based on a comparison in the embedding space, and a given one of one or more previously generated speaker embeddings for one or more users satisfies a threshold, the user may be determined to correspond to the user associated with the given one of one or more previously generated speaker embeddings. Otherwise, the user may be identified as a guest user.

[0040] In these embodiments, one or more previously generated speaker embeddings for one or more users may be generated based on output(s) generated based on the processing of one or more instances of audio data containing speech utterances from one or more users. For example, one or more previously generated speaker embeddings may be based on the average (or other combination) of multiple different embeddings generated based on the processing of corresponding instances of audio data containing corresponding speech utterances from one or more users. Additionally, in these embodiments, the SID model may be a text-independent (TI) SID model, a text-dependent (TD) SID model, and / or any other suitable SID model.

[0041] For example, in TD SID, one or more previously generated speaker embeddings for one or more users are generated based on speech utterances containing only one or more specific words or phrases. Additionally, when used, one or more users may be required to say one or more specific words or phrases for one or more TD SID speaker embeddings to be generated using the TD SID model, which can be effectively compared with one or more previously generated TD speaker embeddings for one or more users to determine whether the user who provided the given speech utterance corresponds to a first user or a second user. For example, one or more specific words or phrases in TD speaker recognition may be limited to one or more call phrases configured to call an automated assistant (e.g., hot words and / or trigger words such as "Hey Assistant," "Okay Assistant," and / or "Assistant") or one or more other words or phrases. In contrast, in TI SID, speech utterances processed using the TI SID model are not limited to one or more specific words or phrases. In other words, audio data based on a virtual arbitrary speech utterance can be processed using a TI SID model to generate TI SID speaker embeddings, which can be effectively compared with one or more previously generated TI speaker embeddings for one or more users to determine whether the user who provided the given speech utterance corresponds to a first user or a second user. Additionally, in various embodiments, one or more previously generated TI speaker embeddings for one or more users utilized in the TI SID are generated based on speech utterances that include heterogeneous words and / or phrases and are not limited to call words and / or phrases and / or any other specific speech utterances.

[0042] In additional or alternative embodiments, the system may perform other forms of identification to determine whether the user who provided a given voice utterance corresponds to a first user or a second user. For example, the system may perform face identification, fingerprint identification, and / or other types of biometric identification. For example, vision components of a client device may capture vision data including the face of the user who provided the voice utterance. In this case, the system may process the vision data using a Face Identification (FID) model to generate a face embedding of the user who provided the voice utterance. The face embedding of the user who provided the voice utterance may correspond to an embedding that can be compared in the embedding space with one or more previously generated embeddings corresponding to the face embeddings of one or more users of the computing device to determine whether the user who provided the given voice utterance corresponds to a first user or a second user. Additionally, for example, a fingerprint sensor may receive touch input from the user who provided the given voice utterance and compare the detected fingerprint with the known fingerprints of one or more users of the computing device to determine whether the user who provided the given voice utterance corresponds to a first user or a second user. Additionally, for example, the display interface of a client device may receive touch input from a user corresponding to a specific password of the client device and determine whether the user who provided a given voice utterance corresponds to a first user or a second user. As another example, the system may utilize device identifiers and / or user accounts associated with a computing device and / or an additional computing device communicating with the computing device, and determine whether the user who provided a given voice utterance corresponds to a first user or a second user based on inferring the user from the device identifiers and / or user accounts.

[0043] In block (358), the system determines how to dynamically adapt the execution of a given voice utterance based on determining whether the user who provided the given voice utterance from block (356) corresponds to a first user or a second user. For example, in a repetition of block (358), if the system determines that the user who provided the given voice utterance corresponds to a first user, the system may proceed to block (360). In block (360), the system selects one or more subsets of first execution actions from a set of execution actions from block (354) based on the user who provided the given voice utterance corresponding to a first user. In some embodiments, the system may select one or more subsets of first execution actions based on context signals characterizing the state of the user and / or the state of the user's environment. In block (362), the system enables one or more subsets of first execution actions to be implemented to satisfy the given voice utterance. The system may select one or more subsets of first implementation actions from a set of implementation actions using implementation action models and / or implementation rule(s) specific to the first user (e.g., personalized mapping(s)). For example, the system may select one or more first implementation actions to include in the subset by processing NLU data for a given voice utterance and / or set of implementation actions from block (354), and optionally context signals characterizing the state of the user who provided the given voice utterance and / or the state of the environment where the user is located when the given voice utterance is provided, using implementation action models and / or implementation rule(s) specific to the first user. Creating and utilizing implementation action models and / or implementation rule(s) specific to the given user (e.g., personalized mapping(s)) is described in more detail in relation to FIG. 4.

[0044] However, in a repetition of block (358), if the system determines that the user who provided the given voice utterance corresponds to the second user, the system may proceed to block (364). In block (364), the system selects one or more subsets of second implementation actions from the set of implementation actions from block (354) based on the user who provided the given voice utterance corresponding to the second user. In block (366), the system enables the one or more subsets of second implementation actions to be implemented to satisfy the given voice utterance. Similarly, the system may select one or more subsets of second implementation actions from the set of implementation actions using an implementation action model and / or implementation rule(s), but this is specific to the second user. For example, the system may select one or more second implementation actions to include in a subset by processing NLU data for a set of given voice utterances and / or implementation actions from block (354), and optionally context signals characterizing the state of the user who provided the given voice utterance and / or the state of the environment where the user is located when the given voice utterance is provided, using implementation action models and / or implementation rule(s) specific to a second user. As mentioned above, generating and utilizing implementation action models and / or implementation rule(s) specific to a given user is described in more detail in relation to FIG. 4. In particular, a subset of one or more second implementation actions may include at least one unique implementation action from the set of implementation actions compared to a subset of one or more first implementation actions. In other words, the implementation of the same voice utterance may vary depending on the user who provided the voice utterance.

[0045] Continuing with the above example where a given voice utterance provided by a user corresponds to "I am cold," it is further assumed that the given voice utterance was provided by a first user. In this example, and assuming the given voice utterance was provided in a vehicle environment, a transition action associated with turning on the first user's heated seats may be selected and implemented, because a transition action model and / or transition rule(s) specific to the first user may have learned that the first user typically turns on only the vehicle's heated seats and does not turn on the blower when it is cold. Additionally, assuming the system can access various environmental signals characterizing the state of the vehicle and that these environmental signals indicate that the vehicle's windows are rolled down when the voice utterance is received, a transition action associated with rolling up the windows may be selected and implemented, because a transition action model and / or transition rule(s) specific to the first user may have learned that the first user generally rolls up the windows when it is cold when the windows are rolled down. Additionally, in this example, and assuming the given voice utterance was provided in a home environment, since the first user-specific implementation action model and / or implementation rule(s) may have learned that the first user typically utilizes the thermostat in this way when it is cold, implementation actions associated with raising the thermostat temperature by N degrees (e.g., N is a positive integer) or raising the thermostat temperature to a first specific temperature (e.g., 74 degrees) may be selected and implemented.

[0046] In contrast, it is further assumed that the given voice utterance is provided by a second user. In this example, and assuming the given voice utterance is provided in a vehicle environment, a transition action associated with turning on the second user's heated seat and turning on the blower may be selected and implemented, because the transition action model and / or transition rule(s) specific to the second user may have learned that the second user typically turns on only both the heated seat and the blower of the vehicle when cold. Also, in this example, and assuming the given voice utterance is provided in a home environment, a transition action associated with raising the thermostat temperature by M degrees (e.g., where M is a positive integer different from N) or raising the thermostat temperature to a second specific temperature (e.g., 75 degrees) may be selected and implemented, because the transition action model and / or transition rule(s) specific to the second user may have learned that the second user typically uses the thermostat in this way when cold.

[0047] From blocks (362 and 366), the system returns to block (352) and performs additional iterations of the method (300) of FIG. 3 based on additional given voice utterances received through the computing device. Thus, when different users provide the same voice utterance (e.g., “I am cold” in the example above), the system can dynamically adapt the fulfillment of that same voice utterance based on how different users typically perform different actions in the same or similar contextual scenarios. As a result, the fulfillment of a given voice utterance is improved, and the amount of received user interaction and / or user input is reduced so that the desired fulfillment action is performed in instances where an incorrect fulfillment action is initially performed, because the fulfillment of a given voice utterance may not be specific to the user who provided the given voice utterance.

[0048] In various embodiments, the system may display the reason why one or more fulfillment actions were selected and implemented to satisfy a given voice utterance for corresponding users, which is to be provided to corresponding users for audible and / or visual presentation via a computing device. The display of the reason why one or more fulfillment actions were selected and implemented to satisfy a given voice utterance may be generated using fulfillment action model(s) and / or fulfillment rule(s). Continuing with the above example, assuming the given voice utterance was provided by a first user in a vehicle environment, the display of the reason why one or more fulfillment actions were selected and implemented to satisfy the given voice utterance may correspond to "I turned on the heated seat rather than the blower because it is a typical action to do when it is cold in the car." Synthetic voice audio data including the synthesized voice capturing the display may be audibly rendered to be presented to the first user via the speaker(s) of the computing device and / or text data corresponding to the display may be visually rendered to be presented to the first user via the display of the computing device. In particular, an indication of the reason why one or more fulfillment actions were selected and implemented to satisfy a given voice utterance may be provided only to present a threshold number of times (e.g., at least once) to educate corresponding users, but avoids overwhelming and bothering users with the same information.

[0049] In various embodiments, the system may generate a prompt requesting the user to provide feedback on one or more selected and implemented actions, and may provide the prompt to be presented to the user. Based on user input received from the user in response to the prompt, the system may cause the user to update specific implementation action model(s) and / or implementation rule(s). For example, the system may generate the aforementioned indication (e.g., "I turned on the heated seat, not the fan, because that is what is typically done when it is cold in the car") and additionally or alternatively generate a prompt asking "Is that correct?" In this example, if positive user input is received, the system may generate a positive training instance to bias subsequent implementations of the same voice utterance toward the selection and implementation of one or more first implementation actions. However, if negative user input is received, the system may generate a negative training instance to bias subsequent implementations of the same voice utterance away from the selection and implementation of one or more first implementation actions.

[0050] In an additional or alternative embodiment, the system may utilize one or more subsequent user interactions as feedback for one or more selected and implemented fulfillment actions. For example, if the system selects and implements a fulfillment action associated with turning on the first user's heated seat based on a given voice utterance "I am cold" provided by the first user, but the first user turns off the heated seat subsequently after the implementation of the fulfillment action, the system may generate negative training instances to bias subsequent fulfillments of the same voice utterance away from the selection and implementation of one or more first fulfillment actions (e.g., the fulfillment action associated with turning on the first user's heated seat). In contrast, the system selects and implements a fulfillment action associated with turning on the first user's heated seat based on a given voice utterance "I am cold" provided by the first user, and if the first user does not turn off the heated seat following the implementation of the fulfillment action, the system may generate a positive training instance to bias subsequent fulfillments of the same voice utterance from the selection and implementation of one or more first fulfillment actions (e.g., a fulfillment action associated with turning on the first user's heated seat) based on inferring that the fulfillment action was correctly implemented in response to the given voice utterance.

[0051] It should be understood that while FIG. 3 is described herein with respect to specific utterances and specific fulfillment actions, this is for illustrative purposes only and is not intended to be limiting. Also, it should be understood that while FIG. 3 is described herein with respect to only the first user and the second user, this is also for illustrative purposes only and is not intended to be limiting. For example, the number of users considered in determining the user who provided a given voice utterance may be based on the number of users associated with the computing device. If the user who provided a given voice utterance is not determined to correspond to any known users, the user may be considered a default user, and fulfillment action model(s) and / or fulfillment rule(s) not specific to any user may be utilized to dynamically adapt the fulfillment of a given voice utterance.

[0052] Now, referring to FIG. 4, a flowchart illustrating an exemplary method (400) for creating training instance(s) to train a user-specific execution action model and training the execution action model for subsequent use to dynamically adapt the execution of a given voice utterance provided by the user is shown. For convenience, the operations of the method (400) are described with reference to a system performing the operations. This system of the method (400) is a computing device(s) (e.g., the computing device(s) of FIG. 1, FIG. 5a, FIG. 5b, FIG. 6a and FIG. 6b) (110 1-N It includes at least one processor, at least one memory, and / or other component(s) of the implementation system (120) of FIG. 1, the computing device (710) of FIG. 7, the remote server(s) and / or other computing devices). The operations of the method (400) are shown in a specific order, but this is not intended to be limiting. One or more operations may be rearranged, omitted, and / or added.

[0053] In block (452), the system identifies the occurrence of a user interaction between one or more smart devices and a user in a given time instance among a plurality of time instances, and the user interaction corresponds to one or more implementation actions. The system may consider any user interaction that can be communicably coupled to one or more smart devices and causes the state of one or more smart devices to change as a result of the occurrence of the user interaction. Additionally, one or more implementation actions may correspond to any specific action performed by a user that causes the state of one or more smart devices to change. One or more smart devices may include, for example, the user's computing devices (e.g., mobile computing devices, in-vehicle computing devices, etc.), vehicle components of the user's vehicle (e.g., window defrosters, power windows and doors, radio, blower, heated seat, etc.), the user's Internet of Things (IoT) devices (e.g., smart TV, smart thermostat, smart alarm system, smart appliance, etc.), and / or any other smart devices that the user can interact with and thereby cause the state of the smart devices to change.

[0054] In block (454), the system obtains one or more context signals that characterize the state of the user at a given time instance and / or the state of the user's environment at a given time instance. In other words, the system obtains one or more context signals at the same given time instance in which the occurrence of a user interaction is identified and determines a context scenario in which the user performs one or more actions of the user interaction. One or more context signals characterizing the state of the user at a given time instance may be based on user profile data of the user's user profile(s), application data of application(s) accessible to the system, sensor data instances of sensor data generated by one or more sensors within the user's environment (e.g., vehicle sensor(s), computing device sensor(s), home sensor(s), etc.), and / or any other context signals characterizing the state of the user. Additionally, the state of the user determined based on one or more context signals may include, for example, the user's location or activity (e.g., whether the user is located at the user's primary residence, whether the user is driving a vehicle, etc.), the time of day at the user's location, the day of the week at the user's location, and / or any other state of the user determined based on one or more context signals. Additionally, one or more context signals characterizing the state of the user's environment at a given time instance may be based on sensor data instances of sensor data generated by one or more sensors within the user's environment (e.g., vehicle sensor(s), computing device sensor(s), home sensor(s), etc.) and / or any other context signals characterizing the state of the user's environment.Additionally, the state of the user's environment determined based on one or more context signals may include, for example, ambient noise detected in the user's environment, the ambient indoor and / or outdoor temperature of the user's environment, whether other users are located together in the user's environment (e.g., the user's primary residence, passengers in a vehicle, etc.), and / or any other state of the user's environment determined based on one or more context signals.

[0055] In block (456), the system generates a given user training instance based on user interaction at a given time instance and based on one or more context signals at a given time instance. Additionally, the system may store the given user training instance in a training instance(s) database (e.g., the training instance(s) database (126A) of FIG. 1). A given training instance may include a training instance input and a training instance output. The training instance input may include, for example, (i) one or more context signals that characterize / specify the state of the user at a given time instance or characterize the state of the user's environment at a given time instance, and (ii) a set of implementation actions associated with one or more context signals. In particular, the set of implementation actions includes at least one implementation action corresponding to the user interaction and at least one additional implementation action. In some embodiments, the training instance input may include one or more terms or phrases describing one or more context signals, which may subsequently be provided by the user to describe a context scenario having a low ambient temperature (e.g., terms such as "cold," "freezing," "chilly," etc.). In other words, the training instance input may include a given context scenario indicated by one or more context signals and a set of implementation actions predicted to be associated with the given context scenario. The set of implementation actions predicted to be associated with the given context scenario may be defined by the system, the user, and / or a developer associated with other actors.For example, if the ambient temperature within the user's environment is less than X degrees (e.g., X is a positive integer) as indicated by one or more context signals, the set of implementation actions may include actions that, when implemented, help increase the ambient temperature within the user's environment. Additionally, the training instance output may include one or more implementation actions of user interaction. In particular, one or more implementation actions of user interaction may include each of the implementation actions of the set of implementation actions or a subset thereof. In other words, the training instance output may include implementation actions actually implemented by the user in a given context scenario.

[0056] In some embodiments, as shown in block (456A), the system prompts the user to verify whether one or more implementation actions have been performed based on the user's state at a given time instance and / or the state of the user's environment at a given time instance before generating a given user training instance. For example, assume the case where the user manually turns on a heated seat in a vehicle. The system may identify the user manually turning on the heated seat as a user interaction that includes an implementation action to increase the ambient temperature of the user's environment. Additionally, the system may identify the ambient temperature of the user's environment (e.g., inside the vehicle and / or outside the vehicle) as one or more context signals at a given time instance where the user manually turns on the heated seat. In such embodiments, and before generating a given training instance based on these user interactions, the system may prompt the user to verify whether the user manually turned on the heated seat because the user was cold. In additional or alternative embodiments, the system may simply infer this without prompting the user.

[0057] In block (458), the system determines whether one or more conditions are satisfied to train a user-specific action model. One or more conditions for training a user-specific action model may include, for example, whether there is a threshold quantity of training instances available to train the action model, time of day, day of the week, and / or other conditions. For example, in implementations where the system is implemented locally on the user's computing device, one or more conditions may additionally or alternatively include whether the computing device is charged, whether the computing device has at least a threshold charge state, whether the temperature of the computing device is below a threshold, and whether the computing device is maintained by the user. In an iteration of block (458), if the system determines that one or more conditions are not satisfied, the system proceeds to block (460).

[0058] In block (460), the system determines whether there is a subsequent occurrence of a user interaction (e.g., including one or more identical implementation actions) or an additional user interaction added to a user interaction (e.g., including at least one unique implementation action compared to one or more implementation actions of the user interaction). In an iteration of block (460), if the system determines that there is a subsequent occurrence of a user interaction or an additional user interaction, the system returns to block (452) and performs an additional iteration of the actions of blocks (452-458) for the subsequent occurrence of a user interaction or an additional user interaction to create an additional given user training instance and determine whether one or more conditions are satisfied to train a user-specific implementation action model. In an iteration of block (460) and / or a subsequent iteration, if the system determines that there is no subsequent occurrence of a user interaction or an additional user interaction, the system may return to block (458). Also, in an iteration of block (458), if the system determines that one or more conditions are satisfied, the system proceeds to block (462). In other words, the system may continuously monitor the occurrence of user interactions during the training phase, generate training instances based on these user interactions, and / or continuously monitor the satisfaction of one or more conditions until the system decides to train an implementation action model based on the training instances.

[0059] In block (462), the system trains a user-specific implementation action model based on at least a given training instance. Additionally, the system may store the trained implementation action model in one or more databases accessible to the system (e.g., the implementation action model(s) and / or rule(s) database (127A) of FIG. 1). In block (464), the system utilizes the user-specific implementation action model to respond to a voice utterance received from the user (e.g., as described in relation to FIG. 3). The implementation action model may be any suitable ML model that can be trained in any suitable manner, including, for example, a classification model (e.g., a Support Vector Machine (SVM) model, a k-nearest neighbors (KNN) model, a Naive Bayes model, etc.), a regression model, and / or other types of machine learning (ML) models. For example, when training a transition action model based on a given training instance generated in block (456), the system may use the transition action model to process a set of one or more context signals and transition actions included in the training instance input to generate a predicted output. In this example, the predicted output may correspond to one or more predicted transition actions from a set of transition actions that are predicted to be performed by a user based on a given context scenario defined by one or more context signals. Additionally, one or more predicted transition actions (or their representations) of the predicted output may be compared with one or more transition actions (or their representations) of user interactions included in the training instance output to generate one or more losses. Additionally, the transition action model may be updated based on one or more losses. For example, one or more losses may be backpropagated across the transition action model to update their weight(s).In particular, this process may be repeated for a plurality of additional training instances created in the manner described herein to further train and personalize a user-specific implementation action model. Additionally, this process may be repeated for a plurality of different users to train and personalize corresponding implementation action models specific to those users.

[0060] In additional or alternative implementations, the system may present various context scenarios to the user and may request the user to identify one or more implementation actions to perform given various context scenarios. In such implementations, the system may create training instances to train an implementation action model and / or generate implementation rule(s) based on how the user responds to the various context scenarios. For example, the system may present that the ambient temperature of the vehicle is low (e.g., a context scenario where the user may feel cold) and may list various actions that the user may perform in response to the low ambient temperature (e.g., the training instance inputs of the aforementioned given training instance). Additionally, the system may utilize the user's selection of one or more actions as the training instance output.

[0061] In various implementations, the fulfillment action model and / or fulfillment rule(s) may be updated over time. For example, the system may continuously generate training instances based on occurrences of user interactions identified at various time instances and one or more context signals acquired at these various time instances. The system may update the fulfillment action model and / or fulfillment rule(s) whenever it is determined that one or more conditions are subsequently satisfied. Additionally, for example, the system may update the fulfillment action model and / or fulfillment rule(s) based on changing context scenarios. For example, during the autumn season, a user may roll up one or more windows of a vehicle when it is cold but may not turn on any heated seats. However, as the autumn season transitions into the winter season, the user may not have rolled down the windows and may turn on the heated seats. Therefore, the fulfillment action model and / or fulfillment rule(s) are not only specific to the user at a given time instance but are also updated over time to ensure better fulfillment of voice utterances for the user.

[0062] In various embodiments, the system may assign one or more voice commands to training instances and / or fulfillment rule(s) so that the system may subsequently determine that voice utterances received in the same or similar context scenarios are mapped to different fulfillment actions for different users. For example, one or more voice commands may include one or more terms or phrases describing one or more context signals and may be subsequently provided by the user. For example, terms such as "cold," "very cold," and "chilly" may be assigned to training instances and / or fulfillment rule(s) associated with context signals indicating the low ambient temperature of the user's environment when one or more fulfillment actions of user interaction are performed to increase the low ambient temperature of the environment. Additionally, for example, terms such as "hot," "scorching," and "blistering" may be assigned to training instances and / or fulfillment rule(s) associated with context signals indicating the high ambient temperature of the user's environment when one or more fulfillment actions of user interaction are performed to decrease the high ambient temperature of the environment. Additionally, terms such as, for example, "dark," "dim," and "pitch black" may be assigned to training instances and / or implementation rule(s) associated with context signals indicating that the environment has low ambient light when one or more implementation actions of user interaction are performed to increase the ambient light of the environment. Accordingly, it should be understood that the technology described herein can be utilized not only to train implementation action model(s) and / or generate implementation rule(s) as described herein, but also to assign one or more voice commands to specific context scenarios that a user may encounter.

[0063] In additional or alternative embodiments, the system may generate a personalized mapping for a user that maps one or more voice commands to one or more fulfillment actions (e.g., also referred to herein as fulfillment rule(s)). For example, the system may generate a personalized mapping based on one or more past instances of the user independent of providing any voice utterance that trigger the performance of one or more fulfillment actions corresponding to user interaction in a context scenario. Additionally, the system may determine that a context scenario is correlated with one or more voice commands using various techniques. For example, the system may map different context scenarios represented by different context signals that characterize the state of the user and / or the state of the environment when one or more fulfillment actions are performed in different parts of a lower-dimensional space (e.g., embedding space, semantic space, etc.). Additionally, one or more voice commands may be assigned to different context scenarios in the lower-dimensional space. Accordingly, when a given voice command is subsequently included in a given voice utterance received from a user, the system can determine a given context scenario associated with the given voice command in a low-dimensional space and enable one or more given fulfillment actions previously performed in the given context scenario to be implemented to satisfy the given voice utterance. Additionally, for example, the system can map different context scenarios represented by different context signals that characterize the user's state and / or the state of the environment when one or more fulfillment actions are performed with different intents. Furthermore, one or more voice commands can be assigned to different intents.Accordingly, when a given voice command is subsequently included in a given voice utterance received from a user, the system determines a given context scenario associated with the given voice command determined based on a given intent, and can enable one or more given fulfillment actions previously performed in the given context scenario to be implemented to satisfy the given voice utterance.

[0064] In some versions of these implementation examples, the system may generate a personalized mapping in response to determining that only one or more fulfillment actions have been performed by the user in a context scenario for a threshold quantity of instances (e.g., 1, 3, 5, and / or one or more other positive integers). For example, assume that when the ambient temperature inside the vehicle is between 45 and 60 degrees Fahrenheit, there have been three past instances of the user turning on the vehicle's heated seats rather than the vehicle's vents, and that the three past instances satisfy a threshold quantity of instances. In this example, the system maps the context scenario (e.g., the ambient temperature inside the vehicle is between 45 and 60 degrees Fahrenheit) to a first part of the low-dimensional space and / or associates it with a "cold" intent, and assigns voice commands such as "cold" or "very cold" to the first part of the low-dimensional space and / or associates them with a "cold" intent. Therefore, even if the user has not provided any voice utterances during three past instances, when the user subsequently provides the voice utterance "I am cold" when the ambient temperature inside the vehicle is between 45 and 60 degrees Fahrenheit, the system can utilize personalized mapping to automatically turn on the heated seats.

[0065] In particular, in these implementations, the same voice command may be mapped to different fulfillment actions. Continuing with the example above, it is further assumed that there were three past instances where the user turned on the vehicle's heated seats and the vehicle's blower when the ambient temperature inside the vehicle was below 45 degrees Fahrenheit, and that the three past instances satisfy a threshold quantity of instances. In this example, the system maps the context scenario (e.g., the ambient temperature inside the vehicle is below 45 degrees Fahrenheit) to a heterogeneous second part of the lower-dimensional space and / or associates it with a "cold" intent, and assigns voice commands such as "cold" and "very cold" to a heterogeneous second part of the lower-dimensional space and / or associates them with a "cold" intent. However, based on the different fulfillment actions performed by the user in these three instances (e.g., turning on both the vehicle's heated seats and the vehicle's blower), these past instances may be mapped to a heterogeneous second part of the lower-dimensional space but may also be associated with the same "cold" intent. Therefore, even if the user has not provided any voice utterances during three past instances, when the user subsequently provides the voice utterance "I am cold" when the ambient temperature inside the vehicle is below 45 degrees Fahrenheit, the system can utilize personalized mapping to automatically turn on the heated seats and automatically turn on the blower.

[0066] It should be understood that FIG. 4 is described herein in relation to a specific user interaction having a specific implementation action, but is for illustrative purposes only and is not intended to be limiting. Additionally, FIG. 4 is described herein in relation to training an implementation action model for a single user and / or generating implementation rule(s), but it should also be understood that this is for illustrative purposes only and is not intended to be limiting. For example, it should be understood that a corresponding iteration of the method (400) of FIG. 4 may be implemented to train a corresponding implementation action model and / or generate corresponding implementation rule(s) for a number of different users.

[0067] Now, referring to FIGS. 5a and 5b, various non-limiting examples are illustrated of dynamically adapting the execution of a given voice utterance in a vehicle environment based on the user who provided the given voice utterance. For example, a vehicle (100A) and an in-vehicle computing device (110) from FIG. 1 N It is assumed that the vehicle includes ) and is at least partially shared by a number of users—a first user (102) as shown in FIG. 5a and a second user (104) as shown in FIG. 5b. Additionally, it is assumed that action execution models have been previously trained for both the first user (102) and the second user (104) and / or that execution rules have been previously generated for both the first user (102) and the second user (104). Accordingly, the in-vehicle computing device (110) of the vehicle (100A) N A fulfillment system (e.g., the fulfillment system (120) of FIG. 1) that is at least partially executed in ) may utilize these fulfillment models and / or fulfillment rules specific to the first user (102) and the second user (104) to determine how to dynamically adapt the fulfillment of voice utterances.

[0068] Specifically, referring to FIG. 5a, it is further assumed that the first user (102) is driving the vehicle (100A) and provides a voice utterance (552A) saying "I am cold," and also the in-vehicle computing device (110) of the vehicle (100A) N It is further assumed that the microphone(s) of the voice utterance (552A) generate audio data that captures the voice utterance (552A). In this example, the audio data that captures the voice utterance (552A) may be processed using an ASR model to generate ASR data for the voice utterance (552A) (e.g., recognized text of “I am cold”). Additionally, the ASR data for the voice utterance (552A) may be processed using an NLU model to generate NLU data for the voice utterance (552A) (e.g., recognized intent(s) such as “cold”, “temperature increase”). In particular, the user who provided the voice utterance (552A) may be determined to correspond to the first user (102) using any of the techniques described herein and / or any other suitable techniques.

[0069] In some embodiments, NLU data for a voice utterance (552A) may be processed using a implementation engine to determine a set of implementation actions that help increase the temperature inside the vehicle (100A) when implemented. A set of implementation actions may include, for example, rolling up the windows of the vehicle (100A) to increase the temperature inside the vehicle (100A) (when the windows of the vehicle (100A) are in a state indicating that they are rolled down), turning on the blower of the vehicle (100A) to circulate warm air throughout the vehicle to increase the temperature inside the vehicle (100A), and turning on the heated seat of the seat occupied by the first user (102) to increase the temperature inside the vehicle (100A) and / or the temperature of the first user (102) located in the vehicle (100A), and / or other implementation actions that can be performed to increase the temperature inside the vehicle (100A) and / or the temperature of the first user (102) located in the vehicle (100A).

[0070] In such an embodiment, a set of implementation actions may be processed using an implementation action model and / or implementation rule(s) specific to the first user (102) to select one or more first implementation actions to be implemented to satisfy a voice utterance (552A). One or more context signals obtained from the voice utterance and / or the voice utterance at a time instance provided by the first user (102) may be additionally or alternatively processed using an implementation action model and / or implementation rule(s) specific to the first user (102) in selecting one or more first implementation actions to be implemented to satisfy the voice utterance (552A). For example, one or more first implementation actions may include turning on a heated seat of a seat occupied by the first user (102) to increase the temperature inside the vehicle (100A) and / or the temperature of the first user (102) located in the vehicle (100A), but (e.g., an in-vehicle computing device (110 N (As indicated by the synthesized voice (555A) of “I turned on the heated seat, not the blower” audibly presented to the user through the speaker(s) of the first user (102), it may not include turning on the blower of the vehicle (100A) to circulate warm air throughout the vehicle to increase the temperature inside the vehicle (100A) based on the output generated using a specific implementation action model and / or implementation rule(s) for the first user (102). In other words, based on the output generated using a specific implementation action model and / or implementation rule(s) for the first user (102), the implementation of the voice utterance (552A) is dynamically adapted to the implementation action(s) that the first user (102) would manually perform in the same or similar context scenario.

[0071] In an additional or alternative embodiment, one or more first implementation actions to be implemented to satisfy a voice utterance (552A) may be directly selected without considering a set of implementation actions (e.g., using implementation rule(s). For example, one or more first implementation actions may be mapped to various voice commands that cause one or more first implementation actions to be implemented when detected in a specific context scenario. For example, when creating one or more implementation rules specific to the first user (102), a context scenario matching that illustrated in FIG. 5a (e.g., when the first user (102) is cold in the vehicle (100A)) may be encountered. Thus, voice commands including terms such as "cold," "very cold," etc. may be pre-assigned to one or more first implementation actions for the first user (102).

[0072] In contrast, and specifically with reference to FIG. 5b, it is further assumed that a second user (104) is driving a vehicle (100A) and provides a voice utterance (552B) of “I am cold.” In particular, the voice utterance (552B) provided by the second user (104) in FIG. 5b is identical to the voice utterance (552A) provided by the first user (102) in FIG. 5a. The voice utterance (552B) of “I am cold” provided by the second user (104) may be processed in the same or similar manner as described above with respect to the voice utterance (552A) in FIG. 5a. However, one or more second implementation actions implemented to satisfy the voice utterance (552B) in FIG. 5b are different from one or more first implementation actions implemented to satisfy the voice utterance (552A) in FIG. 5a. For example, one or more second execution actions may include turning on a heated seat of a seat occupied by a second user (104) to increase the temperature inside the vehicle (100A) and / or the temperature of a first user (102) located in the vehicle (100A), and also an execution action specific to the second user (104). Based on the output generated using the model and / or implementation rule(s) (e.g., in-vehicle computing device (110) N This may include turning on the blower of the vehicle (100A) to circulate warm air throughout the vehicle to increase the temperature inside the vehicle (100A), as indicated by the synthesized voice (554B) of “heated seat and blower turned on” audibly presented to the user through the speaker(s) of the vehicle (100A). In other words, based on the output generated using a specific implementation action model and / or implementation rule(s) for the second user (104), the implementation of the voice utterance (552B) is dynamically adapted to the implementation action(s) that the second user (104) would manually perform in the same or similar context scenario.

[0073] FIGS. 5a and FIGS. 5b both describe a first user (102) and a second user (104) driving the vehicle (100A) in separate time instances, but it should be understood that this is for illustrative purposes only and is not intended to be limiting. For example, the techniques described herein may be utilized when both the first user (102) and the second user (104) are located together in the vehicle environment. As a non-limiting example, it is assumed that the first user (102) is driving the vehicle (100A) and the second user (104) is a passenger of the vehicle (100A). In other words, the first user (102) may be associated with the "driver" zone of the vehicle (100A), and the second user (104) may be associated with the "front passenger" zone of the vehicle (100A). These zones may be determined based on the occupancy sensor(s) of the vehicle (100A), and / or any of the techniques described herein and / or any other suitable technique for identifying users in the vehicle (100A). Additionally, it is assumed that the first user (102) provides the same voice utterance from FIG. 5a and FIG. 5b. In this example, the heated seat of the first user (102) in the "driver" zone may be turned on, but other heated seats of the vehicle may not be turned on and the blower may not be turned on as described in connection with one or more first implementation actions of FIG. 5a. In some of these examples, the heated seat of the second user (104) and the blower associated with the "front passenger" zone may be turned on as described in connection with one or more second implementation actions of FIG. 5b. However, any blower associated with the "driver zone" may remain off. Additionally, it is assumed that the second user (104) provides the same voice utterance from FIG. 5a and FIG. 5b (e.g., before or after the first user (102) provides the same voice utterance from FIG. 5a and FIG. 5b).In this example, the heated seat of the second user (104) in the "front passenger" zone can be turned on, and the blower associated with the "front passenger" zone can be turned on as described in relation to FIG. 5b. Thus, in various embodiments, one or more implementation actions implemented for different users located together in the environment may be limited to the zone determined to be occupied by the user who provided the voice utterance.

[0074] Now, referring to FIGS. 6a and 6b, various non-limiting examples are illustrated for dynamically adapting the execution of a given voice utterance in a primary residential environment based on the user who provided the given voice utterance. The floor plan illustrated in FIGS. 6a and 6b includes a plurality of rooms (660-672). A plurality of computing devices ( 110 2-6 ) is placed across at least some of the rooms. Computing device (110 2-6Each of the following may implement an instance of a transfer system client (e.g., the transfer system client (114) of FIG. 1) configured in a selected embodiment of the present disclosure and may include one or more input devices, such as microphone(s), capable of capturing voice utterances by a nearby user. For example, computing devices taking the form of interactive standalone speakers are placed in the room (660) which is the kitchen of FIG. 6a and 6b (e.g., kitchen computing device (1102)), the room (664) which is the bedroom of FIG. 6a and 6b (e.g., bedroom computing device (1104)), and the room (666) which is the living room of FIG. 6a and 6b (e.g., living room computing device (1104)). Additionally, computing devices taking the form of so-called "smart" televisions are placed in the study room (662) of FIGS. 6a and 6b (e.g., study TV (1103)), and the kitchen room (660) of FIGS. 6a and 6b (e.g., kitchen TV (1106)). Various users of the primary residence have multiple computing devices (1102 -6 It can interact with ) to allow other smart device(s) in the primary residence to be controlled (e.g., causing smart device(s) to change state(s).

[0075] The smart devices of the primary residence illustrated in FIGS. 6a and 6b may include, for example, one or more smart lights(s) in the rooms (660-672) of the primary residence, one or more smart thermostat(s) in the rooms (660-672) of the primary residence, smart appliance(s) of the kitchen 660 in FIGS. 6a and 6b (e.g., smart oven, smart toaster, smart refrigerator, etc.), smart appliance(s) of the laundry room 672 in FIGS. 6a and 6b (e.g., smart washing machine, smart dryer, etc.), smart lock(s) on the door(s) of the primary residence, a smart alarm system of the primary residence (e.g., smart contact sensor(s), smart camera(s), smart motion sensor(s), etc.), and / or any other smart devices. While specific smart devices are described herein for the home environment of FIGS. 6a and 6b, it should be understood that this is for illustrative purposes only and is not intended to be limiting. For example, referring again briefly to FIGS. 5a and 5b, the vehicle (100A) may also include a plurality of smart devices. These smart devices include smart windows of the vehicle (100A), smart locks of the vehicle (100A), a smart defroster of the vehicle (100A), smart heated seats of the vehicle (100A), and an in-vehicle computing device (110) of the vehicle N Smart infotainment systems operating in the vehicle, smart blower systems of the vehicle, and / or other smart devices that can be integrated with the vehicle (100A) may be included, but are not limited to. In such examples, user interactions corresponding to one or more execution actions may include changing the state of any of these smart devices.

[0076] Although not shown in FIGS. 6a and 6b, a plurality of computing devices (106 2-6) can be coupled to communicate with smart devices in a primary residence and / or other resources (e.g., the Internet) through one or more wired or wireless WANs and / or LANs (e.g., through the network(s) (195) of FIG. 1). Additionally, other computing devices—in particular, mobile devices, e.g., smartphones, tablets, laptops, wearable devices, etc.—can also be carried by one or more people in the home, e.g., and may or may not be connected to the same WANs and / or LANs. The configuration of computing devices and smart devices shown in FIG. 6a and 6b is merely one example; It should be understood that more or fewer and / or different computing devices and / or smart devices may be placed across any number of different rooms and / or areas of a residence and / or in locations other than residential residences (e.g., businesses, hotels, public places, airports, vehicles, and / or other locations or spaces).

[0077] For the example of FIGS. 6a and 6b, it is assumed that the home environment is at least partially cohabited by a number of users, such as the first user (102) and the second user (104), as illustrated in FIGS. 6a and 6b. Additionally, it is assumed that implementation action models have been previously trained for both the first user (102) and the second user (104) and / or that implementation rules have been previously created for both the first user (102) and the second user (104). Accordingly, the computing devices (110) of the residence 2-6 A implementation system that is partially executed in at least one or more of the following (e.g., the implementation system (120) of FIG. 1) may utilize these implementation models and / or implementation rules specific to the first user (102) and the second user (104) to determine how to dynamically adapt the implementation of voice utterances.

[0078] Specifically, referring to FIG. 6a, it is further assumed that a first user (102) is located in the room (666), which is the living room of FIG. 6a and FIG. 6b, and provides a voice utterance (652A) of “dark,” and that microphone(s) of the living room computing device (1105) generate audio data that captures the voice utterance (652A). In this example, the audio data that captures the voice utterance (652A) may be processed in the same or similar manner as described in relation to FIG. 5a and FIG. 5b to determine one or more fulfillment actions to be performed in response to receiving the voice utterance (652A). In particular, the user who provided the voice utterance (652A) may be determined to correspond to the first user (102) using any of the techniques described herein and / or any other suitable techniques. As a result, one or more first implementation actions may be implemented to make the living room less dark, such as turning on the smart lights in the living room to 100% brightness (e.g., as indicated by the synthesized voice (654A) saying “Lights turned on to 100% brightness”).

[0079] Specifically, referring to FIG. 6b, it is further assumed that a second user (104) is located in the study room (662) of FIG. 6a and FIG. 6b and provides a voice utterance (652B) of “dark,” and that the microphone(s) of the study TV (1103) generate audio data that captures the voice utterance (652B). In this example, the audio data capturing the voice utterance (652B) may be processed in the same or similar manner as described in relation to FIG. 5a and FIG. 5b to determine one or more fulfillment actions to be performed in response to receiving the voice utterance (652B). In particular, the user who provided the voice utterance (652A) may be determined to correspond to the second user (104) using any of the techniques described herein and / or any other suitable techniques. As a result, one or more second implementation actions can be implemented to make the study less dark, such as turning on the smart light in the study but only turning on the light at 50% brightness (as indicated by the synthesized speech (654B) of “turned on the light to 50% brightness”).

[0080] In particular, the voice utterances provided by the first user (102) and the second user (104) in FIGS. 6a and 6b are identical. However, the fulfillment actions implemented in response to the voice utterances differ in that the living room lights are turned on to 100% brightness based on the first user (102) providing the voice utterance (652A), and the study lights are turned on to 50% brightness based on the second user (104) providing the voice utterance (652B). In some embodiments, this difference in fulfillment actions may be based on different fulfillment action models previously trained for both the first user (102) and the second user (104) and / or fulfillment rules previously generated for both the first user (102) and the second user (104), as described in relation to FIGS. 5a and 5b. In other embodiments, this difference in execution action may be based on different zones where voice utterances are received (e.g., the "living room" zone for the first user (102) and the "study" zone for the second user (104)). Nevertheless, the technique described herein is utilized to dynamically adapt the execution of the same voice utterance.

[0081] Now, referring to FIG. 7, a block diagram of an exemplary computing device (710) that may be optionally utilized to perform one or more aspects of the technologies described herein is shown. In some embodiments, one or more computing devices, one or more vehicles, and / or other component(s) may include one or more components of the exemplary computing device (710).

[0082] The computing device (710) typically includes at least one processor (714) that communicates with a number of peripheral devices via a bus subsystem (712). These peripheral devices may include, for example, a storage subsystem (724) including a memory subsystem (725) and a file storage subsystem (726), user interface output devices (720), user interface input devices (722), and a network interface subsystem (716). The input and output devices allow user interaction with the computing device (710). The network interface subsystem (716) provides an interface to external networks and is coupled to corresponding interface devices within other computing devices.

[0083] User interface input devices (722) may include pointing devices such as keyboards, mice, trackballs, touchpads, or graphic tablets, audio input devices such as scanners, touchscreens integrated into displays, voice recognition systems, microphones, and / or other types of input devices. Generally, the use of the term “input device” is intended to include all possible types of devices and methods for inputting information into a computing device (710) or a communication network.

[0084] The user interface output device (720) may include a non-visual display such as a display subsystem, a printer, a fax machine, or an audio output device. The display subsystem may include a flat-panel device such as a cathode ray tube (CRT) or a liquid crystal display (LCD), a projection device, or some other mechanism for generating a visible image. The display subsystem may also provide a non-visual display, for example, through audio output devices. Generally, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device (710) to a user or another machine or computing device.

[0085] The storage subsystem (724) stores programming and data configurations that provide the functions of some or all of the modules described herein. For example, the storage subsystem (724) may include logic for implementing various components illustrated in FIG. 1 and FIG. 2, as well as performing selected embodiments of the methods disclosed herein.

[0086] These software modules are generally executed by the processor (714) alone or in combination with other processors. The memory (725) used in the storage subsystem (724) may include multiple memories, including a main random access memory (RAM) (730) for storing instructions and data during program execution and a read-only memory (ROM) (732) for storing fixed instructions. The file storage subsystem (726) may provide permanent storage for program and data files and may include a hard disk drive, a floppy disk drive, a CD-ROM drive, an optical drive, or a removable media cartridge, along with associated removable media. Modules implementing the functions of specific embodiments may be stored by the file storage subsystem (726) in the storage subsystem (724) or in other machines accessible by the processor(s) (714).

[0087] The bus subsystem (712) provides a mechanism for various components and subsystems of the computing device (710) to communicate with each other as intended. Although the bus subsystem (712) is schematically illustrated as a single bus, alternative embodiments of the bus subsystem (712) may use multiple buses.

[0088] The computing device (710) may be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the constantly changing nature of computers and networks, the description of the computing device (710) illustrated in FIG. 7 is intended only as a specific example for the purpose of illustrating some embodiments. Many other configurations of the computing device (710) may have more or fewer components than the computing device illustrated in FIG. 7.

[0089] In situations where the systems described herein collect personal information about users, or otherwise monitor, or use personal and / or monitored information, users may be provided with the opportunity to control whether programs or features collect user information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or user's current geographic location), or to control whether and / or how to receive content from a content server that may be more relevant to the user. Additionally, certain data may be processed in one or more ways before being stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that personally identifiable information cannot be determined about the user, or the user's geographic location may be generalized from where geographic location information is obtained (such as at the city, zip code, or state level) so that the user's specific geographic location cannot be determined. Thus, the user may have control over how information about the user is collected and / or used.

[0090] In some embodiments, a method is provided that is implemented by one or more processors, and the method comprises the steps of receiving a given voice utterance from a user and through a computing device—the given voice utterance is provided while the user is located in the user's vehicle—; and determining a set of implementation actions based on processing the given voice utterance. Each implementation action included in the set of implementation actions helps to satisfy the given voice utterance when implemented. The method further comprises the steps of determining whether the user who provided the given voice utterance corresponds to a first user or a second user; and in response to the determination that the user who provided the given voice utterance corresponds to a first user: selecting a subset of one or more first implementation actions from the set of implementation actions based on the determination that the user who provided the given voice utterance corresponds to a first user; and enabling the one or more subsets of first implementation actions to be implemented to satisfy the given voice utterance.

[0091] These and other embodiments of the technology disclosed in this specification may optionally include one or more of the following features.

[0092] In some embodiments, the method may further include the step of selecting one or more subsets of second implementation actions from a set of implementation actions based on the determination that the user who provided the given voice utterance corresponds to a second user. The second user may be an additional user in addition to the first user, and the subset of one or more second implementation actions may include at least one implementation action not included in the subset of one or more first implementation actions. The method may further include the step of enabling one or more second implementation actions to satisfy the given voice utterance.

[0093] In some versions of such embodiments, the first user and the second user may be located together in the vehicle when a given voice utterance is received. Additionally, implementing one or more subsets of first implementation actions to satisfy a given voice utterance may include implementing one or more first implementation actions for a first zone of the vehicle among a plurality of zones of the vehicle occupied by the first user, and implementing one or more subsets of second implementation actions to satisfy a given voice utterance may include implementing one or more second implementation actions for a second zone of the vehicle among a plurality of zones of the vehicle occupied by the second user.

[0094] In an additional or alternative version of such an embodiment, the step of selecting one or more subsets of first implementation actions from a set of implementation actions based on a user who provided a given voice utterance corresponding to a first user may include the step of processing a set of implementation actions to select one or more subsets of first implementation actions using a first user implementation action model specific to the first user and / or one or more first user implementation rules specific to the first user. In some additional version of such an embodiment, the step of selecting one or more subsets of second implementation actions from a set of implementation actions based on a user who provided a given voice utterance corresponding to a second user may include the step of processing a set of implementation actions to select one or more subsets of second implementation actions using a second user implementation action model specific to the second user and / or one or more second user implementation rules specific to the second user.

[0095] In some embodiments, the method may further include the step of, following the implementation of one or more first implementation actions to satisfy a given voice utterance: providing an indication of the reason why one or more first implementation actions are implemented to satisfy a given voice utterance to be presented to a first user.

[0096] In some embodiments, the step of processing a given speech utterance may include: using an Automatic Speech Recognition (ASR) model to process audio data capturing the given speech utterance to generate ASR data for the given speech utterance; and using a Natural Language Understanding (NLU) model to process the ASR data for the given speech utterance to generate NLU data for the given speech utterance. In some versions of such embodiments, determining a set of implementation actions based on processing the given speech utterance may include determining a set of implementation actions based on the NLU data for the given speech utterance.

[0097] In some embodiments, the step of determining whether a user who provided a given voice utterance corresponds to a first user or a second user may include the step of processing audio data capturing a given voice utterance using a Speaker Identification (SID) model to determine whether a user who provided a given voice utterance corresponds to a first user or a second user. In some versions of such embodiments, the step of processing audio data capturing a given voice utterance using a SID model to determine whether a user who provided a given voice utterance corresponds to a first user or a second user may include, in response to determining that the voice utterance is not long enough to perform a Text Independent (TI) SID, the step of processing audio data using a Text Dependent (TD) SID model as a SID model to generate a TD speaker embedding; the step of comparing the TD speaker embedding with one or more stored TD speaker embeddings in a TD SID embedding space; and the step of determining whether a user who provided a given voice utterance corresponds to a first user or a second user based on comparing the TD speaker embedding with one or more stored TD speaker embeddings. In some additional versions of such embodiments, determining the identity of the user who provided the given speech utterance by processing audio data that captures the given speech utterance using a SID model, in response to determining that the speech utterance is of sufficient length to perform TI SID, may include: processing audio data using the TI SID model as a SID model to generate a TI speaker embedding; comparing the TI speaker embedding with one or more stored TI speaker embeddings in the TI SID embedding space; and determining whether the user who provided the given speech utterance corresponds to a first user or a second user based on comparing the TI speaker embedding with one or more stored TI speaker embeddings.In some versions of such an embodiment, audio data capturing a given voice utterance may be generated through one or more microphones of a computing device.

[0098] In some embodiments, the step of determining whether a user who provided a given voice utterance corresponds to a first user or a second user may include: processing vision data capturing the user who provided the given voice utterance to generate a face embedding using a face identification (FID) model; comparing the face embedding with one or more stored face embeddings in a face embedding space; and determining whether the user who provided the given voice utterance corresponds to a first user or a second user based on comparing the face embedding with one or more stored face embeddings. In some versions of such embodiments, the vision data capturing the user who provided the given voice utterance may be generated through one or more vision components of a computing device or an additional computing device coupled to the computing device for communication.

[0099] In some embodiments, the step of determining whether the user who provided the given voice utterance corresponds to the first user or the second user may include the step of utilizing a corresponding device identifier of a computing device or an additional computing device communicateably coupled to the computing device to determine whether the user who provided the given voice utterance corresponds to the first user or the second user.

[0100] In some embodiments, the step of determining whether the user who provided the given voice utterance corresponds to the first user or the second user may include the step of utilizing a corresponding user account of a computing device or an additional computing device communicateably coupled to the computing device to determine whether the user who provided the given voice utterance corresponds to the first user or the second user.

[0101] In some embodiments, a method is provided that is implemented by one or more processors, and the method comprises the steps of: receiving a given voice utterance from a user and through a computing device—the given voice utterance is provided while the user is located at the user's residence—; and determining a set of implementation actions based on processing the given voice utterance. Each implementation action included in the set of implementation actions helps to satisfy the given voice utterance when implemented. The method further comprises the steps of: determining whether the user who provided the given voice utterance corresponds to a first user or a second user; and in response to the determination that the user who provided the given voice utterance corresponds to a first user: selecting a subset of one or more first implementation actions from the set of implementation actions based on the determination that the user who provided the given voice utterance corresponds to a first user; and enabling the one or more subset of first implementation actions to be implemented to satisfy the given voice utterance.

[0102] In some embodiments, a method is provided that is implemented by one or more processors and comprises the steps of: identifying the occurrence of a user interaction between one or more smart devices and a user in a given time instance of a plurality of time instances, wherein the user interaction corresponds to one or more implementation actions; acquiring one or more context signals that characterize the state of the user in a given time instance and / or characterize the state of the user's environment in a given time instance; generating a given training instance based on the user interaction and based on one or more context signals; training a user-specific implementation action model based on at least a given training instance in response to determining that one or more training conditions are satisfied; and utilizing a user-specific implementation action model in response to voice utterances received from the user.

[0103] These and other embodiments of the technology disclosed in this specification may optionally include one or more of the following features.

[0104] In some embodiments, the method may further comprise, before generating a given training instance based on user interaction and based on one or more context signals: generating a prompt requesting the user to verify whether one or more implementation actions have been performed based on the user's state at a given time instance and / or the state of the user's environment at a given time instance; making the prompt available to be presented to the user; and receiving user input in response to the prompt verifying that one or more implementation actions have been performed based on the user's state at a given time instance and / or the state of the user's environment at a given time instance. In some versions of such embodiments, generating a given training instance based on user interaction and based on one or more context signals may be done in response to receiving verification that one or more implementation actions have been performed based on the user's state at a given time instance and / or the state of the user's environment at a given time instance.

[0105] In some embodiments, the step of generating a given training instance based on user interaction and based on one or more context signals may include, for the given training instance, determining a training instance input, wherein the training instance input comprises (i) one or more context signals characterizing the state of the user at a given time instance and / or the state of the user's environment at a given time instance, and (ii) a set of implementation actions associated with one or more context signals; and determining a training instance output, wherein the training instance output comprises one or more implementation actions of the user interaction. In some versions of such embodiments, the step of training a user-specific implementation action model based on at least the given training instance may include, using the implementation action model, processing the training instance input to generate a predicted output; comparing the predicted output with the training instance output to generate one or more losses; and updating the implementation action model based on one or more losses.

[0106] In some embodiments, one or more training conditions may include a quantity of training instances available to train a performance action model, a time of day, and / or a day of the week.

[0107] In some embodiments, the method may further include the steps of: identifying the occurrence of an additional user interaction between an additional user and one or more smart devices in an additional given time instance among a plurality of time instances, wherein the additional user interaction corresponds to one or more additional fulfillment actions, and the one or more additional fulfillment actions include at least one fulfillment action that is different from the one or more fulfillment actions; acquiring one or more additional context signals that characterize an additional state of the additional user in an additional given time instance and / or characterize an additional environment state of the additional user in an additional given time instance; generating an additional given training instance based on the additional user interaction and based on one or more additional context signals; training an additional fulfillment action model specific to the additional user based on at least the additional given training instance in response to determining that one or more training conditions are satisfied; and utilizing an additional fulfillment action model specific to the additional user in response to a voice utterance received from the additional user.

[0108] In some embodiments, a method is provided that is implemented by one or more processors and comprises the steps of: identifying the occurrence of a user interaction between one or more smart devices and a user in a given time instance of a plurality of time instances, wherein the user interaction corresponds to one or more implementation actions; acquiring one or more context signals that characterize the state of the user in a given time instance and / or characterize the state of the user's environment in a given time instance; generating a prompt that requests the user to verify whether one or more implementation actions have been performed based on the state of the user in a given time instance and / or the state of the user's environment in a given time instance; making the prompt available to be presented to the user; receiving user input in response to the prompt that verifies that one or more implementation actions have been performed based on the state of the user in a given time instance and / or the state of the user's environment in a given time instance; generating one or more implementation action rules specific to the user in response to receiving user input that verifies that one or more implementation actions have been performed based on the state of the user in a given time instance and / or the state of the user's environment in a given time instance; and making the one or more implementation action rules specific to the user available in response to voice utterances received from the user.

[0109] These and other embodiments of the technology disclosed in this specification may optionally include one or more of the following features.

[0110] In some embodiments, the method comprises: identifying the occurrence of an additional user interaction between one or more smart devices and an additional user in an additional given time instance of a plurality of time instances, wherein the additional user interaction corresponds to one or more additional implementation actions, and the one or more additional implementation actions include at least one implementation action that is different from the one or more implementation actions; acquiring one or more additional context signals that characterize an additional state of the additional user in an additional given time instance and / or characterize an additional environment state of the additional user in an additional given time instance; generating an additional prompt that requests the additional user to verify whether one or more additional implementation actions have been performed based on the additional state of the additional user in an additional given time instance and / or the additional environment state of the additional user in an additional given time instance; causing the additional prompt to be provided to be presented to the additional user; and receiving an additional user input in response to the additional prompt that verifies that one or more additional implementation actions have been performed based on the additional state of the additional user in an additional given time instance and / or the additional environment state of the user in an additional given time instance. The method may further include: a step of generating one or more additional fulfillment action rules specific to an additional user in response to receiving additional user input verifying that one or more additional fulfillment actions have been performed based on an additional state of an additional user and / or an additional state of an additional environment in an additional given time instance; and a step of utilizing one or more additional fulfillment action rules specific to an additional user to respond to a voice utterance received from an additional user.

[0111] In some embodiments, a method is provided that is implemented by one or more processors and includes the step of generating a personalized mapping for a user of one or more voice commands for one or more fulfillment actions. The step of generating a personalized mapping responds to determining one or more past instances of a user that cause the performance of one or more fulfillment actions in context scenarios determined to be correlated with one or more voice commands, independently of providing any voice utterance. The method further includes the steps of receiving a given voice utterance from a user and after generating the personalized mapping; determining, based on processing the given voice utterance, that the given voice utterance is one or more of voice commands; and using the personalized mapping to automatically implement one or more fulfillment actions in response to determining that the given utterance is one or more of voice commands.

[0112] These and other embodiments of the technology disclosed in this specification may optionally include one or more of the following features.

[0113] In some embodiments, the step of generating a personalized mapping of one or more voice commands for one or more fulfillment actions may include generating a mapping between a voice command semantic representation of one or more voice commands and a fulfillment action semantic representation of one or more fulfillment actions in a semantic space and based on one or more past instances; and storing the mapping in one or more databases.

[0114] In some embodiments, the step of generating a personalized mapping of one or more voice commands for one or more fulfillment actions may include generating a mapping between one or more intents of voice commands and one or more fulfillment actions based on one or more past instances; and storing the mapping in one or more databases.

[0115] In some embodiments, a method is provided that is implemented by one or more processors and includes the steps of: identifying the occurrence of a user interaction between one or more smart devices and a user at a given time instance of a plurality of time instances, wherein the user interaction corresponds to one or more fulfillment actions; and acquiring one or more context signals that characterize a context scenario of the user. The one or more context signals that characterize a context scenario of the user characterize the state of the user at a given time instance and / or the state of the user's environment at a given time instance. The method further includes the step of identifying one or more voice commands to be assigned to a context scenario of the user. One or more voice commands cause one or more fulfillment actions of the user interaction to be implemented when subsequently provided by the user. The method includes the steps of: receiving an instance of a given voice utterance from the user and at a subsequent time instance of a plurality of time instances, wherein the subsequent time instance follows the given time instance; determining, based on processing the instance of the given voice utterance, that the instance of the given voice utterance contains one or more voice commands; and determining that one or more fulfillment actions are implemented to satisfy the instance of the given voice utterance. and further includes a step of enabling one or more fulfillment actions to satisfy an instance of a given voice utterance.

[0116] These and other embodiments of the technology disclosed in this specification may optionally include one or more of the following features.

[0117] In some embodiments, the method may further include the step of identifying an additional occurrence of additional user interaction between one or more smart devices and an additional user in an additional subsequent time instance of a plurality of time instances, wherein the additional user interaction corresponds to one or more additional fulfillment actions, and the one or more additional fulfillment actions include at least one fulfillment action different from one or more fulfillment actions, and the additional subsequent time instance follows a given time instance; and the step of acquiring one or more additional context signals characterizing an additional context scenario of the additional user. The one or more additional context signals characterizing an additional context scenario of the additional user may characterize an additional state of the additional user in an additional subsequent time instance and / or characterize an additional environment state of the additional user in an additional subsequent time instance, and the additional context scenario of the additional user may correspond to a context scenario of the user. The method may further include the step of identifying one or more voice commands to be assigned to an additional context scenario of the additional user. One or more voice commands may cause one or more additional fulfillment actions of the additional user interaction to be implemented when subsequently provided by the additional user. The method may further include the steps of: receiving an additional instance of a given voice utterance from an additional user and in another subsequent time instance among a plurality of time instances, wherein the other subsequent time instance follows the additional subsequent time instance; determining, based on processing the additional instance of the given voice utterance, that the additional instance of the given voice utterance includes one or more voice commands; determining that one or more additional fulfillment actions are implemented to satisfy the additional instance of the given voice utterance; and the step of implementing one or more additional fulfillment actions to satisfy the additional instance of the given voice utterance.

[0118] Additionally, some embodiments include one or more processors of one or more computing devices (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s)), and / or tensor processing unit(s) (TPU(s))), wherein one or more processors are operable to execute instructions stored in associated memory, and the instructions are configured to cause the execution of any of the methods described above. Some embodiments also include one or more non-transient computer-readable storage media storing computer instructions executable by one or more processors to perform any of the methods described above. Some embodiments also include a computer program product comprising instructions executable by one or more processors to perform any of the methods described above.

[0119] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein.

Claims

Claim 1 A method implemented by one or more processors, comprising: receiving a given voice utterance from a user via a computing device, wherein the given voice utterance is provided while the user is located within the user's vehicle; determining a set of implementation actions based on processing the given voice utterance, wherein each implementation action included in the set of implementation actions facilitates satisfying the given voice utterance upon implementation, and each implementation action included in the set of implementation actions is associated with controlling a corresponding component of the vehicle; determining whether the user who provided the given voice utterance corresponds to a first user or a second user; in response to determining that the user who provided the given voice utterance corresponds to a first user: selecting a subset of one or more first implementation actions from the set of implementation actions based on the user who provided the given voice utterance corresponding to the first user, wherein the subset of one or more first implementation actions is associated with controlling at least a first component of the vehicle; and enabling the one or more subset of first implementation actions to be implemented to satisfy the given voice utterance. A method comprising: in response to determining that the user who provided the given voice utterance corresponds to a second user: selecting a subset of one or more second implementation actions from a set of implementation actions based on the user who provided the given voice utterance corresponding to the second user, wherein the second user is an additional user other than the first user, and the subset of one or more second implementation actions is associated with controlling at least a second component of the vehicle that is not controlled when the subset of one or more first implementation actions is implemented; and a step of enabling the one or more second implementation actions to satisfy the given voice utterance. Claim 2 A method according to claim 1, wherein the first user and the second user are located together in a vehicle when a given voice utterance is received, and the step of implementing one or more first implementation actions to satisfy the given voice utterance includes the step of implementing one or more first implementation actions for a first area of ​​the vehicle occupied by the first user among a plurality of areas of the vehicle, and the step of implementing one or more second implementation actions to satisfy the given voice utterance includes the step of implementing one or more second implementation actions for a second area of ​​the vehicle occupied by the second user among a plurality of areas of the vehicle. Claim 3 In claim 1, one or more first implementation actions from the set of implementation actions based on the user who provided the given voice utterance corresponding to the first user. of A method comprising the step of selecting a subset, wherein the step of processing the set of implementation actions using a first user implementation action model specific to the first user and / or one or more first user implementation rules specific to the first user to select a subset of one or more first implementation actions. Claim 4 In paragraph 3, one or more second implementation actions from the set of implementation actions based on the user who provided the given voice utterance corresponding to the second user. of A method comprising the step of selecting a subset, wherein the step of processing the set of implementation actions using a second user implementation action model specific to the second user and / or one or more second user implementation rules specific to the second user to select a subset of one or more second implementation actions. Claim 5 A method according to claim 1, further comprising the step of, in response to a user who provided the given voice utterance determining that the user corresponds to the first user, enabling a subset of the one or more first implementation actions to satisfy the given voice utterance: providing an indication of the reason why the one or more first implementation actions were implemented to satisfy the given voice utterance to the first user. Claim 6 The method of claim 1, wherein the step of processing the given voice utterance comprises: processing audio data capturing the given voice utterance to generate ASR data for the given voice utterance using an Automatic Voice Recognition (ASR) model; and processing ASR data for the given voice utterance to generate NLU data for the given voice utterance using a Natural Language Understanding (NLU) model. Claim 7 In claim 6, the step of determining a set of execution actions based on processing the given voice utterance comprises determining a set of execution actions based on NLU data for the given voice utterance. Claim 8 A method according to claim 1, wherein the step of determining whether the user who provided the given voice utterance corresponds to a first user or a second user comprises the step of processing audio data capturing the given voice utterance to determine whether the user who provided the given voice utterance corresponds to a first user or a second user using a Speaker Identification (SID) model. Claim 9 In claim 8, the step of processing audio data capturing the given speech utterance to determine whether the user who provided the given speech utterance corresponds to the first user or the second user by using the SID model comprises: in response to determining that the speech utterance is not long enough to perform a text-independent (TI) SID, the step of processing the audio data to generate a TD speaker embedding by using a text-independent (TD) SID model as the SID model; the step of comparing the TD speaker embedding with one or more stored TD speaker embeddings in a TD SID embedding space; and the step of determining whether the user who provided the given speech utterance corresponds to the first user or the second user based on comparing the TD speaker embedding with the one or more stored TD speaker embeddings. Claim 10 In claim 9, the step of processing audio data capturing the given voice utterance to determine the identity of the user who provided the given voice utterance using the SID model comprises, in response to determining that the voice utterance is of sufficient length to perform TI SID: processing the audio data to generate a TI speaker embedding using the TI SID model as the SID model; comparing the TI speaker embedding with one or more stored TI speaker embeddings in the TI SID embedding space; and determining whether the user who provided the given voice utterance corresponds to a first user or a second user based on comparing the TI speaker embedding with one or more stored TI speaker embeddings. Claim 11 In claim 8, the method wherein audio data capturing the given voice utterance is generated through one or more microphones of a computing device. Claim 12 The method of claim 1, wherein the step of determining whether the user who provided the given voice utterance corresponds to a first user or a second user comprises: processing vision data that captures the user who provided the given voice utterance to generate a face embedding using a face identification (FID) model; comparing the face embedding with one or more stored face embeddings in a face embedding space; and determining whether the user who provided the given voice utterance corresponds to a first user or a second user based on comparing the face embedding with one or more stored face embeddings. Claim 13 In claim 12, the method of capturing vision data for the user who provided the given voice utterance is generated through one or more vision components of a computing device or an additional computing device communicably coupled to the computing device. Claim 14 A method according to claim 1, wherein the step of determining whether the user who provided the given voice utterance corresponds to the first user or the second user comprises: utilizing the corresponding device identifier of a computing device or an additional computing device communically coupled to the computing device to determine whether the user who provided the given voice utterance corresponds to the first user or the second user. Claim 15 A method according to claim 1, wherein the step of determining whether the user who provided the given voice utterance corresponds to the first user or the second user comprises the step of utilizing the corresponding user account of a computing device or an additional computing device communically coupled to the computing device to determine whether the user who provided the given voice utterance corresponds to the first user or the second user. Claim 16 A method implemented by one or more processors, comprising: receiving a given voice utterance from a user via a computing device, wherein the given voice utterance is provided while the user is at the user's residence; determining a set of implementation actions based on processing the given voice utterance, wherein each implementation action included in the set of implementation actions facilitates satisfying the given voice utterance upon implementation, and each implementation action included in the set of implementation actions is associated with controlling the corresponding Internet of Things (IoT) device and using the corresponding set of parameters; determining whether the user who provided the given voice utterance corresponds to a first user or a second user; in response to determining that the user who provided the given voice utterance corresponds to a first user: selecting a subset of one or more first implementation actions from the set of implementation actions based on the user who provided the given voice utterance corresponding to the first user, wherein the subset of one or more first implementation actions is associated with controlling the corresponding IoT device using the first set of parameters; and enabling the one or more subset of first implementation actions to be implemented to satisfy the given voice utterance. A method comprising: in response to determining that the user who provided the given voice utterance corresponds to a second user: selecting a subset of one or more second implementation actions from a set of implementation actions based on the user who provided the given voice utterance corresponding to the second user, wherein the second user is an additional user other than the first user, and the subset of one or more second implementation actions is associated with controlling the corresponding IoT device using a second parameter set different from a first parameter set; and a step of enabling the one or more second implementation actions to satisfy the given voice utterance. Claim 17 In paragraph 16, the method wherein the first user and the second user are located together in a residence when the given voice utterance is received. Claim 18 In claim 16, the step of selecting a subset of one or more first implementation actions from a set of implementation actions based on a user who provided a given voice utterance corresponding to the first user comprises processing the set of implementation actions using at least one of a first user implementation action model specific to the first user and one or more first user implementation rules specific to the first user to select a subset of actions of the one or more first implementation actions, and the step of selecting a subset of one or more second implementation actions from a set of implementation actions based on a user who provided a given voice utterance corresponding to the first user comprises processing the set of implementation actions using at least one of a second user implementation action model specific to the second user and one or more second user implementation rules specific to the second user to select a subset of actions of the one or more second implementation actions. Claim 19 In claim 16, the method further comprises, following the step of making a subset of one or more first implementation actions to satisfy the given voice utterance in response to a user who has provided the given voice utterance determining that such user corresponds to the first user: making an indication of the reason why the one or more first implementation actions were implemented to satisfy the given voice utterance to be presented to the first user. Claim 20 A method implemented by one or more processors, comprising: generating a personalized mapping of one or more voice commands for one or more fulfillment actions for a user, wherein the step of generating the personalized mapping is in response to determining one or more past instances of the user that cause the one or more fulfillment actions to be performed in context scenarios determined to be correlated with the one or more voice commands, regardless of providing any voice utterance; receiving a given voice utterance after generating the personalized mapping from the user; determining, based on processing the given voice utterance, that the given voice utterance is one of one or more of the voice commands; and in response to determining that the given voice utterance is one of one or more voice commands, using the personalized mapping so that the one or more fulfillment actions are automatically implemented. Claim 21 In claim 20, the step of generating a personalized mapping of one or more voice commands for one or more execution actions comprises: generating a mapping between a voice command semantic representation of one or more voice commands and an execution action semantic representation of one or more execution actions in a semantic space based on one or more past instances; and storing the mapping in one or more databases. Claim 22 A method according to claim 20, wherein the step of generating a personalized mapping of one or more voice commands for one or more fulfillment actions comprises: generating a mapping between the intent of the one or more voice commands and the one or more fulfillment actions based on one or more past instances; and storing the mapping in one or more databases. Claim 23 A method implemented by one or more processors, comprising: identifying the occurrence of a user interaction between one or more smart devices and a user in a given time instance of a plurality of time instances, wherein the user interaction corresponds to one or more implementation actions; acquiring one or more context signals characterizing the user's context scenario, wherein the one or more context signals characterizing the user's context scenario characterize the state of the user in the given time instance and / or characterize the state of the user's environment in the given time instance; and identifying one or more voice commands to be assigned to the user's context scenario, wherein the one or more voice commands cause one or more implementation actions of the user interaction to be implemented when subsequently provided by the user. A method comprising: receiving an instance of a given voice utterance from the user in a subsequent time instance of the plurality of time instances, wherein the subsequent time instance follows the given time instance; determining, based on processing the instance of the given voice utterance, that the instance of the given voice utterance includes one or more of the voice commands; determining that the one or more implementation actions are implemented to satisfy the instance of the given voice utterance; and the step of enabling the one or more implementation actions to satisfy the instance of the given voice utterance. Claim 24 In paragraph 23, the step of identifying an additional occurrence of an additional user interaction between one or more smart devices and an additional user in an additional subsequent time instance of the plurality of time instances, wherein the additional user interaction corresponds to one or more additional fulfillment actions, wherein the one or more additional fulfillment actions include at least one fulfillment action different from the one or more fulfillment actions, and the additional subsequent time instance follows the given time instance; the step of obtaining one or more additional context signals characterizing an additional context scenario of the additional user, wherein the one or more additional context signals characterizing an additional context scenario of the additional user characterize an additional state of the additional user in an additional subsequent time instance and / or characterize an additional environment state of the additional user in an additional subsequent time instance, and the additional context scenario of the additional user matches the context scenario of the user; the step of identifying one or more voice commands to be assigned to an additional context scenario of the additional user, wherein the one or more voice commands cause one or more additional fulfillment actions of the additional user interaction to be implemented when subsequently provided by the additional user; A method comprising: receiving an additional instance of the given voice utterance from the additional user in another subsequent time instance of a plurality of time instances, wherein the other subsequent time instance follows the additional subsequent time instance; determining, based on processing the additional instance of the given voice utterance, that the additional instance of the given voice utterance includes one or more of the voice commands; determining that the one or more additional fulfillment actions are implemented to satisfy the additional instance of the given voice utterance; and further comprising the step of implementing the one or more additional fulfillment actions to satisfy the additional instance of the given voice utterance. Claim 25 A system comprising: one or more processors; and a memory for storing instructions, wherein when the instructions are executed, the one or more processors are configured to perform the operations of any one of claims 1 through 24. Claim 26 A non-transient computer-readable storage medium for storing instructions, wherein the instructions, when executed, cause one or more processors to perform the operations of any one of claims 1 through 24. Claim 27 A computing device for a vehicle, comprising: one or more processors; and a memory for storing instructions, wherein when the instructions are executed, the one or more processors are configured to perform the operations of any one of claims 1 to 24. Claim 28 delete Claim 29 delete Claim 30 delete Claim 31 delete Claim 32 delete Claim 33 delete Claim 34 delete

Images