Dynamic adaptation to the fulfillment of a given speech utterance based on the user who provided the given speech utterance.

Abstract

and determining how to execute the voice utterance based on a user who provided the voice utterance. For example, an embodiment may receive a voice utterance from a user, determine a set of fulfillment actions for the voice utterance, and determine whether the user who provided the voice utterance corresponds to a first user or a second user. Further, in response to determining that the user corresponds to the first user, an embodiment may select a subset of first fulfillment actions from the set and cause the first subset of fulfillment actions to be performed to satisfy the voice utterance. Further, in response to determining that the user corresponds to a second user, an embodiment may select a different subset of second fulfillment actions from the set and cause the second subset of fulfillment actions to be performed to satisfy the voice utterance.

Description

[Background technology]

[0001] Humans may engage in human-computer interaction using interactive software applications referred to herein as “Automated Assistants” (also known as “Chatbots,” “Interactive Personal Assistants,” “Intelligent Personal Assistants,” “Personal Voice Assistants,” “Conversational Agents,” etc.). For example, a human (who may be referred to as a “User” when interacting with an Automated Assistant) may provide speech natural language input (i.e., speech utterances) directed to an Automated Assistant by providing natural language text (e.g., typed) that can, in some cases, be converted to text and then processed, and / or directed to the Automated Assistant. These speech utterances and / or typed inputs often contain commands directed to the Automated Assistant. The Automated Assistant generally responds to these commands by providing responsive user interface output(s) (e.g., audible user interface output(s) and / or visual user interface output(s), controlling smart devices(s), and / or performing other actions(s).

[0002] These automated assistants typically rely on a pipeline of components to interpret and respond to these spoken and / or typed inputs. For example, an automatic speech recognition (ASR) engine can process audio data corresponding to a user's spoken utterances to generate ASR output, such as a transcript of the utterance (i.e., a sequence of terms and / or other tokens). Furthermore, a natural language understanding (NLU) engine can process the ASR output (or typed input) to generate NLU output, such as a user's intent, when providing spoken utterances and optionally slot values ​​to parameters associated with the intent. Additionally, a fulfillment engine can process the NLU output and use it to obtain fulfillment output, such as responsive user interface output, responsive smart device control commands, and / or other responsive actions.

[0003] In some cases, these automated assistants may achieve specific voice utterances in a particular way. For example, suppose an automated assistant is running at least partially in the in-vehicle computing device of a given user's vehicle, and the user directs the voice utterance "cold" to the automated assistant. In this example, the given automated assistant can utilize the aforementioned pipeline to respond to the voice utterance, resulting in a fulfillment action of turning on the blower in the vehicle to satisfy the voice utterance. However, the given user who provided the voice utterance may have only wanted the heated seats in the vehicle to be turned on, not the blower, which may differ from other users' preferences. As a result, the automated assistant has performed the wrong fulfillment action, and the given user may need to manually turn off the blower, turn on the heated seats, and / or initiate additional interaction with the automated assistant to have the desired fulfillment action performed. As a result, computer resources are wasted in these interactions based on the execution of incorrect fulfillment actions (and optionally, any additional interactions resulting from the execution of incorrect fulfillment actions), and natural resources are wasted in this interaction based on turning on the blower when it was not the desired fulfillment action. [Overview of the project]

[0004] The embodiments described herein relate to dynamically adapting the fulfillment of a given speech utterance based on the user who provided the speech utterance, and optionally based on the contextual scenario in which the speech utterance is received. In some embodiments, a system (including, for example, a processor and memory) can receive a given speech utterance from a user, determine a set of fulfillment actions for the given speech utterance, and determine whether the user who provided the given speech utterance corresponds to a first user or a second user. Furthermore, in response to determining that the user corresponds to a first user, the system can select a subset of first fulfillment actions from the set and have the system perform the subset of first fulfillment actions to satisfy the given speech utterance. Furthermore, in response to determining that the user corresponds to a second user, the system can select a different subset of second fulfillment actions from the set and have the system perform the subset of second fulfillment actions to satisfy the given speech utterance. In particular, a subset of the second(multiple) fulfillment actions may include at least one unique fulfillment action compared to a subset of the first(multiple) fulfillment actions. Thus, the fulfillment of the same speech utterance delivered by different users can be dynamically adapted based on whether the speech utterance is delivered by a first or second user.

[0005] For example, suppose a first user is driving a vehicle and provides the voice utterance "It's hot," and the vehicle's onboard computing device's microphone(s) generate audio data that captures the voice utterance. In this example, the audio data capturing the voice utterance can be processed using an automatic speech recognition (ASR) model to generate ASR data for the voice utterance (e.g., the recognized text of "It's hot"). Furthermore, the ASR data for the voice utterance can be processed using a natural language understanding (NLU) model to generate NLU data for the voice utterance (e.g., the recognized intent(s) such as "It's hot," "Turn the temperature down," etc.). In particular, various audio-based technologies (e.g., speaker identification (SID) and / or other audio-based technologies) and / or non-audio-based technologies (e.g., facial recognition (FID), user account identification, device identifier identification, and / or other non-audio-based technologies) described herein can be used to determine that the user providing the voice utterance corresponds to the first user.

[0006] In some embodiments, the NLU data of a voice utterance can be processed at implementation to determine a set of fulfillment actions that facilitate lowering the temperature inside the vehicle. The set of fulfillment actions may include, for example, lowering the vehicle's windows (if the state indicates that the vehicle's windows are up) to lower the temperature inside the vehicle, turning on the vehicle's blower to circulate cool air throughout the vehicle to lower the temperature inside the vehicle, and / or other fulfillment actions that can be performed to lower the temperature inside the vehicle. In this example, the set of fulfillment actions may be processed using a fulfillment action model and / or fulfillment rules (e.g., personalized mappings) specific to the first user to select a first fulfillment action (or more) to be performed to satisfy the voice utterance. For example, the first fulfillment action (or more) may include lowering the vehicle's windows (or more), but may not include turning on the vehicle's blower to circulate cool air throughout the vehicle.

[0007] In contrast, suppose a second user is driving the same vehicle in a subsequent time instance and provides the same voice utterance, “It’s hot.” ​​The voice utterance can be processed in the same or similar way to determine a set of fulfillment actions. However, in this example, the set of fulfillment actions can be processed using a fulfillment action model and / or fulfillment rule(s) that are specific to the second user (e.g., not the first user) to select the second fulfillment action(s) to be performed to satisfy the voice utterance. For example, the second fulfillment action(s) could include lowering the vehicle's windows(s) and turning on the vehicle's blower to circulate cool air throughout the vehicle.

[0008] In various embodiments, the system may generate corresponding training instances that can be used later when training the fulfillment action model and / or fulfillment rules described above. These corresponding training instances may be specific to different users and may be generated based on corresponding user interactions with various smart devices performed by different users in different contextual scenarios. For example, suppose a first user, while riding in a vehicle, causes the vehicle's windows to be lowered at a given time instance among several time instances. Furthermore, suppose the vehicle's ambient temperature sensor indicates that the temperature inside the vehicle is 80 degrees Fahrenheit (e.g., a contextual signal characterizing the state of the vehicle's environment), which represents a “hot” contextual scenario at a given time instance. In this example, a given training instance can be generated based on the fulfillment action(s) manually performed by the user in this “hot” contextual scenario. A given training instance may include training instance outputs and training instance inputs. In this example, the training instance input may include a set of fulfillment actions that can be performed to lower the temperature inside the vehicle, and the contextual scenario in which the fulfillment actions were performed (e.g., a “hot” contextual scenario) (or its representation). Furthermore, the training instance output may include the fulfillment action of a user interaction performed by the user (e.g., lowering the vehicle's window(s)) (or its representation). This process can be repeated for multiple users in multiple different contextual scenarios to generate corresponding training instances. In some of these embodiments, the system may assign one or more voice commands (e.g., corresponding to semantic terms or phrases of the contextual scenario, such as “hot,” “burning,” or “on fire”) to each of the different contextual scenarios.

[0009] In some versions of these embodiments, the system may train a fulfillment action model(s) based on corresponding training instances. For example, the system may use a fulfillment action model for a first user to process training instance inputs corresponding to a set of fulfillment actions that can be performed to lower the temperature inside a vehicle, and contextual scenarios in which the fulfillment actions were performed to produce a predicted output. The predicted output may correspond to a subset of predicted first fulfillment actions(s) that the first user is expected to perform given the available fulfillment actions in a “hot” contextual scenario. Furthermore, the predicted output can be compared to fulfillment actions of user interactions performed by the user to produce one or more losses (e.g., across a distribution of actions(s)). In addition, the fulfillment action model for the first user may be updated based on one or more losses (e.g., via backpropagation). In these embodiments, the system may train only a fulfillment action model(s) in response to determining that one or more training conditions described herein are met.

[0010] In various embodiments, the system may generate personalized mappings (e.g., fulfillment rules) to be used in addition to, or instead of, the fulfillment action models (or models) described above. For example, the system may generate personalized mappings based on one or more past instances of a user that trigger the execution of one or more fulfillment actions corresponding to a user interaction in a contextual scenario, and independently of providing a voice utterance. Furthermore, the system may use various techniques to determine that a contextual scenario correlates with one or more voice commands. For example, the system may map different contextual scenarios, represented by different contextual signals that characterize the user's state and / or the state of the environment, to different parts of a lower-dimensional space (e.g., embedding space, semantic space, etc.) when one or more fulfillment actions are performed. Furthermore, one or more commands may be assigned to different contextual scenarios in a lower-dimensional space. Therefore, when a given voice command is subsequently included in a given voice utterance received from a user, the system can determine a given contextual scenario associated with the given voice command in a lower-dimensional space, and have one or more given fulfillment actions previously performed in the given contextual scenario performed to satisfy the given voice utterance. Furthermore, for example, the system can map different contextual scenarios, represented by different contextual signals characterizing the user's state and / or the state of the environment, to different parts of a lower-dimensional space (e.g., embedding space, semantic space, etc.) when one or more fulfillment actions are performed. Additionally, one or more voice commands can be assigned to different intentions.Therefore, when a given voice command is included later in a given voice utterance received from a user, the system can determine a given context scenario associated with the given voice command, determined based on a given intent, and have one or more given fulfillment actions previously performed in the given context scenario be carried out to satisfy the given voice utterance.

[0011] In some versions of these embodiments, the system may generate personalized mappings only in response to determining that one or more fulfillment actions have been performed by the user a threshold number of times (e.g., once, three times, five times, and / or one or any other positive integer greater than one) in a contextual scenario. For example, suppose there have been three past instances where the user lowered the vehicle's window(s) without turning on the vehicle's blower, when the ambient temperature inside the vehicle is between 70 and 80 degrees Fahrenheit, and the three past instances satisfy a threshold number of instances. In this example, the system may map the contextual scenario (e.g., the ambient temperature inside the vehicle being between 70 and 80 degrees Fahrenheit) to a first part of a lower-dimensional space and / or a first part associated with the "hot" intention, and assign voice commands such as "hot" or "burning" to the first part of the lower-dimensional space and / or the first part associated with the "hot" intention. Therefore, even if the user did not provide any voice utterances during the three previous instances, if the user later provides the voice utterance "It's hot" when the ambient temperature inside the vehicle is between 70 and 80 degrees Fahrenheit, the system can use personalized mapping to automatically lower the window(s). The system may also consider the ambient temperature outside the vehicle in these examples.

[0012] In particular, in these embodiments, the same voice command may be mapped to different fulfillment actions. Continuing the above example, we further assume there have been three past instances where the user lowered the vehicle's window(s) and turned on the vehicle's blower when the ambient temperature inside the vehicle exceeded 80 degrees Fahrenheit, and we further assume that these three past instances meet the threshold quantity of instances. In this example, the system can map the contextual scenario (e.g., the ambient temperature inside the vehicle exceeding 80 degrees Fahrenheit) to a completely different second part of a lower-dimensional space and / or a different second part associated with the “hot” intention, and assign voice commands such as “hot” or “burning” to a completely different second part of a lower-dimensional space and / or a different second part associated with the “hot” intention. However, based on the different fulfillment actions performed by the user in these three instances (e.g., lowering the vehicle's windows and turning on the vehicle's blower), these past instances may be mapped to a completely different second part of a lower-dimensional space, but may be associated with the same "hot" intent for different fulfillment actions performed in slightly different but similar contextual scenarios (e.g., when the ambient temperature inside the vehicle is above 80 degrees Fahrenheit, rather than between 70 and 80 degrees Fahrenheit). Therefore, even if the user did not provide any voice utterances during the three past instances, if the user later provides a voice utterance of "hot" when the ambient temperature inside the vehicle exceeds 80 degrees Fahrenheit, the system can use the personalized mapping to automatically lower the windows and automatically turn on the blower.

[0013] The above example illustrates speech utterances received in a vehicle environment; however, this is for illustrative purposes only and not intended to be limiting. For example, the technology described herein can be used in any environment where a given computing device is used by multiple users to perform speech utterance fulfillment. For example, the technology described herein can be used in home environments, workplace environments, school environments, commercial environments (e.g., hotels and / or other businesses), and / or any other environments. Furthermore, it should be noted that the fulfillment actions performed to satisfy speech utterances may vary from environment to environment. For example, fulfillment actions that can be performed in a home environment may include controlling various Internet of Things (IoT) devices and / or changing the state of other devices, while fulfillment actions that can be performed in the vehicle environment described above may include controlling various components and / or systems of the vehicle.

[0014] Furthermore, while the above example illustrates how to determine whether the user providing the voice utterance is the first or second user, it should be understood that this is for illustrative purposes only and not intended to be limiting. For example, the number of users considered when determining the user providing the voice utterance may be based on the number of users associated with the computing device receiving the voice utterance (e.g., the vehicle-mounted computing device in the above example). Furthermore, if it is determined that the user providing the voice utterance does not correspond to any of the users associated with the computing device, the user may be considered a default user or a guest user, and a default fulfillment model and / or default fulfillment rule (e.g., default mapping) may be used to determine how the voice utterance is fulfilled.

[0015] Furthermore, while the above examples illustrate different time instances and involve a first and second user providing the same voice utterance while each is driving a vehicle, please understand that this is for illustrative purposes only and not intended as an limitation. For example, various environments can be divided into multiple zones, and multiple users may be located in the same place but in different zones within the same environment. For instance, a vehicle environment may be divided into a "driver" zone, a "passenger" zone, etc., while a home environment may be divided into a "living room" zone, a "kitchen" zone, etc. In these instances, fulfillment actions may be limited to specific zones occupied by the user providing the voice utterance. For example, suppose the first user is driving a vehicle, while the second user is a passenger in the vehicle. In this instance, if the first user provides the voice utterance "it's hot," only the windows in the "driver" zone may be lowered, while if the second user provides the voice utterance "it's hot," only the blower associated with the "passenger" zone may be turned on. In other instances, windows may not be able to be lowered, and blowers may be considered a more conservative fulfillment action (for example, the second user may not want to lower the windows, while the first user does not care about the blower), so only the blowers in the "driver" and "passenger" zones may be turned on. Furthermore, or alternatively, joint fulfillment action models and / or joint fulfillment rules may be used in this type of scenario.

[0016] By using the techniques described herein, various technical advantages can be achieved. As one non-limiting example, the techniques described herein enable a system to train fulfillment action models (plural) based on respective user interactions (plural) executed in respective context scenarios (plural), and / or generate fulfillment rules (plural) for voice commands (plural) for fulfillment actions (plural) that are specific to each user (e.g., personalized mappings (plural)). These fulfillment action models (plural) and / or fulfillment rules (plural) can be later utilized to personalize the fulfillment of voice utterances, such that different fulfillment actions (plural) can be performed for different users based on receiving the same voice utterance from different users. As a result, the amount of interaction and / or user input in cases where incorrect fulfillment is performed can be reduced, thereby saving computing resources on a computing device, canceling incorrect fulfillment, and / or performing correct fulfillment based on the reduced amount of subsequent interaction. Additionally, the amount of natural resources wasted in cases where incorrect fulfillment is performed can be reduced.

[0017] The above description is provided as an overview of only some of the embodiments disclosed herein. Those embodiments and other embodiments are described in more detail herein.

Brief Description of the Drawings

[0018] [Figure 1] Shows a block diagram of an exemplary hardware and software environment that illustrates various aspects of the present disclosure and in which the embodiments disclosed herein can be implemented. [Figure 2] Shows an exemplary process flow of the exemplary hardware and software environment of FIG. 1 according to various embodiments. [Figure 3]A flowchart is provided illustrating an exemplary method for dynamically adapting the fulfillment of a given speech utterance based on the user who provided the speech utterance, according to various embodiments. [Figure 4] A flowchart is provided illustrating exemplary methods for generating training instances to train a user-specific fulfillment action model according to various embodiments, and for training the fulfillment action model for subsequent use when dynamically adapting the fulfillment of a given speech utterance provided by the user. [Figure 5A] This document presents various non-limiting examples of dynamically adapting the fulfillment of a given speech utterance in a vehicle environment based on the user who provided the given speech utterance, using a variety of embodiments. [Figure 5B] This document presents various non-limiting examples of dynamically adapting the fulfillment of a given speech utterance in a vehicle environment based on the user who provided the given speech utterance, using a variety of embodiments. [Figure 6A] This document presents various non-limiting examples of dynamically adapting the fulfillment of a given speech utterance in a primary living environment, based on the user who provided the given speech utterance, through various embodiments. [Figure 6B] This document presents various non-limiting examples of dynamically adapting the fulfillment of a given speech utterance in a primary living environment, based on the user who provided the given speech utterance, through various embodiments. [Figure 7] This document illustrates exemplary architectures of computing devices in various embodiments. [Modes for carrying out the invention]

[0019] Referring now to Figure 1, an environment in which one or more selected embodiments of this disclosure may be implemented is shown. An exemplary environment is a plurality of computing devices 110 1~NThis includes a fulfillment system 120, a vehicle 100A, one or more original equipment manufacturer (OEM) applications 181, one or more first-party applications 182, and one or more third-party applications 183. These components 110 1~N Each of 120, 181, 182, and 183 may communicate via one or more networks, for example, generally shown in 195. One or more networks may include wired or wireless networks such as local area networks (LANs) including Wi-Fi, Bluetooth, near-field communication, and / or other LANs, wide area networks (WANs) including the Internet, and / or any other networks to facilitate communication between the components shown in Figure 1.

[0020] In various embodiments, the user accesses the computing device 110 1~N One or more of these can be operated to interact with other components shown in Figure 1. Computing device 110 1~N Examples include desktop computing devices, laptop computing devices, tablet computing devices, mobile phone computing devices, optional standalone speaker devices with displays, and vehicle-mounted computing devices for vehicle 100A (e.g., 110A). N The vehicle may include a vehicle-mounted communication system, a vehicle-mounted entertainment system, and / or a vehicle-mounted navigation system as illustrated in the diagram, or a wearable device including a computing device such as a head-mounted display ("HMD") or a "smart" watch that provides an immersive computing experience of augmented reality ("AR") or virtual reality ("VR"). Additional and / or alternative computing devices may be provided.

[0021] Computing device 110 1~NAnd each of the fulfillment system 120 may include one or more memories for storing data and software applications (e.g., one or more of the OEM applications 181, one or more of the first party applications 182, and / or one or more of the third party applications 183), one or more processors for accessing the data and executing the software applications, and other components for facilitating communication via one or more of the networks 195. The computing device 110 1~N And / or operations performed by one or more of the fulfillment system 120 may be distributed across multiple computer systems. For example, the fulfillment system 120 may be implemented as a computer program that is executed only on one or more computers at one or more locations communicatively coupled to each other via, for example, one or more of the networks 195, or as a computer program distributed across one or more computers. Further or alternatively, the computing device 110 1~N And / or operations performed by one or more of the fulfillment system 120 may be executed locally on a computing system. For example, the fulfillment system 120 (or an instance thereof) may be implemented locally on each of the computing devices 110 1~N Each.

[0022] Component 110 1~N One or more of 120, 181, 182, and 183 may include various different components that can be used, for example, to dynamically adapt the fulfillment of a given voice utterance based on the user who provided the given voice utterance as described herein. For example, the computing device 1101 may detect and process user input (e.g., voice utterances, typed input, and / or touch input) directed to the computing device 1101 using a user interface engine 111 1をIt may include. As another example, computing device 1101 may include one or more sensors 1121 to generate corresponding sensor data. One or more sensors may include, for example, a Global Positioning System ("GPS") sensor for generating GPS data, a visual component for generating visual data within the field of view of a visual component, a microphone for generating audio data based on speech utterances captured in the environment of computing device 1101, and / or other sensors for generating corresponding sensor data.

[0023] As yet another example, computing device 1101 may operate an input processing engine 1131 (which may be standalone or part of another application, such as an automated assistant application) to process various user inputs received by computing device 1101. For example, the input processing engine 1131 may capture speech utterances and process the audio data generated by the microphone(s) of client device 1101 using an automated speech recognition (ASR) model(s) (e.g., a recurrent neural network (RNN) model, a transformer model, and / or other ML models capable of performing ASR) to produce an ASR output. Furthermore, the input processing engine 1131 may process the ASR output (or typed input) using a natural language understanding (NLU) model(s) (e.g., long short-term memory (LSTM), gated recurrent units (GRU), and / or any other type of RNN, or other ML models capable of performing NLU) and grammar-based NLU rules(s) to produce an NLU output. Furthermore, the input processing engine 1131 can process at least the NLU output using fulfillment action models and / or fulfillment rules (e.g., personalized mappings) to obtain one or more candidate responses in response to user input, such as actions to be performed by the automated assistant based on user input, and content items to be provided to present to the user based on user input. These fulfillment action models and / or fulfillment rules can be stored in the fulfillment action model and / or rule database 127A, trained in the manner described herein, and later made available.In embodiments where text content is audibly rendered in response to spoken or typed input, the user interface engine 1111 can process the text content using a text-to-speech model to generate synthesized speech audio data, including computer-generated synthesized speech that captures the content. The synthesized speech audio data can be audibly rendered for presentation to the user via the speaker(s) of the computing device 1101. In embodiments where visual content is visually rendered in response to spoken or typed input, the user interface engine 1111 can visually render the visual content for presentation to the user via the display of 1101.

[0024] In various embodiments, the ASR output may include, for example, one or more speech hypotheses (e.g., terminology hypotheses and / or transcription hypotheses) predicted to correspond to the user's speech activity and / or speech utterances captured in the audio data, one or more corresponding predicted values ​​for each of the one or more speech hypotheses (e.g., probability, log-likelihood, and / or other values), a plurality of phonemes predicted to correspond to the user's speech activity and / or speech utterances captured in the audio data, and / or other ASR outputs. In some versions of these embodiments, the input processing engine 1131 may be allowed to select one or more of the speech hypotheses as recognized text corresponding to speech utterances (e.g., based on the corresponding predicted values).

[0025] In various embodiments, the NLU output may include annotated recognized text, for example, one or more annotations of the recognized text for one or more terms (e.g., all) of the recognized text. For example, the input processing engine 1131 may employ a speech tagger (not shown) configured to annotate terms with their grammatical roles. Alternatively, the input processing engine 1131 may employ an entity tagging program (not shown) configured to annotate entity references in one or more segments of the recognized text. Entity references may include references to people (e.g., literary figures, celebrities, public figures, etc.), organizations, places (real and virtual), etc. In some embodiments, data about entities may be stored in one or more databases, such as a knowledge graph (not shown). In some embodiments, the knowledge graph may include nodes representing known entities (and, in some cases, entity attributes), and edges connecting the nodes and representing relationships between entities. Entity tagging programs can annotate references to entities at a high level of granularity (for example, to enable the identification of all references to an entity class such as a person) and / or at a low level of granularity (for example, to enable the identification of all references to a specific entity such as a particular person). Entity tagging programs may rely on the content of natural language input to resolve a particular entity, and / or may optionally communicate with knowledge graphs and / or other entity databases to resolve a particular entity.

[0026] Furthermore, or alternatively, the input processing engine 1131 may employ a coreference resolver (not shown) configured to group, or "cluster," references to the same entity based on one or more contextual cues. As one non-limiting example, the coreference resolver may be used to resolve the term "that" in the natural language input "What is that light?" to a specific light or indicator associated with the operation of vehicle 100A, based on corresponding sensor data instances generated by vehicle sensors that produced a specific light or indicator associated with the operation of vehicle 100A being generated. In some embodiments, one or more components utilized by the input processing engine 1131 may depend on annotations from one or more other components utilized by the input processing engine 1131. For example, in some embodiments, an entity tagging program may depend on annotations from the coreference resolver when annotating all references to a particular entity. Also, for example, in some embodiments, the coreference resolver may depend on annotations from an entity tagging program when clustering references to the same entity.

[0027] As yet another example, computing device 1101 may interact with the fulfillment system 120 by operating a fulfillment system client 1141 (which may be standalone or part of another application, such as an automated assistant application). Furthermore, additional computing device 110 N This may take the form of a vehicle-mounted computing device in vehicle 100A. Although not shown, an additional computing device 110 N It may include the same or similar components as computing device 1101. For example, additional computing device 110 NThis may include instances of a user interface engine for detecting and processing user input, one or more sensors for generating corresponding vehicle sensor data instances for vehicle sensor data, an input processing engine, and / or a fulfillment system client for interacting with the fulfillment system 120. In this example, one or more sensors may include a tire pressure sensor for generating tire pressure data for the vehicle 100A, an airflow sensor for generating airflow data for the vehicle 100A's air conditioning system, a vehicle speed sensor for generating vehicle speed data for the vehicle 100A, an energy sensor for generating energy source data for the vehicle 100A's energy source, a transmission sensor for generating transmission data for the vehicle 100A's transmission, and / or the vehicle 100A and / or the vehicle-mounted computing device 110 of the vehicle 100A. N It may include vehicle sensors such as any other sensors integrated into it. Furthermore, Figure 1 shows the computing device 1101 and the vehicle-mounted computing device 110 N Only examples are shown, but please understand that these are for illustrative purposes only, and additional or alternative computing devices may be provided (as will be explained with respect to Figures 6A and 6B, for example).

[0028] In various embodiments, the fulfillment system 120 may include an interface engine 121, an input processing engine 122, a user interaction engine 123, a context engine 124, a user identification engine 125, a training instance engine 126, a training engine 127, and a fulfillment engine 128, as shown in Figure 1. In some embodiments, one or more of the engines 121-128 of the fulfillment system 120 may be omitted. In some embodiments, one or more of the engines 121-128 of the fulfillment system 120 may be combined. In some embodiments, one or more of the engines 121-128 of the fulfillment system 120 may be computed to a computing device 110 1~N It may be implemented in a component that is partially or exclusively remotely executed from one or more of the fulfillment systems 120. In some embodiments, one or more of the engines 121-128 of the fulfillment system 120, or any operating part thereof, may be the computing device 110 1~N It can be implemented within a component that is executed locally, partially or exclusively, by one or more of these.

[0029] Referring to Figure 2, an exemplary process flow of the exemplary hardware and software environment of Figure 1 is shown. For the example, assume that audio data 201A capturing the user's voice utterance is received by the fulfillment system 120. Input processing engine 113 1~N / 122 can process audio data to generate processed input data 202. For example, input processing engine 113 1~N / 122 can use the ASR model to process audio data 201A and generate ASR data of the speech utterances captured in audio data 201A. Furthermore, input processing engine 113 1~N / 122 can use an NLU model to process the ASR data of the speech utterance captured in audio data 201A and generate NLU data of the speech utterance captured in audio data 201A. Furthermore, input processing engine 113 1~N / 122 can provide the fulfillment engine with audio data 201A, ASR data of the speech utterances captured in audio data 201A, and / or NLU data of the speech utterances captured in audio data 201A as processed input data 202.

[0030] Furthermore, the user identification engine 125 can process audio data 201A and / or non-audio data 201B (e.g., visual data capturing the user who provided the speech utterance, user profile data of the user profile of the user who provided the speech utterance, device identifier data of the computing device of the user who received the speech utterance, etc.) to determine the user identity 225 of the user who provided the speech utterance. As will be explained in more detail with respect to Figure 3, the user identification engine can use audio-based identification techniques to determine the user identity 225 (e.g., text-dependent (TD) speaker identification (SID), text-independent (TI) SID), and / or use non-audio-based identification techniques to determine the user identity 225 (e.g., facial recognition (FID), user profile identification, device identifier identification, etc.). In particular, when determining the user identity 225, the user identification engine 125 can utilize various user embeddings (e.g., stored in the user embedding database 125A). These various embeddings may include TD speaker embeddings, TI speaker embeddings, facial embeddings, and / or other embeddings. The generation and use of these embeddings in determining the user identity 225 will be described in more detail herein (for example, with respect to Figure 3). Furthermore, the user identification engine 125 can provide user identification to the fulfillment engine 128.

[0031] Furthermore, the fulfillment engine 128 can process at least the processed input data 202 and the user identity 225 to determine one or more fulfillment actions to be performed in response to receiving a voice utterance. In some embodiments, the fulfillment engine 128 can process at least the processed input data 202 and the user identity 225 to determine a set of fulfillment actions. For example, the fulfillment engine 128 can generate one or more structured requests and send one or more of the structured requests to various applications to obtain the fulfillment actions included in the set. For example, the fulfillment engine 128 can generate one or more structured requests and send one or more of the structured requests to one or more OEM applications 181 (e.g., in embodiments where the user is located in a vehicle 100A), one or more first-party applications 182, and / or one or more third-party applications 183. As used herein, the term “first-party application” may refer to a software application developed and / or maintained by the same entity that develops and / or maintains the fulfillment system 120 described herein. Furthermore, as used herein, the term “third-party application” may refer to a software application or system developed and / or maintained by an entity different from the entity that develops and / or maintains the fulfillment system 120 described herein. In response to one or more structured requests, one or more OEM applications 181 may transmit OEM data 281 to the fulfillment engine 128, one or more first-party applications 182 may transmit first-party data 282 to the fulfillment engine 128, and / or one or more third-party applications 183 may transmit third-party data 283 to the fulfillment engine 128.Based on OEM data 281, first-party data 282, and / or third-party data 283, the fulfillment engine 128 can determine a set of fulfillment actions.

[0032] In these embodiments, the fulfillment engine 128 can further process a set of fulfillment actions to determine one or more fulfillment actions to be performed in response to the receipt of a speech utterance. The fulfillment engine 128 can optionally further process context signals obtained via the context engine 124 (e.g., user signals(s) characterizing the user's state and / or environment signals(s) characterizing the user's environment state). For example, the fulfillment engine 128 can use a fulfillment action model and / or fulfillment rule(s) (e.g., stored in a fulfillment action model(s) and / or rule(s) database 127A) that are specific to the user who provided the speech utterance, as indicated by the user identity 225, to process one or more fulfillment actions to be performed in response to the receipt of a speech utterance and optionally context signals from the set of fulfillment actions. Furthermore, the fulfillment engine 128 can fulfill the speech utterance by having one or more fulfillment actions performed in response to the receipt of the speech utterance. In other words, in these embodiments, the fulfillment system 120 can initially determine a set of fulfillment actions using conventional techniques. However, the fulfillment system 120 can later process the set of fulfillment actions and select one or more fulfillment actions using a fulfillment action model and / or fulfillment rule(s) that are specific to the user who provided the voice utterance, as indicated by the user identity 225. This enables the fulfillment system 120 to personalize the fulfillment of the voice utterance for the user who provided the voice utterance, as indicated by the user identity 225.

[0033] In particular, a fulfillment action model specific to the user who provided the voice utterance, as indicated by user identity 225, may be pre-trained, and / or a fulfillment rule(s) specific to the user who provided the voice utterance, as indicated by user identity 225, may be pre-generated. For example, before receiving the voice utterance captured in audio data 201A, the user interaction engine 123 can identify a user interaction with one or more smart devices in a given time instance among several time instances. The user interaction may correspond to one or more fulfillment actions 223. In various embodiments, the one or more fulfillment actions 223 of the user interaction may be limited to those that change the state of one or more smart devices in some way. The user interaction engine 123 can provide one or more fulfillment actions 223 to the training instance engine 126.

[0034] Furthermore, the context engine 124 can acquire one or more context signals that characterize the state of the user's environment, indicated by user signals 224A and / or environment signals 224B, at a given time instance of the user interaction. User signals 224A and / or environment signals 224B can be used to define a context scenario, which can later be used to bias the interpretation of subsequent speech utterances received in the context scenario. The context engine 124 can provide user signals 224A and environment signals 224B to the training instance engine 126. These context signals and context scenarios are described in more detail herein (for example, with respect to Figure 4). As described above, during inference, the context engine 124 can provide context signals to the fulfillment engine 128.

[0035] The training instance engine 126 can generate a training instance 226 by processing one or more fulfillment actions 223 corresponding to a user interaction, as well as user signals 224A and / or environmental signals 224B. Furthermore, the training instance engine 126 can store the training instance 226 in the training instance database 126A. Furthermore, the training engine 127 can train a fulfillment action model specific to the user who provided the speech utterance and / or generate one or more fulfillment action models specific to the user who provided the speech utterance, based on the training instance 226 and optionally one or more additional training instances generated in the same or similar manner. Training a fulfillment action model specific to the user who provided the speech utterance and / or generating one or more fulfillment rules specific to the user who provided the speech utterance will be described in more detail herein (for example, with respect to Figure 4).

[0036] Referring here to Figure 3, a flowchart is shown illustrating an exemplary method 300 for dynamically adapting the fulfillment of a given speech utterance based on the user who provided the given speech utterance. For convenience, the operation of method 300 will be described with reference to the system on which the operation is performed. The system of method 300 comprises at least one processor, at least one memory, and / or computing devices (for example, computing devices 110 in Figures 1, 5A, 5B, 6A, and 6B). 1~N This includes the fulfillment system 120 in Figure 1, the computing device 710 in Figure 7, the remote server(s), and / or other computing device(s). The operations of Method 300 are shown in a specific order, but this is not intended to be limiting. One or more operations may be rearranged, omitted, and / or added.

[0037] In block 352, the system receives a given speech utterance from a user and via a computing device. For example, a user can provide a given speech utterance to a vehicle-mounted computing device while the user is located in the user's vehicle, and the given speech utterance can be captured as audio data generated via the microphone(s) of the vehicle-mounted computing device (as described, for example, with respect to Figures 5A and 5B). Alternatively, for example, a user can provide a given speech utterance to a mobile computing device or a standalone computing device while the user is located in the user's primary residence, and the given speech utterance can be captured as audio data generated via the microphone(s) of the mobile computing device or a standalone computing device (as described, for example, with respect to Figures 6A and 6B).

[0038] In block 354, the system determines a set of fulfillment actions based on processing a given speech utterance. Each fulfillment action included in the set of fulfillment actions facilitates the given speech utterance when performed. For example, as shown in block 354A, the system may use an ASR model to process audio data capturing a given speech utterance to generate ASR data for the given speech utterance. Furthermore, as shown in block 354B, the system may use an NLU model to process the ASR data for a given speech utterance to generate NLU data for the given speech utterance. For example, suppose a given speech utterance provided by a user corresponds to "cold". In this instance, the system may process audio data capturing the speech utterance to generate the recognized text "cold" as the ASR data for the given speech utterance. Furthermore, the system may process the recognized text "cold" to identify one or more intentions for the given speech utterance, such as "cold" or "increase fever". Furthermore, the system may determine a set of fulfillment actions based on at least one or more intentions. In this instance, assuming a given voice utterance is received in a vehicle environment, the set of fulfillment actions could include, for example, raising the vehicle's windows, turning on the vehicle's heated seats, turning on the vehicle's blower, and / or any other actions that could be taken to raise the temperature inside the vehicle. In contrast, assuming a given voice utterance is received in a home environment, the set of fulfillment actions could include, for example, raising the thermostat temperature of the primary residence corresponding to the home environment to a first temperature, raising the thermostat temperature of the primary residence to a second temperature, and / or any other actions that could be taken to raise the temperature inside the primary residence.

[0039] In block 356, the system determines whether the user who provided a given speech utterance corresponds to a first user or a second user. The system may utilize various techniques to determine whether the user who provided a given speech utterance corresponds to a first user or a second user. In some embodiments, the system may perform speaker identification (SID) to determine whether the user who provided a given speech utterance corresponds to a first user or a second user. For example, the system may use an SID model(s) to process audio data capturing a given speech utterance and generate a speaker embedding for the user who provided the given speech utterance. The generated speaker embedding can be compared in the embedding space with one or more previously generated speaker embeddings for one or more users of the computing device (e.g., embeddings associated with the user profiles of one or more users). For example, if a distance criterion determined based on a comparison within the embedding space between a generated speaker embedding and a given one or more previously generated speaker embeddings for one or more users satisfies a threshold, then the user can be determined to correspond to a user associated with one or more given previously generated speaker embeddings. Otherwise, the user may be identified as a guest user.

[0040] In these embodiments, one or more previously generated speaker embeddings for one or more users can be generated based on the output(s) generated from processing one or more instances of audio data containing speech utterances, each from one or more users. For example, one or more previously generated speaker embeddings may be based on the average (or other combination) of several different embeddings, each generated from processing corresponding instances of audio data containing corresponding speech utterances, each from one or more users. Furthermore, in these embodiments, the SID model may be a text-independent (TI) SID model, a text-dependent (TD) SID model, and / or any other suitable SID model.

[0041] For example, in TD SID, one or more previously generated speaker embeddings for one or more users are generated based on speech utterances containing only one or more specific words or phrases. Furthermore, during use, one or more users may be required to utter one or more specific words or phrases for one or more TD SID speaker embeddings generated using the TD SID model, which can be effectively compared to one or more previously generated TD speaker embeddings for one or more users in order to determine whether the user who provided a given speech utterance corresponds to a first user or a second user. For example, one or more words or phrases in TD speaker recognition may be constrained to one or more invocation phrases configured to invoke an automated assistant (e.g., hotwords and / or trigger words such as "Hi, Assistant," "Got it, Assistant," and / or "Assistant") or one or more other words or phrases. In contrast, in TI SID, speech utterances processed using the TI SID model are not constrained to one or more specific words or phrases. In other words, audio data based on any virtually spoken utterance can be processed using the TI SID model to generate TI SID speaker embeddings, which can be effectively compared to one or more previously generated TI speaker embeddings for one or more users to determine whether the user providing a given speech utterance corresponds to a first user or a second user. Furthermore, in various embodiments, one or more previously generated TI speaker embeddings for one or more users utilized in TI SID are generated based on speech utterances that are not limited to completely different words and / or phrases, words and / or phrases of invocation, and / or any other specific speech utterances.

[0042] In additional or alternative embodiments, the system may perform other forms of identification to determine whether the user providing a given speech utterance corresponds to a first user or a second user. For example, the system may perform facial recognition, fingerprint recognition, and / or other types of biometric authentication. For instance, the visual component of a client device may capture visual data including the face of the user providing the speech utterance. In this instance, the system may use a facial recognition (FID) model to process the visual data and generate a facial embedding of the user providing the speech utterance. The facial embedding of the user providing the speech utterance may correspond to one or more previously generated embeddings in the embedding space that correspond to facial embeddings of one or more users of the computing device in order to determine whether the user providing the given speech utterance corresponds to a first user or a second user. Alternatively, for example, a fingerprint sensor may receive touch input from the user providing a given speech utterance and compare the detected fingerprint to known fingerprints of one or more users of the computing device in order to determine whether the user providing the given speech utterance corresponds to a first user or a second user. Furthermore, for example, the display interface of a client device may receive touch input from a user corresponding to a specific password on the client device in order to determine whether the user who provided a given voice utterance corresponds to a first user or a second user. As another example, the system may use device identifiers and / or user accounts associated with a computing device and / or additional computing devices communicating with the computing device to determine whether the user who provided a given voice utterance corresponds to a first user or a second user, based on inferring the user from the device identifier and / or user account.

[0043] In block 358, the system determines how to dynamically adapt the fulfillment of a given speech utterance based on its determination from block 356 whether the user who provided the given speech utterance corresponds to a first user or a second user. For example, if, during an iteration of block 358, the system determines that the user who provided the given speech utterance corresponds to a first user, the system may proceed to block 360. In block 360, based on the determination that the user who provided the given speech utterance corresponds to a first user, the system selects one or more subsets of first fulfillment actions from a set of fulfillment actions from block 354. In some embodiments, the system may select one or more subsets of first fulfillment actions based on contextual signals characterizing the user's state and / or the state of the user's environment. In block 362, the system has one or more subsets of first fulfillment actions performed to satisfy the given speech utterance. The system can select one or more subsets of a first fulfillment action from a set of fulfillment actions using a fulfillment action model and / or fulfillment rule(s) (e.g., personalized mapping(s)) that are specific to the first user. For example, the system can use a fulfillment action model and / or fulfillment rule(s) that are specific to the first user to select one or more first fulfillment actions to include in a subset by processing a given speech utterance and / or a set of fulfillment actions from block 354, and optionally, NLU data for context signals characterizing the state of the user who provided the given speech utterance and / or the state of the environment in which the user is located when the given speech utterance is provided. Generating and utilizing a fulfillment action model(s) and / or fulfillment rule(s) (e.g., personalized mapping(s)) that are specific to a given user will be described in more detail with respect to Figure 4.

[0044] However, during an iteration of block 358, if the system determines that the user who provided a given speech utterance corresponds to a second user, the system may proceed to block 364. In block 364, based on the determination that the user who provided the given speech utterance corresponds to a second user, the system selects one or more subsets of second fulfillment actions from the set of fulfillment actions from block 354. In block 366, the system has one or more subsets of second fulfillment actions perform to satisfy the given speech utterance. Similarly, the system may select one or more subsets of second fulfillment actions from the set of fulfillment actions using a fulfillment action model and / or fulfillment rule(s) that are specific to the second user. For example, the system may use a fulfillment action model and / or fulfillment rule(s) specific to a second user to process a given speech utterance and / or a set of fulfillment actions from block 354, and optionally, NLU data for context signals characterizing the state of the user who provided the given speech utterance and / or the state of the environment in which the user is located when the given speech utterance is provided, to select one or more second fulfillment actions to include in a subset. As described above, generating and utilizing a fulfillment action model(s) and / or fulfillment rule(s) specific to a given user will be described in more detail with respect to Figure 4. In particular, a subset of one or more second fulfillment actions may include at least one unique fulfillment action from the set of fulfillment actions, compared to a subset of one or more first fulfillment actions. In other words, the fulfillment of the same speech utterance may differ depending on the user who provided the speech utterance.

[0045] Continuing with the above example where a given voice utterance provided by a user corresponds to "cold," we further assume that the given voice utterance was provided by a first user. In this example, assuming the given voice utterance was provided in a vehicle environment, then a fulfillment action model and / or fulfillment rule(s) specific to the first user may have learned that the first user typically turns on only the heated seats of the vehicle and not the blower when it is cold, so a fulfillment action associated with turning on the heated seats of the first user may be selected and performed. Furthermore, assuming the system has access to various environmental signals characterizing the state of the vehicle, and that the environmental signals indicate that the vehicle's windows are down when the voice utterance is received, then a fulfillment action model and / or fulfillment rule(s) specific to the first user may have learned that the first user typically raises the windows when it is cold, so a fulfillment action associated with raising the windows may be selected and performed. Furthermore, assuming in this example that a given speech utterance was provided in a home environment, a fulfillment action model and / or fulfillment rule(s) specific to the first user may then be selected and implemented, such as raising the thermostat temperature by N degrees (e.g., N is a positive integer), or raising the thermostat temperature to a first specific temperature (e.g., 74 degrees), because the first user may have learned that they typically use the thermostat in this way when it is cold.

[0046] In contrast, let us further assume that a given speech utterance was provided by a second user. In this example, assuming that the given speech utterance was provided in a vehicle environment, then a fulfillment action model and / or fulfillment rule(s) specific to the second user may then be selected and performed, as this second user may have learned that when it is cold, they typically only turn on both the heated seats and the blower in the vehicle, and therefore, the fulfillment action associated with turning on the heated seats and the blower for the second user may be selected and performed. Furthermore, assuming in this example that a given speech utterance was provided in a home environment, a fulfillment action model and / or fulfillment rule(s) specific to the second user may then be selected and implemented, as the second user may have learned that they typically use the thermostat in this way when it is cold. This fulfillment action may involve raising the thermostat temperature by M degrees (e.g., M is a positive integer different from N), or raising the thermostat temperature to a second specific temperature (e.g., 75 degrees).

[0047] From blocks 362 and 366, the system returns to block 352 and performs additional iterations of method 300 in Figure 3 based on additional given speech utterances received via the computing device. Thus, if different users provide the same speech utterance (e.g., "cold" in the example above), the system can dynamically adapt the fulfillment of that same speech utterance based on how different users typically cause different actions to be performed in the same or similar contextual scenarios. Consequently, the fulfillment of a given speech utterance is improved, thereby reducing the amount of user interaction and / or user input received, and the fulfillment of a given speech utterance may not be unique to the user who provided the given speech utterance, thus causing the desired fulfillment action to be performed in the instance where the incorrect fulfillment action is first performed.

[0048] In various embodiments, the system may provide a display, audibly and / or visually, to the corresponding user via a computing device, of the reasons why one or more fulfillment actions were selected and performed to satisfy a given speech utterance by the corresponding user. The display of the reasons why one or more fulfillment actions were selected and performed to satisfy a given speech utterance may be generated using a fulfillment action model and / or fulfillment rule. Continuing the above example, assuming a given speech utterance is provided by a first user in a vehicle environment, the display of the reasons why one or more fulfillment actions were selected and performed to satisfy the given speech utterance might correspond to "I turned on your heated seats, but I didn't turn on the blower because that's what you normally do when it's cold in the car." Synthetic speech audio data, including synthesized speech capturing the display, can be audibly rendered for presentation to the first user via the computing device's speaker, and / or text data corresponding to the display can be visually rendered for presentation to the first user via the computing device's display. In particular, while displaying the reasons why one or more fulfillment actions were selected and performed to satisfy a given speech utterance may be provided only to educate the corresponding user by presenting it a threshold number of times (e.g., at least once), it is possible to avoid overwhelming and confusing the user with the same information.

[0049] In various embodiments, the system may generate and provide prompts that require the user to provide feedback on one or more fulfillment actions selected and performed. Based on user input received from the user in response to the prompts, the system may update user-specific fulfillment action models and / or fulfillment rules. For example, the system may generate the above display (e.g., "You turned on your heated seats, but you didn't turn on the blower. That's what you normally do when it's cold in the car.") and further or instead generate the prompt, "Is that correct?". In this example, if positive user input is received, the system can then generate positive training instances to bias subsequent fulfillments of the same speech utterance toward the selection and performance of one or more first fulfillment actions. However, if negative user input is received, the system can then generate negative training instances to bias subsequent fulfillments of the same speech utterance toward avoiding the selection and performance of one or more first fulfillment actions.

[0050] In additional or alternative embodiments, the system may utilize one or more subsequent user interactions as feedback regarding one or more selected and performed fulfillment actions. For example, the system selects and performs a fulfillment action associated with the first user turning on a heated seat based on a given speech utterance "cold" provided by a first user, but if the first user turns off the heated seat following the performance of the fulfillment action, the system can then generate negative training instances to bias subsequent fulfillment of the same speech utterance by avoiding the selection and performance of one or more first fulfillment actions (e.g., fulfillment actions associated with the first user turning on a heated seat). In contrast, if the system selects and performs a fulfillment action associated with turning on the heated seat of the first user based on a given speech utterance "cold" provided by the first user, and the first user does not turn off the heated seat following the performance of the fulfillment action, then the system can generate positive training instances to bias subsequent fulfillments of the same speech utterance toward the selection and performance of one or more first fulfillment actions (e.g., fulfillment actions associated with turning on the heated seat of the first user), based on the inference that the fulfillment action was performed correctly in response to the given speech utterance.

[0051] Figure 3 illustrates a specific utterance and a specific fulfillment action as described herein, but it should be understood that this is for illustrative purposes only and not intended to be limiting. Furthermore, Figure 3 illustrates only the first and second users as described herein, but it should be understood that this is also for illustrative purposes only and not intended to be limiting. For example, the number of users is considered when determining that the user who provided a given speech utterance may be based on the number of users associated with the computing device. If the user who provided a given speech utterance is not determined to correspond to any known user, then the user may be considered a default user, and a fulfillment action model and / or fulfillment rule(s) that are not specific to any particular user may be used to dynamically adapt the fulfillment of a given speech utterance.

[0052] Referring here to Figure 4, a flowchart is shown illustrating an exemplary method 400 for generating training instances to train a user-specific fulfillment action model and for subsequent use in dynamically adapting the fulfillment of a given speech utterance provided by the user. For convenience, the operation of method 400 will be described with reference to the system on which the operation is performed. The system of method 400 comprises at least one processor, at least one memory, and / or computing devices (for example, computing devices 110 in Figures 1, 5A, 5B, 6A, and 6B). 1~N This includes the fulfillment system 120 in Figure 1, the computing device 710 in Figure 7, the remote server(s), and / or other computing device(s). The operations of Method 400 are shown in a specific order, but this is not intended to be limiting. One or more operations may be rearranged, omitted, and / or added.

[0053] In block 452, the system identifies the occurrence of a user interaction with one or more smart devices in a given time instance among multiple time instances, and the user interaction corresponds to one or more fulfillment actions. The system may consider any user interaction that is communicatively coupled to one or more smart devices and changes the state of one or more smart devices as an occurrence of user interaction. Furthermore, one or more fulfillment actions may correspond to any specific action performed by the user that changes the state of one or more smart devices. One or more smart devices may include, for example, the user's computing devices (e.g., mobile computing devices, vehicle-mounted computing devices, etc.), vehicle components of the user's vehicle (e.g., window defrosters, power windows and power doors, radios, blowers, heated seats, etc.), the user's Internet of Things (IoT) devices (e.g., smart TVs, smart thermostats, smart alarm systems, smart home appliances, etc.), and / or any other smart devices that the user interacts with and thereby changes the state of.

[0054] In block 454, the system obtains one or more context signals characterizing the user's state in a given time instance and / or the state of the user's environment in a given time instance. In other words, the system obtains one or more context signals in the same given time instance in which the occurrence of a user interaction is identified to determine the contextual scenario that led the user to perform one or more fulfillment actions of the user interaction. One or more context signals characterizing the user's state in a given time instance may be based on user profile data from the user's user profile(s), application data from applications(s) accessible to the system, sensor data instances of sensor data generated by one or more sensors in the user's environment (e.g., vehicle sensors(s), computing device sensors(s), home sensors(s), etc.), and / or any other context signals characterizing the user's state. Furthermore, the user's state, determined based on one or more contextual signals, may include, for example, the user's location or activity (e.g., whether the user is located at their primary residence, whether the user is driving a vehicle), the time of day at the user's location, the day of the week at the user's location, and / or any other state of the user determined based on one or more contextual signals. In addition, one or more contextual signals characterizing the state of the user's environment at a given time instance may be based on sensor data instances of sensor data generated by one or more sensors in the user's environment (e.g., vehicle sensors, computing device sensors, home sensors, etc.) and / or any other contextual signals characterizing the state of the user's environment.Furthermore, the state of the user's environment, determined based on one or more contextual signals, may include, for example, ambient noise detected in the user's environment, the ambient indoor and / or outdoor temperature in the user's environment, whether other users are located in the same place as the user's environment (e.g., within the primary residence such as the user or passengers in a vehicle), and / or any other state of the user's environment determined based on one or more contextual signals.

[0055] In block 456, the system generates a given user training instance based on user interactions in a given time instance and on one or more context signals in a given time instance. Furthermore, the system can store a given user training instance in a training instance(or more) database (e.g., training instance(or more) database 126A in Figure 1). A given training instance may include a training instance input and a training instance output. The training instance input may include, for example, (i) one or more context signals that characterize the state of the user in a given time instance and / or the state of the user's environment in a given time instance, and (ii) a set of fulfillment actions associated with one or more context signals. In particular, the set of fulfillment actions includes at least one or more fulfillment actions corresponding to a user interaction and at least one additional fulfillment action. In some embodiments, the training instance input may include one or more terms or phrases that describe one or more context signals and may be later provided by the user to describe a low ambient temperature context scenario (e.g., "cold," "freezing," "chilly," etc.). In other words, a training instance input may include a given contextual scenario indicated by one or more contextual signals, and a set of fulfillment actions that are expected to be relevant to the given contextual scenario. The set of fulfillment actions expected to be relevant to the given contextual scenario can be defined by the system's associated developers, users, and / or other parties. For example, if the ambient temperature of the user's environment is less than X degrees (e.g., X is a positive integer), as indicated by one or more contextual signals, then the set of fulfillment actions may include actions that, when implemented, facilitate raising the ambient temperature of the user's environment.Furthermore, the training instance output may include one or more fulfillment actions of a user interaction. In particular, one or more fulfillment actions of a user interaction may include each of a set of fulfillment actions or a subset thereof. In other words, the training instance output may, in effect, include the fulfillment actions performed by a user in a given contextual scenario.

[0056] In some embodiments, as shown in block 456A, the system prompts the user to verify whether one or more fulfillment actions were performed based on the user's state in a given time instance and / or the state of the user's environment in a given time instance, before generating a given user training instance. For example, suppose the user manually turns on a heated seat in a vehicle. The system can identify the occurrence of the user manually turning on the heated seat as a user interaction that includes a fulfillment action to raise the ambient temperature of the user's environment. Furthermore, the system can identify the ambient temperature of the user's environment (e.g., inside and / or outside the vehicle) as one or more contextual signals in a given time instance when the user manually turned on the heated seat. In these embodiments, before generating a given training instance based on this user interaction, the system can prompt the user to verify whether the user manually turned on the heated seat because they were cold. In additional or alternative embodiments, the system may simply infer this without prompting the user.

[0057] In block 458, the system determines whether one or more conditions are met for training a fulfillment action model specific to the user. These conditions may include, for example, whether there are training instances of a threshold quantity available for the fulfillment action model, time, day of the week, and / or other conditions. For example, in embodiments where the system is implemented locally on the user's computing device, the conditions may also include, or instead, whether the computing device is charging, whether the computing device has a charge state of at least a threshold, whether the computing device's temperature is below a threshold, and whether the computing device is being held by the user. If, during an iteration of block 458, the system determines that one or more conditions are not met, the system proceeds to block 460.

[0058] In block 460, the system determines whether there are subsequent occurrences of the user interaction (e.g., including the same one or more fulfillment actions) or whether there are additional user interactions in addition to the user interaction (e.g., including at least one unique fulfillment action compared to one or more fulfillment actions of the user interaction). If, during an iteration of block 460, the system determines that there are subsequent occurrences of the user interaction or additional user interactions, the system may return to block 452 and perform additional iterations of the operations in blocks 452-458 with respect to the subsequent occurrences of the user interaction or additional user interactions to generate an additional given user training instance and determine whether one or more conditions are met to train a fulfillment action model that is unique to the user. If, during an iteration of block 460 and / or a subsequent iteration, the system determines that there are no subsequent occurrences of the user interaction or additional user interactions, the system may return to block 458. Furthermore, during an iteration of block 458, if the system determines that one or more conditions are met, the system proceeds to block 462. In other words, the system continuously monitors for user interactions during the training phase and may generate training instances based on those user interactions, and / or continuously monitors for one or more conditions to be met until the system decides to train fulfillment actions based on the training instances.

[0059] In block 462, the system trains a user-specific fulfillment action model based on at least a given training instance. Furthermore, the system can store the trained fulfillment action model in one or more databases accessible to the system (e.g., a fulfillment action model database 127A, which may contain multiple fulfillment action models in Figure 1 and / or rules). In block 464, the system uses the user-specific fulfillment action model (as described, for example, with respect to Figure 3) to respond to speech utterances received from the user. The fulfillment action model can be any suitable ML model that can be trained in any suitable way, including, for example, classification models (e.g., support vector machine (SVM) models, K-nearest neighbor (KNN) models, naive Bayes models, etc.), regression models, and / or other types of machine learning (ML) models. For example, when training fulfillment actions based on a given training instance generated in block 456, the system can use the fulfillment action model to process one or more contextual signals and a set of fulfillment actions included in the training instance input to produce a predictive output. In this example, the prediction output may correspond to one or more fulfillment actions from a set of fulfillment actions that a user is expected to perform, based on a given contextual scenario defined by one or more contextual signals. Furthermore, one or more predicted fulfillment actions (or their representations) in the prediction output can be compared to one or more fulfillment actions (or their representations) of user interactions included in the training instance output to generate one or more losses. In addition, the fulfillment action model can be updated based on one or more losses. For example, one or more losses can be backpropagated across the entire fulfillment action model to update its weights.In particular, this process can be repeated for multiple additional training instances generated in the manner described herein to further train and personalize user-specific fulfillment action models. Furthermore, this process can be repeated for multiple different users to train and personalize corresponding fulfillment action models specific to those users.

[0060] In additional or alternative embodiments, the system may present a variety of contextual scenarios to the user and require the user to identify one or more fulfillment actions that they would perform given the various contextual scenarios. In these embodiments, the system may generate training instances for training a fulfillment action model and / or generating fulfillment rules(s) based on how the user responds to the various contextual scenarios. For example, the system may present a low ambient temperature for the vehicle (e.g., a contextual scenario in which the user would get cold) and list a variety of actions the user could take in response to the low ambient temperature (e.g., the training instance input to the given training instance described above). Furthermore, the system may have one or more user selections of actions available as the training instance output.

[0061] In various embodiments, the fulfillment action model and / or fulfillment rules(s) can be updated over time. For example, the system can continuously generate training instances based on the occurrence of user interactions identified in various time instances and / or one or more contextual signals acquired in those various time instances. The system may update the fulfillment action model and / or fulfillment rules(s) whenever it determines that one or more conditions will subsequently be met. Alternatively, for example, the system can update the fulfillment action model and / or fulfillment rules(s) based on changing contextual scenarios. For instance, during the fall season, a user might raise one or more windows in a vehicle when it's cold, but not turn on any of the heated seats. However, as the fall season changes to the winter season, the user might never lower the windows but turn on the heated seats. Thus, the fulfillment action model and / or fulfillment rules(s) are not only specific to the user in a given time instance, but are also updated over time to ensure better fulfillment of the user's speech utterances.

[0062] In various embodiments, the system can assign one or more voice commands to a training instance and / or fulfillment rule(s) so that the system can later determine that voice utterances received in the same or similar contextual scenarios map to different fulfillment actions for different users. For example, one or more voice commands may describe one or more contextual signals and include one or more terms or phrases that may later be provided by the user. For example, terms such as "cold," "freezing," and "chilly" may be assigned to a training instance and / or fulfillment rule(s) associated with a contextual signal indicating a low ambient temperature in the user's environment when one or more fulfillment actions in a user interaction are performed to raise a low ambient temperature in the environment. Similarly, terms such as "hot," "burning," and "scorching" may be assigned to a training instance and / or fulfillment rule(s) associated with a contextual signal indicating a high ambient temperature in the user's environment when one or more fulfillment actions in a user interaction are performed to lower a high ambient temperature in the environment. Furthermore, terms such as “dark,” “dim,” and “pitch black” can be assigned to trained instances and / or fulfillment rules associated with contextual signals indicating that an environment has low ambient light when one or more fulfillment actions of a user interaction are performed to increase the ambient light of the environment. Therefore, it should be understood that the techniques described herein can be used not only to train fulfillment action models and / or fulfillment rules as described herein, but also to assign one or more voice commands to specific contextual scenarios that a user may encounter.

[0063] In additional or alternative embodiments, the system can generate personalized mappings for the user (e.g., also referred to herein as fulfillment rules) that map one or more voice commands to one or more fulfillment actions. For example, the system can generate personalized mappings based on one or more past instances of the user that trigger the execution of one or more fulfillment actions corresponding to user interactions in a contextual scenario, and without regard to the provision of voice utterances. Furthermore, the system can use various techniques to determine that a contextual scenario correlates with one or more voice commands. For example, the system can map different contextual scenarios, represented by different contextual signals that characterize the user's state and / or the state of the environment when one or more fulfillment actions are performed, to different parts of a lower-dimensional space (e.g., embedding space, semantic space, etc.). Furthermore, one or more commands can be assigned to different contextual scenarios in a lower-dimensional space. Therefore, when a given voice command is subsequently included in a given voice utterance received from a user, the system can determine a given contextual scenario associated with the given voice command in a lower-dimensional space, and have one or more given fulfillment actions previously performed in the given contextual scenario performed to satisfy the given voice utterance. Furthermore, for example, the system can map different contextual scenarios, represented by different contextual signals characterizing the user's state and / or the state of the environment, to different parts of a lower-dimensional space (e.g., embedding space, semantic space, etc.) when one or more fulfillment actions are performed. Additionally, one or more voice commands can be assigned to different intentions.Therefore, when a given voice command is included later in a given voice utterance received from a user, the system can determine a given context scenario associated with the given voice command, determined based on a given intent, and have one or more given fulfillment actions previously performed in the given context scenario be carried out to satisfy the given voice utterance.

[0064] In some versions of these embodiments, the system may generate personalized mappings only in response to determining that one or more fulfillment actions have been performed by the user a threshold number of times (e.g., once, three times, five times, and / or one or any other positive integer greater than one) in a contextual scenario. For example, suppose there have been three past instances where the user turned on the heated seats of the vehicle but did not turn on the blower, when the ambient temperature inside the vehicle was between 45 and 60 degrees Fahrenheit, and the system may assume that these three past instances satisfy the threshold number of instances. In this example, the system may map the contextual scenario (e.g., the ambient temperature inside the vehicle being between 45 and 60 degrees Fahrenheit) to a first part of a lower-dimensional space and / or a first part associated with the "cold" intent, and assign voice commands such as "cold" or "freezing" to the first part of the lower-dimensional space and / or the first part associated with the "cold" intent. Therefore, even if the user did not provide any voice utterances during the three previous instances, if the user later provides the voice utterance "cold" when the ambient temperature inside the vehicle is between 45 and 60 degrees Fahrenheit, the system can use personalized mapping to automatically turn on the heated seats.

[0065] In particular, in these embodiments, the same voice command may be mapped to different fulfillment actions. Continuing the above example, let us further assume there have been three past instances where the user turned on the vehicle's heated seats and the vehicle's blower when the ambient temperature inside the vehicle was below 45 degrees Fahrenheit, and that the three past instances satisfy the instance threshold quantity. In this example, the system can map the contextual scenario (e.g., the ambient temperature inside the vehicle being below 45 degrees Fahrenheit) to a completely different second part of a lower-dimensional space, and / or a different second part associated with the "cold" intention, assigning voice commands such as "cold" or "freezing" to a completely different second part of a lower-dimensional space, and / or a different second part associated with the "cold" intention. However, based on the different fulfillment actions the user performs in these three instances (e.g., turning on both the vehicle's heated seats and the vehicle's blower), these past instances may be mapped to a completely different second part of a lower-dimensional space, but may still be associated with the same "cold" intention. Therefore, even if the user did not provide any voice utterances during the three previous instances, if the user later provides the voice utterance "cold" when the ambient temperature inside the vehicle is below 45 degrees Fahrenheit, the system can use personalized mapping to automatically turn on the heated seats and the blower.

[0066] Figure 4 illustrates a specific user interaction with a particular fulfillment action, but it should be understood that this is for illustrative purposes only and not intended as an limitation. Furthermore, Figure 4 illustrates training a fulfillment action model and / or generating a fulfillment rule(s) for a single user, but again, this is for illustrative purposes only and not intended as an limitation. For example, it should be understood that corresponding iterations of Method 400 in Figure 4 can be performed to train corresponding fulfillment action models and / or generate corresponding fulfillment rules(s) for multiple other users.

[0067] Referring here to Figures 5A and 5B, various non-limiting examples are shown of dynamically adapting the fulfillment of a given speech utterance in a vehicle environment based on the user who provided the given speech utterance. For example, vehicle 100A is the vehicle-mounted computing device 110 in Figure 1. N Assume that the vehicle is at least partially shared by multiple users (a first user 102 shown in Figure 5A and a second user 104 shown in Figure 5B). Furthermore, assume that the action fulfillment model has been previously trained for both the first user 102 and the second user 104, and / or that the fulfillment rules have been previously generated for both the first user 102 and the second user 104. Thus, at least partially of the vehicle-mounted computing device 110 of the vehicle 100A N A fulfillment system running on (for example, the fulfillment system 120 in Figure 1) can utilize these fulfillment models and / or fulfillment rules, which are specific to the first user 102 and the second user 104, when determining how to dynamically adapt the fulfillment of speech utterances.

[0068] Specifically, referring to Figure 5A, let us further assume that the first user 102 is driving the vehicle 100A and provides the voice utterance 552A "It's cold", and the vehicle-mounted computing device 110 of the vehicle 100A N It is further assumed that the microphone(s) generate audio data that captures the speech utterance 552A. In this example, the audio data capturing the speech utterance 552A can be processed using an ASR model to generate ASR data of the speech utterance 552A (e.g., the recognized text "cold"). Furthermore, the ASR data of the speech utterance 552A can be processed using an NLU model to generate NLU data of the speech utterance 552A (e.g., the recognized intent(s) such as "cold" or "turn up the temperature"). In particular, the user who provided the speech utterance 552A can be determined to correspond to the first user 102 using any of the techniques described herein and / or any other suitable techniques.

[0069] In some embodiments, NLU data for voice utterance 552A can be processed at implementation to determine a set of fulfillment actions that facilitate raising the temperature inside vehicle 100A. The set of fulfillment actions may include, for example, raising the windows of vehicle 100A (when the windows of vehicle 100A are in a state indicating that the windows are down) to raise the temperature inside vehicle 100A; turning on the blower of vehicle 100A to circulate warm air throughout the vehicle to raise the temperature inside vehicle 100A; turning on the heated seat of the seat occupied by the first user 100A to raise the temperature inside vehicle 100A and / or the temperature of the first user 102 located inside vehicle 100A; and / or other fulfillment actions that can be performed to raise the temperature inside vehicle 100A and / or the temperature of the first user 102 located inside vehicle 100A.

[0070] In this embodiment, a set of fulfillment actions can be processed using a fulfillment action model and / or fulfillment rules (or multiple) specific to the first user 102 (e.g., personalized mappings) to select one or more first fulfillment actions to be performed to satisfy the speech utterance 552A. The speech utterance and / or one or more contextual signals obtained in a time instance in which the speech utterance is provided by the first user 102 can be processed using a fulfillment action model and / or fulfillment rules (or multiple) specific to the first user 102 when selecting one or more first fulfillment actions to be performed to satisfy the speech utterance 552A. For example, one or more first fulfillment actions may include turning on a heated seat occupied by a first user 102 to raise the temperature inside the vehicle 100A and / or the body temperature of the first user 102 located in the vehicle 100A, but may not include turning on a blower in the vehicle 100A to circulate warm air throughout the vehicle and raise the temperature inside the vehicle 100A based on the output generated using a fulfillment action model and / or fulfillment rule(s) specific to the first user 102 (e.g., vehicle-mounted computing device 110). N As indicated by the synthesized voice 554A, which is presented audibly to the user via the speaker(s)(or speakers), "I turned on your heated seat, but I didn't turn on the blower." In other words, based on the output generated using a fulfillment action model and / or fulfillment rule(s) that are specific to the first user 102, the fulfillment of the voice utterance 552A is dynamically adapted to the fulfillment action(s) that the first user 102 would manually perform in the same or similar contextual scenario.

[0071] In further or alternative embodiments, one or more first fulfillment actions performed to satisfy a speech utterance 552A can be selected directly without considering a set of fulfillment actions (e.g., using fulfillment rules). For example, one or more first fulfillment actions may be mapped to various speech commands that, when detected in a particular contextual scenario, cause one or more first fulfillment actions to be performed. For example, when generating one or more fulfillment rules specific to a first user 102, a contextual scenario matching the scenario shown in Figure 5A (e.g., the first user 102 is cold in vehicle 100A) may be encountered. Thus, speech commands containing terms such as "cold" or "freezing" can be pre-assigned to one or more first fulfillment actions for the first user 102.

[0072] In contrast, referring to Figure 5B, we further assume that a second user 104 is driving vehicle 100A and provides the voice utterance 552B, “It’s cold.” In particular, the voice utterance 552B provided by the second user 104 in Figure 5B is the same as the voice utterance 552A provided by the first user 102 in Figure 5A. The voice utterance 552B, “It’s cold,” provided by the second user 104 can be processed in the same way as, or similarly to, the voice utterance 552A in Figure 5A described above. However, it should be noted that one or more second fulfillment actions performed to satisfy voice utterance 552B in Figure 5B are different from one or more first fulfillment actions performed to satisfy voice utterance 552A in Figure 5A. For example, one or more second fulfillment actions may include turning on a heated seat occupied by a second user 104 to raise the temperature inside the vehicle 100A and / or the body temperature of a first user 102 located inside the vehicle 100A, or, based on the output generated using a fulfillment action model and / or fulfillment rule(s) specific to the second user 104, turning on a blower in the vehicle 100A to circulate warm air throughout the vehicle and raise the temperature inside the vehicle 100A (e.g., vehicle-mounted computing device 110) N As indicated by the synthesized voice 554B, “I have turned on your heated seat and blower,” which is presented audibly to the user via the speaker(s). In other words, based on the output generated using a fulfillment action model and / or fulfillment rule(s) specific to the second user 102, the fulfillment of the voice utterance 552B is dynamically adapted to fulfillment actions(s) that the second user 104 would manually perform in the same or similar contextual scenario.

[0073] It should be understood that Figures 5A and 5B are illustrative and not intended to be limiting, as they illustrate a first user 102 and a second user 104 driving vehicle 100A together in separate time instances. For example, the techniques described herein are available when both the first user 102 and the second user 104 are located in the same place within the vehicle environment. As one non-limiting example, suppose the first user 102 is driving vehicle 100A and the second user 104 is a passenger in vehicle 100A. In other words, the first user 102 may be associated with the “driver” zone of vehicle 100A, and the second user 104 may be associated with the “passenger seat” zone of vehicle 100A. These zones may be determined based on the vehicle 100A’s occupancy sensors, and / or any techniques described herein, and / or any other suitable techniques for identifying users of vehicle 100A. We further assume that the first user 102 provides the same voice utterances from Figures 5A and 5B. In this example, the first user 102's heated seat in the "driver" zone may be turned on, but other heated seats in the vehicle may not be turned on, and the blower may not be turned on as described with respect to one or more first fulfillment actions in Figure 5A. In some of these examples, the second user 104's heated seat and the blower associated with the "passenger" zone may be turned on as described with respect to one or more second fulfillment actions in Figure 5B, however, any blower associated with the "driver" zone may remain off. Furthermore, we further assume that the second user 102 provides the same voice utterances from Figures 5A and 5B (for example, before or after the first user 102 provides the same voice utterances from Figures 5A and 5B). In this example, as explained with respect to Figure 5B, the heated seat for the second user 102 in the "passenger seat" zone may be turned on, and the blower associated with the "passenger seat" zone may be turned on.Therefore, in various embodiments, one or more fulfillment actions performed for different users located in the same place within the environment may be limited to the zone that is determined to be occupied by the user who provided the voice utterance.

[0074] Referring here to Figures 6A and 6B, various non-limiting examples are shown of dynamically adapting the fulfillment of a given speech utterance in a primary residential environment based on the user who provided the given speech utterance. The floor plans shown in Figures 6A and 6B include multiple rooms 660–672. Multiple computing devices 110 2~6 It is deployed in at least some of the rooms. Computing device 110 2~6 Each of these may implement an instance of a fulfillment system client (e.g., fulfillment system client 114 in Figure 1) configured in a selected aspect of the present disclosure and may include one or more input devices, such as microphones, that can capture utterances spoken by nearby users. For example, computing devices in the form of interactive standalone speakers are deployed in room 660, which is the kitchen in Figures 6A and 6B (e.g., kitchen computing device 1102), room 664, which is the bedroom in Figures 6A and 6B (e.g., bedroom computing device 1104), and room 666, which is the living room in Figures 6A and 6B (e.g., living room computing device 1104). Furthermore, computing devices in the form of so-called "smart" televisions are deployed in room 662, which is the study in Figures 6A and 6B (e.g., study TV 1103), and room 660, which is the kitchen in Figures 6A and 6B (e.g., kitchen TV 1106). Various users of the main residence may use multiple computing devices 110 2~6 It can interact with and control other smart devices (or multiple devices) in the main residence (for example, by changing the state (or status) of smart devices (or multiple devices)).

[0075] The smart devices in the main residence shown in Figures 6A and 6B include, for example, smart lights in one or more of rooms 660-672 of the main residence, smart thermostats in one or more of rooms 660-672 of the main residence, smart appliances in kitchen 660 (multiple) (e.g., smart oven, smart toaster, smart refrigerator, etc.), smart appliances in laundry room 672 (multiple) (e.g., smart washing machine, smart dryer, etc.), smart locks on the doors of the main residence, and a smart alarm system for the main residence (e.g., smart contact sensors, smart...). This may include cameras (or more), smart motion sensors (or more), and / or any other smart devices. Certain smart devices are described herein with respect to Figures 6A and 6B, but it should be understood that this is for illustrative purposes only and not intended to be limiting. For example, with regard to Figures 5A and 5B, vehicle 100A may also include several smart devices. These smart devices include the smart windows of vehicle 100A, the smart locks of vehicle 100A, the smart defrosters of vehicle 100A, the smart heated seats of vehicle 100A, and the vehicle-mounted computing device 110 of the vehicle. N This may include, but is not limited to, a smart infotainment system operating on the vehicle, a smart blower system for the vehicle, and / or other smart devices that can be integrated with the vehicle 100A. In these examples, a user interaction corresponding to one or more fulfillment actions may include changing the state of one of these smart devices.

[0076] Although not shown in Figures 6A and 6B, there are multiple computing devices 106 2~6These devices may be connected to one or more wired or wireless WANs and / or LANs (for example, via network(s) 195 in Figure 1) in a communicative manner, to smart devices in the primary residence, and / or to other resources (e.g., the Internet). Furthermore, other computing devices (in particular mobile devices such as smartphones, tablets, laptops, and wearable devices) may also be present, for example, carried by one or more people within the home, and may or may not be connected to the same WAN and / or LAN. It should be understood that the configurations of computing devices and smart devices shown in Figures 6A and 6B are merely examples, and more or fewer and / or different computing devices and / or smart devices may be deployed across any number of other rooms and / or areas of the primary residence and / or in locations other than the residence (e.g., businesses, hotels, public places, airports, vehicles, and / or other places or spaces).

[0077] For Figures 6A and 6B, we assume that the home environment is at least partially inhabited by multiple users (the first user 102 and the second user 104 shown in Figures 6A and 6B). Furthermore, we assume that a fulfillment action model has been previously trained for both the first user 102 and the second user 104, and / or fulfillment rules have been previously generated for both the first user 102 and the second user 104. Thus, at least partially in the primary residence, computing device 110 2~6 A fulfillment system running on one or more of these (for example, the fulfillment system 120 in Figure 1) can utilize these fulfillment models and / or fulfillment rules, which are specific to the first user 102 and the second user 104, when determining how to dynamically adapt the fulfillment of speech utterances.

[0078] Specifically, referring to Figure 6A, we further assume that a first user 102 is located in room 666, which is the living room in Figures 6A and 6B, and provides the voice utterance 652A, “It’s dark,” and that the microphone(s) of the living room computing device 1105 generate audio data that captures the voice utterance 652A. In this example, the audio data capturing the voice utterance 652A can be processed in the same way as or similarly to the methods described with respect to Figures 5A and 5B to determine one or more fulfillment actions to be performed in response to receiving the voice utterance 652A. In particular, the user who provided the voice utterance 652A can be determined to correspond to the first user 102 using any of the techniques described herein and / or any other suitable techniques. As a result, one or more first fulfillment actions may be performed to make the living room darker, such as turning on the living room smart lights to 100% brightness (for example, as indicated by the synthesized voice 654A, “I have turned on the lights to 100% brightness”).

[0079] Specifically, referring to Figure 6B, we further assume that a second user 104 is located in room 662, which is the study in Figures 6A and 6B, and provides the voice utterance 652B "It's dark," and that the microphone(s) of the study television 1103 generate audio data that captures the voice utterance 652B. In this example, the audio data capturing the voice utterance 652B can be processed in the same way as or similarly to the methods described with respect to Figures 5A and 5B to determine one or more fulfillment actions to be performed in response to receiving the voice utterance 652B. In particular, the user who provided the voice utterance 652A can be determined to correspond to the second user 104 using any of the techniques described herein and / or any other suitable techniques. As a result, one or more second fulfillment actions may be performed to make the study darker, such as turning on the smart lights in the study to 50% brightness (for example, as indicated by the synthesized voice 654B, "I have turned the lights on to 50% brightness").

[0080] In particular, the voice utterances provided by the first user 102 and the second user 104 in Figures 6A and 6B are the same. However, the fulfillment actions performed in response to the voice utterances differ in that, based on the first user 102 providing voice utterance 652a, the living room light is turned on to 100% brightness, and based on the second user 104 providing voice utterance 652B, the study light is turned on to 50% brightness. In some embodiments, this difference in fulfillment actions may be based on different fulfillment action models previously trained for both the first user 102 and the second user 104, as described with respect to Figures 5A and 5B, and / or the fulfillment rule may be based on having been previously generated for both the first user 102 and the second user 104. In other embodiments, this difference in fulfillment actions may be based on different zones in which the speech utterance is received (e.g., the “living room” zone for the first user 102 and the “location” zone for the second user 104). Nevertheless, the techniques described herein are used to dynamically adapt the fulfillment of the same speech utterance.

[0081] Referring now to Figure 7, a block diagram of an exemplary computing device 710 that may be optionally used to perform one or more embodiments of the technology described herein is shown. In some embodiments, one or more computing devices, one or more vehicles, and / or other components may include one or more components of the exemplary computing device 710.

[0082] The computing device 710 typically includes at least one processor 714 that communicates with several peripheral devices via a bus subsystem 712. These peripheral devices may include, for example, a storage subsystem 724 including a memory subsystem 725 and a file storage subsystem 726, a user interface output device 720, a user interface input device 722, and a network interface subsystem 716. The input and output devices enable user interaction with the computing device 710. The network interface subsystem 716 provides an interface to an external network and is coupled to a corresponding interface device in another computing device.

[0083] The user interface input device 722 may include pointing devices such as keyboards, mice, trackballs, touchpads, and graphics tablets; audio input devices such as scanners, touchscreens integrated into displays, and speech recognition systems; microphones; and / or other types of input devices. In general, the use of the term “input device” is intended to include all possible types of devices and methods for inputting information into the computing device 710 or a communication network.

[0084] The user interface output device 720 may include non-visual displays such as a display subsystem, printer, fax machine, or audio output device. The display subsystem may include flat-panel devices such as cathode ray tubes (CRTs) or liquid crystal displays (LCDs), projection devices, or several other mechanisms for creating visible images. The display subsystem may also provide non-visual displays, such as via an audio output device. Generally, the use of the term “output device” is intended to include all possible types of devices and methods for outputting information from the computing device 710 to a user or another machine or computing device.

[0085] The storage subsystem 724 stores programming and data structures that provide some or all of the functionality of the modules described herein. For example, the storage subsystem 724 may include logic that performs selected embodiments of the methods disclosed herein, and logic that implements the various components shown in Figures 1 and 2.

[0086] These software modules typically run on processor 714 alone or in combination with other processors. The memory 725 used by the storage subsystem 724 may include several memories, including main random access memory (RAM) 730 for storing instructions and data during program execution, and read-only memory (ROM) 732 for storing fixed instructions. The file storage subsystem 726 can provide persistent storage of program files and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of a particular embodiment may be stored by the file storage subsystem 726 within the storage subsystem 724, or on other machines accessible by the processor 714(or more).

[0087] The bus subsystem 712 provides a mechanism that enables various components and subsystems of the computing device 710 to communicate with each other as intended. Although the bus subsystem 712 is schematically shown as a single bus, alternative embodiments of the bus subsystem 712 may use multiple buses.

[0088] The computing device 710 may be of various types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing systems or computing devices. Because computers and networks are constantly changing, the description of the computing device 710 shown in Figure 7 is intended only as a specific example to illustrate several embodiments. Many other configurations of the computing device 710 are conceivable, having more or fewer components than the computing device shown in Figure 7.

[0089] Wherever the systems described herein may collect or monitor personal information about a user, or may use personal and / or monitored information, the user may have the opportunity to control whether the program or feature collects user information (e.g., information about the user's social networks, social behavior or activities, occupation, user preferences, or current geographical location), or whether and / or how they receive content from a content server that may be more relevant to the user. Furthermore, certain data may be processed in one or more ways so that personally identifiable information is removed before it is stored or used. For example, a user's identity may be processed in such a way that personally identifiable information cannot be determined, or if location information is obtained (e.g., city, zip code, or state level), the user's geographical location may be generalized so that the user's specific location cannot be determined. Thus, the user may have control over how information is collected and / or used about them.

[0090] In some embodiments, a method is provided which is carried out by one or more processors, the method comprising receiving a given speech utterance from a user and via a computing device, the given speech utterance being provided by the user while the user is located in the user's vehicle, and determining a set of fulfillment actions based on receiving and processing the given speech utterance. Each fulfillment action included in the set of fulfillment actions facilitates the given speech utterance when performed. The method further comprises determining whether the user who provided the given speech utterance corresponds to a first user or a second user, and in response to determining that the user who provided the given speech utterance corresponds to a first user, selecting one or more subsets of first fulfillment actions from the set of fulfillment actions based on the user who provided the given speech utterance corresponding to a first user, and having one or more subsets of first fulfillment actions perform to satisfy the given speech utterance.

[0091] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features:

[0092] In some embodiments, the method may further include, in response to the user who provided a given speech utterance determining that they correspond to a second user, selecting one or more subsets of second fulfillment actions from a set of fulfillment actions based on the user who provided the given speech utterance corresponding to a second user. The second user may be an additional user added to the first user, and the one or more subsets of second fulfillment actions may include at least one fulfillment action not included in the one or more subsets of fulfillment actions. The method may further include having one or more second fulfillment actions performed to satisfy a given speech utterance.

[0093] In some versions of these embodiments, the first user and the second user may be located in the same place within the vehicle when a given speech utterance is received. Furthermore, performing one or more subsets of first fulfillment actions to satisfy a given speech utterance may include performing one or more first fulfillment actions with respect to a first zone of the vehicle occupied by the first user, among several zones of the vehicle, and performing one or more subsets of second fulfillment actions to satisfy a given speech utterance may include performing one or more second fulfillment actions with respect to a second zone of the vehicle occupied by the second user, among several zones of the vehicle.

[0094] In additional or alternative versions of those embodiments, selecting one or more subsets of first fulfillment actions from a set of fulfillment actions based on the fact that a user providing a given speech utterance corresponds to a first user may include processing a set of fulfillment actions using a first user fulfillment action model and / or one or more first user fulfillment rules specific to the first user to select one or more subsets of first fulfillment actions. In some additional versions of those embodiments, selecting one or more subsets of second fulfillment actions from a set of fulfillment actions based on the fact that a user providing a given speech utterance corresponds to a second user may include processing a set of fulfillment actions using a second user fulfillment action model and / or one or more second user fulfillment rules specific to the second user to select one or more subsets of second fulfillment actions.

[0095] In some embodiments, the method may further include causing a subset of one or more first fulfillment actions to be performed to satisfy a given speech utterance, followed by providing, for presentation to a first user, a display of the reasons why one or more first fulfillment actions were performed to satisfy the given speech utterance.

[0096] In some embodiments, processing a given speech utterance may include using an automatic speech recognition (ASR) model to process audio data capturing the given speech utterance to generate ASR data for the given speech utterance, and using a natural language understanding (NLU) model to process the ASR data for the given speech utterance to generate NLU data for the given speech utterance. In some versions of those embodiments, determining a set of fulfillment actions based on processing a given speech utterance may include determining a set of fulfillment actions based on the NLU data for the given speech utterance.

[0097] In some embodiments, determining whether a user who provided a given speech utterance corresponds to a first user or a second user may involve using a speaker identification (SID) model to process audio data capturing a given speech utterance and determining whether a user who provided a given speech utterance corresponds to a first user or a second user. In some versions of those embodiments, using an SID model to process audio data capturing a given speech utterance to determine whether the user who provided the given speech utterance corresponds to a first user or a second user may include, in response to determining that the speech utterance is not long enough to perform a text-independent (TI) SID, using a text-dependent (TD) SID model to process the audio data to generate a TD speaker embedding, comparing the TD speaker embedding within the TD SID embedding space to one or more stored TD speaker embeddings, and determining whether the user who provided the given speech utterance corresponds to a first user or a second user based on the comparison of the TD speaker embedding with one or more stored TD speaker embeddings. In some further versions of those embodiments, using an SID model to process audio data capturing a given speech utterance to determine the identity of the user who provided the given speech utterance may include, in response to determining that the speech utterance is long enough to perform a TI SID, processing the audio data to generate a TI speaker embedding using a TI SID model as the SID model, comparing the TI speaker embedding with one or more stored TI speaker embeddings within the TI SID embedding space, and determining, based on the comparison of the TI speaker embedding with one or more stored TI speaker embeddings, whether the user who provided the given speech utterance corresponds to a first user or a second user.In some versions of these embodiments, audio data capturing a given speech utterance may be generated via one or more microphones in a computing device.

[0098] In some embodiments, determining whether a user providing a given speech utterance corresponds to a first user or a second user may involve using a facial recognition (FID) model to process visual data capturing the user providing the given speech utterance to generate a face embedding, comparing the face embedding to one or more stored face embeddings within the face embedding space, and determining whether the user providing the given speech utterance corresponds to a first user or a second user based on the comparison of the face embedding to one or more stored face embeddings. In some versions of those embodiments, the visual data capturing the user providing the given speech utterance may be generated via one or more visual components of a computing device or an additional computing device communicatively coupled to the computing device.

[0099] In some embodiments, determining whether a user who provided a given speech utterance corresponds to a first user or a second user may involve using a corresponding device identifier of a computing device or an additional computing device communicatively coupled to a computing device to determine whether a user who provided a given speech utterance corresponds to a first user or a second user.

[0100] In some embodiments, determining whether a user who provided a given speech utterance corresponds to a first user or a second user may involve using a corresponding user account on a computing device, or on an additional computing device communicatively coupled to a computing device, to determine whether a user who provided a given speech utterance corresponds to a first user or a second user.

[0101] In some embodiments, a method is provided which is carried out by one or more processors, the method comprising receiving a given speech utterance from a user and via a computing device, the given speech utterance being provided by the user while located within the user's primary residence, and determining a set of fulfillment actions based on receiving and processing the given speech utterance. Each fulfillment action included in the set of fulfillment actions facilitates the given speech utterance when performed. The method further comprises determining whether the user who provided the given speech utterance corresponds to a first user or a second user, and in response to determining that the user who provided the given speech utterance corresponds to a first user, selecting one or more subsets of first fulfillment actions from the set of fulfillment actions based on the user who provided the given speech utterance corresponding to a first user, and having one or more subsets of first fulfillment actions perform to satisfy the given speech utterance.

[0102] In some embodiments, methods are provided that are carried out by one or more processors, the methods include: identifying the occurrence of a user interaction with one or more smart devices in a given time instance of a plurality of time instances, the user interaction corresponding to one or more fulfillment actions; obtaining one or more contextual signals that characterize the state of the user in a given time instance and / or the state of the user's environment in a given time instance; generating a given training instance based on the user interaction and based on one or more contextual signals; training a user-specific fulfillment action model for at least a given time instance in response to a determination that one or more training conditions are met; and using the user-specific fulfillment action model in responding to speech utterances received from the user.

[0103] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features:

[0104] In some embodiments, the method may further include, based on user interaction and one or more contextual signals, generating a prompt requesting the user to verify whether one or more fulfillment actions were performed based on the user's state in a given time instance and / or the state of the user's environment in a given time instance, before generating a given training instance; providing the prompt to the user for presentation; and, in response to the prompt, receiving user input verifying that one or more fulfillment actions were performed based on the user's state in a given time instance and / or the state of the user's environment in a given time instance. In some versions of those embodiments, generating a given training instance based on user interaction and one or more contextual signals may be in response to receiving verification that one or more fulfillment actions were performed based on the user's state in a given time instance and / or the state of the user's environment in a given time instance.

[0105] In some embodiments, generating a given training instance based on user interaction and on one or more context signals may include determining, for a given training instance, the training instance input, which includes (i) one or more context signals characterizing the user's state at a given time instance and / or the state of the user's environment at a given time instance, and (ii) a set of fulfillment actions associated with one or more context signals, and determining the training instance output, which includes one or more fulfillment actions for a user interaction. In some versions of those embodiments, training a user-specific fulfillment action model based on at least a given training instance may include using the fulfillment action model to process the training instance input to generate a predictive output, comparing the predictive output to the training instance output to generate one or more losses, and updating the fulfillment action model based on one or more losses.

[0106] In some embodiments, one or more training conditions may include the number of training instances available for training the fulfillment action model, the time of day, and / or the day of the week.

[0107] In some embodiments, the method may further include identifying the occurrence of an additional user interaction with one or more smart devices of an additional user in an additional given time instance of a plurality of time instances, wherein the additional user interaction corresponds to one or more additional fulfillment actions, and the one or more additional fulfillment actions include at least one fulfillment action distinct from one or more fulfillment actions; obtaining one or more additional context signals that characterize an additional state of the additional user in the additional given time instance and / or an additional state of the additional user's environment in the additional given time instance; generating an additional given training instance based on the additional user interaction and based on one or more additional context signals; training an additional fulfillment action model specific to the additional user based on at least the additional given training instance in response to a determination that one or more training conditions are met; and making the additional fulfillment action model specific to the additional user available when responding to speech utterances received from the additional user.

[0108] In some embodiments, a method is provided which is carried out by one or more processors, the method being to identify the occurrence of a user interaction with one or more smart devices in a given time instance of a plurality of time instances, the user interaction being to identify one or more fulfillment actions, to obtain one or more context signals that characterize the user's state in a given time instance and / or the state of the user's environment in a given time instance, and to generate a prompt requesting the user to verify whether one or more fulfillment actions were performed based on the user's state in a given time instance and / or the state of the user's environment in a given time instance. This includes: providing a prompt to the user; receiving user input in response to the prompt that verifies that one or more fulfillment actions were performed based on the user's state in a given time instance and / or the state of the user's environment in a given time instance; generating one or more user-specific fulfillment action rules in response to receiving user input that verifies that one or more fulfillment actions were performed based on the user's state in a given time instance and / or the state of the user's environment in a given time instance; and using one or more user-specific fulfillment action rules when responding to speech utterances received from the user.

[0109] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features:

[0110] In some embodiments, the method involves identifying the occurrence of an additional user interaction with one or more smart devices by an additional user in an additional given time instance among multiple time instances, wherein the additional user interaction corresponds to one or more additional fulfillment actions, and the one or more additional fulfillment actions include at least one fulfillment action distinct from one or more other fulfillment actions; obtaining one or more additional context signals that characterize an additional state of the additional user in the additional given time instance and / or an additional state of the additional user's environment in the additional given time instance; and verifying whether the one or more additional fulfillment actions were performed based on the additional state of the additional user in the additional given time instance and / or an additional state of the additional user's environment in the additional given time instance. The further includes generating an additional prompt requesting to do something, providing additional prompts to be presented to additional users, receiving additional user input in response to the additional prompts verifying that one or more additional fulfillment actions were performed based on additional states of the additional user in an additional given time instance and / or on additional states of the user's environment in an additional given time instance, generating one or more additional fulfillment action rules specific to the additional user in response to the receipt of additional user input verifying that one or more additional fulfillment actions were performed based on additional states of the additional user in an additional given time instance and / or on additional states of the user's environment in an additional given time instance, and using one or more additional fulfillment action rules specific to the additional user when responding to speech utterances received from the additional user.

[0111] In some embodiments, a method is provided which is carried out by one or more processors, the method comprising generating a personalized mapping for a user of one or more voice commands to one or more fulfillment actions. Generating the personalized mapping is in response to determining one or more past instances of a user that caused the execution of one or more fulfillment actions in a contextual scenario that is determined to correlate with one or more voice commands, regardless of providing a voice utterance. The method further comprises receiving a given voice utterance from the user, and following the generation of the personalized mapping, determining, based on processing the given voice utterance, that the given voice utterance is one of one or more voice commands, and, in response to determining that the given utterance is one of one or more voice commands, automatically executing one or more fulfillment actions using the personalized mapping.

[0112] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features:

[0113] In some embodiments, generating personalized mappings of one or more voice commands to one or more fulfillment actions may include generating mappings between voice command semantic representations of one or more voice commands and fulfillment action semantic representations of one or more fulfillment actions, within a semantic space and based on one or more past instances, and storing the mappings in one or more databases.

[0114] In some embodiments, generating personalized mappings of one or more voice commands to one or more fulfillment actions may include generating mappings between the intent of one or more voice commands and the fulfillment actions of one or more fulfillment actions based on one or more past instances, and storing the mappings in one or more databases.

[0115] In some embodiments, a method is provided which is carried out by one or more processors, the method comprising identifying the occurrence of a user interaction with one or more smart devices in a given time instance of a plurality of time instances, the user interaction corresponding to one or more fulfillment actions, and training one or more context signals that characterize the user's contextual scenario. The one or more context signals that characterize the user's contextual scenario characterize the user's state in a given time instance and / or characterize the state of the user's environment in a given time instance. The method further includes identifying one or more voice commands to assign to the user's contextual scenario. The one or more voice commands cause one or more fulfillment actions of the user interaction to be carried out when later provided by the user. The method is to receive a given instance of speech utterance from a user, in a subsequent time instance of a plurality of time instances, the subsequent time instance further includes, based on receiving and processing the given instance of speech utterance, determining that the given instance of speech utterance contains one or more speech commands, deciding to perform one or more fulfillment actions to satisfy the given instance of speech utterance, and performing one or more fulfillment actions to satisfy the given instance of speech utterance.

[0116] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features:

[0117] In some embodiments, the method may further include identifying, in a further subsequent time instance of a plurality of time instances, an additional occurrence of an additional user interaction with one or more smart devices of an additional user interaction, the additional user interaction corresponding to one or more additional fulfillment actions, the one or more additional fulfillment actions comprising at least one fulfillment action distinct from one or more fulfillment actions, and the further subsequent time instance following a given time instance, and obtaining one or more additional context signals characterizing an additional context scenario of the additional user. One or more additional context signals characterizing an additional context scenario of the additional user may characterize an additional state of the additional user in a further subsequent time instance, and / or characterize an additional state of the additional user's environment in a further subsequent time instance, and the additional context scenario of the additional user may coincide with the user's context scenario. The method may further include identifying one or more voice commands to assign to the additional context scenario of the additional user. One or more voice commands may, when later provided by the additional user, cause one or more fulfillment actions of a user interaction less than or equal to a certain value. The method is to receive an additional instance of a given speech utterance from an additional user, and in a further subsequent time instance of a plurality of time instances, and further, based on receiving and processing the additional instance of the given speech utterance, the further subsequent time instance determines that the additional instance of the given speech utterance contains one or more speech commands, decides to perform one or more additional fulfillment actions to satisfy the additional instance of the given speech utterance, and may further include performing one or more additional further fulfillment actions to satisfy the additional instance of the given speech utterance.

[0118] Furthermore, some embodiments include one or more processors in one or more computing devices (e.g., a central processing unit (or multiple CPUs), a graphics processing unit (or multiple GPUs), and / or a tensor processing unit (or multiple TPUs)), where one or more processors are operable to execute instructions stored in associated memory, and the instructions are configured to cause the execution of any of the methods described above. Some embodiments also include one or more non-temporary computer-readable storage media for storing computer instructions that can be executed by one or more processors to perform any of the methods described above. Some embodiments also include a computer program product that includes instructions that can be executed by one or more processors to perform any of the methods described above.

[0119] It should be understood that all combinations of the concepts described above, and any additional concepts described in more detail herein, are intended to be part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are intended to be part of the subject matter disclosed herein.

Claims

1. A method carried out by one or more processors, Receiving a given speech utterance from a user and via a computing device, wherein the given speech utterance is provided and received while the user is located within the user's vehicle. Determining a set of fulfillment actions based on processing a given speech utterance, wherein each fulfillment action included in the set of fulfillment actions facilitates the fulfillment of the given speech utterance when performed, and each fulfillment action included in the set of fulfillment actions is associated with controlling a corresponding component of the vehicle. To determine whether the user who provided the given speech utterance corresponds to a first user or a second user, In response to the user who provided the given speech utterance being determined to correspond to the first user, Based on the user who provided the given voice utterance corresponding to the first user, the selection of one or more subsets of first fulfillment actions from the set of fulfillment actions, wherein the subset of the one or more first fulfillment actions is associated with controlling at least a first component of the vehicle. Performing the subset of the one or more first fulfillment actions in order to satisfy the given speech utterance, In response to the user who provided the given speech utterance determining that they correspond to the second user, Based on the user who provided the given voice utterance corresponding to the second user, the selection of one or more subsets of second fulfillment actions from the set of fulfillment actions, wherein the second user is an additional user added to the first user, and the one or more subsets of second fulfillment actions are associated with controlling at least a second component of the vehicle that is not included in the one or more subsets of first fulfillment actions. Performing the subset of the one or more second fulfillment actions in order to satisfy the given speech utterance, Methods that include...

2. When the given voice utterance is received, the first user and the second user are located in the same place within the vehicle. Performing one or more subsets of the first fulfillment actions to satisfy the given voice utterance includes performing one or more of the first fulfillment actions with respect to a first zone of the vehicle occupied by the first user, among a plurality of zones of the vehicle. Performing one or more subsets of the second fulfillment actions to satisfy the given voice utterance includes performing one or more of the second fulfillment actions with respect to a second zone of the vehicle occupied by the second user, among the plurality of zones of the vehicle. The method according to claim 1.

3. Based on the fact that the user who provided the given speech utterance corresponds to the first user, selecting one or more subsets of the first fulfillment actions from the set of fulfillment actions is: Processing the set of fulfillment actions using a first user fulfillment action model specific to the first user, and / or one or more first user fulfillment rules specific to the first user, and selecting a subset of the one or more first fulfillment actions. The method according to claim 1, including the method described in claim 1.

4. Based on the fact that the user who provided the given speech utterance corresponds to the second user, selecting one or more subsets of the second fulfillment actions from the set of fulfillment actions is: Processing the set of fulfillment actions using a second user fulfillment action model specific to the second user, and / or one or more second user fulfillment rules specific to the second user, and selecting a subset of the one or more second fulfillment actions. The method according to claim 3, including the method described in claim 3.

5. In response to the user who provided the given speech utterance being determined to correspond to the first user, Following the execution of one or more subsets of the first fulfillment actions to satisfy the given speech utterance, To provide, for presentation to the first user, an indication of the reason why one or more of the first fulfillment actions were performed to satisfy the given speech utterance. The method according to claim 1, further comprising:

6. Processing the given speech utterance is Using an automatic speech recognition (ASR) model, the audio data capturing the given speech utterance is processed to generate ASR data of the given speech utterance. Using a natural language understanding (NLU) model, the ASR data of the given speech utterance is processed to generate NLU data of the given speech utterance. The method according to claim 1, including the method described in claim 1.

7. Determining the set of fulfillment actions based on processing the given speech utterance is Based on the NLU data of the given speech utterance, the set of fulfillment actions is determined. The method according to claim 6, including the method described in claim 6.

8. Determining whether the user who provided the given speech utterance corresponds to the first user or the second user is: Using a speaker identification (SID) model, process the audio data capturing the given speech utterance to determine whether the user who provided the given speech utterance corresponds to the first user or the second user. The method according to claim 1, including the method described in claim 1.

9. Using the SID model, the audio data capturing the given speech utterance is processed to determine whether the user who provided the given speech utterance corresponds to the first user or the second user. In response to determining that the aforementioned speech utterance is not long enough to perform a text-independent (TI) SID, Using a text-dependent (TD) SID model as the SID model, the audio data is processed to generate TD speaker embeddings. Within the TD SID embedding space, the TD speaker embedding is compared with one or more stored TD speaker embeddings, Based on comparing the TD speaker embedding with one or more stored TD speaker embeddings, it is determined whether the user who provided the given speech utterance corresponds to the first user or the second user. The method according to claim 8, including the method described in claim 8.

10. Using the SID model, the audio data capturing the given speech utterance is processed to determine the identity of the user who provided the given speech utterance. In response to determining that the aforementioned speech utterance is of sufficient length to perform TI SID, Using the TI SID model as the SID model, the audio data is processed to generate TI speaker embeddings. Within the TI SID embedding space, the TI speaker embedding is compared with one or more stored TI speaker embeddings, Based on comparing the TI speaker embedding with one or more stored TI speaker embeddings, it is determined whether the user who provided the given speech utterance corresponds to the first user or the second user. The method according to claim 9, including the method described in claim 9.

11. The method according to claim 8, wherein the audio data capturing the given speech utterance is generated via one or more microphones of the computing device.

12. Determining whether the user who provided the given speech utterance corresponds to the first user or the second user is: Using a facial recognition (FID) model, the visual data capturing the user who provided the voice utterance is processed to generate a face embedding. In the face implantation space, the face implantation is compared with one or more stored face implantations, Based on comparing the face embedding with one or more stored face embeddings, it is determined whether the user who provided the given speech utterance corresponds to the first user or the second user. The method according to claim 1, including the method described in claim 1.

13. The method according to claim 12, wherein the visual data capturing the user who provided the given speech utterance is generated via one or more visual components of the computing device or an additional computing device communicatively coupled to the computing device.

14. Determining whether the user who provided the given speech utterance corresponds to the first user or the second user is: Using the corresponding device identifier of the computing device, or any additional computing device communicatively coupled to the computing device, it is determined whether the user who provided the given voice utterance corresponds to the first user or the second user. The method according to claim 1, including the method described in claim 1.

15. Determining whether the user who provided the given speech utterance corresponds to the first user or the second user is: Using the corresponding user count of the computing device, or any additional computing device communicatively coupled to the computing device, it is determined whether the user who provided the given voice utterance corresponds to the first user or the second user. The method according to claim 1, including the method described in claim 1.

16. A method carried out by one or more processors, Receiving a given speech utterance from a user and via a computing device, wherein the given speech utterance is provided and received while the user is located within the user's primary residence. Determining a set of fulfillment actions based on processing the given speech utterance, wherein each fulfillment action included in the set of fulfillment actions facilitates the fulfillment of the given speech utterance at the time of execution, and at least one fulfillment action included in the set of fulfillment actions is associated with controlling the settings of a corresponding IoT (Internet-of-Thing) device. To determine whether the user who provided the given speech utterance corresponds to a first user or a second user, In response to the user who provided the given speech utterance determining that it corresponds to the first user, Based on the user who provided the given voice utterance corresponding to the first user, the selection of one or more subsets of first fulfillment actions from the set of fulfillment actions, wherein the subset of one or more first fulfillment actions is associated with controlling a corresponding IoT device using a first set of settings. Performing the subset of the one or more first fulfillment actions in order to satisfy the given speech utterance, In response to the user who provided the given speech utterance determining that they correspond to the second user, Based on the user who provided the given voice utterance corresponding to the second user, the selection of one or more subsets of second fulfillment actions from the set of fulfillment actions, wherein the second user is an additional user added to the first user, and the one or more subsets of second fulfillment actions are associated with controlling the corresponding IoT device using a second set of settings different from the first set of settings. Performing the subset of the one or more second fulfillment actions in order to satisfy the given speech utterance, Methods that include...

17. Based on the fact that the user who provided the given speech utterance corresponds to the first user, one or more subsets of first fulfillment actions are selected from the set of fulfillment actions. This includes processing the set of fulfillment actions using a first user fulfillment action model specific to the first user and / or one or more first user fulfillment rules specific to the first user, and selecting a subset of the one or more first fulfillment actions. Based on the fact that the user who provided the given speech utterance corresponds to the second user, selecting one or more subsets of the second fulfillment actions from the set of fulfillment actions is: This includes processing the set of fulfillment actions using a second user fulfillment action model specific to the second user and / or one or more second user fulfillment rules specific to the second user, and selecting a subset of the one or more second fulfillment actions. The method according to claim 16.

18. In response to the user who provided the given speech utterance being determined to correspond to the first user, Following the execution of one or more subsets of the first fulfillment actions to satisfy the given speech utterance, To provide, for presentation to the first user, an indication of the reason why one or more of the first fulfillment actions were performed to satisfy the given speech utterance. The method according to claim 16, further comprising:

19. It is a system, One or more processors, A memory that stores instructions for causing one or more processors to execute the method according to any one of claims 1 to 18 during execution. A system equipped with these features.

20. A computer-readable storage medium that stores instructions for causing one or more processors to perform the method according to any one of claims 1 to 18 during execution.

21. A vehicle-mounted computing device, One or more processors, A memory that stores instructions for causing one or more processors to execute the method according to any one of claims 1 to 18 during execution. A vehicle-mounted computing system comprising the above.

Images