Selective registration to automated assistant

Abstract

The invention relates to selective registration to an automated assistant. Techniques are described herein for dialog-based registration of individual users with single-mode recognition and / or multi-mode recognition by an automated assistant, and determining how to respond to a request by a particular user based on the fact that the particular user has registered and / or recognized. Conversation-based registration enables a user to register themselves (or others) through human-to-machine conversations with an automated assistant, unlike requiring operation of a graphical user interface for personal registration.

Description

Description of the case This application is a divisional application of the patent application with application number 201880094608.1, which was submitted to the China Patent Office on December 14, 2020, and the invention name is "Selective Registration to Automatic Assistant". Background Art

[0001] A person may participate in a human-computer conversation with an interactive software application referred to herein as an "automated assistant" (also referred to as a "chatbot," "interactive personal assistant," "intelligent personal assistant," "personal voice assistant," "conversational agent," "virtual assistant," etc.). For example, a human (who may be referred to as a "user" when interacting with an automated assistant) may provide commands, queries, and / or requests (collectively referred to herein as "queries") using free-form natural language input, which may include spoken utterances and / or typed free-form natural language input that are converted to text and then processed.

[0002] Different users may control and / or have access to different resources via the automated assistant. For example, a trusted user may have permission to cause the automated assistant to perform various actions that an untrusted user may not necessarily be able to perform, such as controlling smart appliances (e.g., lights, thermostats, locks, etc.). As another example, the automated assistant may have the ability to present various content to the user. Some of this content (such as personal documents, calendar data, etc.) may be protected, and the automated assistant will only present the content after identifying / authenticating the requesting user. The automated assistant may deny access to the same protected content to unidentified or unauthorized users. Other content may not necessarily be protected, but may be inappropriate for some users. For example, a child may be prevented from asking the automated assistant to initiate playback of content for which parental discretion is recommended.

[0003] Configuring existing automated assistants to distinguish between different individuals currently requires manual interaction with a graphical user interface, e.g., turning on a "voice match" feature. As a result, other individuals who may lack sufficient knowledge or motivation to access such an interface may never register with the voice match feature. Furthermore, particularly in noisy environments, or in scenarios where multiple speakers have similar voices / accents, using only voice matching techniques (also referred to herein as "speaker recognition processing") to distinguish between different individuals may not be sufficiently reliable.

[0004] In many cases, before an automated assistant can interpret and respond to a user's request, it must first be "invoked," for example, using a predefined verbal invocation phrase commonly referred to as a "hot word" or "wake-up word." Therefore, many automated assistants operate in what is referred to herein as a "limited hot word listening state" or "default listening state," in which they always "listen" to audio data sampled by a microphone for a limited (or restricted, or "default") set of hot words. Any utterances captured in the audio data other than the default set of hot words will be ignored. Once the automated assistant has been invoked with one or more hot words in the default set of hot words, it can operate in a state referred to herein as a "speech recognition state," in which, for at least some time intervals after the invocation, the automated assistant performs speech-to-text ("STT") processing on the audio data sampled by the microphone to generate text input, which is then semantically processed to determine and fulfill the user's intent. Existing automated assistants can typically only be invoked using one or more default hot words, which are the same regardless of whether the requesting user is recognized. Summary of the invention

[0005] This document describes techniques for improving security processes in an automated assistant through selective registration, wherein a user's registration to the automated assistant unlocks one or more features of the automated assistant that were not available to the user prior to registration. In particular, the following techniques are described for conversation-based registration of individual users for single-mode and / or multi-mode identification through an automated assistant, and determining how to respond to a request from a particular user based on the fact that the particular user has been registered and / or identified. Conversation-based registration enables a user to register himself (or others) through a human-machine dialogue with the automated assistant, as opposed to requiring operation of a graphical user interface to perform a personal registration. The example implementations described herein improve security by placing users at different trust levels, wherein access to functions of the automated assistant that may be considered sensitive (such as controlling home appliances and / or accessing protected data) is limited based on the trust level.

[0006] The techniques described herein may often (but not exclusively) be used on what are referred to herein as "assistant devices." Assistant devices are computing devices designed primarily to facilitate human-computer conversations between a user and an automated assistant. Many assistant devices take the form of stand-alone interactive speakers, which are becoming increasingly common. Stand-alone interactive speakers are typically placed in high-traffic locations, such as kitchens, living rooms, conference rooms, and the like, and are often interacted with by multiple different people (e.g., family members, co-workers, visitors, and the like).

[0007] While it is possible to register any individual who has ever interacted with an assistant device, doing so may have various disadvantages. Information that can be used to identify an individual (referred to herein as "distinguishing attributes of a user"), such as the "voice profiles" and "visual profiles" described herein, may need to be stored locally on the assistant device. In many cases, assistant devices are resource-constrained (e.g., relatively small memory and / or processing power) for economic and / or technical reasons. Therefore, storing data indicating the distinguishing attributes of a large number of users may require an excessive amount of the assistant device's limited memory. In addition, if a particular individual's interaction with the assistant device is likely to be minimal, for example, they are a temporary visitor to a household in which the assistant device is deployed, then registering that individual may be wasteful. In addition, such temporary visitors may not wish to be registered, for example, because they do not want data indicating their distinguishing attributes to be stored on other people's assistant devices.

[0008] Thus, in various embodiments, before a heretofore unknown individual is enrolled into an automated assistant using the techniques described herein, the automated assistant may determine whether the individual meets one or more "automated assistant enrollment criteria." These criteria may include, for example, a threshold number of different human-computer conversational sessions that the individual has conducted with the automated assistant on the same assistant device or on one or more computing devices on a collaborative ecosystem of computing devices controlled by a "host" user (e.g., a person who controls / configures the ecosystem of devices, such as an owner, householder, etc.). Additionally or alternatively, these criteria may include a threshold number of conversational turns that have occurred between the individual and the automated assistant.

[0009] In order to determine whether an individual meets the automated assistant registration criteria, one or more distinguishing attributes of the individual may be identified, for example, based on signals generated by one or more hardware sensors integrated with or otherwise communicatively coupled to the assistant device. These hardware sensors may include, for example, visual sensors (e.g., cameras, passive infrared sensors, etc.), pressure sensors (e.g., microphones, ultrasonic sensors, etc.), or wireless receivers that can detect wireless signals (e.g., Wi-Fi, Bluetooth, ZigBee, Z-wave, RFID, visual markers) emitted by a mobile device carried by the individual. Based on the distinguishing attributes of the identified individual, historical interaction data (e.g., logs maintained by or on behalf of the automated assistant) may be analyzed to identify one or more previous human-to-computer dialog sessions in which the same individual exchanged conversations with the automated assistant (e.g., using the same assistant device or another computing device in the same collaborative ecosystem of computing devices). Based on this analysis, if the automated assistant registration criteria are met, the automated assistant may initiate a routine that will be referred to herein as a "human-to-computer dialog enrollment routine."

[0010] During the human-computer dialogue registration routine, the automated assistant may provide a natural language output including instructions for causing the user to perform various actions, such as facilitating future identification of the user by capturing and storing data indicating the distinguishing attributes of the user. For example, during the visual registration routine, the automated assistant may instruct the user to reposition the user's face into a plurality of different postures and use a visual sensor to capture the user's face in the plurality of different postures. Capturing a plurality of diverse and different images of the user's face may enable the creation of a "visual profile" of the user. This visual profile may be used, for example, to detect / identify the user using facial recognition processing in the future. In some embodiments, the user's visual profile may include some combination of the plurality of images and / or some combination of features extracted from the plurality of images. Additionally or alternatively, in some embodiments, the visual profile may be "baked into" a machine learning classifier / model (e.g., a convolutional neural network). Future images may be used as input to be applied to such a classifier / model, and output generated based on the model may indicate the identity of the user.

[0011] In addition to or in lieu of visual registration, in some embodiments, an automated assistant configured with selected aspects of the present disclosure may trigger a voice registration routine. During the voice registration routine, the automated assistant may instruct the user to speak various words and / or phrases. These words or phrases may be selected because they are suitable for generating a "voice profile" for the user. The user's utterances of these words / phrases may be used to construct a voice profile, and the voice profile can be used, for example, in conjunction with subsequently captured audio data to perform speaker recognition. Like visual profiles, voice profiles may take various forms, such as data indicative of user utterances, features extracted from user utterances, parameters of a trained machine learning classifier / model, and the like.

[0012] In some embodiments, once a user enrolls, the user's identity (e.g., unique identifier, username, etc.) can be stored in one or more databases (e.g., local to the assistant device, or in a remote cloud infrastructure) in association with data indicating one or more distinguishing attributes of the user. In some embodiments, these distinguishing features can be stored as an "enrollment" embedding generated from visual / pressure sensor data, which is applied as input to a machine learning model (such as various types of neural networks). These distinguishing features can be detected later, such as during a subsequent human-machine conversation session between the user and the automated assistant, and used to determine the user's identity, thereby effectively authenticating the user to the automated assistant.

[0013] For example, in some embodiments, visual sensor data and / or pressure sensor data captured with an as-yet-unrecognized individual can be applied to the same machine learning model to generate a new embedding. This new embedding can be compared to previously stored enrollment embeddings (e.g., determining the Euclidean distance between them) to determine whether the embedding of the approaching individual is similar enough to one of the existing enrollment embeddings to reliably match the approaching individual to a previously enrolled individual.

[0014] In various embodiments, the registration of a user may unlock one or more features of the automated assistant that were not available to the user prior to registration. These features may be available to the user after the user is identified based on the user's registration. For example, in some embodiments, one or more dynamic or custom hot words may be activated so that the user, when later identified (e.g., using speaker recognition and / or facial recognition), can use these dynamic hot words to invoke the automated assistant in addition to or in place of default hot words that are also available to unidentified users. Additionally or alternatively, in various embodiments, other features of the automated assistant (or features associated with the automated assistant) may be unlocked to registered users. For example, these features may include the ability to enable the automated assistant to perform one or more actions that cannot be performed when requested by an unregistered user, such as changing one or more parameters of a smart appliance, accessing protected data, ordering and / or ordering services, making payments, and the like.

[0015] In some embodiments, the identification of registered users can generate a confidence measure. For example, in some embodiments, users may be required to perform both speaker identification and facial recognition to register. Later, when such a user approaches the assistant device, it may be the case that the hardware sensors of the assistant device or hardware sensors associated with the assistant device are unable to capture enough data to perform both speaker identification and facial recognition with high confidence, for example, because the camera is malfunctioning, the computing device does not have a camera, the user is mumbling or speaking too softly to enable reliable speaker identification, etc. In some such embodiments, users may nonetheless be identified with limited confidence. Such users may be granted limited access to various features of the automated assistant rather than full access if they are identified with greater confidence.

[0016] For example, in some embodiments, detected users may be placed in "levels" or "bins" of trust. A first or highest trust level may be assigned to a user for whom facial recognition and / or speaker recognition (or recognition based on wireless signals emitted by the user) generates a confidence metric that satisfies a first threshold. A second trust level may be assigned to a user for whom facial recognition and / or speaker recognition generates a confidence metric that satisfies a second threshold but does not satisfy the first threshold. A third trust level may be assigned to a user for whom facial recognition and / or speaker recognition generates a confidence metric that satisfies a third threshold but does not satisfy the first or second thresholds. And so on, until the user cannot be identified at all, in which case they may be assigned the lowest trust level (e.g., "Guest"). In various embodiments, each trust level may unlock various features of the automated assistant for the user. For example, a user assigned to the first trust level (i.e., voice recognition / speaker recognition generates a relatively high confidence metric) may gain unfettered access to features of the automated assistant that may be considered sensitive, such as controlling home appliances and / or accessing protected data. In contrast, users assigned to the lowest trust levels may be considered "guests" and may be denied access entirely or only be allowed access to features of the automated assistant that are considered non-sensitive (e.g., weather forecasts, sports scores, movie schedules, etc.).

[0017] In some embodiments, a method performed by one or more processors is provided, the method comprising: executing an automated assistant at least in part on one or more computing devices; processing one or more sensor signals generated by one or more hardware sensors integrated with one or more of the computing devices; based on the processing, identifying one or more distinguishing attributes of a user within the range of the one or more hardware sensors; based on the one or more distinguishing attributes, analyzing historical interaction data to identify one or more previous human-computer dialogue sessions in which the user exchanged conversations with the automated assistant using one or more of the computing devices; based on the identified one or more previous human-computer dialogue sessions, determining that the user meets automated assistant registration criteria; and in response to determining that the user meets the automated assistant registration criteria, participating in a human-computer dialogue registration routine in which the user is requested to register with the automated assistant, wherein the registration of the user includes storing the identity of the user in association with data indicating one or more of the distinguishing attributes of the user in one or more databases, and wherein the registration of the user unlocks one or more features of the automated assistant that were not available to the user prior to the registration.

[0018] In various embodiments, the one or more hardware sensors may include a visual sensor, and the one or more distinguishing attributes may include a visual profile of the user. In various embodiments, the visual profile of the user can be used in conjunction with a sensor signal generated by the visual sensor or another visual sensor to identify the user using facial recognition processing.

[0019] In various embodiments, the one or more hardware sensors may include a microphone, and the one or more distinguishing attributes may include a voice profile of the user. In various embodiments, the voice profile of the user can be used in conjunction with the sensor signal generated by the microphone or another microphone to identify the user using speaker recognition processing.

[0020] In various embodiments, the one or more distinguishing attributes may include a signal transmitted by a mobile device carried by the user. In various embodiments, the one or more unlocked features may include activating one or more hot words that can be used to invoke the automated assistant. In various embodiments, the one or more unlocked features may include one or more responsive actions that can be performed by the automated assistant. In various embodiments, the one or more unlocked features may include access to protected content.

[0021] In various implementations, the automated assistant registration criteria may include a threshold number of human-computer conversation sessions between the user and the automated assistant using one or more of the computing devices. In various implementations, the automated assistant registration criteria may include a threshold number of conversation turns in a human-computer conversation session between the user and the automated assistant using one or more of the computing devices.

[0022] In various implementations, the human-computer dialogue routine may include: instructing the user to reposition the user's face to a plurality of postures; and using a visual sensor to capture the user's face in the plurality of postures.

[0023] In addition, some embodiments include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause the execution of any of the foregoing methods. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions that can be executed by one or more processors to perform any of the foregoing methods.

[0024] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the beginning of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented.

[0026] Figure 2 Depicted are examples of state machines that may be implemented by an automated assistant configured with selected aspects of the present disclosure, according to various implementations.

[0027] Figure 3A , Figure 3B , Figure 4A and Figure 4B Additional example situations are depicted in which the disclosed techniques may be employed.

[0028] Figure 5 An example of assigning a trust level to a user based on a confidence metric associated with an identification of the individual is schematically depicted.

[0029] Figure 6 A flow chart illustrating an example method according to implementations disclosed herein is depicted.

[0030] Figure 7 A flow chart illustrating an example method according to implementations disclosed herein is depicted.

[0031] Figure 8 An example architecture for a computing device is shown. DETAILED DESCRIPTION

[0032] Now go to Figure 1 , illustrates an example environment in which the techniques disclosed herein may be implemented. The example environment includes one or more client computing devices 106. Each client device 106 may execute a respective instance of an automated assistant client 108, which may also be referred to herein as the "client portion" of the automated assistant. One or more cloud-based automated assistant components 119 (which may also be referred to herein as the "server portion" of the automated assistant) may be implemented on one or more computing systems (collectively referred to as "cloud" computing systems), wherein the one or more computing systems are communicatively coupled to the client devices 106 via one or more local area networks and / or wide area networks (e.g., the Internet), generally indicated at 115.

[0033] In various embodiments, an instance of automated assistant client 108, through its interaction with one or more cloud-based automated assistant components 119, can form what appears to be a logical instance of automated assistant 120 from the user's perspective, and the user can participate in a human-computer dialogue with automated assistant 120. Such an instance of automated assistant 120 is Figure 1 108 is shown as a dashed line in FIG. 106. Thus, it should be understood that each user who engages with an automated assistant client 108 executing on a client device 106 may actually engage with his or her own logical instance of automated assistant 120. For the sake of brevity and simplicity, the "automated assistant (term)" used herein to "serve" a particular (e.g., host) user will refer to the combination of an automated assistant client 108 executing on a client device 106 operated by the user and one or more cloud-based automated assistant components 119 (which may be shared among multiple automated assistant clients 108). It should also be understood that in some embodiments, automated assistant 120 may respond to requests from any user, regardless of whether the user is "served" by a particular instance of the automated assistant 120.

[0034] The one or more client devices 106 may include, for example, one or more of a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a stand-alone interactive speaker (which in some cases may include a visual sensor), a smart home appliance such as a smart TV (or a standard TV equipped with a network dongle with automated assistant functionality), and / or a user's wearable device including a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided. Some client devices 106, such as a stand-alone interactive speaker (or "smart speaker"), may take the form of an assistant device that is primarily designed to facilitate a conversation between a user and an automated assistant 120. Some such assistant devices may take the form of a stand-alone interactive speaker with a display, which may or may not be a touch screen.

[0035] In some embodiments, the client device 106 may be equipped with one or more visual sensors 107 having one or more fields of view, but this is not required. The one or more visual sensors 107 may take various forms, such as digital cameras, passive infrared ("PIR") sensors, stereo cameras, RGBd cameras, and the like. The one or more visual sensors 107 may be used, for example, by an image capture module 111 to capture image frames (still images or videos) of the environment in which the client device 106 is deployed. These image frames may then be analyzed, for example, by a visual cue module 1121 to detect user-provided visual cues contained in the image frames. These visual cues may include, but are not limited to, gestures, gazes at specific reference points, facial expressions, predefined movements of the user, and the like. These detected visual cues may be used for various purposes, such as invoking the automated assistant 120 and / or causing the automated assistant 120 to take various actions.

[0036] Additionally or alternatively, in some implementations, the client device 106 may include one or more proximity sensors 105. The one or more proximity sensors may take various forms, such as a passive infrared ("PIR") sensor, a radio frequency identification ("RFID"), a component that receives a signal transmitted from another nearby electronic component (e.g., a Bluetooth signal from a nearby user's client device, a high or low frequency sound transmitted from the device, etc.), etc. Additionally or alternatively, the visual sensor 107 and / or the microphone 109 may also be used as a proximity sensor, for example, by visually and / or audibly detecting the user's proximity.

[0037] As described in more detail herein, the automated assistant 120 participates in a human-computer conversation session with one or more users via user interface input and output devices of one or more client devices 106. In some embodiments, the automated assistant 120 may participate in a human-computer conversation session with a user in response to a user interface input provided by a user via one or more user interface input devices of one of the client devices 106. In some of those embodiments, the user interface input is explicitly directed to the automated assistant 120. For example, a user may verbally provide (e.g., type, speak) a predetermined invocation phrase, such as "OK, Assistant" or "Hey, Assistant", to cause the automated assistant 120 to actively begin listening to or monitoring typed text. Additionally or alternatively, in some embodiments, the automated assistant 120 may be invoked based on one or more detected visual cues, alone or in combination with a verbal invocation phrase.

[0038] In many embodiments, the automated assistant 120 can utilize speech recognition to convert speech from the user into text, and therefore, for example, respond to the text by providing search results, general information, and / or taking one or more response actions (e.g., playing media, starting a game, ordering food, etc.). In some embodiments, the automated assistant 120 can additionally or alternatively respond to speech without converting speech to text. For example, the automated assistant 120 can convert speech input into embedding, into entity representation (indicating one or more entities present in the speech input) and / or other "non-text" representations, and operate on such non-text representations. Therefore, the embodiments described herein as operating based on text converted from speech input can additionally and / or alternatively directly operate on speech input and / or other non-text representations of speech input.

[0039] Each of the client computing device 106 and the computing device operating the cloud-based automated assistant component 119 may include one or more memories for storing data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. The operations performed by the client computing device 106 and / or the automated assistant 120 may be distributed across multiple computer systems. The automated assistant 120 may be implemented as a computer program running on one or more computers in one or more locations coupled to each other over a network, for example.

[0040] As described above, in various embodiments, client computing device 106 may operate automated assistant client 108 or the “client portion” of automated assistant 120. In various embodiments, automated assistant client 108 may include voice capture module 110, image capture module 111 described above, visual prompt module 1121, and / or invocation module 113. In other embodiments, one or more aspects of voice capture module 110, image capture module 111, visual prompt module 112, and / or invocation module 113 may be implemented separately from automated assistant client 108, for example, via one or more cloud-based automated assistant components 119. For example, in Figure 1 In the example, there is also a cloud-based visual cue module 1122 that can detect visual cues in the image data.

[0041] In various embodiments, the voice capture module 110, which can be implemented using any combination of hardware and software, can interface with hardware such as microphone 109 to capture an audio recording of the user's speech. Various types of processing can be performed on this audio recording for various purposes. In some embodiments, the image capture module 111, which can be implemented using any combination of hardware or software, can be configured to interface with the camera 107 to capture one or more image frames (e.g., digital photos) corresponding to the field of view of the visual sensor 107.

[0042] In various embodiments, the visual cue module 1121 (and / or cloud-based visual cue module 1122) may be implemented using any combination of hardware or software and may be configured to analyze one or more image frames provided by the image capture module 111 to detect one or more visual cues captured in and / or across the one or more image frames. The visual cue module 1121 may employ a variety of techniques to detect visual cues. For example, the visual cue module 1122 may use one or more artificial intelligence (or machine learning) models that are trained to generate outputs indicative of user-provided visual cues detected in the image frames.

[0043] The voice capture module 110 can be configured to capture the user's voice, for example, via the microphone 109, as described above. Additionally or alternatively, in some embodiments, the voice capture module 110 can be further configured to convert the captured audio into text and / or other representations or embeddings, for example, using speech-to-text ("STT") processing technology. Additionally or alternatively, in some embodiments, the voice capture module 110 can be configured to convert the text into computer-synthesized speech, for example, using one or more speech synthesizers. However, in some cases, because the client device 106 may be relatively limited in computing resources (e.g., processor cycles, memory, battery, etc.), the voice capture module 110 local to the client device 106 can be configured to convert a limited number of different spoken phrases (especially phrases that invoke the automated assistant 120) into text (or other forms, such as lower-dimensional embeddings). Other voice inputs can be sent to a cloud-based automated assistant component 119, which can include a cloud-based text-to-speech conversion ("TTS") module 116 and / or a cloud-based STT module 117.

[0044] In various embodiments, the invocation module 113 may be configured to determine whether to invoke the automated assistant 120, for example, based on output provided by the voice capture module 110 and / or the visual cue module 1121 (which may be combined with the image capture module 111 in a single module in some embodiments). For example, the invocation module 113 may determine whether the user's utterance meets the invocation phrase that should initiate a human-machine dialogue session with the automated assistant 120. In some embodiments, the invocation module 113 may analyze data indicating the user's utterance, such as an audio recording or a feature vector (e.g., embedding) extracted from the audio recording, alone or in combination with one or more visual cues detected by the visual cue module 1121. In some embodiments, the threshold used by the invocation module 113 to determine whether to invoke the automated assistant 120 in response to the voiced utterance may be lowered when a particular visual cue is also detected or when the speaker is identified as a registered user. Therefore, even when a user provides a spoken utterance that is different from the correct invocation phrase "OKassistant" but phonetically similar to a certain extent, the utterance can still be accepted as the correct invocation phrase when detected in conjunction with visual cues (e.g., the speaker waving, the speaker looking directly at the visual sensor 107, etc.) or detected based on a registered user.

[0045] In some embodiments, the invocation module 113 can use one or more on-device invocation models stored, for example, in the on-device model database 114 to determine whether the utterance and / or visual cue qualifies as an invocation. Such on-device invocation models can be trained to detect variations in invocation phrases / gestures. For example, in some embodiments, an on-device invocation model (e.g., one or more neural networks) can be trained using training examples, each training example comprising an audio recording (or extracted feature vectors) of an utterance from a user and data indicating one or more image frames and / or detected visual cues captured simultaneously with the utterance.

[0046] exist Figure 1 In the embodiment, the on-device model database 114 can store one or more on-device call models 1141-114 NIn some embodiments, default on-device invocation model 1141 can be trained to detect one or more default invocation phrases or hot words, such as those previously mentioned (e.g., "OK, Assistant," "Hey, Assistant," etc.), in an audio recording or other data indicative thereof. In some such embodiments, these models can be always available and can be used to transition automated assistant 120 to a general listening state, in which any audio recording captured by voice capture module 110 (at least for a period of time after the invocation) can be processed using other components of automated assistant 120 as described below (e.g., on client device 106 or through one or more cloud-based automated assistant components 119).

[0047] Additionally, in some implementations, the on-device model database 114 may at least temporarily store one or more additional “context call models” 1142-114 N These scenarios call for models 1142-114 N The contextual invocation models 1142-1144 may be used and / or available (eg, activated) by the invocation module 113 in certain contexts (such as when a registered user is detected). N The context-specific hotwords may be trained to detect one or more context-specific hotwords, for example, in an audio recording or other data indicative thereof. In some embodiments, the context-specific hotwords 1142-114 N Can be selectively downloaded as needed from, for example, a dynamic hotword engine 128 forming part of a cloud-based automated assistant component 119, which will be described in more detail below. In some embodiments, each contextual invocation model 114 can be available to a different registered user (identifying a speaker as a registered user can constitute a "context" as used herein). Thus, each registered user can customize their own library of invocation phrases.

[0048] In various embodiments, when the calling module 113 uses the context calling model 1142-114 N When various dynamic hot words are detected, it can convert the automated assistant 120 to the general listening state described above. Additionally or alternatively, the call module 113 can convert the automated assistant 120 to a context-specific state, wherein one or more context-specific response actions are performed with or without converting the automated assistant 120 to the general listening state. In many cases, the audio data that triggers the automated assistant 120 to transition to a context-specific state may not be transmitted to the cloud. Instead, one or more context-specific response actions can be performed entirely on the client device 106, which can reduce response time and the amount of information transmitted to the cloud, which can be beneficial from a privacy perspective.

[0049] The cloud-based TTS module 116 can be configured to utilize the virtually unlimited resources of the cloud to convert text data (e.g., natural language responses formulated by the automated assistant 120) into computer-generated speech output. In some implementations, the TTS module 116 can provide the computer-generated speech output to the client device 106 for direct output, for example, using one or more speakers. In other implementations, the text data (e.g., natural language responses) generated by the automated assistant 120 can be provided to the speech capture module 110, which can then convert the text data into computer-generated speech that is output locally.

[0050] The cloud-based STT module 117 can be configured to utilize the virtually unlimited resources of the cloud to convert the audio data captured by the voice capture module 110 into text, which can then be provided to the intent matcher 135. In some embodiments, the cloud-based STT module 117 can convert the audio recording of the voice into one or more phonemes, and then convert the one or more phonemes into text. Additionally or alternatively, in some embodiments, the STT module 117 can employ a state decoding diagram. In some embodiments, the STT module 117 can generate multiple candidate text interpretations of the user's utterance. In some embodiments, the STT module 117 can weight or bias a particular candidate text interpretation higher than other candidate text interpretations depending on whether there are visual cues detected at the same time.

[0051] The automated assistant 120 (particularly the cloud-based automated assistant component 119) may include an intent matcher 135, the aforementioned TTS module 116, the aforementioned STT module 117, and other components described in more detail below. In some embodiments, one or more modules and / or modules of the automated assistant 120 may be omitted, combined, and / or implemented in a component separate from the automated assistant 120. In some embodiments, to protect privacy, one or more components of the automated assistant 120 (such as the natural language processor 122, the TTS module 116, the STT module 117, etc.) may be implemented at least partially on the client device 106 (e.g., excluded from the cloud).

[0052] In some embodiments, the automated assistant 120 generates response content in response to various inputs generated by a user of one of the client devices 106 during a human-machine conversation session with the automated assistant 120. The automated assistant 120 may provide the response content (e.g., over one or more networks when separate from the user's client device) for presentation to the user as part of the conversation session. For example, the automated assistant 120 may generate response content in response to free-form natural language input provided via the client device 106. As used herein, free-form input (also referred to as "free-form natural language input" or "natural language input") is input formulated by a user and is not limited to a set of options presented to the user for selection.

[0053] As used herein, a "conversation session" may include a logically self-consistent conversation of one or more messages between a user and automated assistant 120 (and, in some cases, other human participants). Automated assistant 120 may distinguish between multiple conversation sessions with a user based on various signals, such as the passage of time between sessions, changes in user context (e.g., location, before / during / after a scheduled meeting, etc.) between sessions, detection of one or more intervening interactions between the user and a client device other than the conversation between the user and the automated assistant (e.g., the user switches apps for a while, the user leaves, and then later returns to a standalone voice-activated product), locking / hibernation of client devices between sessions, and changes in client devices used to interface with one or more instances of automated assistant 120, etc.

[0054] The intent matcher 135 may be configured to determine the user's intent based on input provided by the user (e.g., vocal utterances, visual cues, etc.) and / or based on other signals (e.g., sensor signals and online signals (e.g., data obtained from a Web service), etc.). In some embodiments, the intent matcher 135 may include a natural language processor 122 and the cloud-based visual cue module 1122 described above. In various embodiments, the cloud-based visual cue module 1122 may operate similarly to the visual cue module 1121, except that the cloud-based visual cue module 1122 may have more resources at its disposal. In particular, the cloud-based visual cue module 1122 may detect visual cues that may be used by the intent matcher 135 alone or in combination with other signals to determine the user's intent. In some embodiments in which the speaker is identified as a registered user, the evolving context of an ongoing conversation between the registered user and the automated assistant 120 may be utilized to respond to the user's natural language input. For example, the natural language input of the registered user may be automatically disambiguated based on the conversation history between the registered user and the automated assistant 120. This conversation history may be unavailable to other registered users or to other users entirely.

[0055] Natural language processor 122 may be configured to process natural language input generated by a user via client device 106, and may generate annotation output (e.g., in text form) for use by one or more other components of automated assistant 120. For example, natural language processor 122 may process natural language free-form input generated by a user via one or more user interface input devices of client device 106. The generated annotation output includes one or more annotations of the natural language input and / or one or more (e.g., all) terms in the natural language input.

[0056] In some embodiments, the natural language processor 122 is configured to recognize and annotate various types of grammatical information in natural language input. For example, the natural language processor 122 may include a morphological module that can separate individual words into morphemes and / or annotate these morphemes, for example, with the category of morphemes. The natural language processor 122 may also include a part-of-speech tagger that is configured to annotate words with the grammatical role of the words. For example, the part-of-speech tagger can mark each word with the part of speech of the word (such as "noun", "verb", "adjective", "pronoun", etc.). In addition, for example, in some embodiments, the natural language processor 122 may additionally and / or alternatively include a dependency parser (not shown) configured to determine the syntactic relationship between words in the natural language input. For example, the dependency parser can determine which words modify other words and the subject and verb of the sentence, etc. (e.g., parse tree), and can annotate such dependencies.

[0057] In some implementations, the natural language processor 122 may additionally and / or alternatively include a coreference resolver (not shown) that is configured to group or "cluster" references to the same entity based on one or more contextual cues. For example, a coreference resolver may be used to resolve the term "there" in the natural language input "I liked Hypothetical Café last time we ate there" to "Hypothetical Café".

[0058] In some embodiments, one or more components of the natural language processor 122 can rely on annotations from one or more other components of the natural language processor 122. For example, in some embodiments, a named entity tagger can rely on annotations from a coreference resolver and / or a dependency resolver to annotate all mentions of a particular entity. Similarly, for example, in some embodiments, a coreference resolver may rely on annotations from a dependency resolver to cluster references to the same entity. In some embodiments, when processing a particular natural language input, one or more components of the natural language processor 122 can determine one or more annotations using relevant previous inputs and / or other relevant data outside of the particular natural language input.

[0059] The intent matcher 135 can use various techniques to determine the user's intent, for example, based on the output from the natural language processor 122 (which can include annotations and words of the natural language input) and / or based on the output from the visual prompt module (e.g., 1121 and / or 1122). In some embodiments, the intent matcher 135 can access one or more databases (not shown) that include, for example, multiple mappings between grammars, visual prompts, and response actions (or more generally, intents). In many cases, these grammars may be selected and / or learned over time and may represent the most common intents of the user. For example, a grammar "play <artist>(play <artist>)" can be mapped to an intent that invokes the following response action, which causes the <artist>The music of (<artist>) is played on the client device 106 operated by the user. Another syntax "[weather|forecast] today" can match the user query, such as "what's the weather today" and "what's the forecast for today?"

[0060] In addition to or in lieu of grammars, in some embodiments, intent matcher 135 may employ one or more trained machine learning models, either alone or in combination with one or more grammars and / or visual cues. These trained machine learning models may also be stored in one or more databases and may be trained to recognize intent by, for example, embedding data indicative of user utterances and / or any detected user-provided visual cues into a reduced dimensionality space and then determining which other embeddings (and therefore, intents) are closest, for example using techniques such as Euclidean distance, cosine similarity, etc.

[0061] As in "play <artist>As seen in the example grammars, some grammars have slots that can be filled with slot values ​​(or "parameters") (e.g. <artist>). Slot values ​​can be determined in a variety of ways. Users typically provide slot values ​​on their own initiative. For example, for the syntax "Order me a <topping>pizza (order for me <topping>pizza), the user might say "order me a sausage pizza", in which case the autofill slot <topping>Additionally or alternatively, if the user invokes a syntax that includes slots to be filled with slot values, without the user actively providing the slot values, automated assistant 120 can request those slot values ​​from the user (e.g., "what type of crust do you want on your pizza?"). In some implementations, automated assistant 120 can request the slot values ​​based on visual prompt module 112. 1-2 The detected visual cue fills the slot with the slot value. For example, the user can say something like "Order me this many cat bowls" while holding up three fingers toward the visual sensor 107 of the client device 106. Or, the user can say something like "Find me more movies like this" while holding up a DVD box of a particular movie.

[0062] In some embodiments, the automated assistant 120 can act as an intermediary between a user and one or more third-party computing services 130 (or "third party agent" or "agents"). These third-party computing services 130 can be independent software processes that receive input and provide responsive output. Some third-party computing services can take the form of third-party applications that may or may not be running on a computing system separate from the computing system on which the cloud-based automated assistant component 119 is running. One user intent that the intent matcher 135 can recognize is to engage with a third-party computing service 130. For example, the automated assistant 120 can provide access to an application programming interface ("API") to a service for controlling smart devices. A user can invoke the automated assistant 120 and provide a command such as "I'd like to turn the heating on." The intent matcher 135 can map this command to a grammar that triggers the automated assistant 120 to engage with the third-party service to turn on the user's heating. Third-party service 130 may provide automated assistant 120 with a list of minimum slots that need to be filled in order to implement (or "resolve") a command to turn on the heat. In this example, the slots may include the temperature to which the heat is to be set and the duration for which the heat is to be turned on. Automated assistant 120 may generate and provide to the user (via client device 106) natural language output requesting the parameters of the slots.

[0063] The fulfillment module 124 can be configured to receive the predicted / estimated intent and associated slot value (whether actively provided by the user or requested from the user) output by the intent matcher 135 and fulfill (or "resolve") the intent. In various embodiments, the fulfillment (or "resolution") of the user's intent can, for example, generate / obtain various fulfillment information (also referred to as "response" information or "resolution information") through the fulfillment module 124. As described below, in some embodiments, the fulfillment information can be provided to a natural language generator (shown as "NLG" in some figures) 126, which can generate a natural language output based on the fulfillment information.

[0064] Fulfillment (or "resolution") information can take a variety of forms, as intent can be fulfilled (or "resolved") in a variety of ways. Assuming a user asks for pure information, such as "Where were the outdoor shots of 'The Shining' filmed?" the user's intent can be determined as a search query, for example, by intent matcher 135. The intent and content of the search query can be provided to fulfillment module 124, such as Figure 1 As shown in , the fulfillment module can communicate with one or more search modules 150 configured to search a corpus of documents and / or other data sources (e.g., a knowledge graph, etc.) to obtain response information. The fulfillment module 124 can provide data indicating the search query (e.g., the text of the query, a reduced dimension embedding, etc.) to the search module 150. The search module 150 can provide: response information, such as GPS coordinates; or other more specific information, such as "Timberline Lodge, Mt. Hood, Oregon." The response information can form part of the fulfillment information generated by the fulfillment module 124.

[0065] Additionally or alternatively, fulfillment module 124 may be configured to receive the user's intent and any slot values ​​provided by the user or determined using other means (e.g., the user's GPS coordinates, user preferences, etc.), for example, from intent matcher 135, and trigger a response action. The response action may include, for example, ordering goods / services, starting a timer, setting a reminder, initiating a phone call, playing media, sending a message, etc. In some such embodiments, the fulfillment information may include slot values ​​associated with the fulfillment, a confirmation response (which in some cases may be selected from predetermined responses), etc.

[0066] The natural language generator 126 can be configured to generate and / or select natural language output (e.g., words / phrases designed to mimic human speech) based on data obtained from various sources. In some embodiments, the natural language generator 126 can be configured to receive as input fulfillment information associated with the fulfillment of an intent and generate natural language output based on the fulfillment information. Additionally or alternatively, the natural language generator 126 can receive information (e.g., required slots) from other sources such as third-party applications, which can be used to compose natural language output for a user.

[0067] In some embodiments, registration engine 132 can be configured to perform selected aspects of the present disclosure relating to conversation-based registration of individual users for unimodal and / or multimodal identification by an automated assistant, and specific responses to requests of a particular user based on the fact that the particular user has been registered and / or identified. In some embodiments, registration engine 132 can be operably coupled to a registration database 133 that stores information about registered users that can be used to authenticate registered users by automated assistant 120.

[0068] The registration database 133 may include various information about registered users, such as visual attributes and / or auditory attributes that distinguish registered users. For example, the registration database 133 may store, for example, visual and / or voice profiles of registered users. In some embodiments, the registration database 133 may be hosted in the cloud, for example, on the same or different computing system as the cloud-based automatic assistant component 119. In some such embodiments, the information stored in the registration database 133 may be associated with a specific client device and / or associated with a specific user with which the registered user is registered. Additionally or alternatively, in some embodiments, the registration database 133 (and in some cases, all or part of the registration engine 132) may be hosted on the client device 106, for example so that users can be registered and / or registered users can be authenticated using resources local to the client device 106, rather than cloud-based resources.

[0069] Registration database 133 (or another database) may also store what will be referred to herein as "historical interaction data." This information can be used by registration engine 132 to determine whether to request that a particular individual be registered with automated assistant 120 using the techniques described herein. Historical interaction data may include information indicating past interactions between automated assistant 120 and different individuals. In some implementations, historical interaction data may include data indicating distinguishing attributes of these individuals, such as a temporary voice profile and / or a visual profile.

[0070] By maintaining this information in registration database 133 over time, registration engine 132 is able to determine that a heretofore unidentified individual has satisfied one or more "auto-attendant registration criteria." The auto-attendant registration criteria may include, for example, a threshold number of different human-machine conversational sessions that the individual has conducted with the auto-attendant 120 on the same client device 106 or on one or more client devices in an ecosystem of client devices. Additionally or alternatively, the registration criteria for the auto-attendant may include a threshold number of conversational turns occurring between the individual and the auto-attendant.

[0071] Once it is determined that a particular speaker meets the automated assistant registration criteria, the automated assistant 120 may initiate a routine referred to herein as a "human-machine conversation registration routine." During the human-machine conversation registration routine, the automated assistant 120 may provide natural language output that includes instructions for causing the user to perform various actions, such as by capturing and storing data indicative of distinguishing attributes of the user to facilitate future identification of the user. For example, the automated assistant 120 may instruct the individual to move their head into various poses so that the registration engine 132 may, for example, use images captured by the camera 107 to establish a "visual" profile of the individual. Additionally or alternatively, the automated assistant 120 may instruct the individual to speak one or more words or phrases, such as using audio data captured by the microphone 109, which the registration engine 132 may use to establish an "auditory" profile of the individual. These visual and / or auditory profiles may be stored in the registration database 133 and used in the future to authenticate the speaker.

[0072] In various embodiments, the registration engine 132 may be configured to process one or more sensor signals generated by one or more hardware sensors (e.g., 105-109) integrated with the client device 106. Based on the processing, the registration engine 132 may be configured to identify one or more distinguishing attributes of the user within the range of the one or more hardware sensors. Based on the one or more distinguishing attributes, the registration engine 132 may analyze historical interaction data (e.g., stored in the registration database 133) to identify one or more previous human-computer dialogue sessions in which the user exchanged dialogues with the automated assistant 120. For example, in some embodiments, the registration database 133 may include voice profile and / or visual profile information collected from previous interactions with an unknown user. This voice profile and / or visual profile information may include, for example, an audio recording captured by the microphone 109 during the last interaction with the unknown user, one or more digital images of the unknown user captured using the camera 107, a combination of both, or one or more embeddings generated from the audio recording and / or digital image.

[0073] Based on the identified one or more previous human-computer dialogue sessions, the registration engine 132 can determine that the user meets the automated assistant registration criteria. For example, in some embodiments, the automated assistant registration criteria may include that the user has more than one, more than two, or more than n contacts with the automated assistant (n is a positive integer). Additionally or alternatively, the automated assistant registration criteria may include that the unknown user has contacted the automated assistant more than a certain threshold number of times during a predetermined time interval, such as during a week, ten days, a month, a quarter, a day, an hour, etc. Additionally or alternatively, the automated assistant registration criteria may include, for example, that the unknown user has a certain threshold number of dialogue turns with the automated assistant 120 in a single dialogue session and / or across multiple dialogue sessions. In some such embodiments, if the voice and / or face of a specific unknown user is not detected again within a predetermined time interval, it can be erased (e.g., from the registration database 133). And in some embodiments, all or part of the registration database 133 may be stored only on the client device 106, for example, to protect the privacy of the unknown user.

[0074] In various embodiments, once registration engine 132 determines that a user meets automated assistant registration criteria, registration engine 132 may trigger and / or engage in a human-machine dialog registration routine. During the human-machine dialog registration routine, the automated assistant may request a heretofore unknown user to register with the automated assistant 120, for example, using audio output and / or visual output. In some embodiments, registration of a user may include storing the user's identity (which may be provided by the user in some cases) in association with data indicating one or more distinguishing attributes of the user (e.g., a voice profile and / or a visual profile) in one or more databases (e.g., 133). In some embodiments, a newly registered user may be matched with a pre-existing online profile, for example, which the newly registered user uses to engage with his or her own instance of the automated assistant 120.

[0075] In various embodiments, the registration of a user by the registration engine 132 may unlock one or more features of the automated assistant 120 that were not available to the user prior to registration. For example, a registered user may be able to invoke the automated assistant 120 using one or more hot words in one or more contextual invocation models 114. In some such embodiments, a newly registered user may be able to customize their own library of hot words that they may use to invoke the automated assistant in the future. As another example, various trust levels may be provided to a registered user, for example, by a controlling or "host" user who controls / owns the client device 106. For example, an unknown user may be able to engage with the automated assistant 120 using the client device 106, but may not be able to cause the automated assistant 120 to perform various actions, such as accessing a calendar and / or other documents controlled by the host user, or controlling smart appliances such as smart lights and / or thermostats. In contrast, a registered user may be placed in one or more trust levels, each of which may enable the registered user to gain access to various resources and / or features, such as those just mentioned.

[0076] Figure 2 Schematically depicts an example state machine according to various embodiments that may be implemented by an automated assistant (e.g., 120) and / or an assistant device (e.g., 106) configured with selected aspects of the present disclosure. At the upper left is a "default inactive state" in which the automated assistant 120 may reside when not being engaged by a user. In the default inactive state, one or more microphones of one or more client devices (106) may be activated and audio data captured thereby may be analyzed using the techniques described herein. In response to detecting one or more default call words (e.g., by the call module 113 and / or the visual cue module 112 based on the default call model 1141, for example Figure 2 If an utterance other than the default hotword is heard (e.g., “DIW” in the voice prompt, also referred to herein as a “hotword”) (e.g., “OK, Assistant” or “Hey, Assistant”), automated assistant 120 may transition to a “General Listening State”. Utterances other than the default hotword (e.g., ambient dialogue, etc.) may be ignored and not processed.

[0077] In the general listening state, the automated assistant 120 can capture the audio data spoken after the default call word and transition to the "General Processing" state. In the general processing state, the automated assistant 120 can process the data indicating the audio input, as previously described with reference to Figure 1 The above includes STT processing, natural language processing, intent matching, fulfillment, etc. Once the processing is completed, the automated assistant 120 may transition back to the default inactive state. If no audio input is received after the default call word is detected, the timeout ( Figure 2 ) can transition the automated assistant 120 from the general listening state back to a default inactive state, e.g., so that subsequent utterances not intended for processing by the automated assistant are not captured or processed.

[0078] As previously mentioned, the techniques described herein facilitate context-specific hotwords that can be activated and detected to transition the automated assistant 120 to various different states, such as a general listening state, or other context-specific states in which the automated assistant 120 performs various actions. In some implementations, the vocabulary of invocation words that can be spoken to transition the automated assistant 120 from a default inactive state to a general listening state can be expanded, at least temporarily (e.g., for a limited amount of time, until the context no longer applies, etc.), in certain circumstances.

[0079] For example, in Figure 2 In some embodiments, a first context-specific signal CS1 can transition the automated assistant 120 from a default inactive state to a first context-specific listening state "CSLS1". In CSLS1, the automated assistant 120 can listen for a default invocation word ("DIW") and a first context-specific hotword ("C1 hotword"). If either is detected, the automated assistant 120 can transition to a general listening state as described above. Thus, in the first context-specific listening state, the vocabulary of hotwords that transition the automated assistant 120 to the general listening state is expanded to include the default invocation word and the first context-specific hotword. Moreover, in some embodiments, if a sufficient amount of time has passed without an activated hotword being detected while the automated assistant 120 is in the first context-specific listening state, a timeout ("TO") can transition the automated assistant 120 back to the default inactive state.

[0080] Additionally or alternatively, in some embodiments, in certain contexts, automated assistant 120 may be able to transition to a general listening state, for example using an expanded vocabulary of hot words, or to a context-specific state in which one or more context-specific actions may be performed. Figure 2 In response to a second context signal ("CS2"), automated assistant 120 may transition from a default inactive state to a second context-specific listening state "CSLS2". In this second context-specific listening state, automated assistant 120 may, for example, detect one or more default call words and / or in some cases detect one or more second context-specific hotwords ("C2hotword(s)"). A (C2 Hot Words A )") to transition to a general listening state, wherein the one or more second context-specific hot words effectively expand the vocabulary capable of transitioning the automated assistant 120 to the general listening state.

[0081] Additionally or alternatively, automated assistant 120 may transition from the second context-specific state ("CSLS2") to one or more states in which, for example, in response to one or more additional second context-specific hotwords ("C2hotword(s)") B (C2 Hot Words B )") to perform one or more second context-specific response actions ("2 nd In some implementations, a specific second context-specific hotword may be mapped to a specific second context-specific response action, but this is not required. Although not described in detail for clarity, Figure 2 , but in some implementations, after performing these one or more second context-specific response actions, automated assistant 120 can transition back to a default inactive state.

[0082] In some implementations, in certain contexts, the automated assistant 120 may no longer listen for default hotwords. Instead, the automated assistant 120 may only listen for context-specific hotwords and perform responsive actions. For example, in Figure 2 In response to the Mth context signal ("CS M ”), the automated assistant 120 can switch from the default inactive state to the Mth context-specific listening state (M is a positive integer) “CSLSM”. In this state, the automated assistant 120 can listen to the Mth context-specific hotword (“C M In response to detecting one or more Mth context-specific hot words, automated assistant 120 may perform one or more Mth context-specific response actions ("Mth context-specific resp. act.").

[0083] In various implementations, automated assistant 120 can activate context-specific hotwords in various ways. For example, and with reference to Figure 1 and Figure 2 In some embodiments, when switching to a specific context, the automated assistant 120 may download one or more context-specific machine learning models or classifiers (e.g., 1142, 1143, ..., 1144) from the dynamic hotword engine 128, for example. N ), such as neural networks, hidden Markov models, etc., which are pre-trained to detect hot words to be activated in this specific context. For example, assume that in a specific context, the vocabulary that converts the automated assistant 120 from a default inactive state to a general listening state is expanded to include the word "howdy". In various embodiments, the automated assistant 120 can obtain a classifier, for example from a database 129 available to the dynamic hot word engine 128, that is trained to generate an output indicating whether the word "howdy" is detected. In various embodiments, the classifier can be binary (e.g., output "1" if a hot word is detected, otherwise output "0"), or a probability can be generated. If the probability meets a certain confidence threshold, the hot word can be detected.

[0084] Figure 3A and Figure 3B 1 illustrates how an unknown user 101 can communicate with an automated assistant instance ( Figure 3A An example of a human-machine conversation session occurs between a client computing device 306 (not shown in FIG. B) via a microphone and speaker (depicted as a separate interactive speaker, but this is not meant to be limiting) of the client computing device 306. One or more aspects of the automated assistant 120 may be implemented on the computing device 306 and / or on one or more computing devices in network communication with the computing device 306. The client device 306 includes a camera 307 and may also include a microphone (in FIG. 3A to FIG. 3B Not depicted, see Figure 1 ).

[0085] exist Figure 3A , user 101 invokes automated assistant 120 using a default invocation phrase, "Hey, Assistant." The user then provides spoken natural language input, "What is the capital of Kentucky?" Because this request seeks non-sensitive public information, automated assistant 120 performs a search and responds, "The capital of Kentucky is Frankfort." Next, automated assistant 120 determines, for example, via registration engine 132, whether unknown user 101 has met automated assistant registration criteria. For example, automated assistant 120 may compare an audio recording of what user 101 said with one or more voice profiles stored in registration database 133 in association with one or more unknown speakers. Additionally or alternatively, automated assistant 120 may compare one or more digital images captured by camera 307 with one or more visual profiles (e.g., digital images, embeddings thereof, etc.) stored in registration engine 133 in association with one or more unknown speakers.

[0086] In this example, automated assistant 120 determines that user 101 has met automated assistant criteria (e.g., because this is the nth time that user 101 has been detected engaging with automated assistant 120). Therefore, the automated assistant requests the user to participate in the registration routine, "...I see you're a regular here. Would you like to enroll with me to unlock various features?..." User 101 responds affirmatively.

[0087] exist Figure 3B , the registration routine begins. The automated assistant 120 prompts the user as follows: "Repeat after me: 'Sally sells seashells on the seashore". The user's audible response provides audio data to the registration engine 132, which can be analyzed to form a robust voice profile that the automated assistant 120 can use to authenticate the user 101 in the future.

[0088] Next, the automated assistant 120 prompts the user ("Great. Now look right at me...good, now look to your right...good, now to your left.... "). As the user 101 cooperates, the automated assistant 120 operates the camera 307 to capture a series of digital images 360. 1-3 Each of these digital images 360 captures the head of user 101 from a different angle, as a result of audio instructions being provided to user 101 to look in different directions. Using these multiple images, registration engine 132 is able to build a robust visual profile that automated assistant 120 can use moving forward to authenticate the now-registered user 101.

[0089] Figure 4A and Figure 4B Another example scenario is depicted that illustrates what happens when a user is identified as a registered user. Figure 4A , user 101A is not recognized by automated assistant 120, which is executed at least in part on client device 406. Although client device 406 is depicted as a standalone interactive speaker, this is not meant to be limiting. Client device 406 again includes camera (visual sensor) 407 and microphone (not depicted).

[0090] exist Figure 4A , an unknown user 101A invokes automated assistant 120 with the invocation phrase "Hey Assistant" and then provides the natural language input "Please open the garage door". Because user 101 is not recognized, automated assistant 120 determines, for example, through registration engine 132, that the unknown user is not allowed to operate the smart garage door. Therefore, automated assistant 120 responds, "I'm sorry, I'm afraid I'm not authorized to do that for unknown users".

[0091] On the contrary, Figure 4B In the example, the registered user 101B is identified based on one or more of her visual and / or audible inputs matching a corresponding visual profile and / or voice profile in the registration database 133. When the registered and identified user 101B makes the same request, the automated assistant 120 follows and responds “OK, opening the garage door”.

[0092] Even though facial and voice recognition technology continues to evolve, it is still not perfect. The appearance of a user may change over time. Teenage users may experience voice changes, while other users may change their voices due to illness, etc. Moreover, in order to prevent some assistant devices from becoming too expensive, relatively cheap cameras, microphones, and other components may be used, which are not as reliable as their more expensive counterparts. Therefore, through the technology described herein, in some embodiments, users can be partially identified, or they can be identified with a measure of confidence that falls below absolute certainty. In various embodiments, rather than simply placing partially identified users in a fully trusted mode or a fully untrusted mode, partially identified users can be placed in one or more intermediate trust levels (or boxes), where they can be provided with access to some (but not necessarily all) features that would otherwise be provided to fully identified users.

[0093] Figure 5 An example of how a user can be placed in one of a plurality of trust levels based on how reliably the user is identified is schematically shown. Figure 5 , a client device 506A configured with selected aspects of the present disclosure includes a camera 107, a microphone 109, and one or more wireless receivers 566. As previously described, the one or more wireless receivers 566 may be implemented with a variety of wireless communication technologies, including but not limited to Wi-Fi, Bluetooth, ZigBee, Z-wave, infrared communication, radio frequency (e.g., RFID), etc. User 101 may also bring his or her own client device 506B, which may be connected to a wireless network. Figure 5 In the embodiment, the user may carry a smart phone, although this is not meant to be limiting. For example, the user may carry other computing devices, such as smart glasses, smart watches, etc.

[0094] exist Figure 5 , client device 506A may detect distinguishing features of user 101 in three domains. Camera 107 may capture one or more digital images. Microphone 109 may capture audio data. And wireless receiver 566 may capture wireless signals transmitted by client device 506B, which include, for example, data that can be used to identify the identity of user 101. Based on the one or more distinguishing attributes, registration engine 132 may determine in which of a plurality of trust levels user 101 should be placed.

[0095] exist Figure 5 There are N (positive integer) trust levels. The highest trust level N gives the user virtually unfettered access to features provided by automated assistant 120, such as controlling home appliances (e.g., lights, thermostats, smart locks, smart garage door openers), accessing protected documents (e.g., calendars or other personal documents of the host user), and accessing third-party applications (e.g., Figure 1 150 in ). Full access to third-party applications may allow user 101 to enable automated assistant 120 to make purchases (e.g., order a pizza, book a rideshare, etc.) from a third-party application without requiring user 101 to provide payment information each time. The highest level of trust also provides users with access to an enhanced library of hotwords, which may allow trusted users to invoke automated assistant 120 using hotwords other than the default hotwords (e.g., reference Figure 2 described above).

[0096] The next highest trust level, N-1, may provide user 101 with fewer features of automated assistant 120 than the highest trust level, N, but may still allow user 101 to perform some actions. For example, user 101 may have limited control over appliances, such that user 101 may, for example, adjust lights (which may not be considered particularly sensitive), but may not adjust more sensitive appliances, such as a smart garage or smart locks. User 101 may also have limited access to documents controlled by the host user, such as read-only access, or have access only to documents of low sensitivity. User 101 may also be able to engage automated assistant 120 to get in touch with third-party applications, but may be required to provide payment information before authorizing a purchase.

[0097] As indicated by the ellipses, there may be any number of trust levels. At the bottom, there is a lowest trust level ("Guest"), which may be provided to a user 101 who is barely identifiable or unidentifiable based on user 101's distinguishing attributes. In the lowest trust level, user 101 may not be able to control appliances, may not be able to access any documents controlled by the host user, or may not be able to engage automated assistant 120 with third-party applications. Also, user 101 may only be able to invoke automated assistant 120 using default hot words. However, user 101 may still be able to engage automated assistant 120 to perform various non-sensitive operations, such as answering general questions, performing Internet searches, and the like. In various embodiments, an authorized user, such as a host user, may be able to freely adjust the trust level such that various trust levels are unlocked (except Guest). Figure 5 Various features other than the configuration shown in .

[0098] As an example, assume that one or more digital images captured by camera 107 and wireless signals detected by wireless receiver 566 are analyzed by registration engine 132 to identify user 101 with a very high confidence level. Further assume that audio data captured by microphone 109 is analyzed by registration engine 132 to identify user 101 with a slightly lower confidence level. Figure 5 , this scenario is represented by the top arrow from the registration engine 132 to the trust level N. Three confidence metrics (1, 0.7, 1) corresponding to visual match, voice match, and wireless detection, respectively, are used to map the user 101 to the trust level N. In this example, the confidence metrics fall between 0 and 1, but this is not meant to be limiting.

[0099] As another example, assume that one or more digital images captured by camera 107 yield a slightly lower confidence metric of 0.8, audio data captured by microphone 109 yields a confidence metric of 0.6, and wireless receiver 566 detects no wireless signal. Figure 5 , this scenario is represented by the middle arrow from registration engine 132 to trust level N-1. Three confidence metrics (0.8, 0.6, 0.0) corresponding to visual match, voice match, and wireless detection, respectively, are used to map user 101 to trust level N-1. Because no wireless signal is detected, and the other confidence metrics are slightly lower than the previous example, registration engine 132 places user 101 in trust level N-1, which means that user 101 has less access to the features of automated assistant 120 than in the previous example.

[0100] As yet another example, assume that one or more digital images captured by camera 107 yield a very low confidence metric of 0.2, audio data captured by microphone 109 also yields a very low confidence metric of 0.1, and wireless receiver 566 detects no wireless signal. Figure 5 , this scenario is represented by the bottom arrow from registration engine 132 to trust level N-1. Because no wireless signals are detected and other confidence metrics are very low, registration engine 132 places user 101 in trust level 0, which means that user 101 is considered a guest and can have access to a minimum number of features of automated assistant 120.

[0101] Figure 5 The examples are not intended to be limiting, but rather illustrate one possible way to assign users into bins or trust levels and provide appropriate access to features based on these assignments. For example, a host user may not allow all registered users to access the highest trust level. However, the host user may only allow automatically registered users to access, for example, the second highest level of trust. The host user may need to provide explicit instructions to allow any registered user to be assigned to the highest trust level. Alternatively, the highest level of trust may be maintained only for the host user.

[0102] Figure 6 600 according to embodiments disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include various components of various computer systems, such as one or more components of automated assistant 120. Furthermore, although the operations of method 600 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0103] At block 602, the system may operate automated assistant 120 at least partially on a computing device (e.g., client device 106, 306, 406, 506). For example, and as described above, in many cases, automated assistant 120 may be implemented partially on client device 106 and partially on the cloud (e.g., cloud-based automated assistant component 119). At block 604, the system may monitor hardware sensor signals generated by sensors (such as camera 107, microphone 109, proximity sensor 105, and / or wireless receiver 566), for example, as evidence of a user who may intend to engage with automated assistant 120.

[0104] If at block 606, the system detects one or more hardware sensor signals, then at block 608, the system, for example via the registration engine 132, may process the detected hardware signals to identify distinguishing attributes of the user that triggered the hardware sensor signals. As previously described, these distinguishing attributes may include aspects of the user's physical appearance, voice, and / or information provided by wireless signals transmitted by a mobile device carried by the user.

[0105] If no distinguishing features are identified at block 610, the method 600 returns to block 604. However, if distinguishing features are identified at block 610, the method 600 may proceed to block 612. At block 612, the system, such as via the registration engine 132, may analyze historical interaction data, such as historical interaction data stored in the registration database 133 or elsewhere (e.g., in local memory of the client device 106, 306, 406, 506). For example, the registration engine 132 may compare the distinguishing attributes identified at blocks 608-610 with distinguishing attributes captured from unknown users in the past.

[0106] At block 614, a determination may be made as to whether the distinguishing attributes identified at blocks 608-610 match any attributes stored in registration database 133. If the answer is no, at block 616, the system may record the distinguishing attributes identified at blocks 608-610, for example, in registration database 133, for example, so that these distinguishing attributes may be used in the future to determine whether to register an unknown user. However, if the answer at block 614 is yes, at block 618, a determination may be made as to whether the automated assistant registration criteria are met. For example, a determination may be made as to whether the user has participated in a threshold number of conversation sessions with automated assistant 120, or whether the user has participated in a threshold number of conversation turns with automated assistant 120.

[0107] If the answer at block 618 is no, then method 600 may proceed to block 617. At block 617, the system may increment a count associated with the matching distinguishing feature. For example, if the unknown user has been encountered once before, and the threshold for automatic registration is three different conversation sessions, then the count associated with the distinguishing feature may be incremented to two. Method 600 may then return to block 604. However, if the answer at block 618 is yes, then automated assistant 120 may engage in a human-machine conversation registration routine similar to Figure 3A and Figure 3B as well as Figure 7 Thereafter, the user may now be considered registered.

[0108] Figure 7 An example method 700 for practicing an automated assistant registration routine according to various embodiments is depicted. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include various components of various computer systems. Furthermore, although the operations of method 700 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0109] At box 702, the system may request the user to speak one or more words or phrases. At box 704, the system may generate a voice profile based on audio data generated from the spoken one or more words or phrases of box 702. At box 706, the system may instruct the user to reposition the user's face into multiple poses, for example, so that multiple digital images can be captured with a visual sensor at box 708. At box 710, the system may generate a visual profile based on the captured digital images. For example, the system may train one or more convolutional neural networks based on the captured digital images. At box 712, the system may store the identity of the user in association with the voice profile and / or the visual profile, for example, in the registration database 133.

[0110] Where certain embodiments discussed herein may collect or use personal information about a user (e.g., user data extracted from other electronic communications, information about a user's social network, a user's location, a user's time, a user's biometric information, and a user's activities and demographic information, relationships between users, etc.), the user is provided with one or more opportunities to control whether information is collected, whether personal information is stored, whether personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use a user's personal information only after receiving explicit authorization from the relevant user.

[0111] For example, a user is provided with control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. One or more options are presented to each user whose personal information is to be collected to allow control over the collection of information related to that user, providing permission or authorization as to whether information is collected and which portions of the information are collected. For example, one or more such control options may be provided to a user over a communications network. In addition, certain data may be processed in one or more ways before it is stored or used so as to remove personally identifiable information. As an example, the identity of a user may be processed so that personally identifiable information cannot be determined. As another example, a user's geographic location may be generalized to a larger area so that the user's specific location cannot be determined.

[0112] Figure 8 810, which can optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client computing device, the user-controlled resource engine 134, and / or other components can include one or more components of the example computing device 810.

[0113] The computing device 810 typically includes at least one processor 814 that communicates with a plurality of peripheral devices via a bus subsystem 812. These peripheral devices may include: a storage subsystem 824, which includes, for example, a memory subsystem 825 and a file storage subsystem 826; a user interface output device 820; a user interface input device 822; and a network interface subsystem 816. The input and output devices allow a user to interact with the computing device 810. The network interface subsystem 816 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0114] User interface input devices 822 may include: a keyboard; a pointing device (such as a mouse, trackball, touchpad, or graphics tablet); a scanner; a touch screen incorporated into a display; an audio input device (such as a voice recognition system, a microphone); and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computing device 810 or into a communication network.

[0115] User interface output devices 820 may include a display subsystem, a printer, a fax machine, or a non-visual display (such as an audio output device). The display subsystem may include: a cathode ray tube (CRT); a flat panel device (such as a liquid crystal display (LCD)); a projection device; or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways to output information from the computing device 810 to a user or to another machine or computing device.

[0116] The storage subsystem 824 stores programs and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 824 may include an executable Figure 6 to Figure 7 Selected aspects of the method and implementation Figure 1 The logic of the various components is depicted in .

[0117] These software modules are typically executed by the processor 814 alone or in combination with other processors. The memory 825 used in the storage subsystem 824 may include multiple memories, including a main random access memory (RAM) 830 for storing instructions and data during program execution and a read-only memory (ROM) 832 in which fixed instructions are stored. The file storage subsystem 826 can provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain embodiments may be stored in the storage subsystem 824 by the file storage subsystem 826, or in other machines that can be accessed by the processor 814.

[0118] The bus subsystem 812 provides a mechanism for the various components and subsystems of the computing device 810 to communicate with each other as intended. Although the bus subsystem 812 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0119] The computing device 810 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, for the purpose of illustrating some embodiments, Figure 8 The description of computing device 810 depicted in FIG. 8 is intended only as a specific example. Many other configurations of computing device 810 may have more Figure 8 The computing devices shown may have greater or fewer components.

[0120] Although several embodiments have been described and shown herein, a variety of other means and / or structures for performing functions and / or obtaining results and / or one or more of the advantages described herein may be utilized, and each such variation and / or modification is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the specific application for which this teaching is used. Using only routine experiments, those skilled in the art will recognize or be able to determine many equivalents of the specific embodiments described herein. Therefore, it should be understood that the foregoing embodiments are given only by way of example, and within the scope of the attached claims and their equivalents, the embodiments may be implemented in a manner different from that specifically described and claimed. The embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, if such features, systems, articles, materials, kits, and / or methods are not mutually contradictory, the scope of the present disclosure includes any combination of two or more such features, systems, articles, materials, kits, and / or methods.< / topping> < / topping> < / topping> < / artist> < / artist> < / artist> < / artist>

Claims

1. A method implemented by one or more processors, the method comprising: requesting a user of the computing device to speak one or more words or phrases; for each of the one or more words or phrases, capturing corresponding audio data, the corresponding audio data capturing the user speech, wherein the corresponding audio data is generated via one or more microphones of the computing device; generating a voice profile of the user based on the corresponding audio data; instructing the user to position the user's face into a plurality of poses; for each of the plurality of gestures, capturing corresponding visual data of the user's face, wherein the corresponding visual data is generated via one or more visual components of the computing device; generating a visual profile of the user based on the corresponding visual data; storing both the voice profile of the user and the visual profile of the user in association with the user; and After storing both the voice profile of the user and the visual profile of the user in association with the user: The voice profile of the user and / or the visual profile of the user is caused to be used to authenticate the user to a computing resource at the computing device.

2. The method according to claim 1, further comprising: requesting an additional user of the computing device to speak the one or more words or phrases; for each of the one or more words or phrases, capturing additional corresponding audio data, the additional corresponding audio data capturing the additional user utterance, wherein the additional corresponding audio data is generated via the one or more microphones of the computing device; generating an additional voice profile for the additional user based on the additional corresponding audio data; instructing the additional user to position the additional user's face into the plurality of poses; for each of the plurality of gestures, capturing additional corresponding visual data of the additional user's face, wherein the additional corresponding visual data is generated via the one or more visual components of the computing device; generating an additional visual profile for the additional user based on the additional corresponding visual data; storing both the additional voice profile of the additional user and the additional visual profile of the additional user in association with the additional user; and After storing both the additional voice profile of the additional user and the additional visual profile of the additional user in association with the additional user: The additional voice profile of the additional user and / or the additional visual profile of the additional user is caused to be used to authenticate the additional user to the computing resource and / or additional computing resources at the computing device.

3. The method according to claim 2, wherein: The user is associated with a first trust level, wherein the additional user is associated with a trust permission level, and wherein the first trust level is different than the second trust level.

4. The method according to any one of the preceding claims, further comprising: Prior to requesting the user of the computing device to speak one or more of the words or phrases: A registration routine is initiated to register the user with the automated attendant.

5. The method according to claim 4, wherein: Initiating the registration routine to register the user with the automated attendant is responsive to determining: The user has participated in a threshold number of different human-machine conversation sessions with the automated assistant; or The user has engaged in a threshold number of conversation turns with the automated assistant.

6. A method according to any one of the preceding claims, wherein: The voice profile of the user includes audio features extracted from the corresponding audio data, the audio features distinguishing the user from other users.

7. A method according to any one of the preceding claims, wherein: Each of the plurality of gestures corresponds to a different gesture, and wherein the corresponding visual data for each of the plurality of gestures captures the user's face in a different gesture.

8. The method according to claim 7, wherein: The visual profile of the user comprises visual features extracted from the corresponding visual data, the visual features distinguishing the user from other users.

9. A method according to any one of the preceding claims, wherein: The speech profile is a speech embedding, which is a non-text representation of the user's speech.

10. A computing device comprising: one or more processors; as well as A memory storing instructions which, when executed by the one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 9.

11. A non-transitory computer-readable storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations according to any one of claims 1 to 9.

Images