Cross-device data synchronization based on simultaneous hot word triggering

Abstract

This document describes technologies for cross-device data synchronization. A method includes: implementing a first instance of an automated assistant at least in part on a first computing device; receiving audio data capturing a spoken utterance; processing the audio data using a machine learning model to generate a prediction output indicating a probability of one or more hot words present in the audio data; determining that the prediction output satisfies a threshold; in response to determining that the prediction output satisfies the threshold, performing arbitration with at least one other computing device implementing at least one other instance of the automated assistant at least in part; and in response to performing arbitration with the at least one other computing device, initiating synchronization of user data or configuration data between the first instance of the automated assistant and the at least one other instance of the automated assistant.

Description

Background Art

[0001] Humans can engage in human-computer conversations using interactive software applications, referred to herein as "automated assistants" (also referred to herein as "digital agents," "interactive personal assistants," "intelligent personal assistants," "assistant applications," "dialog agents," and the like). For example, humans (who may be referred to as "users" when they interact with the automated assistant) can provide commands and / or requests to the automated assistant using spoken natural language input (i.e., utterances), which in some cases can be converted to text and then processed by providing textual (e.g., typed) natural language input and / or by touch and / or non-verbal physical movement (e.g., gestures, eye gaze, facial movement, and the like). The automated assistant responds to the requests by providing responsive user interface output (e.g., auditory and / or visual user interface output), controlling one or more smart devices, and / or controlling one or more functions of the device implementing the automated assistant (e.g., controlling other applications of the device).

[0002] As described above, many automated assistants are configured to interact via spoken utterances. To protect user privacy and / or conserve resources, the automated assistant avoids executing one or more automated assistant functions based on all spoken utterances present in audio data detected by a microphone of a client device implementing (at least in part) the automated assistant. Instead, specific processing based on spoken utterances occurs only in response to determining that a specific condition exists.

[0003] For example, many client devices including automated assistants and / or connected to automated assistant interfaces include a hotword detection model. When the microphone of such a client device is not disabled, the client device can use the hotword detection model to continuously process audio data detected via the microphone to generate a prediction output indicating whether one or more hotwords (including multi-word phrases), such as "Hey Assistant," "Okay Assistant," and / or "Assistant." When the prediction output indicates the presence of a hotword, any audio data that follows within a threshold amount of time (and optionally determined to include voice activity) can be processed by one or more on-device and / or remote automated assistant components, such as a speech recognition component, a speech activity detection component, and the like. In addition, a natural language understanding engine can be used to process the recognized text (from the speech recognition component), and / or actions can be performed based on the natural language understanding engine output. For example, an action can include generating and providing a response and / or controlling one or more applications and / or smart devices. However, when the prediction output indicates the absence of a hotword, the corresponding audio data will be discarded without any further processing, thereby saving resources and protecting user privacy.

[0004] One or more components of these automated assistants may run on edge (client) devices to provide increased privacy and / or reduced latency. However, as automated assistants become increasingly decentralized and spread across various client devices, the data used by the automated assistants (e.g., for personalization, speaker recognition, etc.) becomes more isolated. In the case of using an automated assistant without any cloud-based component (e.g., without a cloud-based account), the data used by the automated assistant running on the user's various client devices may not be available or may be out of date on one or more of the client devices. Summary of the Invention

[0005] Some implementations disclosed herein relate to improving the performance of machine learning models, improving device performance and capabilities, and improving the user experience provided by devices by synchronizing data across devices based on simultaneous hotword triggers. As described in more detail herein, cross-device data synchronization is provided so that user data is appropriately synchronized between client devices associated with a particular user. In some implementations, such synchronization can help automated assistants provide a consistent experience to users when they interact with different devices. In other implementations, such synchronization can provide data synchronization between one or more existing devices of a user and a newly purchased device of the user.

[0006] In some implementations, cross-device data synchronization can be performed in a setting where there are multiple hotword-enabled devices (e.g., an automated assistant) that are not centrally connected (e.g., one or more devices are not connected to the internet or have an account or connection with a cloud service, or one device has an account and the others do not, or the devices have accounts with different cloud providers). In some implementations, a local determination is made as to whether a user has multiple devices that can be synchronized.

[0007] In some implementations, cross-device data synchronization can be performed in settings where devices are connected to a public cloud account, but there is sensitive data (e.g., speaker recognition models) that is not stored in the cloud. In some implementations, synchronization is triggered based on interaction with the automated assistant device using standard hot words to wake the automated assistant device from a low-power state. In some implementations, when the hot word event is triggered, other devices that can handle queries from the same user are discovered. In some implementations, on-device data, including personalized models, preferences, and the like, can be seamlessly shared across the user's devices in a fully on-device scenario, allowing a consistent experience across all of the user's devices and enabling data to be synchronized in a peer-to-peer manner. In some implementations, the cross-device data synchronization process can be used to simplify the setup process for new devices that are not tied to any existing cloud account.

[0008] In various implementations, a first instance of an automated assistant may be at least partially implemented in an inactive state on a first computing device operated by a user; while in the inactive state, audio data capturing spoken utterances of the user may be received via one or more microphones of the first computing device; the audio data may be processed using a machine learning model to generate a prediction output indicating a probability that one or more hot words are present in the audio data; a determination may be made that the prediction output satisfies a threshold indicating that one or more hot words are present in the audio data; in response to determining that the prediction output satisfies the threshold, arbitration may be performed with at least one other computing device that at least partially implements at least one other instance of the automated assistant; and in response to performing arbitration with the at least one other computing device, synchronization of user data or configuration data may be initiated between the first instance of the automated assistant on the first computing device and at least one other instance of the automated assistant on the at least one other computing device, the user data including data based on one or more interactions with the user at the first computing device, the one or more interactions occurring prior to receiving the audio data.

[0009] In some implementations, before initiating synchronization of user data, it can be determined that permission has been received from the user to perform synchronization of user data between a first instance of an automated assistant on a first computing device and at least one other instance of an automated assistant on at least one other computing device.

[0010] In some implementations, prior to initiating synchronization of user data, a match can be determined between registered user profiles stored on each of the first computing device and the at least one other computing device.

[0011] In some implementations, before initiating synchronization of the user data, a match can be determined between attributes of the user identified as a joint secret and stored on each of the first computing device and the at least one other computing device. In some implementations, the joint secret can be based on the user's date of birth, the user's email address, the user's phone number, or the user's preferences.

[0012] In some implementations, the synchronized user data may include a personalized hotword model, a personalized speech recognition model, or a personalized query interpretation model. In some implementations, the synchronized user data may include the user's most recent queries. In some implementations, the synchronized user data may include the user's preferences. In some implementations, the synchronized user data may include the user's private knowledge graph. In some implementations, the user data may include data selected for synchronization based on the capabilities of each of the first computing device and at least one other computing device. In some implementations, the synchronized configuration data may include settings of the first computing device or settings of at least one other computing device.

[0013] In some additional or alternative implementations, a first instance of the automated assistant may be implemented at least partially in an inactive state on a first computing device operated by a user, and a second instance of the automated assistant may be implemented at least partially in an inactive state on a second computing device operated by the user; when in the inactive state, first audio data may be received via one or more microphones of the first computing device, and second audio data may be received via one or more microphones of the second computing device, the first audio data and the second audio data each capturing a spoken utterance of the user; the first audio data received via the one or more microphones of the first computing device may be processed at the first computing device to generate a first prediction output indicating a probability of one or more synchronized hotwords being present in the first audio data received via the one or more microphones of the first computing device; the second audio data received via the one or more microphones of the second computing device may be processed at the second computing device to generate a second prediction output indicating a probability of one or more synchronized hotwords being present in the first audio data received via the one or more microphones of the second computing device. The invention also provides a method for determining a probability of one or more synchronization hot words present in the second audio data received; determining that the first prediction output satisfies a first threshold indicating that the one or more synchronization hot words are present in the first audio data, and determining that the second prediction output satisfies a second threshold indicating that the one or more synchronization hot words are present in the second audio data; in response to determining that the first prediction output satisfies the first threshold and the second prediction output satisfies the second threshold, arbitration can be performed between the first instance of the automatic assistant on the first computing device and the second instance of the automatic assistant on the second computing device; and in response to performing arbitration between the first instance of the automatic assistant on the first computing device and the second instance of the automatic assistant on the second computing device, synchronization of user data or configuration data can be initiated between the first instance of the automatic assistant on the first computing device and the second instance of the automatic assistant on the second computing device, the user data including data based on one or more interactions with the user at the first computing device before receiving the first audio data, and data based on one or more interactions with the user at the second computing device before receiving the second audio data.

[0014] In some implementations, in response to determining that the first predicted output satisfies a first threshold and the second predicted output satisfies a second threshold, a first instance of the automated assistant on the first computing device and a second instance of the automated assistant on the second computing device can be transitioned to an active state.

[0015] In some implementations, before initiating synchronization of user data, it can be determined that permission has been received from the user to perform synchronization of user data between a first instance of an automated assistant on a first computing device and a second instance of an automated assistant on a second computing device.

[0016] In some implementations, before initiating synchronization of user data, a match can be determined between registered user profiles stored on each of the first computing device and the second computing device. In some implementations, before initiating synchronization of user data, a match can be determined between attributes of the user identified as a joint secret and stored on each of the first computing device and the second computing device.

[0017] In some additional or alternative implementations, a first instance of an automated assistant may be at least partially implemented in an inactive state on a first computing device operated by a user; while in the inactive state, audio data capturing spoken utterances of the user may be received via one or more microphones of the first computing device; the audio data may be processed using a machine learning model to generate a prediction output indicating a probability that one or more hot words are present in the audio data; a determination may be made that the prediction output satisfies a threshold indicating that one or more hot words are present in the audio data; in response to determining that the prediction output satisfies the threshold, at least one other computing device that at least partially implements at least one other instance of the automated assistant may be identified, the at least one other computing device also having received the audio data; and in response to identifying the at least one other computing device, synchronization of user data or configuration data may be initiated between the first instance of the automated assistant on the first computing device and at least one other instance of the automated assistant on the at least one other computing device, the user data including data based on one or more interactions with the user at the first computing device prior to receiving the audio data.

[0018] The above description is provided as an overview of some implementations of the present disclosure. Further descriptions of those implementations and other implementations are described in more detail below.

[0019] Various implementations may include a non-transitory computer-readable storage medium storing instructions that are executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and / or a tensor processing unit (TPU)) to perform a method such as one or more of the methods described above. Other implementations may include an automated assistant client device (e.g., a client device that includes at least an automated assistant interface for interfacing with a cloud-based automated assistant component) that includes a processor that is operable to execute stored instructions to perform a method such as one or more of the methods described herein. Other implementations may include a system of one or more servers that include one or more processors that are operable to execute stored instructions to perform a method such as one or more of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1A and Figure 1B Depicted are example process flows according to various implementations, demonstrating aspects of the present disclosure.

[0021] Figure 2 A block diagram depicting the example environment, including Figure 1A and Figure 1B Various components of the present invention may be implemented in which the implementation disclosed herein may be implemented.

[0022] Figure 3 A flow chart illustrating an example method for cross-device data synchronization based on simultaneous hotword triggers is depicted.

[0023] Figure 4 A flow chart illustrating an example method for cross-device data synchronization based on simultaneous hotword triggers is depicted.

[0024] Figure 5 An example architecture for a computing device is depicted. DETAILED DESCRIPTION

[0025] Figure 1A and Figure 1B An example process flow illustrating various aspects of the present disclosure is depicted. The client device 110 Figure 1A , and includes a Figure 1A Components within the box of . The machine learning engine 122A can receive audio data 101 corresponding to spoken words detected via one or more microphones of the client device 110 and / or other sensor data 102 corresponding to speechless physical movements (e.g., gestures and / or movements, body postures and / or body movements, eye gaze, facial movements, mouth movements, etc.) detected via one or more non-microphone sensor components of the client device 110. The one or more non-microphone sensors can include a camera or other visual sensor, a proximity sensor, a pressure sensor, an accelerometer, a magnetometer, and / or other sensors. The machine learning engine 122A processes the audio data 101 and / or other sensor data 102 using a machine learning model 152A to generate a prediction output 103. As described herein, the machine learning engine 122A can be a hot word detection engine 122B or an alternative engine, such as a voice activity detector (VAD) engine, an endpoint detector engine, and / or other engine.

[0026] In some implementations, when the machine learning engine 122A generates the prediction output 103, it can be stored locally on the client device in the on-device storage 111 and optionally associated with the corresponding audio data 101 and / or other sensor data 102. In some versions of those implementations, the prediction output can be retrieved by the gradient engine 126 for use in generating the gradient 106 at a later time, such as when one or more conditions described herein are met. The on-device memory 111 can include, for example, read-only memory (ROM) and / or random access memory (RAM). In other implementations, the prediction output 103 can be provided to the gradient engine 126 in real time.

[0027] Client device 110 may determine whether to initiate a currently dormant automated assistant function (e.g., Figure 2 The automated assistant 295 may include: avoiding initiating a currently dormant automated assistant function and / or shutting down a currently active automated assistant function using the assistant activation engine 124. The automated assistant function may include: generating speech recognition of recognized text, generating a natural language understanding (NLU) output of NLU, generating a response based on the recognized text and / or the NLU output, transmitting audio data to a remote server, and / or transmitting the recognized text to a remote server. For example, assuming that the predicted output 103 is a probability (e.g., 0.80 or 0.90) and the threshold at box 182 is a threshold probability (e.g., 0.85), if the client device 110 determines at box 182 that the predicted output 103 (e.g., 0.90) meets the threshold (e.g., 0.85), the assistant activation engine 124 may initiate the currently dormant automated assistant function.

[0028] In some implementations, and as Figure 1B , the machine learning engine 122A may be a hotword detection engine 122B. It is noteworthy that various automated assistant functions, such as the on-device speech recognizer 142, the on-device NLU engine 144, and / or the on-device fulfillment engine 146, are currently dormant (i.e., as indicated by the dashed lines). Furthermore, it is assumed that the predicted output 103 generated using the hotword detection model 152B and based on the audio data 101 satisfies the threshold at block 182, and that the voice activity detector 128 detects user speech directed to the client device 110.

[0029] In some versions of these implementations, assistant activation engine 124 activates on-device speech recognizer 142, on-device NLU engine 144, and / or on-device fulfillment engine 146 as currently dormant automated assistant functionality. For example, on-device speech recognizer 142 can use on-device speech recognition model 142A to process audio data 101 for a spoken utterance including the hotword "good assistant" and an additional command and / or phrase following the hotword "good assistant" to generate recognized text 143A, on-device NLU engine 144 can use on-device NLU model 144A to process recognized text 143A to generate NLU data 145A, on-device fulfillment engine 146 can use on-device fulfillment model 146A to process NLU data 145A to generate fulfillment data 147A, and client device 110 can use fulfillment data 147A in performing 150 one or more actions in response to audio data 101.

[0030] In other versions of these implementations, assistant activation engine 124 activates a sole on-device fulfillment engine 146, without activating on-device speech recognizer 142 and on-device NLU engine 144, to process various commands, such as “no,” “stop,” “cancel,” and / or other commands that can be processed without on-device speech recognizer 142 and on-device NLU engine 144. For example, on-device fulfillment engine 146 processes audio data 101 using on-device fulfillment model 146A to generate fulfillment data 147A, and client device 110 can use fulfillment data 147A in performing 150 one or more actions in response to audio data 101. Furthermore, in versions of these implementations, the assistant activation engine 124 may initially activate a currently dormant automatic function to verify that the decision made at block 182 is correct (e.g., that the audio data 101 in fact includes the hotword “good assistant”) by initially activating only the on-device speech recognizer 142 to determine that the audio data 101 includes the hotword “good assistant”, and / or the assistant activation engine 124 may transmit the audio data 101 to one or more servers (e.g., the remote server 160) to verify that the decision made at block 182 is correct (e.g., that the audio data 101 in fact includes the hotword “good assistant”).

[0031] Return to Steering Figure 1AIf, at block 182, client device 110 determines that predicted output 103 (e.g., 0.80) fails to satisfy a threshold value (e.g., 0.85), assistant activation engine 124 may refrain from initiating currently dormant automated assistant functionality and / or shut down any currently active automated assistant functionality. Furthermore, if, at block 182, client device 110 determines that predicted output 103 (e.g., 0.80) fails to satisfy a threshold value (e.g., 0.85), at block 184 client device 110 may determine whether to receive further user interface input. For example, the further user interface input may be an additional spoken utterance including a hotword, an additional speechless physical movement acting as a proxy for the hotword, an actuation of an explicit automated assistant invocation button (e.g., a hardware button or a software button), a sensed "squeeze" of client device 110 (e.g., invoking the automated assistant when client device 110 is squeezed with at least a threshold amount of force), and / or other explicit automated assistant invocations. If, at block 184 , the client device 110 determines that no further user interface input has been received, the client device 110 may stop recognizing corrections and end at block 190 .

[0032] However, if the client device 110 determines at block 184 that further user interface input has been received, then at block 186 the system may determine whether the further user interface input received at block 184 includes a correction (e.g., a user-mediated or user-provided correction) that contradicts the decision made at block 182. If the client device 110 determines at block 186 that the further user interface input received at block 184 does not include a correction, then the client device 110 may stop identifying the correction and end at block 190. However, if the client device 110 determines at block 186 that the further user interface input received at block 184 includes a correction that contradicts the initial decision made at block 182, then the client device 110 may determine the ground truth output 105.

[0033] In some implementations, gradient engine 126 can generate gradient 106 based on predicted output 103 relative to ground truth output 105. For example, gradient engine 126 can generate gradient 106 based on comparing predicted output 103 to ground truth output 105. In some versions of those implementations, client device 110 stores predicted output 103 and corresponding ground truth output 105 locally in on-device storage 111, and gradient engine 126 retrieves predicted output 103 and corresponding ground truth output 105 to generate gradient 106 when one or more conditions are met. The one or more conditions can include, for example, that the client device is charging, that the client device has at least a threshold charge state, that the temperature of the client device (based on one or more on-device temperature sensors) is below a threshold, and / or that the client device is not being held by a user. In other versions of those implementations, client device 110 provides predicted output 103 and ground truth output 105 to gradient engine 126 in real time, and gradient engine 126 generates gradient 106 in real time.

[0034] In addition, the gradient engine 126 can provide the generated gradients 106 to the on-device machine learning training engine 132A. Upon receiving the gradients 106, the on-device machine learning training engine 132A uses the gradients 106 to update the on-device machine learning model 152A. For example, the on-device machine learning training engine 132A can utilize backpropagation and / or other techniques to update the on-device machine learning model 152A. It should be noted that in some implementations, the on-device machine learning training engine 132A can utilize batch processing techniques to update the on-device machine learning model 152A based on the gradients 106 and additional gradients determined locally on the client device 110 based on the additional corrections.

[0035] In addition, client device 110 can transmit the generated gradients 106 to remote system 160. When remote system 160 receives gradients 106, remote training engine 162 of remote system 160 uses gradients 106 and additional gradients 107 from additional client devices 170 to update the global weights of global speech recognition model 152A1. Additional gradients 107 from additional client devices 170 can each be generated based on the same or similar techniques as described above with respect to gradients 106 (but based on corrections to the local recognition specific to those client devices).

[0036] The update distribution engine 164 can provide updated global weights and / or the updated global speech recognition model itself to the client device 110 and / or other client devices in response to one or more conditions being met, as indicated by 108. The one or more conditions can include, for example, a threshold duration and / or amount of training since the updated weights and / or updated speech recognition model were last provided. The one or more conditions can additionally or alternatively include, for example, a measured improvement in the updated speech recognition model and / or a threshold duration of time since the updated weights and / or updated speech recognition model were last provided. When the updated weights are provided to the client device 110, the client device 110 can replace the weights of the on-device machine learning model 152A with the updated weights. When the updated global speech recognition model is provided to the client device 110, the client device 110 can replace the on-device machine learning model 152A with the updated global speech recognition model.

[0037] In some implementations, the on-device machine learning model 152A is transmitted (e.g., by the remote system 160 or other component) for storage and use at the client device 110 based on the geographic region and / or other attributes of the client device 110 and / or the user of the client device 110. For example, the on-device machine learning model 152A can be one of N available machine learning models for a given language, but can be trained based on corrections specific to a particular geographic region and provided to the client device 110 based on the client device 110 being primarily located in that particular geographic region.

[0038] Now turn Figure 2 , in an implementation, a client device 110 is shown, wherein Figure 1A and Figure 1B The machine learning engines on various devices are included as part of (or in communication with) the automated assistant client 240. The corresponding machine learning models are also shown with Figure 1A and Figure 1B For simplicity, Figure 2 Not shown in Figure 1A and Figure 1B other components. Figure 2 Shows how the automated assistant client 240 utilizes Figure 1A and Figure 1B An example of various on-device machine learning engines and their corresponding machine learning models to perform various actions.

[0039] Figure 2The client device 110 in FIG. 1 is shown as having one or more microphones 211, one or more speakers 212, one or more cameras and / or other visual components 213, and a display 214 (e.g., a touch-sensitive display). The client device 110 may also include pressure sensors, proximity sensors, accelerometers, magnetometers, and / or other sensors for generating other sensor data in addition to the audio data captured by the one or more microphones 211. The client device 110 at least optionally implements an automated assistant client 240. Figure 2 In the example of FIG, automated assistant client 240 includes on-device hotword detection engine 122B, on-device speech recognizer 142, on-device natural language understanding (NLU) engine 144, and on-device fulfillment engine 146. Automated assistant client 240 also includes voice capture engine 242 and visual capture engine 244. Automated assistant client 140 can include additional and / or alternative engines, such as a voice activity detector (VAD) engine, an endpoint detector engine, and / or other engines.

[0040] One or more cloud-based automated assistant components 280 may optionally be implemented on one or more computing systems (collectively, "cloud" computing systems) that are communicatively coupled to client device 110 via one or more local area networks and / or wide area networks (e.g., the Internet), indicated generally at 290. Cloud-based automated assistant components 280 may be implemented, for example, via a cluster of high-performance servers.

[0041] In various implementations, an instance of automated assistant client 240, by virtue of its interaction with one or more cloud-based automated assistant components 280, can form what appears, from the user's perspective, to be a logical instance of automated assistant 295 that the user can use to engage in human-computer interactions (e.g., verbal interactions, gesture-based interactions, and / or touch-based interactions).

[0042] The client device 110 may be, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device in a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker, a smart home appliance such as a smart TV (or a standard TV equipped with a network-connected dongle with automated assistant capabilities), and / or a user's wearable device that includes a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client devices may be provided.

[0043] The one or more visual components 213 can take various forms, such as a single-image camera, a stereo camera, a LIDAR component (or other laser-based component), a radar component, etc. The one or more visual components 213 can be used, for example, by the visual capture engine 242 to capture visual frames (e.g., image frames, laser-based visual frames) of the environment in which the client device 110 is deployed. In some implementations, such visual frames can be used to determine whether a user is present in the vicinity of the client device 110 and / or the distance of the user (e.g., the user's face) relative to the client device 110. Such determination can be used, for example, to determine whether to activate a user interface. Figure 2 The various on-device machine learning engines depicted, and / or other engines.

[0044] The voice capture engine 242 can be configured to capture user voice and / or other audio data captured via the microphone 211. In addition, the client device 110 may include a pressure sensor, a proximity sensor, an accelerometer, a magnetometer, and / or other sensors for generating other sensor data in addition to the audio data captured via the microphone 211. As described herein, such audio data and other sensor data can be used by the hot word detection engine 122B and / or other engines to determine whether to initiate one or more currently dormant automatic assistant functions, avoid initiating one or more currently dormant automatic assistant functions, and / or shut down one or more currently active automatic assistant functions. The automatic assistant function may include an on-device speech recognizer 142, an on-device NLU engine 144, an on-device fulfillment engine 146, and additional and / or alternative engines. For example, the on-device speech recognizer 142 can utilize the on-device speech recognition model 142A to process audio data that captures spoken utterances to generate recognized text 143A corresponding to the spoken utterances. The on-device NLU engine 144 optionally utilizes an on-device NLU model 144A to perform on-device natural language understanding on the recognized text 143A to generate NLU data 145A. The NLU data 145A may, for example, include an intent corresponding to the spoken utterance and optionally parameters (e.g., slot values) for the intent. Furthermore, the on-device fulfillment engine 146 generates fulfillment data 147A based on the NLU data 145A, optionally utilizing an on-device fulfillment model 146A. The fulfillment data 147A may define local and / or remote responses (e.g., answers) to the spoken utterance, interactions with locally installed applications based on the spoken utterance, commands transmitted to Internet of Things (IoT) devices (directly or via corresponding remote systems) based on the spoken utterance, and / or other resolution actions performed based on the spoken utterance. The fulfillment data 147A is then provided for local and / or remote execution / implementation of the determined actions to resolve the spoken utterance. Execution may include, for example, rendering local and / or remote responses (e.g., visual and / or auditory rendering (optionally utilizing a local text-to-speech module)), interacting with locally installed applications, transmitting commands to IoT devices, and / or other actions.

[0045] Display 214 may be used to display recognized text 143A and / or further recognized text 143B from on-device speech recognizer 122, and / or one or more results from implementation 150. Display 214 may also be one of the user interface output components through which the visual portion of a response from automated assistant client 240 is rendered.

[0046] In some implementations, the cloud-based automated assistant component 280 may include a remote ASR engine 281 that performs speech recognition, a remote NLU engine 282 that performs natural language understanding, and / or a remote fulfillment engine 283 that generates fulfillment. A remote fulfillment module may also optionally be included that performs remote fulfillment based on fulfillment data determined locally or remotely. Additional and / or alternative remote engines may be included. As described herein, in various implementations, on-device speech processing, on-device NLU, on-device fulfillment, and / or on-device fulfillment may be prioritized, at least due to the latency and / or network usage reduction they provide when parsing spoken utterances (due to the fact that no client-server round trip is required to parse spoken utterances). However, one or more cloud-based automated assistant components 280 may be utilized at least selectively. For example, such components may be used in parallel with on-device components and, when local components fail, the output from such components may be utilized. For example, in certain circumstances (e.g., due to the relatively limited resources of the client device 110), the on-device fulfillment engine 146 may fail, and in such circumstances, the remote fulfillment engine 283 may utilize the more robust resources of the cloud to generate fulfillment data. Remote fulfillment engine 283 may operate in parallel with on-device fulfillment engine 146 and utilize its results when on-device fulfillment fails, or may be invoked in response to determining a failure of on-device fulfillment engine 146 .

[0047] In various implementations, an NLU engine (on-device and / or remote) may generate NLU data comprising one or more annotations of the recognized text and one or more (e.g., all) terms of the natural language input. In some implementations, the NLU engine is configured to recognize and annotate various types of grammatical information in the natural language input. For example, the NLU engine may include a morphology module that can separate individual words into morphemes and / or annotate morphemes, for example, with categories of morphemes. The NLU engine may also include a part-of-speech tagger configured to annotate terms with their grammatical roles. Furthermore, for example, in some implementations, the NLU engine may additionally and / or alternatively include a dependency parser configured to determine syntactic relationships between terms in the natural language input.

[0048] In some implementations, the NLU engine may additionally and / or alternatively include an entity tagger configured to annotate entity references in one or more segments, such as references to people (including, for example, literary characters, celebrities, public figures, etc.), organizations, places (real and imagined), etc. In some implementations, the NLU engine may additionally and / or alternatively include a coreference resolver (not depicted) configured to group or "cluster" references to the same entity based on one or more contextual clues. In some implementations, one or more components of the NLU engine may rely on annotations from one or more other components of the NLU engine.

[0049] The NLU engine may also include an intent matcher configured to determine the intent of a user participating in an interaction with the automated assistant 295. The intent matcher may use various techniques to determine the user's intent. In some implementations, the intent matcher may access one or more local and / or remote data structures that include, for example, multiple mappings between grammars and response intents. For example, the grammars included in the mappings may be selected and / or learned over time and may represent common intents of users. For example, one grammar, "play <artist>," may be mapped to an intent that invokes a response action that causes music by <artist> to be played on the client device 110. Another grammar, "today's [weather|forecast]," may match user queries such as "what's the weather like today" and "what's the forecast for today?" In addition to or in lieu of grammars, in some implementations, the intent matcher may utilize one or more trained machine learning models, either alone or in combination with one or more grammars. These trained machine learning models can be trained to recognize intents, for example, by embedding the recognized text from the spoken utterance into a reduced dimensionality space, and then determining which other embeddings (and therefore intents) are closest, for example, using techniques such as Euclidean distance, cosine similarity, etc. As can be seen in the "play <artist>" example grammar above, some grammars have slots (e.g., <artist>) that can be filled with slot values ​​(or "parameters"). Slot values ​​can be determined in a variety of ways. Users typically actively provide slot values. For example, for the grammar "order me <toppings> pizza", the user might say the phrase "order me sausage pizza", in which case the slot <toppings> is automatically filled. Other slot values ​​can be inferred based on, for example, user location, currently rendered content, user preferences, and / or other cues.

[0050] A fulfillment engine (local and / or remote) can be configured to receive the predicted / estimated intent output by the NLU engine, as well as any associated slot values, and fulfill (or "resolve") the intent. In various implementations, the fulfillment (or "resolve") of the user intent can result in, for example, the generation / obtaining of various fulfillment information (also referred to as fulfillment data) by the fulfillment engine. This can include determining local and / or remote responses (e.g., answers) to the spoken utterance, interactions with locally installed applications based on the spoken utterance, commands transmitted to Internet of Things (IoT) devices (directly or via corresponding remote systems) based on the spoken utterance, and / or other resolution actions performed based on the spoken utterance. The on-device fulfillment can then initiate local and / or remote execution / implementation of the determined actions to resolve the spoken utterance.

[0051] Figure 3 A flowchart illustrating an example method 300 for synchronizing data across devices based on simultaneous hotword triggers is depicted. For convenience, the operations of method 300 are described with reference to a system performing the operations. The system of method 300 includes one or more processors and / or other components of a client device. Furthermore, while the operations of method 300 are presented in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0052] At block 310, the system executes, at least in part, a first instance of an automated assistant in an inactive state on a first computing device operated by a user. In implementations, the first computing device can run an on-device automated assistant stack that may not be connected to a central cloud account. In some implementations, the first computing device may not be connected to the internet.

[0053] At block 320 , while in the inactive state, the system receives audio data capturing spoken utterances of the user via one or more microphones of the first computing device.

[0054] At box 330, the system processes the audio data received at box 320 using a machine learning model to generate a prediction output indicating a probability of one or more hot words being present in the audio data. The machine learning model may be, for example, an on-device hot word detection model and / or other machine learning model. The machine learning model may be a deep neural network or any other type of model and may be trained to recognize one or more hot words. In addition, the generated output may be, for example, a probability and / or other likelihood measure. In some implementations, the hot word may be a hot word used to initiate a query (e.g., "good assistant"). In other implementations, the hot word may be a hot word used as part of a setup process (e.g., a hot word such as "Initiate device setup" for initiating setup of a new device, a hot word such as "Perform device sync" for triggering data synchronization, or a hot word used to train a speaker recognition model for a user).

[0055] At block 340, the system determines whether the prediction output generated at block 330 satisfies a threshold indicating the presence of one or more hot words in the audio data. If, in an iteration of block 340, the system determines that the prediction output generated at block 330 does not satisfy the threshold, the system proceeds to block 350 and the process ends. On the other hand, if, in an iteration of block 340, the system determines that the prediction output generated at block 330 does satisfy the threshold, the system proceeds to block 360.

[0056] Still referring to block 340 , in this example, assume that the predicted output generated at block 330 is a probability and that the probability must be greater than 0.85 to satisfy the threshold at block 340 , and the predicted probability is 0.88. Based on the predicted probability 0.88 satisfying the threshold 0.85, the system proceeds to block 360 .

[0057] At block 360, in response to determining at block 340 that the predicted output satisfies the threshold, the system performs arbitration with at least one other computing device that at least partially implements at least one other instance of the automated assistant. In some implementations, the at least one other computing device also receives audio data capturing the spoken utterance. For example, the at least one other computing device can be in the same room or otherwise physically close to the user and / or the first computing device, such that the at least one other computing device can also receive audio data capturing the utterance via one or more microphones.

[0058] Still referring to block 360, in some implementations, the system can use a central server to perform arbitration. In other implementations, the system can use a local communication protocol such as Bluetooth or Wi-Fi to perform arbitration. In implementations, the arbitration process can use various factors (e.g., nearest device, query content, highest hot word probability, etc.) to determine which device should be selected to handle the query, while suppressing other devices (i.e., not responding to the query).

[0059] At block 370, the system determines whether permission has been received from the user to perform synchronization of user data between the first instance of the automated assistant on the first computing device and at least one other instance of the automated assistant on at least one other computing device.

[0060] Still referring to block 370, in some implementations, the system can request user permission (e.g., at least on the first instance of synchronizing data between devices) to synchronize data between two or more devices. The system can store device identification information (e.g., MAC address) in response to the user granting permission to allow future automatic synchronization.

[0061] Still referring to box 370, in other implementations, the system can automatically determine that permission exists to perform data synchronization based on a secure handshake protocol that verifies that each device is owned by the same user. In some implementations, this handshake can involve a matching process between registered speaker profiles or some other attribute of the user identified as a joint secret. For example, device A can declare that it will use the user's date of birth as a secret. Device A then encrypts a random string using the date of birth. If device B has a secret available with the same value, it can decrypt the string and return its value to the other device as confirmation that they belong to the same user. This process can be performed symmetrically, so that devices A and B both use the same secret or different secrets to authenticate each other.

[0062] Still referring to block 370, if, in an iteration of block 370, the system determines that permission has not been received from the user to perform synchronization of user data between the first instance of the automated assistant on the first computing device and at least one other instance of the automated assistant on at least one other computing device, the system proceeds to block 350, and the process ends. On the other hand, if, in an iteration of block 370, the system determines that permission has been received from the user to perform synchronization of user data between the first instance of the automated assistant on the first computing device and at least one other instance of the automated assistant on at least one other computing device, the system proceeds to block 380.

[0063] Still referring to block 370, in other implementations, instead of (or in addition to) determining whether permission has been received from the user, the system determines whether a match exists between the registered user profiles stored on each of the first computing device and the at least one other computing device. If, in an iteration of block 370, the system determines that a match does not exist between the registered user profiles stored on each of the first computing device and the at least one other computing device, the system proceeds to block 350, and the process ends. On the other hand, if, in an iteration of block 370, the system determines that a match exists between the registered user profiles stored on each of the first computing device and the at least one other computing device, the system proceeds to block 380.

[0064] Still referring to block 370, in other implementations, instead of (or in addition to) determining whether permission has been received from the user, the system determines whether there is a match between the user attributes identified as the joint secret and stored on each of the first computing device and the at least one other computing device. If, in an iteration of block 370, the system determines that there is no match between the user attributes identified as the joint secret and stored on each of the first computing device and the at least one other computing device, the system proceeds to block 350 and the process ends. On the other hand, if, in an iteration of block 370, the system determines that there is a match between the user attributes identified as the joint secret and stored on each of the first computing device and the at least one other computing device, the system proceeds to block 380. In implementations, the joint secret can be based on the user's date of birth, the user's email address, the user's phone number, and / or user preferences (e.g., provided during the setup process), etc.

[0065] Still referring to block 370 , in other implementations, where non-sensitive data, such as device-level settings, is to be synchronized, the system may proceed to block 380 without determining whether permission exists to perform data synchronization.

[0066] At block 380, the system initiates synchronization of user data between the first instance of the automated assistant on the first computing device and at least one other instance of the automated assistant on at least one other computing device. In an implementation, the synchronization can be a two-way (or N-way, in the case of N devices) transfer of data. In an implementation, the user data includes data based on one or more interactions with the user at the first computing device, the one or more interactions occurring before receiving the audio data.

[0067] Still referring to box 380, in some implementations, the synchronized user data may include a personalized hot word model, a personalized speech recognition model, or a personalized query interpretation model. In some implementations, the synchronized user data may include queries recently issued by the user. In some implementations, the synchronized user data may include the user's preferences. In some implementations, the synchronized user data may include the user's private knowledge graph. In some implementations, the user data may include data selected for synchronization based on the capabilities of each of the first computing device and at least one other computing device. For example, a device that cannot play videos may not need information about the user's video viewing history. In addition, devices that may be resource-constrained in terms of storage may receive only selected data. In implementations, some synchronized data (e.g., private knowledge graphs) may be merged and other synchronized data (e.g., speech recognition models) may be combined, or which version of the data to retain may be determined, for example, based on criteria indicating quality, such as the number of personalized examples exposed to the speech recognition model.

[0068] Still referring to block 380 , in other implementations, the system may synchronize configuration data (eg, device-specific configuration information, settings, or preferences) instead of or in addition to synchronizing user data.

[0069] Figure 4 A flowchart illustrating an example method 400 for cross-device data synchronization based on simultaneous hotword triggers is depicted. For convenience, the operations of method 400 are described with reference to a system performing the operations. The system of method 400 includes one or more processors and / or other components of a client device. Furthermore, while the operations of method 400 are presented in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0070] At block 410, the system implements, at least in part, a first instance of an automated assistant in an inactive state on a first computing device operated by a user, and implements, at least in part, a second instance of an automated assistant in an inactive state on a second computing device operated by the user.

[0071] At block 420, while in the inactive state, the system receives first audio data via one or more microphones of the first computing device and receives second audio data via one or more microphones of the second computing device. In an implementation, the first audio data and the second audio data each capture a spoken utterance of a user.

[0072] In box 430, the system processes the first audio data received in box 420 via one or more microphones of the first computing device at the first computing device to generate a first prediction output that indicates a probability that one or more synchronization hot words are present in the first audio data received via the one or more microphones of the first computing device.

[0073] In box 440, the system processes the second audio data received in box 420 via one or more microphones of the second computing device at the second computing device to generate a second prediction output that indicates a probability that one or more synchronization hot words are present in the second audio data received via the one or more microphones of the second computing device.

[0074] At block 450, the system determines whether the first prediction output generated at block 430 satisfies a first threshold indicating the presence of one or more sync hotwords in the first audio data, and determines whether the second prediction output generated at block 440 satisfies a second threshold indicating the presence of one or more sync hotwords in the second audio data. If, in an iteration of block 450, the system determines that neither the first prediction output generated at block 430 satisfies the first threshold indicating the presence of one or more sync hotwords in the first audio data nor the second prediction output generated at block 440 satisfies the second threshold indicating the presence of one or more sync hotwords in the second audio data, the system proceeds to block 460 and the process ends. On the other hand, if, in an iteration of block 450, the system determines that both the first prediction output generated at block 430 satisfies the first threshold indicating the presence of one or more sync hotwords in the first audio data and the second prediction output generated at block 440 satisfies the second threshold indicating the presence of one or more sync hotwords in the second audio data, the system proceeds to block 470.

[0075] Still referring to block 450, in this example, assume that the first predicted output generated at block 430 is a probability and that the probability must be greater than 0.85 to satisfy the first threshold at block 450, and that the predicted probability is 0.88. Furthermore, assume that the second predicted output generated at block 440 is a probability and that the probability must be greater than 0.85 to satisfy the second threshold at block 450, and that the predicted probability is 0.91. Based on the first predicted output of 0.85 satisfying the first threshold of 0.85 and the second predicted output of 0.91 satisfying the second threshold of 0.85, the system proceeds to block 470.

[0076] At block 470, in response to determining at block 460 that the first predicted output satisfies the first threshold and the second predicted output satisfies the second threshold, the system performs arbitration between the first instance of the automated assistant on the first computing device and the second instance of the automated assistant on the second computing device. In some implementations, in response to determining at block 460 that the first predicted output satisfies the first threshold and the second predicted output satisfies the second threshold, the system may also transition the first instance of the automated assistant on the first computing device and the second instance of the automated assistant on the second computing device to an active state.

[0077] At block 480, the system determines whether permission has been received from the user to perform synchronization of user data between the first instance of the automated assistant on the first computing device and the second instance of the automated assistant on the second computing device. If, in an iteration of block 480, the system determines that permission has not been received from the user to perform synchronization of user data between the first instance of the automated assistant on the first computing device and the second instance of the automated assistant on the second computing device, the system proceeds to block 460, and the process ends. On the other hand, if, in an iteration of block 480, the system determines that permission has been received from the user to perform synchronization of user data between the first instance of the automated assistant on the first computing device and the second instance of the automated assistant on the second computing device, the system proceeds to block 490.

[0078] Still referring to block 480, in other implementations, instead of (or in addition to) determining whether permission has been received from the user, the system determines whether a match exists between the registered user profiles stored on each of the first computing device and the second computing device. If, in an iteration of block 480, the system determines that a match does not exist between the registered user profiles stored on each of the first computing device and the second computing device, the system proceeds to block 460, and the process ends. On the other hand, if, in an iteration of block 480, the system determines that a match exists between the registered user profiles stored on each of the first computing device and the second computing device, the system proceeds to block 490.

[0079] Still referring to block 480, in other implementations, instead of (or in addition to) determining whether permission has been received from the user, the system determines whether there is a match between the user attributes identified as the joint secret and stored on each of the first computing device and the second computing device. If, in an iteration of block 480, the system determines that there is no match between the user attributes identified as the joint secret and stored on each of the first computing device and the second computing device, the system proceeds to block 460 and the process ends. On the other hand, if, in an iteration of block 480, the system determines that there is a match between the user attributes identified as the joint secret and stored on each of the first computing device and the second computing device, the system proceeds to block 490. In implementations, the joint secret can be based on the user's date of birth, the user's email address, the user's phone number, and / or user preferences, etc.

[0080] Still referring to block 480, in other implementations, where non-sensitive data, such as device-level settings, is to be synchronized, the system may proceed to block 490 without determining whether permission exists to perform data synchronization.

[0081] At block 490, the system initiates synchronization of user data between the first instance of the automated assistant on the first computing device and the second instance of the automated assistant on the second computing device. In an implementation, the user data can include data based on one or more interactions with the user at the first computing device prior to receiving the first audio data, and data based on one or more interactions with the user at the second computing device prior to receiving the second audio data.

[0082] Still referring to block 490, in some implementations, the synchronized user data may include a personalized hotword model, a personalized speech recognition model, or a personalized query interpretation model. In some implementations, the synchronized user data may include the user's most recent queries. In some implementations, the synchronized user data may include the user's preferences. In some implementations, the synchronized user data may include the user's private knowledge graph. In some implementations, the user data may include data selected for synchronization based on the capabilities of each of the first computing device and the second computing device.

[0083] Still referring to block 490, in other implementations, the system may synchronize configuration data (eg, device-specific configuration information, settings, or preferences) instead of or in addition to synchronizing user data.

[0084] Figure 5is a block diagram of an example computing device 510 that can optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of a client device, a cloud-based automated assistant component, and / or other components can include one or more components of the example computing device 510.

[0085] The computing device 510 typically includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524 (including, for example, a memory subsystem 525 and a file storage subsystem 526), ​​a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow a user to interact with the computing device 510. The network interface subsystem 516 provides an interface to an external network and couples to corresponding interface devices in other computing devices.

[0086] User interface input devices 522 may include a keyboard, a pointing device (such as a mouse, trackball, touchpad, or graphics tablet), a scanner, a touch screen incorporated into a display, an audio input device (such as a voice recognition system, a microphone, and / or other types of input devices). In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computing device 510 or over a communication network.

[0087] User interface output device 520 can include a display subsystem, a printer, a fax machine, or a non-visual display (such as an audio output device). The display subsystem can include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visual image. The display subsystem can also provide a non-visual display such as via an audio output device. Typically, the term "output device" is used to include devices and methods for outputting information from computing device 510 to a user or to all possible types of another machine or computing device.

[0088] The storage subsystem 524 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 524 may include software that performs selected aspects of the methods disclosed herein and implements Figure 1A and Figure 1B The logic of the various components is depicted in .

[0089] These software modules are typically executed by processor 514 alone or in combination with other processors. The memory subsystem 525 used in the storage subsystem 524 may include multiple memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 may provide persistent storage for program and data files and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations may be stored by the file storage subsystem 526 in the storage subsystem 524 or in another machine accessible to the processor 514.

[0090] The bus subsystem 512 provides a mechanism for the various components and subsystems of the computing device 510 to communicate with each other as intended. Although the bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0091] The computing device 510 may be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5 The description of the computing device 510 depicted in FIG is intended only as a specific example for the purpose of illustrating some implementations. Many other configurations of the computing device 510 are possible with more Figure 5 The computing devices depicted may have greater or fewer components.

[0092] In cases where the systems described herein collect or otherwise monitor personal information about a user or may utilize personal and / or monitoring information, the user may be provided with an opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, the user's preferences, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from a content server. Furthermore, certain data may be processed in one or more ways before being stored or used to eliminate personally identifiable information. For example, the user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location (such as city, zip code, or state level) may be generalized where geographic location information is obtained so that the user's specific geographic location cannot be determined. Thus, the user may have control over how information about the user is collected and / or used.

[0093] Although several implementations have been described and illustrated herein, various other devices and / or structures for performing the functions and / or obtaining the results and / or one or more advantages described herein may be utilized, and each of such variations and / or modifications is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the specific application or applications in which the teachings are used. Those skilled in the art will recognize or be able to determine using only routine experimentation many equivalents to the specific implementations described herein. It will therefore be understood that the foregoing implementations are presented by way of example only, and within the scope of the appended claims and their equivalents, implementations may be practiced in a manner other than as specifically described and claimed. Implementations of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, if two or more such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, any combination of such two or more features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

Claims

1. A method implemented by one or more processors, the method comprising: executing, at least in part, a first instance of an automated assistant in an inactive state on a first computing device operated by a user; while in the inactive state, receiving, via one or more microphones of the first computing device, audio data capturing spoken utterances of the user; Processing the audio data using a machine learning model to generate a prediction output indicating a probability of one or more hot words being present in the audio data; determining that the predicted output satisfies a threshold indicating the presence of the one or more hot words in the audio data; In response to determining that the predicted output satisfies the threshold, performing arbitration with at least one other computing device that at least partially implements at least one other instance of the automated assistant; as well as In response to performing arbitration with the at least one other computing device: selecting user data for synchronization based on respective capabilities of each of the first computing device and the at least one other computing device; determining that there is a match between attributes of the user identified as a joint secret and stored on each of the first computing device and the at least one other computing device; as well as Initiate a bidirectional synchronization of the user data between a first instance of the automated assistant on the first computing device and at least one other instance of the automated assistant on the at least one other computing device, wherein the bidirectional synchronization occurs at the first computing device and the at least one other computing device, and the user data includes data based on one or more interactions with a user at the first computing device, the one or more interactions occurring prior to receipt of the audio data.

2. The method according to claim 1 further includes, before initiating bidirectional synchronization of the user data, determining that permission has been received from the user to perform synchronization of the user data between the first instance of the automated assistant on the first computing device and at least one other instance of the automated assistant on the at least one other computing device. 3 . The method of claim 1 , further comprising, before initiating the bidirectional synchronization of the user data, determining that a match exists between registered user profiles stored on each of the first computing device and the at least one other computing device.

4. The method according to claim 1, wherein The joint secret is based on the user's date of birth, the user's email address, the user's phone number, or the user's preferences.

5. The method according to claim 1, wherein The synchronized user data includes a personalized hot word model, a personalized speech recognition model, or a personalized query interpretation model.

6. The method according to claim 1, wherein The synchronized user data includes the most recent queries issued by the user.

7. The method according to claim 1, wherein The synchronized user data includes the user's preferences.

8. The method according to claim 1, wherein The synchronized user data includes the user's private knowledge graph.

9. The method according to any one of claims 1 to 8, further comprising: In response to performing arbitration with the at least one other computing device, synchronization of configuration data is initiated, the configuration data including settings of the first computing device or settings of the at least one other computing device.

10. A method implemented by one or more processors, the method comprising: executing, at least in part, a first instance of an automated assistant in an inactive state on a first computing device operated by a user; while in the inactive state, receiving, via one or more microphones of the first computing device, audio data capturing spoken utterances of the user; Processing the audio data using a machine learning model to generate a prediction output indicating a probability that one or more sync hotwords are present in the audio data; determining that the predicted output satisfies a threshold indicating that the one or more sync hotwords are present in the audio data; In response to determining that the predicted output satisfies the threshold, identifying at least one other computing device at least partially implementing at least one other instance of the automated assistant, the at least one other computing device also having received the audio data; as well as In response to identifying the at least one other computing device: selecting user data for synchronization based on respective capabilities of each of the first computing device and the at least one other computing device; determining that there is a match between attributes of the user identified as a joint secret and stored on each of the first computing device and the at least one other computing device; as well as Initiate a bidirectional synchronization of the user data between a first instance of the automated assistant on the first computing device and at least one other instance of the automated assistant on the at least one other computing device, wherein the bidirectional synchronization occurs at the first computing device and the at least one other computing device, and the user data includes data based on one or more interactions with a user at the first computing device prior to receipt of the audio data.

11. The method according to claim 10, wherein: The synchronized user data includes a personalized hot word model, a personalized speech recognition model, or a personalized query interpretation model.

12. The method according to claim 10, wherein: The synchronized user data includes the most recent queries issued by the user.

13. The method according to any one of claims 10 to 12, wherein: The synchronized user data includes the user's preferences.

14. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 13.

15. A computer-readable storage medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 13.

16. A system comprising a processor, a computer-readable memory, one or more computer-readable storage media, and program instructions collectively stored on the one or more computer-readable storage media, the program instructions being executable to perform the method according to any one of claims 1 to 13.

Images