Assistant interaction data that prevents non-transitory storage and / or erase storage of assistant interaction data

Abstract

The invention relates to assistant interaction data that prevents non-transitory storage and / or erase storage of the assistant interaction data. Embodiments disclosed herein are directed to techniques for selectively allowing and / or inhibiting non-transitory storage of one or more instances of assistant interaction data for a session round between a user and an automated assistant. Embodiments are additionally or alternatively directed to techniques for retroactively erasing assistant interaction data from non-transitory storage of previous assistant interactions.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 202180005458.4, with a filing date of January 7, 2021. Background Art

[0003] Humans may participate in human-computer interactions with interactive software applications, referred to herein as "automated assistants." For example, a person (who may be referred to as a "user" when interacting with an automated assistant) may provide input to the automated assistant, which may cause the automated assistant to generate and provide responsive output to control one or more smart devices, and / or perform one or more other functions. The input provided by the user may be, for example, touch input (e.g., via a touch screen), gestures (e.g., detected via a camera), and / or spoken natural language input (i.e., utterances detected via a microphone), which in some cases may be converted into text (or other semantic representations) and then further processed.

[0004] In many cases, the automated assistant includes an automated assistant client that is executed locally by an assistant client device (a client device that includes an assistant client application, also referred to as an assistant device) and directly engaged by a user, and a cloud counterpart that utilizes the more robust resources of the cloud to help the automated assistant client respond to the user input. For example, the automated assistant client can provide the cloud counterpart with an audio recording of the user's spoken utterances (and / or a textual conversion thereof), and optionally data indicating an account associated with the user. The cloud counterpart can perform various processing on the spoken utterances (and / or a textual conversion thereof) to return a response result to the automated assistant client, which can then provide a corresponding output to the user.

[0005] With user permission, assistant interaction data from interactions between a user and an automated assistant may be non-transiently stored in a computer-readable medium of a cloud counterpart and / or an assistant device. For example, an instance of interaction data may be stored in response to a conversation turn between a user and an automated assistant, such as a conversation turn including a user verbal input of "what's the weather in Louisville" and an assistant response of "85 and sunny". An instance of interaction data may include, for example, audio data capturing the user's verbal input, a text transcription thereof (e.g., generated using a speech-to-text model), natural language understanding (NLU) data based on the text transcription (e.g., intent (e.g., "today's weather") and / or parameters (e.g., the city of Louisville), and / or data characterizing the assistant's response (e.g., the response itself and / or the subject of the response). An instance of interaction data may be non-transiently stored, where it is stored for a duration that exceeds the duration required to generate and provide a response output, control one or more smart devices, and / or perform one or more other functions.

[0006] With the user's permission, the non-transiently stored assistant interaction data can be used for various purposes, such as improving future interactions between the user and the assistant. For example, the assistant interaction data can be used to infer topics of interest to the user (e.g., based on topics included in the assistant response data and / or NLU data), as well as updates related to those topics that are proactively provided to the user via the automated assistant. Such proactive updates can result in more efficient (e.g., shorter duration) automated assistant interactions because the need for the user to provide input requesting updates is eliminated. As another example, the assistant interaction data can be used to train machine learning models (such as speech recognition models and / or NLU models) used by the automated assistant to make them more accurate and / or more robust. Thus, various technical benefits can be achieved through the non-transient storage and use of assistant interaction data. However, for some interactions, non-transient storage of assistant interaction data may be undesirable for various considerations. Summary of the invention

[0007] Embodiments disclosed herein are directed to techniques for selectively enabling and / or disabling non-transitory storage of one or more instances of assistant interaction data for a conversation turn between a user and an automated assistant.

[0008] Some of these embodiments actively prevent the storage of assistant interaction data in a conversation turn in response to a specific user input received at the assistant device. In some versions of these embodiments, whether to actively prevent or allow the storage of assistant interaction data may depend on how the user calls the assistant at the beginning of the conversation turn. In other words, in some of these versions, when the assistant is called in one or more first ways, the storage of assistant interaction data may be prevented, and when the assistant is called in one or more second ways, the storage of assistant interaction data may be allowed. As an example, some of these versions may actively prevent such storage of conversation turns after a specific assistant call phrase (e.g., "secret assistant"), while allowing such storage of conversation turns after a specific other assistant call phrase (e.g., "OK assistant"). As another example, some of these versions may additionally or alternatively actively prevent such storage of conversation turns after a specific assistant call interaction with a software or hardware call button (e.g., long press), while allowing such storage of conversation turns after a specific other assistant call interaction with a software or hardware call button (e.g., short press). As another example, some of these versions may additionally or alternatively actively block such storage of conversational turns following certain invocation-free and touchless assistant invocation interactions (e.g., gesture-based invocation interactions detected via a camera of an assistant device), while allowing such storage of conversational turns following certain other invocation-free and touchless assistant invocation interactions. For example, the assistant may be invoked based on local processing of an image from a camera and determining that a user is making a certain gesture, and optionally, the presence of other conditions in the image (e.g., the gaze of the user making the gesture is directed toward the camera and / or the body and / or head posture of the user making the gesture is facing the camera). In this case, if the gesture is detected as the user placing their finger in front of their mouth in a shushing gesture, such storage may be actively blocked, while if the gesture is detected as the user waving, such storage may be allowed. Note that in some of the above versions, such storage may always be actively blocked for all conversational turns except after certain non-default (i.e., standard "out-of-the-box") invocation phrases, after non-default invocation interactions, and / or after certain phrases. For example, the default may be to actively prevent such storage unless the user utilizes a non-default invocation phrase such as "OK non-privateassistant."

[0009] In some additional or alternative versions of these embodiments, such storage may be proactively prevented for conversation turns following a particular spoken phrase provided after an invocation (e.g., "don't store this," "don't store anything for the next 5 minutes"), while allowed for conversation turns not following a particular spoken phrase provided after an invocation.

[0010] Regardless of the technology used to activate the active blocking of such storage, the blocking of such storage can continue until one or more termination conditions are met, such as a threshold amount of time (e.g., a fixed time or a time specified in a user input), the end of a conversation session, and / or the end of a conversation round. In addition, in various embodiments, audible and / or visual prompts are provided during the blocking of such storage to allow for guiding the user / assistant conversation so that the user can be aware of the continued blocking of such storage while preventing interference with the user / assistant conversation. For example, during the entire duration of the active blocking of such storage, visual prompts can be provided via the display of the assistant device and / or via the light emitting diode of the assistant device, and can be stopped when such blocking is terminated (e.g., in response to the occurrence of a termination condition). In these and other ways, audible and / or visual prompts can guide user / assistant interactions to ensure that the user knows when storage is blocked and when storage is allowed. This can notify the user when storage is blocked, thereby reducing the occurrence of inputs provided by the user for blocking storage when storage has been blocked. Therefore, the occurrence of unnecessary extensions of conversations can be prevented while preventing unnecessary use of resources when processing such inputs.

[0011] In various embodiments, storage of assistant interaction data for interacting with any one of the multiple assistant devices may be blocked in response to corresponding user input detected at only a subset (e.g., only one) of the multiple assistant devices of the coordinated ecosystem of assistant devices associated with the user. In these and other ways, user input detected at a first assistant device for blocking storage of a conversation round may still block storage at a second assistant device. This can ensure that if the second assistant device processes the utterance received during the conversation round (e.g., due to an error in device arbitration and / or due to the user moving the location closer to the second assistant device), storage is blocked at the second assistant device. In some of those various embodiments, each of the multiple assistant devices may present a corresponding audible and / or visual prompt during the blocking of such storage. In these and other ways, the audible and / or visual prompts may guide the user / assistant interaction to ensure that the user knows when storage is blocked and when storage is allowed on a device-by-device basis. This can reduce the occurrence of a user providing input for blocking storage when storage is already blocked.

[0012] Some embodiments disclosed herein additionally or alternatively erase stored instances of assistant interaction data retroactively in response to specific user input received at an assistant device. As an example, some of these embodiments may retroactively clear any assistant interaction data timestamped within the past week in response to a verbal input to the automated assistant of "delete everything from the past week." For example, those instances of assistant interaction data that are timestamped within a week of the current time and associated with the account of the user providing the verbal input may be erased. The account of the user providing the verbal input may be determined using speaker recognition technology, facial matching technology, and / or other recognition technology. Similarly, when a corresponding assistant interaction occurs, an instance of assistant interaction data may be stored in association with an account based on identifying the account using similar technology.

[0013] In some embodiments, when a verbal input causes assistant interaction data to be retroactively erased from a previous assistant interaction, any future actions that were queued for future execution based on the previous assistant interaction but have not yet been executed can be maintained. In other words, these future actions will still be executed in the future. For example, a previous assistant interaction in which the verbal input "turn on the livingloom lights at 10:00" can queue a future action to cause the livingloom lights to transition to the on state at 10:00. The future action can persist even if the associated assistant interaction data is erased, so the livingloom lights will still transition to the on state at 10:00.

[0014] In some embodiments, the verbal input indicates a desire to erase assistant interaction data, but fails to specify a time period for erasing such data. For example, the verbal input may be "delete what I just said", rather than "everything from the past X temporal period" (e.g., hours, minutes, days). In some of these embodiments, the assistant interaction data to be erased may be determined based on those instances of assistant interaction data identified from the most recent conversation session, and the identified instances may be erased. It may be determined that instances of assistant interaction data belong to the same conversation session based on one or more considerations, such as comparisons between the attributes of the instances. For example, assume a first instance of the first round of a person / assistant conversation and a second instance of the second round of a person / assistant conversation. Whether the two instances are determined to belong to the same conversation session may be based on a comparison of the timestamps of the two instances and / or a comparison of the topics of the two instances. For example, the timestamp of the first instance may indicate the time of the human input and / or assistant response (and / or action) of the first round, and the timestamp of the second instance may similarly indicate the time of the human input and / or assistant response (and / or action) of the second round. The closer the two timestamps are in time, the better it can indicate that the two instances belong to the same session. For example, if the temporal proximity meets a threshold, they can be considered to belong to the same session. It can be additionally or alternatively considered whether the subject matter of the two instances is the same and / or similar.

[0015] In some embodiments, a conversation session can include a logically independent exchange of one or more messages between a user and an automated assistant. Distinguishing between multiple conversation sessions can be based on various signals, such as the passage of time between sessions, changes in user context between sessions (e.g., location, before / during / after a scheduled meeting, etc.), detection of one or more intervening interactions between the user and the assistant device other than a conversation between the user and the automated assistant (e.g., the user switches applications for a period of time, the user walks away and then later returns to a standalone voice-activated product), assistant device locking / sleeping between sessions, changing the assistant device used to interface with one or more instances of the automated assistant, and the like.

[0016] The foregoing is merely an overview of some embodiments disclosed herein. These and / or other embodiments will be described in more detail below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 An example environment is shown in which embodiments disclosed herein may be implemented.

[0018] Figure 2A , Figure 2B , Figure 2C and Figure 2D Each shows an example interaction between a user and an assistant device according to various embodiments disclosed herein.

[0019] Figure 3 is a flow diagram illustrating an example method for proactively preventing storage of assistant interaction data for a conversation turn in response to certain user input received at an assistant device.

[0020] Figure 4 is a flow chart illustrating an example method of retroactively erasing a stored instance of assistant interaction data in response to specific user input received at an assistant device.

[0021] Figure 5 An example architecture for a computing device is shown. DETAILED DESCRIPTION

[0022] First turn Figure 1, assistant device 110 may include one or more microphones, one or more speakers, and optionally one or more cameras and / or other visual components and / or displays (e.g., a touch-sensitive display). Assistant device 110 at least selectively executes automated assistant client 120. Automated assistant client 120 may include, for example, an on-device automatic speech recognition (ASR) engine 122, an on-device natural language understanding (NLU) engine 124, an on-device text-to-speech (TTS) engine 126, an on-device implementation engine 128, an on-device invocation engine 130, an on-device authentication engine 132, and / or an on-device interaction data engine 134. Automated assistant client 120 may include additional and / or alternative engines, such as a voice activity detector (VAD), an endpoint detector, and / or other engines.

[0023] One or more cloud-based automated assistant components 140 may optionally be implemented on one or more computing systems (collectively referred to as "cloud" computing systems) that are communicatively coupled to the assistant device via one or more local area networks and / or wide area networks (e.g., the Internet) 108. Cloud-based automated assistant components 140 may be implemented, for example, by a high-performance server cluster. Cloud-based automated assistant components 140 are described in more detail below.

[0024] In various embodiments, an instance of automated assistant client 120, optionally through its interaction with cloud-based automated assistant component 140, can form what appears to be a logical instance of automated assistant 100 from the perspective of a user, who can interact with automated assistant 100 (e.g., verbally, gesture-based, and / or touch-based).

[0025] Assistant device 110 may be, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker (optionally with a display and / or a camera), a smart appliance such as a smart TV (or a standard TV equipped with a network-connected dongle with automated assistant capabilities), and / or a wearable device of a user including a computing device (e.g., a watch of a user with a computing device, glasses of a user with a computing device, a virtual or augmented reality computing device). Additional and / or alternative assistant devices may be provided.

[0026] The optional visual component of assistant device 110 can take various forms, such as a thematic camera, a stereo camera, a LIDAR component (or other laser-based component), a radar component, etc. One or more visual components can be used to capture visual frames (e.g., image frames, laser-based visual frames) of the environment in which the assistant device is deployed. The visual frames can be used by, for example, on-device invocation engine 130, on-device authentication engine 132, and / or other engines of automated assistant 100 (e.g., cloud-based invocation engine 150 and / or cloud-based authentication engine 152).

[0027] In some embodiments, the on-device call engine 130 can use such a visual frame to determine whether a user (any user or any of one or more registered users) is present near the assistant device, to determine whether the user is providing a touchless call gesture, and / or to determine the distance of the user (e.g., the user's face) relative to the assistant device. For example, the on-device call engine 130 can use such a determination to determine whether to activate the on-device ASR engine 122 and / or other assistant components. For example, the on-device call engine 130 can use one or more call models 162A-162N (e.g., a neural network model) to process the visual frame to determine whether the user captured in the visual frame is providing a touchless call gesture. For example, the call engine 130 can process the visual frame when determining whether the user is providing a touchless call gesture and which touchless call gesture is being provided. As described herein, one or more touchless call gestures can be performed to actively prevent the storage of assistant interaction data, and one or more other touchless call gestures can be performed to allow the storage of assistant interaction data. The on-device invocation engine 130 can provide an indication of which touchless invocation gesture is being provided to the on-device interaction data engine 134, and the on-device interaction data engine 134 can use the indication to determine whether to actively prevent storage of assistant interaction data for a conversation turn following the touchless invocation, or to allow storage of such assistant interaction data. For example, the on-device interaction data engine 134 can use the indication to determine whether to store such assistant interaction data locally in an on-device interaction data database 166 located on one or more computer-readable media local to the assistant device 110. As another example, the on-device interaction data engine 134 can transmit the indication (or related data) to the cloud-based automated assistant component 140 to allow or prevent cloud-based storage of such assistant interaction data by the cloud-based interaction data engine 154.

[0028] In some embodiments, on-device authentication engine 132 can additionally or alternatively utilize such visual frames to determine an account associated with a user who is in the visual frame and is interacting with assistant device 110 (e.g., determined based on the visual frame indicating that the user's gaze, body gesture, and / or head gesture is directed to assistant device 110). For example, on-device authentication engine 132 can process the visual frame using one or more on-device authentication models 164A-164N (e.g., neural network models) stored locally at assistant device 110 to determine whether features of the user captured in the visual frame sufficiently match stored features of an account registered with assistant device 110. For example, one or more on-device authentication models 164A-164N can be used to process the visual frame (e.g., capture at least a portion of the user's face) to generate a facial embedding of the user's face, and compare the facial embedding to a stored facial embedding previously generated during a registration process for the user account. If the comparison indicates a sufficient match (e.g., less than a threshold distance in the embedding space), authentication engine 132 can determine that the upcoming or ongoing interaction is from a user associated with the account.

[0029] As described herein, a user may provide a verbal utterance that expresses an intent to erase assistant interaction data of the user's previous assistant interactions. In some of these embodiments, the on-device authentication engine 132 may provide an indication of the user's account to the on-device interaction data engine 134, and the on-device interaction data engine 134 may utilize the indication to ensure that interaction data stored in association with the account is erased. For example, the on-device interaction data engine 134 may utilize the indication to identify assistant interaction data stored locally in the on-device interaction data database 166 associated with the account. As another example, the on-device interaction data engine 134 may transmit the indication (or related data) to the cloud-based automated assistant component 140 to enable the cloud-based interaction data engine 154 to identify and erase assistant interaction data stored in the remote interaction database 176 associated with the account.

[0030] In addition to or in lieu of invoking automated assistant 100 based on visual frames, on-device invocation engine 130 can invoke automated assistant 100 based on detecting the occurrence of an assistant invocation phrase and / or the occurrence of one or more hardware and / or software button actuations. For example, on-device invocation engine 130 can invoke automated assistant 100 in response to detecting a verbal assistant invocation phrase such as "Hey assistant," "OK assistant," "assistant," "Secret assistant," and / or "Private assistant." On-device invocation engine 130 can use one or more on-device invocation models 162A-162N to continuously process (e.g., if not in "inactive" mode) a stream of audio data frames based on output from one or more microphones of assistant device 110 to monitor for the occurrence of assistant invocation phrases. For example, one or more first on-device invocation models 162A-162N can be used to process audio data frames to monitor for one or more first invocation phrases (e.g., those phrases that cause active blocking of storage of assistant interaction data). One or more second on-device invocation models 162A-162N may be used in parallel to process audio data frames to monitor for one or more second invocation phrases (e.g., those phrases that cause assistant interaction data to be stored). While monitoring for the occurrence of a spoken assistant invocation phrase, the on-device invocation engine 130 discards (e.g., after temporarily storing in a buffer) any audio data frame that does not include a spoken invocation phrase. However, when the on-device invocation engine 130 detects the occurrence of a spoken invocation phrase in a processed audio data frame, the on-device invocation engine 130 may invoke the automated assistant 100. As used herein, "invoking" an automated assistant may include causing one or more previously inactive functions of the automated assistant to be activated. For example, invoking an automated assistant may include causing one or more local engines and / or cloud-based automated assistant components to further process the audio data frame based on the audio data frame in which the invocation phrase was detected, and / or one or more subsequent audio data frames (where no further processing of the audio data frame occurred prior to the invocation). For example, the local and / or cloud-based components may process the captured audio data using an ASR model in response to the invocation of the automated assistant.

[0031] As described herein, detection of a particular invocation phrase may result in active blocking of storage of assistant interaction data, and detection of a particular other invocation phrase may result in allowing storage of assistant interaction data. The on-device invocation engine 130 may provide an indication to the on-device interaction data engine 134 that provides which phrase was invoked, and the on-device interaction data engine 134 may utilize the indication to determine whether to actively block storage of assistant interaction data for a conversation turn following a touchless invocation, or to instead allow storage of such assistant interaction data. For example, the on-device interaction data engine 134 may utilize the indication to determine whether to store such assistant interaction data locally in an on-device interaction data database 166, which is located on one or more computer-readable media local to the assistant device 110. As another example, the on-device interaction data engine 134 may transmit the indication (or related data) to the cloud-based automated assistant component 140 to allow or block cloud-based storage of such assistant interaction data by the cloud-based interaction data engine 154. In embodiments where active prevention of storage of assistant interaction data is based on the manner in which the assistant is invoked, the interaction data engine 134 can utilize an indication of the manner to prevent storage of any assistant interaction data prior to even providing verbal input and / or other data that generates assistant interaction data. In these and other ways, the interaction data engine 134 can ensure that assistant interaction data is only temporarily stored (if at all) for the amount of time necessary to resolve the corresponding user request. This can minimize (or even eliminate) the amount of time that memory and / or other resources are used when temporarily storing assistant interaction data.

[0032] In addition to determining the account of the interacting user based on the visual frame, or in lieu of determining the account of the interacting user based on the visual frame, the on-device authentication engine 132 can process the audio frame to determine the account. For example, the on-device authentication engine 132 can process the audio frame using one or more of the on-device authentication models 164A-164N (e.g., a neural network model) stored locally at the assistant device 110 to determine whether the characteristics of the voice input captured in the audio frame sufficiently match the stored characteristics of the account registered with the assistant device 110. For example, the audio frame can be processed using one or more of the on-device authentication models 164A-164N to generate a speaker embedding, and the speaker embedding can be compared to the stored speaker embedding previously generated during the registration process of the user account. If the comparison indicates a sufficient match (e.g., less than a threshold distance in the embedding space), the authentication engine 132 can determine that the upcoming or ongoing interaction is from the user associated with the account. The audio frame can include an audio frame that captures the invocation phrase (e.g., text-dependent speaker recognition) and / or an audio frame after the invocation (e.g., text-independent speaker recognition).

[0033] As described herein, a user may provide a verbal utterance expressing an intent to erase assistant interaction data of the user's previous assistant interactions. In some of these embodiments, on-device authentication engine 132 may provide an indication of the user's account to on-device interaction data engine 134, and on-device interaction data engine 134 may utilize the indication to ensure that interaction data stored in association with the account is erased.

[0034] The ASR engine 122 on the device can process the audio data capturing the spoken utterance using the on-device speech recognition model to generate the recognized text corresponding to the spoken utterance. The optional on-device NLU engine 124 of the assistant device performs on-device natural language understanding on the recognized text to generate NLU data. The on-device NLU engine 124 can optionally generate NLU data using one or more on-device NLU models. The NLU data may include, for example, the intent corresponding to the spoken utterance and optional parameters (e.g., slot values) for the intent. For example, the NLU data may indicate whether the utterance expresses the intent to retroactively erase the assistant interaction data and / or actively prevent the storage of the assistant interaction data. When the NLU data indicates such an intent, the NLU engine 124 may provide an indication of the intent to the interaction data engine 134 for appropriate processing. The NLU data may also include parameters for such an intent, such as the duration of actively preventing storage and / or the time period during which the assistant interaction data will be erased. Such parameters may also be provided to the interaction data engine 134 for processing. As described herein, in the absence of such parameters, the interaction data engine 134 may utilize various techniques to automatically determine the duration or time period, and may optionally cause an audible and / or visual indication of this automatic determination to be provided.

[0035] In addition, the optional on-device implementation engine 128 of the assistant device generates implementation data using the NLU data. The on-device implementation engine 128 can optionally generate implementation data using one or more on-device implementation models. The implementation data can define local and / or remote responses (e.g., answers) to spoken utterances, interactions performed with locally installed applications based on spoken utterances, commands transmitted to Internet of Things (IoT) devices (directly or via corresponding remote systems) based on spoken utterances, and / or other parsing actions performed based on spoken utterances. The implementation data is then provided for local and / or remote execution / implementation of the determined actions to parse spoken utterances. Implementation can include, for example, presenting local and / or remote responses (e.g., visual and / or audible presentation (optionally using the on-device TTS engine 126)), interacting with locally installed applications, transmitting commands and / or other actions to IoT devices. The optional on-device TTS engine 126 can generate synthetic speech using one or more on-device TTS models. The implementation engine 128 can utilize the on-device TTS engine 126 to generate synthetic speech for specific audible local responses. The TTS engine 126 can also be used to generate audible prompts described herein.

[0036] The optional display of assistant device 110 can be used to present the various visual prompts described herein and / or can be one of the user interface output components by which the visual portion of the response from automated assistant client 120 is presented. The visual prompts can additionally or alternatively be presented via light emitting diodes and / or other visual output devices. In addition, audible prompts can be presented via a speaker of assistant device 110.

[0037] In some embodiments, cloud-based automated assistant component 140 may include remote ASR engine 142 that performs speech recognition, remote NLU engine 144 that performs natural language understanding, remote TTS engine 146 that generates synthesized speech, remote implementation engine 148 that generates implementation data, remote invocation engine 150 that determines whether and / or how to invoke assistant 100 (or verifies the determination of invocation engine 130 on the device), authentication engine 152 that determines the account (if any) of the interacting user (or verifies the determination of authentication engine 132 on the device), and interaction data engine 154. Interaction data engine 154 may selectively store interaction data in remote interaction data database 176. In addition, interaction data engine 154 may retroactively erase assistant interaction data for an account from remote interaction data database 176 based on instructions received from interaction data engine 134 and / or self-determination using other techniques described herein. Furthermore, the interaction data engine 154 can additionally or alternatively actively prevent (or allow) storage of assistant interaction data for a dialog turn at the remote interaction database 176 based on instructions received from the interaction data engine 134 and / or self-determination using other techniques described herein. A remote execution module can also optionally be included that performs remote execution based on locally or remotely determined implementation data.

[0038] Additional and / or alternative remote engines may be included. In various embodiments, when provided on an assistant device, speech processing on the device, NLU on the device, implementation on the device, and / or execution on the device may be prioritized, at least due to the latency and / or network usage reduction they provide when parsing spoken utterances (due to the lack of client-server round trips to parse spoken utterances). However, one or more cloud-based automated assistant components may be at least selectively utilized. For example, such components may be used in parallel with components on the device, and outputs from such components may be used when local components fail. For example, in certain circumstances (e.g., due to relatively limited resources of client 160), an implementation engine on the device may fail, and in this case, a remote implementation engine may utilize more robust resources of the cloud to generate implementation data. A remote implementation engine may operate in parallel with an implementation engine on the device, and utilize its results when the implementation fails on the device, or may be called in response to determining that the implementation fails on the device.

[0039] In various embodiments, the NLU engine (on-device and / or remote) can generate an annotated output including one or more annotations of the recognized text and one or more (e.g., all) terms of the natural language input. In some embodiments, the NLU engine is configured to recognize and annotate various types of grammatical information in the natural language input. For example, the NLU engine may include a morphological module that can separate individual words into morphemes and / or annotate morphemes, for example, with categories of morphemes. The NLU engine may also include a speech tagger that is configured to annotate terms with the grammatical roles of the terms. In addition, for example, in some embodiments, the NLU engine may additionally and / or alternatively include a dependency parser that is configured to determine syntactic relationships between terms in the natural language input.

[0040] In some embodiments, the NLU engine may additionally and / or alternatively include an entity tagger configured to annotate entity references in one or more segments, such as references to people (e.g., including literary characters, celebrities, public figures, etc.), organizations, locations (real and fictional), etc. In some embodiments, the NLU engine may additionally and / or alternatively include a coreference resolver (not shown) configured to group or "cluster" references to the same entity based on one or more context clues. In some embodiments, one or more components of the NLU engine may rely on annotations from one or more other components of the NLU engine.

[0041] The NLU engine may also include an intent matcher that is configured to determine the intent of a user participating in an interaction with automated assistant 100. The intent matcher may use various techniques to determine the intent of a user. In some embodiments, the intent matcher may access one or more local and / or remote data structures that include, for example, multiple mappings between grammars and response intents. For example, the grammars included in the mappings may be selected and / or learned over time and may represent common intents of users. For example, a grammar "play <artist>(play <artist>)" can be mapped to an intent that invokes a response action that makes <artist>music is played on the assistant device. Another grammar "[weather|forecast] today" can match user queries such as "what's the weather today" and "what's the forecast for today?". In addition to or in lieu of grammars, in some embodiments, the intent matcher can use one or more trained machine learning models alone or in combination with one or more grammars. These trained machine learning models can be trained to recognize intents, for example by embedding recognized text from spoken utterances into a reduced dimensionality space, and then determining which other embeddings (and therefore intents) are closest, for example using techniques such as Euclidean distance, cosine similarity, etc. As shown above, the "play <artist>As shown in the example grammars, some grammars have slots that can be filled with slot values ​​(or "parameters") (e.g. <artist>). Slot values ​​can be determined in a variety of ways. Users often actively provide slot values. For example, for the syntax "Order me a <topping>pizza", the user might say the phrase "order me a sausage pizza", in which case the slot <topping>Other slot values ​​may be inferred based on, for example, user location, currently presented content, user preferences, and / or other cues.

[0042] An implementation engine (local and / or remote) may be configured to receive the predicted / estimated intent output by the NLU engine and any associated slot values, and implement (or "parse") the intent. In various embodiments, implementation (or "parse") of the user's intent may result in, for example, the generation / obtaining of various implementation information (also referred to as implementation data) by the implementation engine. This may include determining local and / or remote responses (e.g., answers) to spoken utterances, interactions with locally installed applications performed based on spoken utterances, commands transmitted to Internet of Things (IoT) devices (directly or via corresponding remote systems) based on spoken utterances, and / or other parsed actions performed based on spoken utterances. On-device implementation may then allow local and / or remote execution / implementation of the determined actions to parse the spoken utterance.

[0043] As described herein, in various embodiments, assistant interaction data may include data generated by one or more of the above-described assistant device 110 and / or cloud-based automated assistant component 140 (such as an ASR engine, an NLU engine, and / or an implementation module) (or data based on such generated data). When stored non-transiently, the assistant interaction data may be stored at the cloud-based automated assistant component 140 (e.g., in database 176) and / or the assistant device 110. Thus, when erased (or prevented from being stored), the erasure (or prevention of storage) may be at the cloud-based automated assistant component 140 and / or the assistant device 110.

[0044] refer to Figure 2A , Figure 2B , Figure 2C and Figure 2D , shows an example interaction between a user 201 and an assistant device 205, and demonstrates aspects of various embodiments disclosed herein.

[0045] exist Figure 2A , user 201 provides three spoken utterances 252A1, 252A2, and 252A3, and corresponding assistant responses 254A1, 254A2, and 254A3 are audibly (and optionally visually) presented by assistant device 205. Each spoken utterance and assistant response pair (e.g., 252A1 and 254A1 are a pair) can be a turn of conversation, and in Figure 2A In the example of , relatively little time can elapse between turns (e.g., less than 5 seconds between completion of response 254A1 and utterance 252A2, and similarly for response 254A2 and utterance 252A3). Spoken utterance 252A4 indicates an intent to erase assistant interaction data for one or more previous assistant interactions, but fails to indicate any time period (e.g., date range, hour range, minute range) for which to erase assistant interaction data. As a result, in some embodiments, optionally, if a recent conversation session ended within a threshold time of receiving spoken utterance 252A4, the automated assistant can erase assistant interaction data for only those interactions of that recent conversation session. Figure 2A In the example of , based on a determination that all three spoken utterances and all three assistant responses form part of a single conversation session, assistant interaction data for any of all three spoken utterances 252A1, 252A2, and 252A3 and any of all three corresponding assistant responses 254A1, 254A2, and 254A3 can be erased. Such a determination can be made (e.g., by on-device interaction data engine 134 or remote interaction data engine 154) based on all three turns occurring in close temporal proximity to one another (e.g., each turn past an initial turn starting within X seconds of the end of the previous turn) and / or based on similarity in the subject matter of the turns (e.g., based on all turns having weather intent, as determined based on NLU data). In Figure 2A , a visual cue 206 ("check mark") is provided on the assistant device's display 205A to visually indicate that the request of spoken utterance 252A4 was performed. In addition, an assistant response 254A4 is also audibly provided and indicates that all three turns of assistant interaction data are to be deleted in response to the request of spoken utterance 252A4, thereby removing any ambiguity about the action taken in response to the ambiguous spoken utterance 252A4.

[0046] exist Figure 2B , the user provides three spoken utterances 252B1, 252B2, and 252B3, and corresponding assistant responses 254B1, 254B2, and 254B3 are audibly (and optionally visually) presented by assistant device 205. Each spoken utterance and assistant response pair (e.g., 252A1 and 254A1) can be a turn of conversation, and in Figure 2A In the example of , relatively little time may have elapsed between the first two rounds (the first round of 252B1 and 254B1, and the second round of 252B2 and 254B2), but a relatively large time gap (e.g., 30 seconds or more) may occur between the second and third rounds. Spoken utterance 252B4 indicates an intent to erase assistant interaction data for one or more previous assistant interactions, but fails to indicate any time period (e.g., a date range, an hour range, a minute range) for erasing assistant interaction data. As a result, in some embodiments, optionally, if a recent conversation session ends within a threshold time of receiving spoken utterance 252B4, the automated assistant may erase assistant interaction data for only those interactions of that recent conversation session. Figure 2B In the example of , based on determining that the most recent turn itself forms the most recent conversation session, and the first two turns are part of a previous conversation session, the assistant interaction data for only the most recent turn (i.e., 252B3 and 254B3) can be erased. This determination can be based on a time gap between the third turn and the second turn and / or based on the third turn being related to a different topic than the topics of the first two turns (e.g., weather). Figure 2B , a visual cue 206 (a "check mark") is provided on display 205B of assistant device 205 to visually indicate that the request of spoken utterance 252B4 was performed. In addition, an assistant response 254B4 is also audibly provided and indicates that in response to the request of spoken utterance 252B4, only the most recent turn of assistant interaction data will be deleted (i.e., by indicating its subject matter), thereby eliminating any ambiguity about the action taken in response to the ambiguous spoken utterance 252B4. Note that in Figure 2B In the example of , if user 201 also wants to delete assistant interaction data from the previous turn, user 201 may provide a further utterance to cause such deletion. For example, the further utterance may be "forget everything I said for the last 5 minutes" or "forget the weather questions as well", either of which may be interpreted (e.g., using the NLU engine and / or the interaction data engine) as a request for erasure. Figure 2B earlier round.

[0047] Figure 2C The example of provides an example of spoken input 252C2 that causes assistant interaction data to be retroactively erased from a previous assistant interaction (i.e., the interactions of 252C1, 254C1) while maintaining future actions for reminder notifications that were queued for future execution but have not yet been executed.

[0048] exist Figure 2C , a user provides spoken utterance 252C1 requesting that an action be performed (providing a reminder notification) in the future (tomorrow morning at 6:00 AM). A corresponding assistant response 254C1 is audibly (and optionally visually) presented by assistant device 205 and confirms that the action is queued for future execution. Spoken utterance 252C2 is provided shortly after assistant response 254C1 (e.g., 10 seconds), and indicates an intent to erase assistant interaction data for one or more previous assistant interactions, but fails to indicate any time period (e.g., date range, hour range, minute range) for erasing assistant interaction data. As a result, in some embodiments, optionally, if a recent conversation session ends within a threshold time of receiving spoken utterance 252C2, the automated assistant may erase assistant interaction data for only those interactions of that recent conversation session. Figure 2C In the example of , assistant interaction data for only the most recent turn (i.e., 252C1 and 254C1) can be erased based on determining that the most recent turn itself forms the most recent conversation session. However, it is worth noting that actions queued for future execution are not erased, but rather remain queued for future execution (i.e., the reminder notification will still be presented at 6:00 AM tomorrow). Figure 2C , a visual cue 206 ("check mark") is provided on display 205B of assistant device 205 to visually indicate that the request of spoken utterance 252C2 was executed. In addition, an assistant response 254C2 is also audibly provided and indicates that in response to the request of spoken utterance 254C2, only the assistant interaction data of the most recent interaction will be deleted, and indicates that the action will remain queued for future execution, thereby eliminating any ambiguity as to what will be deleted and / or whether the action will remain queued for future execution.

[0049] Figure 2D The example of provides an example of a spoken input 252D1 invoking an automated assistant and including a specific invocation phrase ("secret assistant") that actively prevents non-transient storage of assistant interaction data. Figure 2D In the embodiment of the present invention, a symbol 207 is visually provided to provide an indication of such blocking, and will continue to be presented until the termination condition occurs. In other words, the symbol 207 can be provided for the entire duration of the non-transient storage of the assistant interaction data that is actively blocked. Optionally, an assistant prompt 254D1 can be audibly provided to also provide an indication of the start of such blocking. Additionally or alternatively, during the duration of the blocking, a continuous audible prompt can be continuously provided, such as continuous bird sounds, waves crashing, or other sounds. Due to the active blocking triggered by the verbal input 252D1, the assistant interaction data associated with the verbal input 252D2 and / or the assistant response 254D2 will never be non-transiently stored. It should be noted that in some embodiments, an audible and / or visual prompt can be provided additionally or alternatively to provide an indication of when the non-transient storage of the assistant interaction data is occurring (i.e., when it is not blocked).

[0050] Figure 3 is a flow chart illustrating an example method for proactively blocking storage of assistant interaction data for a turn of a conversation in response to certain user input received at an assistant device. For convenience, the operations of the flow chart are described with reference to a system performing the operations. The system may include various components of various computer systems, such as one or more components of assistant device 110 or cloud-based automated assistant component 140. Furthermore, while the operations of method 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0051] At box 352, the system determines to prevent non-transient storage of assistant interaction data based on user input received at the assistant device. In some embodiments, the system prevents non-transient storage of assistant interaction data only for interactions at the assistant device. In some other embodiments (and optionally depending on the input), the system prevents non-transient storage of assistant interaction data for interactions at the assistant device and for interactions at one or more (e.g., all) assistant devices in the user's assistant device ecosystem (e.g., those assistant devices linked to the user's account and optionally linked to the same structure as the assistant device). In some embodiments, box 352 includes sub-box 352A or sub-box 352B.

[0052] At sub-box 352A, the system determines to prevent non-transient storage of assistant interaction data based on whether the call to the assistant is a specific call. For example, when one or more specific calls are detected, these calls can cause the system to determine to prevent non-transient storage of assistant interaction data, and the system can prevent non-transient storage in response to determining that the call is one of those specific calls. For example, the call can be a spoken call phrase. The system can use each of a plurality of different call phrase detection models to process the audio data, each call phrase detection model being trained for a different subset of call phrases (e.g., only a corresponding single call phrase) to generate a corresponding metric, each metric indicating whether a corresponding call phrase subset is present in the audio data. If the corresponding metric generated by the model trained for preventing non-transient storage of specific call phrases (e.g., "Private assistant") indicates that one of those call phrases is detected, the system can determine to prevent non-transient storage of assistant interaction data. The specific call can additionally or alternatively include a touchless gesture call, an actuation of a specific hardware or software button, and / or an actuation of a hardware or software button in a specific manner (e.g., a long press or a double click).

[0053] At subblock 352B, the system determines to prevent non-transient storage of the assistant interaction data based on processing of the spoken utterance following the invocation. For example, the system can generate recognized text by performing ASR on the audio data capturing the spoken utterance, and then utilizing NLU to determine that the recognized text expresses an intent to prevent non-transient storage of the assistant interaction data.

[0054] At box 354, the system determines one or more termination conditions for stopping the prevention of non-transient storage of assistant interaction data. In other words, the system will continue to prevent non-transient storage of assistant interaction data until one or more conditions are met. In some embodiments, box 354 includes sub-box 354A and / or sub-box 354B.

[0055] At sub-box 354A, the system determines a termination condition based on the call and / or spoken utterance of box 352. For example, a spoken call phrase of "secret assistant" can be associated with a termination condition of "passage of 30 seconds" or "termination of current dialog session", while a spoken call phrase of "top secret assistant" can be associated with a termination condition of "passage of 1 hour". The system can select the corresponding termination condition as one of the termination conditions. As another example, the spoken utterance after the call can specify a duration of blocking, and the duration can be a parameter determined using NLU, and the system can use the expiration of the duration as the termination condition. For example, if the spoken utterance is "secret mode for 10 minutes", the expiration of the duration of 10 minutes can be used as the termination condition.

[0056] At sub-block 354B, the system additionally or alternatively determines one or more default conditions. The system may optionally always determine some default conditions that, if they occur, cease blocking. For example, one such condition may be if another call is detected that is not a call to block non-transient storage of assistant interaction data. In some embodiments, when block 354A is not executed, the system may additionally or alternatively determine some default conditions. For example, if the spoken utterance or call does not specify a termination condition, a default termination condition may be determined, such as the passage of a default duration or the termination of the current conversation session.

[0057] At optional block 356, the system causes one or more audible and / or visually perceptible prompts to be presented at the assistant device, and optionally, at one or more additional assistant devices of the assistant device ecosystem. For example, in embodiments that also prevent storage of interactions at those assistant devices, the prompts may also be presented at the additional assistant devices. In some embodiments, the system causes the prompts to be presented continuously for the entire duration that storage is prevented (i.e., until a "yes" determination is made at block 358, as described below).

[0058] At block 358, the system determines whether to terminate the blocking of non-transient storage of assistant interaction data. The system can determine whether to terminate based on whether one or more conditions of block 354 have been satisfied. In some embodiments and / or for some conditions, the occurrence of any single condition or multiple conditions will cause the system to determine to terminate. In some other embodiments and / or for some other conditions, multiple conditions must occur simultaneously for the system to determine to terminate.

[0059] Box 358 can be performed iteratively (e.g., every second or at other frequencies). As long as the decision at box 358 is not to terminate the block, the system will prevent non-transient storage of assistant interaction data for interactions at the assistant device and optional additional assistant devices. For example, at box 362, the system can process the received spoken utterance and generate a response (audible and / or visual) and / or perform a responsive action (e.g., control a smart device), and do so without non-transient storage of assistant interaction data. One or more iterations of box 362 can be performed. For example, an iteration of box 362 can be a turn of a conversation, and depending on the termination condition, multiple turns of a conversation can occur through multiple iterations of box 362.

[0060] In some embodiments, box 362 includes sub-box 362A and / or sub-box 362B.

[0061] At sub-box 362A, the system determines whether the further received spoken utterance is from the same user who provided input at box 352, and if it is determined that it is from the same user, only non-transient storage of the corresponding assistant interaction data is blocked. Therefore, for sub-box 362A, the storage is blocked for the user who provided input at box 352, and storage for other users is not blocked. In determining whether the further received spoken utterance is from the same user, speech recognition and / or facial recognition can be performed based on the further spoken utterance, and can also be performed when receiving input at box 352. Only when these two recognitions indicate that the same user provided input at box 352 and provided a further spoken utterance at box 362, can it be determined that the further received spoken utterance is from the same user.

[0062] At sub-block 362B, the system presents all or part of the audible response using a synthesized voice generated based on a particular voice that is specifically used to prevent non-transient storage. For example, the synthesized voice may have one or more voice features that distinguish it from other synthesized voices, and the synthesized voice may be utilized only when non-transient storage of assistant interaction data is prevented, at least for the user and optionally for a group (e.g., all) of users. For example, one or more prosodic attributes of the synthesized voice may be different from all other synthesized voices used by the automated assistant for the user and / or group of users (e.g., all users). Thus, hearing such a synthesized voice may serve as a prompt to the user (in addition to or in lieu of the prompt at block 356) to inform the user that preventing storage is ongoing. This may prevent the user from unnecessarily requesting prevention of non-transient storage again and / or requesting retroactive erasure of the instance interaction (e.g., using the method described below). Figure 4 This can prevent wasting of computing resources and / or prolonging the duration of the assistant / user interaction.

[0063] If the decision at block 358 is yes in an iteration, the system proceeds to block 360. At block 360, the system re-enables non-transient storage of assistant interaction data and stops presentation of the perceptible prompt (if any) of block 356. Non-transient storage of assistant interaction data may continue until specific input is received again in another iteration of block 352.

[0064] Method 300 is shown and described as actively preventing storage of assistant interaction data for a turn of conversation in response to specific user input received at the assistant device. However, it should be noted that method 300 can be adapted to instead allow storage of assistant interaction data for a turn of conversation in response to specific user input received at the assistant device. For example, non-transient storage of assistant interaction data can be prevented by default, input to box 352 can be dedicated to allowing non-transient storage of assistant interaction data, and a termination condition determined at box 354 can indicate when storage of assistant interaction data will cease to be allowed.

[0065] Additionally, instead of or in addition to preventing non-transient storage of assistant interaction data, in some embodiments, the input of box 352 can prevent at least some (e.g., any) personal data from being used to generate at least some (e.g., all) responses and / or perform at least some (e.g., all) response actions at box 362. For example, in addition to preventing non-transient storage of assistant interaction data at box 362, box 362 can also include preventing personal data (i.e., data stored in association with and private to a user account) from being utilized in generating at least some (or even all) responses. For example, if the spoken utterance received at box 362 is "what's on my calendar", an "error" response can be provided because the user's calendar data is personal data and cannot be used to generate a response. As another example, if the spoken utterance received at box 362 is "play some music", a song can be played, but the song will be selected independently of any personal data of the user (i.e., it will not be selected based on the user's music preferences).

[0066] Figure 4 is a flow chart illustrating an example method for retroactively erasing a stored instance of assistant interaction data in response to specific user input received at an assistant device. For convenience, the operations of the flow chart are described with reference to a system performing the operations. The system may include various components of various computer systems, such as one or more components of assistant device 110 or cloud-based automated assistant component 140. Furthermore, although the operations of method 400 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0067] At box 452, the system receives audio data capturing the user's speech via the microphone of the assistant device.

[0068] At block 454 , the system performs ASR on the audio data to generate recognized text of the spoken utterance.

[0069] At block 456, the system determines, based on the recognized text of the spoken utterance, that the recognized text expresses an intent to erase assistant interaction data for one or more previous assistant interactions. For example, the system can perform NLU to determine the intent.

[0070] At block 458, the system identifies instances of assistant interaction data to be erased. Block 458 may include one or more sub-blocks 458A, 458B, and 458C.

[0071] At subbox 458A, the system identifies instances of assistant interaction data based on recognized text of the utterance. For example, the utterance can include a time period (e.g., "delete everything from the last week"), and the time period can be used to identify instances of assistant interaction data that fall within the time period. For example, a timestamp of the assistant interaction data can be utilized. Processing of the recognized text using an NLU can be performed to identify the time period (e.g., the time period can be a parameter output by the NLU processing).

[0072] At sub-block 458B, the system identifies the instance based on the instance being part of a recent conversation session. Sub-block 458B may optionally be performed in place of sub-block 458A when the system determines that the recognized text fails to specify a time period.

[0073] At sub-block 458C, the system determines the user's account and identifies the instance based on storing the instance in association with the account. In other words, the system can identify those instances of the user's interaction (from the assistant device and / or other assistant devices) that provided the spoken utterance, rather than simply all instances from the assistant device (which may include those from other users). The system can use speaker recognition, facial recognition, and / or other verification techniques to determine the user's account.

[0074] At box 462, the system erases the instance of the assistant interaction data.

[0075] At optional block 464, the system presents an output that confirms the erasure and optionally provides the number of erased instances and / or an indication of the erased instances. For example, the output may identify the number of erased conversation turns and / or conversation sessions. As another example, the output may identify the topic and / or other summary of the erased conversation turns and / or conversation sessions.

[0076] In various embodiments, examples of assistant interaction data described herein may include, for example, audio data capturing spoken input, text transcriptions thereof (e.g., generated using a speech-to-text model), natural language understanding (NLU) data based on the text transcriptions (e.g., intents (e.g., "today's weather") and / or parameters (e.g., the city of Louisville), and / or data characterizing assistant responses (e.g., the response itself and / or the subject of the response). Examples of assistant interaction data may additionally or alternatively include images, videos, and / or other visual data captured during a conversation session and / or data based on processing such image data. For example, during a conversation session with an assistant device with a camera, a user may hold an object in front of the camera and provide a spoken utterance of "what is this". The automated assistant may process the visual data using a neural network model to generate a predicted classification of the object and generate a response to the spoken utterance based on the predicted classification. In such an example, the assistant interaction data may include the visual data and / or the predicted classification. Examples of assistant interaction data may additionally or alternatively include audio data that does not capture the user's spoken utterance, and / or other data based on processing such audio data. For example, during a conversation session via an assistant device, a user may provide a verbal utterance of "what is this". In such examples, the assistant interaction data may include the audio data and / or the identified song.

[0077] Embodiments of general automated assistants have been described herein, through which users can participate in various conversations to complete various tasks. For example, users can use automated assistants to control smart devices, play music, make calls, create calendar entries, create reminders, query weather, and / or perform additional and / or alternative tasks across other verticals. In some embodiments, the various techniques described herein can be used in combination with one or more non-automated assistant systems and / or with automated assistants dedicated to interacting with users only for a single task or only for a single vertical. For example, a device and / or application can be dedicated to receiving impaired speech, processing impaired speech to generate synthesized unimpaired speech, and presenting synthesized unimpaired speech. With such an example, non-transient storage of interactive data (e.g., impaired speech and / or unimpaired speech from a conversation turn) can be selectively allowed and / or prohibited depending on how the device and / or application are called. Additionally or alternatively, for such an example, any stored interactive data can be additionally or alternatively erased retroactively in response to specific user input received at a device or application.

[0078] Reference now Figure 5 , a block diagram of an example computing device 510 is shown, which can optionally be used to perform one or more aspects of the technology described herein. In some embodiments, one or more of the assistant device, cloud-based automated assistant components, and / or other components can include one or more components of the example computing device 510.

[0079] The computing device 510 typically includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524, including, for example, a memory subsystem 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow a user to interact with the computing device 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0080] User interface input devices 522 may include a keyboard, a pointing device such as a mouse, a trackball, a touch pad or graphics tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into computing device 510 or onto a communication network.

[0081] User interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanisms for creating a visual image. The display subsystem may also provide a non-visual display, for example, through an audio output device. Generally speaking, the use of the term "output device" is intended to include devices and modes of all possible types of information output from computing device 510 to a user or to another machine or computing device.

[0082] The storage subsystem 524 stores programs and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 524 may include logic to perform selected aspects of the methods disclosed herein and to implement various components.

[0083] These software modules are typically executed by the processor 514 alone or in conjunction with other processors. The memory subsystem 525 used in the storage subsystem 524 may include multiple memories, including a main random access memory (RAM) 510 for storing instructions and data during program execution and a read-only memory (ROM) 512 for storing fixed instructions. The file storage subsystem 526 may provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of a particular embodiment may be stored in the storage subsystem 524 by the file storage subsystem 526, or in other machines accessible to the processor 514.

[0084] The bus subsystem 512 provides a mechanism for the various components and subsystems of the computing device 510 to communicate with each other as intended. Although the bus subsystem 512 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0085] Computing device 510 can be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5 The description of computing device 510 depicted in FIG. 5 is intended only as a specific example for illustrating some embodiments. Many other configurations of computing device 510 may have more Figure 5 The computing devices depicted may have greater or fewer components.

[0086] Where the systems described herein collect or otherwise monitor personal information about a user, or where personal and / or monitored information (e.g., messages exchanged in a message exchange thread) may be utilized, the user may be provided with an opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, preferences of the user, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from a content server. In addition, before storing or using specific data, it may be processed in one or more ways to remove personally identifiable information. For example, the identity of the user may be processed so that the user's identity information cannot be determined, or the geographic location of the user may be summarized (e.g., city, zip code, or state level) where geographic location information is obtained so that the user's specific geographic location cannot be determined. Thus, the user may control how information about the user is collected and / or used. For example, in some embodiments, a message exchange thread participant operating a message exchange client configured with selected aspects of the present disclosure may choose not to monitor message exchange thread content, for example, so that a participant who desires to transition a message exchange client to a private state or out of a private state may do so manually.

[0087] In some embodiments, a method is provided, the method comprising processing audio data using a speech recognition model to generate recognized text of a user's spoken utterance captured in the audio data. The audio data is detected via one or more microphones of an assistant device. The method also includes determining that the recognized text expresses an intention to erase assistant interaction data of one or more previous assistant interactions of the user. The method also includes determining an account associated with the user providing the spoken utterance. The method also includes, in response to determining that the recognized text expresses an intention to erase the assistant interaction data: identifying one or more instances of assistant interaction data for the user in one or more computer-readable media, based on storing the one or more instances in association with the determined account; and erasing the identified one or more instances of assistant interaction data from the one or more computer-readable media.

[0088] These and other implementations of the technology disclosed herein may include one or more of the following features.

[0089] In some embodiments, the method further includes determining that the recognized text fails to indicate any time period for erasing the assistant interaction data. In those embodiments, identifying one or more instances of assistant interaction data further includes: in response to determining that the recognized text fails to indicate any time period for erasing the assistant interaction data: further identifying one or more instances of assistant interaction data based on determining that each of the one or more instances is part of a recent conversation session between the user and the automated assistant. In some versions of these embodiments, the one or more instances of assistant interaction data include a first instance and a second instance, and the method further includes determining that each of the one or more instances is part of a recent conversation session between the user and the automated assistant. Determining that each of the one or more instances is part of a recent conversation session between the user and the automated assistant may include: determining that a first timestamp stored in association with the first instance is within a threshold duration of a second timestamp stored in association with the second instance; and comparing the current time with the first timestamp and / or the second timestamp. In some of these versions, determining that each of the one or more instances is part of a recent conversation session between the user and the automated assistant is also based on determining that a first topic stored in association with the first instance is within a threshold semantic distance of a second topic stored in association with the second instance. For example, a first embedding may be generated for a first descriptor of a first topic (e.g., using Word2Vec or other semantic embedding models), a second embedding may be generated for a second descriptor of a second topic, and the semantic distance may be the distance between the first embedding and the second embedding in the embedding space. As another example, the semantic distance may be based on how many edges there are in the shortest path between the first topic and the second topic in a knowledge graph or other classification structure. Optionally, the first instance includes: first audio data capturing a first utterance of the user, a first transcription of the first utterance of the user, first natural language understanding data generated based on the first transcription, and / or first response data representing an assistant response to the first utterance. Optionally, the second instance includes: second audio data capturing a second utterance of the user, a second transcription of the second utterance of the user, second natural language understanding data generated based on the second transcription, and / or second response data representing an assistant response to the second utterance.

[0090] In some embodiments, determining an account associated with a user providing the spoken utterance includes: performing speaker recognition using the audio data and / or call audio data immediately preceding the audio data; and determining the account associated with the user based on performing the speaker recognition. In some versions of these embodiments, performing speaker recognition includes generating an embedding at the assistant device based on processing the audio data and / or call audio data. In some of these versions, determining the account associated with the user based on performing speaker recognition includes matching the embedding with a pre-stored embedding stored locally at the assistant device, and determining that the matched pre-stored embedding is stored in association with the account.

[0091] In some embodiments, the method further includes, after identifying one or more instances of assistant interaction data: causing an audible output and / or visual output to be presented at the assistant device, the audible output and / or visual output confirming the erasure of the identified one or more instances of assistant interaction data from the one or more computer-readable media. In some versions of those embodiments, the method further includes generating an audible output and / or visual output to include: a number of the one or more instances of the assistant interaction data; and / or a corresponding indication for each instance of assistant interaction data. In some of these versions, the corresponding indication for each instance of assistant interaction data includes a corresponding topic for each instance of assistant interaction data.

[0092] In some embodiments, each instance of assistant interaction data includes data from a corresponding single turn of the user-assistant conversation.

[0093] In some embodiments, the method further includes determining that the recognition text indicates a time period for erasing assistant interaction data. In some of these embodiments, in response to determining that the recognition text indicates a time period for erasing assistant interaction data, identifying one or more instances of assistant interaction data further includes: further identifying the one or more instances of assistant interaction data based on determining that the corresponding timestamps stored in association with the one or more instances all fall within the time period for erasing assistant interaction data. In some versions of these embodiments, determining the account associated with the user providing the spoken utterance includes: performing speaker recognition using audio data and / or call audio data immediately preceding the audio data; and determining the account associated with the user based on performing speaker recognition.

[0094] In some embodiments, a given instance of the one or more instances of assistant interaction data corresponds to a conversation turn that causes a future action to be queued for execution at a future time before the current time. In some of these embodiments, erasing the identified one or more instances of assistant interaction data from one or more computer-readable media occurs without affecting the execution of the future action at the future time. In some versions of these embodiments, the future action includes controlling a smart device and / or causing a notification to be presented.

[0095] In some embodiments, the user input is a particular touchless gesture detected in one or more visual frames generated by a camera of the assistant device, and detection of an alternative touchless gesture enables non-transient storage of assistant interaction data for at least one assistant interaction immediately following the alternative touchless gesture.

[0096] In some embodiments, causing the assistant response to be presented at the assistant device includes causing an audible portion of the assistant response to be presented in a particular synthesized voice. Causing the audible portion to be presented in the particular synthesized voice is responsive to determining to prevent non-transient storage of the assistant interaction data. The particular synthesized voice is used only when, at least to the user, non-transient storage of the assistant interaction data is prevented.

[0097] In some embodiments, the method further comprises determining, based on the user input, to additionally prevent use of any personal data in generating the assistant response until one or more termination conditions are satisfied. In some of these embodiments, determining the assistant response in response to the spoken utterance comprises, in response to determining to additionally prevent use of any personal data in generating the assistant response, determining the assistant response independent of any personal data.

[0098] In some embodiments, a method is provided, the method comprising determining to prevent non-transient storage of assistant interaction data based on user input received at an assistant device until one or more termination conditions are met. The method also comprises: in response to determining to prevent non-transient storage of assistant interaction data, and until the one or more termination conditions are met: causing the assistant device to present a visually perceptible and / or audibly perceptible prompt. The method also comprises, while preventing non-transient storage of assistant interaction data and before the one or more termination conditions are met: processing audio data using a speech recognition model to generate a recognition text of a user's spoken utterance captured in the audio data, wherein the audio data is detected via one or more microphones of the assistant device; based on the recognition text, determining an assistant response in response to the spoken utterance; causing the assistant response to be presented at the assistant device; and in response to determining to prevent non-transient storage of assistant interaction data, preventing non-transient storage of audio data, recognition text, and / or assistant response. The method also comprises allowing non-transient storage of assistant interaction data in response to determining that the one or more termination conditions are met.

[0099] These and other implementations of the technology disclosed herein may include one or more of the following features.

[0100] In some embodiments, the assistant device is one of a plurality of assistant devices in an assistant device ecosystem associated with the user. In some of these embodiments, the method further includes, in response to determining to prevent non-transient storage of assistant interaction data, preventing non-transient storage of any assistant interaction data from interactions with any assistant device in the ecosystem. In some versions of those embodiments, the method further includes: in response to determining to prevent non-transient storage of assistant interaction data, and until the one or more termination conditions are met: causing each of the plurality of assistant devices other than the assistant device to present a corresponding visually perceptible and / or audibly perceptible prompt. In some of these versions, causing each of the plurality of assistant devices other than the assistant device to present a corresponding visually perceptible and / or audibly perceptible prompt includes transmitting, by the assistant device to the plurality of assistant devices other than the assistant device, a command causing the presentation of a corresponding visually perceptible and / or audibly perceptible prompt. In some other versions of those versions, causing each of the multiple assistant devices except the assistant device to present a corresponding visually perceptible and / or audibly perceptible prompt includes transmitting, by a remote assistance server in communication with the assistant device, to the multiple assistant devices except the assistant device, a command causing the corresponding visually perceptible and / or audibly perceptible prompt to be presented.

[0101] In some embodiments, non-transient storage of assistant interaction data is prevented only for an account associated with the user providing the user input. In some of these embodiments, the method further comprises: determining an account associated with the user providing the user input for which non-transient storage is prevented; and preventing non-transient storage of the audio data, recognized text, and / or assistant response is further responsive to determining that the spoken input matches a speaker embedding stored for the account.

[0102] In some embodiments, the one or more termination conditions include the passage of a threshold amount of time and / or completion of the dialog session following user input.

[0103] In some embodiments, the user input is actuation of a hardware or software button in a given manner, and optionally, actuation of the hardware or software button in an alternative manner causes non-transient storage of assistant interaction data for at least one assistant interaction immediately following the actuation in the alternative manner.

[0104] In some embodiments, the user input is a spoken specific assistant invocation phrase, and optionally, the alternate spoken assistant invocation phrase pattern causes non-transient storage of assistant interaction data for at least one assistant interaction immediately following the alternate spoken assistant invocation phrase.

[0105] In some embodiments, a method implemented by one or more processors is provided, the method including determining, based on user input received at an assistant device, to prevent the use of any personal data when generating an assistant response until one or more termination conditions are met. The method also includes, before the one or more termination conditions are met: processing audio data using a speech recognition model to generate a recognized text of the user's spoken utterance captured in the audio data; based on the recognized text, determining an assistant response in response to the spoken utterance; and causing the assistant response to be presented at the assistant device. Determining the assistant response in response to the spoken utterance includes, in response to determining that any personal data is additionally prevented from being used when generating the assistant response, determining an assistant response independent of any personal data. The method also includes, in response to determining that the one or more termination conditions are met: allowing the use of personal data to determine the assistant response.

[0106] These and other implementations of the technology disclosed herein may include one or more of the following features.

[0107] In some embodiments, causing the assistant response to be presented at the assistant device includes causing an audible portion of the assistant response to be presented in a particular synthesized voice. Causing the audible portion to be presented in the particular synthesized voice is responsive to determining to prevent non-transient storage of assistant interaction data. Optionally, at least to the user, the particular synthesized voice is used only when non-transient storage of assistant interaction data is prevented.

[0108] In some embodiments, a method implemented by one or more processors is provided, the method comprising determining, based on user input received at an assistant device, to prevent non-transient storage of assistant interaction data until one or more termination conditions are met. The method also includes, while preventing non-transient storage of assistant interaction data and before the one or more termination conditions are met: processing audio data detected via a microphone of the assistant device using a speech recognition model to generate recognized text of the user's spoken utterance captured in the audio data; based on the recognized text, determining an assistant response in response to the spoken utterance; causing the assistant response to be presented at the assistant device; and in response to determining to prevent non-transient storage of assistant interaction data, preventing non-transient storage of audio data, recognized text, and / or assistant response. The method also includes, in response to determining that the one or more termination conditions are met: allowing non-transient storage of assistant interaction data.

[0109] In addition, some embodiments include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause execution of any of the methods disclosed herein. Some embodiments include at least one non-transitory computer-readable storage medium storing computer instructions that can be executed by one or more processors to perform any of the methods disclosed herein.< / topping> < / topping> < / artist> < / artist> < / artist> < / artist>

Claims

1. A method implemented by one or more processors, the method comprising: processing audio data using a speech recognition model to generate recognized text of a user's spoken utterance captured in the audio data, wherein the audio data is detected via one or more microphones of the assistant device; determining that the recognized text expresses an intent to erase assistant interaction data for one or more previous assistant interactions of the user; determining an account associated with the user providing the spoken utterance; In response to determining that the recognized text expresses an intention to erase the assistant interaction data: identifying, for the user in one or more computer-readable media, the one or more instances of assistant interaction data based on the one or more instances being stored in association with the determined account; as well as The identified one or more instances of assistant interaction data are erased from the one or more computer-readable media.

2. The method according to claim 1, further comprising: determining that the recognized text fails to indicate any time period for erasing the assistant interaction data; Wherein, identifying one or more instances of the assistant interaction data further comprises, in response to determining that the recognized text fails to indicate any time period for erasing the assistant interaction data: One or more instances of the assistant interaction data are identified further based on determining that each of the one or more instances is part of a recent conversational session between the user and the automated assistant.

3. The method according to claim 2, wherein: The one or more instances of assistant interaction data include a first instance and a second instance, and further include: Determining that each of the one or more instances is part of a most recent conversation session between the user and the automated assistant, wherein determining that each of the one or more instances is part of a most recent conversation session between the user and the automated assistant comprises: determining that a first timestamp stored in association with the first instance is within a threshold duration of a second timestamp stored in association with the second instance; and A current time is compared with the first timestamp and / or the second timestamp.

4. The method according to claim 2, wherein: Determining that each of the one or more instances is part of a most recent conversation session between the user and the automated assistant is further based on: A determination is made that a first topic stored in association with the first instance is within a threshold semantic distance of a second topic stored in association with the second instance.

5. The method according to claim 3, in, The first instance includes: first audio data capturing a first utterance of the user; a first transcription of the first utterance of the user; first natural language understanding data generated based on the first transcription; and / or first response data characterizing an assistant response to the first utterance; and Wherein, the second instance includes: second audio data, which captures a second utterance of the user; a second transcription of the second utterance of the user; second natural language understanding data, which is generated based on the second transcription; and / or second response data, which represents the assistant's response to the second utterance.

6. The method according to claim 1, wherein: Determining an account associated with the user providing the spoken utterance includes: performing speaker recognition using the audio data and / or the preceding audio data; and An account associated with the user is determined based on performing the speaker recognition.

7. The method according to claim 6, in, Performing the speaker recognition includes generating an embedding based on processing the audio data and / or the call audio data at the assistant device; and Wherein determining the account associated with the user based on performing the speaker recognition includes matching the embedding with a pre-stored embedding stored locally at the assistant device, and determining that the matched pre-stored embedding is stored in association with the account.

8. The method of claim 1, further comprising, after identifying one or more instances of the assistant interaction data: Causing presentation of an audible output and / or a visual output at the assistant device that confirms erasing the identified one or more instances of assistant interaction data from the one or more computer-readable media.

9. The method according to claim 8, further comprising: generating the audible output and / or the visual output to include: the number of the one or more instances of the assistant interaction data; and / or A corresponding indication of each of the instances of the assistant interaction data.

10. The method according to claim 9, wherein: The corresponding indication for each of the instances of the assistant interaction data includes a corresponding topic for each of the instances of the assistant interaction data.

11. The method according to claim 1, wherein: Each of the instances of the assistant interaction data includes data from a corresponding single user-assistant conversation turn.

12. The method according to claim 1, further comprising: determining a time period during which the recognized text indicates erasure of the assistant interaction data; Wherein, identifying the one or more instances of the assistant interaction data further comprises, in response to determining that the recognized text indicates erasing the time period of the assistant interaction data: The one or more instances of the assistant interaction data are further identified based on determining that corresponding timestamps stored in association with the one or more instances all fall within the time period for erasing the assistant interaction data.

13. The method according to claim 12, wherein: Determining an account associated with the user providing the spoken utterance includes: performing speaker recognition using the audio data and / or the preceding audio data; and An account associated with the user is determined based on performing the speaker recognition.

14. The method according to any one of claims 1 to 13, wherein: A given instance of the one or more instances of the assistant interaction data corresponds to a conversation turn that queues a future action for execution at a future time prior to a current time, and wherein an occurrence of erasing the identified one or more instances of the assistant interaction data from the one or more computer-readable media does not affect execution of the future action at the future time.

15. The method according to claim 14, wherein: The future action includes controlling the smart device and / or causing presentation of a notification.

16. A method implemented by one or more processors, the method comprising: determining, based on user input received at the assistant device, to prevent utilization of any personal data in generating an assistant response until one or more termination conditions are satisfied; Before the one or more termination conditions are met: processing audio data using a speech recognition model to generate recognized text of a user's spoken utterance captured in the audio data, wherein the audio data is detected via one or more microphones of the assistant device; determining an assistant response in response to the spoken utterance based on the recognized text, wherein determining the assistant response in response to the spoken utterance comprises: determining the assistant response independently of any personal data in response to determining that utilization of any personal data in generating the assistant response is additionally prevented; and causing the assistant response to be presented at the assistant device; and In response to determining that the one or more termination conditions are satisfied: Enables the use of personal data to determine assistant responses.

17. The method according to claim 16, wherein: Causing presentation of the assistant response at the assistant device includes: Cause an audible portion of the assistant response to be presented in a particular synthesized voice, wherein causing the audible portion to be presented in the particular synthesized voice is responsive to determining that non-transient storage of assistant interaction data is prevented, and wherein, at least to the user, the particular synthesized voice is utilized only when non-transient storage of assistant interaction data is being prevented.

18. A method implemented by one or more processors, the method comprising: determining, based on user input received at the assistant device, to prevent non-transitory storage of the assistant interaction data until one or more termination conditions are satisfied; While preventing non-transient storage of the assistant interaction data, and before satisfying the one or more termination conditions: processing audio data using a speech recognition model to generate recognized text of a user's spoken utterance captured in the audio data, wherein the audio data is detected via one or more microphones of the assistant device; determining an assistant response in response to the spoken utterance based on the recognized text; causing the assistant response to be presented at the assistant device; and In response to determining to prevent non-transient storage of the assistant interaction data, preventing non-transient storage of the audio data, the recognized text, and / or the assistant response; and In response to determining that the one or more termination conditions are satisfied: Allows non-transient storage of assistant interaction data.

19. A device comprising one or more processors and a memory, the memory being operably coupled to the one or more processors, wherein the memory stores instructions, and in response to execution of the instructions by the one or more processors, the instructions cause the one or more processors to perform the method according to any one of claims 1 to 18.

20. At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 18.

Images