Encryption and / or decryption of audio data using speaker features

Abstract

Implementations relate to encrypting audio data with utterance features generated from the audio data. Some of those implementations include generating an utterance feature from a portion of the audio data, encrypting at least a portion of the audio data using the utterance feature, and providing the encrypted audio data to one or more applications for decryption. The audio data is decrypted for further processing using speaker features previously generated from the utterance. Other implementations relate to receiving a speaker feature and comparing the speaker feature to an utterance feature generated from audio data. The audio data is provided to a target application that provides speaker features that match utterance features generated from the audio data.

Description

Background Art

[0001] A human may engage in a human-computer conversation with an interactive software application referred to herein as an "automated assistant" (also referred to as a "digital agent," "chatbot," "interactive personal assistant," "intelligent personal assistant," "assistant application," "conversational agent," etc.). For example, a human (who may be referred to as a "user" when they interact with an automated assistant) may provide commands and / or requests to the automated assistant using spoken natural language input (i.e., utterances) and / or by providing textual (e.g., typed) natural language input, which in some cases may be converted to text and then processed. The automated assistant responds to the request by providing responsive user interface output, which may include audible and / or visual user interface output.

[0002] As described above, many automated assistants are configured to interact via spoken utterances (such as an invocation instruction followed by a spoken query). To protect user privacy and / or conserve resources, the user must typically explicitly invoke the automated assistant, which will then fully process the spoken utterance. The explicit invocation of the automated assistant typically occurs in response to receiving some user interface input at a client device. The client device includes an assistant interface that provides an interface to a user of the client device for interfacing with the automated assistant (e.g., receiving spoken and / or typed input from the user, and providing audible and / or graphical responses), and interfaces with one or more additional components that implement the automated assistant (e.g., a remote server device that processes the user input and generates an appropriate response).

[0003] Certain user commands or requests will only be fully processed and responded to if the automated assistant authenticates the user making the request. For example, in order to maintain the security of personal data, user authentication may be required for requests that access the user's personal data when generating a response and / or incorporate the user's personal data in the response. For example, in order to appropriately respond to requests such as "what's on my calendar for tomorrow" (which requires access to the requesting user's personal calendar data and inclusion of the personal calendar data in the response) and "message Vivian I'm running late" (which requires access to the requesting user's personal contact data). As another example, in order to maintain the security of smart devices (e.g., smart thermostats, smart lights, smart locks), user authentication may be required for requests that cause control of one or more of such smart devices.

[0004] Various techniques for user authentication of automated assistants have been utilized. For example, when authenticating a user, some automated assistants utilize text-dependent techniques (TD) that are constrained to an invocation phrase for the assistant (e.g., "OK Assistant" and / or "Hey Assistant"). Using such techniques, a registration process is performed in which the user is explicitly prompted to provide one or more instances of spoken utterances of the invocation phrase to which the TD features are constrained. Speaker features (e.g., speaker embeddings) for the user can then be generated by processing instances of audio data, where each instance in the instances captures a corresponding one of the spoken utterances in the spoken utterances. For example, speaker features can be generated by processing each instance in the instances of audio data using a TD machine learning model to generate a corresponding speaker embedding for each utterance in the utterance. Speaker features can then be generated as a function of the speaker embeddings, and the speaker features can be stored (e.g., on a device) for use in TD techniques. For example, the speaker features can be cumulative speaker embeddings that are functions of the speaker embeddings (e.g., averages). It has also been proposed to utilize text-independent (TI) techniques as a supplement or alternative to TD techniques. TI features are not constrained to a subset of phrases as in TD. Like TD, TI can also exploit the user's speaker characteristics, and those features can be generated based on user utterances obtained through an enrollment process and / or other verbal interactions, although more instances of user utterances may be needed to generate useful TI speaker features.

[0005] After the speaker features are generated, the speaker features can be used to verify whether the spoken utterance was spoken by the user. For example, when the user speaks another spoken utterance, the audio data that captures the spoken utterance can be processed to generate utterance features, those utterance features are compared to the speaker features, and based on the comparison, it is determined whether to authenticate the speaker. As a specific example, a speaker recognition model can be used to process the audio data to generate an utterance embedding, and the utterance embedding can be compared to the user's previously generated speaker embedding to determine whether to verify the user as the speaker of the spoken utterance. For example, if the distance metric between the generated utterance embedding and the user's speaker embedding meets a threshold, the user can be verified to be the user who spoke the spoken utterance. Such verification can be used as a criterion (e.g., a unique criterion) for authenticating a user.

[0006] Accordingly, various TD and / or TI methods have been proposed for use by automated assistants and / or other applications to determine whether and / or how to process audio data that captures spoken utterances and is provided to the application for processing. Summary of the invention

[0007] Implementations disclosed herein relate to ensuring the security of audio data that captures a user's spoken utterances. For example, various implementations utilize utterance features and / or speaker features (e.g., generated using TD and / or TI methods) to ensure that an unencrypted version of the audio data is accessible only to components (e.g., processors, operating systems, and / or applications) that the user has authorized to access and / or process their spoken utterances.

[0008] Some implementations disclosed herein relate to generating speech features from audio data including one or more terms spoken by a user, and utilizing the speech features to encrypt the audio data. For example, a speech feature can be generated based on at least processing a portion of the audio data including at least a portion of the spoken speech, and the speech feature can be used as an encryption key for encrypting the portion of the audio data. For example, TD technology can be used to process a call portion of the audio data to generate a TD speech feature, and the TD speech feature is used to encrypt a portion of the audio data, which includes an additional portion of the spoken speech before and / or after the call portion. Some of those implementations include processing a stream of audio data to monitor the occurrence of a call phrase. In response to detecting the occurrence of a call phrase during monitoring, a portion of the audio data (e.g., audio data including at least a portion of the spoken speech before, after, and / or including the call phrase) is encrypted, and the encrypted audio data is then provided to one or more components for further processing. Only components that can access speaker features derived from one or more features of a previous spoken speech previously spoken by the same user providing the speech can decrypt the encrypted audio data in order to process the spoken speech and / or other features of the decrypted audio data. In these and other ways, only components that have access to the user's speaker characteristics can decrypt the encrypted audio data and generate an unencrypted version of the audio data, thereby ensuring the security of the audio data that captures the user's spoken utterances. As a specific example, if the encrypted audio data is provided to a particular application (e.g., an automated assistant application or other application), the particular application can decrypt the encrypted audio data only if the user has previously provided explicit or implicit authorization for the particular application to process their spoken utterances. For example, a particular application may have access to the user's speaker characteristics that can be used as a decryption key, and may have such access rights as a result of the user performing a registration process with the application, granting the application access to speaker characteristics generated by a registration process performed by another application and / or an operating system, and / or granting the application access rights to process utterances while the application is in use.

[0009] In some implementations, a digital signal processor (DSP) may process audio data of a limited length (e.g., audio data including a detected 10-second utterance, a set file length, a variable but limited file length) that is stored in a buffer that only the DSP has access to. When the DSP detects an occurrence of an utterance in the audio data included in the buffer, it may process the audio data to generate an utterance feature indicating a speaker who has spoken the utterance. For example, the DSP may generate a vector indicating a voice profile of a user who spoke a phrase included in the audio data. The DSP may generate features based on a portion of the utterance that includes an invocation phrase (e.g., a portion of the audio data that includes the user saying "Hey Assistant") to generate a vector indicating that the user spoke the phrase, and / or generate features based on a portion of the audio data before and / or after the invocation phrase ("Turn on the kitchen lights" after "Hey Assistant" in the utterance) to generate an utterance feature vector for a speaker who spoke a non-invocation phrase portion of the audio data.

[0010] In some implementations, only a portion of the available audio data may be used to generate speech features for the audio data. For example, in some implementations, the audio data may include audio captured by multiple microphones. In those cases, the DSP may only utilize a subset of the audio channels to generate speech features that may be used to encrypt the entire audio data. Thus, by not requiring the DSP (which may have limited computing power and / or resources) to process all captured audio data, computing resources may be conserved.

[0011] In some implementations, the stored vectors and / or invocation signatures can be used to encrypt audio data preceding and / or following an invocation phrase. For example, a user can utter the phrase "Hey Assistant, turn on the lights," and the DSP can generate an invocation signature from the portion of the audio data that includes "Hey Assistant." The generated signature can then be used to encrypt the portion of the audio data that includes the user uttering "turn on the lights," which the DSP can then provide to one or more applications without providing unencrypted audio data.

[0012] Once the encrypted audio data is provided, one or more applications can process the encrypted audio data to determine whether the application can decrypt the audio data based on the stored features of the user (or other users) who spoke the phrase. In the case where the application has access to one or more vectors of the user who spoke the invocation phrase, for example, the vector can be used as a key to decrypt the audio data. If the application can decrypt the audio data, then it can further process the audio data (e.g., perform automatic speech recognition, natural language processing, speech to text processing) to perform one or more actions. If the application cannot decrypt the audio data because, for example, the application cannot access the vector that successfully decrypted the audio data, then the data can remain secure and the application will not be able to determine what is further included in the audio data.

[0013] Thus, the implementations disclosed herein alleviate the need to determine which applications to provide spoken utterances to, thereby reducing the processing time and power consumption required by the DSP. In addition, the implementations described herein improve security measures by reducing access to spoken utterances by applications to which the user may be interested in not providing audio data. Thus, the implementations described herein reduce the computational requirements of components (e.g., DSPs) that may have limited computing resources, while maintaining the security of audio data received from the user.

[0014] In some implementations, features determined based on the invocation phrase and / or other portions of the spoken utterance may not completely match the stored vectors representing the user speaker speaking the utterance. For example, due to differences in the terms spoken in the spoken phrase, the prosodic characteristics (e.g., pitch, speech rate) of the user speaking the invocation phrase, the presence of background noise, and / or other factors, the invocation features generated from the spoken utterance may not completely match the stored features accessible to the application receiving the encrypted audio data. In those cases, the application may decrypt the spoken utterance using one or more algorithms that can be decrypted using fuzzy matching (e.g., vectors within a threshold distance to the stored vectors). For example, in some cases, an utterance feature vector generated from at least a portion of the audio data may not completely match a speaker feature vector previously generated for the user, even though the two vectors may include the same speaker speaking the same phrase. However, using one or more fuzzy matching techniques, the stored speaker feature vectors may be used to decrypt the audio data even if the audio data was encrypted using non-matching vectors.

[0015] In some implementations, an obfuscated version of a portion of the audio data may be generated prior to encrypting the audio data. For example, prior to encrypting the audio data, the audio data may first be processed to reduce background noise, omit at least a portion of the audio data (e.g., omit a portion including sensitive information), and / or otherwise processed to limit the audio data provided to the application. Thus, after decryption, the application may still not be able to access all of the information included in the audio data, thereby further improving the security of the user by limiting access to potentially sensitive information that may be included in the audio data.

[0016] Other implementations disclosed herein relate to selectively providing audio data to one or more applications based on determining that a call feature provided by the application matches a feature generated from a spoken utterance (either a complete match or, using a fuzzy matching algorithm, an approximate match). For example, the DSP may process audio data including a spoken utterance of a user to generate features indicative of the spoken utterance. In response to identifying a call phrase included in the utterance, the DSP may provide a notification that audio data is available to one or more applications. One or more applications may provide a call feature (e.g., a previously stored vector), and the DSP may compare the provided vector with a vector generated based on the spoken utterance. In the event that the generated vector (or other feature) matches the provided vector (or other feature), the DSP may provide the audio data (or an obfuscated version of the audio data) to the application that provided the matching vector. Thus, the audio data is not encrypted, but is provided only to the application that provides a vector that matches the vector generated from the spoken utterance. By selectively providing the audio data only to those applications that can utilize the audio data, the resource consumption required to transmit the audio data to the application is reduced, thereby improving the performance of the computing device.

[0017] The above description is provided as an overview of only some of the implementations disclosed herein. Those implementations and other implementations are described in more detail herein.

[0018] It should be understood that all combinations of the aforementioned concepts and additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Depicted is a block diagram of an example environment showing various aspects of the disclosure and in which implementations disclosed herein may be implemented.

[0020] Figure 2A An example of text-dependent speaker verification in accordance with one or more implementations disclosed herein is depicted.

[0021] Figure 2B Another example of text-dependent speaker verification in accordance with one or more implementations disclosed herein is depicted.

[0022] Figure 3A A flow chart illustrating a method of providing encrypted audio data to one or more applications according to one or more implementations disclosed herein.

[0023] Figure 3B A flow chart illustrating a method for decrypting audio data using a user's speech features.

[0024] Figure 4A A method of providing audio data to one or more applications in response to receiving speaker characteristics from the one or more applications is illustrated.

[0025] Figure 4B A method of receiving audio data in response to providing speaker characteristics of a user is illustrated.

[0026] Figure 5 An example architecture is illustrated that can be used to implement one or more of the methods described herein. DETAILED DESCRIPTION

[0027] First go to Figure 1 , illustrating an example environment in which various implementations may be performed. Figure 1 An assistant device 100 (i.e., a client device executing an automated assistant client and / or via which an automated assistant may be otherwise accessed) is included, which executes an instance of an automated assistant client 120. One or more cloud-based automated assistant components may be implemented on one or more computing systems (collectively, "cloud" computing systems) that are communicatively coupled to assistant device 100 via one or more local area networks and / or wide area networks (e.g., the Internet). An instance of automated assistant client 120, optionally via interaction with one or more cloud-based automated assistant components, may form what appears to be a logical instance of an automated assistant from a user's perspective, with which the user may engage in a human-computer dialogue.

[0028] Assistant device 100 may be, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a vehicle computing device (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker, a smart appliance such as a smart TV, and / or a wearable device including a computing device (e.g., a watch with a computing device, glasses with a computing device, a virtual or augmented reality computing device).

[0029] The assistant device 100 can be used by one or more users in a home, business, or other environment. In addition, one or more users can register with the assistant device 100 and have corresponding user accounts that can be accessed via the assistant device 100. With the permission of the associated user, the text-related (TD) speaker features described herein can be generated and stored for each of the registered users (e.g., associated with their corresponding user profiles). For example, a TD speaker feature vector constrained to the term "OK Assistant" can be generated, which can be stored in association with the first registered user and has corresponding speaker features specific to the first registered user - and the TD speaker features constrained to the term "OK Assistant" can be stored in association with the second registered user and have corresponding speaker features specific to the second registered user. The TD technology described herein can be used to authenticate the speech as coming from a specific user (rather than from another registered user or a guest user). Optionally, TI technology, speaker verification, facial verification, and / or other verification technologies (e.g., PIN entry) can be used additionally or alternatively to authenticate a specific user. Furthermore, encryption and / or decryption of audio data may be performed utilizing speaker features generated via TD and / or TI techniques, as further described herein.

[0030] Additional and / or alternative assistant devices may be provided, and in some of those implementations, speaker characteristics of a particular user may be shared among assistant devices for which the user is a registered user. In various implementations, assistant device 100 may optionally operate one or more other applications (e.g., applications 130 and 140) in addition to automated assistant 120, such as a messaging client (e.g., SMS, MMS, online chat), a browser, etc. In some of those various implementations, one or more of the other applications may optionally interface with automated assistant 100 (e.g., via an application programming interface), or include their own instances of automated assistant applications (which may also interface with any cloud-based automated assistant components).

[0031] The automated assistant 120 conducts a human-computer dialogue session with the user via the user interface input and output devices of the assistant device 100. In order to protect user privacy and / or save resources, in many cases, the user must usually explicitly call the automated assistant 120 before the automated assistant will fully process the spoken utterance. The explicit call to the automated assistant 100 may occur in response to a certain user interface input received at the assistant device 100. For example, the user interface input that can call the automated assistant 100 via the assistant device 100 may optionally include the actuation of the hardware and / or virtual buttons of the assistant device 100. In addition, the automated assistant client may include one or more local engines, such as a call engine that is operable to detect the presence of one or more verbal general call wake-up words. The call engine may call the automated assistant 120 in response to detecting one of the verbal call wake-up words. For example, the call engine may call the automated assistant 100 in response to detecting a verbal call wake-up word such as "HeyAssistant", "OK Assistant" and / or "Assistant". The invocation engine may continuously process (e.g., when not in an "inactive" mode) a stream of audio data frames based on the output of one or more microphones of the assistant device 100 to monitor for the occurrence of a spoken invocation phrase. While monitoring for the occurrence of a spoken invocation phrase, the invocation engine discards (e.g., after temporary storage in a buffer) any audio data frames that do not include a spoken invocation phrase. However, when the invocation engine detects the occurrence of a spoken invocation phrase in a processed audio data frame, the invocation engine may invoke the automated assistant 120. As used herein, "invoking" the automated assistant 120 may include causing one or more previously inactive functions of the automated assistant 120 to be activated. For example, invoking the automated assistant 120 may include causing one or more local engines and / or cloud-based automated assistant components to further process an audio data frame based on which an invocation phrase is detected, and / or one or more subsequent audio data frames (whereas no further processing of the audio data frames occurred prior to the invocation). For example, the local and / or cloud-based components may process the captured audio data using an ASR model in response to the invocation of the automated assistant 120.

[0032] In some implementations, multiple automated assistants can be executed on assistant device 100, and the invocation phrase spoken for each automated assistant can be different. For example, a first automated assistant can have an invocation phrase "OK Assistant 1", and when a user speaks the phrase "OK Assistant 1", the first automated assistant can be invoked so that the first automated assistant can process additional audio data before and / or after the invocation phrase. Similarly, when the user speaks a second invocation phrase such as "OK Assistant 2", a second automated assistant also executing on assistant device 100 can be invoked, whereby the second automated assistant can process additional audio data before and / or after the invocation phrase. As further described herein, the audio data can be encrypted using speech features generated from a portion of the audio data that includes the user speaking the invocation phrase. Continuing with the previous example, in some implementations, audio data including a user saying “OK Assistant 1” can be encrypted using utterance features generated from a portion of the audio data including the user saying “OK Assistant 1” such that only a first automated assistant (e.g., an automated assistant invoked with “OK Assistant 1”) that has access to the speaker features generated from the user saying “OK Assistant 1” can decrypt the audio data.

[0033] Figure 1 The automated assistant client 120 in FIG. 1 is illustrated as including an automatic speech recognition (ASR) engine 122, a natural language understanding (NLU) engine 124, a text-to-speech (TTS) engine 126, and a fulfillment engine 128. In some implementations, one or more of the illustrated engines may be omitted (e.g., instead implemented solely by the cloud-based automated assistant component 140) and / or additional engines may be provided (e.g., the invocation engine described above).

[0034] The ASR engine 122 may process audio data capturing a spoken utterance to generate a recognition of the spoken utterance. For example, the ASR engine 122 may utilize one or more ASR machine learning models to process the audio data to generate a prediction of the recognized text corresponding to the utterance. In some of those implementations, the ASR engine 122 may generate a corresponding confidence metric for each of the one or more recognized terms indicating a confidence that the predicted term corresponds to the spoken utterance.

[0035] TTS engine 126 can convert text into synthesized speech, and can rely on one or more speech synthesis neural network models to do so. For example, TTS engine 126 can be used to convert a text response into audio data, which includes a synthesized version of the text, and the synthesized version is rendered in an audible manner via the hardware speakers of assistant device 100.

[0036] The NLU engine 124 determines the semantic meaning of the audio and / or the text converted from the audio by the ASR engine, and determines assistant actions corresponding to those semantic meanings. In some implementations, the NLU engine 124 determines the assistant action as the intent and / or parameter determined based on the recognition of the ASR engine 122. In some cases, the NLU engine 124 can parse the intent and / or parameter based on a single utterance of the user, and in other cases, prompts can be generated based on unresolved intent and / or parameters, those prompts are rendered to the user, and the NLU engine 124 uses the user response to those prompts to parse the intent and / or parameters. In those cases, the NLU engine 124 can optionally work in conjunction with a dialog manager engine (not illustrated) that determines unresolved intent and / or parameters and / or generates corresponding prompts. The NLU engine 124 can use one or more NLU machine learning models to determine intent and / or parameters.

[0037] Fulfillment engine 128 may cause execution of an assistant action determined by NLU engine 124. For example, if NLU engine 124 determines that the assistant action is "turning on the kitchen lights," fulfillment engine 128 may cause corresponding data to be transmitted (either directly to the lights or to a remote server associated with the manufacturer of the lights) to cause the "kitchen lights" to be "turned on." As another example, if NLU engine 124 determines that the assistant action is "provide a summary of the user's meetings for today," fulfillment engine 128 may access the user's calendar, summarize the user's meetings for the day, and cause the summary to be rendered at assistant device 100 in a visual and / or audible manner.

[0038] Assistant device 100 further includes a digital signal processor (DSP) 110 that can process incoming audio data and perform initial processing on the audio data. For example, although DSP 110 and automated assistant 120 are illustrated as separate components, at least a portion of the components of automated assistant 120 can be performed by DSP 110, such as initial ASR, STT processing, NLU, and / or other processing of incoming audio data. DSP 110 includes a buffer 115 that can store a portion of audio data captured by, for example, one or more microphones of assistant device 100. In some implementations, DSP 110 may be limited in computing power and / or functionality such that the audio data is at least partially processed only by DSP 110 before being provided to automated assistant 120. Thus, for security purposes, the received audio data can be temporarily stored in buffer 115, processed by a call engine to determine whether the captured audio data requires further processing (e.g., whether the audio data includes a request from the user that requires fulfillment of the request), and / or provided to one or more applications 130 and 140 executing on assistant device 100.

[0039] DSP 110 further includes a speech-to-text (STT) engine 120 that can generate text based on incoming audio data stored in buffer 115. For example, audio data including a user speaking one or more requests can be captured by a microphone of assistant device 100, the audio data can be stored in buffer 115, and STT engine 120 can process the stored audio data to generate a text representation of the user's utterance. Buffer 115 may be limited in its capacity to store audio data, such that at a given time, STT engine 120 can only access a limited amount of audio data (e.g., 10 seconds of audio data, audio data of a limited file size).

[0040] In addition, the DSP 110 may include one or more components that can analyze the incoming audio data (and / or text representing the incoming audio data) and determine whether the audio data includes one or more phrases spoken by the user, such as an invocation phrase. In the case where the DSP 110 determines that the audio data includes an invocation phrase, the DSP 110 can provide the audio data to the automated assistant 120 for further processing. Thus, the DSP 110 can determine that the audio data includes a request for the automated assistant to perform one or more actions before the audio data is forwarded to the automated assistant 120 and / or other components (e.g., applications 130 and 140). Similarly, before the audio data is provided to the applications 130 and / or 140, the audio data can be processed to determine which applications 130 and 140 are authorized to access the audio data.

[0041] DSP 110 further includes a feature generation engine 145 that can generate one or more speech features for the audio data stored in buffer 115. Speaker generation engine 145 can generate speaker features for a user using instances of audio data, each instance of audio data capturing a corresponding spoken utterance of the user during normal non-enrolled interaction with an automated assistant via one or more corresponding assistant devices. For example, in response to determining that a recognized term of a spoken utterance captured by a portion of an instance of audio data (determined using speech recognition performed on the audio data) corresponds to a particular text associated with the TD technology being used, speaker generation engine 145 can generate a speech feature for the user using the portion of the instance of audio data capturing the spoken utterance.

[0042] In the TD technique, one or more previously generated speaker embeddings of a user are generated based on a spoken utterance that includes only one or more specific words or phrases. Moreover, in use, the user must say the one or more specific words or phrases in order to generate one or more TD speaker embeddings using the TD model, which can be effectively compared with one or more previously generated TD speaker embeddings of the user to determine whether the spoken utterance is from a specific user (e.g., the user of the assistant device 100 or another user associated with the assistant device 100). For example, the one or more specific words or phrases in the TD speaker identification may be constrained to one or more invocation phrases configured to invoke the automated assistant (e.g., hot words and / or trigger words, such as, for example, "Hey Assistant", "OK Assistant", and / or "Assistant") or one or more warm words described herein. In contrast, for the TI technique, the spoken utterance processed using the TI model is not constrained to one or more specific words or phrases. In other words, audio data based on virtually any spoken utterance can be processed using the TI model to generate a TI speaker embedding that can be effectively compared to one or more previously generated TI speaker embeddings of a user (e.g., via feature comparison engine 155) to determine whether the spoken utterance is from the same user (e.g., the user of assistant device 100 or another user associated with assistant device 100). Furthermore, in various implementations, the one or more previously generated TI speaker embeddings of a user utilized in the TI technique are generated based on spoken utterances that include disparate words and / or phrases, and are not limited to invocation words and / or phrases, warm words, and / or any other particular spoken utterance.

[0043] As an example, refer to Figure 2A, a flow chart is provided that illustrates the use of a TD model to generate speech features. Speech 205A is provided by a buffer 115, which can store audio data captured by one or more microphones of a client device. Speech 205A includes a first portion 206A "OK Assistant", which can be an invocation phrase indicating that the user is interested in interacting with the automated assistant. In addition, speech 205A includes a second portion 207A, which indicates what actions the user is interested in being performed by the automated assistant and / or by one or more other applications. As illustrated, the feature generation module 145 can generate speech features 210A from the first portion 206A to generate, for example, a vector representing the speech features. Thus, in this example, using TD technology, speech features are generated based on a portion of the audio data that includes a user saying the invocation phrase. As further described herein, speech features 210A can be used to encrypt all or part of the audio data that includes spoken speech.

[0044] As an example of TI discourse feature generation, refer to Figure 2B , a flow chart is provided that illustrates the use of the TI model to generate speech features. The speech 205B is provided to the feature generation module 145, which then generates speech features 210B based on the audio data. Figure 2A In contrast to the method illustrated in , the utterance feature 210B is not constrained to a particular phrase, but is independent of what the user said. As further described herein, the TI utterance feature can be used to encrypt audio data, such as the entire spoken utterance 105B and / or a portion of the audio data.

[0045] refer to Figure 3A , provides a flow chart that illustrates one or more methods described herein. The steps of the illustrated method can be performed by Figure 1 In some implementations, one or more steps of method 300 may be omitted and / or one or more additional steps may be included in method 300.

[0046] At step 305A, audio data including an invocation phrase is received. In some implementations, the audio data may be received by a component that shares one or more characteristics with the DSP 110. For example, the audio data may be captured by a microphone of the assistant device 100 and stored in a buffer 115. In some implementations, the audio data may be initially processed by the DSP 110 to determine whether the audio data includes an invocation phrase. In the event that an invocation phrase has been detected (e.g., by processing a text representation of the audio data generated by the STT engine 120 to determine whether the user has spoken an invocation phrase such as “Hey Assistant”), the DSP 110 may provide the audio data and / or the text representation of the audio data to one or more other components, such as the automated assistant 120, the application 130, and / or the application 140.

[0047] At step 310A, speaker features are generated from the audio data. In some implementations, speaker features may be generated by a component that shares one or more characteristics with the feature generation engine 145. For example, in some implementations, the feature generation engine 145 may generate TD features that are generated based on one or more specific terms and / or phrases (e.g., generated based on a portion of the audio data that includes a user saying an invocation phrase). In some implementations, the feature generation engine 145 may generate TI features based on other portions of the audio data that do not include specific phrases and / or terms (e.g., "turn on the kitchen lights") and / or other portions that include additional phrases and / or terms in addition to the invocation phrase. In both cases, the speaker features may include one or more vectors generated based on a model used to analyze the audio data.

[0048] Reference again Figure 1 , DSP 110 further includes an audio data encryption engine 155. In some implementations, audio data encryption engine 155 can encrypt audio data using one or more features as encryption keys. For example, speech features (and / or vectors representing speech features) generated by feature generation engine 145 can be used as encryption keys for audio data stored in buffer 115. In some implementations, since DSP 110 is a separate component that is not accessible to other components of assistant device 115, DSP 110 can prevent other components from accessing received audio data by only allowing encrypted audio data to be provided to other components such as automated assistant 120, application 130, and / or application 140.

[0049] Reference again Figure 3A, at step 315A, the audio data is encrypted using the speaker features generated at step 310. In some implementations, the speaker features can be generated based on the user saying a particular phrase (e.g., using a TD-generated speaker feature generated from the audio of the user saying "Hey assistant") and / or from the TI-generated speaker features. Other components with access to previously generated speaker features (e.g., the user's speaker features generated during the enrollment period) can decrypt the audio data by using the speaker features as a key for encryption.

[0050] Reference again Figure 2A , the generated speech features 210A are provided to the audio data encryption engine 150 along with a portion of the audio data 207A. In some implementations, the entire audio data 205A may be provided to the audio data encryption engine 150 for encryption. In some implementations, the speech features 210A may be used to encrypt only a portion of the audio data (e.g., a single channel of multi-channel audio data, an obfuscated version of the audio data, a portion of the audio data that does not include an invocation phrase). The resulting encrypted audio data 215A may be provided to the application communication engine 160 for transmission to one or more applications, as further described herein. For example, one or more applications may attempt to decrypt the audio data using one or more stored speaker features generated using TD techniques.

[0051] Reference again Figure 2B , the generated speech features 210B are provided to the audio data encryption engine 150 together with the audio data 205B. In this case, the entire audio data is provided for encryption. However, Figure 2A As in the case of audio data in the TI, only a portion of the audio data may be provided for encryption. The resulting encrypted audio data 215 is provided to the application communication engine 160 for transmission to one or more applications for further processing, as described herein. For example, one or more applications may attempt to decrypt the audio data using one or more stored speaker features generated using the TI technique.

[0052] Reference again Figure 1, the DSP 110 further includes an application communication engine 160. The application communication engine 160 can communicate with one or more applications such as application 130, application 140, and / or automated assistant 120 to indicate to the receiving application that the audio data is available. For example, the audio data can be stored in buffer 115, and the STT engine 120 can generate a text representation of the audio data. In some implementations, the DSP 110 can further determine one or more applications that can utilize the audio data, including the automated assistant 120, based on one or more terms in the text representation. For example, the audio data may include a user saying "OK Assistant, book me a flight using Application 1 (OK, Assistant, use application 1 to book a flight for me)", and the DSP 110 can determine that the user refers to application 130. In this case, the application communication engine 160 can communicate directly with the application 130 and provide an indication that the audio data is available, and further, the application 130 may be able to fulfill the request included in the audio data. In some implementations, an indication may first be provided to automated assistant 120 that may determine, based on one or more signals from DSP 110, that an application may be capable of performing one or more tasks represented by the user's utterance captured in the audio data. For example, DSP 110 may generate a signal that may be provided to automated assistant 120 via application communication engine 160 indicating that the user has spoken the term "Application 1" and that the audio data may be directed to application 130 based on the limited information provided to automated assistant 120.

[0053] In some implementations, the DSP 110 may provide the encrypted audio data to the automated assistant 120. In this case, the automated assistant 120 may be able to decrypt the audio data and / or may further facilitate the transmission of the audio data to one or more applications. For example, the DSP 110 may generate encrypted audio data that is encrypted based on speech features generated from a portion of the audio data including a user saying the phrase "OK Assistant" (e.g., an invocation phrase of the automated assistant 120). The automated assistant 120 may then provide the encrypted audio data to the application 130 and / or the application 140 for further processing. Thus, in some implementations, the DSP 110 may only transmit to the automated assistant, which may then determine to which application(s) the encrypted audio data should be provided.

[0054] Based on the indication provided by the DSP 110 via the application communication engine 160, one or more applications receiving the indication can determine that the audio data points to the application. For example, in some implementations, the DSP 110 can provide an indication that the audio data is available to all and / or multiple applications. In addition, the DSP 110 can further provide an indication that the application that the user has indicated should process the oral speech included in the audio data. For example, the user can say the name of a specific application (for example, "book aflight using Application 1 (booking a flight using application 1)" includes the application name "Application (application)"). Moreover, for example, the user can say one or more keywords indicating the type of the request proposed, and the DSP 110 can provide an indication of the keyword and an indication that the audio data is available (for example, providing "flight (flight)" and the user has said "Book me a flight (booking a flight for me)"). In some implementations, as described above, the automated assistant 120 can be used as an intermediary between the DSP 110 and the application that can receive the encrypted audio data.

[0055] At step 320A, the encrypted audio data is provided to one or more applications. In some implementations, the encrypted audio data may be provided to multiple applications by the DSP 110. In some implementations, the encrypted audio data may be selectively provided to one or more applications based on preliminary processing by the DSP 110. For example, the DSP 110 (or automated assistant 120) may determine that one or more applications are the target applications of the spoken utterance based on one or more terms included in the audio data. In response, the DSP 110 may provide an indication of the presence of new audio data only to those applications that are likely to be the target of the user's utterance.

[0056] Reference again Figure 1 As mentioned above, assistant device 100 further includes application 130 and application 140. In some implementations, assistant device 100 can be executed via one or more processors of assistant device 100. In some implementations, assistant device 100 can have more than Figure 1 . For example, assistant device 100 may have an email application, a ride-sharing application, and / or a calendar application, all of which are executed on assistant device 100. Applications may be executed in the foreground, or may be executed in the background. Thus, in some cases, an application may be actively accessed and / or utilized by a user of assistant device 100, or may be executed in a standby mode such that the application can receive and / or send information without the user actively accessing the application.

[0057] In some implementations, the DSP 110 may communicate with one or more of the applications 130 and 140. For example, the application communication engine 160 may provide encrypted audio data to the application 130, and independently provide the same encrypted audio data to the application 140. Each may include one or more components that may process the received audio data, and may further include a corresponding fulfillment engine that is capable of performing one or more tasks in response to receiving audio data including a user speaking a request to perform a task. In some implementations, one or more of the applications 130 and / or 140 may further communicate with the automated assistant 120, such that, for example, the DSP 110 may send at least a portion of the instructions to the automated assistant 120 for preliminary processing, and then the automated assistant 120 may forward the instructions (or different instructions) to the application. For example, the automated assistant 120 may receive audio data from the DSP 110, process the audio data using the TTS engine 126 to generate a text representation of the spoken utterance, encrypt the audio data, and provide the encrypted audio data to the application 130. Also, for example, rather than DSP 110 providing encrypted audio data directly to application 130, DSP 110 may first generate a text representation of the utterance, encrypt the text representation, and provide the encrypted text representation (and / or other intermediate representation of the audio data, such as a spectrogram).

[0058] In some implementations, the audio data (or the textual representation of the audio data) can be obfuscated prior to encryption. For example, the STT engine 120 can first generate a textual representation of the audio data, and one or more components can determine whether the utterance includes one or more terms that are not required to be provided to the application, omit those terms, and encrypt the remainder of the textual representation. Moreover, for example, portions of the audio data can be omitted (or otherwise obfuscated) prior to encryption to remove or obfuscate at least a portion of the audio data that is not required by the target application.

[0059] As Figure 3AIn an example of the method illustrated in , a user may speak the phrase "OK Assistant, send a message using Application 1". Audio data of the user speaking the phrase may be stored in buffer 115 for further processing. Feature generation engine 145 may then generate speaker features based on the portion of the user speaking "OK Assistant". Alternatively or additionally, STT engine 120 may generate a text representation of the audio data. Based on the speaker features generated by feature generation engine 145, audio data encryption engine 150 may encrypt the audio data, the text representation of the audio data, and / or an obfuscated version of the audio data and / or the text representation. The encrypted audio data and / or the encrypted text representation may then be provided to one or more other components, such as automated assistant 120, application 130, and / or application 140. In this way, unencrypted audio data is available only to DSP 110 and is not available to other components. In some implementations, only a portion of the audio data may be encrypted, and other portions of the audio data may be transmitted without encryption. For example, in some cases, a portion of the audio data may include sensitive information, and only that portion of the audio data may be encrypted before the audio data or an intermediate representation of the audio data is provided to one or more other components for further processing. Thus, in order for another application to access the portion with the sensitive information, speaker characteristics may be required that allow the application to decrypt those portions of the audio data.

[0060] refer to Figure 3B , a flow chart is provided that illustrates one or more methods that may be performed by one or more applications, such as application 130 and application 140, executing on assistant device 100. In some implementations, one or more steps of the illustrated method may be omitted and / or one or more additional steps may be performed. In some implementations, one or more steps of the illustrated method may be performed by one or more other components, such as automated assistant 120.

[0061] At step 305B, an application (e.g., application 130 and / or application 140) receives the encrypted audio data and / or an encrypted textual representation of the audio data - intact or obfuscated as described herein. As previously described, the audio data has been encrypted using speaker features determined by DSP 110 (e.g., by feature generation engine 145 and encrypted by audio data encryption engine 150). For example, feature generation engine 145 can generate a vector indicating the voice characteristics of a user who said "OK Assistant" and generate a TD vector unique to the utterance. Before providing the encrypted audio data to one or more applications 130 and / or 140, audio data encryption engine 150 can encrypt the audio data using the generated vector as a key.

[0062] The application 130 includes (or has access to) a database 132 that includes one or more speech features (e.g., vectors representing speaker features). In some cases, the speaker features may be based on previous interactions with the user. For example, a user may be asked to say a phrase (e.g., "OK Assistant") multiple times during a registration period, and a vector may be generated that is an average of the speech, such that the resulting vector is an approximation of the prosodic features of the user when the particular phrase is spoken. Thus, in some cases, the database 132 may include text-dependent speech features that represent a user saying a particular phrase. In some cases, a vector may be generated that is independent of the user saying a particular phrase, and thus the resulting vector may be text-independent. This type of vector may be generated based on the user saying various phrases, and then the vector may be averaged so that the resulting vector represents a general prosodic feature that is unique to the user saying the phrase. As described herein, "speech features" and / or "feature vectors" may include TD or TI speech features of a user, such that for a given user, a unique vector (TD or TI) that can be used to encrypt and / or decrypt audio data may be generated.

[0063] Application 140 further includes (or can access) a database 142, which includes one or more feature vectors of the user. As with database 132, the feature vectors can be TD or TI, and each feature vector is unique to the user. In some implementations, database 132 and database 142 can be the same database, wherein application 130 can access one or more vectors, and application 140 can access one or more vectors. Thus, a database including multiple vectors of multiple users can be maintained, wherein access rights to each vector are selectively granted to specific applications. For example, automated assistant 120 can maintain a database and selectively grant access rights to speaker feature vectors based on the rights granted by the user.

[0064] In some implementations, the application communication engine 160 can provide the encrypted audio data to the applications 130 and / or 140, along with one or more indications of the user who spoke the phrase. For example, the first encrypted audio data can be provided with an indication of "User 1", and the second encrypted audio data can be provided with an indication of "User 2". In some implementations, one or more of the applications 130 and 140 can access the feature vector of "User 1" via a corresponding database. Additionally or alternatively, one or more of the applications 130 and 140 can access the feature vector of "User 2" and / or the feature vectors of both "User 1" and "User 2" via a corresponding database.

[0065] As an example, User 1 may speak a phrase captured in the audio data, where the phrase includes the term "OK Assistant". DSP 110 may generate a feature vector based on User 1 speaking "OK Assistant" and utilize the feature vector to encrypt the audio data. In some implementations, DSP 110 may also determine an indication of the identity of the speaker of the utterance (i.e., User 1) and provide the audio data to both application 130 and application 140. Continuing with this example, application 130 may access a feature vector of User 1 generated based on the user's previous speaking of "OK Assistant" so that the vector accessible to application 130 is the same (or similar) as the vector used by DSP 110 to encrypt the audio data. In some implementations, DSP 110 may provide the audio data to both applications 130 and 140, as well as an indication of the user who spoke the phrase captured in the audio data. Thus, application 130 may check in database 132 to determine whether application 130 has access to a feature vector for User 1. Similarly, application 140 may check in database 142 to determine whether application 140 has access to the feature vector for User 1. In some cases, only application 130 has access to the feature vector for User 1, and application 140 has not been granted access to the same feature vector.

[0066] At step 310B, the receiving application (e.g., application 130 and / or 140) identifies one or more speaker characteristics stored in a database accessible to the application (e.g., database 132 or database 142, depending on the receiving application). In some implementations, the application may identify multiple potential speaker characteristics, each potential speaker characteristic associated with a unique user. In the event that DSP 110 provides an indication of the user who spoke the phrase captured in the audio data, the application may determine whether it has access to the user's utterance characteristics.

[0067] In some implementations, the encrypted audio data may be provided with an indication of the type of application that can process the audio data. In those cases, the application may first determine whether it can process the audio data before checking for speaker characteristics. For example, the encrypted audio data may be provided to an application 130, which may be a messaging application, and an application 140, which may be a flight booking application. The encrypted audio data may be provided to both applications along with an indication that the audio data is related to booking a flight (e.g., "Book a flight to Miami" or "Book a flight to Miami using Application 2"), such as an indication that the audio data is a "book flight" request. In this case, for example, the application 130 may not perform any additional processing and / or may not identify potential speaker characteristics based on determining that it will not be able to process the audio data even if it is decrypted. In contrast, an application 140 configured to handle a request to book a flight may identify one or more speaker characteristics based on determining that the request is likely to be handled by the application 140.

[0068] Reference again Figure 1 , application 130 includes a decryption engine 136 that can use speaker features to decrypt audio data. For example, application 130 can receive encrypted audio data that is encrypted using speech features derived from the audio data. The encrypted audio data can then be provided to application 130. In response, decryption engine 136 can identify one or more speaker feature vectors in database 132 that were previously generated based on speech from a user (e.g., based on the user saying the phrase "OK Assistant" multiple times during a registration period). Using one or more techniques, decryption engine 136 can use the identified speaker features to attempt to decrypt the audio data.

[0069] Reference again Figure 3B, at step 315B, the application uses the identified speaker features to attempt to decrypt the audio data. For example, the application 130 can access the first speaker features of the first user and use one or more encryption / decryption techniques to attempt to decrypt the encrypted audio data. In some implementations, decryption can be performed using one or more fuzzy algorithms. For example, the speech features used by the DSP 110 to encrypt the audio data may not be the same as the speech features accessible to the application. However, using one or more fuzzy matching algorithms, the application can use similar speech features to attempt to decrypt the encrypted audio data. Thus, the application does not require a complete match between the encryption key (i.e., the speech features used by the DSP 110 to encrypt the audio data) and the decryption key (i.e., the speaker features used by the application to attempt to decrypt the audio data) for decryption of the audio data.

[0070] As a working example, database 132 may include speaker feature vectors for each of User 1, User 2, and User 3. Similarly, database 142 may include speaker feature vectors for each of User 4 and User 5. User 3 may utter "OK ​​Assistant, send a message to Bob," which is then encrypted using speech features derived from the audio data. At step 305B, both application 130 and application 140 receive the encrypted audio data. At step 310B, each of the applications identifies the user's speaker features in a corresponding database (e.g., database 132 of application 130 and database 142 of application 140). For example, at step 310B, application 130 may identify the speaker features of User 1, and application 140 may identify the speaker features of User 4. At step 315B, application 130 may use speaker characteristics of User 1 to attempt to decrypt the encrypted audio data, and application 140 may use speaker characteristics of User 4 to attempt to decrypt the audio data.

[0071] At decision block 320B, application 130 and / or application 140 may determine whether the speaker features used to attempt to decrypt the encrypted audio data have successfully decrypted the audio data. If not, then at step 325B, different speaker features may be identified by the application and used to attempt to decrypt the encrypted audio data at step 315B. If the audio data has been successfully decrypted, then at step 330B, the audio data may be processed by the application. For example, continuing with the previous example, because the utterance is from User 3, neither application 130 nor application 140 can successfully decrypt the audio data using the speaker features of User 1 and User 4, respectively. At step 325B, application 130 may identify another speaker feature vector (e.g., speaker features of User 2), and application 130 may identify another speaker feature vector (e.g., speaker features of User 5). Since neither User 2 nor User 5 is the speaker of the utterance, neither application can successfully decrypt the audio data. At step 325B, application 140 does not have any other speaker feature vectors to use to attempt to decrypt the audio data, and therefore cannot process the audio data to perform one or more tasks requested by the user. Application 130 identifies the speaker feature vector for User 3 and attempts to use the vector to decrypt the encrypted audio data at step 315B. At step 320B, application 130 recognizes that the audio data was successfully decrypted and processes the audio data at step 330B. The audio data, including the audio of the user saying "Send a message to Bob", is processed, and one or more tasks are performed (e.g., providing an interface to the user, automatically sending a message).

[0072] In some implementations, instead of encrypting the audio data using one or more speech features generated from at least a portion of the audio data, the DSP 110 can notify one or more applications that the audio data is available before sending the audio data. In response, the one or more applications can provide speaker features that can be compared to the speech features generated from the audio data by the feature generation engine 145, and if the provided speaker features match the speech features (either completely or approximately, using one or more fuzzy logic matching techniques), the audio data can be provided to the application that provided the matching speaker features. Figure 1, the feature comparison engine 155 may compare the speaker features to one or more stored speaker features to generate a metric indicating the similarity between the speaker features and the one or more stored speaker features. For example, the feature generation engine 145 may generate a speaker feature for a user saying a phrase such as "Hey Assistant." The speaker features may be stored in a database that includes speaker features for other users saying the phrase "Hey Assistant." Thus, for each user, a unique set of features is generated for each user that indicates the prosodic characteristics of the user's voice. When an utterance is received by the DSP 110, the feature generation engine 145 may generate a speaker feature for the utterance and provide the new speaker feature to the feature comparison engine 155. The feature comparison engine 155 may generate a metric for each stored speaker feature set that indicates the similarity between a particular stored speaker feature and the newly received speaker feature.

[0073] As an example, during a registration period, speaker features may be generated based on a user saying an invocation phrase such as “Hey assistant”. Features generated for each iteration of a user saying the invocation phrase may be utilized (e.g., averaged) to generate a speaker feature set that may be compared to other speaker features and / or other features generated from audio data (e.g., speech features). For example, the speaker features may be stored in a database, and when the user subsequently says the invocation phrase, the feature generation engine 145 may generate speech features for that instance. The stored speaker features may be compared to the speech features, and the stored speaker features may be utilized to determine whether a particular speaker feature set is generated from audio data of the same user as the user speaking in the audio data currently stored in the buffer 115. In the event that an application has access to matching speaker features, it may be determined that the application is authorized to receive audio data, and the application communication engine 160 may provide unencrypted audio data to the application.

[0074] refer to Figure 4A , a flow chart is provided that illustrates an example method of providing audio data to an authorized application. At step 405A, audio data including an invocation phrase is received. The audio data may be temporarily stored in buffer 115 and is accessible only to DSP 110. Thus, the audio data is not available to any other component until it is determined that the application is authorized to receive the audio data. At step 410A, speaker features are generated from the audio data. In some implementations, step 410A may be coupled to Figure 3A410A shares one or more characteristics with the feature generation engine 145. For example, the utterance features may be generated using TD techniques (e.g., generated on a portion of the audio data that includes the user uttering the invocation phrase) and / or the utterance features may be generated using TI techniques (e.g., on a portion or all of the audio data, regardless of the utterance). Thus, step 410A may be performed by a component that shares one or more characteristics with the feature generation engine 145.

[0075] At step 415A, a notification is provided to one or more applications that audio data is available. For example, in some implementations, one or more components of the DSP 110 may determine the intent of the request in the audio data and determine which application(s) may be configured to fulfill the request. In some implementations, the DSP 110 may identify the name of the application included in the utterance and provide a notification directly to the application. For example, for the utterance of "Hey Assistant, set a reminder to call the office on Tuesday", the STT engine 120 may generate a text representation of the utterance, and one or more components may determine that "set a reminder" is a task that can be fulfilled by a calendar application. Thus, in some implementations, the application communication engine 160 may identify one or more applications (e.g., application 130 and / or application 140), and further provide one or both applications with audio data related to "set areminder" that is available. In some implementations, the application communication engine 160 may send notifications only to the calendar application. In some implementations, the application communication engine 160 may send requests to all applications. Also, for example, for an utterance of “Hey Assistant, set a reminder using Application 1,” the application communication engine 160 may send notifications only to the application, and not to other applications not indicated by the user.

[0076] At step 420A, one or more applications provide speaker characteristics previously generated from audio data of one or more users speaking a phrase. For example, as previously described, a user may be requested to speak a phrase during a registration period, and a speaker characteristic may be generated from the user utterance. In some implementations, the generated speaker characteristics may be selectively provided to one or more applications based on permissions granted by the user. For example, a user may grant application 130 access to the audio data, and in response, the user's speaker characteristics may be provided to application 130, which may be stored in database 132. Similarly, a user may not grant application 140 access to the audio data, and thus, by not providing the user's speaker characteristics to application 140, application 140 does not have access to the same speaker characteristics as application 130.

[0077] At step 420A, one or more speaker features are provided to the DSP 110. For example, at step 415A, a notification may be provided to the application 130 that audio data is available. In response, the application 130 may provide one or more sets of speaker features (e.g., speaker feature vectors) stored in the database 132. At step 425A, the speech features generated from the audio data are compared to the speaker features received from the application using one or more techniques (e.g., exact match and / or fuzzy match techniques). For example, the speaker feature vector may be provided to the DSP 110, and the feature comparison engine 155 may compare the provided speaker features to the speech features generated by the feature generation engine 145. If the features match (e.g., exact match, approximate match), then at step 430A, the audio data may be provided to the application that provided the speaker features.

[0078] refer to Figure 4B , a flow chart is provided that illustrates a method of receiving audio data in response to providing speaker characteristics. In some implementations, Figure 4B The method can be obtained by Figure 1 The application 130 and / or 140 may be executed by components that share one or more characteristics. In some implementations, one or more steps may be omitted, combined, and / or one or more additional steps may be included in the method.

[0079] At step 405B, a notification is received that audio data is available. As described herein, the notification may be provided by a component that shares one or more characteristics with the DSP 110. For example, the application communication engine 160 may provide a notification that includes an indication of a target application for a request included in the audio data (e.g., "application 1"), an indication of the type of request (e.g., "messaging request"), and / or may simply include an indication that audio data is available. In some implementations, the notification may further include an indication of a speaker of an utterance included in the audio data (e.g., "User 1") and / or other information that may be used by the application to determine whether the available audio data may include a request that may be fulfilled by the application.

[0080] At step 410B, speaker characteristics are provided in response to the notification that audio data is available. In some implementations, applications (e.g., applications 130 and / or 140) can all be provided with notification and can each independently provide speaker characteristics for one or more users. For example, application 130 can access speaker characteristics for User 1 and User 2 stored in database 132. Also, for example, application 140 can access speaker characteristics for User 3 stored in database 142. In some implementations, each application can provide one or more speaker characteristics to DSP 110 in response to receiving notification from DSP 110 that audio data is available.

[0081] At step 415B, audio data is received in response to providing speaker characteristics. As previously described, the audio data can be provided to an application that has provided speaker characteristics matching the speech characteristics generated from the audio data at step 410B. For example, continuing with the previous example, if DSP 110 provides notification that audio data is available, and the audio data includes speech spoken by User 2, then application 130 will only be provided with the audio data after providing speaker characteristics of User 2. At step 420B, the received audio data is processed. The audio data can be processed by a fulfillment engine such as fulfillment engines 134 and 144 of the receiving application.

[0082] Figure 55 is a block diagram of an example computing device 510 that may optionally be used to perform one or more aspects of the techniques described herein. The computing device 510 generally includes at least one processor 514 that communicates with a plurality of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524 (including, for example, a memory subsystem 525 and a file storage subsystem 526), ​​a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input devices and output devices allow for user interaction with the computing device 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0083] The user interface input device 522 may include a keyboard, a pointing device (such as a mouse, a trackball, a touch pad or a graphics tablet, a scanner, a touch screen integrated into a display), an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computing device 510 or onto a communication network.

[0084] The user interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device (such as a liquid crystal display (LCD)), a projection device, or some other mechanism for producing a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. Generally speaking, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 510 to a user or another machine or computing device.

[0085] The storage subsystem 524 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 524 may include a computer program that executes FIG. 3A to FIG. 4B Selected aspects of the method and / or for implementing Figure 1 The logic of the various components depicted in .

[0086] These software modules are typically executed by the processor 514 alone or in combination with other processors. The memory 525 used in the storage subsystem 524 may include multiple memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 may provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations may be stored by the file storage subsystem 526 in the storage subsystem 524, or in other machines accessible to the processor 514.

[0087] The bus subsystem 512 provides a mechanism for the various components and subsystems of the computing device 510 to communicate with each other as intended. Although the bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0088] Computing device 510 can be of different types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Figure 5 The depicted description of computing device 510 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computing device 510 are possible, and these configurations are similar to those of FIG. Figure 5 Computing devices are depicted as having more or fewer components than those shown.

[0089] In some implementations, a method implemented by one or more processors is provided herein, and the method includes the following steps: processing a stream of audio data detected via one or more microphones of a client device to monitor for a spoken invocation phrase. In response to detecting an occurrence of a spoken invocation phrase in a portion of the audio data, the method further includes the following steps: processing at least the portion of the audio data to generate a speech feature; encrypting at least a portion of the audio data using the speech feature as an encryption key to generate encrypted audio data; and outputting the encrypted audio data without outputting any unencrypted form of at least a portion of the audio data.

[0090] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0091] In some implementations, at least a portion of the audio data includes preceding audio data that precedes the portion of the audio data and / or succeeding audio data that follows the portion of the audio data. In some implementations, at least a portion of the audio data includes a portion of the audio data that is detected as including an occurrence of the spoken invocation phrase. In some implementations, the one or more processors are comprised of a digital signal processor (DSP) of the client device, and wherein outputting the encrypted audio data includes outputting the encrypted audio data to an additional processor of the client device.

[0092] In some implementations, processing the call portion of the audio data to generate the utterance features includes: processing the utterance features using a text-dependent speaker-verified machine learning model to generate utterance vectors; and generating the utterance features based on the utterance vectors. In some of those implementations, generating the utterance features based on the utterance vectors includes using the utterance vectors as the utterance features.

[0093] In some implementations, outputting the encrypted audio data causes an application executing on the client device to attempt to decrypt the audio data using one or more pre-stored speaker features accessible to the application. In some of those implementations, the one or more pre-stored speaker features are generated during a registration process with the application or with an operating system of the client device. In other of those implementations, the application uses approximate speaker feature matching when attempting to decrypt the audio data using the one or more pre-stored speaker features.

[0094] In some implementations, the method further includes, prior to encrypting at least the portion of the audio data, generating an obfuscated version of at least the portion of the audio data, wherein encrypting the audio data includes encrypting the obfuscated version of at least the portion of the audio data. In some of those implementations, generating the obfuscated version of at least the portion of the audio data includes omitting at least a segment of at least the portion of the audio data to generate the obfuscated version of at least the portion of the audio data. In other of those implementations, generating the obfuscated version of the portion of the audio data includes enhancing at least some of the portion of the audio data with noise to generate the obfuscated version of the portion of the audio data.

[0095] In some implementations, at least a portion of the audio data includes previous audio data that precedes the portion of the audio data, and wherein the previous audio data is retrieved from a local buffer accessible only to the one or more processors.

[0096] In another aspect, a method implemented by one or more processors is provided, and the method includes the following steps: processing a stream of audio data detected via one or more microphones of a client device to detect a spoken invocation phrase; processing at least a portion of the audio data to generate an utterance feature; and receiving one or more speaker features provided by a requested target included in the audio data; determining whether the received speaker features match the utterance features. The method further includes: in response to determining that the speaker features match the utterance features, outputting the audio data to the requested target; and in response to determining that the speaker features do not match the utterance features, suppressing outputting the audio data to the requested target.

[0097] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0098] In some implementations, the one or more processors include a digital signal processor, and wherein the target is executed on a processor separate from the digital signal processor. In some of those implementations, the target of the request is an additional processor, an application, or an operating system. In some implementations, the method further includes: identifying a target of the request based on the audio data; and providing one or more notifications indicating the target of the request. In some of those implementations, the one or more notifications include a type of the request.

[0099] In another aspect, a method implemented by one or more processors is provided, and the method includes the following steps: receiving encrypted audio data of an utterance spoken by a user, wherein the encrypted audio data is encrypted using one or more utterance features of the user; decrypting the audio data using one or more vectors generated from previous occurrences of the user speaking the one or more utterances to generate decrypted audio data; processing the decrypted audio data; and causing one or more computer actions to be performed based on the processing of the decrypted audio data.

[0100] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0101] In some implementations, one or more speech features are generated based on a user speaking an invocation phrase included in the audio data. In some implementations, one or more vectors are generated during a registration process. In some implementations, the audio data is received from a digital signal processor (DSP). In some of those implementations, the audio data is received by an application executing on one or more processors separate from the DSP.

[0102] Various implementations may include a non-transitory computer-readable storage medium storing instructions that are executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and / or a tensor processing unit (TPU)) to perform a method such as one or more of the methods described herein. Other implementations may include an automated assistant client device (e.g., a client device that includes at least an automated assistant interface for interfacing with a cloud-based automated assistant component), the automated assistant client device including a processor that is operable to execute the stored instructions to perform a method such as one or more of the methods described herein. Still other implementations may include a system of one or more servers that include one or more processors that are operable to execute the stored instructions to perform a method such as one or more of the methods described herein.

[0103] Where certain implementations discussed herein may collect or use personal information about a user (e.g., user data extracted from other electronic communications, information about a user's social network, a user's location, a user's time, a user's biometric information, and a user's activities and demographic information, relationships between users, etc.), the user is provided with one or more opportunities to control whether information is collected, whether personal information is stored, whether personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use user personal information only after receiving explicit authorization from the relevant user to collect, store, and / or use the user's personal information.

[0104] For example, a user is provided with control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. One or more options are presented to each user whose personal information is to be collected to allow control over the collection of information associated with that user, providing permission or authorization as to whether to collect information and as to which portions of the information are to be collected. For example, one or more such control options may be provided to a user via a communications network. In addition, specific data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. As an example, the identity of a user may be processed so that no personally identifiable information can be determined. As another example, a user's geographic location may be generalized to a larger area so that the user's specific location cannot be determined.

[0105] Although several implementations have been described and shown herein, various other means and / or structures for performing functions and / or obtaining results and / or one or more advantages described herein may be used, and each of such changes and / or modifications is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which this teaching is used. Those skilled in the art will recognize or confirm many equivalents of the implementations described herein using only routine experiments. Therefore, it should be understood that the aforementioned implementations are presented only by way of example, and it should be understood that within the scope of the attached claims and their equivalents, implementations may be practiced in a manner other than specifically described and claimed. The implementations of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure if such features, systems, articles, materials, kits, and / or methods do not contradict each other.

Claims

1. A method implemented by one or more processors of a client device, the method comprising: processing a stream of audio data detected via one or more microphones of the client device to detect a spoken invocation phrase; In response to detecting an occurrence of the spoken invocation phrase in a portion of the audio data: processing at least the portion of the audio data to generate utterance features; encrypting at least a portion of the audio data using the speech feature as an encryption key to generate encrypted audio data; as well as The encrypted audio data is output without outputting any unencrypted form of at least a portion of the audio data.

2. The method of claim 1, wherein: At least part of the audio data includes preceding audio data preceding the part of the audio data and / or succeeding audio data following the part of the audio data.

3. A method as claimed in any preceding claim, wherein: At least a portion of the audio data comprises the portion of the audio data detected as including the occurrence of the spoken invocation phrase.

4. A method as claimed in any preceding claim, wherein: The one or more processors are comprised of a digital signal processor (DSP) of the client device, and wherein outputting the encrypted audio data comprises outputting the encrypted audio data to an additional processor of the client device.

5. A method as claimed in any preceding claim, wherein: Processing the call portion of the audio data to generate the utterance features comprises: processing the utterance features using a text-dependent speaker-verified machine learning model to generate an utterance vector; and The utterance feature is generated based on the utterance vector.

6. The method of claim 5, wherein: Generating the utterance feature based on the utterance vector includes using the utterance vector as the utterance feature.

7. A method as claimed in any preceding claim, wherein: Outputting the encrypted audio data causes an application executing on the client device to attempt to decrypt the audio data using one or more pre-stored speaker characteristics accessible to the application.

8. The method of claim 7, wherein: The one or more pre-stored speaker characteristics are generated during a registration process with the application or with an operating system of the client device.

9. The method of claim 7, wherein: The application uses approximate speaker feature matching when attempting to decrypt the audio data using the one or more pre-stored speaker features.

10. The method of any preceding claim, further comprising: Prior to encrypting at least part of said audio data: generating an obfuscated version of at least part of the audio data, Wherein, encrypting the audio data comprises encrypting at least part of the obfuscated version of the audio data.

11. The method of claim 10, wherein: Generating the obfuscated version of at least part of the audio data comprises: At least one segment of at least a portion of the audio data is omitted to generate the obfuscated version of at least a portion of the audio data.

12. The method of claim 10, wherein: Generating the obfuscated version of the portion of the audio data comprises: At least some of the portion of the audio data is enhanced with noise to generate the obfuscated version of the portion of the audio data.

13. A method as claimed in any preceding claim, wherein: At least a portion of the audio data includes previous audio data preceding the portion of the audio data, and wherein the previous audio data is retrieved from a local buffer accessible only to the one or more processors.

14. A method implemented by one or more processors of a client device, the method comprising: processing a stream of audio data detected via one or more microphones of the client device to detect a spoken invocation phrase; processing at least a portion of the audio data to generate utterance features; receiving one or more speaker characteristics provided by a requested target included in audio data; determining whether the received speaker characteristics match the utterance characteristics; In response to determining that the speaker characteristic matches the utterance characteristic: outputting the audio data to the requested destination; and In response to determining that the speaker characteristic does not match the utterance characteristic: Outputting the audio data to the target of the request is suppressed.

15. The method of claim 14, wherein: The one or more processors include a digital signal processor, and wherein the target is executed at least in part on a processor separate from the digital signal processor.

16. The method according to any one of claims 14 and 15, wherein: The target of the request is an additional processor, an application, or an operating system.

17. The method of any one of claims 14 to 16, further comprising: identifying the target of the request based on the audio data; as well as One or more notifications are provided indicating the target of the request.

18. The method of claim 17, wherein: The one or more notifications include a type of the request.

19. A method implemented by one or more processors of a client device, the method comprising: Receiving encrypted audio data of an utterance spoken by a user, wherein the encrypted audio data is encrypted using one or more utterance features of the user; decrypting the audio data using one or more vectors generated from previous occurrences of the user uttering one or more utterances to generate decrypted audio data; processing the decrypted audio data; and Based on processing the decrypted audio data, one or more computer actions are caused to be performed.

20. The method of claim 19, wherein: The one or more utterance features are generated based on the user speaking an invocation phrase included in the audio data.

21. A computer program comprising instructions which, when executed by one or more processors of a computing system, cause the computing system to perform a method as claimed in any preceding claim.

22. One or more computing devices configured to perform the method of any one of claims 1 to 20.

Images