Leveraging a sandboxed feature detection process to ensure the security of captured audio and / or other sensor data

Abstract

To improve security of sensor data that is at least selectively processed by a feature detection process of an application installed on a client device.SOLUTION: The feature detection process is sandboxed to restrict the egress of data from a component. Once the feature detection process determines that a feature has been detected in sensor data, an interactor process is provided with the sensor data and / or additional sensor data. The sensor data and / or the additional sensor data are provided directly by an operating system and not via the feature detection process.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] It relates to utilizing a sandboxed feature detection process that ensures the security of captured audio and / or other sensor data. [Background technology]

[0002] Humans can engage in human-computer interactions with interactive software applications referred to herein as “automated assistants” (also referred to as “digital agents,” “interactive personal assistants,” “intelligent personal assistants,” “assistant applications,” “conversational agents,” etc.). For example, a human (who may be referred to as a “user” when interacting with an automated assistant) can provide commands and / or requests to the automated assistant using natural language input (i.e., speech) that may be converted to text and then processed by providing oral, but in some cases textual (e.g., typed), natural language input, and / or through touch and / or non-speech body movements (e.g., hand gestures, gaze, facial movements, etc.). The automated assistant responds to the request by providing responsive user interface output (e.g., audible and / or visual user interface output), controlling one or more smart devices, and / or controlling one or more functions of a device implementing the automated assistant (e.g., controlling other applications on the device).

[0003] As described above, many automated assistants are configured to be interacted with via spoken utterances. To protect user privacy and / or conserve resources, the automated assistant refrains from performing one or more automated assistant functions based on all spoken utterances present in audio data detected via a microphone of a client device that (at least partially) implements the automated assistant. Rather, certain processing based on spoken utterances occurs only in response to a determination that certain conditions exist.

[0004] For example, many client devices include and / or interface with automated assistants and include a hot word detection model. If the microphone of such a client device is not muted, the client device can continuously process audio data detected via the microphone using the hot word detection model to generate a predictive output indicating the presence of one or more hot words (including compound word phrases), such as “Hey, Assistant,” “Okay, Assistant,” and / or “Assistant.” If the predictive output indicates the presence of a hot word, any audio data that follows within a threshold amount of time (and optionally is determined to contain a speech activity) can be processed by one or more on-device and / or remote automated assistant components, such as a speech recognition component, a speech activity detection component, etc. Audio data predicted to contain a hot word can also be processed by other on-device and / or remote automated assistant components. Furthermore, recognized text (from the speech recognition component) can be processed using a natural language understanding engine and / or actions can be taken based on the natural language understanding engine output. Actions can include, for example, generating and providing a response and / or controlling one or more applications and / or smart devices. Other hot words (e.g., "no," "stop," "cancel," "volume up," "volume down," "next track," "previous track," etc.) may be mapped to various commands, and if the predicted output indicates that one of these hot words is present, the mapped command may be processed by the client device. However, if the predicted output indicates that the hot word is not present, the corresponding audio data will be discarded without any further processing, thereby preserving resources and user privacy.

[0005] A user can install one or more automated assistant applications or other applications on a client device. If the installed application includes hotword detection capabilities and the corresponding rights are granted to the application during installation, the installed application will have at least selective access to audio data captured via the client device's microphone. This allows the application to process the audio data, for example, in determining whether a hotword is present in the audio data. However, allowing an application unchecked access to audio data may expose security vulnerabilities, such as the leakage of audio data (or data derived from audio data) in which no hotwords were detected. These security vulnerabilities may be exacerbated in situations where the application is controlled by a malicious entity. More generally, security vulnerabilities may be exposed by applications that can process sensor data (e.g., audio data, image data, location data, and / or other sensor data) while operating in the background and / or under many (or all) conditions. Summary of the Invention [Problem to be solved by the invention]

[0006] Implementations disclosed herein are directed to improving the security of sensor data (e.g., audio data) that is at least selectively processed by feature detection processes (e.g., hotword detection processes and / or speaker verification processes) of applications installed on client devices. [Means for solving the problem]

[0007] In some of these implementations, the feature detection process runs in a sandbox environment, such as an isolated process in an operating system, controlled by the operating system of the client device. In other words, the operating system controls the constraints imposed by the sandbox, but the feature detection process itself can be controlled by the application that utilizes the feature detection process (e.g., the feature detection process can be part of the application and work in cooperation with other non-sandboxed processes of the application).

[0008] Additionally, the operating system controls the provisioning of sensor data to the sandboxed feature detection process and prevents the sandboxed feature detection process from sending sensor data. Rather, the operating system provides the sensor data (and / or other sensor data) directly (i.e., not through the sandboxed feature detection process) to the application's non-sandboxed interactor process in response to the feature detection process indicating that a feature was detected in the sensor data. As an example, if the feature detection process is a hotword detection process and indicates that a hotword is detected in a portion of audio data detected via the client device's microphone, the operating system can provide the non-sandboxed interactor process with that portion of audio data as well as portions of audio data preceding and / or following that portion. Preventing the sandboxed feature detection process from sending sensor data and instead having the operating system provide the sensor data directly improves security. For example, the sandboxed feature detection process can be prevented from sending previous sensor data (or data derived therefrom) that was provided to the sandboxed feature detection process and determined not to contain a feature, under the guise of providing sensor data. For example, it can prevent encoding such prior sensor data (or data derived therefrom) into the outgoing sensor data.

[0009] Moreover, in some implementations, the sandboxed feature detection process can be allowed to send only a limited amount of data, only data that matches a defined schema, and / or only when a feature is detected. In these and other ways, limiting when and / or what data can be sent, for example, by mitigating the opportunity for sending prior sensor data (and / or data derived therefrom), improves the security of the sensor data. As described herein, in various implementations, a human-perceptible indication can be provided when the sandboxed feature detection process indicates that it has detected a feature, when it sends data, and / or when sensor data is provided to an interactor process. For example, the perceptible indication can be a graphical and / or audible affordance indicating the type of sensor data (e.g., a picture of a microphone when the sensor data is audio data). Optionally, the perceptible indication can additionally or alternatively be selected to identify or reveal the application. In these and other ways, a user can confirm through the perceptible indication that the corresponding sensor data is being accessed by an application, further ensuring the security of the sensor data.

[0010] In various implementations, additional and / or alternative techniques can be utilized to further mitigate the risk of the sandboxed feature detection process releasing prior sensor data (or data derived therefrom) that was provided to the sandboxed feature detection process and determined not to contain features. For example, the operating system can cause the sandboxed feature detection process's memory, which may store such data, to be wiped at intervals. For example, the operating system can force a restart of the sandboxed feature detection process at intervals and / or fork the sandboxed feature detection process at intervals.

[0011] As mentioned above, some implementations disclosed herein are directed to improving security for audio data captured by a client device and provided to a component (also referred to as an "interactor process") based on the identification of hotwords in the audio data. The hotword detection process operates within a "sandbox" such that the outgoing sensor data from the hotword detection process is restricted. Once the sandboxed hotword detector determines the presence of a hotword, the data is provided to the component or application that will utilize the sensor data. Thus, the audio data, or audio data stream, is not directly accessible by the interactor process until detection of a specific hotword occurs.

[0012] By sandboxing the hotword detection process, unauthorized transmission of data is mitigated. The hotword detection process receives audio data for analysis and then sends one or more indications that a hotword has been detected. However, the hotword detection process is restricted from sending the audio data itself, but instead indicates to the interaction manager that one or more components have been invoked by the hotword. The interaction manager then grants the interactor access to the audio stream. For example, the hotword detection process may receive a snippet of audio data that is likely to contain a hotword. In response to determining the presence of a hotword, the hotword detection process may, by virtue of the sandbox, be authorized to send only an indication that a hotword is present (e.g., a single-bit signal). In some implementations, the hotword detection process may be authorized to send additional, but limited, data, such as an indication of the user who uttered the hotword, the uttered hotword, and / or additional information that does not specifically include audio data. By limiting the hotword detection process to the transmission of a limited number of bytes of information, unauthorized transmission of data can be further mitigated. Once a hotword is detected by the hotword detection process, the voice interaction manager may provide the audio data, as well as optionally audio data preceding and / or following the audio data, to the interactor. For example, the interactor process can be provided with the audio data in which the hotword was detected, as well as a stream of audio data following such audio data. The interactor process can then further process and act upon the received audio data. The interactor process can be non-sandboxed. For example, the interactor process can operate within the permissions granted by the user when the application was installed and would not be constrained to the extent of the restrictions imposed on a sandboxed hotword detection process.

[0013] To better improve security, the hotword detection process can be forced by the operating system to erase its memory at intervals. This can ensure that any data stored in memory by the hotword detection process is limited to data generated since the last memory erase. This can prevent a malicious hotword detection process from storing audio data or data derived from audio data and attempting to illicitly transmit such stored data. As described above, to mitigate the illicit transmission of such stored data, the sandbox can have restrictions on when, how much, and / or what type of data can be transmitted. However, forcing the hotword detection process to erase its memory can additionally or alternatively mitigate the illicit transmission of such stored data. For example, forcing memory erasure can be used in combination with restrictions on data transmission, thereby mitigating the opportunity for the hotword detection process to illicitly attempt to encode stored data into something that appears to be valid data for transmission. As an example, one or more components of the operating system can erase memory accessible to the hotword detection process at regular or irregular intervals to restrict access to audio data. In some implementations, this can be achieved by the operating system forcing the hotword detection process to restart. In some additional or alternative implementations, this can be achieved by the operating system utilizing forking to spawn a new hotword detection process and reaping the previous hotword verification process, thereby clearing the memory of any previous hotword detection process. Forking allows a new process to be spawned for the hotword detection process without requiring additional overhead components (e.g., libraries, configuration information) to be reloaded into the sandbox's memory.Thus, forking can allow for the effective clearing of memory in a more resource-efficient manner than completely restarting the hotword detection process (which would require reloading overhead components). The new hotword detection process therefore does not have access to the audio data that was accessible by the previous hotword detection process, which may be terminated once its replacement has been created.

[0014] As mentioned above, in some implementations, it may be desirable to notify a user when audio data is being provided to an application. Such an indication can improve the security of the audio data by informing the user when an application is accessing the audio data (and optionally which applications are accessing the audio data) and allowing the user to identify and remove any applications accessing the audio data at inappropriate times. However, because audio data may be continuously provided to the hotword detection process (at least when certain contextual conditions are met) to enable monitoring for the occurrence of hotwords, providing an indication when the hotword detection process is processing the audio data would result in the user always being provided with an indication that the audio data is being processed. For example, a device may have a graphical interface that allows an indication to be displayed to the user when an application is accessing the audio data. However, displaying an indication when the hotword detection process is processing the audio data would be undesirable because it would effectively render the indicator useless (i.e., it would always show the microphone as active), thereby reducing its effectiveness in improving the security of the audio data. Thus, implementations disclosed herein provide an indication to the user that audio data is being provided to an application and / or interactor process only once a hotword is detected by the sandboxed hotword detection process, causing the operating system to provide the corresponding audio data to a non-sandboxed process of the application.

[0015] Accordingly, these implementations can promote audio data security by providing a cue that enables a user to be aware when audio data is being provided to a non-sandboxed process. Moreover, through utilization of the sandboxed hotword detection process and related techniques disclosed herein, the security of audio data provided to the sandboxed hotword detection process can also be ensured, while avoiding the need to provide a cue when audio data is being provided only to the sandboxed hotword detection process. Also, avoiding the need to provide a cue when only the sandboxed hotword detection process allows the cue to be meaningful to the user.

[0016] Various examples are described herein regarding the processing of audio data using a sandboxed hotword detection process. However, implementations disclosed herein can process audio data using additional and / or alternative processes. For example, a speaker identification process can operate within the sandbox in conjunction with the hotword detection process. The speaker identification process can process audio data detected by the hotword detection process as containing hotwords to perform text-dependent speaker identification (TDSID). An indication of the user account, if any, determined from the TDSID to have provided the hotword can optionally be provided as part of the limited data permitted to be sent from the sandbox.

[0017] Further, implementations disclosed herein can additionally and / or alternatively be utilized in sandboxing other processes that process additional and / or alternative sensor data. For example, implementations can require a gaze and / or gesture detection process to operate in the sandbox process. The gaze and / or gesture detection process can at least selectively process image data to determine whether a user's gaze or a user's gesture is intended to invoke one or more components. For example, an application (e.g., an assistant application) can be invoked in response to detecting a user's gaze directed at the client device and lasting longer than a threshold duration. When the sandboxing detection process determines that a particular gaze and / or gesture has been detected, it can provide an indication to the operating system, and in response, the operating system can provide image data, subsequent image data, and / or audio data to the application's corresponding interactor process. Restrictions on data transmission can be imposed on the sandbox to prevent unauthorized transmission of image data (or data derived therefrom) by the detection process. Additionally, an indication that the image data is being processed can be given when the operating system provides the image data to an interactor process, but not when it is provided only to a secure sandboxed detection process.

[0018] As another example, an application's geofence entry detection process can be forced to operate within a sandbox. The geofence entry detection process can at least selectively process GPS and / or other location data to determine whether a client device has entered one or more geofences. When the sandboxed geofence entry detection process determines that a particular geofence has been entered, it can provide an indication to the operating system, and in response, the operating system can provide the location data to the application's corresponding interactor process. To prevent unauthorized emission of location data (or data derived therefrom) by the detection process, restrictions on data emission can be imposed on the sandbox. Furthermore, an indication that the location data is being processed can be provided when the operating system provides the location data to an interactor process, but not when it is provided only to the secure geofence entry detection process.

[0019] The above description is provided merely as an overview of some implementations disclosed herein. These and other implementations of the present technology are disclosed in further detail below.

[0020] It should be understood that all combinations of the above concepts, and additional concepts described in more detail herein, are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. [Brief explanation of the drawings]

[0021] [Figure 1] 1 illustrates an example environment in which implementations disclosed herein may be implemented. [Figure 2] 1 illustrates an example interface that may be provided via a client device. [Figure 3] 1. This shows an example of an interaction that can occur between the components illustrated in FIG. [Figure 4]1 illustrates a flowchart of an example method according to various implementations described herein. [Figure 5] 10 illustrates a flowchart of another example method according to various implementations described herein. [Figure 6] 1 illustrates example architectures for computing devices according to various implementations. DETAILED DESCRIPTION OF THE INVENTION

[0022] 1 illustrates an example environment in which implementations described herein may be implemented. The environment includes a client device 110 having an operating system 105. The client device 110 may optionally utilize a digital signal processor (DSP) 115 to process audio data and / or other sensor data. In some implementations, the DSP 115 can be utilized by the operating system 105 and / or by applications installed on the operating system 105 to perform some low-power processing of the sensor data. For example, the DSP 115 can be utilized to at least selectively process captured audio data to determine a likelihood that the audio data includes human speech (e.g., voice activity detection) and / or to determine a likelihood that the audio data includes any of one or more hot words.

[0023] The operating system may have access to one or more buffers 150 that store audio data while it is being processed by one or more components. The operating system 105 may store a portion of the audio data in one or more buffers 150 and provide access to at least a portion of the audio data and / or the buffers 150 to the DSP 115. For example, the interaction manager 120 may store the audio data as it is provided, while imposing limits on the amount of data stored during processing by the DSP 115 and / or the hotword detection process 125 (e.g., storage size of the data, set duration of the audio data). In instances where the interactor process 135 is given permission to access the audio data, at least a portion of the audio data stored in the buffer 150 may be provided to the hotword detection process 125. For example, any audio data in the buffer 150 may be provided to the hotword detection process 125, as well as being given access to the input stream of the microphone 140. In some implementations, this may include audio uttered before and / or after the hotword.

[0024] In implementations in which a DSP 115 is included and utilized to determine the likelihood that audio data contains human speech and / or the likelihood that audio data contains hotwords, providing such audio data (and optionally preceding and / or subsequent audio data) to other processes not running on the DSP 115 may be conditioned on meeting a likelihood threshold. For example, the DSP 115 may be utilized to perform initial hotword detection on the audio data, and if the initial hotword detection indicates that a hotword is present, the audio data may be provided to a hotword detection process 125 running within the sandbox 130 and may utilize a high-power processor (relative to the DSP 115). The DSP 115 may be low-power (relative to other processors) and may utilize a small footprint and low robustness and / or accuracy models (relative to the models utilized by the sandboxed hotword detection process) in performing the initial hotword detection. Although the initial hotword detection performed on the DSP 115 may trigger excessively (i.e., have many false positives), many of those false positives will be caught by the more robust and / or accurate sandboxed hotword detection process 125. Thus, the initial hotword detection process can effectively serve as an initial, loose filter, such that the sandboxed hotword detection process 125 does not need to analyze all captured audio data. This can conserve power resources because the initial hotword detection process utilizes the DSP 115, a less resource-intensive processor than utilized by the sandboxed hotword detection process 125. It is noted that in implementations in which a DSP 115 is included and utilized to perform initial hotword detection, sandboxing of the initial hotword detection by the DSP 115 may not be necessary to ensure the security of the audio data. This may be due, for example, to hardware limitations of the DSP 115 that prevent robust processing of the audio data and / or prevent robust storage of the resulting data from the processing, and / or limitations on the sending of data from the initial detection by the DSP 115 (e.g., only an indication that a hot word was initially detected).

[0025] As referenced above, the hotword detection process 125 is included within a sandbox 130 to isolate the hotword detection process 125 from other processes running on the operating system 105 and to constrain the ingress and egress of data to and from the hotword detection process 125. For example, the sandbox 130 can limit the ingress of data to the hotword detection process 125 to audio data and, optionally, limited other data (e.g., a confidence measure determined by the initial hotword detection process). As another example, the sandbox 130 can limit the egress of data to sending only a certain amount of bits in a given egress instance, can limit the frequency of regression instances, and / or can require that egress instances conform to a certain data schema. Although the hotword detection process 125 can be part of (e.g., controlled by) an application 170 running on the operating system 105, the hotword detection process 125 will be constrained by the sandbox 130 restrictions imposed by the operating system 105. Application 170 further includes interactor process 135, which receives audio data and performs one or more tasks based on the input sensor data, such as performing one or more tasks based on the presence of hotwords in the audio data. Operating system 105 further includes interaction manager 120, which coordinates the flow of sensor data between various components of operating system 105 and application 170. For example, interaction manager 120 may provide permission to interactor process 135 to access the sensor data and / or may receive one or more indications from hotword detection process 125 that hotwords have been detected in the audio data.

[0026] In some implementations, a sandbox controlled by an operating system can prevent network access to processes running within the sandbox. For example, the hotword detection process 125 may be restricted from accessing the network (e.g., restricted from accessing the client device's network interface) to further improve security and prevent the transmission of audio data. In some cases, an interactor process may have network access and may send audio data after the audio data has been sent to the interactor process by the operating system. In various implementations, a sandbox controlled by an operating system limits what operating system functionality is available for exploitation by processes running within the sandbox. For example, the operating system (e.g., the interaction manager 120) can allow processes running within the sandbox to exploit only certain application programming interfaces (APIs) of the operating system and / or only certain aspects of the APIs, while preventing access to other APIs and / or other aspects of the APIs. When allowing a process to utilize only certain aspects of an API, a proxy API may be used (e.g., implemented by interaction manager 120) that interfaces between the process and the API, where the proxy API is an intermediary that allows utilization of certain aspects of the API while preventing utilization of other aspects of the API. As one particular example, an operating system may allow a process to access all or aspects of a base API that are required for running apps within the operating system.As another particular example, the operating system may additionally or alternatively enable a process to access all or aspects of an API that enables interaction with interaction manager 120, an API that provides access to microphone audio data, and / or an API that enables exposure of certain data to other sandboxed processes (e.g., sandboxed processes that can leverage certain data for federated learning). APIs or API aspects for which access is not explicitly enabled may be completely inaccessible to processes operating within the sandbox.

[0027] The client device 110 includes a microphone 140 for capturing audio data, a camera 165 for capturing video and / or images, and a GPS component 160. Each of these components is a sensor that captures and provides sensor data. In some implementations, one or more of the components may be absent. The microphone 140 may, in some implementations, include an array of multiple microphones and may include near-field and / or far-field microphones. In some implementations, the audio data captured via the microphone 140 is continuously provided to the interaction manager 120. The client device 110 further includes a display 145, which may be utilized to provide a graphical interface to the user. In some implementations, the graphical interface may optionally include an indication that the sensor data is being utilized by one or more applications. For example, with reference to FIG. 2 , an example interface 300 is provided. The interface 300 may include one or more graphical elements that change appearance and / or appear when sensor data is provided to the application 170. For example, indicator 305 may appear and / or change appearance (e.g., a different image, change color, change size) when a non-sandboxed process of application 170 utilizes audio data from microphone 140. Additionally, indicator 310 may appear and / or change appearance when a non-sandboxed process of application 170 accesses image data from camera 165. In some implementations, GPS 160 may capture location data, and one or more indicators may appear when a non-sandboxed process of application 170 accesses the location data. In some implementations, notification 315 may be provided to a user when a non-sandboxed process of application 170 accesses audio data, and notification 320 may be provided when a non-sandboxed process of application 170 accesses video and / or image data.It is noted that notifications 315 and 320 not only indicate that the corresponding sensor data is being accessed, but also indicate the corresponding application that is accessing the sensor data. In some implementations, notification 315 can be provided in place of indicator 305, and notification 320 can be provided in place of indicator 310. In some other implementations, notification 315 can be provided in response to a user selection of indicator 305, and notification 320 can be provided in response to a user selection of indicator 310.

[0028] 1. Referring to Figure 3, one example of interactions that may occur between the components illustrated in Figure 1 is illustrated. As illustrated, feature data (e.g., audio data, image data, location data) continuously flows from sensors 180 of client device 110 to operating system 105. As the audio data is received by operating system 105, it is captured for further analysis (see arrow #1). Operating system 105 may store a portion of the audio data in one or more buffers 150 and provide access to at least a portion of the audio data and / or buffers 150 to DSP 115 (see arrow #2).

[0029] The digital signal processor (DSP) 115 receives the audio data from the interaction manager 120 and determines whether the audio data contains human speech. The DSP may be a low-power circuit that is always active or always active when certain contextual conditions are met (e.g., at a certain time, when the client device 110 is in a certain state, etc.). The DSP 115 can determine the likelihood that the audio data contains human speech and / or the likelihood that the audio data contains hotwords. In cases where language is likely to be detected (e.g., a likelihood score meeting a threshold), the audio or a portion of the audio may be provided to the hotword detection process 125 for further analysis to determine whether the detected language contains hotwords. Thus, the initial hotword detection process can effectively serve as an initial, loose filter, so that the sandboxed hotword detection process 125 does not need to analyze all captured audio data. However, as a tradeoff for consuming minimal resources, the DSP 115 may downsize the incoming stream of audio data so that the DSP's analysis is less robust than the hotword detection process 125. In some implementations, such as those where power consumption is not a consideration, the DSP 115 may not be present at all, and the captured audio data may be provided directly by the interaction manager 120 to the hotword detection process 125. In some implementations, in addition to or instead of utilizing the DSP 115 to process the audio, a portion of the audio data may be provided to a remote device for further analysis, such as detecting the presence of hotwords with a more robust detector.

[0030] In some implementations, at least a portion of the audio data is provided to the DSP 115 to allow the DSP 115 to detect the likelihood of language in the audio data (see arrow #2). Analysis by the DSP 115 may be triggered by a high false positive rate, for example, due to background noise and / or other sounds not intended to invoke the application (see arrow #3). Furthermore, because the DSP 115 is a low-power device, the audio channel may be downsized to allow for faster processing time with minimized resource consumption. In some implementations, the DSP 115 may use one or more neural networks to determine the likelihood that the audio data contains human language. If the likelihood measure meets a threshold, a trigger may be provided to the interaction manager 120.

[0031] The hotword detection process 125 utilizes one or more hotword detection models to determine whether the audio data contains one or more hotwords. In some implementations, the hotword detection process 125 may recognize specific hotwords that invoke the assistant application (e.g., "Okay, Assistant," "Hey, Assistant") or other applications 170. In some cases, the hotword detection process 125 may recognize different sets of hotwords in different contexts (e.g., time of day) or based on the running application (e.g., foreground application). For example, if a music application is currently playing music, the automated assistant may recognize additional hotwords such as "music pause," "volume up," and "volume down."

[0032] While continuous processing of audio data may be necessary to recognize hotword utterances in the audio data, unwanted access to audio data from one or more applications may expose security vulnerabilities, such as data leakage and eavesdropping. Additionally, this access may lead to compromised data privacy and information security, as a person near the client device 110 may carry on a conversation not directed at the microphone 140 and directed to the interactor process 135 and sent to the operating system 105. Continuous access of audio data obtained via the microphone 140 may occur as a result of an unintentional or intentional configuration of the interactor process 135 to leak audio data not wanted by the user. In either case, the application 170 may become vulnerable to security and privacy losses. Such vulnerabilities may be exacerbated if a malicious entity configures an application to continue accessing audio data obtained via the microphone. Therefore, notifications and / or warnings provided to the user when an application is accessing sensor data may improve security measures by ensuring that the user is aware of when sensor data is being transmitted.

[0033] As mentioned above, an interface provided to the user via a display on client device 110 may indicate when the microphone or other sensor is active and alert the user via an icon or other visual or audio indication. For example, referring again to FIG. 2, indicators 305 and 310 and / or notifications 315 and 320 may be displayed when audio and / or video data is being utilized by an application. However, this is impractical in cases where audio data is being utilized to detect hotwords and is not being processed by the application. For example, in cases where audio data is stored in buffer 150 for further analysis by DSP 115 and / or hotword detection process 125 for hotword detection purposes, the indication of audio data being provided to the application may be constant. This is undesirable because the user may not be aware of what applications are accessing the audio data based on an indication that the microphone is on. This may be undesirable because, additionally or alternatively, when the DSP 115 and / or hotword detection process 125 are processing the voice data, the voice data is prevented from being sent to a remote device (e.g., due to sandboxing of the hotword detection process 125 and restrictions on the DSP 115), and the user may not have security concerns with such local-only processing. Furthermore, the DSP 115 will often trigger on non-speech voice data, leading to a significant number of false positive triggers, and will have the microphone indication "on" for a significant amount of time when no voice data is being sent to the interactor process 135. Therefore, the indication is preferably provided only once a hotword is detected and access to the buffered voice data and / or voice stream from the microphone 140 is provided to the agent application via the interactor process 135.

[0034] To prevent voice data from being provided to an application without authorization, hotword detection process 125 is contained within secure sandbox 130. Sandbox 130 regulates what data is provided to an application's interactor process, thus mitigating security concerns associated with applications that intercept or exfiltrate voice data without the user's knowledge. Therefore, hotword detection process 125 may be restricted in terms of what information it sends to interactor process 135. For example, hotword detection process 125 may receive a portion of the voice data stored in buffer 150 to determine whether a hotword is present in the voice data. If hotword detection process 125 determines that a hotword is present, an indication of the hotword may be provided to interaction manager 120, indicating that one or more applications have been invoked by the user via the hotword. Once the voice data is provided to interactor process 135, an interface may be updated to provide an indication that the voice data is being accessed. Thus, the user is alerted when an application is using audio data without the drawback of the "microphone in use" indication always being active or more active than when audio data is being used by an application other than the operating system 105.

[0035] Once a possible human speech is detected, a trigger (arrow #4) is sent to the hotword detection process 125 to indicate that the DSP 115 has detected a threshold probability of human speech in the audio data. The trigger may be provided on (or instead of) at least a portion of the audio data (e.g., a portion of the audio data stored in a buffer). The hotword detection process 125 is sandboxed to restrict data transmission and determines whether the audio data contains a hotword. If a hotword is detected, the hotword detection process 125 provides confirmation of the hotword to the interaction manager 120 (arrow #5). In some implementations, the data transmission may include only an indication that a hotword was detected (i.e., "yes / no"). In some implementations, the hotword detection process 125 may provide additional information to the interaction manager 120, such as information about the user who uttered the hotword. In some implementations, the hotword detection process 125 may provide confirmation of the presence of the hotword based on one or more other conditions, such as a particular application being accessed or a particular time of day. In some implementations, the hotword detection process 125 may send a confirmation whenever a hotword is detected, and the interaction manager 120 or another component may determine whether some other condition is met.

[0036] As an example, the operating system 105 may record a small snippet of audio data captured by the microphone 140, which is stored in the buffer 150. The DSP 115 may analyze the audio data and determine that the audio data contains a threshold probability of human speech. The interaction manager 120 may then provide the recorded audio data to the hotword detection process 125, which is contained within the sandbox 130. Based on the audio data, the hotword detection process 125 may determine that the audio data contains the hotword "Okay, Assistant." Because the hotword detection process 125 is sandboxed 130, it cannot provide the audio data directly to the interactor process 135, which may be configured to further process the audio data. Instead, the hotword detection process 125 may send an indication to the interaction manager 120 that a hotword has been uttered by the user. The interaction manager 120 may then grant that application 170 access to the interactor process 135. Once the interactor process 135 is provided with access to the audio data, an indication that the microphone 140 is processing the audio data may be provided to the user via the display 145, as described herein.

[0037] In some implementations, the hotword detection process 125 may provide additional information about the hotword utterance to the interaction manager 120 and / or directly to the interactor process 135. This may include, for example, information about the user who uttered the keyword. In some implementations, the sending of information may be limited to a certain number of bytes of information. Thus, the hotword detection process 125 may not be permitted (by the sandbox 130) to provide enough data to effectively transmit any audio data. For example, the hotword detection process 125 may provide an indication that is below a size threshold, such as less than 10 bytes. Such a limitation may allow the hotword detection process 125 to, for example, provide an indication of the speaker of the hotword while not having enough message space to send meaningful audio data.

[0038] In some implementations, the sandbox 130 may restrict the output from the hotword detection process 125 to a particular format or data schema, such that it is constrained to a particular type of data. In some implementations, any instructions provided by the hotword detection process 125 may be encrypted to better ensure that other applications and / or components do not illicitly intercept communications between the hotword detection process 125 and the interaction manager 120. The instructions may include, for example, a flag indicating that a keyword was uttered, an indication of the keyword uttered, user information associated with the user who uttered the hotword, and / or other indication that a hotword was detected.

[0039] Once hotword detection process 125 determines that a hotword has been uttered in the voice data and further provides instructions to interaction manager 120, as described above, confirmation may be provided to operating system 105 that the voice data may be recorded and / or provided to one or more components. Referring again to Figure 3, the confirmation (arrow #6) may include acknowledging that operating system 105 may begin recording additional voice data (arrow #7) and / or send already stored voice data to interactor process 135 for additional analysis. As illustrated, hotword detection process 125 does not provide the voice data directly, but instead the voice data is provided to interactor process 135 via interaction manager 120.

[0040] In some implementations, interactor process 135 may be provided only with previously captured audio data. In some implementations, interactor process 135 may be provided only with audio data captured after the utterance of a hotword. For example, the audio data may include something the user says that is unrelated to invoking the hotword detection process, which hotword detection process 125 determines is not a hotword. Once a hotword (e.g., "Okay, Assistant") is identified in the audio data, interactor process 135 may be provided with stored audio data that occurs after the hotword and / or with additional audio captured from microphone 140. In some implementations, interactor process 135 may be provided with additional audio data that occurs before the utterance of the hotword.

[0041] As an example, a user may utter the phrase "Okay, Assistant, turn on the lights." Interaction manager 120 may receive all or a portion of the voice data and, optionally, send it to DSP 115 to determine whether it contains human speech. Once a threshold likelihood of language is detected, the voice data and / or a portion of the voice data may be provided to hot word detection process 125. The hot word detection process may then determine that "Okay, Assistant" is a hot word and send an indication that the term is included to interaction manager 120. Interaction manager 120 may then provide access to the voice data and / or additional voice data for further processing, such as performing speech recognition.

[0042] In some implementations, interactor process 135 may be provided access to the audio data only if one or more additional conditions are met. For example, a hotword detection process may determine that the hotword "volume up" was uttered in the audio data and send an instruction to interaction manager 120. Interaction manager 120 may then determine whether the application that is the target of the hotword (e.g., a music application) is currently active before granting the application access to the audio stream. In some implementations, the conditions for granting access to the audio data may be, for example, the device that captured the audio data, the location where the audio data was captured, the time the audio data was captured, and / or the identity of the user who uttered the hotword.

[0043] In some implementations, to further enhance security by limiting the ability of the hotword detection process 125 to export information not intended for the interactor, one or more components of the hotword detection process 125 and / or the interaction manager 120 may clear the memory of the hotword detection process 125 to ensure that it only has information it immediately needs. In some implementations, the interaction manager 120 may have a process scheduler 155 that controls the hotword detection process 125. At intervals, the process scheduler 155 may spawn a new hotword detection process 130. This may be via forking, whereby a new verification service is spawned while additional libraries utilized by the verification service remain in memory. Such a process reduces the overhead required to create a new verification service. Once the new service is created, the process running the original hotword detection process 125 may be terminated. Thus, the new service does not have access to any of the previous information that was accessible to the original hotword detection process 125.

[0044] In some implementations, instructions and / or other data sent by the hotword detection process 125 can be stored for further verification that such data does not contain more information than permitted by the sandbox (e.g., to ensure the security of audio data). For example, when the hotword detection process sends data, the contents of the sent data, as well as a corresponding timestamp indicating when the data was sent, can be stored in an entry locally on the client device. The entry can later be reviewed by one or more security components or humans to further ensure that the sandbox is appropriate and does not permit the sending of additional information, such as audio data. For example, the entry can be securely transmitted from the client device to a remote server for review by security experts.

[0045] 4 depicts a flowchart illustrating an example method 400 for processing audio data to identify hot words. For convenience, the operations of method 400 are described with reference to a system that performs the operations, such as the systems illustrated in FIGS. 1 and 2. The system of method 400 includes one or more processors and / or other components of a client device. Moreover, while the operations of method 400 are illustrated in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added. As described herein, operating system 105 may be executing via one or more processors of a device, such as client device 110 and / or one or more cloud-based computer systems.

[0046] In step 405, the captured audio data is provided to a sandboxed feature detection process. The feature detection process may share one or more characteristics with the hotword detection process 125. In some implementations, only a portion of the captured audio data is provided to the feature detection process. For example, the feature detection process may receive audio data of a certain size or duration. In some implementations, the DSP 115 may first process the audio data to determine whether it contains human speech and then provide the audio data to the feature detection process (e.g., the hotword detection process 125). The feature detection process resides in a sandbox that restricts the sending of data from the process. Some components, such as the interaction manager 120 and the interactor process 135, are non-sandboxed, in that they are not restricted from sending and / or receiving data.

[0047] At step 410, an indication of the voice features detected by the sandboxed feature detection process is provided to the operating system and / or components executing through the operating system. In some implementations, the indication is restricted based on the sandbox in which the feature detection process is located. For example, with reference to FIG. 1 , the hotword detection process 125 may provide an indication to the interaction manager 120 that a hotword has been detected. The indication may include additional information, such as the identity of the user who uttered the hotword. In some implementations, the sending of information from the feature detection process may be restricted by a specific predefined data schema. In some implementations, the sending of information from the feature detection process may be limited by size, such as an indication being less than 10 bytes. By limiting the information allowed to be provided by the voice feature detection process, voice data is restricted from being provided directly from the feature detection process to one or more components.

[0048] At step 415, the captured voice data is provided to the non-sandboxed interactor process 135. The voice feature detection process is restricted from sending the voice data directly, as described above. Instead, an intermediary, such as the interaction manager 120, sends the voice data to the authorized interactor process 135. Thus, the voice data utilized by the hotword detection process 125 cannot be sent out from the service. In some implementations, to further ensure that the voice feature detection process cannot send the voice data, memory accessible by the voice feature detection process may be periodically wiped and / or the process may be terminated and restarted. This may occur at regular or irregular intervals to ensure that another non-sandboxed component cannot unauthorizedly send data. In some implementations, the operating system may utilize forking to spawn new processes, as described herein. Wiping memory at irregular intervals may ensure a higher level of security by preventing applications from determining when memory is wiped and leaking data before the memory is wiped. Irregular intervals may include clearing the memory once a certain amount of data has been received, whenever the client device 110 is inactive, and / or once the DSP 115 makes an initial voice detection.

[0049] 5 presents a flowchart illustrating an example method 500 for processing sensor data to identify features using a sandboxed detection process. For convenience, the operations of method 500 are described with reference to a system that performs the operations, such as the systems illustrated in FIGS. 1 and 3. This system of method 500 includes one or more processors and / or other components of a client device. Moreover, while the operations of method 500 are illustrated in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0050] In step 505, sensor data is provided to a sandboxing feature detector process. In some implementations, the sensor data may be audio data captured by a microphone on the client device, such as microphone 140 on client device 110. In some implementations, the sensor data may be video data captured by one or more cameras 165 on client device 110. For example, an operating system, which may include one or more of the components of FIG. 1, may receive image data captured by sensor 180. The image data may include, for example, a user gesture and / or one or more other features indicating that the user is interested in interacting with an application. At least a portion of the image data may be provided to a hotword detection process, which may determine whether certain features are present in the image data, such as a user looking at the device, interacting with the device, making a gesture, and / or other visual features that may be present in the image data. In some implementations, the sensor data may include location data captured via a GPS component and utilized to determine whether the device is in a location that should trigger one or more applications.

[0051] In step 510, the feature detection process provides an indication that a feature has been detected in the sensor data. Step 510 may share one or more characteristics with step 410 of Figure 4. In some implementations, the detected feature may be, for example, audio data, video data, location data, and / or other sensor data captured via one or more components of the client device.

[0052] In step 515, the audio data is provided to an interactor process. The interactor process may share one or more characteristics with interactor process 135. For example, the interactor process may be non-sandboxed in that the output of data from the process is not restricted in the same way as feature detection process 125. In some implementations, step 515 may share one or more characteristics with step 415 of FIG. 4, except that the sensor data may include, for example, audio data, image data, location data, and / or other captured sensor data.

[0053] While many examples and descriptions herein are primarily directed to capturing audio data for hotword verification, a similar process may be utilized using video data. Video data from camera 165 may be analyzed, for example, to determine whether an identified gesture is the video equivalent of a “hotword” (e.g., a gesture and / or feature by a user indicating an interest in interacting with one or more components). This may include, for example, a hand swiping motion to indicate that a particular action should be initiated by the client device. Also, for example, the sensor data described in FIG. 5 may be location data captured via a GPS component. Feature detection process 125 may examine the location data to determine whether a trigger location is identified, and one or more other components, such as interaction manager 120, may provide additional location data to the interactor process in response to determining that the required location has been detected.

[0054] As an example, a user may look at a device or a location on the device for a desired amount of time. Image data may be provided to the operating system 105 from a sensor 180 (e.g., a camera) and provided to a detection process running in the sandbox that can process the image data and determine, for example, whether the user is looking at the device. Once the presence of a user action is detected, the interactor process 135 may be provided with the image data and / or additional image data for further analysis.

[0055] 6 is a block diagram of an example computer system 610. Computer system 610 typically includes at least one processor 614 that communicates with several peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624, including, for example, memory 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input / output devices allow user interaction with computer system 610. Network interface subsystem 616 provides an interface to external networks and is coupled to corresponding interface devices in other computer systems.

[0056] The user interface input devices 622 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad or a graphics tablet, a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and means of inputting information into the computer system 610 or onto a communications network.

[0057] The user interface output devices 620 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and means for outputting information from the computer system 610 to a user or to another machine or computer system.

[0058] Storage subsystem 624 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 624 may include logic to perform selected aspects of method 400, method 500, and / or implement one or more of client device 110, operating system 105, operating system-executed interaction manager 120 and / or one or more of its components, interactor process 135, and / or any other engines, modules, chips, processors, applications, etc. described herein.

[0059] These software modules are typically executed by the processor 614 alone or in combination with other processors. The memory 625 used in the storage subsystem 624 can include several memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 can provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of an implementation may be stored in the storage subsystem 624 by the file storage subsystem 626 or on another machine accessible by the processor 614.

[0060] Bus subsystem 612 provides a mechanism for allowing the various components and subsystems of computer system 610 to communicate with each other as intended. Although bus subsystem 612 is illustrated schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0061] The computer system 610 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computer system 610 depicted in Figure 6 is intended as a specific example only for purposes of illustrating some implementations. Many other configurations of the computer system 610 are possible, having more or fewer components than the computer system depicted in Figure 6.

[0062] In situations where the systems described herein may collect or utilize personal information about users (or participants, as they are often referred to herein), users may be given the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographic location) or whether and / or how to receive content from content servers that may be more relevant to the user. Also, some data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that personally identifiable information cannot be determined to the user, or the user's geographic location may be generalized to the extent that geographic location information is available (e.g., to the city, zip code, or state level) so that the user's specific geographic location cannot be determined. Thus, users may have control over how information is collected and / or used about them.

[0063] In some implementations, a method implemented by a processor of a client device is provided, the method including providing, by an operating system of the client device, captured audio data to a sandboxed audio feature detection process sandboxed by the operating system. The method further includes receiving, by the operating system and from the sandboxed audio feature detection process, an indication that audio features have been detected by the sandboxed audio feature detection process. The method further includes, in response to receiving the indication, sending, by the operating system, the captured audio data to an interactor process. The operating system restricts the sandboxed audio feature detection process from sending the captured audio data to the interactor process.

[0064] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0065] In some implementations, the method further includes terminating and restarting the audio feature detection process by the operating system at intervals. In some variations of these implementations, the terminating and restarting of the audio feature detection process is at irregular intervals. In some variations of these implementations, the intervals are based on corresponding received indications that audio features have been detected in the audio data.

[0066] In some implementations, the method further includes forking, by the operating system and at intervals, a sandboxed audio feature detection process within the sandbox.

[0067] In some implementations, the method further includes controlling, by the operating system, the sandbox to prevent the sandboxed audio feature detection process from sending the captured audio. In some variations of these implementations, the controlling includes restricting data sending from the sandboxed audio feature detection process. In some variations, restricting data sending includes limiting instances of data sending to data that meets a size threshold. For example, meeting the size threshold can include being equal to or less than a certain amount of bytes, such as 16 bytes, 10 bytes, or 4 bytes. In some additional or alternative variations, restricting data sending includes limiting data sending to data that matches a defined data schema.

[0068] In some implementations, the method further includes, in response to receiving the instruction, providing a notification indicating non-sandboxed processing of the audio data, wherein the notification may be suppressed or otherwise not provided during processing of the audio data by the sandboxed audio feature detection process.

[0069] In some implementations, a method performed by a processor of a client device is provided, and includes providing, by an operating system of the client device, sensor data to a sandboxed feature detection process running on the client device within a sandbox controlled by the operating system. The sensor data is based on output from one or more sensors of the client device and / or one or more sensors communicatively coupled to the client device (e.g., via Bluetooth or other wireless modality). The method further includes receiving, by the operating system and from the sandboxed feature detection process, an indication that a feature has been detected by the sandboxed feature detection process. In response to receiving the indication, the method further includes sending, by the operating system, the sensor data to a non-sandboxed interactor process. The operating system restricts the sandboxed feature detection process from sending the sensor data.

[0070] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0071] In some implementations, the sensor data includes image data and / or audio data. In some implementations where the sensor data includes image data, the feature is a user gesture, a user gaze fixation, a pose (head and / or torso) with a certain characteristic, and / or the co-occurrence of a gesture, a gaze fixation, and / or a pose with a certain characteristic.

[0072] In some implementations, the method further includes terminating and restarting the sandboxed feature detection process by the operating system and at intervals.

[0073] In some implementations, the method further includes forking a sandboxed feature detection process within the sandbox by the operating system and at intervals.

[0074] In some implementations, the method further includes restricting, by the operating system, the sandboxed feature detection process from sending the captured sensor data. In some variations of these implementations, restricting the sandboxed feature detection process from sending the captured sensor data includes restricting data sending from the sandboxed feature detection process. In some variations of these implementations, restricting data sending includes restricting instances of data sending to data that meets a size threshold and / or restricting data sending to data that matches a defined data schema.

[0075] In some implementations, the method further includes, in response to receiving the instruction, providing a notification indicating non-sandboxed processing of the sensor data. The notification can be suppressed or otherwise not provided during processing of the sensor data by the sandboxed audio feature detection process. The notification can indicate a type of sensor data and / or can indicate (or be selectable to indicate) an application that controls the interactor process and optionally also controls the sandboxed feature detection process.

[0076] Various implementations may include a non-transitory computer-readable storage medium having stored thereon instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and / or a tensor processing unit (TPU)) to perform methods such as one or more of the methods described herein. Other implementations may include a client device including a processor operable to execute stored instructions to perform methods such as one or more of the methods described herein. [Explanation of symbols]

[0077] 105 Operating Systems 110 client devices 115 Digital Signal Processor (DSP) 120 Interaction Manager 125 Hotword detection process 130 Sandbox 135 Interactor Processes 140 microphone 145 displays 150 buffers 155 Process Scheduler 160 GPS 165 cameras 170 Applications 180 sensors 300 Interface 305 indicator 310 indicator 315 Notification 320 notifications 610 Computer Systems 612 Bus Subsystem 614 processor 616 Network Interface Subsystem 620 User Interface Output Device 622 User Interface Input Devices 624 Memory Subsystem 625 Memory Subsystem 626 File Storage Subsystem 630 Random Access Memory (RAM) 632 Read-Only Memory (ROM)

Claims

1. receiving sensor data in an application-controlled sandboxed feature detection process from an operating system of a client device; the sandboxed feature detection process is executed within a sandbox on the client device within the constraints of the sandbox imposed by the operating system; the sensor data is based on output from one or more sensors of the client device; processing the sensor data through the sandboxed feature detection process; determining whether features are present in the sensor data based on processing the sensor data; if it is determined that the feature is present in the sensor data, providing an indication to the operating system that the feature is present in the sensor data; A method comprising:

2. receiving, in the application, at a non-sandboxed interactor process controlled from the operating system in response to providing the instruction to the operating system; and The method of claim 1 further comprising:

3. transmitting, by the non-sandboxed interactor process, the at least a portion of the sensor data over a network to one or more remote devices; 3. The method of claim 2, further comprising:

4. receiving, at the non-sandboxed interactor process, from the sandboxed feature detection process, emitted data generated by the sandboxed feature detection process based on the processing of the sensor data and / or based on further processing of the sensor data and within constraints imposed by the sandbox; 3. The method of claim 2, further comprising:

5. The method of claim 4 , wherein at least a portion of the outgoing data is generated by the sandboxed feature detection process based on further processing of the sensor data.

6. 6. The method of claim 5, wherein the sensor data includes audio data, the features include hot words, the further processing includes processing the audio data using a speaker identification model, and the at least some of the transmitted data includes an indication of a user who uttered the hot words.

7. The method of claim 1 , wherein the sensor data includes audio data.

8. The method of claim 7 , wherein the features include hot words.

9. The method of claim 1 , wherein the sensor data includes image data.

10. The method of claim 9 , wherein the feature comprises a gesture of the user.

11. The method of claim 9 , wherein the characteristic comprises a user's gaze directed toward the client device.

12. A computer readable medium storing instructions that, when executed, cause the method of any one of claims 1 to 11 to be performed.

13. A client device comprising a memory for storing instructions and one or more processors operable to execute said instructions to perform the method of any one of claims 1 to 11.

Images