Safety audio indicator for electronic devices

Abstract

This disclosure relates to safety audio indicators for electronic devices. Aspects of this subject matter relate to electronic devices having speakers and input components. The electronic device can generate a safety audio indicator to be output by one or more speakers, indicating the operational status of one or more input components. The safety audio indicator can be generated by a safety audio processor. A safety microphone or other safety transducer can also be used, or alternatively, to confirm whether the audio indicator has been output during the expected output time. Other aspects of this subject matter relate to electronic devices having speakers and the ability to pair with other electronic devices. The electronic device can use an audio pairing signal from the speaker of the electronic device to pair with another electronic device. The electronic device can also detect and alter or disrupt attempts to use the speaker of the electronic device to emit a deceptive audio pairing signal.

Description

Cross-references to related applications

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 757,771, filed February 12, 2025, entitled "Secure Audio Indicators for Electronic Devices," the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] This specification relates in its entirety to electronic devices, including, for example, safety audio indicators for electronic devices. Background Technology

[0003] Electronic devices are typically equipped with cameras and microphones to capture images, video, and audio. An indicator light is usually located at or near the camera, and this light is activated to indicate that video or audio is being recorded. Some electronic devices use microwave communication to pair with other electronic devices to enable communication between the paired devices. Attached Figure Description

[0004] Specific features of this subject matter are set forth in the appended claims. However, for illustrative purposes, several aspects of this subject matter are illustrated in the following figures.

[0005] Figure 1 Example physical environments of electronic devices including speakers and input components are illustrated according to various aspects of the technology of this subject.

[0006] Figure 2 A schematic block diagram illustrating an example electronic device having a speaker and input components according to various aspects of the subject matter is shown.

[0007] Figure 3 A schematic block diagram illustrating an example electronic device having a speaker, a security microphone, and an input component according to various aspects of the subject matter is shown.

[0008] Figure 4 A schematic block diagram illustrating an example secure audio processor based on various aspects of the technology in this subject matter is shown.

[0009] Figure 5 A schematic block diagram illustrating another example of a secure audio processor based on various aspects of the technology in this subject matter is shown.

[0010] Figure 6 A schematic block diagram illustrating an example device having a secure audio processor and a secure audio input processor according to various aspects of the subject matter is shown.

[0011] Figure 7 This is a flowchart illustrating exemplary operations that can be performed to provide a safe audio indicator based on various aspects of the technology of this subject.

[0012] Figure 8 This is a flowchart of other exemplary operations that can be performed to provide a safe audio indicator, based on various aspects of the technology of this subject.

[0013] Figure 9 This is a flowchart illustrating exemplary operations for security monitoring of audio indicators, based on various aspects of the technology in this subject matter.

[0014] Figure 10 A schematic block diagram illustrating another example of a secure audio processor based on various aspects of the technology in this subject matter is shown.

[0015] Figure 11A and Figure 11B Example spectrograms of audio signals before and after modification of a portion of the audio signal are illustrated according to various aspects of the techniques described in this subject matter.

[0016] Figure 12 Examples of secure audio processors configured to switchably bypass masking operations are illustrated according to various aspects of the techniques in this subject matter.

[0017] Figure 13 Examples of secure audio processors configured to switch masking operations on and off are illustrated according to various aspects of the technology in this subject matter.

[0018] Figure 14 This is a flowchart illustrating exemplary operations for secure audio pairing based on various aspects of the technology in this subject matter.

[0019] Figure 15 A flowchart illustrating an example process, performed by an application to control an electronic device, according to one or more specific implementations of the subject matter, is shown.

[0020] Figure 16 A flowchart illustrating another example process, which can be executed by an application to control an electronic device to perform a method according to one or more specific implementations of the subject matter, is shown.

[0021] Figure 17 Examples of devices based on one or more specific implementations of the subject matter are shown.

[0022] Figure 18 Examples of one or more specific implementations of systems based on the techniques of this subject are provided.

[0023] Figure 19 Examples of one or more specific implementations of the technology according to this subject matter are illustrated. Figure 15The ladder diagram corresponding to the operation.

[0024] Figure 20 Examples of one or more specific implementations of the technology according to this subject matter are illustrated. Figure 16 The ladder diagram corresponding to the operation.

[0025] Figure 21 Examples of electronic systems that can be used to implement one or more specific embodiments of the subject matter are illustrated. Detailed Implementation

[0026] The detailed description shown below is intended as a description of various configurations of the subject matter and is not intended to represent the only configuration in which the subject matter can be practiced. The accompanying drawings are incorporated herein and form part of the detailed description. The detailed description includes specific details intended to provide a thorough understanding of the subject matter. However, it will be clear and apparent to those skilled in the art that the subject matter is not limited to the specific details shown herein and can be practiced without such specific details. In some cases, well-known structures and components are shown in block diagram form in order to avoid obscuring the concepts of the subject matter.

[0027] Electronic devices, including portable electronic devices (such as mobile phones, portable music players, tablet computers, laptop computers), wearable devices (such as smartwatches, smart glasses, head-mounted devices (HMDs), headphones, earphones, other wearable devices, etc.), typically include one or more input components, such as microphones and / or cameras.

[0028] In some devices, visual indicators, such as indicator lights, are provided that can be activated when the input component is in use to indicate (e.g., to the user of the device and / or others around the device) that audio and / or video recording is in progress. However, in some devices (e.g., compact devices with limited housing space), providing indicator lights and / or providing them in a manner visible to the user of the device (e.g., and / or others) when activated may be impractical. For example, when the device is worn on the user's wrist or close to the user's eyes, it may be difficult to position the indicator light in a location easily visible to the user. In some use cases, indicator lights may not be the most efficient way to indicate that the input component is in use, regardless of whether they are provided. For example, for visually impaired individuals, audio and / or haptic indicators may be more effective than indicator lights in providing indication.

[0029] According to one or more specific embodiments of the subject matter, a first audio indicator may be output by the speaker of a device to indicate that one or more input components of the device are in use. A second audio indicator may be output by the speaker of the device to indicate that one or more input components are no longer in use. The aspects disclosed in this subject matter can help securely ensure that when one or more input components are activated, the first audio indicator is indeed output by the speaker of the device, and / or that when these input components are still in use, the second audio indicator is not output by the speaker (e.g., to deceive).

[0030] For example, the first and second audio indicators can be generated in a manner that prevents the first audio indicator from being blocked from output and / or prevents the second audio indicator from being copied or spoofed (e.g., by malware or third-party applications running on the device). In this way, a secure audio indicator can be provided.

[0031] In one or more embodiments, such a security audio indicator may be generated by a security audio processor that controls the output of a device's speaker. For example, the device's hardware may be configured such that only the security audio processor (e.g., not the device's main processor) is communicatively coupled to the speaker. In this way, any spoofed version of the audio indicator can be removed by the security audio processor before the speaker can output that spoofed version. As an example, the security audio indicator may include low-frequency (e.g., less than 100 Hz and / or infrasound) output generated by the security audio processor, or a spatialized audio output spatialized by the security audio processor to be perceived at a specific location relative to the user / wearer of the device. In one or more embodiments, a security microphone may be used to confirm that the audio indicator is output as expected. For example, a security microphone may be provided by providing a direct (e.g., tamper-proof) path from the microphone to a security audio input processor that performs the confirmation that the audio indicator is output as expected.

[0032] As discussed in further detail below, the presence of a secure audio processor controlling the speaker output of a device (e.g., where the device's hardware can be configured such that only the secure audio processor is communicatively coupled to the speaker) facilitates other secure audio functions of the device, such as secure audio pairing operations. For example, an electronic device may be provided with the ability to generate audio output including encoded information (such as encoded passwords, credentials, authorization information, or pairing information) in a manner that prevents malicious software (or third-party applications) running on the electronic device from generating spoofed audio output with additional encoded information. For example, in a normal operating scenario, audio output including encoded information, utilizing a valid password from a valid source, can be generated by the secure audio processor controlling the speaker output of the device. If another (e.g., insecure) process at the device attempts to output audio with encoded information, the secure audio processor can modify at least a portion of the audio signal before all encoded information can be successfully output by the speaker.

[0033] For example, a secure audio processor may include a detector that mimics at least a portion of a decoder used by a receiving device to encode information. The detector may detect a first portion of a non-secure audio signal indicating that encoded information and / or information that can be interpreted as encoded information is about to be emitted by a speaker, and may enable the secure audio processor to modify at least a portion of a second portion of the non-secure audio signal before it is emitted by the speaker of the electronic device, to modify the encoded information (e.g., and / or audio formatting information, such as one or more acoustically encoded acknowledgment pulses). Modifications to the encoded information and / or audio formatting information may include, but are not limited to: suppressing temporal and frequency localization regions of the signal (e.g., through adaptive filters), adding to the signal to change how the encoded information is interpreted, nonlinear modifications to the signal (such as differentiating the high and low portions of signal samples), and / or timing alterations (such as time shifts or phase distortion).

[0034] Figure 1 The image shows an exemplary electronic device including a speaker and one or more input components. Figure 1 In the example, the electronic device 100 has been implemented using a housing small enough to be portable and carried or worn by the user. For example, Figure 1 The electronic device 100 can be a handheld electronic device (such as a tablet computer, cellular phone, or smartphone), a wearable device (such as a smartwatch, smart glasses, head-mounted device (HMD), pendant device, headlamp device), etc. Figure 1In the example, electronic device 100 includes a display, such as display 110 mounted to housing 106 (e.g., a frame). Electronic device 100 may include one or more input components, such as a touchscreen, buttons, switches, knobs, crowns, one or more microphones 115, one or more cameras (such as camera 119), and / or other input components incorporated into display 110. Input components may be located on or behind display 110, and / or on, within, or behind portions of housing 106. Display 110 and / or housing 106 may include one or more openings to accommodate one or more buttons, speakers, microphones, light sources, and / or cameras (as an example). In various specific implementations, display 110 may be an opaque display or a transparent display (e.g., through which a user can directly view their physical environment, and the displayed content may be displayed and / or projected onto the display, such as in a form combined with the user's direct view of the physical environment).

[0035] exist Figure 1 In the example, housing 106 (e.g., frame and / or shell) includes openings such as opening 108, opening 111, and / or opening 118. For example, opening 108 may form a port for one or more corresponding speakers 114. Figure 1 In the example, opening 108 forms a speaker port for a speaker 114 disposed within housing 106. In one or more embodiments, one or more speakers in speaker 114 may be safety speakers, as discussed in further detail below.

[0036] In various specific embodiments, housing 106 and / or display 110 may also include other openings, such as openings for one or more input components, such as one or more microphones 115, one or more pressure sensors, one or more cameras 119 (e.g., visible light cameras, infrared cameras, color imagers, monochrome imagers), depth sensors, and / or other components that receive signals from and / or provide signals to the environment outside housing 106. Figure 1 In the example, opening 111 forms a port for the corresponding microphone 115. In one or more embodiments, one or more microphones among the microphones 115 may be security microphones, as discussed in further detail below. Figure 1In the example, opening 118 forms a port for camera 119. Some input components, such as microphone 115 and / or camera 119, may have the ability to capture, record, and / or store information from the environment surrounding electronic device 100. In one or more embodiments, one or more speakers in speaker 114 may be used to securely generate audio output 129, which includes audio indicators (e.g., ringing, tinkling, clicking, or other predetermined notification sounds) to indicate when one or more of microphone 115 and / or camera 119 begins or ends capturing information.

[0037] Because the information captured by microphone 115 and / or camera 119 may include private and / or personal information about the user and / or other people around electronic device 100, it may be desirable to be able to securely control and / or confirm the output of these audio indicators, as discussed in further detail below.

[0038] Furthermore, it may be desirable to prevent electronic device 100 from using audio outputs (such as audio output 129) from electronic device 100 to perform unauthorized pairing with other electronic devices (such as electronic device 131 or electronic device 133). For example, in some implementations, electronic device 100 may be configured to include encoded information in audio output 129, such as encoded pairing information (e.g., numeric, alphanumeric, or other forms of passwords, credentials, or other authorization and / or pairing information), for pairing electronic device with another electronic device such as electronic device 131 (e.g., an audio output device such as a speaker, smart speaker, headset, or earphone, another media output device such as a television, monitor, etc., or an accessory device such as a smartphone or tablet device, which may have higher power, processing, and / or communication capabilities than electronic device 100). Electronic device 131 may include a microphone 132 for receiving audio output 129 and may include processing circuitry configured to execute decoder 135 to decode the encoded pairing information corresponding to audio output 129 in the microphone signal from microphone 132 and use the pairing information to pair electronic device 131 with electronic device 100. For example, the audio encoding / decoding sequence could be part of a direct device authentication process between the two devices. Once pairing is complete using the encoded pairing information in audio output 129, electronic devices 100 and 131 can wirelessly exchange information (e.g., using electromagnetic signals such as Bluetooth or WiFi signals). For example, electronic device 100 can receive audio data captured by microphone 132 of electronic device 131.

[0039] When the user of electronic device 100 and / or electronic device 131 approves pairing, these audio-based pairing operations can be legitimately performed by electronic devices 100 and 131. However, because audio output 129 is insecure once it leaves speaker 114, malware (e.g., malware on electronic device 100 or another device such as electronic device 133) may be able to capture audio output 129 and deduce the audio format that electronic device 131 expects to receive encoded pairing information. Malware on electronic device 100 can reproduce this deduced audio format to attempt illegitimate pairing between electronic device 100 and electronic device 131 (e.g., pairing not approved by the user of electronic device 100 and / or electronic device 131, pairing without the user's knowledge, and / or pairing without the knowledge of the operating system of electronic device 100).

[0040] Because unauthorized pairing of electronic devices 100 and 131 could grant malware at electronic device 100 access to microphone 132, camera, other hardware features, and / or software features (e.g., including information stored in memory) at electronic device 131, it may be desirable to securely control the ability of speaker 114 to include encoded information in audio output 129, as discussed in further detail below. The ability to securely control speaker 114 to include, exclude, or wholly or partially modify encoded information in audio output 129 can also help prevent unauthorized pairing of electronic device 100 with malicious devices such as electronic device 133 (e.g., a third-party device associated with a third-party application at electronic device 100 that may attempt to covertly pair with malicious electronic device 133 using the audio pairing capabilities of electronic device 100, thereby enabling malicious electronic device 133 to execute code 139 for extracting information from paired electronic device 100) (e.g., via malware at electronic device 100). For example, if a malicious device is not prevented (e.g., covertly) from pairing with electronic device 100 by the secure audio pairing technology described herein, the malicious device may be able to access one or more input components such as a microphone and / or camera, location information, contact information, and / or other personal and / or private information stored at electronic device 100.

[0041] Despite Figure 1The diagram shows three openings 108, one opening 111, and one opening 118, but this is merely illustrative. One, two, or more openings 108 may be provided on any of the various surfaces of the housing 106 for one, two, or more speakers 114. Similarly, one, two, or more openings 111 may be provided on any of the various surfaces of the housing 106 for one, two, or more microphones 115. Figure 1 In the example, a single opening 118 is provided for a single camera 119. However, this is merely illustrative. One, two, or more openings 118 may be provided on any of the various surfaces of the housing 106 for one, two, or more cameras 119.

[0042] In some implementations, one or more sets of openings and / or groups of openings in housing 106 may be aligned with a single port of an input component within housing 106. Housing 106, sometimes referred to as a chassis, enclosure, or frame, may be formed of plastic, glass, ceramic, fiber composites, metals (e.g., stainless steel, aluminum, etc.), other suitable materials, or any combination of two or more of these materials.

[0043] In one or more embodiments, two or more speakers 114 are operable to spatialize the audio output 129 of the electronic device 100 as perceived by the user or wearer of the electronic device 100 as originating from a location remote from the speakers 114 and / or the electronic device 100 (e.g., including a perceived location within the user's head and / or a perceived location in the external environment of the electronic device outside the user's head and remote from the electronic device 100).

[0044] Figure 1The configuration of the electronic device 100 is merely illustrative. In other embodiments, the electronic device 100 may be a computer, such as a computer integrated into a display (such as a computer monitor), a laptop computer, a media player, a gaming device, a navigation device, a computer monitor, a television, a headset, an earphone, a smartphone, a tablet computer, or other electronic equipment having at least one speaker and one or more input components. As discussed herein, in some embodiments, the electronic device 100 may be provided in the form of a wearable device (such as a smartwatch, a head-mounted device, or smart glasses). In one or more embodiments, the housing 106 may include one or more interfaces for mechanically coupling the housing 106 to one or more structures 123 (e.g., straps, arms, or other attachment mechanisms) configured to secure the housing 106 to a wearer (e.g., to the wearer's wrist or head). In one or more embodiments, when the electronic device 100 is worn by a user, one or more cameras of the electronic device 100 (such as camera 119) may be oriented toward the user's field of view (e.g., to facilitate recording video and / or capturing images that roughly correspond to what the user is watching).

[0045] Figure 2 Example architectures, implementable by electronic devices, are illustrated according to one or more specific implementations of the techniques described herein. For illustrative purposes, Figure 2 The architecture is described as being composed of Figure 1 The electronic device 100 is implemented, such as by one or more processors and / or memories of the electronic device; however, suitable portions of the architecture may be implemented by any other wearable electronic device. However, not all depicted components are usable in all embodiments, and one or more embodiments may include additional or different components compared to those shown in the figures. Variations in the arrangement and type of these components may be made without departing from the spirit or scope of the claims set forth herein. Additional components, different components, or fewer components may be provided.

[0046] Figure 2 The various parts of the architecture can be implemented in software, firmware, and / or hardware, including by one or more processors and a memory device containing instructions that, when executed by the processor, cause the processor to perform the operations described herein. For example, in Figure 2In the text, the rectangular box may indicate that the camera 119, microphone 115, display 110, processor 200, security audio processor 206, and speaker 114 may be hardware components, while the trapezoidal box may indicate that the application 202, system process 204, processing block 208, and security audio generator 210 may be implemented in software, including by one or more processors and a memory device containing instructions, which, when executed by the processor, cause the processor to perform the operations described herein.

[0047] exist Figure 2 In the example, electronic device 100 includes a first circuit (such as processor 200) and a second circuit (such as secure audio processor 206). As shown, secure audio processor 206 can provide audio signals to speaker 114 so that speaker 114 can output corresponding audio content. Figure 2 As shown, processor 200 can be securely isolated from speaker 114. For example, electronics 100 may not be provided with any direct communication path from processor 200 to speaker 114 (e.g., no communication path without passing through secure audio processor 206). In this way, secure audio processor 206 can be provided, which can securely control all audio outputs from some or all of speakers 114. In various embodiments, processor 200 and secure audio processor 206 may be implemented in separate physical processing architectures (e.g., on separate substrates, or as physically separated parts of a single substrate). In one or more other embodiments, secure audio processor 206 may be implemented in software, such as by implementing secure audio processor as a containerized processing environment that can be executed by processor 200 but cannot be modified by processor 200.

[0048] As shown, processor 200 can run one or more processes (e.g., execute their instructions), such as one or more applications 202 and / or one or more system processes 204. For example, system process 204 may be a process controlled by the operating system of electronic device 100. Application 202 and / or system process 204 can generate various audio signals for output by speaker 114, which is securely isolated from the processor 200 running these processes. As an example, application 202 can generate audio signals corresponding to audio content of media (e.g., music, podcasts, audiobooks, video content, and / or soundtracks of game applications), application sounds (e.g., ringtones, button sounds, click sounds, alarm sounds, notification sounds), and / or various other audio content. System process 204 can generate audio signals corresponding to system sounds (such as ringtones, button sounds, click sounds, alarm sounds, notification sounds) and / or various other audio content. Figure 2As shown, the secure audio processor 206 and speaker 114 can be securely isolated from the application 202 and system process 204 (e.g., the application 202 and system process 204 may not be able to access the secure audio processor 206 or directly control the speaker 114).

[0049] As shown, audio signals generated at processor 200 (e.g., by application 202 and / or system process 204) may be provided to secure audio processor 206 (e.g., instead of being directly provided to speaker 114). Secure audio processor 206 may include one or more processing blocks 208 that process audio signals received from processor 200. As an example, processing block 208 may apply one or more filters, noise reduction processes, volume adjustments, reverberation adjustments, spatialization operations, masking operations, distortion operations, detection operations, monitoring operations, speaker protection operations, and / or other audio signal processing operations to the received audio signal, and provide the resulting processed audio signal to speaker 114 for output.

[0050] As an example, a secure audio processor 206 (e.g., processing block 208) may remove (e.g., using a high-pass filter, low-pass filter, or band-pass filter) audio content from an audio signal received from processor 200 from one or more frequencies and / or one or more frequency ranges. For example, secure audio processor 206 (e.g., processing block 208) may remove low-frequency audio content, such as audio content with frequencies close to or below a cutoff frequency (e.g., a cutoff frequency located at or near the lower end of the human audible range, such as located at or near 20 Hz, or located at or near 100 Hz). For example, processing block 208 may remove low-frequency infrasound content that is inaudible to the human ear. Removing audio content at certain frequencies before providing the processed audio signal to speaker 114 for output can serve any of a variety of purposes. These purposes may include speaker protection (e.g., preventing the output of audio content that would cause speaker 114 to operate outside the speaker's approved or safe operating range), speaker efficiency (e.g., preventing unnecessary power consumption and thermal effects at speaker 114), and / or audio security (e.g., preventing speaker 114 from outputting unauthorized audio content). In one or more embodiments, the secure audio processor 206 may process the audio signal received from the processor 200 to remove audio content from the frequency range (e.g., the low-frequency range) reserved for audio indicators used by the microphone 115 and / or camera 119. In one or more embodiments, the secure audio processor 206 may process the audio signal received from the processor 200 to detect attempts to output unauthorized encoded content (e.g., passwords) in the audio output 129 from the speaker 114, and modify the audio signal (e.g., by suppressing temporal and frequency localization regions of the signal, adding to the signal to alter the interpretable encoded information, nonlinear modifications to the signal (such as differentiating the high and low portions of signal samples) and / or timing changes (such as time shifts or phase distortions), as discussed herein) to prevent the successful output of unauthorized encoded content.

[0051] like Figure 2 As shown, the secure audio processor 206 may also include a secure audio generator 210. For example, the secure audio generator 210 may generate audio signals in response to changes in the use of the microphone 115 and / or camera 119 detected by the processor 200, for use by the speaker 114 to output an audio indicator. As another example, the secure audio generator 210 may generate audio signals for outputting encoded content, such as for use with another device (e.g., Figure 1The secure audio generator 210 is paired with a password for the electronic device 131. In one or more embodiments, the secure audio generator 210 may be hard-coded in the secure audio processor. In one or more other embodiments, the secure audio generator 210 may be implemented in software stored in the secure memory of the secure audio processor 206 (e.g., memory inaccessible to the processor 200).

[0052] Processor 200 (e.g., system process 204 running on processor 200) may be configured to monitor the operational status of microphone 115 and / or camera 119 at electronic device 100. As an example, the operational status of microphone 115 and / or camera 119 may include an active or in-use state, an inactive or off state, a secure-use state (e.g., being used by a secure process at electronic device), or an insecure-use state (e.g., being used by an insecure process at electronic device). As an example, insecure processes at electronic device may include processes associated with third-party applications (e.g., applications provided by developers other than the provider of the electronic device's operating system), and / or any (e.g., first-party or third-party) process that stores input data (e.g., input data from microphone 115 and / or camera 119) and / or sends input data from the device (e.g., to another remote device).

[0053] In one or more embodiments, one or more secure processes at electronic device 100 can obtain input data from microphone 115 and / or camera 119 without providing an audio indicator. For example, a secure process at electronic device 100 may be a process that has received explicit permission from the user of electronic device 100 to obtain input data and does not store or send the input data. For example, a voice assistant application at electronic device 100 may have explicit user permission to obtain microphone signals from one or more microphones 115 and provide the microphone signals to a local machine learning model at electronic device 100, which has been trained to recognize trigger phrases for the microphone signals. In one or more embodiments, once a trigger phrase has been detected, electronic device 100 may output an audio indicator that microphone 115 is about to be used by an insecure process, and may then capture additional microphone signals and send these microphone signals or their encoded versions to a server for recognition of subsequent voice input. Subsequently, once the recording of subsequent voice input is complete, electronic device 100 may output another audio indicator to indicate that microphone 115 is no longer being used to capture data sent from the device.

[0054] As another example, a computer vision application at electronic device 100 may have explicit user permission to acquire image data from one or more cameras 119 and provide that image data to a local computer vision model at electronic device 100, which has been trained to recognize one or more predetermined objects (e.g., physical objects in the physical world, such as cars, bicycles, product logos, signs, etc.) in the image data. In one or more implementations, when a predetermined object is detected, electronic device 100 may generate audio or visual augmentation data for output to a user of electronic device 100 without storing or sending image data from the device.

[0055] When processor 200 determines that the operational state of some or all of the microphone 115 and / or camera 119 has changed (e.g., the microphone 115 and / or camera 119 has been activated, deactivated, accessed by an insecure process, or released by an insecure process), processor 200 may responsively provide a command to secure audio processor 206 to generate an audio indication of the corresponding change in the operational state of the microphone 115 and / or camera 119. For example, when the microphone 115 and / or camera 119 is activated or begins to be used by an insecure process, processor 200 may provide a first command to secure audio processor 206 to generate a first audio indicator, and when the microphone 115 and / or camera 119 is deactivated or released by an insecure process, processor 200 may provide a second command to secure audio processor 206 to generate a second audio indicator.

[0056] In response to receiving a first command from processor 200, security audio generator 210 may generate an audio signal (e.g., an audio indicator signal) that includes the audio content of a first audio indicator, indicating that microphone 115 and / or camera 119 is being used and / or is being used by an insecure process at electronic device 100. In response to receiving a second command from processor 200, security audio generator 210 may generate an audio signal (e.g., an audio indicator signal) that includes the audio content of a second audio indicator, indicating that microphone 115 and / or camera 119 is no longer being used and / or is no longer being used by an insecure process at electronic device 100.

[0057] Because the security audio processor 206 processes all audio output by the speaker 114, and therefore removes any spoofed versions of the audio indicator signal, only the audio indicator signal generated by the security audio generator 210 can be output by the speaker 114. In this way, the electronic device 100 is configured to provide security audio indicators (e.g., security audio notifications) for the status (or status changes) of the input components of the electronic device.

[0058] In one or more embodiments, electronic device 100 may include a plurality of speakers 114 and is capable of spatializing audio signals before they are output by the speakers 114. In this way, electronic device 100 can generate audio outputs that are to be perceived by a user or wearer of electronic device 100 as originating from any of a variety of three-dimensional spatial locations around electronic device 100, including locations remote from the speakers 114 themselves (e.g., spatially separated from them). In one or more embodiments, secure audio processor 206 can process audio signals received from processor 200 by spatializing them to be perceived at a first location (e.g., inside the user's head), and can spatialize audio indicator signals generated by secure audio generator 210 to be perceived at a second location different from the first location (e.g., outside the user's head and remote from the speakers 114). In this way, electronic device 100 can use the spatial distribution of audio outputs to distinguish securely generated audio notifications of the state (or state changes) of the input components of the electronic device from other audio outputs from electronic device 100 (e.g., application and / or system-generated sounds generated at processor 200).

[0059] In one or more specific implementations, a user of electronic device 100 may wish to connect electronic device 100 with another electronic device (such as...). Figure 1 Pairing electronic device 100 with electronic device 131. When (e.g., at processor 200) a request to pair electronic device 100 with electronic device 131 is received, processor 200 may respond by providing a command to secure audio processor 206 to generate secure audio pairing output including encoded information (e.g., encoded password, credentials or authorization information or pairing information).

[0060] In response to receiving a command from processor 200, secure audio generator 210 may generate an audio signal (e.g., a secure audio pairing signal) that includes encoded content. In one or more embodiments, secure audio processor 206 (e.g., secure audio generator 210) may include additional content in the secure audio pairing signal, such as: media content (e.g., music or melody) configured to make the overall audio output 129 including the encoded content sound pleasing to the user; and / or audio formatting content, such as one or more acknowledgment pulses, start pulses (e.g., configured to indicate the start of encoded content in the audio signal), one or more end pulses (e.g., configured to indicate the end of encoded content in the audio signal), and / or one or more synchronization pulses (e.g., configured to allow electronic device 131 to synchronize the timing of the encoded content).

[0061] In some implementations, the secure audio generator 210 adapts the encoded content based on the presence and form of the supplementary content (e.g., a musical sequence of notes arranged to please the user). For example, the secure audio generator 210 may generate an audio waveform for the encoded password whose timing and / or frequency content matches and / or complements the supplementary content. In one example where the supplementary content has a basic musical timing sequence of notes, the timing of the added password waveform may be synchronized with such notes. In one example where the supplementary content has a basic musical timing sequence of notes different on the scale, the added password waveform itself may be generated to present musical note characteristics that depend on the notes of the supplementary content (e.g., notes that are the same as and / or complementary to the notes of the supplementary content, and therefore may sound pleasing when combined with the notes of the supplementary content).

[0062] In one or more specific implementations, in a use case where processor 200 (e.g., application 200 executing on processor 200, such as a third-party application, which may be malicious) generates a spoofed secure audio pairing signal for output from speaker 114, secure audio processor 206 (e.g., processing block 208) may modify at least a portion of the spoofed secure audio processing signal to prevent encoded content (e.g., and / or audio formatting content) from being successfully included in the audio output from speaker 114.

[0063] Because the secure audio processor 206 processes all audio output from the speaker 114 and thus removes any spoofed versions of the secure audio pairing signal, only the audio pairing signal generated by the secure audio generator 210 can be output by the speaker 114. In this way, the electronic device 100 is configured to provide secure audio pairing for electronic devices.

[0064] exist Figure 2 In the example, the secure audio processor 206 is used to output secure audio indicators and / or secure encoded content using speaker 114. In one or more embodiments, the electronic device 100 may also, or alternatively, use secure microphone circuitry at the electronic device to confirm that speaker 114 has output an audio indicator signal associated with microphone 115 and / or camera 119. For example, Figure 3An embodiment is illustrated in which electronic device 100 includes a secure audio input processor 300 that receives microphone signals from a microphone 115S. In this example, the microphone 115S is a secure microphone because it is communicatively coupled to the secure audio input processor 300 (e.g., via a secure direct hardware communication path 315), and electronic device 100 does not include any communication path from any insecure (e.g., non-microphone) audio source (e.g., source 317, such as an artificial audio data generator that may be executed by malware or other malicious processes, including processes executing on processor 200) to the secure audio input processor 300. For example, audio input to the secure audio input processor 300 could be a hard-coded conductive path (e.g., a conductive trace in a silicon substrate) between the microphone 115 and the secure audio input processor 300. In various specific implementations, one or a subset of microphones in the microphones 115 of the electronic device 100 may be a secure microphone with a secure (e.g., tamper-proof, secure, direct hardware) communication path to the secure audio input processor 300, or all microphones 115 of the electronic device 100 may be secure microphones with a secure (e.g., tamper-proof, secure, direct hardware) communication path to the secure audio input processor 300. Although in Figure 3 The safety microphone 115S is shown, but in one or more other embodiments, another safety transducer (such as a safety inertial measurement unit (IMU) or its sub-components (e.g., accelerometers)), one or more independent accelerometers, and / or features and / or components of the speaker 114 itself (e.g., using safety current / voltage (IV) sensing operations, such as sensing the current and / or voltage of the speaker during the expected output time of the audio indicator and comparing the sensed current and / or voltage with the current and / or voltage of the speaker during the output of the previously stored / measured audio indicator) may be used in place of or as a supplement to the safety microphone 115S to verify the output of the audio indicator. As shown, in some embodiments, the microphone signal from the safety microphone 115S may be provided to other processes and / or components of the electronic device 100 (e.g., including processor 200); however, the input to the safety audio input processor 300 may be safety-protected (e.g., preventing sources, components, and / or processors other than the microphone 115 from providing audio input data to the safety audio input processor 300).

[0065] In one or more embodiments, processor 200 may notify secure audio input processor 300 of the expected output time of an audio indicator associated with an input component of the electronic device. Secure audio input processor 300 may acquire a microphone signal from microphone 115S during the expected output time and use these microphone signals to determine whether the audio indicator is output by speaker 114 during the expected output time (e.g., by determining whether the microphone signal includes a representation of the audio indicator). In various embodiments, secure audio input processor 300 may determine whether the audio indicator is output by speaker 114 during the expected output time by comparing the microphone signal with a previously stored reference microphone signal corresponding to the audio indicator, and / or by providing the microphone signal to a machine learning model at the electronic device, which has been trained to detect the audio indicator in the microphone signal. In this way, regardless of whether the audio indicator is generated by processor 200 (e.g., in embodiments where a secure audio processor is not provided or is unavailable) or by secure audio processor 206, electronic device 100 can verify whether the expected audio indicator is output by speaker 114 during the expected output time.

[0066] In one or more embodiments, the secure audio input processor 300 may also use a microphone signal from the microphone 115S to listen for audio indicator sounds at times other than the expected output time of the audio notification (e.g., all times during which the electronic device is powered on and / or all times during which the microphone and / or camera of the electronic device is in use). For example, the secure audio input processor 300 may be configured to detect audio indicators that are no longer in use (or no longer used by an insecure process) when one or more input components of the electronic device 100 are actually still in use (e.g., being used by an insecure process). In this way, the electronic device 100 may detect deceptive audio indicators that are being output (e.g., in embodiments where the secure audio processor 206 is not provided or is not available to remove such deceptive audio indicators before they are output by the speaker 114).

[0067] In one or more specific embodiments, the secure audio input processor 300 may use multiple secure microphones 115S (and / or other secure transducers) to verify whether the spatialized audio indicator is output at the correct (e.g., secure) spatial location. For example, the secure audio input processor 300 may acquire a microphone signal corresponding to the left channel output (e.g., using a first microphone near the left speaker of the electronics 100), acquire a microphone signal corresponding to the right channel output (e.g., using a second microphone near the right speaker of the electronics 100), and compare the left and right channel outputs to determine the spatial location of the audio outputs from the left and right speakers. In the example discussed herein where the secure audio indicator is output to be perceived within the user's head (e.g., using a mono audio output), the secure audio input processor 300 may compare the left and right channel outputs to determine whether the left and right channel outputs are identical (e.g., mono outputs).

[0068] As discussed herein, the security audio processor 206 may be configured to provide security audio indicators (e.g., security audio notifications) of the status (or status changes) of input components of an electronic device, and / or securely include coded (e.g., pairing) information in the audio output of the speaker 114. Figures 4 to 9 Additional details are illustrated related to safety audio indicators (e.g., safety audio notifications) that provide the status (or status change) of input components of electronic devices. Figures 10 to 14 Additional details related to securely including coded pairing information in the speaker's audio output are illustrated.

[0069] Figure 4 This is a block diagram illustrating an example implementation of the secure audio processor 206. (See diagram for example.) Figure 4 As shown, in some specific implementations, the processing block 208 executed by the security audio processor 206 may include a filter 400 (e.g., a high-pass filter), a spatial processor 402 (e.g., configured to spatialize audio signals for spatial output by the speaker 114), a mixer 404 (e.g., a binaural mixer), and a speaker protection block 406. Figure 4In this example, a security command (e.g., from processor 200) may be received by a security audio processor 206 and provided to a security audio generator 210. In this example, the security audio generator 210 may be a low-frequency (e.g., infrasound) audio generator that generates low-frequency audio indicator signals, such as infrasound frequencies that are inaudible to the human ear or close to the lower end of the human audible frequency range. In one or more specific implementations, the security audio generator 210 may generate an audio indicator that includes an audible portion (e.g., a ringtone) and an infrasound or other extremely low-frequency portion. For example, the audio indicator may include a ringtone with an audible component (e.g., a frequency greater than 100 Hz) and includes (e.g., a segment) of extremely low-frequency components (e.g., infrasound components in the frequency range of 20 Hz–100 Hz and / or frequencies less than 20 Hz). In this example, this (segment) of low-frequency content moves the speaker diaphragm, and although the user / wearer may not hear the resulting low-frequency audio output, it may be perceived as tactile. Figure 2 , Figure 3 and Figure 4 In this arrangement, incoming audio from outside the security domain (e.g., incoming audio generated outside the security audio processor 206) will not be able to produce such a tactile sensation using speaker 114. In one or more embodiments, speaker 114, which outputs low-frequency and / or infrasound audio content, may be a standard speaker (e.g., having a diaphragm actuated in response to a current in a voice coil mounted near one or more magnets). In one or more other embodiments, speaker 114, which outputs low-frequency and / or infrasound audio content, may include features for enhancing the tactile sensation of the low-frequency and / or infrasound output (e.g., an acoustically driven mechanical resonator and / or a multi-degree-of-freedom transducer with a suspended motor).

[0070] As shown, an insecure audio signal (e.g., an audio signal generated at and / or received from processor 200) may also be received at secure audio processor 206 and provided to filter 400. Filter 400 may remove (e.g., filter out) predetermined portions of the insecure audio signal. For example, filter 400 may be a high-pass filter that removes low-frequency audio content from the insecure audio signal. In one or more use cases, this filtering by filter 400 may remove spoofed low-frequency audio indicators generated at processor 200. In this example, filter 400 and secure audio generator 210 may be complementary processing blocks that work together to help ensure that only the low-frequency output from speaker 114 is a secure audio indicator generated by secure audio processor 206. In one or more implementations, filter 400 and / or one or more other processes of secure audio processor 206 may also be used to modify portions of the audio signal that have been determined to include spoofed audio pairing information, as described below (e.g., in conjunction with...). Figures 10 to 14This will be discussed in further detail. Figure 4 In the example, the filtered audio signal can be provided to the spatial processor 402 for spatialization.

[0071] As shown in the figure, the spatialized and filtered audio signal, as well as the audio indicator signal generated by the safety audio generator 210, can be provided to the mixer 404. The mixer 404 can combine the spatialized and filtered audio signal with the audio indicator signal generated by the safety audio generator 210 into one or more combined audio signals (e.g., a left combined audio signal for the left speaker output and a right combined audio signal for the right speaker output) for output. Figure 4 In the example, speaker protection block 406 can process the combined audio signal and make one or more modifications to the combined audio signal to protect speaker 114 from potential damage that may occur due to the output of the combined audio signal (e.g., due to the speaker operating outside a predetermined operating range). As shown, the combined audio signal processed by speaker protection block 406 can then be provided to speaker 114 for speaker 114 output (e.g., as sound and / or vibration generated by speaker 114).

[0072] Figure 4 The example illustrates a specific implementation of the security audio processor 206, in which an audio indicator is output at a frequency different from other audio outputs (e.g., including infrasonic audio indicators) from the speaker 114. The infrasonic audio indicator causes the speaker 114 to generate vibrations that can be felt by the user / wearer of the electronic device 100 but not heard by them. In addition to the security benefits described herein, this type of infrasonic indicator output by the speaker can also be helpful, for example, to blind, deaf, and / or hearing-impaired users / wearers. However, the electronic device 100 may also provide other types of audio indicators. For example, as discussed herein, in one or more implementations, the security audio processor 206 may be configured to spatialize the audio indicator as part of the secure output of the audio indicator.

[0073] Figure 5 An example implementation of a secure audio processor 206 is illustrated for spatializing audio indicators as part of the secure output of the audio indicators. Figure 5In this example, the safety audio generator 210 is implemented as a spatial audio generator. For example, the safety audio generator 210 could be a spatial audio generator for a reserved location, configured to generate spatialized audio signals for an audio indicator that, when output by the speaker 114 of the electronic device 100, are perceived by (e.g., by a user or wearer of the electronic device) as originating from (e.g., a predetermined) three-dimensional location (e.g., a safety location) reserved for an audio indicator associated with an input component of the electronic device 100. In this example, the spatial processor 402 may not be able to spatialize the filtered audio signal from the filter 400 to the reserved (e.g., safety) location. For example, the spatial processor 402 could force all unsafe audio to be spatialized to unsafe locations (e.g., locations not reserved for audio indicators generated by the safety audio generator 210). In this way, the audio indicator output by the speaker 114 can be perceived by the user or wearer at a specific location different from the location where other sounds generated by the device are perceived.

[0074] Figure 6 An example is given, in which Figure 4 or Figure 5 The secure audio processor 206 is provided together with the secure audio input processor 300 (e.g., and the secure microphone 115S, as discussed herein). As shown, in one or more embodiments, commands provided from processor 200 to secure audio generator 210 can also be provided from secure audio processor 206 to secure observer operation 600 of secure audio input processor 300. In this way, secure observer operation 600 can securely know (e.g., via commands received from secure audio processor 206) any expected output time when speaker 114 is expected to output an audio indicator. In this way, secure observer operation 600 can confirm whether the expected audio indicator is output during the expected output time (e.g., by comparing a microphone signal from secure microphone 115S with a previously stored reference microphone signal corresponding to the audio indicator, and / or by providing the microphone signal to a machine learning model that has been trained to detect audio indicators in the microphone signal), and / or detect unauthorized or spoofed audio indicators output at other times (e.g., by detecting a microphone signal that includes a representation of one or more audio indicators when no audio indicator is expected).

[0075] exist Figure 6 In the example, the secure audio processor 206 is provided in combination with the secure audio input processor 300. In one or more other specific embodiments, the secure audio processor 206 may be provided without the secure audio input processor 300 (e.g., as in...). Figure 4 and Figure 5 In the example), or a secure audio input processor 300 may be provided without providing a secure audio processor (e.g., relying on the operation of the secure audio input processor 300 to provide security for the audio indicator through authentication and spoofing detection). In specific implementations (such as...) Figure 6 In a specific implementation where the secure audio processor 206 is provided in combination with the secure audio input processor 300, the secure audio input processor 300 can provide further verification of the secure audio indicator from a second secure compartment or container (e.g., communicatively isolated from the secure audio processor 206). For example, the secure microphone 115S and the secure audio input processor 300 can also verify that the speaker hardware itself has not been hacked.

[0076] Figure 7 A flowchart illustrating an example process for providing a security audio indicator according to one or more specific implementations is provided. For purposes of explanation, this document primarily refers to... Figure 1 and Figure 2 The process 700 is described using electronic devices 100 and speakers 114. However, the process 700 is not limited to... Figure 1 and Figure 2 The electronic device 100 and speaker 114, and one or more blocks (or operations) of process 700 may be performed by one or more other components and other suitable devices. Further, for illustrative purposes, the blocks of process 700 are described herein as occurring sequentially or linearly. However, multiple blocks of process 700 may occur in parallel. Furthermore, the blocks of process 700 need not be performed in the order shown, and / or one or more blocks of process 700 need not be performed and / or may be replaced by other operations.

[0077] exist Figure 7 In the example, at box 702, a system process (e.g., system process 204) running on a processor (e.g., processor 200) of an electronic device (e.g., electronic device 100) may provide a command to a secure audio processor (e.g., secure audio processor 206) at the electronic device to generate audio notifications (e.g., audio indicators) indicating that one or more input components of the electronic device (e.g., one or more microphones and / or cameras) are being used. In one or more implementations, providing the command may include providing the command in response to a determination by the system process that one or more input components of the electronic device are being used by an insecure process at the electronic device. For example, the insecure process may store and / or export information obtained by one or more input components.

[0078] At block 704, the secure audio processor can generate an audio signal in response to a command. In one or more embodiments, generating the audio signal using the secure audio processor may include generating low-frequency audio content for speaker output. For example, the low-frequency audio content may be audio content with frequencies close to or below the human audibility threshold (e.g., close to or below 20 Hz or close to or below 100 Hz). In one or more embodiments, the secure audio processor may also receive from a processor another audio signal (e.g., an unsecured audio signal) including additional low-frequency audio content; remove the additional low-frequency audio content from the other audio signal; and provide the audio signal with the low-frequency audio content and the other audio signal from which the additional low-frequency audio content has been removed to the speaker for simultaneous speaker output. In this way, the secure audio processor can prevent spoofed versions of the low-frequency audio content in the audio signal from being output by the speaker. In this way, the secure audio processor can allow audio generated by application 202 and / or system process 204 to be output from speaker 114, while preventing these applications 202 and / or system processes 204 from including spoofed versions of audio indicators output by the speaker.

[0079] In one or more embodiments, generating an audio signal using a secure audio processor may include: spatializing the audio signal so that it is perceived at a specific spatial location relative to a user of the electronic device when output by a speaker and another speaker. In one or more embodiments, the secure audio processor may also receive another audio signal from a processor; spatialize that other audio signal so that it is perceived at one or more locations other than the specific spatial location of the audio signal when output by a speaker and another speaker; and provide the spatialized audio signal and the other spatialized audio signal to the speaker for simultaneous output by the speaker. As an example, the specific spatial location may be within the user's head. For example, secure audio generator 210 may generate a mono audio version of an audio indicator that, when output by speaker 114, is perceived by the user of electronic device 100 as originating from within the user's head. In one or more embodiments, secure audio processor 206 (e.g., spatial processor 402) may prevent any other audio output to that location (e.g., prevent output as mono audio).

[0080] At box 706, an audio signal can be provided from the secure audio processor to the speakers of the electronic device (e.g., one or more speakers in speaker 114) for speaker output. For example, the processor may be prevented from providing the audio signal directly to the speakers. For example, the electronic device may not have any communication path between the processor and the speakers that does not pass through the secure audio processor. For example, there may not be a conductive trace, wire, or other physical communication connection directly from the processor 200 to the speaker 114.

[0081] In one or more embodiments, process 700 may further include obtaining a microphone signal using a microphone of an electronic device (e.g., a security microphone 115S) during the expected output time of the audio signal output by the speaker; and having the processor use the microphone signal to confirm that the audio signal is output by the speaker during the expected output time.

[0082] Figure 8 A flowchart illustrating another example process 800 for providing a security audio indicator according to one or more specific implementations is provided. For illustrative purposes, this document primarily refers to... Figure 1 and Figure 2 The process 800 is described using electronic devices 100 and speakers 114. However, the process 800 is not limited to... Figure 1 and Figure 2 The electronic device 100 and speaker 114 are included, and one or more blocks (or operations) of process 800 may be performed by one or more other components and other suitable devices. Further, for illustrative purposes, the blocks of process 800 are described herein as occurring sequentially or linearly. However, multiple blocks of process 800 may occur in parallel. Furthermore, the blocks of process 800 need not be performed in the order shown, and / or one or more blocks of process 800 need not be performed and / or may be replaced by other operations.

[0083] exist Figure 8 In the example, at box 802, a first circuit (e.g., processor 200) of an electronic device (e.g., electronic device 100) can run a process (e.g., application 202 and / or system process 204) that generates a first audio signal for output by a speaker (e.g., speaker 114) that is securely isolated from the first circuit.

[0084] At block 804, a second circuit of the electronic device (e.g., a secure audio processor 206) can receive the first audio signal from the first circuit. For example, the second circuit can receive the first audio signal from the first circuit via a communication path from the first circuit to the second circuit (e.g., a conductive trace on a substrate and / or in a flexible circuit or wire).

[0085] At block 806, a second circuit (e.g., processing block 208) may process the first audio signal from the first circuit. For example, the second circuit may process the first audio signal by removing first low-frequency content from the first audio signal (e.g., using filter 400). As another example, the second circuit may process the first audio signal by spatializing it so that it is perceived at a first location (e.g., a first location relative to the user or wearer of the electronic device) using spatial processor 402.

[0086] At box 808, the second circuitry can provide the processed first audio signal to the speaker for output. The speaker can then output first audio content based on the first audio signal. The second circuitry and the speaker can be safely isolated from the process.

[0087] At block 810, the second circuitry may generate a second audio signal for speaker output in response to a change in the use of one or more input components of the electronic device (e.g., microphone 115 and / or camera 119) detected by the first circuitry. For example, the second circuitry may generate the second audio signal in response to a command from the first circuitry. In one or more embodiments, a change in the use of one or more input components may include initiating an unsafe process at the electronic device to use one or more input components. In one or more embodiments, a change in use may include terminating an unsafe process at the electronic device to use one or more input components.

[0088] In one or more embodiments (e.g., including embodiments where the second circuit processes the first audio signal by removing first low-frequency content from the first audio signal), the second audio signal may include second low-frequency audio content generated by the second circuit. For example, the second low-frequency audio content may include infrasound content inaudible to the human ear. In one or more embodiments, the first low-frequency audio content removed from the first audio signal may include other infrasound content inaudible to the human ear. In one or more use cases, the first low-frequency audio content removed from the first audio signal may include a spoofed version of the second low-frequency audio content generated by the first circuit and the second circuit.

[0089] In one or more embodiments (e.g., including embodiments in which the second circuit processes the first audio signal by spatializing the first audio signal as being perceived at the first location), the second circuit may also spatialize the second audio signal as being perceived at a second location different from the first location (e.g., a location outside the user's head, such as a location away from the speaker that outputs the second audio signal).

[0090] In one or more embodiments, process 800 may also include using a security microphone circuit (e.g., security microphone 115S or other security transducers, and security audio input processor 300) to confirm the output made by the speaker corresponding to the second audio signal (e.g., at or during the expected output time of the second audio signal).

[0091] Figure 9 A flowchart illustrating an example process for confirming the output of an audio indicator, according to one or more specific implementations, is provided. For illustrative purposes, this document primarily refers to... Figure 1 and Figure 2 The process 900 is described using electronic devices 100 and speakers 114. However, the process 900 is not limited to... Figure 1 and Figure 2 The electronic device 100 and speaker 114 are included, and one or more blocks (or operations) of process 900 may be performed by one or more other components and other suitable devices. Further, for illustrative purposes, the blocks of process 900 are described herein as occurring sequentially or linearly. However, multiple blocks of process 900 may occur in parallel. Furthermore, the blocks of process 900 need not be performed in the order shown, and / or one or more blocks of process 900 need not be performed and / or may be replaced by other operations.

[0092] exist Figure 9 In the example, at box 902, one or more processors (e.g., processor 200 and / or secure audio processor 206) of an electronic device (e.g., electronic device 100) can generate audio notifications (e.g., audio indicators) that indicate that one or more input components of the electronic device (e.g., one or more microphones 115 and / or camera 119) are in use (e.g., being used by an insecure process at the electronic device).

[0093] At block 904, a secure audio input processor (e.g., secure audio input processor 300) may receive a microphone signal from a microphone (e.g., microphone 115S) via a secure direct hardware communication path between the microphone and the secure audio input processor during the expected output time of the audio notification. In one or more embodiments, one or more input components include at least one additional microphone (e.g., another microphone 115) having a communication path to one or more processors. In one or more embodiments, one or more input components include a camera (e.g., camera 119). In one or more embodiments, one or more input components may include a microphone.

[0094] At block 906, the secure audio input processor can determine, based on a microphone signal, whether an audio notification is output by the speaker of the electronic device during the expected output time. In one or more embodiments, one or more processors can terminate the operation of one or more input components in response to the secure audio input processor determining that an audio notification is not output by the speaker during the expected output time.

[0095] In one or more embodiments, one or more processors include: a first processor (e.g., processor 200) securely isolated from the speaker and configured to run an application (e.g., application 202) that generates an audio signal for speaker output; and a second processor (e.g., secure audio processor 206) configured to: receive the audio signal from the first processor and provide at least one version of the audio signal (e.g., a processed version processed by processing block 208) to the speaker for output; and generate another audio signal corresponding to an audio notification for speaker output during the expected output time.

[0096] In one or more embodiments, process 900 may further include utilizing a secure audio input processor to receive a signal from a microphone while one or more input components are in use (e.g., at a time other than the expected output time of an audio notification); determining, based on the other microphone signal, that the speaker has output another notification (e.g., another audio indicator) indicating that one or more input components are no longer in use; and providing a signal to one or more processors indicating that the speaker has output a spoofed audio notification. In one or more embodiments, one or more processors may terminate the operation of one or more input components and / or provide an alarm in response to the secure audio input processor determining that the speaker has output a spoofed audio notification.

[0097] Figure 10 This is a block diagram illustrating an example implementation of a secure audio processor 206 in a configuration for preventing spoofing of secure audio pairing outputs. (See diagram for example.) Figure 10 As shown, the processing block executed by the secure audio processor 206 (e.g., Figure 2 The specific implementation of the processing block 208 may include a masking block 1000, a detector 1002, and a combination of Figure 4 The mixer 404 is described. In this example, filter 400, space processor 402, and speaker protection block 406 are omitted. However, it should be understood that masking block 1000 and detector 1002 may be implemented together with filter 400, space processor 402, and / or speaker protection block 406 (e.g., in a specific implementation where a security audio processor is configured to provide both a security audio indicator and a security audio pair). For example, masking block 1000 may be switchably coupled to filter 400 before or after it. Figure 4 , Figure 5 and / or Figure 6 The signal processing path of the secure audio processor (e.g., a control signal generated by the detector in response to when the detector 1002 detects a spoof audio indicator) is bypassed when the detector 1002 does not detect a spoof audio indicator.

[0098] exist Figure 10 In the example, the secure audio generator 210 can generate an audio signal that includes encoded content (such as encoded passwords, credentials, authorization information, or pairing information). For example, in a normal operating scenario, such as a use case where a user expects to pair electronic device 100 with another device (such as electronic device 131), the secure audio generator 210 can generate an audio signal with valid encoded information and provide the audio signal with valid encoded information to speaker 114 for output. Valid encoded information may include one or more encoded bits corresponding to passwords, credentials, authorization information, pairing information, etc., and / or audio formatting information. For example, audio formatting information may include one or more pulses or other audio features indicating the start of encoded information (e.g., one or more trigger pulses configured to trigger a detector at another device to begin decoding operation), the end of encoded information, and / or synchronization information. For example, one or more start pulses and / or one or more end pulses may be configured to trigger a decoder at another device (e.g., the pairing target device). Figure 1 The decoder 135 decodes the encoded bits in the audio output from the speaker 114 and uses the decoded bits to perform a pairing operation. One or more synchronization pulses can also be used by a decoder at another device to synchronize the timing of the encoded bits.

[0099] As discussed herein, the signal carrying password and audio formatting information may include an auxiliary signal, such as a musical melody designed to please the user. In such cases, the generated signal carrying password and audio formatting information may be additionally matched with the auxiliary signal. Matching may include timing adjustments to such a signal to match events (e.g., notes) in the auxiliary signal, and / or generating waveforms that are pleasing in themselves when played in sync with the auxiliary signal; for example, the password may be a musically matching and / or complementary sequence of notes to the musical auxiliary signal. In such cases, the "password playback" generated by the secure audio generator 210 (which may include password and formatting signal generation operations) may receive the auxiliary signal itself to establish timing (e.g., detect notes) and introduce the auxiliary signal back into the password playback, appropriately synchronizing it with the password and formatting signal.

[0100] As shown in the figure, the secure audio processor 206 can also be configured to prevent an insecure audio source 1004 outside the secure audio processor 206 (e.g., an application 202 running on the processor 200, which could be malware or a third-party application) from successfully outputting an audio signal that includes spoofed encoded information. For example, if the insecure audio source 1004 attempts to output audio with an encoded password, the secure audio processor 206 (e.g., masking block 1000) can modify one or more portions of the audio signal before the entire encoded password sequence can be successfully output by the speaker 114.

[0101] For example, as shown, an audio signal 1006 (e.g., a non-secure audio signal) from processor 200 may be provided to both masking block 1000 and detector 1002 (e.g., to determine by detector 1002 whether the content in the signal can be interpreted as a cipher by a decoder). For example, detector 1002 may mimic at least a portion of a decoder (e.g., decoder 135) used by a receiving device for encoded information from secure audio generator 210. When audio signal 1006 includes spoofed encoded information (e.g., a spoofed audio pairing signal, which may include one or more information bits encoded into the audio signal), detector 1002 may detect a first portion (e.g., in time) of audio signal 1004 indicating that encoded information (e.g., encoded bits) is about to be transmitted. For example, the first portion of audio signal 1006 may include audio formatting information, such as a start pulse indicating the start of encoded information in audio signal 1006.

[0102] When detector 1002 detects audio formatting information, detector 1002 may cause masking block 1000 to modify at least a portion of the second part of the audio signal 1006 before the second part of the unsafe audio signal is emitted by speaker 114. When detector 1002 does not detect audio formatting information, detector 1002 may cause masking block to be turned off or bypassed to allow audio signal 1006 to pass to speaker 114 (e.g., in some specific implementations, after passing through filter 400, spatialization processor 402, mixer 404 and / or speaker protection block 406).

[0103] In one or more specific implementations, detector 1002 may detect audio formatting information in audio signal 1006 while (e.g., in parallel) the first portion of audio signal 1006 is being output by speaker 114. For example, when detector 1002 detects audio formatting information, detector 1002 may cause masking block 1000 to modify one or more (e.g., temporally) subsequent portions of audio formatting and / or encoding information in audio signal 1006. By modifying only the later portions of the audio signal while the earlier portions of the audio signal are being output by the speaker in this manner, detector 1002 may corrupt or obfuscate enough information in the encoded audio signal to prevent successful operation of the encoded information therein at a remote device without delaying (e.g., attempting to detect and / or modify / block the entire encoded sequence) legitimate audio output from processor 200.

[0104] For example, Figure 11A A spectrum diagram 1100 is shown, illustrating audio formatting information in the form of audio pulses 1104 that can be output by speaker 114. Figure 11AA spectrogram 1102 of the modified version 1004M of the audio pulse 1104 after modification by the masking block 1000 is also shown. In various examples, the masking block 1000 can apply various modifications to the audio pulse 1104. As an example, the masking block 1000 can be implemented as a filtering function, a scaling function, an adding function, a time offset function, a mute function, or a combination of one or more of these and / or other audio processing functions. For example, the filtering function can be a linear filtering function (e.g., a notch filter, a bandpass filter, a bandstop filter, a combination of these filters, etc.) or a nonlinear filter that masks the bit representation of the sample (e.g., Float32->Int16->Float32, or mantissa or exponent bit manipulation). For example, a scaling function can reduce or nonlinearly modify a portion of the audio signal. For example, an adding function can add additional content to the audio signal. In one or more embodiments, the modification can include a combination of filtering, scaling, and adding functions (e.g., scaling or nonlinearly modifying a notched portion of the audio signal by notching it, and then adding the scaled portion back to the notched signal). For example, a time-shifting function can time-shift one or more portions of an audio signal so that the portion of the audio signal still exists in the audio signal, but does not appear at the time when the decoder expects that portion of the audio signal (e.g., by applying an all-pass phase-modified filter with a large group delay in the expected frequency band). In one or more implementations, the modification can be configured to avoid violating subsequent constraints on the speaker's audio output (e.g., avoiding the generation of a modified audio signal in which a portion is cut off by the subsequent speaker protection block 406, resulting in unpleasant distortion from the speaker).

[0105] For example, to prevent more sophisticated attackers from anticipating modifications applied by the masking block 1000 and pre-distorting the spoofed audio signal, ensuring that the modifications applied by the masking block 1000 actually recover the intended encoded information, the modifications applied by the masking block 1000 can be different at different times. For example, the modification applied at any given time can be one of several predetermined modifications selected (e.g., by the security audio processor, detector, or masking block, e.g., randomly). In this way, attackers are prevented from anticipating modifications to their spoofed output.

[0106] In one or more specific implementations, the audio pulse 1104 modified by the masking block 1000 may be an end pulse or a confirmation pulse at or near the end of the portion of the audio signal 1006 that includes coded information, such as... Figure 11B A wider view of the spectrum diagrams 1100 and 1102 is shown.

[0107] For example, such as Figure 11BAs shown, an audio signal with encoded information may include an audio pulse 1105 in its first portion P1 (e.g., this audio pulse is configured as a start pulse or trigger pulse, which, when received by a decoder of another device (such as via the microphone of that other device), causes the decoder to search for and decode one or more encoded bits in the incoming audio stream). As shown, the audio signal may include encoded bits 1106 (e.g., bit pulses) and other audio content 1108 in its second portion P2. For example, the other audio content 1108 may include media content (e.g., music, podcast content, or other media content) and / or synchronization content (e.g., one or more synchronization pulses that may be used by the receiving device to synchronize audio signal content to the decoder). As shown, in one or more embodiments, different portions of an audio signal with encoded content (e.g., a legitimate audio signal with encoded content, or a deceptive audio signal that mimics a legitimate audio signal) may include different portions in different frequency bands.

[0108] exist Figure 11B In the example, different portions of the audio signal containing audio content are contained in different frequency bands. For example, audio pulses 1105 and 1104 may be transmitted in a first frequency band F1, the coded bit may be transmitted in a second frequency band F2 below the first frequency band F1, and the additional audio content 1108 may be transmitted in a third frequency band F3 below the second frequency band. Although in this example, frequency band F2 is lower than frequency band F1 and frequency band F3 is lower than frequency band F3, this is merely illustrative, and audio pulses 1105 and 1104, coded bit 1106, and audio content 1108 may be distributed in other frequency spatial arrangements (e.g., audio pulses 1105 and 1104 may be transmitted in frequency ranges that are higher than, lower than, overlap with, or deviate from the frequency range of transmitting coded bit 1106 and / or audio content 1108, and / or coded bit 1106 may be transmitted in frequency ranges that are higher than, lower than, overlap with, or deviate from the frequency range of transmitting audio content 1108). In some examples, audio pulses 1105 and 1104, code bit 1106, and audio content 1108 can be transmitted in the same frequency range.

[0109] exist Figure 11B In the example, audio pulse 1104 is modified by masking block 1000. In this example, a decoder at another device can receive audio pulse 1105, encoded bit 1106, and other audio content 1108, but encoded bit 1106 is prevented from being processed to prevent audio pulse 1104 (e.g., an acknowledgment pulse) from being detected by the other device. However, this is merely illustrative, and as combined with... Figure 10 Other modifications may be made to the discussed content, including modifications to the coded bit 1106 itself and / or to one or more synchronization pulses.

[0110] exist Figure 11B In the example, for the sake of simplicity in the current description, audio pulses 1104 and 1105, coded bit 1106, and other audio content 1108 appear identical in the spectrogram. However, it should be understood that bit pulse 1106 may have temporal and / or frequency characteristics of the coded bit and may differ from each other and / or from audio formatting pulses 1104 and 1105, and / or from other audio content 1108. Furthermore, audio pulses 1104 and 1105 may differ from each other, and other audio content 1108 may include audio content in pulse (e.g., sync pulse) and non-pulse forms.

[0111] As discussed herein, detector 1002 can be configured to control whether masking block 1000 applies modifications to audio signal 1006 received from processor 200, based on whether an indicator of encoded content (e.g., audio formatting information, such as audio pulse 1105 configured as a start pulse) is detected in the audio signal, by allowing the masking block to be bypassed or by enabling and disabling the masking block itself. For example, Figure 12 One specific implementation is illustrated, in which detector 1002 controls switch 1200 (e.g., a software switch) to switch between a path bypassing masking block 1000 (e.g., when detector 1002 does not detect audio pulse 1105 or another indicator of encoded content) and a path through masking block 1000 (e.g., when detector 1002 detects audio pulse 1105 or another indicator of encoded content). In this example, masking block 1000 may be operational even when its output is not used (e.g., not supplied to speaker 114). Figure 13 Another specific implementation is illustrated, in which detector 1002 controls masking block 1000 itself (e.g., by changing filter coefficients or other parameters to turn the masking operation of masking block 1000 on and off).

[0112] Figure 14 A flowchart illustrating an example process for providing secure audio pairing according to one or more specific implementations is provided. For illustrative purposes, this document primarily refers to... Figure 1 , Figure 2 , Figure 10 , Figure 12 and / or Figure 13 The process 1400 is described using electronic devices 100 and speakers 114. However, the process 1400 is not limited to... Figure 1 , Figure 2 , Figure 10 , Figure 12 and / or Figure 13The electronic device 100 and speaker 114 are included, and one or more blocks (or operations) of process 1400 may be performed by one or more other components and other suitable devices. Further, for illustrative purposes, the blocks of process 1400 are described herein as occurring sequentially or linearly. However, multiple blocks of process 1400 may occur in parallel. Furthermore, the blocks of process 1400 need not be performed in the order shown, and / or one or more blocks of process 1400 need not be performed and / or may be replaced by other operations.

[0113] exist Figure 14 In the example, at box 1402, an audio signal (e.g., audio signal 1006) can be received from the first circuitry of the electronic device (e.g., processor 200) at the second circuitry (e.g., electronic device 100) of the electronic device (e.g., security audio processor 206) for output by a speaker (e.g., speaker 114) that is securely isolated from the first circuitry.

[0114] At box 1404, a second circuit (e.g., detector 1002 of the security audio processor 206) may (e.g., in the first part of the audio signal, such as...) Figure 10 The part P1) detects an indicator (e.g., audio pulse 1105, such as a start pulse, trigger pulse, or a series or set of start and / or trigger pulses or sounds) in the encoded content (e.g., encoded bit 1106) of the audio signal (e.g., its second part, such as the subsequent part P2 that is temporally after the first part). For example, the encoded content may include an indicator (e.g., an audio pulse, such as a start pulse, a trigger pulse, or a series or set of start and / or trigger pulses or sounds) for connecting an electronic device to another electronic device (e.g., Figure 1 The password paired with electronic device 131 or electronic device 133.

[0115] At block 1406, in response to the detection, at least a portion (e.g., a second portion) of the audio signal can be modified (e.g., by a second circuit, such as by a masking block 1000 of the secure audio processor 206) to generate a modified portion of the audio signal (e.g., a modified second portion, such as a modified second portion including a modified audio pulse 1104M).

[0116] At block 1408, an audio signal including modified portions (e.g., a first portion of the audio signal and a modified second portion of the audio signal) may be provided to a speaker for output (e.g., output from speaker 114) (e.g., by a second circuit). For example, providing the audio signal including the modified portions to the speaker may include providing the first portion of the audio signal and the modified second portion of the audio signal to the speaker for output, which may include providing the first portion of the audio signal to the speaker for output while (e.g., in parallel) detecting an indicator (e.g., of upcoming encoded content) in the first portion of the audio signal.

[0117] In one or more embodiments, the indicator detecting an audio signal (e.g., its first portion) at block 1404 may include detecting one or more pulses (e.g., audio pulse 1105, such as a start pulse, a trigger pulse, or a series or set of start and / or trigger pulses or sounds) in the audio signal (e.g., its first portion), which are configured to trigger another electronic device (e.g., Figure 1 The decoder (e.g., decoder 135 or decoder 137) at electronic device 131 or electronic device 133 performs decoding operations for decoding the encoded content. For example, the one or more pulses may include at least one pulse (e.g., a start pulse and / or an acknowledgment pulse configured to indicate the start of encoded content in the audio signal), the at least one pulse being configured to indicate one or more of the start or acknowledgment of encoded content in the audio signal. In another example, the one or more pulses may also include one or more synchronization pulses (e.g., in other audio content 1108), the one or more synchronization pulses being configured to allow another device to synchronize the timing of the encoded content.

[0118] In one or more embodiments, modifying at least a portion (e.g., a second portion) of an audio signal may include distorting a subsequent pulse (e.g., an end pulse or a subsequent acknowledgment pulse, such as audio pulse 1104) following at least one pulse and in the audio signal (e.g., its second portion), the subsequent pulse being configured to indicate the end or subsequent acknowledgment of encoded content in the audio signal. In one or more embodiments, modifying at least a portion (e.g., a second portion) of an audio signal to generate a modified (e.g., second) portion of the audio signal may include distorting at least some of the encoded content. For example, the encoded content may include multiple encoded bits (e.g., encoded bit 1106), and distorting at least some of the encoded content may include modification (e.g., filtering, or other modifications, such as combining...) Figure 10 (As discussed) at least some of the multiple coded bits. In one or more embodiments, modifying at least a portion (e.g., a second portion) of an audio signal may include modifying the (e.g., second) portion of the audio signal using a first modification operation selected substantially randomly from a plurality of modification operations.

[0119] In one or more specific embodiments, the audio signal may include an indicator in a first frequency range (e.g., F1) of a first portion (e.g., P1) of the audio signal, include coded content (e.g., coded bit 1106) in a second frequency range (e.g., F2) that is different from (e.g., higher, lower, or deviated from) the first frequency range and is distributed across the first portion (e.g., P1) and the second portion (e.g., P2) of the audio signal, and include media content (e.g., additional audio content 1108) in a third frequency range (e.g., F3) that is different from (e.g., higher, lower, or deviated from) the second frequency range and is distributed across the first portion and the second portion of the audio signal.

[0120] In one or more embodiments, the audio signal may have been generated by an application (e.g., audio playback source 1004, such as application 202) running on a first circuit of the electronic device, and received by a second circuit via the electronic device's application programming interface (e.g., API 1890). In one or more embodiments, the other electronic device may be a wearable electronic device (e.g., electronic device 133) manufactured by a provider other than the electronic device's provider (e.g., a third-party provider), and the wearable electronic device includes code (e.g., code 139) for retrieving information (e.g., microphone signals, images, sensor signals, location information, stored data, etc.) from the electronic device after pairing.

[0121] exist Figure 14In the example, the audio signal could be an illegal or deceptive audio signal from an untrusted or insecure process at the electronic device. In one or more use cases, the electronic device may also (e.g., via a secure audio generator 210) generate a legitimate secure audio pairing signal for speaker output. For example, process 1400 may also include a second circuitry of the electronic device generating a secure audio pairing signal comprising: a legitimate indicator (e.g., a legitimate pulse 1105, whose legitimacy may stem from the user and / or the electronic device's operating system being aware of and / or approving the generation of the secure audio pairing signal) in a first frequency range (e.g., FR1) of a first portion (e.g., P1) of the secure audio pairing signal; and a legitimate indicator (e.g., a legitimate pulse 1105, whose legitimacy may derive from the user and / or the electronic device's operating system being aware of and / or approving the generation of the secure audio pairing signal) in a second frequency range (e.g., FR2) that is different from (e.g., higher, lower, or deviating from) the first frequency range and distributed across the first and second portions (e.g., P2) of the secure audio pairing signal. The following are considered as legitimate media content (e.g., legitimate code bit 1106, whose legitimacy may stem from the user and / or the electronic device's operating system being aware of and / or approving the generation of the secure audio pairing signal); and legitimate media content in a third frequency range (e.g., FR3) that is different from (e.g., higher, lower, or deviating from) the second frequency range and distributed across the first and second portions of the secure audio pairing signal (e.g., legitimate audio content 1108, whose legitimacy may stem from the user and / or the electronic device's operating system being aware of and / or approving the generation of the secure audio pairing signal); and the secure audio pairing signal being provided to the speaker for output by the second circuitry.

[0122] In one or more specific implementations, generating a secure audio pairing signal may include generating legally encoded content based at least in part on legal media content (e.g., by generating an audio waveform for the legally encoded content whose timing and / or frequency content matches and / or complements the legal media content). For example, a second circuit may receive or obtain one or more bits to be encoded in an audio signal and generate an audio signal to encode the one or more bits based on the timing and / or frequency characteristics of the legal media content.

[0123] Specific embodiments within the scope of this disclosure include a computer-readable storage medium encoded and organized for an application (e.g., Figure 17 Instructions (referring to the application of 1760) that, when executed by one or more processing units, control electronic devices (e.g., ...). Figure 17 Device 1750) performs Figure 7 , Figure 8 , Figure 9 and / or Figure 14 Methods Figure 15 Methods Figure 16 The methods and / or one or more other processes and / or methods described herein.

[0124] It should be recognized that the application of 1760 ( Figure 17 The application 1760 (shown in the diagram) can be any suitable type of application, including one or more of the following: browser applications, applications used as execution environments for plugins, widgets, or other applications, fitness applications, health applications, digital payment applications, media applications, social networking applications, pairing applications, hearing aid applications, messaging applications, and / or map applications. In some embodiments, application 1760 is an application pre-installed on device 1750 at the time of purchase (e.g., a first-party application). In other embodiments, application 1760 is an application provided to device 1750 via operating system update files (e.g., a first-party or second-party application). In some embodiments, application 1760 is an application provided via an app store. In some embodiments, the app store can be an app store pre-installed on device 1750 at the time of purchase (e.g., a first-party app store). In some embodiments, the app store is a third-party app store (e.g., an app store provided by another app store, downloaded via a network, and / or read from a storage device).

[0125] refer to Figure 15 and Figure 19 Information is obtained at S1760 (e.g., S1510). In some embodiments, at S1510, information is obtained from at least one hardware component of device 1750. In some embodiments, at S1510, information is obtained from at least one software module (e.g., instruction set) of device 1750. In some embodiments, at S1510, information is obtained from at least one hardware component external to device 1750 (e.g., peripheral device, accessory device, pairing device, and / or server). In some embodiments, the information obtained at S1510 includes location information, time information, image information, notification information, user information, environmental information, electronic device status information, microphone signals, images, health information, weather information, media information, historical information, event information, hardware information, and / or motion information. In one or more embodiments, the information obtained at S1510 may include images captured by external hardware components and / or microphone signals captured by external hardware components. In some implementations, in response to obtaining information at S1510 and / or thereafter, application 1760 provides information to the system (e.g., S1520).

[0126] In some implementations, the system (e.g., Figure 18 The system 1810 shown is an operating system hosted on device 1750. In some implementations, the system (e.g., Figure 18 The system 1810 shown is an external device (e.g., a server, peripheral device, accessory, pairing device, and / or personal computing device) that includes an operating system.

[0127] refer to Figure 16 and Figure 20 Application 1760 obtains information (e.g., S1630). In some embodiments, the information obtained at S1630 includes location information, time information, notification information, images, microphone signals, user information, health information, environmental information, electronic device status information, weather information, media information, historical information, event information, hardware information, and / or motion information. In one or more embodiments, the information obtained at S1630 may include images from one or more cameras, microphone signals from one or more microphones, and motion information from one or more accelerometers, gyroscopes, magnetometers, and / or GPS components. In one or more embodiments, the information obtained at S1630 may include information stored in the memory of device 1750. In one or more embodiments, the information obtained at S1630 may be obtained in response to (e.g., at S1520) providing first information (e.g., encoded audio information) to the operating system. In response to obtaining information at S1630 and / or thereafter, application 1760 performs an operation using the information (e.g., S1640). In some implementations, the operations performed at S1640 include: providing notifications based on information, transmitting messages based on information, displaying information, controlling the user interface based on information, sending information to a remote device or system, controlling the user interface based on information, and / or invoking the API of system 1810 based on information. In one or more specific implementations, the operations performed at S1640 include those described herein. Figures 2 to 10 , Figure 12 , Figure 13 and / or Figure 14 One or more of the operations described.

[0128] In some implementations, execution is performed in response to a trigger. Figure 15 Methods and / or Figure 16 The method involves one or more steps. In some implementations, triggering includes detecting an event, receiving a notification from system 1810, user input, and / or responding to a call to an API provided by system 1810.

[0129] In some implementations, when the instructions of application 1760 are executed, control device 1750 executes them by calling an application programming interface (API) (e.g., API 1890) provided by system 1810. Figure 15 Methods and / or Figure 16 The method. In some implementations, application 1760 executes without calling API 1890. Figure 15 Methods and / or Figure 16 At least a part of the method.

[0130] In some implementation schemes, Figure 15 Methods and / or Figure 16 One or more steps of the method involve calling the API (e.g., API 1890) using one or more parameters defined by the API. In some implementations, one or more parameters include constants, keys, data structures, objects, object classes, variables, data types, pointers, arrays, lists, or pointers to functions or methods and / or references to data or other items to be passed via the API in another way.

[0131] refer to Figure 17 Example 1750 is shown. In some embodiments, device 1750 is a personal computing device, smartphone, smartwatch, fitness tracker, head-mounted display (HMD) device, media device, public utility, speaker, television, and / or tablet computer. Device 1750 includes application 1760 and operating system (not shown) (e.g., Figure 18 System 1810 is shown. Application 1760 includes application implementation instructions 1770 and API call instructions 1780. System 1810 includes API 1890 and implementation instructions 1800. It should be understood that device 1750, application 1760 and / or system 1810 may include... Figure 17 and Figure 18 The examples illustrate more, fewer, and / or different components.

[0132] In some implementations, application implementation instructions 1770 is a software module comprising a set of one or more computer-readable instructions. In some implementations, the set of one or more instructions in instructions 1770 corresponds to one or more operations performed by application 1760. For example, when application 1760 is a messaging application, application implementation instructions 1770 may include operations for receiving and transmitting messages. In some implementations, application implementation instructions 1770 communicate with API call instructions to access API 1890 (… Figure 18 (As shown) communicates with system 1810.

[0133] In some implementations, API call instruction 1780 is a software module that includes a set of one or more computer-executable instructions.

[0134] In some implementations, implementing instruction 1800 is a software module that includes a set of one or more computer-executable instructions.

[0135] In some implementations, API 1890 is a software module comprising one or more computer-executable instructions. In some implementations, API 1890 provides an interface that allows different sets of instructions (e.g., API call instruction 1780) to access and / or use one or more functions, methods, procedures, data structures, classes, and / or other services provided by implementation instructions 1800 of system 1810. For example, API call instruction 1780 can access features of implementation instruction 1800 through one or more API calls or references opened by API 1890 (e.g., embodied by function or method calls), and can pass data and / or control information using one or more parameters via API calls or references. In some implementations, API 1890 allows application 1760 to use services provided by a software development kit (SDK) library. In some implementations, application 1760 combines calls to functions or methods provided by the SDK library and API 1890, or uses data types or objects defined in the SDK library and provided by API 1890. In some implementations, API call instruction 1780 makes an API call via API 1890 to access and use the features of implementation instruction 1800 specified by API 1890. In such implementations, implementation instruction 1800 may return a value to API call instruction 1780 via API 1890 in response to the API call. This value may report to application 1760 the capabilities or status of hardware components of device 1750, including those capabilities or statuses related to aspects such as input capabilities and status, output capabilities and status, processing capabilities, power status, storage capacity and status, and / or communication capabilities. In some implementations, API 1890 is implemented in part by firmware, microcode, or other low-level logic executed in part on the hardware components.

[0136] In some implementations, API 1890 allows the developer of API call instruction 1780 (which may be a third-party developer) to utilize features provided by implementation instruction 1800. In such implementations, one or more sets of API call instructions (e.g., including API call instruction 1780) may exist to communicate with implementation instruction 1800. In some implementations, API 1890 allows multiple sets of API call instructions written in different programming languages ​​to communicate with implementation instruction 1800 (e.g., API 1890 may include features for translating calls and returns between implementation instruction 1800 and API call instruction 1780), and API 1890 is implemented in a specific programming language. In some implementations, API call instruction 1780 calls APIs from different providers, such as one set of APIs from an OS provider, another set of APIs from a plugin provider, and / or another set of APIs from another provider (e.g., a software library provider) or the creator of another set of APIs.

[0137] Examples of API 1890 may include one or more of the following: pairing APIs (e.g., for establishing a secure connection, such as with an accessory), device detection APIs (e.g., for locating nearby devices, such as media devices and / or smartphones), payment APIs, UIKit APIs (e.g., for generating user interfaces), location detection APIs, locator APIs, map APIs, health sensor APIs, sensor APIs, messaging APIs, push notification APIs, streaming APIs, collaboration APIs, video conferencing APIs, app store APIs, advertising service APIs, web browser APIs (e.g., WebKit APIs), transportation APIs, networking APIs, WiFi APIs, Bluetooth APIs, NFC APIs, UWB APIs, fitness APIs, smart home APIs, contact transfer APIs, photo APIs, camera APIs, and / or image processing APIs. In some implementations, a sensor API is an API for accessing data associated with sensors of device 1750. For example, a sensor API may provide access to raw sensor data. Alternatively, a sensor API may provide data derived (and / or generated) from raw sensor data. In some implementations, sensor data includes temperature data, image data, video data, audio data, heart rate data, IMU (Inertial Measurement Unit) data, LiDAR data, location data, GPS data, and / or camera data. In some implementations, sensors include one or more of accelerometers, temperature sensors, infrared sensors, optical sensors, heart rate sensors, barometers, gyroscopes, proximity sensors, and / or biometric sensors.

[0138] In some embodiments, implementation instruction 1800 is a system (e.g., an operating system, a server system) software module (e.g., a set of computer-readable instructions) configured to perform an operation in response to an API call received via API 1890. In some embodiments, implementation instruction 1800 is configured to provide an API response (via API 1890) as a result of processing the API call. For example, implementation instruction 1800 and API call instruction 1780 may each be any of an operating system, library, device driver, API, application, or other module. It should be understood that implementation instruction 1800 and API call instruction 1780 may be the same or different types of software modules. In some embodiments, implementation instruction 1800 is at least partially embodied in firmware, microcode, or other hardware logic.

[0139] In some implementations, implementation instruction 1800 returns a value via API 1890 in response to an API call from API call instruction 1780. While API 1890 defines the syntax and result of the API call (e.g., how to reference the API call and what the API call can do), API 1890 may not reveal how implementation instruction 1800 performs the function specified by the API call. Various API calls are transmitted via one or more application programming interfaces between API call instruction 1780 and implementation instruction 1800. Transmitting API calls may include issuing, initiating, referencing, calling, receiving, returning, and / or responding to function calls or messages. In other words, transmission may describe the action performed by either API call instruction 1780 or implementation instruction 1800. In some implementations, function calls or other references to API 1890 transmit and / or receive one or more parameters via parameter lists or other structures.

[0140] In some implementations, implementation instruction 1800 provides more than one API, each API providing a different view or aspect of the functionality implemented by implementation instruction 1800. For example, one API of implementation instruction 1800 may provide a first set of functions and be exposed to third-party developers, while another API of implementation instruction 1800 may be hidden (e.g., not exposed) and provide a subset of the first set of functions, and also provide another set of functions, such as test or debug functions not in the first set of functions. In some implementations, implementation instruction 1800 calls one or more other components via lower-level APIs, and is therefore both a set of API call instructions and a set of implementation instructions. It should be recognized that implementation instruction 1800 may include additional functions, methods, classes, data structures, and / or other features not specified through API 1890 and not available for API call instruction 1780. It should be recognized that API call instruction 1780 may be on the same system as implementation instruction 1800, or may be remotely located and accessed via a network using API 1890. In some implementations, the implementation instructions 1800, API 1890, and / or API call instructions 1780 are stored in a machine-readable medium, which includes any means for storing information in a machine-readable form (e.g., a computer or other data processing system). For example, a machine-readable medium may include a magnetic disk, optical disk, random access memory, read-only memory, and / or flash memory devices.

[0141] In some implementations, method 700, method 800, method 900 and / or method 1400 are executed at a first computer system (as described herein) via a system process (e.g., an operating system process, a server system process) that is different from one or more applications executed and / or installed on the first computer system.

[0142] In some embodiments, methods 700, 800, 900, and / or 1400 are executed at a first computer system (as described herein) by an application different from a system process. In some embodiments, the application's instructions, when executed, control the first computer system to execute methods 700, 800, 900, and / or 1400 by invoking an application programming interface (API) provided by the system process. In some embodiments, the application executes at least a portion of methods 700, 800, 900, and / or 1400 without invoking the API.

[0143] In some implementations, the application can be any suitable type of application, including one or more of the following: browser applications, applications used as execution environments for plugins, widgets or other applications, pairing applications, fitness applications, health applications, digital payment applications, media applications, social networking applications, camera applications, recording applications, chat applications, messaging applications and / or map applications.

[0144] In some embodiments, the application is an application pre-installed on the first computer system at the time of purchase (e.g., a first-party application). In some embodiments, the application is an application provided to the first computer system via operating system update files (e.g., a first-party application). In some embodiments, the application is an application provided via an app store. In some embodiments, the app store is pre-installed on the first computer system at the time of purchase (e.g., a first-party app store) and allows the download of one or more applications. In some embodiments, the app store is a third-party app store (e.g., an app store provided by another device, downloaded via a network, and / or read from a storage device). In some embodiments, the application is a third-party application (e.g., an application (app) provided by an app store, downloaded via a network, and / or read from a storage device). In some embodiments, the application controls the first computer system to execute method 700, method 800, method 900, and / or method 1400 by calling an application programming interface (API) provided by a system process using one or more parameters.

[0145] In some implementations, at least one API is a software module (e.g., a set of computer-readable instructions) that provides an interface that allows different sets of instructions (e.g., API call instructions) to access and use one or more functions, methods, procedures, data structures, classes, and / or other services provided by an implementation set of instructions from a system process. The API may define one or more parameters passed between the API call instructions and the implementation instructions.

[0146] As described above, in some implementations, the application controls the first computer system to execute method 700, method 800, method 900 and / or method 1400 by calling an application programming interface (API) provided by a system process using one or more parameters.

[0147] In some implementations, exemplary APIs provided by system processes include one or more of the following: pairing API (e.g., for establishing a secure connection, such as with an accessory), device detection API (e.g., for locating nearby devices, such as media devices and / or smartphones), payment API, UIKit API (e.g., for generating user interfaces), VisionKit API (e.g., for extracting text from images), location detection API, locator API, map API, health sensor API, sensor API, messaging API, push notification API, video conferencing API, app store API, advertising service API, web browser API (e.g., WebKit API), networking API, WiFi API, Bluetooth API, NFC API, UWB API, fitness API, smart home API, contact transfer API, photo API, camera API, microphone API, and / or image processing API.

[0148] In some implementations, API 1890 defines a first API call that can be provided by API call instruction 1780, wherein the definition of the first API call specifies call parameters (e.g., an image and / or text extracted from the image).

[0149] In some implementations, API 1890 defines a first API call response that can be provided to an application by API call instruction 1780, wherein the first API call response includes one or more rankings of one or more menu items.

[0150] In some embodiments, the implementation instruction set is a system software module (e.g., a set of computer-readable instructions) configured to perform operations in response to receiving an API call via an API. In some embodiments, the implementation instruction set is configured to provide an API response (via an API) as a result of processing an API call. In some embodiments, the implementation instruction set is included in a device (e.g., 1750) running the application. In some embodiments, the implementation instruction set is included in an electronic device separate from the device running the application.

[0151] As described above, one aspect of this technology is the collection and use of data obtainable from specific and legitimate sources to provide user information in association with the provision of secure audio indicators and / or protective audio pairings. This disclosure contemplates that, in some instances, the collected data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data may include demographic data, location-based data, online identifiers, telephone numbers, email addresses, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other personal information.

[0152] This disclosure recognizes that the use of such personal information data in the techniques of this invention can be used to benefit users. For example, personal information data can be used to provide secure audio indicators and / or protective audio pairings. Therefore, the use of such personal information data facilitates transaction processing (e.g., online transaction processing). Furthermore, this disclosure envisions other uses for personal information data that benefit users. For example, health and fitness data can be used according to user preferences to provide insights into their overall health status, or it can be used as positive feedback to individuals using the technology to pursue health goals.

[0153] This disclosure anticipates that entities responsible for collecting, analyzing, disclosing, transmitting, storing, or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, it is expected that such entities will implement and consistently apply privacy practices generally recognized as meeting or exceeding industry or governmental requirements for protecting user privacy. Such information regarding the use of personal data should be prominently displayed and readily accessible to users, and should be updated as data collection and / or use change. Users' personal information should be collected only for lawful use. Furthermore, such collection / sharing should only occur after receiving user consent or other lawful grounds provided for in applicable law. Additionally, such entities should consider taking any necessary steps to protect and safeguard the right to access such personal information data and to ensure that other entities with access to personal information data comply with the privacy policies and procedures of other entities. Moreover, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. Furthermore, policies and practices should be tailored to the specific types of personal information data collected and / or accessed, and made applicable to applicable laws and standards, including specific jurisdictional considerations that may be used to impose higher standards. For example, in the United States, the collection or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly.

[0154] Regardless of the foregoing, this disclosure also anticipates implementation schemes for users to selectively block the use or access to personal information data. That is, this disclosure anticipates providing hardware and / or software components to prevent or block access to such personal information data. For example, with the provision of a secure audio indicator and / or protective audio pairing, this technology can be configured to allow users to opt-in or opt-out at any time during or after service registration to participate in the collection of personal information data. In addition to providing opt-in and opt-out options, this disclosure also anticipates providing notifications related to access to or use of personal information. For example, users may be notified when downloading an application that their personal information data will be accessed, and then reminded again just before the application accesses the personal information data.

[0155] Furthermore, the intent of this disclosure is that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once data is no longer needed, this risk can be minimized by restricting data collection and deleting data. Additionally, and where applicable, including in certain health-related applications, data deidentification can be used to protect user privacy. Deidentification can be facilitated, where appropriate, by removing identifiers, controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data among users), and / or other methods (such as differentiated privacy).

[0156] Therefore, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, it also contemplates that various embodiments can be implemented without access to such personal information data. That is, various embodiments of the present invention will not become inoperable due to the absence of all or part of such personal information data.

[0157] Figure 21 An electronic system 2100 is illustrated that can be used to implement one or more specific embodiments of the subject matter. The electronic system 2100 may be... Figure 1 One or more of the illustrated electronic devices 100, and / or may be a part thereof. Electronic system 2100 may include various types of computer-readable media and interfaces for various other types of computer-readable media. Electronic system 2100 includes a bus 2108, one or more processing units 2112, system memory 2104 (and / or buffers), ROM 2110, persistent storage device 2102, input device interface 2114, output device interface 2106, and one or more network interfaces 2116, or subsets and variations thereof.

[0158] Bus 2108 generally represents all system buses, peripheral bus, and chipset bus that communicatively connect multiple internal devices of electronic system 2100. In one or more embodiments, bus 2108 communicatively connects one or more processing units 2112 to ROM 2110, system memory 2104, and persistent storage device 2102. One or more processing units 2112 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes disclosed in this subject matter. In different embodiments, one or more processing units 2112 may be a single processor or a multi-core processor.

[0159] ROM 2110 stores static data and instructions required by one or more processing units 2112 and other modules of electronic system 2100. On the other hand, persistent storage device 2102 can be a read-write memory device. Persistent storage device 2102 can be a non-volatile memory cell that stores instructions and data even when electronic system 2100 is off. In one or more embodiments, mass storage devices (such as disks or optical discs and their corresponding disk drives) can be used as persistent storage device 2102.

[0160] In one or more embodiments, a removable storage device (such as a floppy disk, flash drive, and its corresponding disk drive) may be used as persistent storage device 2102. Like persistent storage device 2102, system memory 2104 may be a read-write memory device. However, unlike persistent storage device 2102, system memory 2104 may be volatile read-write memory, such as random access memory. System memory 2104 may store any instructions and data that one or more processing units 2112 may need during operation. In one or more embodiments, the processes disclosed in this subject matter are stored in system memory 2104, persistent storage device 2102, and / or ROM 2110. One or more processing units 2112 retrieve instructions to be executed and data to be processed from these various memory units to execute the processes of one or more embodiments.

[0161] Bus 2108 is also connected to input device interface 2114 and output device interface 2106. Input device interface 2114 enables a user to communicate information to electronic system 2100 and select commands. Input devices that can be used with input device interface 2114 may include, for example, a microphone, an alphanumeric keypad, and a pointing device (also known as a "cursor control device"). Output device interface 2106 may, for example, enable the display of images generated by electronic system 2100. Output devices that can be used with output device interface 2106 may include, for example, printers and display devices such as liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic light-emitting diode (OLED) displays, flexible displays, flat panel displays, solid-state displays, projectors, speakers or speaker modules, or any other device for outputting information. One or more embodiments may include a device that functions as both an input device and an output device, such as a touchscreen. In these embodiments, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, verbal, or tactile input.

[0162] Finally, as Figure 21 As shown, bus 2108 also couples electronic system 2100 to one or more networks and / or one or more network nodes via one or more network interfaces 2116. In this way, electronic system 2100 may be part of a computer network (such as a LAN, wide area network (“WAN”), or intranet), or may be part of a network of various networks (such as the Internet). Any or all components of electronic system 2100 may be used in conjunction with the disclosures herein.

[0163] Computer-readable storage media can be any storage medium that can be read, written, or otherwise accessed by general-purpose or special-purpose computing devices, including any processing electronics and / or processing circuitry capable of executing instructions. For example, without limitation, computer-readable media can include any volatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. Computer-readable media can also include any non-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM, NVRAM, flash memory, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, track memory, FJG, and Millipede memory.

[0164] Additionally, computer-readable storage media may include any non-semiconductor memory, such as optical disc storage devices, magnetic disk storage devices, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions. In one or more embodiments, the tangible computer-readable storage medium may be directly coupled to a computing device, while in other embodiments, the tangible computer-readable storage medium may be indirectly coupled to a computing device, for example, via one or more wired connections, one or more wireless connections, or any combination thereof.

[0165] Instructions can be directly executable or can be used to develop executable instructions. For example, instructions can be implemented as executable or non-executable machine code, or as high-level language instructions that can be compiled to produce executable or non-executable machine code. Additionally, instructions can be implemented as data, or may include data. Computer executable instructions can also be organized in any format, including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, etc. As those skilled in the art will recognize, details including, but not limited to, the number, structure, sequence, and organization of instructions can vary significantly without altering the underlying logic, functionality, processing, and output.

[0166] While the above discussion primarily concerns microprocessors or multi-core processors that execute software, one or more specific implementations are executed by one or more integrated circuits such as ASICs or FPGAs. In one or more specific implementations, such integrated circuits execute instructions stored on the circuit itself.

[0167] The various functions described above can be implemented in digital electronic circuits, computer software, firmware, or hardware. This technology can be implemented using one or more computer program products. Programmable processors and computers can be included in or packaged as mobile devices. These processes and logical flows can be executed by one or more programmable processors and one or more programmable logic circuits. General-purpose and special-purpose computing devices, as well as storage devices, can be interconnected via communication networks.

[0168] Some specific implementations include electronic components, such as microprocessors, storage devices, and memories, that store computer program instructions in machine-readable or computer-readable media (or computer-readable storage media, machine-readable media, or machine-readable storage media). Examples of such computer-readable media include RAM, ROM, read-only optical discs (CD-ROM), recordable optical discs (CD-R), rewritable optical discs (CD-RW), read-only digital versatile optical discs (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and / or solid-state hard disk drives, high-density optical discs, any other optical or magnetic media, and floppy disks. Computer-readable media can store computer programs that can be executed by at least one processing unit and include a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as machine code generated by a compiler, and files that include higher-level code that can be executed by a computer, electronic component, or microprocessor using an interpreter.

[0169] While the above discussion primarily concerns microprocessors or multi-core processors that execute software, some implementations are performed by one or more integrated circuits such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some implementations, such integrated circuits execute instructions stored on the circuit itself.

[0170] As used in this specification and any claim of this patent application, the terms "computer," "processor," and "memory" refer to electronic or other technical devices. These terms exclude persons or groups of persons. For the purposes of this specification, the terms "display" or "being displayed" mean displaying on an electronic device. As used in this specification and any claim of this application, the terms "computer-readable medium" and "computer-readable medium" are entirely limited to tangible, touchable objects that store information in a form readable by a computer. These terms do not include any wireless signals, wired download signals, or any other transient signals.

[0171] Many of the features and applications described above can be implemented as software processes that specify a set of instructions to be recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more processing units (e.g., one or more processors, processor cores, or other processing units), the instructions cause the one or more processing units to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard disk drives, EPROMs, etc. Computer-readable media do not include carrier waves and electrical signals transmitted wirelessly or via wired connections.

[0172] In this specification, the term "software" is intended to include firmware residing in read-only memory or applications stored in magnetic storage devices, which can be read into memory for processing by a processor. Similarly, in some embodiments, multiple software aspects disclosed herein may be implemented as sub-parts of a larger program while retaining the different software aspects disclosed herein. In some embodiments, multiple software aspects may also be implemented as independent programs. Finally, any combination of independent programs that collectively implement the software aspects described herein is within the scope of this disclosure. In some embodiments, when installed to run on one or more electronic systems, a software program defines one or more specific machine implementations that execute and perform the operations of the software program.

[0173] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, objects, or other units suitable for use in a computing environment. Computer programs may, but do not necessarily, correspond to files in a file system. A program may be stored as a part of a file containing other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in its description, or in multiple coordinating files (e.g., a file storing one or more modules, subroutines, or code portions). Computer programs can be deployed to execute on a single computer or on multiple computers located at the same site or distributed across multiple sites and interconnected via a communication network.

[0174] It should be understood that any particular order or hierarchy of the boxes in the process disclosed in this invention is an example of the exemplary method. Based on design preferences, it should be understood that a particular order or hierarchy of the boxes in the process may be rearranged or all illustrated boxes may be executed. Some boxes within these boxes may be executed simultaneously. For example, in some cases, multitasking and parallel processing may be advantageous. Furthermore, the division of various system components in the above embodiments should not be construed as requiring such division in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0175] The preceding description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, this claim is not intended to be limited to the aspects shown herein, but rather to be consistent with the language of the claim, wherein references to elements in singular values ​​are not intended to mean “one and only one,” but rather “one or more,” unless specifically indicated. Unless otherwise specifically stated, the term “some” means one or more. Male pronouns (e.g., his) include female and neutral (e.g., her and its), and vice versa. Titles and subtitles (if any) are used for convenience only and do not limit the disclosure of this subject matter.

[0176] The predicates “configured to,” “operable to,” and “programmed to” do not imply any specific tangible or intangible modification to a particular subject but are intended to be used interchangeably. For example, a component or a processor configured to monitor and control operations may also mean that the processor is programmed to monitor and control operations or that the processor is operable to monitor and control operations. Similarly, a processor configured to execute code can be interpreted as a processor programmed to execute code or operable to execute code.

[0177] The phrase "aspect" does not imply that this aspect is essential to the present subject matter or that this aspect applies to all configurations of the present subject matter. Disclosures relating to an aspect may apply to all configurations, or one or more configurations. The phrase "aspect" may refer to one or more aspects, and vice versa. The phrase "configuration" does not imply that this configuration is essential to the present subject matter or that this configuration applies to all configurations of the present subject matter. Disclosures relating to a configuration may apply to all configurations, or one or more configurations. The phrase "configuration" may refer to one or more configurations, and vice versa.

[0178] The word “example” is used in this document to mean “used as an example or illustration.” Any aspect or design described in this document as an “example” is not necessarily to be construed as superior or advantageous to any other aspect or design.

[0179] On one hand, the term "linkage" can refer to a direct connection. On the other hand, the term "linkage" can refer to an indirect connection.

[0180] Terms such as top, bottom, front, back, side, horizontal, and vertical refer to any frame of reference, not the usual gravitational frame of reference. Therefore, such terms can extend upward, downward, diagonally, or horizontally within a gravitational frame of reference.

[0181] All structural and functional equivalents of elements throughout the various aspects described herein that are known or later become apparent to those skilled in the art are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be made public, regardless of whether such disclosure is expressly stated in the claims. No claim element should be interpreted in accordance with 35 USC §112(f) unless the element is expressly stated using the phrase “means for…” or, in the case of a method claim, using the phrase “steps for…”. Additionally, terms such as “comprising,” “having,” etc., are used to a certain extent in the specification or claims, and such terms are intended to be included in a manner similar to how the term “comprising” is interpreted when used as a transitional word in a claim.

Claims

1. A method, the method comprising: A system process running on the processor of an electronic device provides commands to a security audio processor at the electronic device to generate an audio notification that one or more input components of the electronic device are in use; In response to the command, the secure audio processor generates an audio signal; as well as The audio signal is provided from the secure audio processor to the speaker of the electronic device for output by the speaker.

2. The method of claim 1, wherein the processor is prevented from directly providing audio signals to the speaker.

3. The method of claim 1, wherein providing the command comprises: The command is provided in response to a system process determining that one or more input components of the electronic device are being used by an insecure process at the electronic device, wherein the insecure process stores or exports information obtained by the one or more input components.

4. The method of claim 1, wherein the one or more input components include at least one of a camera or a microphone.

5. The method of claim 1, wherein generating the audio signal using the secure audio processor includes generating low-frequency audio content for output by the speaker.

6. The method of claim 5, further comprising the secure audio processor: Receive another audio signal from the processor, which includes additional low-frequency audio content; Remove the additional low-frequency audio content from the other audio signal; and The audio signal having the low-frequency audio content and another audio signal having the additional low-frequency audio content removed are provided to the speaker for simultaneous output by the speaker.

7. The method of claim 1, wherein generating the audio signal using the secure audio processor comprises: The audio signal is spatialized so that it is perceived at a specific spatial location relative to the user of the electronic device when output by the speaker and another speaker.

8. The method of claim 7, further comprising the secure audio processor: Receive another audio signal from the processor; The other audio signal is spatialized so that, when output by the speaker and the other speaker, it is perceived at one or more locations other than the specific spatial location of the audio signal; and The spatialized audio signal and another spatialized audio signal are provided to the speaker so that the speaker can output them simultaneously.

9. The method according to claim 1, further comprising: During the output time of the audio signal from the speaker, a microphone signal is obtained using the microphone of the electronic device; as well as The processor uses the microphone signal to confirm that the audio signal is output by the speaker during the output time.

10. An electronic device, the electronic device comprising: A first circuit, configured to run a process, generates a first audio signal for output from a speaker that is securely isolated from the first circuit; and The second circuit is configured as follows: Receive the first audio signal from the first circuit; Process the first audio signal from the first circuit; The processed first audio signal is provided to the speaker for output; as well as A second audio signal is generated in response to a change in the use of one or more input components of the electronic device detected by the first circuit, for output by the speaker.

11. The electronic device of claim 10, wherein the second circuit is configured to generate the second audio signal in response to a command from the first circuit.

12. The electronic device of claim 10, wherein the variation in use of said one or more input components includes: Initiating an insecure process at the electronic device to use the one or more input components, or terminating the insecure process at the electronic device to use the one or more input components.

13. The electronic device of claim 10, wherein the second circuit is configured to process the first audio signal by removing a first low-frequency content from the first audio signal, and wherein the second audio signal includes second low-frequency audio content generated by the second circuit.

14. The electronic device of claim 13, wherein the second low-frequency audio content includes infrasound content that is inaudible to the human ear.

15. The electronic device of claim 10, wherein the second circuit is configured to: The first audio signal is processed by spatializing it so that it is perceived at a first location, and The second audio signal is spatialized so that it is perceived at a second location different from the first location.

16. The electronic device of claim 10, further comprising a security microphone circuit configured to acknowledge an output by the speaker corresponding to the second audio signal.

17. The electronic device of claim 10, wherein the second circuit and the speaker are securely isolated from the process.

18. An electronic device, the electronic device comprising: One or more input components; speaker; microphone; One or more processors, the one or more processors being configured to generate audio notifications indicating that the one or more input components are in use; and Safe audio input processor; and A secure direct hardware communication path between the microphone and the secure audio input processor, wherein the secure audio input processor is configured to: During the expected output time of the audio notification, a microphone signal is received from the microphone via the secure direct hardware communication path; and Based on the microphone signal, determine whether the audio notification is output by the speaker during the expected output time.

19. The electronic device of claim 18, wherein the one or more processors are configured to terminate operation of the one or more input components in response to the security audio input processor determining that the audio notification was not output by the speaker during the expected output time.

20. The electronic device of claim 18, wherein the one or more processors comprise: A first processor, securely isolated from the speaker and configured to run an application that generates audio signals for output by the speaker; and A second processor, configured to: The audio signal is received from the first processor, and at least one version of the audio signal is provided to the speaker for output; and Generate another audio signal corresponding to the audio notification for output by the speaker during the expected output time.

21. The electronic device of claim 18, wherein the one or more input components include at least one additional microphone having a communication path to the one or more processors.

22. The electronic device of claim 18, wherein the secure audio input processor is further configured to: Receive another microphone signal from the microphone while one or more of the input components are in use; Based on the other microphone signal, it is determined that the speaker has output another notification indicating that the one or more input components are no longer in use; and Provide the one or more processors with a signal indicating that the speaker has output a deceptive audio notification.

23. The electronic device of claim 18, wherein the one or more input components include the microphone.

Images