Privacy security-based audio processing method and system, device, and storage medium

Abstract

The application provides a privacy security-based audio processing method and system, equipment and a storage medium. The method is applied to smart glasses, the smart glasses comprising a microphone array, a behavior recognition sensor module and a variable speaker; the method comprises the following steps: determining an external sound source coordinate based on a sound signal collected by the microphone array; determining a user's attention degree and a privacy protection requirement level based on user behavior information collected by the behavior recognition sensor module; determining an audio signal volume adjustment amount and a beam width based on the external sound source coordinate, the attention degree and the privacy protection level, so as to process an initial audio signal and play the processed audio signal through the variable speaker. The application can comprehensively process audio from multiple aspects such as environmental risk, user state, content sensitivity and user privacy protection preference, realize personalized dynamic privacy protection, and balance the user's privacy protection and audio experience.

Description

Technical Field

[0001] This invention relates to the field of audio processing technology, and in particular to a privacy-preserving audio processing method, system, device, and storage medium. Background Technology

[0002] With the rapid development of smart wearable devices, audio devices such as headphones and smart glasses have become an important part of daily life.

[0003] However, traditional audio devices have significant shortcomings in terms of privacy protection, environmental adaptability, and human-computer interaction. Existing noise reduction technologies mainly target ambient noise across the entire frequency range, but lack the ability to directionally suppress interfering sound sources from specific directions. When using headphones or smart glasses, users often face problems such as audio data leakage, environmental noise interference, and insufficient personalized experience, and it is difficult to achieve both privacy protection and a good audio experience. Summary of the Invention

[0004] This invention provides a privacy-preserving audio processing method, system, device, and storage medium to address the challenge of balancing privacy protection and audio experience when using headphones or smart glasses.

[0005] In a first aspect, embodiments of the present invention provide a privacy-preserving audio processing method applied to smart glasses, the smart glasses including a microphone array, a behavior recognition sensor module, and a variable speaker; the method includes: The coordinates of the external sound source are determined based on the sound signals collected by the microphone array. Based on user behavior information collected by the behavior recognition sensor module, the user's level of attention and privacy protection needs are determined. Based on the coordinates of the external audio source, the level of attention, and the level of privacy protection, the volume adjustment and beamwidth of the audio signal are determined to process the initial audio signal and play the processed audio signal through a variable speaker.

[0006] In one possible implementation, based on user behavior information collected by the behavior recognition sensor module, the user's attention level and privacy protection requirement level are determined, including: The user's level of attention to the external sound source is determined based on the cumulative duration of the user's gaze in the target direction, the angle between the user's gaze direction and the direction of the external sound source, and the change in pupil diameter. Based on facial recognition confidence level, blink frequency, audio signal content classification, and distance to external sound sources, the user's privacy protection needs level is determined.

[0007] In one possible implementation, the formula for calculating the level of attention is:

[0008] in, This measures the user's level of attention towards the target direction within a monitoring period. The cumulative duration of time the user gazes in the direction of the target. For the duration of the monitoring period, The angle between the user's line of sight and the direction of the external sound source. This represents the change in pupil diameter. This is the baseline value for pupil diameter. , , As weight.

[0009] In one possible implementation, before determining the user's attention level and privacy protection needs based on user behavior information collected by the behavior recognition sensor module, the following is also included: Acquire multiple sets of historical usage data from the user; each set of usage data includes user behavior information and the corresponding user input volume adjustment amount and audio output direction; Based on multiple sets of historical usage data, the weights in the formula for calculating attention level are optimized.

[0010] In one possible implementation, the user's privacy protection requirement level is determined based on facial recognition confidence, blink frequency, audio signal content classification, and distance to the external sound source, including: Calculate the environmental risk coefficient based on the confidence level of facial recognition; Calculate user state coefficients based on blink frequency; Calculate sound source sensitivity based on audio signal content classification and distance to external sound sources; By inputting environmental risk coefficient, user status coefficient, and audio source sensitivity into the privacy level decision tree, the user's privacy protection requirement level is obtained.

[0011] In one possible implementation, the volume adjustment and beamwidth of the audio signal are determined based on the coordinates of the external sound source, the level of attention, and the privacy protection level, including: The volume adjustment amount of the audio signal is calculated based on the volume adjustment amount formula; where the volume adjustment amount formula is:

[0012] in, This is the volume adjustment amount. This is the initial volume. Distance to external sound source The angle between the user's line of sight and the direction of the external sound source. The degree of attention a user pays to the direction of the target. , , This is the adjustment coefficient; The directional difference level is determined based on the difference between the horizontal angle of the sound source and the horizontal angle of the user's line of sight, and the beamwidth corresponding to the directional difference level is used as the beamwidth of the audio signal.

[0013] In one possible implementation, determining the coordinates of an external sound source based on sound signals acquired by a microphone array includes: The coordinates of the external sound source are determined by solving the sound signal collected by the microphone array based on the time delay estimation algorithm and the least squares method.

[0014] Secondly, embodiments of the present invention provide a privacy-preserving audio processing system for use in smart glasses, the smart glasses including a microphone array, a behavior recognition sensor module, and a variable speaker; the system includes: The sound source determination module is used to determine the coordinates of external sound sources based on the sound signals collected by the microphone array. The requirement determination module is used to determine the user's attention level and privacy protection requirement level based on the user behavior information collected by the behavior recognition sensor module. The audio processing module is used to determine the volume adjustment and beamwidth of the audio signal based on the coordinates of the external sound source, the level of attention, and the level of privacy protection, so as to process the initial audio signal and play the processed audio signal through a variable speaker.

[0015] Thirdly, embodiments of the present invention provide an apparatus including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in the first aspect or any possible implementation of the first aspect above.

[0016] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in the first aspect or any possible implementation thereof.

[0017] This invention provides a privacy-preserving audio processing method, system, device, and storage medium. It utilizes a microphone array and behavior recognition sensors in smart glasses to perform multimodal perception of the environment and user state, extracting user attention and privacy needs. It also precisely calculates the volume adjustment and beamwidth of the audio signal, comprehensively considering environmental risks, user state and content sensitivity, and user privacy preferences to achieve personalized dynamic privacy protection, balancing user privacy with audio experience. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the implementation of the privacy-preserving audio processing method provided in this embodiment of the invention. Figure 2 This is a schematic diagram of the privacy-preserving audio processing system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the device provided in an embodiment of the present invention. Detailed Implementation

[0020] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0021] To make the objectives, technical solutions, and advantages of the present invention clearer, specific embodiments will be described below in conjunction with the accompanying drawings.

[0022] See Figure 1 The document illustrates a flowchart of the implementation of a privacy-preserving audio processing method provided in an embodiment of the present invention, applied to smart glasses. The smart glasses include a microphone array, a behavior recognition sensor module, and a variable speaker. The method is described in detail below: Step 101: Determine the coordinates of the external sound source based on the sound signals collected by the microphone array.

[0023] In this embodiment, traditional devices struggle to accurately track the location of the sound source, causing the sound field to easily deviate from the target area and posing a high risk of privacy leakage. This embodiment employs the TDOA algorithm and a microphone array to calculate the three-dimensional coordinates of the sound source in real time. When the user moves or the sound source location changes (such as when a speaker switches during a meeting), the system updates the sound source coordinates 10-20 times per second, driving the beam direction to adjust in real time and preventing sound energy leakage.

[0024] Step 102: Based on the user behavior information collected by the behavior recognition sensor module, determine the user's attention level and privacy protection requirement level.

[0025] In this embodiment, the behavior recognition sensor module may include a miniature camera and an infrared sensor to track the user's eye movements and gaze direction in real time, and collect the user's head posture, blink frequency, and pupil diameter. It can also determine whether there are strangers nearby through facial recognition.

[0026] Traditional volume control relies solely on ambient noise intensity, ignoring user intent and resulting in either excessive noise reduction or insufficient protection. To address this, this embodiment utilizes a behavior recognition sensor module to track the user's eye movements (horizontal angle θ, vertical angle φ) and head posture (yaw angle, pitch angle) to quantify the intensity of the user's attention to a specific sound source. Furthermore, it combines physiological state (blink frequency, pupil diameter) with environmental risk (probability of strangers approaching) to categorize privacy needs into low, medium, and high levels.

[0027] A user's attention level directly reflects their auditory priority for a specific sound source, while the privacy level quantifies the combined threat of privacy breaches arising from environmental risks and the user's state. Based on the user's attention level and privacy protection needs, effective information that the user is interested in can be precisely highlighted in complex scenarios, while effectively protecting the privacy information that the user does not want to disclose.

[0028] Step 103: Based on the coordinates of the external sound source, the level of attention, and the level of privacy protection, determine the volume adjustment and beamwidth of the audio signal to process the initial audio signal and play the processed audio signal through a variable speaker.

[0029] In this embodiment, volume adjustment can be specifically considered based on distance attenuation, direction matching, and attention enhancement. The closer the sound source, the higher the volume, but it is constrained by sound masking to prevent close-range leakage. The smaller the angle between the sound source and the line of sight, the higher the volume priority. Furthermore, when the user continues to pay attention, the target sound source is amplified further.

[0030] Beamwidth control can be achieved by using a narrow beam to concentrate sound energy and reduce lateral leakage under high privacy requirements, and by expanding coverage and maintaining basic audibility under low privacy requirements.

[0031] For example, when a user is looking in a certain direction (such as a speaker at a conference), the system enhances the audio in that direction by using a narrow beam and volume boost, while suppressing interference from other directions; when the user's attention is diverted (such as when multitasking), the system automatically expands the beam to ±30° to balance the priority of multiple audio sources and prevent excessive volume from a single direction from causing auditory fatigue; when the audio content is identified as privacy-sensitive (such as keywords "password" or "transfer"), the system automatically switches to narrow beam mode, and even if the environmental risk level is low, it still forces a -15dB sound pressure level attenuation.

[0032] Furthermore, smart glasses can incorporate adjustable sound-blocking materials. At a high level of privacy, electrostriction mechanisms can be used to increase material density and improve sound resistance. For example, polyurethane can be used; applying voltage causes the polyurethane matrix to exhibit an electrostrictive effect, compressing the bubble volume and increasing material density. This increases the complexity of the sound wave penetration path and enhances blocking efficiency.

[0033] This invention utilizes the microphone array and behavior recognition sensors of smart glasses to perform multimodal perception of the environment and user status, extracting user attention and privacy needs, and accurately calculating the volume adjustment and beamwidth of the audio signal. It comprehensively processes audio based on environmental risks, user status and content sensitivity, and user privacy protection preferences to achieve personalized dynamic privacy protection, balancing user privacy protection and audio experience.

[0034] In one possible implementation, based on user behavior information collected by the behavior recognition sensor module, the user's attention level and privacy protection requirement level are determined, including: The user's level of attention to the external sound source is determined based on the cumulative duration of the user's gaze in the target direction, the angle between the user's gaze direction and the direction of the external sound source, and the change in pupil diameter. Based on facial recognition confidence level, blink frequency, audio signal content classification, and distance to external sound sources, the user's privacy protection needs level is determined.

[0035] In this embodiment, the user's level of attention is determined by a combination of physiological behavioral characteristics and spatial relationships. The cumulative duration reflects the intensity of the user's sustained attention in a specific direction. The longer the duration (e.g., accounting for 80% of the total duration), the higher the priority of the sound source, triggering a higher volume boost. For example, continuously staring at the speaker in a meeting for more than 5 seconds indicates that the user is highly focused on the sound source. Brief glances (<200ms) may be unconscious actions, and the cumulative duration can eliminate momentary interference.

[0036] The angle between the user's line of sight and the direction of the external sound source directly reflects the spatial consistency between the user's visual focus and the sound source. The smaller the angle, the more focused the user is on the target direction. If the sound source is directly in front of the user's line of sight (θ=0°), the sound energy focusing needs to be maximized; if θ=45°, the beam needs to be expanded or the gain reduced.

[0037] Pupil dilation and constriction are physiological indicators of cognitive load and emotional state. Pupil dilation may indicate that the user is interested in the audio content (such as hearing key information), and audio clarity needs to be improved. Prolonged pupil constriction may indicate fatigue, and the volume should be automatically reduced to avoid auditory stress.

[0038] The system captures facial features through a camera to determine whether people in the surrounding area are authorized users. When a stranger is detected and the confidence level is ≥80%, it is identified as a high-risk scenario and the highest privacy level is triggered.

[0039] Blinking frequency is a quantitative indicator of fatigue and inattention. A normal frequency is 12-20 times per minute, while abnormal values ​​may indicate a decrease in the user's awareness of privacy. If the blinking frequency is >25 times per minute (lasting for 1 minute), the user is considered to be fatigued, and the privacy level is automatically increased. High-frequency blinking (such as 30 times per minute when nervous) may be accompanied by inattention, requiring enhanced suppression of lateral sound leakage.

[0040] By using Automatic Speech Recognition (ASR) and Natural Language Processing (NLP), it is possible to determine in real time whether audio content contains sensitive information. Even if the environmental risk is low, if sensitive words are detected, the privacy level will still be forcibly raised to "high".

[0041] The distance to the sound source directly affects the risk of sound leakage. The closer the distance, the easier it is for an eavesdropper to capture clear audio.

[0042] These parameters allow for the elimination of interference, enhancing only the audio sources that users truly care about, and triggering privacy protection in a tiered manner based on three dimensions: content, environment, and user status. In one possible implementation, the formula for calculating the level of attention is:

[0043] in, This measures the user's level of attention towards the target direction within a monitoring period. The cumulative duration of time the user gazes in the direction of the target. For the duration of the monitoring period, The angle between the user's line of sight and the direction of the external sound source. This represents the change in pupil diameter. This is the baseline value for pupil diameter. , , As weight.

[0044] In this embodiment, the user's attention weight to a specific sound source is comprehensively quantified by considering the proportion of fixation time, consistency of gaze direction, and pupil physiological response.

[0045] The fixation duration percentage reflects the proportion of sustained attention a user pays to the target direction. Brief glances (such as unintentional glances) have little impact on attention weight. Noise needs to be eliminated by accumulating the duration. The longer the fixation time, the higher the user's interest or need for the target, and the higher the level of attention.

[0046] The more aligned the user's gaze is with the direction of the sound source, the more likely they are to actively focus on that source, thus increasing its priority. The difference between the current pupil diameter and the baseline value reflects the relative magnitude of pupil dilation or constriction. Pupil dilation is usually associated with focus, interest, or cognitive effort, while persistent pupil constriction may indicate fatigue or distraction.

[0047] By quantifying gaze duration, gaze direction consistency, and pupil physiological response in three dimensions, it is possible to eliminate noise interference, capture the user's true intentions, adjust audio output in real time, and balance privacy and experience.

[0048] In one possible implementation, before determining the user's attention level and privacy protection needs based on user behavior information collected by the behavior recognition sensor module, the following is also included: Acquire multiple sets of historical usage data from the user; each set of usage data includes user behavior information and the corresponding user input volume adjustment amount and audio output direction; Based on multiple sets of historical usage data, the weights in the formula for calculating attention level are optimized.

[0049] In this embodiment, by analyzing users' historical behavior and manual operation records, the implicit correlation between users' actual needs and volume adjustment and audio output direction can be discovered, and users' privacy protection preferences can be learned, thereby optimizing the weight allocation of the attention calculation model and improving the system's adaptability.

[0050] Specifically, a regression model can be used to fit historical data, and the weights of each item in the formula can be adjusted (…). , , This allows the model output to approximate the user's true preferences. In one possible implementation, the user's privacy protection requirement level is determined based on facial recognition confidence, blink frequency, audio signal content classification, and distance to the external sound source, including: Calculate the environmental risk coefficient based on the confidence level of facial recognition; Calculate user state coefficients based on blink frequency; Calculate sound source sensitivity based on audio signal content classification and distance to external sound sources; By inputting environmental risk coefficient, user status coefficient, and audio source sensitivity into the privacy level decision tree, the user's privacy protection requirement level is obtained.

[0051] In this embodiment, the face recognition confidence level refers to the degree of matching between the facial features captured by the system through the camera and the pre-stored database, expressed as a percentage (0%~100%), reflecting the probability of detecting a stranger. For example, a confidence level ≥80% indicates a high-risk stranger. The formula for calculating the environmental risk coefficient can be:

[0052] in This is the noise interference factor, taken as 1.5 in noisy environments and 1.0 in quiet environments.

[0053] Frequent blinking (>25 times / minute) indicates fatigue or distraction, requiring an increased level of protection. The formula for calculating the user's state coefficient is:

[0054] The formula for calculating sound source sensitivity is:

[0055] Where d is the distance to the sound source, the closer the distance is to d, the higher the sensitivity weight.

[0056] The expression for the privacy level decision tree is:

[0057] All of the above parameter values ​​can be adjusted according to actual needs.

[0058] In one possible implementation, the volume adjustment and beamwidth of the audio signal are determined based on the coordinates of the external sound source, the level of attention, and the privacy protection level, including: The volume adjustment amount of the audio signal is calculated based on the volume adjustment amount formula; where the volume adjustment amount formula is:

[0059] in, This is the volume adjustment amount. This is the initial volume. Distance to external sound source The angle between the user's line of sight and the direction of the external sound source. The degree of attention a user pays to the direction of the target. , , This is the adjustment coefficient; The directional difference level is determined based on the difference between the horizontal angle of the sound source and the horizontal angle of the user's line of sight, and the beamwidth corresponding to the directional difference level is used as the beamwidth of the audio signal.

[0060] In this embodiment, the decision logic for volume adjustment includes: distance priority: the closer the sound source, the higher the volume, but due to the limitation of sound-blocking materials, excessive leakage is prevented; direction matching: the smaller the angle between the sound source and the line of sight (i.e., the direction the user is focusing on), the greater the volume increase; attention enhancement: when the user continues to look at a sound source, the volume priority of that direction is further increased.

[0061] The variable radiation mode loudspeaker dynamically adjusts beamforming parameters using phase array technology. The beamwidth adaptively adjusts based on the sound source distance and the user's line-of-sight stability, especially at close range (d < 1m). d <1m: Narrow beam (±5°), concentrates acoustic energy to reduce leakage, suitable for long distances (d>3m). d>3m: Wide beam (±30°), covering a larger area but with reduced gain. For beam pointing, if the direction of the sound source is consistent with the user's line of sight, it is directly aligned with the sound source coordinates; if there is competition from multiple sound sources, it is preferentially pointed to the direction where the user's gaze time is the longest, and other directions are suppressed by a filter.

[0062] During use, reinforcement learning models can be used to record users' manual volume adjustment behaviors (such as actively increasing the volume in a certain direction), gradually optimizing the adjustment coefficients and realizing personalized strategies.

[0063] In one possible implementation, determining the coordinates of an external sound source based on sound signals acquired by a microphone array includes: The coordinates of the external sound source are determined by solving the sound signal collected by the microphone array based on the time delay estimation algorithm and the least squares method.

[0064] In this embodiment, the microphone array is arranged in a ring (6 microphones, radius 5cm) to ensure 360° coverage. The microphone spacing must meet the wavelength requirements; for example, the highest frequency of 8kHz corresponds to a wavelength of 4.3cm, and the spacing should be ≤2cm to avoid aliasing.

[0065] The time delay estimation algorithm calculates the sound source location based on the time difference of sound arrival at different microphones. By comparing the time delay differences of signals received by multiple pairs of microphones, the spatial coordinates of the sound source can be inferred. In sound source localization, the least squares method is used to establish a nonlinear hyperbolic equation system based on the time delay differences of multiple pairs of microphones. This system is then transformed into a linear least squares problem for iterative solution, achieving high-precision localization. This approach strikes a balance between accuracy, real-time performance, and noise resistance, providing a foundation for subsequent audio processing.

[0066] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0067] The following are system embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0068] Figure 2 A schematic diagram of the privacy-preserving audio processing system provided in an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown, and are described in detail below: like Figure 2 As shown, a privacy-preserving audio processing system 2 is applied to smart glasses, which include a microphone array, a behavior recognition sensor module, and a variable speaker; the privacy-preserving audio processing system 2 includes: The sound source determination module 21 is used to determine the coordinates of an external sound source based on the sound signals collected by the microphone array; The requirement determination module 22 is used to determine the user's attention level and privacy protection requirement level based on the user behavior information collected by the behavior recognition sensor module. The audio processing module 23 is used to determine the volume adjustment amount and beamwidth of the audio signal based on the coordinates of the external sound source, the level of attention and the level of privacy protection, so as to process the initial audio signal and play the processed audio signal through a variable speaker.

[0069] In one possible implementation, the requirements determination module 22 is specifically used for: The user's level of attention to the external sound source is determined based on the cumulative duration of the user's gaze in the target direction, the angle between the user's gaze direction and the direction of the external sound source, and the change in pupil diameter. Based on facial recognition confidence level, blink frequency, audio signal content classification, and distance to external sound sources, the user's privacy protection needs level is determined.

[0070] In one possible implementation, the formula for calculating the level of attention is:

[0071] in, This measures the user's level of attention towards the target direction within a monitoring period. The cumulative duration of time the user gazes in the direction of the target. For the duration of the monitoring period, The angle between the user's line of sight and the direction of the external sound source. This represents the change in pupil diameter. This is the baseline value for pupil diameter. , , As weight.

[0072] In one possible implementation, the requirements determination module 22 is also used for: Before determining the user's attention level and privacy protection needs based on user behavior information collected by the behavior recognition sensor module, multiple sets of historical usage data of the user are obtained; each set of usage data includes user behavior information and the corresponding user input volume adjustment amount and audio output direction; Based on multiple sets of historical usage data, the weights in the formula for calculating attention level are optimized.

[0073] In one possible implementation, the requirements determination module 22 is specifically used for: Calculate the environmental risk coefficient based on the confidence level of facial recognition; Calculate user state coefficients based on blink frequency; Calculate sound source sensitivity based on audio signal content classification and distance to external sound sources; By inputting environmental risk coefficient, user status coefficient, and audio source sensitivity into the privacy level decision tree, the user's privacy protection requirement level is obtained.

[0074] In one possible implementation, the audio processing module 23 is specifically used for: The volume adjustment amount of the audio signal is calculated based on the volume adjustment amount formula; where the volume adjustment amount formula is:

[0075] in, This is the volume adjustment amount. This is the initial volume. Distance to external sound source The angle between the user's line of sight and the direction of the external sound source. The degree of attention a user pays to the direction of the target. , , This is the adjustment coefficient; The directional difference level is determined based on the difference between the horizontal angle of the sound source and the horizontal angle of the user's line of sight, and the beamwidth corresponding to the directional difference level is used as the beamwidth of the audio signal.

[0076] In one possible implementation, the sound source determination module 21 is specifically used for: The coordinates of the external sound source are determined by solving the sound signal collected by the microphone array based on the time delay estimation algorithm and the least squares method.

[0077] This invention utilizes the microphone array and behavior recognition sensors of smart glasses to perform multimodal perception of the environment and user status, extracting user attention and privacy needs, and accurately calculating the volume adjustment and beamwidth of the audio signal. It comprehensively processes audio based on environmental risks, user status and content sensitivity, and user privacy protection preferences to achieve personalized dynamic privacy protection, balancing user privacy protection and audio experience.

[0078] Figure 3 This is a schematic diagram of the device provided in an embodiment of the present invention. Figure 3 As shown, the device 3 in this embodiment includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30. When the processor 30 executes the computer program 32, it implements the steps in the various privacy-based audio processing method embodiments described above, for example... Figure 1Steps 101 to 103 are shown. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each module / unit in the above system embodiments, for example... Figure 2 The functions of modules / units 21 to 23 shown.

[0079] For example, the computer program 32 can be divided into one or more modules / units, which are stored in the memory 31 and executed by the processor 30 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 32 in the device 3. For example, the computer program 32 can be divided into... Figure 2 Modules / units 21 to 23 are shown.

[0080] The device 3 may be a desktop computer, laptop, handheld computer, or cloud server, etc. The device 3 may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of device 3 and does not constitute a limitation on device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, the device may also include input / output devices, network access devices, buses, etc.

[0081] The processor 30 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0082] The memory 31 can be an internal storage unit of the device 3, such as a hard disk or memory of the device 3. The memory 31 can also be an external storage device of the device 3, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the device 3. Furthermore, the memory 31 can include both internal storage units and external storage devices of the device 3. The memory 31 is used to store the computer program and other programs and data required by the device. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0083] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0084] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0085] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0086] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / device and method can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0087] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0088] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0089] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various privacy-preserving audio processing method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately added to or subtracted from the content as required by the legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium may not include electrical carrier signals and telecommunication signals.

[0090] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A privacy-preserving audio processing method, characterized in that, Applied to smart glasses, the smart glasses including a microphone array, a behavior recognition sensor module, and a variable speaker; the method includes: The coordinates of the external sound source are determined based on the sound signals collected by the microphone array. Based on user behavior information collected by the behavior recognition sensor module, the user's level of attention and privacy protection needs are determined. Based on the external sound source coordinates, the level of attention, and the level of privacy protection, the volume adjustment and beamwidth of the audio signal are determined to process the initial audio signal and play the processed audio signal through the variable speaker. The user behavior information collected by the behavior recognition sensor module is used to determine the user's attention level and privacy protection needs level, including: The user's level of attention to the external sound source is determined based on the cumulative duration of the user's gaze in the target direction, the angle between the user's gaze direction and the direction of the external sound source, and the change in pupil diameter. Based on facial recognition confidence level, blink frequency, audio signal content classification, and distance to external sound sources, the user's privacy protection requirement level is determined; The formula for calculating attention level is: in, This measures the user's level of attention towards the target direction within a monitoring period. The cumulative duration of time the user gazes in the direction of the target. For the duration of the monitoring period, The angle between the user's line of sight and the direction of the external sound source. This represents the change in pupil diameter. This is the baseline value for pupil diameter. , , As weight; The step of determining the volume adjustment and beamwidth of the audio signal based on the external sound source coordinates, the attention level, and the privacy protection level includes: The volume adjustment amount of the audio signal is calculated based on the volume adjustment amount formula; wherein, the volume adjustment amount formula is: in, This is the volume adjustment amount. This is the initial volume. Distance to external sound source The angle between the user's line of sight and the direction of the external sound source. The degree of attention a user pays to the direction of the target. , , This is the adjustment coefficient; The directional difference level is determined based on the difference between the horizontal angle of the sound source and the horizontal angle of the user's line of sight, and the beamwidth corresponding to the directional difference level is used as the beamwidth of the audio signal.

2. The privacy-preserving audio processing method according to claim 1, characterized in that, Before determining the user's attention level and privacy protection needs based on the user behavior information collected by the behavior recognition sensor module, the following steps are also included: Acquire multiple sets of historical usage data from the user; each set of usage data includes user behavior information and the corresponding user input volume adjustment amount and audio output direction; Based on the aforementioned multiple sets of historical usage data, the weights in the formula for calculating attention level are optimized.

3. The privacy-preserving audio processing method according to claim 1, characterized in that, The method for determining a user's privacy protection needs level based on facial recognition confidence, blink frequency, audio signal content classification, and distance to external sound sources includes: Calculate the environmental risk coefficient based on the confidence level of facial recognition; Calculate user state coefficients based on blink frequency; Calculate sound source sensitivity based on audio signal content classification and distance to external sound sources; The environmental risk coefficient, the user status coefficient, and the sound source sensitivity are input into the privacy level decision tree to obtain the user's privacy protection requirement level.

4. The privacy-preserving audio processing method according to claim 1, characterized in that, The determination of external sound source coordinates based on sound signals acquired by the microphone array includes: The coordinates of the external sound source are determined by solving the sound signal collected by the microphone array based on the time delay estimation algorithm and the least squares method.

5. A privacy-preserving audio processing system, characterized in that, The system is applied to smart glasses, which include a microphone array, a behavior recognition sensor module, and a variable speaker; the system includes: The sound source determination module is used to determine the coordinates of external sound sources based on the sound signals collected by the microphone array. The requirement determination module is used to determine the user's attention level and privacy protection requirement level based on the user behavior information collected by the behavior recognition sensor module. An audio processing module is used to determine the volume adjustment amount and beamwidth of an audio signal based on the coordinates of the external sound source, the level of attention, and the level of privacy protection, so as to process the initial audio signal and play the processed audio signal through the variable speaker. The requirements definition module is specifically used for: The user's level of attention to the external sound source is determined based on the cumulative duration of the user's gaze in the target direction, the angle between the user's gaze direction and the direction of the external sound source, and the change in pupil diameter. Based on facial recognition confidence level, blink frequency, audio signal content classification, and distance to external sound sources, the user's privacy protection requirement level is determined; The formula for calculating attention level is: in, This measures the user's level of attention towards the target direction within a monitoring period. The cumulative duration of time the user gazes in the direction of the target. For the duration of the monitoring period, The angle between the user's line of sight and the direction of the external sound source. This represents the change in pupil diameter. This is the baseline value for pupil diameter. , , As weight; The audio processing module is specifically used for: The volume adjustment amount of the audio signal is calculated based on the volume adjustment amount formula; wherein, the volume adjustment amount formula is: in, This is the volume adjustment amount. This is the initial volume. Distance to external sound source The angle between the user's line of sight and the direction of the external sound source. The degree of attention a user pays to the direction of the target. , , This is the adjustment coefficient; The directional difference level is determined based on the difference between the horizontal angle of the sound source and the horizontal angle of the user's line of sight, and the beamwidth corresponding to the directional difference level is used as the beamwidth of the audio signal.

6. An apparatus comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 4 above.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 4 above.

Images