Directive pickup system for intelligent glasses

Abstract

The invention discloses a directional pickup system for intelligent glasses, which relates to the technical field of intelligent glasses and comprises heterogeneous microphone array modules used for being arranged on two sides of glasses legs to form spatial asymmetric array digital microphones and realizing coverage of + / -90 degrees in a horizontal plane and + / -30 degrees in a vertical plane by utilizing a binaural sound source positioning principle. The receiver is used for receiving acoustic signals from the environment. The adaptive beam control module calculates a sound source azimuth angle based on a dynamic beam forming algorithm and the delay inequality of microphone array signals, adjusts digital beam pointing in real time, and updates the frequency to be greater than or equal to 50Hz; according to the multistage noise suppression module, a bone conduction sensor is integrated at a nose pad of a glasses frame, sound source positioning is assisted through jaw vibration signals, environmental noise is suppressed, and a three-stage noise reduction mode is set; and the execution module is used for executing switching of pickup modes based on sound sources with different distances, wherein the pickup modes comprise a near-field mode and a far-field mode. According to the invention, through fusion of bone conduction and air conduction, environment pickup is effectively and adaptively carried out.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of smart glasses, and specifically to a directional sound pickup system for smart glasses. Background Art

[0002] Smart glasses, a new generation of wearable smart devices, are gradually moving from concept to reality, demonstrating enormous potential for application in a wide range of fields. They integrate multiple functionalities, including eyewear, display, audio, and interaction, aiming to provide users with a more convenient and intelligent interactive experience. Smart glasses boast ease of use, compact size, and diverse functionality, meeting user needs in daily life, work, and study.

[0003] Existing smart glasses' sound pickup systems have the following drawbacks: 1. Inadequate ambient noise suppression: Traditional single microphones or non-directional microphone arrays easily absorb ambient noise (such as wind noise and traffic noise), resulting in reduced speech clarity; 2. Limited directional control: Fixed beamforming algorithms cannot dynamically adapt to changes in the location of the sound source, and the pickup directionality fails when the user's head turns or the sound source moves; 3. Near-field speech distortion: When picking up sounds at close range, the high sound pressure level leads to signal saturation, and when picking up sounds at long distances, the signal-to-noise ratio is insufficient;

[0004] Therefore, developing a highly directional and adaptive environment smart glasses sound pickup system is a technical problem that needs to be solved urgently by people in this technical field. Summary of the Invention

[0005] Based on this, the purpose of the present invention is to provide a directional sound pickup system for smart glasses to solve the technical problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A directional sound pickup system for smart glasses includes a heterogeneous microphone array module, an adaptive beam control module, a multi-stage noise suppression module, and an execution module. The heterogeneous microphone array module is used to arrange two MEMS digital microphones on each side of the temple to form a spatially asymmetric array. The system uses the principle of binaural sound source localization to achieve coverage of ±90° in the horizontal plane and ±30° in the vertical plane, and is used to receive sound wave signals from the environment.

[0008] The adaptive beam control module calculates the sound source azimuth based on the dynamic beamforming algorithm and the time delay difference (TDOA) of the microphone array signal, and adjusts the digital beam pointing in real time with an update frequency of ≥50Hz;

[0009] The multi-level noise suppression module integrates a bone conduction sensor at the nose pad of the frame, uses jaw vibration signals to assist in sound source positioning, suppresses ambient noise, and sets three levels of noise reduction mode;

[0010] The execution module is used to switch the sound pickup mode based on the sound sources of different distances, including the near-field mode and the far-field mode.

[0011] Preferably, the near-field mode is set to a distance of <50 cm, the bone conduction sensor is enabled as the main microphone, the voice spectrum is reconstructed through the vibration signal, and the saturation of the air conduction microphone is avoided; a nonlinear compression algorithm is used to limit the dynamic range to prevent distortion of strong near-field signals.

[0012] Preferably, the far-field mode is set to a distance ≥ 50 cm, switches to microphone array beamforming, and improves spatial resolution through the super-resolution algorithm MUSIC; combined with the glasses posture sensor data, compensates for the beam deviation caused by head rotation.

[0013] Preferably, the three-level noise reduction mainly includes:

[0014] Level 1: Separates user voice from ambient noise through correlation analysis of bone conduction signals and air conduction signals;

[0015] Second stage: Using the improved minimum variance distortionless response (MVDR) algorithm, a main lobe is formed in the target direction, and nulling is used to suppress noise in the interference direction.

[0016] Level 3: The backend deep learning model LSTM performs spectrum masking on the residual noise.

[0017] Preferably, the dynamic beamforming algorithm includes:

[0018] Step 1: Perform a 200Hz-8kHz bandpass filter on the four microphone signals to suppress low-frequency wind noise and high-frequency harmonics;

[0019] Step 2: Calculate the time delay difference of the four microphones and estimate the azimuth of the sound source using the generalized cross-correlation method (GCC-PHAT).

[0020] Step 3: Generate a dynamic beam weight vector based on the sound source azimuth, and perform weighted summation of the four signals to form the main lobe.

[0021] Step 4: Adaptively match the bone conduction signal and the main lobe signal to eliminate residual noise;

[0022] Step 5: Post-process the output signal through the LSTM model to repair high-frequency loss.

[0023] Preferably, the surface of the bone conduction sensor in contact with the skin is made of medical-grade silicone material, and the contact area between the bone conduction sensor and the skin is ≥10 mm².

[0024] Preferably, the execution module is integrated at the end of the temple and includes a DSP chip, a Bluetooth 5.3 module and a lithium polymer battery.

[0025] Preferably, the glasses posture sensor data is acquired through a gyroscope or an accelerometer.

[0026] Preferably, the MEMS digital microphone supports time-sharing wake-up.

[0027] Preferably, the dynamic beamforming algorithm is based on dynamic power management of voice activity detection (VAD), and the beamforming module is turned off during silent periods.

[0028] In summary, the present invention mainly has the following beneficial effects:

[0029] The present invention achieves an improvement of more than 20dB in noise reduction capability through the fusion of bone conduction and air conduction; dynamic beamforming makes the sound source positioning error less than 5°, meeting the needs of multi-scenario interaction; the system's comprehensive power consumption is less than 100mW, meeting the requirements of all-weather wearable endurance. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a signal processing flow chart of the present invention. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0032] like Figure 1 As shown, a directional sound pickup system for smart glasses includes a heterogeneous microphone array module, an adaptive beam control module, a multi-stage noise suppression module, and an execution module. The heterogeneous microphone array module is used to arrange two MEMS digital microphones on each side of the temple, for a total of four, forming a spatially asymmetric array. The system uses the principle of binaural sound source localization to achieve coverage of ±90° in the horizontal plane and ±30° in the vertical plane, and is used to receive sound wave signals from the environment.

[0033] The adaptive beam control module calculates the sound source azimuth based on the dynamic beamforming algorithm and the time delay difference (TDOA) of the microphone array signal, and adjusts the digital beam pointing in real time with an update frequency of ≥50Hz;

[0034] The multi-level noise suppression module integrates a bone conduction sensor in the nose pad of the frame, which uses jaw vibration signals to assist in sound source positioning, suppress ambient noise, and set three levels of noise reduction mode;

[0035] The execution module is used to switch the pickup mode based on the sound sources of different distances, including near-field mode and far-field mode.

[0036] The near-field mode is set to a distance of <50cm, the bone conduction sensor is enabled as the main microphone, and the voice spectrum is reconstructed through the vibration signal to avoid saturation of the air conduction microphone; a nonlinear compression algorithm is used to limit the dynamic range to prevent distortion of strong near-field signals.

[0037] The far-field mode is set to a distance of ≥50cm, and the microphone array beamforming is switched to improve the spatial resolution through the super-resolution algorithm MUSIC; combined with the glasses posture sensor data, the beam deviation caused by head rotation is compensated.

[0038] The three-level noise reduction mainly includes:

[0039] Level 1: Separates user voice from ambient noise through correlation analysis of bone conduction signals and air conduction signals;

[0040] Second stage: Using the improved minimum variance distortionless response (MVDR) algorithm, a main lobe is formed in the target direction, and nulling is used to suppress noise in the interference direction.

[0041] Level 3: The backend deep learning model LSTM performs spectrum masking on the residual noise.

[0042] Dynamic beamforming algorithms include:

[0043] Step 1: Perform a 200Hz-8kHz bandpass filter on the four microphone signals to suppress low-frequency wind noise and high-frequency harmonics;

[0044] Step 2: Calculate the time delay difference of the four microphones and estimate the azimuth of the sound source using the generalized cross-correlation method (GCC-PHAT).

[0045] Step 3: Generate a dynamic beam weight vector based on the sound source azimuth, and perform weighted summation of the four signals to form the main lobe.

[0046] Step 4: Adaptively match the bone conduction signal and the main lobe signal to eliminate residual noise;

[0047] Step 5: Post-process the output signal through the LSTM model to repair high-frequency loss.

[0048] The contact surface between the bone conduction sensor and the skin is made of medical-grade silicone material, and the contact area between the bone conduction sensor and the skin is ≥10mm².

[0049] The execution module is integrated at the end of the temple and includes a DSP chip, Bluetooth 5.3 module and a lithium polymer battery.

[0050] The glasses attitude sensor data is obtained through a gyroscope or accelerometer.

[0051] The MEMS digital microphone supports time-sharing wake-up.

[0052] The dynamic beamforming algorithm uses dynamic power management based on voice activity detection (VAD), and the beamforming module is turned off during silent periods.

[0053] It should be noted that, in this embodiment, the following is used to illustrate the hardware structure and the overall implementation process:

[0054] Temple A side: microphone M1 (front), microphone M2 (back), 30mm apart;

[0055] Temple B side: microphone M3 (front), microphone M4 (back), 30mm apart;

[0056] Nose pad: Bone conduction sensor S1, the surface in contact with the skin is made of medical-grade silicone material;

[0057] The signal processing flow is as follows Figure 1 As shown;

[0058] (1) Microphone signal input

[0059] Process Description: The smart glasses' microphone array (M1, M2, M3, M4) receives acoustic signals from the environment and converts them into electrical signals. These electrical signals contain a mixture of information such as the user's voice and ambient noise. This step mainly involves acoustic-to-electrical conversion, providing basic data for subsequent processing.

[0060] (2) Pre-filtering

[0061] Process Description: Bandpass filtering is performed on the microphone signal from 200 Hz to 8 kHz to eliminate low-frequency wind noise and high-frequency harmonic interference. The filtered signal is cleaner, which is beneficial for subsequent delay estimation and beamforming.

[0062] Bandpass filtering can be achieved by digital filters, whose transfer function is Satisfied: When In the range of 200Hz-8kHz, ; When outside this range, .

[0063] (3) Time Delay Odds (TDOA) Calculation

[0064] The generalized cross-correlation method GCC-PHAT is used to calculate the time delay difference between microphone pairs (such as M1-M2 and M3-M4) ( The GCC-PHAT algorithm estimates the delay difference by calculating the cross-correlation function between two signals and finding its peak position.

[0065]

[0066] in,

[0067] and are the Fourier transforms of the signals from microphones M1 and M2, respectively, yes The conjugate of is the time delay variable.

[0068] By calculation The peak position of .

[0069] (4) Sound source azimuth estimation

[0070] According to the delay difference ( ) and microphone spacing ( ), and the speed of sound (c), estimate the azimuth of the sound source ( The calculation formula is as follows:

[0071]

[0072] in, It is the delay difference calculated by GCC-PHAT algorithm. is the microphone spacing, is the speed of sound (about 343 m / s in air).

[0073] (5) Dynamic beam weight generation

[0074] According to the azimuth of the sound source ( ), generates a dynamic beam weight vector (W); the weight vector is used to adjust the phase and amplitude of each microphone signal to form a main lobe in the direction of the sound source and a null in the direction of the interference.

[0075] Take the uniform linear array as an example:

[0076] in, is the signal frequency, is the microphone spacing, is the azimuth of the sound source, is the speed of sound; this formula expresses the complex weights that need to be multiplied by each microphone signal to achieve beamforming.

[0077] (6) Beamforming

[0078] The generated beam weight vector (W) is applied to the microphone signal (X) to form the beamformed signal (Y) through weighted summation.

[0079]

[0080] Where N is the number of microphones (4 in this case), and They are The beam weights and signals of each microphone are combined; by weighted summation, the signal from the direction of the sound source can be enhanced and interference from other directions can be suppressed.

[0081] (7) Bone conduction signal fusion

[0082] The signal collected by the bone conduction sensor (S1) and the beamformed signal are adaptively matched filtered; this step uses the correlation between the bone conduction signal and the air conduction signal to further eliminate residual noise and improve speech clarity.

[0083] Adaptive matched filtering can be implemented through methods such as Wiener filtering. Its goal is to minimize the mean square error between the output signal and the desired signal. The core idea is to dynamically adjust the filter coefficients based on the statistical characteristics of the input signal and the desired signal.

[0084] (8) Post-processing

[0085] Process description: Use the LSTM model to post-process the output signal; the LSTM model can learn the long-term dependencies of speech signals, further suppress residual noise, and repair high-frequency loss, making the speech more natural and clear.

[0086] Example 2: It should be noted that, in this example, the following scenario experiment results are presented:

[0087] Scenario 1 (noisy cafe): Speech intelligibility (STOI) improved to 0.87 under 65dB SPL background noise;

[0088] Scenario 2 (wind noise during cycling): At a wind speed of 15 km / h, the voice signal-to-noise ratio (SNR) improved by 12 dB.

[0089] Scenario 3 (meeting recording): Voice pickup at a distance of 3 meters, with keyword recognition accuracy reaching 92%.

[0090] The above embodiments are only for illustrating the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A directional sound pickup system for smart glasses, characterized in that: The system comprises a heterogeneous microphone array module, an adaptive beam control module, a multi-stage noise suppression module, and an execution module. The heterogeneous microphone array module is used to arrange two MEMS digital microphones on each side of the temple to form a spatially asymmetric array. The system uses the binaural sound source localization principle to achieve coverage of ±90° in the horizontal plane and ±30° in the vertical plane, and is used to receive sound wave signals from the environment. The adaptive beam control module calculates the sound source azimuth based on the dynamic beamforming algorithm and the time delay difference (TDOA) of the microphone array signal, and adjusts the digital beam pointing in real time with an update frequency of ≥50Hz; The multi-level noise suppression module integrates a bone conduction sensor at the nose pad of the frame, uses jaw vibration signals to assist in sound source positioning, suppresses ambient noise, and sets three levels of noise reduction mode; The execution module is used to switch the sound pickup mode based on the sound sources of different distances, including the near-field mode and the far-field mode.

2. The directional sound pickup system for smart glasses according to claim 1, characterized in that: The near-field mode is set to a distance of <50cm, the bone conduction sensor is enabled as the main microphone, the voice spectrum is reconstructed through the vibration signal, and the saturation of the air conduction microphone is avoided; a nonlinear compression algorithm is used to limit the dynamic range to prevent distortion of strong near-field signals.

3. The directional sound pickup system for smart glasses according to claim 1, characterized in that: The far-field mode is set to a distance ≥ 50 cm, switches to microphone array beamforming, and improves spatial resolution through the super-resolution algorithm MUSIC; combined with the glasses posture sensor data, compensates for beam deviation caused by head rotation.

4. The directional sound pickup system for smart glasses according to claim 1, characterized in that: The three-level noise reduction mainly includes: Level 1: Separates user voice from ambient noise through correlation analysis of bone conduction signals and air conduction signals; Second stage: Using the improved minimum variance distortionless response (MVDR) algorithm, a main lobe is formed in the target direction, and nulling is used to suppress noise in the interference direction. Level 3: The backend deep learning model LSTM performs spectrum masking on the residual noise.

5. The directional sound pickup system for smart glasses according to claim 1, characterized in that: The dynamic beamforming algorithm includes: Step 1: Perform a 200Hz-8kHz bandpass filter on the four microphone signals to suppress low-frequency wind noise and high-frequency harmonics; Step 2: Calculate the time delay difference of the four microphones and estimate the azimuth of the sound source using the generalized cross-correlation method (GCC-PHAT). Step 3: Generate a dynamic beam weight vector based on the sound source azimuth, and perform weighted summation of the four signals to form the main lobe. Step 4: Adaptively match the bone conduction signal and the main lobe signal to eliminate residual noise; Step 5: Post-process the output signal through the LSTM model to repair high-frequency loss.

6. The directional sound pickup system for smart glasses according to claim 1, characterized in that: The bone conduction sensor and the skin contact surface are made of medical grade silicone material, and the bone conduction sensor and the skin contact area is ≥10mm².

7. The directional sound pickup system for smart glasses according to claim 1, characterized in that: The execution module is integrated at the end of the temple and includes a DSP chip, a Bluetooth 5.3 module and a lithium polymer battery.

8. The directional sound pickup system for smart glasses according to claim 3, characterized in that: The glasses posture sensor data is obtained through a gyroscope or an accelerometer.

9. The directional sound pickup system for smart glasses according to claim 1, characterized in that: The MEMS digital microphone supports time-sharing wake-up.

10. The directional sound pickup system for smart glasses according to claim 5, characterized in that: The dynamic beamforming algorithm is based on dynamic power management of voice activity detection (VAD), and the beamforming module is turned off during silent periods.

Images