Method and apparatus for wind noise attenuation
Through the analysis of spatial selectivity and signal-related attributes, combined with the signal phase alignment and cross-power spectrum of multiple microphones, effective detection and attenuation of wind noise is achieved, and the problem of difficult to distinguish and remove stroke noise in the prior art is solved, and the problem of the existing technology is improved. The cleanliness of microphone signals and the operational performance of the equipment.
Patent Information
- Application Number
- CN202180010243.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-01-24
- Filing Date
- 2021-01-22
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-01-22
AI Technical Summary
The prior art is difficult to effectively reduce wind noise in signals detected by microphones, especially in hearing aids or hands-free communication systems in automobiles, where wind noise and voice are difficult to distinguish by energy or signal-to-noise ratio analysis in the time domain or frequency domain.
Using spatial selectivity and signal-related properties, wind noise detectors are constructed through signal phase alignment and cross-power spectrum analysis of multiple microphones, and classified signals into wind noise, voice mixing, voice and other noises, and based on the normalization of phase difference. Calculate the gain function by variance and signal-to-noise ratio to achieve effective attenuation of wind noise.
Quickly and effectively remove wind noise, maintain voice quality, and improve the operating performance of audio equipment in noisy environments.
Smart Images

Figure CN114930450B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to eliminating or reducing wind noise in a signal detected by a microphone. Background Art
[0002] Wind noise (WN) is a major source of hearing disturbance in many environments, for example, for hearing aids or hands-free communication systems in cars. Wind noise is caused by turbulent airflow hitting the microphone diaphragm, which produces a strong audible signal mainly concentrated in the relatively low-frequency area. Reliable and effective wind noise reduction (WNR) capabilities are important to allow these audio devices or voice communication systems to operate well in noisy conditions.
[0003] However, previous noise suppression methods have failed to adequately remove wind noise, mainly because wind noise and speech are difficult to distinguish through energy or SNR analysis in the time or frequency domain. BRIEF DESCRIPTION OF THE DRAWINGS
[0004] For a more complete understanding of the present disclosure, reference should be made to the following detailed description and accompanying drawings, in which:
[0005] Figure 1 including diagrams of systems for wind noise reduction according to various embodiments of the present invention;
[0006] Figure 2 A flowchart including a method for wind noise reduction according to various embodiments of the present invention;
[0007] FIG. 3 includes diagrams illustrating aspects of the operation of the methods described herein according to various embodiments of the invention;
[0008] Figure 4 including diagrams illustrating aspects of the operation of the methods described herein according to various embodiments of the invention;
[0009] Figure 5 including diagrams illustrating aspects of the operation of the methods described herein according to various embodiments of the invention;
[0010] Figure 6 including diagrams illustrating aspects of the operation of the methods described herein according to various embodiments of the invention;
[0011] Figure 7 including diagrams illustrating aspects of the operation of the methods described herein according to various embodiments of the invention;
[0012] Figure 8 Included are diagrams illustrating aspects of the operation of the methods described herein according to various embodiments of the invention.
[0013] The skilled artisan will appreciate that the elements in the figures are illustrated for simplicity and clarity. It will also be understood that certain actions and / or steps may be described or depicted in a particular order of occurrence, while those skilled in the art will appreciate that such specificity with respect to order is not actually required. It will also be understood that the terms and expressions used herein have the ordinary meanings ascribed to such terms and expressions with respect to the respective fields of investigation and study to which they correspond, unless a specific meaning is otherwise set forth herein. DETAILED DESCRIPTION
[0014] The methods described herein employ spatial selectivity and signal correlation properties at two or more microphones to determine wind noise in a received signal. By exploiting three signal correlation properties present at different microphone locations (wind noise signal that is uncorrelated with the speech signal, wind noise at different locations that is largely uncorrelated, and speech at all microphones that is correlated on a compact microphone array), these methods quickly construct a reliable wind noise detector that classifies the microphone input at any given time into one of four categories (wind noise, wind noise mixed with speech, speech, and noise other than chattering, such as conventional stationary noise).
[0015] In various aspects and based on wind noise detection and / or classification results, the present invention also creates and applies an effective wind noise attenuator for a signal (e.g., two incoming microphone inputs). In some aspects, the attenuation gain factor is derived from the coherence, the phase of the cross power spectrum of the two (or more) microphone inputs, and the probability of speech and wind noise estimated at the wind noise detector. A comfort noise power spectrum generated from the minimum statistics of the two microphone inputs can also be created and applied to the audio signal attenuated by wind noise to eliminate the noise gating effect. The application of the method provided herein removes wind noise quickly and in large quantities while maintaining speech quality.
[0016] In various aspects, the present method embodies various methods and algorithms for two (or more) microphone based wind noise / speech detection and wind noise suppression. Various steps are performed.
[0017] In one method, preprocessing is performed first. In some aspects, speech signals are captured at two microphones in a car, and each microphone signal is phase aligned. Phase alignment is accomplished by a combination of: a geometric method that determines a constant time delay between two signals originating from a speech source (e.g., driver or co-pilot); and a delay calculated at runtime based on the cross-correlation of the two signals. Decision logic is used to determine whether to use a geometrically based static delay or a dynamically calculated runtime delay to phase align the two signals. Unlike previous methods, this method is reliable and more tolerant to inaccurate geometric measurements or speaker (driver / co-pilot) positions in the car.
[0018] Next, create metrics for the measurements of wind noise and speech. Create two metrics: the probability of speech being present and the probability of wind noise being present. In various aspects, these metrics are probabilities because their values range between 0 and 1.
[0019] Unlike previous methods that use energy or SNR (signal-to-noise ratio) for signal classification (e.g., speech, noise, etc.), these probabilities are used for speech / wind noise classification and are derived entirely from the statistics of phase differences in multiple frequency regions. In the method described herein, the normalized variance of the phase difference extended over a certain frequency region is used as a key parameter for distinguishing speech from wind noise. These normalized variances are further used to construct the probability of speech presence and the probability of wind noise presence. This process occurs for each time interval (e.g., 10ms to 20ms) at runtime.
[0020] Then, the speech and wind noise are detected and / or classified. The classifier / detector used in this article uses decision logic (for example, implemented as any combination of hardware or software) to pre-train (or offline train) the decision logic using audio samples including only speech, only wind noise, and mixed speech / wind noise data. At each short time interval (for example, 10ms to 20ms), two metrics are calculated, namely the speech probability and the wind noise probability, which characterize the signal characteristics in different frequency regions. The two metrics are weighted separately and then linearly combined to form a single metric for classification. The single metric is compared with three thresholds representing the speech threshold, the wind noise threshold, and the threshold for the simultaneous occurrence of speech and wind noise. In the example, these thresholds are determined from offline classifier training.
[0021] In various aspects, and in order to enhance the reliability of speech / wind noise classification frame by frame and avoid occasional classification errors (which would lead to annoying wind noise leakage after wind noise is suppressed), the method described herein adopts a majority voting scheme, where each classification result c at frame t is t Together with the (Nl) classification results from the (Nl) previous frames, they are pushed into a circular buffer of length N (e.g., N=10). The signal category decision of the current frame t is made by majority voting, that is, the classification result that appears most frequently in the circular buffer is selected as the final classification result.
[0022] Next, the gain function is derived and applied. Unlike previous gain function construction methods (which only use signal-to-noise ratio (SNR) information), the wind noise gain function used in the method described in this article is a combination of SNR and the normalized variance of phase difference, which also plays a key role in wind noise / voice detection. The combination of SNR and phase information provides spectral and spatial information and is much better than conventional SNR, which only derives a gain function for wind noise attenuation / voice preservation.
[0023] In many of these embodiments, the system includes a first microphone, a second microphone, and a control circuit. The first microphone obtains a first audio signal, and the second microphone obtains a second audio signal. The first microphone is spatially separated from the second microphone.
[0024] The control circuit is coupled to the first microphone and the second microphone, and is configured to continuously and simultaneously segment the first audio signal arriving at the first microphone and the second audio signal arriving at the second microphone into time periods. For each time period, the first audio signal arriving at the first microphone forms a first frame audio signal, and the second audio signal arriving at the second microphone forms a second frame audio signal.
[0025] The control circuit is also configured to time-align the first frame audio signal and the second frame audio signal relative to the target speech source. The time alignment of the first frame audio signal and the second frame audio signal is based on: a measurement result based on static geometric characteristics, which is adjusted by a dynamic cross-correlation evaluation between signals received at the two microphones at runtime.
[0026] The control circuit is also configured to perform a Fourier transform on each of the time-aligned first frame audio signals to generate a first spectrum, and to perform a Fourier transform on the second frame audio signal to generate a second spectrum. Each of the first spectrum and the second spectrum represents a spectrum of one of the two time-aligned microphone signals in each time period.
[0027] The control circuit is further configured to calculate a phase difference between the first spectrum and the second spectrum at each of a plurality of frequencies based on a cross-correlation of the first spectrum and the second spectrum. The control circuit is further configured to determine a normalized variance of the phase difference within a defined frequency range for each time period. The frequency range is calculated based on microphone geometry so that an error margin in calculating the normalized variance of the phase difference is minimized.
[0028] The control circuit is also configured to formulate and evaluate the probability of speech presence and the probability of wind noise presence in each time period based on the normalized variance of the spectral phase difference of the two time-aligned microphone signals. Then, the control circuit is configured to determine the category of each time period in each time period, wherein the category is one of the following: only speech, only wind noise, speech mixed with wind noise, or unknown, wherein the decision logic is used to determine the category and the decision logic is based on a first function, which combines the individual values and combined values of the probability of speech presence and the probability of wind noise presence. The value of the first function is compared with multiple thresholds and a wind noise detection decision is made. Based on the determined category, a wind attenuation action is selectively triggered.
[0029] When the action is to perform wind noise attenuation, the control circuit is configured to calculate a gain or attenuation function based on the normalized variance of the phase difference and the individual phase difference at each of the plurality of frequencies within a predetermined frequency range. Wind noise attenuation is performed in the frequency domain by multiplying the gain or attenuation function with the magnitude of each of the first spectrum and the second spectrum to produce a first spectrum with wind noise removed and a second spectrum with wind noise removed.
[0030] The control circuit is configured to then combine the first spectrum without wind noise and the second spectrum without wind noise to produce a combined spectrum and construct a time domain signal without wind noise by performing an inverse FFT on the combined spectrum.
[0031] The control circuitry, possibly in combination with other entities, may use the time-domain signal to take an action that is one or more of: transmitting the time-domain signal to an electronic device, controlling electronic equipment using the time-domain signal, or interacting with electronic equipment using the time-domain signal.
[0032] In some aspects, the time period is between 10 and 20 milliseconds in length.Other examples are possible.
[0033] In an example, the target speech source includes speech from a person sitting in a vehicle seat. Other examples of speech sources are possible.
[0034] In other examples, the probability that speech is present and the probability that wind noise is present both have values between 0 and 1.
[0035] In other aspects, the determination of the category further utilizes a majority voting method that considers the current decision and the decision sequence in the previous continuous time period. In other examples, the probability of speech existence and the probability of wind noise existence provide a metric that is used to evaluate the degree of speech existence or wind noise existence at each time period.
[0036] In yet other aspects, when the determined decision is that it is wind noise only or wind noise mixed with speech, a wind noise attenuation action is triggered.In still other examples, the threshold is estimated offline by using the amount of speech and wind noise samples in an offline algorithm training phase.
[0037] In an example, the system is at least partially disposed in a vehicle. Other locations are possible. In some examples, the sound source moves, while in other examples, the sound source is stationary or nearly stationary.
[0038] In others of these embodiments, a method for wind noise reduction in a microphone signal is provided.
[0039] The control circuit continuously and simultaneously segments the first audio signal arriving at the first microphone and the second audio signal arriving at the second microphone into time periods, so that for each time period, the first audio signal arriving at the first microphone forms a first frame audio signal and the second audio signal arriving at the second microphone forms a second frame audio signal.
[0040] The control circuitry temporally aligns the first frame audio signal and the second frame audio signal relative to a target speech source. The temporal alignment of the first frame audio signal and the second frame audio signal is based on a measurement result based on static geometric characteristics adjusted by a dynamic cross-correlation evaluation between signals received at the two microphones at runtime.
[0041] The control circuit performs a Fourier transform on each of the time-aligned first frame audio signals to generate a first spectrum, and performs a Fourier transform on the second frame audio signal to generate a second spectrum. Each of the first spectrum and the second spectrum represents the spectrum of one of the two time-aligned microphone signals in each time period.
[0042] The control circuit calculates a phase difference between the first spectrum and the second spectrum at each of a plurality of frequencies based on a cross-correlation of the first spectrum and the second spectrum.
[0043] The control circuit determines, for each time period, a normalized variance of the phase difference within a defined frequency range. The frequency range is calculated based on the microphone geometry so that an error margin in calculating the normalized variance of the phase difference is minimized.
[0044] The control circuit formulates and evaluates the probability of speech presence and the probability of wind noise presence in each time period based on the normalized variance of the spectral phase difference of two time-aligned microphone signals. The control circuit determines the category of each time period in each time period, and the category is one of the following: only speech, only wind noise, speech mixed with wind noise, or unknown. The decision logic is used to determine the category and the decision logic is based on a first function that combines the individual values and combined values of the probability of speech presence and the probability of wind noise presence. The value of the first function is compared with multiple thresholds and a wind noise detection decision is made. Based on the determined category, a wind attenuation action is selectively triggered.
[0045] When the action is to perform wind noise attenuation, the control circuit calculates a gain or attenuation function based on the normalized variance of the phase difference and the individual phase difference at each of the plurality of frequencies within a predetermined frequency range. Wind noise attenuation is performed in the frequency domain by multiplying the gain or attenuation function with the magnitude of each of the first spectrum and the second spectrum to produce a first spectrum with wind noise removed and a second spectrum with wind noise removed.
[0046] The control circuit combines the first spectrum with wind noise removed and the second spectrum with wind noise removed to generate a combined spectrum. The control circuit constructs a time domain signal with wind noise removed by performing an inverse FFT on the combined spectrum.
[0047] An action is taken using the time domain signal, the action being one or more of: transmitting the time domain signal to an electronic device, controlling electronic equipment using the time domain signal, or interacting with electronic equipment using the time domain signal. Other action examples are possible.
[0048] Reference now Figure 1 , describes an example of a system for attenuating wind noise. A vehicle 100 includes a first microphone 102, a second microphone 104, a driver 101, and a passenger 103. The microphones 102 and 104 may be coupled to a control circuit 106.
[0049] Microphones 102 and 104 can be any type of microphones that, in some aspects, detect human speech. In one example, microphones 102 and 104 can be conventional analog microphones that sense human speech signals in the time domain and generate analog signals representing the detected speech. Vehicle 100 is any type of vehicle that transports humans, such as a passenger car or a truck. Other examples are possible. Although two microphones are shown, it should be understood that these methods are applicable to any number of microphones.
[0050] It should be understood that, as used herein, the term "control circuitry" broadly refers to any microcontroller, computer, or processor-based device having a processor, memory, and programmable input / output peripherals that are typically designed to manage the operation of other components and devices. It should also be understood to include common accompanying accessory devices, including memory, transceivers for communicating with other components and devices, etc. These architectural options are well known and understood in the art and do not require further description herein. The control circuitry 106 can be configured (e.g., by using corresponding programming stored in memory, as will be fully understood by those skilled in the art) to implement one or more of the steps, actions, and / or functions described herein.
[0051] The control circuit 106 may be deployed at various locations in the vehicle 100. In one example, the control circuit 106 may be deployed at a vehicle control unit (e.g., a vehicle control unit that controls or monitors various functions at the vehicle 100). In general, the control circuit 106 determines whether wind noise is present in the received microphone signals (as described below) and then selectively removes the wind noise from these signals. After removing the wind noise, the now attenuated microphone signals may be used for other purposes (e.g., performing actions at the vehicle 100).
[0052] Microphones 102 and 104 may be coupled to control circuit 106 via a wired connection or a wireless connection. Microphones 102 and 104 may also be deployed at various locations in vehicle 100 depending on user needs and / or system requirements.
[0053] exist Figure 1 In one example of the operation of the system, the first microphone 102 obtains a first audio signal and the second microphone 104 obtains a second audio signal. The first microphone 102 is spatially separated from the second microphone 104.
[0054] The control circuit 106 is configured to continuously and simultaneously segment the first audio signal arriving at the first microphone 102 and the second audio signal arriving at the second microphone 104 into time periods, so that for each time period, the first audio signal arriving at the first microphone 102 forms a first frame audio signal, and the second audio signal arriving at the second microphone 104 forms a second frame audio signal.
[0055] The control circuit 106 is further configured to align the first frame audio signal and the second frame audio signal in time relative to the target speech source. The time alignment of the first frame audio signal and the second frame audio signal is based on: a measurement result based on static geometric characteristics, which is adjusted by a dynamic cross-correlation evaluation between the signals received at the two microphones during operation.
[0056] The control circuit 106 is further configured to perform a Fourier transform on each of the time-aligned first frame audio signal and the second frame audio signal, perform the Fourier transform on the first frame audio signal to generate a first spectrum, and perform the Fourier transform on the second frame audio signal to generate a second spectrum. Each of the first spectrum and the second spectrum represents the spectrum of one of the two time-aligned microphone signals in each time period.
[0057] The control circuit 106 is further configured to calculate a phase difference between the first spectrum and the second spectrum at each of the plurality of frequencies based on a cross-correlation of the first spectrum and the second spectrum. The control circuit 106 is further configured to determine a normalized variance of the phase difference within a defined frequency range for each time period. The frequency range is calculated based on microphone geometry so that an error margin in calculating the normalized variance of the phase difference is minimized.
[0058] The control circuit 106 is also configured to formulate and evaluate the probability of speech presence and the probability of wind noise presence in each time period based on the normalized variance of the spectral phase difference of the two time-aligned microphone signals. Then, the control circuit 106 is configured to determine the category of each time period in each time period, wherein the category is one of the following: only speech, only wind noise, speech mixed with wind noise, or unknown, wherein the decision logic is used to determine the category and the decision logic is based on a first function, which combines the individual values and combined values of the probability of speech presence and the probability of wind noise presence, wherein the value of the first function is compared with multiple thresholds and a wind noise detection decision is made. Based on the determined category, the wind attenuation action is selectively triggered.
[0059] When the action is to perform wind noise attenuation, the control circuit 106 is configured to calculate a gain or attenuation function based on the normalized variance of the phase difference and the individual phase difference at each of the plurality of frequencies within a predetermined frequency range. Wind noise attenuation is performed in the frequency domain by multiplying the gain or attenuation function with the magnitude of each of the first spectrum and the second spectrum to produce a first spectrum with wind noise removed and a second spectrum with wind noise removed.
[0060] The control circuit 106 is configured to then combine the first spectrum from which wind noise is removed and the second spectrum from which wind noise is removed to produce a combined spectrum and construct a time domain signal from which wind noise is removed by performing an inverse FFT on the combined spectrum.
[0061] The control circuit 106, by itself or in combination with other entities, can use the time domain signal to take an action, which is one or more of the following: transmit the time domain signal (using a transmitter 110) to an electronic device (e.g., an electronic device such as a smart phone, computer, laptop, or tablet), use the resulting time domain signal to control electronic equipment 112 (e.g., electronic equipment in the vehicle 100, such as an audio system, steering system, or braking system), or use the time domain signal to interact with electronic equipment 112. In one example, a user can verbally command a radio to be activated and then control the volume of the radio. Other examples are possible.
[0062] In some aspects, the length of the time period of the signal is between 10 and 20 milliseconds.Other examples are possible.
[0063] In an example, the target speech source includes speech from a driver 101 or a passenger 103 sitting in a vehicle seat. Other examples of speech sources are possible.
[0064] In other examples, the probability that speech is present and the probability that wind noise is present both have values between 0 and 1.
[0065] In other aspects, the determination of the category further utilizes a majority voting method that considers the current decision and the decision sequence in the previous continuous time period. In other examples, the probability of speech existence and the probability of wind noise existence provide a metric that is used to evaluate the degree of speech existence or wind noise existence at each time period.
[0066] In yet other aspects, when it has been determined that the determination is only wind noise or wind noise mixed with voice, the wind noise attenuation action is triggered. In yet other examples, the threshold is estimated offline by using the amount of voice and wind noise samples in the offline algorithm training phase. For example, this can be determined at the factory when the system is initialized.
[0067] In some examples, the sound sources (driver 101 and passenger 103) are moving, while in other examples, the sound sources are stationary or nearly stationary.
[0068] Reference now Figure 2 , an example of a method for wind noise detection and attenuation is described.
[0069] At step 202, spectral analysis is performed. In one example, each 10ms input signal x1(n), x2(n) from the dual microphones is passed through an overlap-add process to formulate a 20ms frame with the previous frame and produce spectrally equivalent values x1(f), x2(f) representing the "raw" data to be processed.
[0070] At step 204, microphone input steering is performed. The algorithm keeps the two microphone inputs x1(f), x2(f) phase aligned. To do this, as part of system initialization, steering vectors derived from microphone geometry are calculated. In some aspects, the geometry-based steering vector formation is similar to but simpler than the steering vector formation used in a fixed beamformer (FBF).
[0071] Regarding microphone geometry, two microphone arrays are mounted inside the vehicle (usually overhead on the center console) in a collinear manner and perpendicular to the center axis of the vehicle. The microphone array geometry is defined by the distance from the driver and co-driver's mouth to the microphone, e.g. Figure 1 DM1 is the distance from the driver 101 to microphone 1 (102). PM2 is the distance from the co-driver or passenger 103 to microphone 2 (104). In practice, it is also assumed that the driver 101 and the front passenger 103 have symmetrical geometric characteristics relative to the central axis of the vehicle, that is, PM1=DM2 and PM2=DM1, etc.
[0072] Assuming that the speech source in the vehicle comes from the driver 101 and the effect of the multipath of the signal propagating to the two microphones 102 and 104 is negligible, the steering vector svl that aligns the phase of the speech signal is determined by the following formula:
[0073]
[0074] τ1, τ2 are the signal propagation delays (in seconds) to microphones 1 and 2. α1, α2 are two factors related to the normalized path loss of the individuals.
[0075] Simplifying the steering vector by assuming that the delay of the signal propagating to the farthest microphone is 0, the steering vector becomes:
[0076]
[0077] where τ is the relative delay of speech arriving at the closer microphone (negative number in seconds).
[0078] Assuming that the (mouth) positions of the driver 101 and passenger 103 are symmetric with respect to the dual microphone array; the same steering vectors are formulated to apply to both the driver 101 and passenger 103 .
[0079] Assuming that the speech source in the vehicle 100 is from the driver and the effect of the multipath of the signal propagating to the two microphones 102 and 104 is negligible, the steering vector sv1 that aligns the phase of the speech signal is determined by the following formula:
[0080]
[0081] τ1, τ2 are the signal propagation delays (in seconds) to microphones 1 and 2. α1, α2 are two factors related to the normalized path loss of the individuals.
[0082] Simplifying the steering vector by assuming that the delay of the signal propagating to the farthest microphone is zero, the steering vector becomes:
[0083]
[0084] where τ is the relative delay of speech arriving at the closer microphone (negative number in seconds).
[0085] Assuming that the (mouth) positions of the driver 101 and passenger 103 are symmetric with respect to the dual microphone array; the same steering vectors are formulated to apply to both the driver and the co-pilot.
[0086] In step 206, signal alignment is performed. Given the steering vectors derived from the microphone geometry, the two microphone signals x1(f), x2(f) originating from the driver or the co-pilot are phase aligned in the viewing direction of the driver and the co-pilot by:
[0087] For Driver 101:
[0088]
[0089] Or for the co-pilot (passenger) 103:
[0090]
[0091] In step 208, dynamic time delay estimation and guidance vector selection are performed. The microphone geometry is measured once and becomes a fixed parameter for each use. However, the distance from the driver 101 and the passenger 103 to the two microphones 102 and 104 may change from time to time. Even the height of the driver / co-pilot may be different, which means that the measured geometry is no longer accurately applicable. Therefore, the relative time delay calculated according to the geometry should be confirmed as a "nominal" value, and there will be a phase alignment error caused by mismatching the geometry.
[0092] To alleviate this problem, the time delay is estimated on the fly via the cross-correlation of the two microphone signals x1(n), x2(n) at each frame in the following way:
[0093]
[0094] Where n and m are the data sample indices.
[0095] The cross-correlation R calculated in the time domain x1x2 (m) Through R x1x1(0) and R x2x2 The geometric mean of (0) is further normalized to become the cross-correlation coefficient. The absolute value of the cross-correlation coefficient is constrained to the interval [0,1]:
[0096]
[0097] 0≤|R x1x2 (m)|≤1
[0098] Therefore, the effective time delay between x1 and x2 in samples can be estimated as follows:
[0099]
[0100] If R x1x2 (τ_d)>thld_R x1x2
[0101] Then τ_d is effective
[0102] otherwise
[0103] τ_d is invalid
[0104] where τ_d, τ, Δ denote the time delay in samples for dynamic, geometric and margin, the margin being the maximum allowed deviation from the geometric τ. x1x2 is the threshold value (e.g. 0.60).
[0105] The delay τ_d, if valid, is converted from samples to seconds to construct the dynamic steering vector:
[0106] τ d =τ_d / f s
[0107]
[0108] where f s is the sampling frequency in Hz.
[0109] For geometric or dynamically constructed steering vectors, the path loss remains the same.
[0110] At each frame, if the calculated dynamic delay is valid, its corresponding steering vector is used for signal alignment; otherwise the geometrically derived steering vector is used. Dynamic τ d The calculation and its application of steering vectors mitigate possible errors in the alignment of the two signals due to mismatched geometrical properties and prevent occasional severe errors in the dynamic time delay induced by numerical analysis.
[0111] At step 210, the coherence and cross-spectrum of the signals are determined. The statistics of the two microphone signals show a large difference between the wind noise and the speech in the vehicle. Useful statistics are best represented by the coherence of the two signals X1(f) and X2(f), defined as:
[0112]
[0113] Where {}* represents the complex conjugate operator.
[0114] Due to the short frame analysis, the cross power spectrum Over time t it is smoothed as:
[0115]
[0116] In one example, the smoothing factor α is set to 0.5.
[0117] The phase of the cross power spectrum is in some ways the most important statistic for wind noise / speech detection. The phase of the cross power spectrum is calculated as follows:
[0118]
[0119] where X1(f) and X2(f) are phase aligned via geometric and dynamic steering vectors as discussed elsewhere herein.
[0120] At step 212, wind noise and speech differentiation (by phase analysis) is performed. In a vehicle, the distinction between wind noise and speech is explored from the phase of the cross-complex spectrum between the two aligned signals X1(f) and X2(f). Because the speech signal is correlated, while the wind noise is not. For speech, the phase of the cross-spectrum is usually very small, especially in the low or medium frequency range (e.g., up to 2kHz). On the other hand, for the case of wind noise, the phase value of the cross-spectrum is much larger, and its variation with time and frequency is random.
[0121] In order to better distinguish wind noise from speech, the analysis frequency range is divided into two regions: the first region [(F_WN), from 10 Hz (F_WN_B) to 500 Hz (F_WN_E)] is mainly used for wind noise detection, and the second region [F_SP, from 600 Hz (F_SP_B) to 2000 Hz (F_SP_E)] is mainly used for speech detection.
[0122] Since the individual phase values at the time / frequency grid are meaningless, a statistical measure is created to characterize the phase. This measure is the normalized variance of the cross-spectral phase and is defined as:
[0123]
[0124] Calculate two phase variances from one of the two frequency regions respectively and
[0125] From region F_WN, f1 = F_WN_B, f2 = F_WN_E (e.g., f1 = 20 Hz, f2 = 500 Hz). From region F_SP, f1=F_SP_B, f2=F_SP_E (e.g. f1=500 Hz, f2=2000 Hz).
[0126] However, the maximum frequency f2 in the region F_SP must be limited such that:
[0127]
[0128] where c and d are the speed of sound and the separation distance between the two microphones.
[0129] Figure 3A shows clear dual-microphone speech recorded in a car without jitter, and Figure 3B Showing dual microphone dithering in a car with no speech present.
[0130] Figure 4 and Figure 5 (The horizontal axis is variance, the vertical axis is the number of occurrences) The normalized phase variance distribution (histogram) in two frequency regions for the clear speech case is presented. and The distribution is restricted to a range close to zero. On the other hand, Figure 6 and Figure 7 As shown in , the two distributions for the wind noise case are spread over a wider interval. It is obvious that speech and wind noise are separable from the perspective of normalized phase variance.
[0131] Furthermore, through the analysis of these statistical data, it can be concluded that wind noise is more easily detected in the frequency region F_WN, while voice is more easily recognized in the frequency region F_SP, especially when wind noise and voice occur simultaneously.
[0132] At step 214, the assignment of speech and wind noise probabilities is performed. To facilitate wind noise / speech detection or recognition, the probabilities of speech and wind noise are calculated as follows:
[0133]
[0134]
[0135] in denote the normalized phase variances from regions F_WN and F_SP, respectively. are thresholds used to determine wind noise probability and speech probability in their associated frequency regions.
[0136] At step 216, decision logic is used to classify wind noise, speech, or a mixture of wind noise and speech.
[0137] The wind noise and voice detection decision logic is calculated as:
[0138]
[0139] Among them, thld_sp, thld_wn, thld_sp_wn are thresholds, α sp and α wn is a weight, and the operator ← is an assignment.
[0140] The instantaneous (i.e., per-frame) classification result c is further denoised by consulting neighboring results. The current value ct at frame t and the (Nl) decision results from the (Nl) previous frames are stored in a circular buffer of length N (e.g., N=10). The final signal class decision for the current frame t is made by a so-called majority vote; the class that appears the most times in the circular buffer is selected.
[0141] C t =majority(c t-N-1 ,C t-N-2 , ...C t )
[0142] Among them C t is the final decision of the signal category at frame t, and c t-N-l ,c t-N-2 ,...c t is the instantaneous category computed for the current frame and (Nl) previous frames.
[0143] Figure 8 The probability estimation and signal classification results for a dual microphone recording are highlighted, where both speech and wind noise are present, except for the beginning and end of the portion where only speech is present. Examples of speech and wind noise are marked in the figure. In this example, the conventional noise category is merged with the speech category, but only wind noise and wind noise mixed with speech are two separate categories. The probability analysis and classification decisions shown in this figure are matched to the true content in the recording (i.e., speech, wind noise, or wind noise mixed with speech). It can be seen that in all aspects, wind noise mixed with speech is almost always correctly selected by the high values of the probability of both wind noise and speech being present, and is not confused with the speech or wind noise categories.
[0144] Wind noise reduction can now be performed. Wind noise reduction occurs when a wind noise detector detects the presence of wind noise. A control circuit that implements wind noise reduction implements or utilizes four functions in various aspects: wind noise image estimation, wind noise reduction gain construction, comfort noise generation, wind noise reduction, and comfort noise injection.
[0145] In step 218, a wind noise image estimation is performed. It is assumed that the wind noise signals at the two microphones 102 and 104 are uncorrelated, while the speech signals are correlated. In addition, the wind noise and speech signals are also uncorrelated. Therefore, the theoretical noise power spectral density (PSD) can be formulated as:
[0146]
[0147] where t, f are the frame and frequency indices.
[0148] However, these assumptions do not always hold. For one reason, the correctness of the assumptions depends on the geometry of the microphones. For example, the farther the microphones are separated, the lower the correlation of the speech signals at the two microphones. The theoretical wind noise PSD is often underestimated. A more reliable and more powerful wind noise PSD is designed as a combination of the theoretical and geometric mean of the automatic PSDs of X1 and X2 weighted by the probabilities of speech and wind noise, as follows:
[0149]
[0150] α=ALPHA(prob wn +(1-prob sp ))
[0151] where ALPHA is a constant (0.4), prob wn 、prob sp is the probability of wind noise and speech associated with the selected gaze direction (towards the driver or the co-pilot).
[0152] Under conditions where the probability of wind noise is high and the probability of speech is low, the wind noise PSD is approximately the same as the geometric mean of the two automatic PSDs of X1 and X2.
[0153] At step 220, the WNR gain function is determined. Two different gain calculations are designed and applied to wind noise reduction. The first one is from a variation of the spectral subtraction method, as follows:
[0154]
[0155] where Φ N (t, f) is the estimated wind noise power spectrum.
[0156] The minimum gain factor usually needs to be much smaller (e.g. -40B) to effectively remove very strong wind noise. To better preserve speech even in the presence of noise, G min In G min_min and G min_max and becomes the normalized phase variance by Function:
[0157]
[0158] Among them G min_min , G min_min Set to -40dB and -20dB respectively, indicating minimum and maximum G min . is based on the frequency range assigned for wind noise detection and the thresholds discussed elsewhere in this paper Compute the normalized phase variance.
[0159] Since large values of the phase of the cross-spectrum are a strong indicator of the presence of wind noise, a second gain function is also derived as:
[0160]
[0161] in is the same threshold used above (regarding probability determination) and is used to calculate the probability prob of wind noise in the specified frequency range wn .
[0162] One advantage of this gain function is that it will ensure deep attenuation on both channels for a time / frequency grid that may have wind noise present because the phase of its associated cross-spectrum is too large.
[0163] The final combined suppression rule for WNR operation is as follows:
[0164]
[0165] At step 222, wind noise reduction is performed and it is applied to both microphone channels, e.g. Figure 1 If the wind noise detector detects a frame as wind noise only, or as a frame of wind noise mixed with speech, WNR will be used and calculated as follows:
[0166] X i (f) = G WN (f)X i (f)+αCn(f),1≤i≤2,f1≤f≤f2
[0167] Where X i(f) represents the complex spectrum of virtual channel i, and Cn(f) is the comfort noise generated in advance. f1, f2 represent the frequency range within which WNR occurs.
[0168] Comfort noise injection into the attenuated signal can also be utilized in the method described herein. Because wind noise is usually deeply suppressed due to very small gain values (e.g., -40 dB). Really smooth comfort noise needs to be created in advance and injected into the point where the signal is severely attenuated. For fixed noise conditions, the comfort noise spectrum is created by a long-term smoothed version of the estimated instantaneous noise. However, since wind noise is strong, large, and may last for a long time, the comfort noise generated in a conventional manner has a noise gating effect and is still similar to wind noise, so it is not suitable for adding back to the signal where the wind noise is reduced.
[0169] For wind noise reduction applications, an alternative and more useful comfort noise was designed with the help of a minimum statistics approach. Minimum statistics, run on both channels, efficiently and effectively locates the minimum over the elapsed time for each frequency considered. It then assembles these unsynchronized minimum grids to formulate a "minimum" background noise for each channel.
[0170] The new comfort noise spectrum (envelope) is the average of the two minimum statistical sets from the two channels:
[0171]
[0172] where channe[i]→Smin[f] represents the minimum power spectrum value at frequency f associated with the i-th channel within the minimum statistical search time.
[0173] As with conventional comfort noise generation, the final comfort noise generation for WNR applications is to apply a minimum statistics derived spectral envelope to a piece of normalized white noise N w (f)
[0174] Cn(f)=CnEn(f)N w (f)
[0175] This new comfort noise generated may actually be suitable for use elsewhere, such as where it is used after echo suppression.
[0176] After removing wind noise from the signals, the signals can be converted back to the time domain and then used for other purposes. For example, the signals can be used to control the operation of other devices in the vehicle. In other examples, the signals can be transmitted to other users or devices. In still other examples, the signals can be processed for other purposes.
[0177] It should be understood that any device described herein (e.g., control circuits, controllers, receivers, transmitters, sensors, any presentation or display device or external device) can use computing devices to implement various functions and operations of these devices. In terms of hardware architecture, such computing devices may include, but are not limited to, processors, memories, and one or more input and / or output (I / O) device interfaces, which are coupled by local interface communication. The local interface may include, for example, but not limited to, one or more buses and / or other wired or wireless connections. The processor may be a hardware device for executing software, especially software stored in a memory. The processor may be a custom or commercially available processor, a central processing unit (CPU), an auxiliary processor in a plurality of processors associated with a computing device, a semiconductor-based microprocessor (in the form of a microchip or chipset), or generally any device for executing software instructions.
[0178] The memory devices described herein may include any one or a combination of the following: volatile memory elements (e.g., random access memory (RAM), such as dynamic RAM (DRAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), video RAM (VRAM), etc.) and / or non-volatile memory elements (e.g., read-only memory (ROM), hard drive, magnetic tape, CD-ROM, etc.). In addition, the memory may incorporate electronic, magnetic, optical and / or other types of storage media. The memory may also have a distributed architecture in which various components are remote from each other but accessible by the processor.
[0179] The software in any memory device described herein may include one or more separate programs, each program including an ordered list of executable instructions for implementing the functions described herein. When constructed as a source program, the program is translated by a compiler, assembler, interpreter, or the like, which may or may not be included in the memory.
[0180] It should be understood that any method described herein can be implemented at least in part as computer instructions stored on a computer medium (e.g., a computer memory as described above), and these instructions can be executed on a processing device (such as a microprocessor). However, these methods can be implemented as any combination of electronic hardware and / or software.
[0181] Preferred embodiments of the invention are described herein, including the best mode known to the inventors for carrying out the invention. It should be understood that the illustrated embodiments are exemplary only and should not be construed as limiting the scope of the invention.
Claims
1. A system for wind noise attenuation, the system comprising: A first microphone is used to obtain a first audio signal; a second microphone, for obtaining a second audio signal; wherein the first microphone and the second microphone are separated in space; a control circuit coupled to the first microphone and the second microphone, wherein the control circuit is configured to: continuously and simultaneously segmenting the first audio signal arriving at the first microphone and the second audio signal arriving at the second microphone into time periods, so that for each time period, the first audio signal arriving at the first microphone forms a first frame audio signal, and the second audio signal arriving at the second microphone forms a second frame audio signal; aligning the first frame audio signal and the second frame audio signal in time relative to the target speech source; Wherein, the time alignment of the first frame audio signal and the second frame audio signal is based on: a measurement result based on a static geometric characteristic, wherein the measurement result based on the static geometric characteristic is adjusted by a dynamic cross-correlation evaluation between signals received at two microphones at runtime; Performing Fourier transform on each of a first frame audio signal and a second frame audio signal that are time-aligned, performing Fourier transform on the first frame audio signal to generate a first spectrum, and performing Fourier transform on the second frame audio signal to generate a second spectrum, wherein each of the first spectrum and the second spectrum represents a spectrum of one of the two time-aligned microphone signals in each time period; calculating a phase difference between the first spectrum and the second spectrum at each of a plurality of frequencies based on a cross-correlation between the first spectrum and the second spectrum; determining, for each time period, a normalized variance of the phase difference within a defined frequency range, wherein the frequency range is calculated based on microphone geometry such that an error margin in the calculation of the normalized variance of the phase difference is minimized; formulating and evaluating the probability of speech presence and the probability of wind noise presence at each time segment based on the normalized variance of the spectral phase difference of the two time-aligned microphone signals; determining a category for each time period at each time period, wherein the category is one of: speech only, wind noise only, speech mixed with wind noise, or unknown, wherein decision logic is used to determine the category and the decision logic is based on a first function that combines individual and combined values of probability of speech presence and probability of wind noise presence, wherein the value of the first function is compared to a plurality of thresholds and a wind noise detection decision is made, wherein based on the determined category, a wind attenuation action is selectively triggered; When the action is to perform wind noise attenuation, a gain or attenuation function is calculated that is based on a normalized variance of the phase difference and the individual phase difference at each of a plurality of frequencies within a predetermined frequency range, and wherein wind noise attenuation is performed in the frequency domain by multiplying the gain or attenuation function with a magnitude of each of the first spectrum and the second spectrum to produce a first spectrum with wind noise removed and a second spectrum with wind noise removed; combining the first spectrum with wind noise removed and the second spectrum with wind noise removed to produce a combined spectrum; Constructing a time domain signal with wind noise removed by performing inverse FFT on the combined spectrum; An action is taken using the time domain signal, the action being one or more of: transmitting the time domain signal to an electronic device, controlling electronic equipment using the time domain signal, or interacting with electronic equipment using the time domain signal.
2. The system according to claim 1, wherein: The length of the time period is between 10 and 20 milliseconds.
3. The system according to claim 1, wherein: The target speech source includes speech from a person sitting in a vehicle seat.
4. The system according to claim 1, wherein: The probability that speech exists and the probability that wind noise exists both have values between 0 and 1.
5. The system according to claim 1, wherein: The determination of the category also utilizes a majority voting method that considers the current decision and the sequence of decisions in previous consecutive time periods.
6. The system according to claim 1, wherein: The probability of speech presence and the probability of wind noise presence provide a metric for evaluating the extent to which speech is present or wind noise is present at each time period.
7. The system according to claim 1, wherein: When it has been determined that there is wind noise only or wind noise mixed with speech, the wind noise attenuation action is triggered.
8. The system according to claim 1, wherein: The threshold is estimated offline by using the amount of speech and wind noise samples in the offline algorithm training phase.
9. The system according to claim 1, wherein: The system is at least partially disposed in a vehicle.
10. The system according to claim 1, wherein: The target speech source moves.
11. A method for wind noise attenuation, the method comprising: In the control circuit: continuously and simultaneously segmenting a first audio signal arriving at the first microphone and a second audio signal arriving at the second microphone into time periods, so that for each time period, the first audio signal arriving at the first microphone forms a first frame audio signal, and the second audio signal arriving at the second microphone forms a second frame audio signal; aligning the first frame audio signal and the second frame audio signal in time relative to the target speech source; Wherein, the time alignment of the first frame audio signal and the second frame audio signal is based on: a measurement result based on a static geometric characteristic, wherein the measurement result based on the static geometric characteristic is adjusted by a dynamic cross-correlation evaluation between signals received at two microphones at runtime; Performing Fourier transform on each of a first frame audio signal and a second frame audio signal that are time-aligned, performing Fourier transform on the first frame audio signal to generate a first spectrum, and performing Fourier transform on the second frame audio signal to generate a second spectrum, wherein each of the first spectrum and the second spectrum represents a spectrum of one of the two time-aligned microphone signals in each time period; calculating a phase difference between the first spectrum and the second spectrum at each of a plurality of frequencies based on a cross-correlation between the first spectrum and the second spectrum; determining, for each time period, a normalized variance of the phase difference within a defined frequency range, wherein the frequency range is calculated based on microphone geometry such that an error margin in the calculation of the normalized variance of the phase difference is minimized; formulating and evaluating the probability of speech presence and the probability of wind noise presence at each time segment based on the normalized variance of the spectral phase difference of the two time-aligned microphone signals; determining a category for each time period at each time period, wherein the category is one of: speech only, wind noise only, speech mixed with wind noise, or unknown, wherein decision logic is used to determine the category and the decision logic is based on a first function that combines individual and combined values of probability of speech presence and probability of wind noise presence, wherein the value of the first function is compared to a plurality of thresholds and a wind noise detection decision is made, wherein based on the determined category, a wind attenuation action is selectively triggered; When the action is to perform wind noise attenuation, a gain or attenuation function is calculated that is based on a normalized variance of the phase difference and the individual phase difference at each of a plurality of frequencies within a predetermined frequency range, and wherein wind noise attenuation is performed in the frequency domain by multiplying the gain or attenuation function with a magnitude of each of the first spectrum and the second spectrum to produce a first spectrum with wind noise removed and a second spectrum with wind noise removed; combining the first spectrum with wind noise removed and the second spectrum with wind noise removed to produce a combined spectrum; Constructing a time domain signal with wind noise removed by performing inverse FFT on the combined spectrum; An action is taken using the time domain signal, the action being one or more of: transmitting the time domain signal to an electronic device, controlling electronic equipment using the time domain signal, or interacting with electronic equipment using the time domain signal.
12. The method according to claim 11, wherein: The length of the time period is between 10 and 20 milliseconds.
13. The method according to claim 11, wherein: The target speech source includes speech from a person sitting in a vehicle seat. The method of claim 11 , wherein the probability that speech exists and the probability that wind noise exists each have a value between 0 and 1.
15. The method according to claim 11, wherein: The determination of the category also utilizes a majority voting method that considers the current decision and the sequence of decisions in previous consecutive time periods.
16. The method of claim 11, wherein the probability of speech presence and the probability of wind noise presence provide a metric for evaluating the extent to which speech is present or wind noise is present at each time period.
17. The method according to claim 11, wherein: When it has been determined that there is wind noise only or wind noise mixed with speech, the wind noise attenuation action is triggered.
18. The method according to claim 11, wherein: The threshold is estimated offline by using the amount of speech and wind noise samples in the offline algorithm training phase.
19. The method according to claim 11, wherein: The control circuit is at least partially arranged in the vehicle.
20. The method according to claim 11, wherein: The target speech source moves.
Citation Information
Patent Citations
System and method for wind detection and suppression
CN103348686A
Double-microphone noise inhibiting method and system
CN104751853A
Method and apparatus for wind noise detection
CN106664486A
Suppressing or reducing effects of wind turbulence
CN109074815A
Detecting presence of wind noise
CN109845289A
