A differential beamforming method, apparatus and storage medium

Abstract

This disclosure relates to a differential beamforming method, apparatus, and storage medium. The differential beamforming method includes: determining a target microphone among a plurality of microphones based on the direction of a line connecting them, wherein the target microphone is the microphone closest to the target direction among the plurality of microphones, the plurality of microphones are on a line, and the target direction is the foremost point along the line connecting the plurality of microphones; determining the wind noise intensity in the environment where the target microphone is located based on the current frame audio signal acquired from the target microphone; and forming a differential beam based on the wind noise intensity. This disclosure can effectively reduce wind noise and enhance the sound signal during differential beamforming.

Description

Technical Field

[0001] This disclosure relates to the field of signal processing technology, and in particular to a differential beamforming method, apparatus and storage medium. Background Technology

[0002] Speech enhancement technology based on differential microphone array (DMA) signal processing is widely used. The key issue is how to robustly reduce environmental noise without damaging the target sound signal.

[0003] In related technologies, among signal processing methods based on differential beamforming, speech enhancement based on first-order and second-order differential beamforming targets the environment of scattered noise fields. The noise judgment is too simplistic. Instead, the order is increased, that is, the number of sensors is increased at the physical level. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a differential beamforming method, apparatus and storage medium.

[0005] According to a first aspect of the present disclosure, a differential beamforming method is provided, the method comprising:

[0006] Based on the direction of the connection between multiple microphones, a target microphone is determined among the multiple microphones. The target microphone is the microphone closest to the target direction among the multiple microphones, wherein the multiple microphones are on a line and the target direction is the foremost part of the direction of the connection between the multiple microphones. Based on the audio signal of the current frame acquired by the target microphone, the wind noise intensity in the environment where the target microphone is located is determined. Differential beamforming is formed according to the wind noise intensity.

[0007] In one embodiment, determining the wind noise intensity in the environment where the target microphone is located based on the current frame audio signal acquired by the target microphone includes:

[0008] Determine the auto-power spectral density function of the current frame audio signal; determine the low-frequency energy value and the centroid value of the spectral sub-band corresponding to the auto-power spectral density function; and determine the wind noise intensity based on the low-frequency energy value and the centroid value of the spectral sub-band.

[0009] In one embodiment, determining the wind noise intensity based on the low-frequency energy value and the centroid value of the spectral sub-band includes:

[0010] If either the low-frequency energy value or the centroid value of the spectral sub-band is greater than the first wind noise threshold, then the wind noise intensity is determined to be the first wind noise intensity; if either the low-frequency energy value or the centroid value of the spectral sub-band is greater than the second wind noise threshold and less than the first wind noise threshold, then the wind noise intensity is determined to be the second wind noise intensity; if both the low-frequency energy value and the centroid value of the spectral sub-band are less than the second wind noise threshold, then the wind noise intensity is determined to be 0; wherein, the first wind noise threshold is greater than the second wind noise threshold, and the first wind noise intensity is greater than the second wind noise intensity; the first wind noise threshold and the second wind noise threshold each include a low-frequency energy threshold value and a threshold value corresponding to the centroid of the spectral sub-band.

[0011] In one embodiment, forming a differential beam based on the wind noise intensity includes:

[0012] If the wind noise intensity is 0, a first-order differential beamforming algorithm is used to form a differential beam; if the wind noise intensity is a second wind noise intensity, a first-order differential beamforming algorithm is used to form a differential beam, and the audio signal below a certain range is set to 0 before the short-time Fourier inverse transform; if the wind noise intensity is a first wind noise intensity, a delay superposition beamforming algorithm is used to form a differential beam in the first frequency band, and a differential beamforming algorithm is used to form a differential beam in the second frequency band, wherein the boundary point between the first frequency band and the second frequency band is estimated by the centroid value of the spectral sub-band.

[0013] In one embodiment, the method further includes:

[0014] The statistics are based on the low-frequency energy value and the centroid value of the spectral sub-band within a consecutive first frame after differential beamforming. If, within the first frame, there is a second consecutive frame where the low-frequency energy value is higher than the low-frequency energy threshold and the centroid value of the spectral sub-band is lower than the centroid value of the spectral sub-band, then a delayed superposition beamforming algorithm is used to form a differential beam in the first frequency band, and a differential beamforming algorithm is used to form a differential beam in the second frequency band. The boundary between the first and second frequency bands is estimated by the centroid value of the spectral sub-band. The low-frequency energy threshold and the centroid value of the spectral sub-band are determined based on the audio signal of the current frame before differential beamforming.

[0015] According to a second aspect of the present disclosure, a differential beamforming apparatus is provided, the apparatus comprising:

[0016] A determining unit is configured to determine a target microphone among multiple microphones based on the direction of a line connecting multiple microphones, wherein the target microphone is the microphone closest to the target direction among the multiple microphones, wherein the multiple microphones are on a line, and the target direction is the foremost point of the line connecting the multiple microphones; it is also configured to determine the wind noise intensity in the environment where the target microphone is located based on the current frame audio signal acquired by the target microphone; a beamforming unit is configured to form a differential beam according to the wind noise intensity.

[0017] In one embodiment, the determining unit determines the wind noise intensity in the environment where the target microphone is located by acquiring the current frame audio signal based on the target microphone in the following manner:

[0018] Determine the auto-power spectral density function of the current frame audio signal; determine the low-frequency energy value and the centroid value of the spectral sub-band corresponding to the auto-power spectral density function; and determine the wind noise intensity based on the low-frequency energy value and the centroid value of the spectral sub-band.

[0019] In one embodiment, the determining unit determines the wind noise intensity based on the low-frequency energy value and the centroid value of the spectral sub-band in the following manner:

[0020] If either the low-frequency energy value or the centroid value of the spectral sub-band is greater than the first wind noise threshold, then the wind noise intensity is determined to be the first wind noise intensity; if either the low-frequency energy value or the centroid value of the spectral sub-band is greater than the second wind noise threshold and less than the first wind noise threshold, then the wind noise intensity is determined to be the second wind noise intensity; if both the low-frequency energy value and the centroid value of the spectral sub-band are less than the second wind noise threshold, then the wind noise intensity is determined to be 0; wherein, the first wind noise threshold is greater than the second wind noise threshold, and the first wind noise intensity is greater than the second wind noise intensity; the first wind noise threshold and the second wind noise threshold each include a low-frequency energy threshold value and a threshold value corresponding to the centroid of the spectral sub-band.

[0021] In one embodiment, the beamforming unit forms a differential beam according to the wind noise intensity in the following manner:

[0022] If the wind noise intensity is 0, a first-order differential beamforming algorithm is used to form a differential beam; if the wind noise intensity is a second wind noise intensity, a first-order differential beamforming algorithm is used to form a differential beam, and the audio signal below a certain range is set to 0 before the short-time Fourier inverse transform; if the wind noise intensity is a first wind noise intensity, a delay superposition beamforming algorithm is used to form a differential beam in the first frequency band, and a differential beamforming algorithm is used to form a differential beam in the second frequency band, wherein the boundary point between the first frequency band and the second frequency band is estimated by the centroid value of the spectral sub-band.

[0023] In one embodiment, the device further includes:

[0024] A statistics unit is used to count the low-frequency energy value and the centroid value of the spectral sub-band within a consecutive first frame after differential beamforming; an adjustment unit is used to form a differential beam in a first frequency band and a differential beamforming algorithm in a second frequency band when, within the first frame, there is a consecutive second frame where the low-frequency energy value is higher than the low-frequency energy threshold and the centroid value of the spectral sub-band is lower than the centroid value of the spectral sub-band. The boundary point between the first and second frequency bands is estimated by the centroid value of the spectral sub-band; the low-frequency energy threshold and the centroid value of the spectral sub-band are determined based on the audio signal of the current frame before differential beamforming.

[0025] According to a third aspect of the present disclosure, a differential beamforming apparatus is provided, comprising:

[0026] Processor; memory used to store processor-executable instructions;

[0027] The processor is configured to execute the method described in the first aspect or any embodiment of the first aspect.

[0028] According to a fourth aspect of the present disclosure, a storage medium is provided, the storage medium storing instructions that, when executed by a processor of a terminal, enable the terminal to perform the method described in the first aspect or any embodiment of the first aspect.

[0029] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: a target microphone pointing in a target direction is determined among multiple microphones, and the wind noise intensity of the environment in which the target microphone is located is determined. Based on the wind noise intensity, a differential beam is formed. Thus, when differential beamforming is performed in an environment with wind noise, a differential beamforming method adapted to the wind noise intensity is used to reduce wind noise, enhance the sound signal, prevent the sound signal of the target microphone from being masked, and improve the system robustness of the sound signal processed by differential beamforming.

[0030] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0031] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0032] Figure 1 This is a schematic diagram of a scenario where a microphone array forms a differential beam, as provided in an embodiment of this disclosure.

[0033] Figure 2 This is a flowchart illustrating a dual-microphone differential beamforming method according to an exemplary embodiment.

[0034] Figure 3 This is a flowchart illustrating the determination of wind noise intensity based on frequency domain data according to an exemplary embodiment.

[0035] Figure 4 This is a flowchart illustrating, according to an exemplary embodiment, the determination of wind noise intensity based on low-frequency energy and the centroid of the spectral sub-band.

[0036] Figure 5 This is a flowchart illustrating the formation of differential beams based on wind noise intensity according to an exemplary embodiment.

[0037] Figure 6 This is a schematic diagram of first-order differential beamforming provided in an embodiment of this disclosure.

[0038] Figure 7 These are spectrograms of the processing results of different beamforming methods provided in the embodiments of this disclosure.

[0039] Figure 8 This is a flowchart illustrating a dual-microphone differential beamforming method according to an exemplary embodiment.

[0040] Figure 9 This is a line graph of the energy of each frame after processing by different beamforming methods provided in the embodiments of this disclosure.

[0041] Figure 10 This is a line graph of the SSC (Single Frame Layout) for each frame after processing by different beamforming methods provided in the embodiments of this disclosure.

[0042] Figure 11 This is a schematic diagram of an application scenario for dual-microphone differential beamforming provided in an embodiment of this disclosure.

[0043] Figure 12 This is a flowchart illustrating a dual-microphone differential beamforming method according to an exemplary embodiment.

[0044] Figure 13 This is a block diagram of a dual-microphone differential beamforming apparatus according to an exemplary embodiment.

[0045] Figure 14 This is a block diagram illustrating a dual-microphone differential beamforming apparatus according to an exemplary embodiment. Detailed Implementation

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure.

[0047] Voice enhancement technology based on microphone array signal processing is widely used in electronic devices such as mobile phones, headphones and smart TVs. The key issue is how to robustly reduce environmental noise without damaging the target sound signal.

[0048] Figure 1 This is a schematic diagram of a scenario where a microphone array forms a differential beam, as provided in an embodiment of this disclosure. Figure 1 As shown, a microphone array, combined with beamforming, can produce different responses to signals from different directions in space. This characteristic can be used to enhance the signal-to-noise ratio (SNR) of the array's output signal. When all noise originates from the spatial sound field, if the array's response in other directions is smaller than its response in the direction of the desired sound source, the SNR of the array's output signal relative to the SNR of the signal observed by a single microphone will naturally be improved. Electronic devices with dual microphones are often used in very complex acoustic environments. One troublesome factor is the various noises that the microphones pick up during calls, including background noise, wind noise, stationary interference, non-stationary interference, and echoes. Wind noise is a special type of interference. Furthermore, since wind noise is generated by turbulence in the airflow around the edge of the device, it results in rapidly changing non-stationary noise signals.

[0049] In view of this, embodiments of the present disclosure provide a differential beamforming method, in which differential beamforming is performed based on the wind noise intensity present in the environment to enhance the target sound signal.

[0050] The differential beamforming method provided in this disclosure is applicable to scenarios involving sound signal enhancement using microphone arrays formed by multiple microphones. For example, the differential beamforming method provided in this disclosure can be applied to electronic devices including microphone arrays. These electronic devices can be wired or wireless headsets, headphones, or hearing aids; short-dangling headsets (such as audio headsets for corporate call centers, industrial applications, and general mobile phones); in-ear headsets with input lines (wires, cables, or other connectors) mounted on or within eyeglass frames; near-eye (NTE) headset displays or headset computing devices; long-dangling headsets for very noisy environments (such as those for industrial, military, and aerospace applications); and gooseneck desktop microphones that can provide cinema or concert hall-quality acoustics. The multi-microphone array includes one or more groups of microphones, each group comprising two microphones.

[0051] The differential beamforming method provided in this disclosure involves multiple microphones connected by a line, the line having a direction. This direction includes a null direction and a target direction. The null direction is the furthest point (directly behind) of the line connecting the multiple microphones, and the target direction is the foremost point (directly in front) of the line connecting the multiple microphones.

[0052] A typical application scenario of this disclosure is a true wireless stereo (TWS) earphone scenario including a dual-microphone array. The dual-microphone array in the TWS earphone includes a call microphone and a feedforward microphone. This disclosure embodiment will be described below using a TWS earphone as an example. For instance, the line connecting the call microphone and the feedforward microphone in the TWS earphone points towards the wearer's mouth, and the direction of the mouth's position is the target direction. In other words, the target direction is the direction from which the feedforward microphone points to the call microphone. The microphone closest to the target direction is the call microphone.

[0053] In one embodiment of the differential beam velocity generation method provided in this disclosure, the target direction of multiple microphones in a microphone array is determined, and the wind noise intensity of the microphone pointed to by the target direction is determined. Based on the wind noise intensity, the sound signal of the microphone pointed to by the target direction is enhanced.

[0054] Figure 2 This is a flowchart illustrating a differential beamforming method according to an exemplary embodiment, such as... Figure 2 As shown, it includes the following steps.

[0055] In step S11, based on the direction of the connection between the multiple microphones, a target microphone is determined among the multiple microphones. The target microphone is the microphone that is closest to the target direction among the multiple microphones. The multiple microphones are on a line, and the target direction is the foremost direction of the connection between the multiple microphones.

[0056] In step S12, the wind noise intensity in the environment where the target microphone is located is determined based on the audio signal of the current frame acquired by the target microphone.

[0057] In step S13, a differential beam is formed based on the wind noise intensity.

[0058] In this embodiment of the disclosure, the wind noise intensity of the environment in which the target microphone is located is detected based on the current frame audio signal collected by the target microphone. Differential beamforming is formed based on the wind noise intensity. Therefore, when performing differential beamforming in an environment with wind noise, a differential beamforming method adapted to the wind noise intensity is used to reduce wind noise, enhance the sound signal, prevent the sound signal of the target microphone from being masked, and improve the system robustness of the sound signal processed by differential beamforming.

[0059] In this embodiment of the disclosure, a target microphone is determined among the multiple microphones based on the direction of the line connecting them. The target microphone is the microphone closest to the target direction among the multiple microphones, wherein the multiple microphones are on a single line, and the target direction is the foremost point along the line connecting the multiple microphones. Two microphones are used for first-order differential beamforming, and their target direction is the direction of the line connecting the feedforward microphone and the call microphone.

[0060] In this embodiment of the disclosure, the target direction is directly in front of the line connecting the two microphones, that is, the direction from which the target voice signal originates, and the zero point direction is directly behind the line connecting the two microphones.

[0061] After the two microphones acquire the input signal of the target sound source, the input signals acquired by the two microphones are subjected to Fourier transform, transforming the input signals of each microphone from time domain signals to frequency domain signals, which are then used as the signals to be processed for differential beamforming.

[0062] In this embodiment, a TWS earphone is used as an example for explanation. The TWS earphone contains a feedforward microphone and a call microphone, receiving two audio signals. The audio signals acquired by the feedforward microphone and the call microphone are f(n) and t(n), respectively. The sampling rate f of the audio signal processing system is... sThe sampling rate is typically 16kHz, 44.1kHz, or 48kHz. Due to hardware and computing power limitations, this disclosure uses a 16kHz sampling rate. Since the sound signal approximately satisfies short-time stationarity characteristics between 10-40ms, its second-order statistics and other information can be used. Therefore, a Short-Time Fourier Transform (STFT) is performed on the received signal, namely, frame windowing and fast Fourier transform (FFT). This disclosure selects each frame as 32ms, so the frame length L = 512; the window function w(n) is a Hanning window, and its length is the same as the frame length; the frame shift is 50% of the frame length, i.e., inc = 256. The two-channel signals sf(n,m) and st(n,m) after frame windowing can be obtained as formulas (1) and (2):

[0063] sf(n,m)=f((m-1)*inc+n)*w(n),0≤n≤(L-1) (1)

[0064] st(n,m)=t((m-1)*inc+n)*w(n),0≤n≤(L-1) (2)

[0065] Where m represents the frame index and n represents the data point index of the m-th frame of audio data. Then, performing FFT on sf(n,m) and st(n,m) respectively yields the spectral data F(k,m) and T(k,m) of the feedforward microphone and the call microphone, where k represents the frequency index.

[0066] In this embodiment of the present disclosure, after performing the above STFT preprocessing on the sound signals collected by multiple microphones, two-channel sound signals are obtained after frame-segmentation and windowing. The auto-power spectral density function corresponding to the two-channel sound signals can be determined. Subsequently, the wind noise intensity is determined based on the auto-power spectral density function, and differential beamforming is performed.

[0067] This embodiment still uses TWS earphones as an example for explanation. F(k,m) and T(k,m) are calculated based on the above preprocessing method. The autopower spectral density φ of F(k,m) and T(k,m) is then calculated. ff (k,m) and φ tt (k,m).

[0068] This disclosure employs an autoregressive time smoothing method to calculate the autopower spectral density φ of F(k,m) and T(k,m). ff (k,m) and φ tt (k,m). In one example, the autopower spectral density φ of F(k,m) and T(k,m) is calculated using formulas (3) and (4). ff (k,m) and φ tt (k,m):

[0069] φ ff (k,m)=αφ ff (k,m-1)+(1-α)F(k,m)F * (k,m) (3)

[0070] φ tt (k,m)=αφ tt (k,m-1)+(1-α)T(k,m)T * (k,m) (4)

[0071] in,(·) * The expression represents complex conjugation, and α is a smoothing factor (α<1). In this disclosure, α<0.7 is used.

[0072] In this embodiment of the disclosure, since the target direction is from the feedforward microphone to the call microphone, and due to limitations in the device's computing power, single-channel wind noise detection is used. In this embodiment of the disclosure, when enhancing the audio signal of the differential beam target microphone, the auto-power spectral density function of the current frame audio signal of the target microphone can be determined, and based on the auto-power spectral density function of the current frame audio signal of the target microphone, the wind noise intensity of the environment where the target microphone is located can be determined.

[0073] The following embodiments of this disclosure describe the process of determining the wind noise intensity of the environment in which the target microphone is located.

[0074] In one embodiment of this disclosure, spectral subband centroids (SSC) can be used for single-channel wind noise detection. Here, SSC can be understood as the average frequency value of the sound signal and wind noise.

[0075] In this embodiment of the disclosure, the low-frequency energy of the current frame is combined with SSC for joint judgment, thereby improving the accuracy of wind noise detection.

[0076] Figure 3 This is a flowchart illustrating a method for determining the wind noise intensity of the environment in which a target microphone is located, according to an exemplary embodiment. Figure 3 As shown, it includes the following steps.

[0077] In step S21, the autopower spectral density function of the current frame audio signal of the target microphone is determined.

[0078] In one example, a TWS earphone is used as an example for illustration. The target microphone in the TWS earphone is the call microphone. Based on the above embodiment, the self-power spectral density function φ of the call microphone can be determined. tt (k,m).

[0079] In step S22, the low-frequency energy value and SSC corresponding to the self-power spectral density function are determined.

[0080] Taking the call microphone as an example, the SSC can be determined as shown in formula (5):

[0081]

[0082] Where k1 and k2 represent the lower and upper limits of the statistical frequency index, respectively. This value is related to the number of FFT points and is generally not normalized.

[0083] Among them, the low-frequency energy value can be understood as the energy of wind noise. Since the energy of wind noise is mainly distributed in the low frequency range of 0-1kHz, it is called low-frequency energy.

[0084] In step S23, the wind noise intensity is determined based on the low-frequency energy value and the centroid value of the spectral sub-band.

[0085] In this embodiment, the energy of wind noise is mainly distributed in the low frequency range, especially the 0-1kHz range, while the SSC (Special Score Concentration) of wind noise is concentrated below 100Hz. Therefore, combining the low-frequency energy and SSC to determine the magnitude of wind noise improves the accuracy of wind noise detection. Based on extensive experimental results, this application derives small and large wind noise thresholds. If either the low-frequency energy or the SSC of the current frame exceeds the large wind noise threshold, the current frame is considered a large wind noise scene; if either the low-frequency energy or the SSC of the current frame exceeds the small wind noise threshold, the current frame is considered a small wind noise scene; only if both the low-frequency energy and the SSC of the current frame are less than the small wind noise threshold is the current frame considered a wind-noise scene. By setting different wind noise thresholds, the audio signal acquired by the microphone can be distinguished into different wind noise scenes, and different beamforming methods can be selected according to different wind noise scenes, thus improving the stability of the system.

[0086] The following embodiments of this disclosure describe the process of determining the wind noise intensity of the target microphone's environment based on low-frequency energy values ​​and spectral sub-band centroid values.

[0087] In this embodiment of the disclosure, a wind noise threshold is set for determining the wind noise intensity. The wind noise threshold includes a low-frequency energy threshold and a threshold corresponding to the centroid of the spectral sub-band. When determining the wind noise intensity of the environment where the target microphone is located based on the low-frequency energy value and the centroid of the spectral sub-band, the wind noise intensity can be determined based on the low-frequency energy value, the centroid of the spectral sub-band, and the wind noise threshold.

[0088] In this embodiment of the disclosure, the current wind noise scene can be determined based on the wind noise intensity as a no-wind-noise scene (wind noise intensity is 0), a high-wind-noise scene (wind noise intensity is greater than the set wind noise intensity (hereinafter referred to as the first wind noise intensity)), and a low-wind-noise scene (wind noise intensity is lower than the set wind noise intensity (hereinafter referred to as the second wind noise intensity)).

[0089] In this embodiment, a first wind noise threshold is set for a first wind noise intensity, and a second wind noise threshold is set for a second wind noise intensity. Each of the first and second wind noise thresholds includes a low-frequency energy threshold value and a threshold value corresponding to the centroid of the spectral sub-band. For example, the small wind noise thresholds corresponding to the second wind noise intensity are φ1 and SSC1, and the large wind noise thresholds corresponding to the first wind noise intensity are φ2 and SSC2.

[0090] Figure 4 This is a flowchart illustrating the determination of wind noise intensity based on low-frequency energy and the centroid of the spectral sub-band, according to an exemplary embodiment. Figure 4 As shown, it includes the following steps.

[0091] In step S31, the low-frequency energy value, the centroid value of the spectral sub-band, the first wind noise threshold, and the second wind noise threshold are determined.

[0092] In this embodiment, the energy of wind noise is mainly distributed in the low frequency range, so the low frequency energy value of the audio signal is extracted for judgment. By judging the relationship between the low frequency energy value and the centroid value of the spectral sub-band and the first wind noise threshold and the second wind noise threshold, it is determined whether the current frame of the audio signal is a scene with no wind noise, a scene with low wind noise, or a scene with high wind noise.

[0093] In step S32a, if either the low-frequency energy value or the centroid value of the spectral sub-band is greater than the first wind noise threshold, then the wind noise intensity is determined to be the first wind noise intensity.

[0094] In this embodiment of the disclosure, if either the low-frequency energy or the SSC of the current frame is greater than the high wind noise threshold, then the current frame is considered to be a high wind noise scene.

[0095] In step S32b, if either the low-frequency energy value or the centroid value of the spectral sub-band is greater than the second wind noise threshold and less than the first wind noise threshold, then the wind noise intensity is determined to be the second wind noise intensity.

[0096] In this embodiment of the disclosure, if either the low-frequency energy or the SSC of the current frame is greater than the low wind noise threshold, then the current frame is considered to be a low wind noise scene.

[0097] In step S32c, if both the low-frequency energy value and the centroid value of the spectral sub-band are less than the second wind noise threshold, then the wind noise intensity is determined to be 0.

[0098] In this embodiment of the disclosure, the current frame is considered to be a wind-noise-free scene only if both the low-frequency energy and the SSC of the current frame are less than the low wind noise threshold.

[0099] Based on numerous experimental results, this disclosure provides that the small wind noise thresholds corresponding to the second wind noise intensity are φ1 and SSC1, and the large wind noise thresholds corresponding to the first wind noise intensity are φ2 and SSC2.

[0100] If either the low-frequency energy or the SSC of the current frame is greater than the high wind noise threshold, the current frame is considered a high wind noise scene; if either the low-frequency energy or the SSC of the current frame is greater than the low wind noise threshold, the current frame is considered a low wind noise scene; only if both the low-frequency energy and the SSC of the current frame are less than the low wind noise threshold, the current frame is considered a windless scene.

[0101] After determining the wind noise intensity in this embodiment, different differential beamforming algorithms can be used to perform differential beamforming based on different wind noise intensities.

[0102] Figure 5 This is a flowchart illustrating the formation of differential beams based on wind noise intensity according to an exemplary embodiment, such as... Figure 5 As shown, it includes the following steps.

[0103] In step S41, the wind noise intensity is determined.

[0104] In step S42a, if the wind noise intensity is 0, a first-order differential beamforming algorithm is used to form a differential beam.

[0105] In this embodiment of the disclosure, if the wind noise detection result is no wind noise, then a standard first-order DMA is used for beamforming. Figure 6 This is a schematic diagram of a first-order differential beamforming provided in an embodiment of this disclosure. For example... Figure 6 As shown, the observation signal from microphone #2 is first delayed (to control the beam shape and directivity factor), then the observation signal from microphone #1 is subtracted from the delayed signal from microphone #2, and finally the result of the subtraction is passed through an equalizer (to compensate for the high-pass effect caused by the differential beamforming) to obtain the output of the first-order differential beamforming.

[0106] In step S42b, if the wind noise intensity is the second wind noise intensity, a first-order differential beamforming algorithm is used to form a differential beam, and the audio signal below a certain range is set to 0 before the short-time Fourier inverse transform.

[0107] In this embodiment of the disclosure, if the wind noise is low, a standard first-order DMA is used for beamforming, and signals below 100-150Hz are set to 0 before ISTFT. This is because the DMA has a weak amplification effect on low-frequency noise at this time, and setting signals below 100-150Hz to zero has almost no effect on the audio signal, and this can effectively avoid time-domain waveform clipping.

[0108] In step S42c, if the wind noise intensity is the first wind noise intensity, a differential beam is formed using a delay superposition beamforming algorithm in the first frequency band, and a differential beam is formed using a differential beamforming algorithm in the second frequency band. The boundary between the first and second frequency bands is estimated by the centroid value of the spectral sub-band.

[0109] In this embodiment, for high wind noise, delay-and-sum (DAS) beamforming is used in the low-frequency band, and DMA beamforming is used in the high-frequency band. Different beamforming methods are used for different wind noise intensities. A more conservative beamforming method is used in high wind noise scenarios, while a more aggressive beamforming method is used in low wind noise scenarios, resulting in better wind noise suppression.

[0110] In this embodiment of the disclosure, the weight coefficients of differential beamforming need to be solved before it can be formed. There are many ways to solve the weight coefficients of differential beamforming, the most classic of which is the null information design method. For a first-order DMA of two microphones: 1. The target direction must be the end-fire direction, i.e., θ g = 0° or 180°, generally defaulting to 0° in the spatial rectangular coordinate system. Signals in this direction are distortion-free, and the normalized beam response is 1; 2. The minimum value direction is θ. null =cos -1 α 1,1 (-1≤α 1,1 <1), the beam response in this direction is β 1,1 Generally, β is taken. 1,1 =0. Then, the system of linear equations is expressed as formula (6):

[0111]

[0112] in,(·) H This indicates the complex conjugate transpose. Let be the steering vector of the dual-microphone linear array, and represent the frequency point. The weight coefficients are obtained by inversion and equivalent infinitesimal approximation, as shown in formula (7):

[0113]

[0114] Where C represents a constant term, ω k =2πf k τ1=τ0(1-α1,1 ), τ2=τ0α 1,1 τ0=d / c. By using the weight vector h(k) and the dual-microphone array received signal vector t(k,m) for filtering, the enhanced signal out(k,m) can be obtained, as shown in formula (8).

[0115] out(k,m)=h H (k)t(k,m) (8)

[0116] The enhanced signal out(k,m) still needs to be transformed by ISTFT to obtain the time-domain output result.

[0117] In the application scenario disclosed herein, the target direction is directly in front of the line connecting the two microphones, and the null direction is directly behind the line connecting the two microphones. At this time, the beam pattern of the first-order DMA is cardioid, with a relatively large white noise gain, which meets the requirements of the scenario. From the above analysis, it can be seen that α... 1,1 =-1, β 1,1 =0, substituting it into formula (7) yields h DMA (k). Furthermore, the most commonly used weight vector for delay-sum beamforming is h. DAS (k)=d(k).

[0118] Furthermore, in this embodiment of the disclosure, the boundary point between the first frequency band and the second frequency band is estimated by the centroid value of the spectral sub-band.

[0119] In one example, the boundary point k between the first and second frequency bands DAS (m) is estimated from SSC using formula (9):

[0120]

[0121] In this embodiment, if the wind noise detection result is no wind noise, standard first-order DMA is used for beamforming. This aggressive beamforming method is chosen because DMA has many advantages: 1. The array response does not change with frequency; 2. A larger array directivity can be obtained with fewer array elements, meaning the array has high directivity and can easily identify and suppress interference and noise signals while increasing useful speech signals; 3. An Nth-order DMA has at most N zeros, which can suppress interference signals in the direction of the zeros. If the wind noise is low, standard first-order DMA is used for beamforming, and signals below 100-150Hz are set to 0 before ISTFT. This is a more aggressive beamforming method, using standard first-order DMA in the high-frequency speech band. Since setting signals below 100-150Hz to zero has almost no effect on the speech signal, signals in the 100-150Hz wind noise band are set to 0. If there is significant wind noise, DAS beamforming is used in the low-frequency band and DMA beamforming is used in the high-frequency band. In this case, a conservative beamforming method is chosen. Using different beamforming methods can effectively enhance the voice signal and suppress wind noise.

[0122] Figure 7 These are spectrograms of the processing results of different beamforming methods provided in the embodiments of this disclosure, such as... Figure 7 As shown in Figures (a)-(d), the recorded signal, the DMA processing result, the DAS combined with DMA processing result (DAS for frequencies below 2kHz), and the DMA processing result with the signal below 100Hz set to zero are respectively. The recorded signal is the unprocessed real sound. 1. Figure (b) shows that DMA processing of wind-noise signals abnormally amplifies low frequencies. Comparing with Figure (d), it can be seen that setting the frequencies below 100Hz to zero can improve this shortcoming without affecting the sound signal. Therefore, when wind noise detection is in a low-wind-noise scenario, using standard first-order DMA for beamforming and setting the signals below 100-150Hz to 0 before ISTFT yields better results. 2. Figure (c) shows that if DAS beamforming is used in the low-frequency range, the noise reduction performance in the low-frequency range is very poor, but it does not damage the sound signal, making it suitable for use in scenarios with high wind noise.

[0123] In this embodiment of the disclosure, in order to ensure that a beamforming method adapted to the current scenario is used in different scenarios, feedback judgment of the sound signal after beamforming processing can be performed based on the low-frequency energy value and the centroid value of the spectral sub-band within the first consecutive frame after the differential beam is formed.

[0124] Figure 8 This is a flowchart illustrating a differential beamforming method according to an exemplary embodiment, such as... Figure 8 As shown, it includes the following steps.

[0125] In step S51, the low-frequency energy value and the centroid value of the spectral sub-band are statistically analyzed based on the first consecutive frames after the differential beam is formed.

[0126] In this embodiment of the disclosure, Figure 9 This is a line graph of the energy of each frame after processing by different beamforming methods provided in the embodiments of this disclosure. Figure 10 This is a line graph of the SSC (Single Frame Layout) for each frame after processing by different beamforming methods provided in the embodiments of this disclosure, such as... Figure 9 and 10 As shown, the energy and SSC results of each frame after processing by different methods can be seen, where Ori, DMA1, DMA+DAS, and DMA2 represent respectively Figure 7 The results (a)-(d) show that energy statistics only calculate the low-frequency band results of 10-1000Hz. We can obtain: 1. The low-frequency energy, from smallest to largest, is Ori, DMA+DAS, DMA2, and DMA1. 2. The SSC, from largest to smallest, is Ori, DMA+DAS, DMA2, and DMA1. The consistency across the four processing methods indicates that the above results meet the expected processing requirements. Specifically, wind noise has a smaller SSC and larger low-frequency energy compared to the sound signal. After beamforming processing, the SSC increases and the low-frequency energy decreases, indicating that the processing results meet expectations. Conversely, if the opposite is true, the processing results do not meet expectations. Therefore, this disclosure adds a feedback path to judge the processing results.

[0127] In step S52, if there is a low-frequency energy value higher than the low-frequency energy threshold for two consecutive frames within the first frame, and the centroid value of the spectral sub-band is lower than the centroid value of the spectral sub-band, then a differential beam is formed in the first frequency band using a delay superposition beamforming algorithm, and a differential beam is formed in the second frequency band using a differential beamforming algorithm.

[0128] The boundary between the first and second frequency bands is estimated by the centroid value of the spectral sub-band.

[0129] In this embodiment of the disclosure, if the beamforming processing result meets expectations, no further adjustments are made. However, in actual use, the desired result is generally not achieved. Figure 7 To address the feedback of the processing results, this disclosure adds a feedback path for judging the processing results. Specifically, it continuously counts the low-frequency energy and SSC results for 15 frames (approximately 0.25 seconds). If the low-frequency energy of 7 consecutive frames or the average result is higher than the result before processing and the SSC is lower than the result before processing, then the beamforming method used in the previous frame is considered to have caused unacceptable low-frequency amplification, and the beamforming method of DMA+DAS should be used in the next frame. A small number of consecutive frames will lead to inaccurate judgment, and the delay cannot be too long; therefore, 15 consecutive frames are selected for counting.

[0130] The differential beamforming method provided in this disclosure improves the robustness and practicality of the system in the face of abnormal amplification of low-frequency environmental noise. Furthermore, the differential beamforming method provided in this disclosure adds a feedback branch to ensure the detection of wind noise, and introduces different beamforming and signal processing methods to suppress noise.

[0131] The differential beamforming method provided in this disclosure enhances the sound signal by using different differential beamforming methods based on logical judgments of the current low-frequency energy and SSC, thereby improving the system's robustness. Furthermore, an additional feedback judgment branch is added to ensure that the appropriate beamforming method is used in different scenarios.

[0132] The following embodiments of this disclosure use a True Wireless Stereo (TWS) headset with a dual-microphone array as an example to illustrate the differential beamforming method involved in the above embodiments of this disclosure.

[0133] Figure 11 This is a schematic diagram illustrating an application scenario of dual-microphone differential beamforming provided in an embodiment of this disclosure, such as... Figure 2 As shown, the following explanation is based on a TWS earphone scenario including a dual-microphone array. During normal wear, the connection between the TWS earphone's call microphone and the feedforward microphone is pointed towards the wearer's mouth. First, a single-channel wind noise detection method is used to determine whether there is wind noise and its magnitude. Then, an appropriate beamforming method is selected. Next, the beamforming results are statistically analyzed to assist in the selection of the beamforming method, thereby enhancing the robustness of the system.

[0134] Figure 12 This is a flowchart illustrating a dual-microphone differential beamforming method according to an exemplary embodiment, such as... Figure 12 As shown, the process includes the following steps: (1) Data preprocessing. Input audio data from the feedforward microphone and the call microphone, perform STFT (frame windowing and FFT) to obtain the corresponding frequency domain signal. (2) Wind noise detection and judgment. Use single-channel wind noise detection coefficients to determine whether there is wind noise and the magnitude of the wind noise in the current frame. (3) Beamforming method selection. Based on the results of step (2), use different DMA methods or methods combined with delay summation beamforming. (4) Statistical result judgment. Detect the results of step (3), compare them with the corresponding coefficients in step (2), and judge whether the processing results meet expectations and whether it is necessary to change the beamforming method.

[0135] This disclosure identifies a target microphone pointing in a specific direction among multiple microphones, acquires the current frame audio signal based on the target microphone, determines the wind noise intensity of the environment in which the target microphone is located, forms different differential beams according to the magnitude of the wind noise intensity, judges the processing results, and then reselects the beamforming method. This method, by using different beamforming methods, can effectively enhance the voice signal, suppress wind noise, and improve system stability.

[0136] Based on the same concept, embodiments of this disclosure also provide a differential beamforming apparatus.

[0137] It is understood that the dual-microphone differential beamforming apparatus provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0138] Figure 13 This is a block diagram 100 of a differential beamforming apparatus according to an exemplary embodiment. (Refer to...) Figure 13 The device includes a determination unit 101, a beamforming unit 102, a statistics unit 103, and an adjustment unit 104.

[0139] The determining unit 101 is used to determine the target microphone among multiple microphones, wherein the target microphone is the microphone that the target direction is pointing to among the multiple microphones; it is also used to determine the wind noise intensity of the environment where the target microphone is located by acquiring the audio signal of the current frame based on the target microphone.

[0140] Beamforming unit 102 is used to form differential beams based on wind noise intensity.

[0141] The statistical unit 103 is used to statistically analyze the low-frequency energy value and the centroid value of the spectral sub-band within the first consecutive frame after the differential beam is formed.

[0142] The adjustment unit 104 is configured to form a differential beam using a delay superposition beamforming algorithm in the first frequency band and a differential beamforming algorithm in the second frequency band if, within the first frame, there is a low-frequency energy value higher than the low-frequency energy threshold and the centroid value of the spectral sub-band is lower than the centroid value of the spectral sub-band. The boundary point between the first and second frequency bands is estimated by the centroid value of the spectral sub-band, and the low-frequency energy threshold and the centroid value of the spectral sub-band are determined based on the audio signal of the current frame before the differential beam is formed.

[0143] In one embodiment, the determining unit 101 determines the wind noise intensity of the environment where the target microphone is located by acquiring the current frame audio signal based on the target microphone in the following manner:

[0144] Determine the auto-power spectral density function of the current frame audio signal; determine the low-frequency energy value and the centroid value of the spectral sub-band corresponding to the auto-power spectral density function; based on the low-frequency energy value and the centroid value of the spectral sub-band, determine the wind noise intensity of the environment where the target microphone is located.

[0145] In one embodiment, the determining unit 101 determines the wind noise intensity of the target microphone's environment based on low-frequency energy values ​​and spectral sub-band centroid values:

[0146] If either the low-frequency energy value or the centroid value of the spectral sub-band is greater than the first wind noise threshold, then the wind noise intensity is determined to be the first wind noise intensity; if either the low-frequency energy value or the centroid value of the spectral sub-band is greater than the second wind noise threshold and less than the first wind noise threshold, then the wind noise intensity is determined to be the second wind noise intensity; if both the low-frequency energy value and the centroid value of the spectral sub-band are less than the second wind noise threshold, then the wind noise intensity is determined to be 0; wherein, the first wind noise threshold is greater than the second wind noise threshold, and the first wind noise intensity is greater than the second wind noise intensity; the first wind noise threshold and the second wind noise threshold each include the low-frequency energy threshold value and the threshold value corresponding to the centroid of the spectral sub-band.

[0147] In one embodiment, the beamforming unit 102 forms a differential beam based on the wind noise intensity in the following manner:

[0148] If the wind noise intensity is 0, a first-order differential beamforming algorithm is used to form a differential beam. If the wind noise intensity is the second wind noise intensity, a first-order differential beamforming algorithm is used to form a differential beam, and the audio signal below a certain range is set to 0 before the short-time Fourier inverse transform. If the wind noise intensity is the first wind noise intensity, a delay superposition beamforming algorithm is used to form a differential beam in the first frequency band, and a differential beamforming algorithm is used to form a differential beam in the second frequency band. The boundary between the first and second frequency bands is estimated by the centroid value of the spectral sub-band.

[0149] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0150] Figure 14 This is a block diagram illustrating an apparatus 200 for a differential beamforming method according to an exemplary embodiment. For example, apparatus 200 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0151] Reference Figure 14 The device 200 may include one or more of the following components: processing component 202, memory 204, power component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.

[0152] Processing component 202 typically controls the overall operation of device 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.

[0153] Memory 204 is configured to store various types of data to support the operation of device 200. Examples of such data include instructions for any application or method operating on device 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0154] The power supply component 206 provides power to the various components of the device 200. The power supply component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 200.

[0155] Multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0156] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when device 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.

[0157] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0158] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of device 200. For example, sensor assembly 214 may detect the on / off state of device 200, the relative positioning of components such as the display and keypad of device 200, changes in the position of device 200 or a component of device 200, the presence or absence of user contact with device 200, the orientation or acceleration / deceleration of device 200, and temperature changes of device 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0159] Communication component 216 is configured to facilitate wired or wireless communication between device 200 and other devices. Device 200 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 216 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0160] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0161] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by a processor 220 of the device 200 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0162] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0163] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0164] It can be further understood that, unless otherwise specified, "connection" includes both direct connections where no other components exist between the two parties and indirect connections where other components exist between them.

[0165] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0166] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0167] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A differential beamforming method, characterized by, The method comprises: determining a target microphone among a plurality of microphones based on the connection direction of the plurality of microphones, the target microphone being the closest microphone to a target direction among the plurality of microphones, wherein the plurality of microphones are connected in a line, and the target direction is the most forward direction of the connection direction of the plurality of microphones; collecting a current frame audio signal based on the target microphone, and determining the wind noise intensity in the environment where the target microphone is located; forming a differential beam based on the wind noise intensity; wherein the forming of the differential beam based on the wind noise intensity comprises: if the wind noise intensity is 0, using a first-order differential beam forming algorithm to form the differential beam, and the wind noise intensity of 0 corresponds to a wind noise-free scene; if the wind noise intensity is a second wind noise intensity, using the first-order differential beam forming algorithm to form the differential beam, and setting the audio signal below a certain range to 0 before inverse short-time Fourier transform, and the second wind noise intensity is less than a set wind noise intensity; if the wind noise intensity is a first wind noise intensity, using a delay and superposition beam forming algorithm to form the differential beam in a first frequency band, and using a differential beam forming algorithm to form the differential beam in a second frequency band, wherein the demarcation point between the first frequency band and the second frequency band is estimated by a spectral sub-band centroid value, and the first wind noise intensity is greater than the set wind noise intensity.

2. The method of claim 1, wherein, The method further comprises: determining a self-power spectral density function of the current frame audio signal; determining a low-frequency energy value and a spectral sub-band centroid value corresponding to the self-power spectral density function; determining the wind noise intensity based on the low-frequency energy value and the spectral sub-band centroid value.

3. The method of claim 2, wherein, The determining of the wind noise intensity based on the low-frequency energy value and the spectral sub-band centroid value comprises: if any one of the low-frequency energy value and the spectral sub-band centroid value is greater than a first wind noise threshold, determining the wind noise intensity as a first wind noise intensity; if any one of the low-frequency energy value and the spectral sub-band centroid value is greater than a second wind noise threshold and less than the first wind noise threshold, determining the wind noise intensity as a second wind noise intensity; if both the low-frequency energy value and the spectral sub-band centroid value are less than the second wind noise threshold, determining the wind noise intensity as 0; wherein the first wind noise threshold is greater than the second wind noise threshold, and the first wind noise intensity is greater than the second wind noise intensity; each of the first wind noise threshold and the second wind noise threshold comprises a low-frequency energy threshold value and a spectral sub-band centroid corresponding threshold value.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: counting the low-frequency energy value and the spectral sub-band centroid value within a first number of continuous frames after the forming of the differential beam; if the low-frequency energy value of a second number of continuous frames within the first number of frames is higher than a low-frequency energy threshold value, and the spectral sub-band centroid value is lower than a spectral sub-band centroid threshold value, using a delay and superposition beam forming algorithm to form the differential beam in a first frequency band, and using a differential beam forming algorithm to form the differential beam in a second frequency band, wherein the demarcation point between the first frequency band and the second frequency band is estimated by the spectral sub-band centroid value. The low-frequency energy threshold and the spectral sub-band centroid threshold are determined based on the current frame audio signal before forming a differential beam.

5. A differential beam forming device, characterized by Comprise: A determination unit is configured to determine a target microphone in a plurality of microphones based on a connection direction of the plurality of microphones, the target microphone being the closest microphone to a target direction in the plurality of microphones, wherein the plurality of microphones are connected in a line, and the target direction is the most forward direction of the connection direction of the plurality of microphones; and determine wind noise intensity in an environment where the target microphone is located based on a current frame audio signal collected by the target microphone. A beam forming unit is configured to form a differential beam according to the wind noise intensity. The beam forming unit forms a differential beam according to the wind noise intensity in the following manner: If the wind noise intensity is 0, a first-order differential beam forming algorithm is used to form a differential beam, and wind noise intensity of 0 corresponds to a wind noise-free scene; If the wind noise intensity is a second wind noise intensity, a first-order differential beam forming algorithm is used to form a differential beam, and the audio signal below a certain range is set to 0 before inverse short-time Fourier transform; If the wind noise intensity is a first wind noise intensity, a delay and superposition beam forming algorithm is used to form a differential beam in a first frequency band, and a differential beam forming algorithm is used to form a differential beam in a second frequency band, wherein the demarcation point between the first frequency band and the second frequency band is estimated by a spectral sub-band centroid value.

6. The apparatus of claim 5, wherein, The determination unit determines the wind noise intensity in the environment where the target microphone is located based on the current frame audio signal collected by the target microphone in the following manner: Determine the auto-power spectral density function of the current frame audio signal; Determine the low-frequency energy value and the spectral sub-band centroid value corresponding to the auto-power spectral density function; Determine the wind noise intensity based on the low-frequency energy value and the spectral sub-band centroid value.

7. The apparatus of claim 6, wherein, The determination unit determines the wind noise intensity based on the low-frequency energy value and the spectral sub-band centroid value in the following manner: If any one of the low-frequency energy value and the spectral sub-band centroid value is greater than a first wind noise threshold, the wind noise intensity is determined to be a first wind noise intensity; If any one of the low-frequency energy value and the spectral sub-band centroid value is greater than a second wind noise threshold and less than the first wind noise threshold, the wind noise intensity is determined to be a second wind noise intensity; If both the low-frequency energy value and the spectral sub-band centroid value are less than the second wind noise threshold, the wind noise intensity is determined to be 0; Wherein, the first wind noise threshold is greater than the second wind noise threshold, and the first wind noise intensity is greater than the second wind noise intensity; The first wind noise threshold and the second wind noise threshold each include a low-frequency energy threshold value and a spectral sub-band centroid corresponding threshold value.

8. The apparatus of any one of claims 5 to 7, wherein, The device further comprises: A statistical unit is configured to count the low-frequency energy value and the spectral sub-band centroid value within a continuous first frame number after forming a differential beam. The adjusting unit is configured to use a delay-and-sum beamforming algorithm to form a differential beam in a first frequency band and use a differential beamforming algorithm to form a differential beam in a second frequency band when there are a continuous second number of frames with low-frequency energy values higher than a low-frequency energy threshold value and spectral sub-band centroid values lower than a spectral sub-band centroid threshold value within the first number of frames, wherein a demarcation point between the first frequency band and the second frequency band is estimated based on the spectral sub-band centroid values. The low-frequency energy threshold value and the spectral sub-band centroid threshold value are determined based on the current frame audio signal before the differential beam is formed.

9. A differential beam forming apparatus characterized by, The device comprises: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the method of any one of claims 1 to 4.

10. A storage medium, characterized by The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor of a device, the device is enabled to execute the method of any one of claims 1 to 4.

Images