Multi-beam joint noise reduction method and device based on white noise gain control

Abstract

This application provides a multi-beam joint noise reduction method and apparatus based on white noise gain control. The method includes: performing preliminary noise reduction on a speech signal acquired by an earphone microphone array using multiple directional beamformers with different white noise gain lower limits to obtain multiple directional beam output signals with different white noise gain lower limits; using a null filter to obtain the null beam output signal corresponding to the speech signal; and using a machine learning-based speech enhancement model to perform post-filtering on the aforementioned beam output signals. This method combines the advantages of multiple directional beamformers with different white noise gain lower limits, and can improve the suppression effect of noise from other directions while extracting the target speech.

Description

Technical Field

[0001] The present application relates to the technical field of speech noise reduction, and in particular to a multi-beam joint noise reduction method and device based on white noise gain control. Background Art

[0002] Headphones have become a common electronic product in daily entertainment and voice communication. In actual applications, the microphone of headphones often picks up a variety of sounds, including the wearer's voice and other noises, such as subway noise, road noise, and noise from other speakers. These noises are often random and non-stationary, which not only affects the call quality, but also affects the performance of voice assistant applications such as voice wake-up and voice recognition.

[0003] Therefore, it is necessary to design a noise reduction solution for headphones to suppress various noises and enhance the wearer's voice. Summary of the invention

[0004] The present application provides a multi-beam joint noise reduction method and device based on white noise gain control, which uses multiple directional beamformers corresponding to different white noise gain lower limits to jointly reduce the noise of speech signals. By combining the advantages of multiple directional beamformers, when extracting the target speech, the beam white noise gain is controlled, and the noise suppression effect in other directions is improved, thereby ultimately improving the speech noise reduction effect.

[0005] In a first aspect, the present application provides a method for determining a beamformer. The beamformer is applied to a headset, and the method comprises:

[0006] Obtaining a lower limit value of a white noise gain corresponding to a target beamformer; wherein the target beamformer is obtained by weighted summing a diagonally loaded super-directional beamformer and a delay-sum beamformer corresponding to the earphone, the diagonally loaded super-directional beamformer and the delay-sum beamformer are determined according to a steering vector corresponding to the earphone, and the steering vector is determined according to a distance between array elements in an extra-ear microphone array of the earphone and a pointing angle corresponding to the beamformer;

[0007] According to the lower limit value, updating a first weight of the diagonally loaded super-directional beamformer in the target beamformer and a second weight of the delay-sum beamformer in the target beamformer;

[0008] An updated target beamformer is determined according to the updated first weight and the updated second weight.

[0009] The above scheme updates the weights of the diagonally loaded super-directional beamformer and the delay-sum beamformer in the target beamformer according to the lower limit of the white noise gain of the target beamformer, so that the white noise gain of the target beamformer can be controlled within the lower limit. Among them, the target beamformer combines the advantages of high white noise gain of the delay-sum beamformer and high directivity of the diagonally loaded super-directional beamformer, and can achieve a certain noise reduction effect while controlling the white noise gain.

[0010] In a possible implementation, the target beamformer includes a directional beamformer or a null beamformer; wherein a directional angle corresponding to the directional beamformer is different from a directional angle corresponding to the null beamformer.

[0011] In a possible implementation manner, updating, according to the lower limit value, a first weight of the diagonally loaded super-directional beamformer in the target beamformer and a second weight of the delay-sum beamformer in the target beamformer includes:

[0012] Determining a regularization parameter of the target beamformer in each frequency band according to the target lower limit value, the white noise gain of the delay-sum beamformer in each frequency band, the white noise gain of the diagonally loaded super-directional beamformer in each frequency band, and the directivity index value of the diagonally loaded super-directional beamformer in each frequency band;

[0013] The updated first weight and the updated second weight are determined according to the regularization parameters corresponding to the respective frequency bands.

[0014] In a second aspect, the present application also provides a method for reducing speech noise. The method is applied to headphones, and the method includes:

[0015] Acquire a voice signal to be processed; the voice signal is collected by a microphone array outside the headset;

[0016] According to the speech signal, a plurality of directional beamformers are used to obtain a plurality of directional beam output signals, and a null-notch beamformer is used to obtain a null-notch beam output signal corresponding to the speech signal; wherein the plurality of directional beam output signals correspond one-to-one to the plurality of directional beamformers, and the plurality of directional beamformers correspond to different lower limits of white noise gain; the plurality of directional beamformers and the null-notch beamformer are both obtained according to the method of the first aspect and its optional implementation manner,

[0017] According to the null-notch beam output signal and the plurality of directional beam output signals, post-filtering is performed using a speech enhancement model to obtain a frequency domain signal after noise reduction;

[0018] A noise-reduced speech signal is obtained according to the noise-reduced frequency domain signal.

[0019] Since directional beamformers with different white noise gain lower limits have different noise suppression capabilities in other directions of speech, for common fixed beamformers, the lower the lower limit is, the smaller the white noise gain of the directional beamformer is, that is, the weaker the suppression capability of the microphone self-noise is, but the better the noise suppression capability in other directions is. When the white noise gain is lower than 0dB, the self-noise will be amplified. Therefore, in the above scheme, multiple directional beamformers with different white noise gain lower limits are used for joint noise reduction, which can combine the advantages of multiple directional beamformers, and while controlling the white noise gain in the noise-reduced speech, the noise suppression effect in other directions can be guaranteed to improve the speech noise reduction effect.

[0020] In a third aspect, the present application further provides a method for training a speech enhancement model, wherein the speech enhancement model is applied to headphones, and the method comprises:

[0021] Obtain noisy speech signals and clean speech signals;

[0022] Using a plurality of directional beamformers corresponding to the earphone, a plurality of directional beam output signals corresponding to the noisy speech signal are obtained, and using a nulling beamformer corresponding to the earphone, a nulling beam output signal corresponding to the noisy speech signal is obtained; wherein the plurality of directional beamformers and the plurality of directional beam output signals correspond one to one, the plurality of directional beamformers and the nulling beamformer are both obtained according to the method in the first aspect and its optional implementation manner, and the lower limits of the white noise gains corresponding to the plurality of directional beamformers are different;

[0023] The parameters of the speech enhancement model are updated according to the multiple directional beam output signals, the null-notched beam output signal and the clean speech signal.

[0024] In a possible implementation, the speech enhancement model is determined according to a GCCRN network.

[0025] In a fourth aspect, the present application further provides a beamformer determination device. The beamformer is applied to a headset, and the determination device includes:

[0026] an acquisition module, configured to acquire a lower limit value of a white noise gain corresponding to a target beamformer; wherein the target beamformer is obtained by weighted summing a diagonally loaded super-directional beamformer and a delay-sum beamformer corresponding to the earphone, wherein the diagonally loaded super-directional beamformer and the delay-sum beamformer are determined according to a steering vector corresponding to the earphone, and the steering vector is determined according to a distance between array elements in a microphone array outside the earphone and a pointing angle corresponding to the beamformer;

[0027] An updating module, configured to update a first weight of the diagonally loaded super-directional beamformer in the target beamformer and a second weight of the delay-sum beamformer in the target beamformer according to the lower limit value;

[0028] The determination module is used to determine an updated target beamformer according to the updated first weight and the updated second weight.

[0029] In a possible implementation, the target beamformer includes a directional beamformer or a null beamformer; wherein a directional angle corresponding to the directional beamformer is different from a directional angle corresponding to the null beamformer.

[0030] In a possible implementation manner, the update module is specifically used to:

[0031] Determining a regularization parameter of the target beamformer in each frequency band according to the target lower limit value, the white noise gain of the delay-sum beamformer in each frequency band, the white noise gain of the diagonally loaded super-directional beamformer in each frequency band, and the directivity index value of the diagonally loaded super-directional beamformer in each frequency band;

[0032] The updated first weight and the updated second weight are determined according to the regularization parameters corresponding to the respective frequency bands.

[0033] In a fifth aspect, the present application further provides a speech noise reduction device. The speech noise reduction device is applied to headphones, and the speech noise reduction device includes:

[0034] An acquisition module, used for acquiring a voice signal to be processed; the voice signal is collected by a microphone array outside the headset;

[0035] A beamforming module, configured to obtain, according to the speech signal, a plurality of directional beam output signals using a plurality of directional beam formers, and to obtain a null-notch beam output signal corresponding to the speech signal using a null-notch beam former; wherein the plurality of directional beam output signals correspond one-to-one to the plurality of directional beam formers, the plurality of directional beam formers and the null-notch beam formers are obtained according to the method in the first aspect and its optional implementation manner, and the plurality of directional beam formers have different white noise gain lower limits;

[0036] A post-filtering module, configured to perform post-filtering using a speech enhancement model according to the null-notch beam output signal and the multiple directional beam output signals to obtain a de-noised speech signal;

[0037] The conversion module is used to obtain a noise-reduced speech signal according to the noise-reduced frequency domain signal.

[0038] In a sixth aspect, the present application further provides a training device for a speech enhancement model. The speech enhancement model is applied to headphones, and the training device comprises:

[0039] An acquisition module, used to acquire noisy speech signals and clean speech signals;

[0040] A beamforming module, used to obtain multiple directional beam output signals corresponding to the noisy speech signal using multiple directional beamformers corresponding to the headset, and to obtain a null-notch beam output signal corresponding to the noisy speech signal using a null-notch beamformer corresponding to the headset; wherein the multiple directional beamformers and the multiple directional beam output signals correspond one to one, the multiple directional beamformers and the null-notch beamformer are both obtained according to the method in the first aspect and its optional implementation manner, and the lower limits of the white noise gains corresponding to the multiple directional beamformers are different;

[0041] An updating module is used to update the parameters of the speech enhancement model according to the multiple directional beam output signals, the null-notch beam output signal and the clean speech signal.

[0042] In a seventh aspect, the present application further provides an earphone, wherein the earphone comprises a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to execute the method described in the second aspect.

[0043] Any of the devices or headphones provided above are used to execute the method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding schemes in the corresponding methods provided above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1This is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0045] Figure 2 is a schematic diagram of a user wearing headphones provided in an embodiment of the present application;

[0046] Figure 3 This is a white noise gain comparison diagram and a directivity index comparison diagram of each beamformer provided in an embodiment of the present application;

[0047] Figure 4 is a beam comparison diagram of each beamformer provided in an embodiment of the present application;

[0048] Figure 5 is a flow chart of a method for determining a beamformer provided in an embodiment of the present application;

[0049] Figure 6 is a beam comparison diagram of a beamformer based on different white noise gain controls provided in an embodiment of the present application;

[0050] Figure 7 It is a spectrogram of a 0-4kHz signal in an actual noisy indoor environment and a spectrogram of a signal processed by different directional beamformers provided in an embodiment of the present application;

[0051] Figure 8 It is a spectrogram of a 0-1kHz signal in an actual noisy indoor environment and a spectrogram of a signal processed by different directional beamformers provided in an embodiment of the present application;

[0052] Fig. 9 This is a beam comparison diagram of two null-notch beamformers with different pointing angles provided in an embodiment of the present application;

[0053] Fig.10 It is a beam diagram and a white noise gain curve diagram of a null-notch beamformer provided in an embodiment of the present application;

[0054] Fig.11 It is a spectrogram of a signal received by a microphone array in an actual noisy indoor environment provided by an embodiment of the present application before and after being processed by a null-notch beamformer;

[0055] Fig.12 This is a schematic diagram of a multi-beam joint noise reduction process based on white noise gain control provided in an embodiment of the present application;

[0056] Fig.13 It is a flow chart of a multi-beam joint noise reduction method based on white noise gain control provided in an embodiment of the present application;

[0057] Fig.14It is a spectrogram of a signal received by a microphone array in an indoor music noise environment, and a spectrogram after the signal is processed by beamformers with different directivity, provided in an embodiment of the present application;

[0058] Fig.15 It is a spectrogram of a signal received by a microphone array in an outdoor noise environment, and a spectrogram after the signal is processed by beamformers with different directivity, provided in an embodiment of the present application;

[0059] Fig.16 It is a flow chart of a method for training a speech enhancement model provided in an embodiment of the present application;

[0060] Fig.17 It is a structural diagram of a GCCRN network provided in an embodiment of the present application;

[0061] Fig.18 is a structural schematic diagram of a beamformer determination device provided in an embodiment of the present application;

[0062] Fig.19 It is a structural schematic diagram of a multi-beam joint noise reduction device based on white noise gain control provided in an embodiment of the present application;

[0063] Fig. 20 It is a structural schematic diagram of a training device for a speech enhancement model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0065] In the description of the embodiments of the present application, words such as "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a concrete way.

[0066] In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the term "multiple" means two or more. For example, multiple systems refers to two or more systems, and multiple screen terminals refers to two or more screen terminals.

[0067] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprises", "has" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0068] Figure 1 Schematic diagram of a signal model of a headset provided by this application. Figure 1 As shown, when user 1 wears headphones in a space and uses the headphones to talk, the earphone's external microphone will collect the user's 1 voice and surrounding environmental noise. The external microphone refers to the microphone located outside the ear when the user wears the headphones.

[0069] Figure 2 A schematic diagram showing a user wearing headphones. Figure 2 As shown in the front view, both sides of the earphone are equipped with an outer ear microphone array with a linear array structure (Mic1 and Mic3 on the left, Mic2 and Mic4 on the right). Figure 2 As shown in the side view, each extra-auricular microphone array includes two microphones.

[0070] Specifically, the signal collected by the i-th external ear microphone in the earphone can be expressed by formula (1).

[0071] x i (n) = s i (n)+d i (n)+w i (n) (1)

[0072] In formula (1), s i (n) represents the user's voice signal collected by the i-th external ear microphone, d i (n) represents the ambient noise signal collected by the i-th external ear microphone, w i (n) represents the self-noise of the ith extra-auricular microphone, i∈[1,M]. M represents the number of microphones in the extra-auricular microphone array.

[0073] Ambient noise can include direct sound and diffuse field noise. Figure 1 , the direct sound can be Figure 1 The sound emitted by the user 2 is the sound collected by the microphone after being propagated in a straight line, and the diffuse field noise can be the sound emitted by the user 2 is the sound collected by the microphone after being reflected.

[0074] Based on the signal representation shown in formula 1, the frequency domain reception model of the i-th extra-auricular microphone can be shown as formula (2).

[0075] X i (k,l)=S i (k,l)+D i (k,l)+W i (k,l) (2)

[0076] In formula (2), X i (k,l),S i (k,l),D i (k,l) and W i (k,l) are x i (n), s i (n), d i (n) and w i (n) corresponds to the kth frequency band of the lth frame (the corresponding frequency is recorded as f k ) short-time spectrum. In practical applications, x can be transformed by fast Fourier transform i (n)Convert to X i (k,l).

[0077] In one example, the headset can use three noise reduction schemes for voice enhancement. The following briefly introduces the three noise reduction schemes.

[0078] The first is a single-microphone solution, in which the earphone is equipped with an extra-ear microphone and the frequency domain signal represented by formula (2) is processed using a traditional spectral subtraction method or a single-channel speech enhancement algorithm based on machine learning.

[0079] This solution is low-cost, but it often fails to achieve good noise suppression effects in high-noise environments. In particular, when the noise includes non-wearer voices or voice-like signals, the solution cannot effectively distinguish between target voices and interfering voices, resulting in a significant decrease in noise suppression effects.

[0080] The second solution is to combine the external ear microphone with the internal auxiliary sensor for noise reduction, and achieve speech enhancement by fusing the signals collected by the external ear microphone and the internal auxiliary sensor. Among them, common internal auxiliary sensors include the internal ear microphone, vibration sensor or acceleration sensor.

[0081] This solution requires internal auxiliary sensors to provide information for auxiliary judgment. Although it can achieve good noise reduction effects, it requires the wearer to wear the headphones correctly to ensure the sound pickup quality of the auxiliary sensors, and the cost is relatively high. When worn incorrectly or the coupling is poor, the fusion scheme of the inner and outer ear signals is mismatched, resulting in a decrease in noise reduction performance. In addition, the price of the internal auxiliary microphone is relatively expensive, which will increase the hardware cost of the headphones.

[0082] The third is a multi-ear microphone solution, which generally places 2 to 4 microphones in a single earphone to form an ear microphone array. This solution first uses beamforming or adaptive beamforming methods to preliminarily enhance the frequency domain signal represented by formula (2), and then uses post-filtering methods to further enhance the beam output information. A common representative method is the time frequency-generalized sidelobe canceller (TF-GSC) algorithm based on the time-frequency domain. This method constructs a directional beam and a null beam to obtain a preliminarily enhanced target signal and noise signal, respectively, and then performs post-filtering to obtain the enhanced information.

[0083] The hardware cost of this solution is low, and the noise reduction performance is significantly improved compared to the single-microphone method. However, this solution still has some problems.

[0084] For example, the directive beam in the scheme is usually formed by a delay and sum beamformer (DSB) and a super directive beamformer (SDB). DSB does not have the problem of white noise gain amplification, but its directivity is poor, the noise suppression ability in all directions is limited, and the sidelobe level in the beam pattern is high. Although SDB has the highest directivity index, that is, good directivity, it has a serious problem of low-frequency noise amplification.

[0085] For another example, the null beam is often formed by differential beamforming or by blocking matrix design. The former also has the problem of low-frequency white noise amplification, and the latter has the problem of too small null width. If the null width is too small, if there is a certain deviation between the preset null direction and the actual mouth direction, the notch performance will drop sharply. These problems will lead to post-filtering speech distortion under low signal-to-noise ratio conditions.

[0086] Specifically, the DSB and SDB used in the third solution may be expressions as shown in formulas (3) and (4), respectively.

[0087]

[0088]

[0089] In the above formulas (3) and (4), v(k,θ s ) represents the steering vector corresponding to the beamformer pointing to the human mouth, v H (k,θ s ) represents v(k,θ s ) is the conjugate transpose of Γ; d(k) is the M×M dimensional diffusion field covariance matrix, and its nth row and mth column element can be calculated according to formula (5).

[0090]

[0091] In formula (5), d mn represents the distance between the nth microphone and the mth microphone, n∈[1,M], m∈[1,M], and c represents the speed of sound.

[0092] Generally, the indicators for measuring the beam performance of the beamformer include directivity index, white noise gain and beam pattern. The directivity index, white noise gain and beam pattern can be determined according to formula (6), formula (7) and formula (8).

[0093]

[0094]

[0095] B(k,θ)=w H (k)v(k,θ) (8)

[0096] In the above formula (6) to formula (8), represents the directivity index, represents the white noise gain, B(k,θ) represents the beam gain in the beam pattern, and w H (k) represents the conjugate transpose of the beamformer expression w(k), and v(k,θ) represents the steering vector pointing in the direction of angle θ. The larger the directivity index, the stronger the beam directivity of the beamformer, and the stronger the noise suppression capability in directions other than the pointing direction. The larger the white noise gain, the less likely the beamformer is to amplify white noise, and the better the robustness.

[0097] SDB has the largest directivity index The delay-sum beamformer has the largest white noise gain compared to other beamformers mentioned in this application. (is a constant ), the corresponding expressions are shown in formula (9) and formula (10) respectively.

[0098]

[0099]

[0100] In order to solve the problem of white noise amplification in SDB, SDB can be improved by performing diagonal loading operation on it to obtain a diagonal-loaded super-directive beamformer-diagonal load (SDB-DL). Specifically, Γ in formula (4) is re-determined according to the diagonal loading ε(k). d (k), i.e. Γ ε (k) = Γ d (k)+ε(k)I M Therefore, the expression of SDB-DL can be shown as formula (10).

[0101]

[0102] Accordingly, the white noise gain expression and directivity expression of SDB-DL can be as shown in formula (12) and formula (13).

[0103]

[0104]

[0105] Although SDB-DL can be corrected by diagonal loading, the loading amounts at different frequencies are different and can be determined based on experience.

[0106] The above-mentioned DSB, SDB and SDB-DL all have certain advantages and disadvantages. According to the expressions and performance index expressions of each beamformer of DSB, SDB and SDB-DL mentioned above, we can get Figure 3 The performance index analysis diagram of the three beamformers is shown in FIG. Figure 3 As shown in Figure a, the white noise gain of SDB near 0Hz is about -30dB, and there is a serious problem of low-frequency white noise amplification; DSB almost does not have low-frequency white noise amplification; SDB-DL has moderate white noise amplification compared with SDB due to the diagonal loading operation. Figure 3 As shown in Figure b, the directivity of SDB is the best, the directivity of DSB is the worst, and the directivity of SDB-DL is degraded compared to SDB.

[0107] Recombination Figure 4 As shown in the beam diagram, DSB has a larger main lobe width and poorer noise suppression capability in other directions; SDB has the narrowest main lobe width and the best noise suppression capability in other directions; SDB-DL is moderate. Figure 4 In the figure, the horizontal axis represents the pointing angle Direction, the left side represents the frequency Frequency, and the right side represents the beam gain.

[0108] Based on the problems existing in the above-mentioned beamformers, an embodiment of the present application provides a beamformer based on white noise gain control, which can achieve as high a directivity index as possible under the premise of accurately controlling the white noise gain. Specifically, the beamformer is a beamformer based on regularized weighting, as shown in formula (14).

[0109]

[0110] In formula (14), α(k) is expressed as a regularization parameter, and the diagonal loading ε(k) can be determined by formula (15). The value range of ε(k) is 10 -6 Up to 10 -2 .

[0111]

[0112] Formula (14) can be further transformed into the form shown in Formula (16), where α ε (k) is the intermediate variable,

[0113]

[0114] From formula (16), we can see that the beamformer w α (k) can be regarded as a linear combination of SDB-DL and DSB, and and are the weighted weights of SDB-DL and DSB respectively.

[0115] According to equations (11), (12) and (16), the beamformer w can be obtained α (k) The corresponding white noise gain As shown in Figure (17).

[0116]

[0117] In formula (17),

[0118] From formula (16), we can get:

[0119]

[0120] According to formula (17), Combined with the above formula The calculation formula for the regularization parameter α(k) can be calculated as shown in formula (19).

[0121]

[0122] According to formula (16) and It can be seen that the weights of the above SDB-DL and DSB are related to the white noise gain, steering vector and regularization parameter of the beamformer. According to formulas (10), (12) and (19), it can be seen that the regularization parameter is related to the white noise gain and steering vector of the beamformer.

[0123] When the pointing angle in the steering vector is an angle pointing to the human mouth, the beamformer is a directional beamformer. When the pointing angle in the steering vector is an angle not pointing to the human mouth, the beamformer is a null-notch beamformer.

[0124] Therefore, when determining the beamformer, a directional beamformer and a null beamformer can be obtained according to different white noise gain lower limits and pointing angles.

[0125] Based on this, an embodiment of the present application provides a method for determining a beamformer, which is used to determine the above-mentioned directional beamformer and null beamformer.

[0126] Figure 5 It is a flow chart of a beamformer determination method provided in an embodiment of the present application.

[0127] like Figure 5 As shown, the method includes the following steps S501 to S503.

[0128] In step S501, a lower limit value of the white noise gain corresponding to the target beamformer is obtained.

[0129] Specifically, the lower limit value of the white noise gain may be 0 dB, -6 dB, or -12 dB. The target beamformer may be the beamformer shown in formula (16). The target beamformer is obtained by weighted summing of the diagonally loaded super-directional beamformer and the delay-sum beamformer corresponding to the earphone.

[0130] Specifically, as shown in formulas (3) and (11), the diagonally loaded super-directional beamformer and the delay-sum beamformer corresponding to the earphone can be determined according to the steering vector corresponding to the earphone.

[0131] Specifically, the steering vector can be determined according to the distance between the array elements in the earphone's extra-ear microphone array and the corresponding pointing angle of the target beamformer, that is, where τ m =d m1 cosθ s / c,θ s Indicates the pointing angle, d m1represents the distance between the mth microphone and the first microphone. The first microphone may be a microphone far from the human mouth or a microphone close to the human mouth in the microphone array.

[0132] Specifically, when the target beamformer is a directional beamformer, θ s It is the angle pointing to the mouth of a person. In one example, the angle pointing to the mouth of a person can be 0 degrees.

[0133] Specifically, when the target beamformer is a null-notch beamformer, a directional angle can be set for the microphone array, or a different directional angle can be set for each microphone in the microphone array. Taking the microphone array containing two microphones as an example, the directional angle of the microphone close to the human mouth can be set to 135 degrees, and the directional angle of the microphone far from the human mouth can be set to 100 degrees. In this way, two beamformers with different directional directions can be obtained.

[0134] In one example, for different frequencies, the lower limit of the white noise gain may be limited. For example, for frequency f k , when the lower limit is lower than SDB-DL at f k When the white noise gain is , the f corresponding to SDB-DL can be used instead. k The white noise gain is used as the lower limit of the white noise gain corresponding to the frequency.

[0135] In step S502, a first weight of the diagonally loaded super-directional beamformer in the target beamformer and a second weight of the delay-and-sum beamformer in the target beamformer are updated according to the lower limit value.

[0136] The first weight refers to the aforementioned The second weight refers to the aforementioned Specifically, the regularization parameter α(k) can be calculated according to formula (19), and then according to formula Calculate the intermediate variable α ε (k), and then the updated first weight and second weight can be obtained.

[0137] In step S503, an updated target beamformer is determined according to the updated first weight and the updated second weight.

[0138] Specifically, the updated target beamformer can be obtained according to the updated first weight and the updated second weight in combination with formula (16).

[0139] In one example, when the white noise gain lower limit of the target beamformer can be set to 0 dB, -6 dB, and -12 dB, three target beamformers with different white noise gain lower limits can be obtained. Figure 6The beamformer beam patterns corresponding to three different lower limits are shown. Figure 6 As shown, the lower the white noise gain lower limit is set, the narrower the low-frequency main lobe width is, and the better the noise suppression effect in other directions is.

[0140] The above beamformer determination method can improve the problem of white noise amplification by controlling the lower limit of the white noise gain of the target beamformer.

[0141] Figure 7 and Figure 8 The microphone signals recorded in an actual noisy indoor environment are given, and the spectrograms of the output signals 0-4kHz and 0-1kHz after being processed by different beamformers are given. Figure 6 Figure a and Figure 7 Figure a shows the spectrogram of the speech frequency domain signal of the corresponding frequency band received by the microphone array. Figure 6 The b-picture and Figure 7 Figure b shows the signal spectrogram after DSB processing the speech domain signal of the corresponding frequency band. Figure 6 The c graph and Figure 7 Figure c shows the signal spectrogram after SDB processes the speech domain signal of the corresponding frequency band. Figure 6 The d graph and Figure 7 The d graph represents the Figure 5 The signal spectrogram after the directional beamformer with a white noise gain lower limit of -6dB determined by the method shown processes the speech domain signal in the corresponding frequency band.

[0142] contrast Figure 7 Middle B picture and Figure 7 As can be seen from Figure d, DSB has a lot of residual noise, while the interference suppression capability of the directional beamformer obtained by the method of the present application is significantly improved. It is calculated that the average noise reduction amount of 0-4kHz can be increased by about 6dB.

[0143] contrast Figure 8 Figure C and Figure 8 As shown in Figure d, the directional beamformer obtained by the method of the present application has no obvious white noise amplification phenomenon below 800Hz, while SDB produces serious white noise amplification phenomenon, which seriously affects the voice quality. Compared with the above two beamformers, the directional beamformer obtained by the method of the present application can effectively suppress noise in other directions without significantly amplifying low-frequency white noise.

[0144] In one example, when the directional directions corresponding to the microphone arrays are different, two beamformers with different directional directions may be obtained. Fig. 9 The beam pattern of the beamformer is shown when two microphones are included in the microphone array. Fig. 9Figure a in the figure corresponds to the beam pattern of the beamformer with a pointing direction of 135 degrees, and figure b corresponds to the beam pattern of the beamformer with a pointing direction of 100 degrees. Fig. 9 It can be seen that except for the null region near 0 degrees, the two beamformers w n1 (k) and w n2 (k) both form a null region in the area around 0 degrees, and the null depths of the two null regions have certain complementary characteristics. n1 (k) In the 0 degree direction, the null depth of the high frequency is shallow, while w n1 (k) The corresponding zero-sag depth is deeper here.

[0145] Therefore, according to w n1 (k) and w n2 (k) The spatial beam pattern characteristics of the two beamformers with different pointing directions can be weighted summed to design a new beamformer w n (k) = α1w n1 (k)+α2w n2 (k), the final null-notched beamformer w n (k). Optionally, the weights α1 and α2 can be set to 0.5.

[0146] Fig.10 The null-notch beamformer w n (k) The beam pattern and the corresponding white noise gain diagram. n (k) A deeper null can be formed in the 0 degree direction and the spatial area of ​​plus or minus 45 degrees, while maintaining a high gain in other directions. At the same time, the full-band white noise gain is controlled within the range of 0dB to 2dB, and there is no problem of white noise gain amplification.

[0147] Fig.11 The spectrogram of the microphone signal recorded in an actual noisy indoor environment before and after being processed by the null-notch beamformer is shown. It can be seen that the null-notch beamformer obtained by the method of the present application effectively suppresses the wearer's vocal component without producing obvious white noise amplification. It should be noted that although the null-notch beamformer does not produce the problem of low-frequency white noise amplification, the ability to suppress target speech in the 0-degree direction below 1kHz is also poor. Therefore, when processing in the post-filtering process, harmonic detection technology can be used to remove the target speech in the null-notch beam output below 1kHz, so that the low-frequency target speech can be retained as much as possible during post-filtering.

[0148] It should be noted that when the number and array of microphones change, the beamformer group {w n,i (k)} i＝1,2,...M, construct a new null-notch beamformer through their linear combination The specific pointing direction and weighting coefficient can be determined in advance based on the number and array of microphones.

[0149] Based on the above-mentioned beamformer determination method, the present application embodiment provides a multi-beam joint noise reduction method based on white noise gain control. Fig.12 As shown in FIG. 1 , the scheme first uses short-time Fourier transform (STFT) to transform the collected speech signals x1(t) and x2(t) into the frequency domain to obtain X1(k, l) and X2(k, l); a plurality of directional beam formers directed to the human mouth designed based on the white noise gain control method are used for preliminary filtering to obtain directional beam output signals (Y corresponding to different white noise gains) 0dB (k,l),Y -6dB (k,l),Y- 12 dB(k,l)); and a null-notch beam former directed to the mouth designed based on the white noise gain control method obtains the null-notch beam output signal corresponding to the collected speech signal (Y null (k,l)); finally, the speech enhancement model based on machine learning is used to perform post-filtering on the directional beam output signals corresponding to the different white noise gains obtained previously according to the null-notch beam output signal to obtain the frequency domain signal Y after noise reduction enh (k, l), and use inverse short-time Fourier transform (ISTFT) to obtain the de-noised speech signal y(t). This solution can solve the problem of low-frequency white noise amplification in the third solution.

[0150] Fig.13 A flow chart of a noise reduction method provided in an embodiment of the present application is shown.

[0151] like Fig.13 As shown, the method includes: the following steps S1301-S1303.

[0152] In step S1301, a speech signal to be processed is obtained.

[0153] The speech signal is collected by a microphone array outside the headset.

[0154] In step S1302, according to the speech signal, a plurality of directional beamformers are used to obtain directional beam output signals corresponding to a plurality of white noise gains, and a null beamformer is used to obtain a null beam output signal corresponding to the speech signal.

[0155] The plurality of directional beamformers correspond to the directional beam output signals corresponding to the plurality of white noise gains one by one, and the plurality of directional beamformers and the nulling beamformer are composed of Figure 5 The method shown is used to obtain the white noise gain, and the lower limits of the white noise gains corresponding to the multiple directional beamformers are different.

[0156] In step S1303, post-filtering is performed using a speech enhancement model according to the null-notch beam output signal and the multiple directional beam output signals to obtain a de-noised speech signal.

[0157] Specifically, the real and imaginary signals of multiple directional beam output signals and the real and imaginary signals of null-notch beam output signals may be input into a speech enhancement model to obtain a noise-reduced frequency domain signal output by the speech enhancement model.

[0158] Among them, the multiple directional beam output signals can be signals corresponding to the corresponding white noise gain lower limits of 0dB, -6dB and -12dB. At this time, the input of the speech enhancement model is 8-channel data. The specific training process of the speech enhancement model will be introduced later and will not be repeated here.

[0159] The de-noised speech signal is obtained according to the de-noised frequency domain signal. Specifically, the de-noised frequency domain signal may be subjected to an inverse fast Fourier transform to obtain the de-noised speech signal.

[0160] The above noise reduction method, based on the advantage that target beamformers with different white noise gain lower limits have different noise suppression effects, uses multiple beamformers with different white noise gain lower limits for multi-beam filtering, and then uses a speech enhancement model for post-filtering, which can improve the noise suppression effects in other directions.

[0161] Fig.14 The spectrograms of the signals received by the microphone array in an indoor music noise environment are shown respectively ( Fig.14 Figure a in the figure), the spectrogram after processing by the directional beamformer based on white noise gain control (WNG lower limit -6dB) ( Fig.14 Figure b in the figure), the spectrogram after processing by the null-notch beamformer based on white noise gain control ( Fig.14 Figure c), and use Fig.13 Spectrogram of the signal obtained by the denoising method shown ( Fig.14 d in Figure ).

[0162] Fig.15 The spectrograms of the signals received by the microphone in an outdoor noise environment are shown respectively ( Fig.15 Figure a in the figure), the spectrogram after processing by the directional beamformer based on white noise gain control (WNG lower limit -6dB) ( Fig.15Figure b in the figure), the spectrogram after processing by the null-notch beamformer based on white noise gain control ( Fig.15 Figure c), and use Fig.13 Spectrogram of the signal obtained by the denoising method shown ( Fig.15 d in Figure ).

[0163] from Fig.14 and Fig.15 It can be seen that the directional beamformer based on white noise gain control can effectively reduce noise, while the null-notch beamformer based on white noise gain control can effectively eliminate the target speech, and the null-notch beamformer can fully retain the information of the music interference signal, thereby ultimately achieving a better speech enhancement effect.

[0164] Fig.16 It is a flow chart of a training method for a speech enhancement model provided in an embodiment of the present application.

[0165] like Fig.16 As shown, the method includes the following steps S1601-S1603.

[0166] In step S1601, a noisy speech signal and a clean speech signal are obtained.

[0167] Noise signals are collected in different environments, such as roads, subways, conference rooms, and other places where the types of noise and the directions of noise waves should be diverse.

[0168] The user's pure voice signal is collected in a quiet environment. When collecting the pure voice signal, the wearing position of the earphone can be appropriately adjusted to enhance the robustness of the model.

[0169] Then, the noise signal and the clean speech signal are mixed according to the signal-to-noise ratio to obtain the noisy speech signal.

[0170] In one example, the noise signal and the clean speech signal may be mixed according to different signal-to-noise ratios to obtain different noisy speech signals.

[0171] In step S1602, multiple directional beamformers corresponding to the earphone are used to obtain multiple directional beam output signals corresponding to the noisy speech signal, and a null beamformer corresponding to the earphone is used to obtain a null beam output signal corresponding to the noisy speech signal.

[0172] The multiple directional beamformers and the multiple directional beam output signals correspond one to one, and the multiple directional beamformers and the null-notch beamformer both adopt Figure 5 The method shown in the figure obtains that the lower limits of the white noise gains corresponding to the multiple directional beamformers are different.

[0173] In step S1603, the parameters of the speech enhancement model are updated according to the multiple directional beam output signals, the null beam output signal and the clean speech signal.

[0174] Specifically, a plurality of directional beam output signals based on white noise gain lower limit control and a null-notch beam output signal based on white noise gain control may be input into the speech enhancement model to obtain the output of the speech enhancement model.

[0175] Then, according to the output of the speech enhancement model and the clean speech signal, the error value is calculated using the loss function, and according to the error value, the parameters of the speech enhancement model are updated using the gradient descent method.

[0176] Among them, the speech enhancement model can adopt a GCCRN (gate complex convolution recursion network) structure, or a gate recurrent unit (GRU) structure and a U-NET structure. Fig.17 Schematic diagram of a GCCRN network structure provided in an embodiment of the present application. Fig.17 As shown in the figure, the GCCRN network includes an encoder encoding network, a recurrent convolutional network, a decoder decoding network and an output layer.

[0177] The encoder has 5 layers, each of which is constructed using the conv2d convolution function. The decoder has 5 layers, each of which includes real and imaginary outputs, and uses the deconv2d deconvolution function. The recurrent convolutional network has a 2-layer LSTM structure, and each LSTM layer uses two sets of identical network structures to reduce the number of model parameters. The output layer is a FC fully connected layer.

[0178] Based on the above Figure 5 The beamformer determination method shown in the figure, an embodiment of the present application provides a beamformer determination device.

[0179] Fig.18 FIG. 1 is a schematic diagram of a structure of a beamformer determination device 1800 provided in an embodiment of the present application. Fig.18 As shown, the determination device 1800 includes: an acquisition module 1801 , an update module 1802 and a determination module 1803 .

[0180] Among them, the acquisition module 1801 is used to obtain the lower limit value of the white noise gain corresponding to the target beamformer; wherein the target beamformer is obtained by weighted summing the diagonally loaded super-directional beamformer and the delay-sum beamformer corresponding to the earphone, the diagonally loaded super-directional beamformer and the delay-sum beamformer are determined according to the steering vector corresponding to the earphone, and the steering vector is determined according to the distance between array elements in the microphone array outside the earphone and the pointing angle corresponding to the beamformer.

[0181] The updating module 1802 is configured to update the first weight of the diagonally loaded super-directional beamformer in the target beamformer and the second weight of the delay-sum beamformer in the target beamformer according to the lower limit value.

[0182] The determination module 1803 is used to determine an updated target beamformer according to the updated first weight and the updated second weight.

[0183] The specific implementation process of each of the above modules can be found in the above invention content or the above Figure 5 The description of the determination method shown will not be repeated here.

[0184] Based on the above Fig.13 The noise reduction method shown, an embodiment of the present application provides a noise reduction device.

[0185] Fig.19 1 is a schematic diagram of the structure of a noise reduction device 1900 provided in an embodiment of the present application. Fig.19 As shown, the noise reduction device 1900 includes: an acquisition module 1901 , a beamforming module 1902 and a post-filtering module 1903 .

[0186] The acquisition module 1901 is used to acquire the voice signal to be processed; the voice signal is collected by the microphone array outside the earphone.

[0187] Among them, the beamforming module 1902 is used to obtain multiple directional beam output signals according to the speech signal using multiple directional beamformers, and to obtain a null beam output signal corresponding to the speech signal using a null beamformer; wherein the multiple directional beam output signals correspond one-to-one to the multiple directional beamformers, and the lower limits of white noise gains corresponding to the multiple directional beamformers are different.

[0188] The post-filtering module 1903 is used to perform post-filtering based on the null-notch beam output signal and the multiple directional beam output signals using a speech enhancement model to obtain a noise-reduced speech signal.

[0189] The specific implementation process of each of the above modules can be found in the above invention content or the above Fig.13 The introduction of the noise reduction method shown in the figure will not be repeated here.

[0190] Based on the above Fig.16 The training method of the speech enhancement model shown, the embodiment of the present application also provides a training device.

[0191] Fig. 20 2 is a schematic diagram of the structure of a training device 2000 provided in an embodiment of the present application. Fig. 20 As shown, the training device 2000 includes: an acquisition module 2001, a beamforming module 2002 and an updating module 2003.

[0192] The acquisition module 2001 is used to acquire a noisy speech signal and a clean speech signal.

[0193] Among them, the beamforming module 2002 is used to use the multiple directional beamformers corresponding to the earphone to obtain the multiple directional beam output signals corresponding to the noisy speech signal, and use the null-notch beamformer corresponding to the earphone to obtain the null-notch beam output signal corresponding to the noisy speech signal; wherein the multiple directional beamformers and the multiple directional beam output signals correspond one to one, and the lower limits of the white noise gains corresponding to the multiple directional beamformers are different.

[0194] The updating module 2003 is used to update the parameters of the speech enhancement model according to the multiple directional beam output signals, the null beam output signal and the clean speech signal.

[0195] The specific implementation process of each of the above modules can be found in the above invention content or the above Fig.16 The introduction in the training method shown will not be repeated here.

[0196] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0197] It is understood that the various numerical numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application. It should be understood that in the embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0198] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the scope of protection of the present application.

Claims

1. A method for determining a beamformer, characterized in that: The beamformer is applied to headphones, and the method comprises: Obtaining a lower limit value of a white noise gain corresponding to a target beamformer; wherein the target beamformer is obtained by weighted summing a diagonally loaded super-directional beamformer and a delay-sum beamformer corresponding to the earphone, the diagonally loaded super-directional beamformer and the delay-sum beamformer are determined according to a steering vector corresponding to the earphone, and the steering vector is determined according to a distance between array elements in an extra-ear microphone array of the earphone and a pointing angle corresponding to the beamformer; According to the lower limit value, updating a first weight of the diagonally loaded super-directional beamformer in the target beamformer and a second weight of the delay-sum beamformer in the target beamformer; An updated target beamformer is determined according to the updated first weight and the updated second weight.

2. The method according to claim 1, characterized in that: The target beamformer includes a directional beamformer or a null beamformer; wherein a directional angle corresponding to the directional beamformer is different from a directional angle corresponding to the null beamformer.

3. The method according to claim 1, characterized in that: The updating, according to the lower limit value, of the first weight of the diagonally loaded super-directional beamformer in the target beamformer and the second weight of the delay-sum beamformer in the target beamformer comprises: Determining a regularization parameter of the target beamformer in each frequency band according to the lower limit value, the white noise gain value of the delay-sum beamformer in each frequency band, the white noise gain value of the diagonally loaded super-directional beamformer in each frequency band, and the directivity index value of the diagonally loaded super-directional beamformer in each frequency band; The updated first weight and the updated second weight are determined according to the regularization parameters corresponding to the respective frequency bands.

4. A speech noise reduction method, characterized in that: The method is applied to headphones, and the method comprises: Acquire a voice signal to be processed; the voice signal is collected by a microphone array outside the headset; According to the speech signal, a plurality of directional beamformers are used to obtain a plurality of directional beam output signals, and a null-notch beamformer is used to obtain a null-notch beam output signal corresponding to the speech signal; wherein the plurality of directional beam output signals correspond to the plurality of directional beamformers one-to-one, the plurality of directional beamformers and the null-notch beamformer are both obtained according to the method of claim 2, and the plurality of directional beamformers have different lower limits of white noise gain; According to the null-notch beam output signal and the multiple directional beam output signals, a speech enhancement model is used to perform post-filtering to obtain a noise-reduced speech signal.

5. A method for training a speech enhancement model, characterized in that: The speech enhancement model is applied to headphones, and the method comprises: Obtain noisy speech signals and clean speech signals; Utilizing the multiple directional beamformers corresponding to the earphone, obtaining the multiple directional beam output signals corresponding to the noisy speech signal, and utilizing the nulling beamformer corresponding to the earphone, obtaining the nulling beam output signal corresponding to the noisy speech signal; wherein the multiple directional beamformers and the multiple directional beam output signals correspond one to one, the multiple directional beamformers and the nulling beamformer are both obtained according to the method of claim 2, and the lower limits of the white noise gains corresponding to the multiple directional beamformers are different; The parameters of the speech enhancement model are updated according to the multiple directional beam output signals, the null-notched beam output signal and the clean speech signal.

6. The method according to claim 5, characterized in that The speech enhancement model is determined according to the GCCRN network.

7. A beamformer determination device, characterized in that: The beamformer is applied to headphones, and the determining device includes: an acquisition module, configured to acquire a lower limit value of a white noise gain corresponding to a target beamformer; wherein the target beamformer is obtained by weighted summing a diagonally loaded super-directional beamformer and a delay-sum beamformer corresponding to the earphone, wherein the diagonally loaded super-directional beamformer and the delay-sum beamformer are determined according to a steering vector corresponding to the earphone, and the steering vector is determined according to a distance between array elements in a microphone array outside the earphone and a pointing angle corresponding to the beamformer; An updating module, configured to update a first weight of the diagonally loaded super-directional beamformer in the target beamformer and a second weight of the delay-sum beamformer in the target beamformer according to the lower limit value; The determination module is used to determine an updated target beamformer according to the updated first weight and the updated second weight.

8. A speech noise reduction device, characterized in that: The speech noise reduction device is applied to headphones, and the speech noise reduction device comprises: An acquisition module, used for acquiring a voice signal to be processed; the voice signal is collected by a microphone array outside the headset; A beamforming module, configured to obtain a plurality of directional beam output signals using a plurality of directional beamformers according to the speech signal, and to obtain a null-notch beam output signal corresponding to the speech signal using a null-notch beamformer; wherein the plurality of directional beam output signals correspond to the plurality of directional beamformers one-to-one, the plurality of directional beamformers and the null-notch beamformer are both obtained according to the method of claim 2, and the plurality of directional beamformers correspond to different lower limits of white noise gain; The post-filtering module is used to perform post-filtering according to the null-notch beam output signal and the multiple directional beam output signals using a speech enhancement model to obtain a noise-reduced speech signal.

9. A training device for a speech enhancement model, characterized in that: The speech enhancement model is applied to headphones, and the training device comprises: An acquisition module, used to acquire noisy speech signals and clean speech signals; A beamforming module, used to obtain multiple directional beam output signals corresponding to the noisy speech signal using multiple directional beamformers corresponding to the headset, and to obtain a null beam output signal corresponding to the noisy speech signal using a null beamformer corresponding to the headset; wherein the multiple directional beamformers and the multiple directional beam output signals correspond one to one, the multiple directional beamformers and the null beamformer are both obtained according to the method of claim 2, and the lower limits of the white noise gains corresponding to the multiple directional beamformers are different; An updating module is used to update the parameters of the speech enhancement model according to the multiple directional beam output signals, the null-notch beam output signal and the clean speech signal.

10. A headset, characterized in that: The headset comprises a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to perform the method of claim 4.

Images