A voice enhancement method, apparatus and electronic device

Abstract

The application discloses a speech enhancement method, device and electronic equipment, the method comprises the following steps: acquiring speech recognition confidence and a to-be-enhanced speech signal, and converting the to-be-enhanced speech signal into a frequency domain signal, wherein the speech recognition confidence is a confidence degree representing that the speech signal contains a user speech instruction; calculating a noise power estimation value based on the speech recognition confidence and the frequency domain signal; and performing adaptive beamforming by using the frequency domain signal and the noise power estimation value to obtain an enhanced speech signal. According to the technical scheme provided by the application, the speech recognition confidence of the speech signal is introduced to adjust the noise power estimation value, so that a more accurate noise power estimation value is obtained, and then the adaptive beamforming algorithm is executed based on the noise power estimation value, so that the speech enhancement effect can be improved in a low signal-to-noise ratio scene, and a more reliable enhanced speech signal is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of signal processing, in particular to a speech enhancement method, device and electronic equipment. BACKGROUND

[0002] With the progress of society and the development of technology, speech has gradually become a convenient human-computer interaction method, and more and more families have smart devices with speech interaction function. However, due to the influence of surrounding environment, device native noise and other factors, the received signal of the microphone of the smart device is usually contaminated by noise, and the speech intelligibility is reduced, resulting in an unsatisfactory speech recognition result and poor user experience.

[0003] At present, the smart device usually collects user speech through a microphone array. On the basis of the microphone array, the sound from a certain direction can be enhanced or suppressed through sound source positioning and beamforming method, so as to enhance the useful sound signal in the noise environment in a directional manner, and suppress noise and improve speech intelligibility. The current speech enhancement method with better effect is adaptive beamforming. Adaptive beamforming relies on adaptive noise estimation and sound source positioning. The more accurate the noise estimation and sound source positioning are, the better the performance of adaptive beamforming is. However, on devices with high noise, such as smart air conditioners and smart range hoods, the signal-to-noise ratio is low, which affects the accuracy of noise estimation and target positioning, and thus affects the speech enhancement effect of adaptive beamforming. Therefore, how to improve the speech enhancement effect in the scene with low signal-to-noise ratio is a problem to be solved. SUMMARY

[0004] Therefore, the embodiments of the present application provide a speech enhancement method, device and electronic equipment, so as to improve the speech enhancement effect in the scene with low signal-to-noise ratio.

[0005] According to a first aspect, the embodiments of the present application provide a speech enhancement method, which comprises: acquiring a speech recognition confidence and a to-be-enhanced speech signal, and converting the to-be-enhanced speech signal into a frequency domain signal, wherein the speech recognition confidence is a confidence degree indicating that the speech signal contains a user speech instruction; calculating a noise power estimation value based on the speech recognition confidence and the frequency domain signal; and performing adaptive beamforming by using the frequency domain signal and the noise power estimation value to obtain an enhanced speech signal.

[0006] Optionally, before the step of calculating the noise power estimation value based on the speech recognition confidence and the frequency domain signal, the method further comprises: judging whether the speech recognition confidence is greater than a preset threshold; and if the speech recognition confidence is greater than the preset threshold, performing the step of calculating the noise power estimation value based on the speech recognition confidence and the frequency domain signal.

[0007] Optionally, the calculating the noise power estimation based on the speech recognition confidence and the frequency domain signal comprises: performing smoothing processing on each frequency point in each microphone channel of the frequency domain signal to obtain a frequency domain smoothed power spectrum of each frequency point in each microphone channel; tracking an initial noise power of each frequency point in each microphone channel based on the obtained frequency domain smoothed power spectrum; calculating a speech existence probability of each frequency point in each microphone channel according to the obtained frequency domain smoothed power spectrum, the initial noise power and the speech recognition confidence; and calculating a noise power estimation of each frequency point in each microphone channel from the frequency domain signal based on the speech existence probability of each frequency point in each microphone channel.

[0008] Optionally, the tracking the initial noise power of each frequency point in each microphone channel based on the obtained frequency domain smoothed power spectrum comprises: comparing the frequency domain smoothed power spectrum of a current frequency point with an initial noise power of a corresponding position frequency point in a previous time speech signal in a current microphone channel of the current time to-be-enhanced speech signal; and taking a smaller value in the comparison result as the initial noise power of the current frequency point; wherein the frequency domain smoothed power spectrum of the current frequency point and the initial noise power of the corresponding position frequency point in the previous time speech signal participating in the comparison are reset in a preset period.

[0009] Optionally, the performing adaptive beamforming on the frequency domain signal and the noise power estimation to obtain an enhanced speech signal comprises: calculating a noise covariance matrix of each frequency point based on the frequency domain signal and the noise power estimation of each frequency point in each microphone channel; calculating an adaptive beamforming weight vector corresponding to each angle by using a plurality of preset angle steering vectors and the noise covariance matrix of each frequency point; decomposing the frequency domain signal by using the adaptive beamforming weight vector corresponding to each angle to obtain a single-path enhanced signal corresponding to each angle; and fusing the single-path enhanced signal corresponding to each angle to obtain the enhanced speech signal.

[0010] Optionally, the fusing the single-path enhanced signal corresponding to each angle to obtain the enhanced speech signal comprises: inputting the single-path enhanced signal corresponding to each angle into a preset neural network model to output a sound source position confidence corresponding to each angle; and performing weighted fusion on the sound source position confidence corresponding to each angle and the single-path enhanced signal corresponding to each angle to obtain the enhanced speech signal.

[0011] Optionally, the method further comprises: performing speech recognition on the enhanced speech signal to obtain a speech instruction; updating the speech recognition confidence based on the recognized speech instruction; and wherein the updated speech recognition confidence is used for participating in a speech enhancement process of a next time speech signal.

[0012] According to a second aspect, an embodiment of the present application provides a speech enhancement device, the device comprising: a signal initialization module configured to obtain a speech recognition confidence and a to-be-enhanced speech signal, and convert the to-be-enhanced speech signal into a frequency domain signal, wherein the speech recognition confidence is a confidence degree indicating whether the speech signal contains a user speech instruction; a noise power estimation module configured to calculate a noise power estimation value based on the speech recognition confidence and the frequency domain signal; and a speech enhancement module configured to perform adaptive beamforming using the frequency domain signal and the noise power estimation value to obtain an enhanced speech signal.

[0013] According to a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, which are in communication connection with each other, and the memory stores computer instructions, and the processor executes the computer instructions to perform the method of the first aspect or any one of the optional implementation manners of the first aspect.

[0014] According to a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to make the computer execute the method of the first aspect or any one of the optional implementation manners of the first aspect.

[0015] The technical scheme provided in the present application has the following advantages:

[0016] The technical scheme provided in the present application creatively introduces a speech recognition confidence for adaptive beamforming. The speech recognition confidence is a confidence degree indicating whether the current speech signal contains a real speech instruction based on the result of text recognition when the speech is recognized by text. First, the to-be-enhanced speech signal is converted into a frequency domain signal to facilitate subsequent execution of a beamforming algorithm. In the adaptive beamforming algorithm, a noise power estimation value needs to be applied, that is, the noise power contained in the current signal is estimated. The embodiment of the present application adjusts the noise power estimation value by introducing the speech recognition confidence of the speech signal, so as to obtain a more accurate noise power estimation value. Finally, the adaptive beamforming algorithm is executed using the frequency domain signal obtained in the first step and the noise power estimation value obtained in the second step, so as to enhance the effect of speech enhancement and obtain a more reliable enhanced speech signal.

[0017] In addition, in the adaptive beamforming algorithm, the embodiment further estimates the real sound source position through the pre-trained neural network model, obtains a plurality of sound source position confidences, and then performs adaptive beamforming through the sound source position confidences, thereby improving the accuracy of sound source positioning and further improving the reliability of the enhanced speech signal. BRIEF DESCRIPTION OF DRAWINGS

[0018] The features and advantages of the present application will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, which are schematic and should not be interpreted in a limiting sense, in which:

[0019] Figure 1 A schematic diagram of steps of a speech enhancement method in an embodiment of the present application is shown;

[0020] Figure 2 A schematic diagram of a flow of a speech enhancement method in an embodiment of the present application is shown;

[0021] Figure 3 Another schematic diagram of a flow of a speech enhancement method in an embodiment of the present application is shown;

[0022] Figure 4 A schematic diagram of a structure of a speech enhancement device in an embodiment of the present application is shown;

[0023] Figure 5 A schematic diagram of a structure of an electronic device in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the drawings of the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0025] Reference will now be made to Figure 1 In an embodiment, a speech enhancement method specifically includes the following steps:

[0026] Step S101: Obtain a speech recognition confidence and a speech signal to be enhanced, and convert the speech signal to be enhanced into a frequency domain signal. The speech recognition confidence is a confidence degree representing that a user speech instruction is contained in the speech signal.

[0027] Step S102: calculate a noise power estimation value based on the speech recognition confidence and the frequency domain signal.

[0028] Step S103: perform adaptive beamforming using the frequency domain signal and the noise power estimation value to obtain an enhanced speech signal.

[0029] Specifically, when performing a speech recognition task, the speech signal is used to recognize a speech instruction in the speech signal and convert the speech instruction into relevant text. Generally, a speech recognition confidence index can be calculated based on the recognized speech instruction. The speech recognition confidence refers to a posterior probability of a user speech instruction obtained through a speech recognition model, and has a value range of 0-1. That is, when the speech recognition confidence is greater than zero, it indicates that the user has a high probability of interacting with the device. When the speech recognition confidence exceeds a preset threshold, it is considered that the device recognizes a user instruction, and the device needs to perform a corresponding operation. In the embodiment of the present application, the speech recognition confidence is introduced into an adaptive beamforming algorithm, so as to enhance the effect of speech enhancement.

[0030] In the embodiment of the present application, a speech signal is first collected through a microphone array. The microphone array of the collection device can be a regular array such as a linear array, a planar array, a circular array, a stereo array and the like, or an irregular array. The number of microphone channels of the microphone array of the device is at least two. The structure information of the microphone array of the device is known by default, that is, the array configuration and the microphone spacing are known. Then, the speech signal is framed and windowed. In the embodiment, each frame of signal is a segment of speech signal to be enhanced. Then, each frame of speech signal to be enhanced is converted from a time domain signal to a frequency domain signal for subsequent algorithm processing. For example, the microphone array of the device has M microphones. The Fourier transform is performed on the t-th frame of speech signal to be enhanced to obtain a frequency domain signal:

[0031] Y(t,f)=[Y1(t,f),Y2(t,f),...,Y M (t,f)] T

[0032] Y m (t,f)(m=1,2,...,M) represents the frequency domain signal of the m-th microphone channel of the microphone array at the t-th frame and the f-th time-frequency point. If the sampling rate is f s , the frame length is N, the frame shift is N / 2, the window length is N, and the Fourier transform point number is N, then the corresponding frequency domain signal has N / 2+1 frequency points, and the frequency interval is f s / N. T represents a transposition operation; the window function can be a commonly used window function such as a Hamming window or a Hanning window.

[0033] The frequency domain signal contains device native noise and possibly user voice instructions (for example, the user voice instruction is a wake-up word spoken through a wake-up device in a smart home scenario). When the user issues a voice instruction, since the frequency energy of the signal received by the microphone array is the sum of the energy of the voice signal and the noise signal, assuming that the noise signal is stationary and fixed, when the signal-to-noise ratio is high, the minimum noise power on each frequency point can be found in a time window through minimum tracking, but in some low signal-to-noise ratio scenarios, most of the frequency points will not meet the condition of the presence of voice and will be determined as the absence of voice, and in fact these frequency points still have the power of the voice signal, so the noise signal power estimation obtained will inevitably contain the voice component, resulting in distortion of the voice obtained by adaptive beamforming, affecting the recognition result. When the embodiment estimates the noise power based on the frequency domain signal, the voice recognition confidence is fed back to the power estimation process, so that the voice recognition confidence can be used to modify the condition of the presence of voice, thereby reducing the voice component in the noise signal power estimation, and further obtaining a more accurate noise power estimation value (the voice recognition confidence received in the embodiment includes but is not limited to the voice recognition confidence of the previous frame of voice signal, the voice recognition confidence of the current frame of voice signal, and the voice recognition confidence of the voice signal of several frames before). Finally, the frequency domain signal and the noise power estimation value are used to execute the adaptive beamforming algorithm in the prior art, so as to output an enhanced voice signal with higher voice intelligibility and recognition rate, thereby improving the user experience.

[0034] Specifically, in an embodiment, the step S102 specifically includes the following steps:

[0035] Step one: smoothing each frequency point in each microphone channel of the frequency domain signal to obtain the frequency domain smoothed power spectrum of each frequency point in each microphone channel.

[0036] Step two: tracking the initial noise power of each frequency point in each microphone channel based on the obtained frequency domain smoothed power spectrum.

[0037] Step three: calculating the voice presence probability of each frequency point in each microphone channel according to the obtained frequency domain smoothed power spectrum, the initial noise power, and the voice recognition confidence.

[0038] Step four: calculating the noise power estimation value of each frequency point in each microphone channel from the frequency domain signal based on the voice presence probability of each frequency point in each microphone channel.

[0039] Specifically, first, each frequency point in each microphone channel of the frequency domain signal needs to be smoothed to obtain the frequency domain smoothed power spectrum of each frequency point in each microphone channel:

[0040] S m (t,f)=αa S m (t-1,f)+(1-α a )|Y m (t,f)| 2

[0041] wherein, S m (t,f) represents the frequency domain smoothed power spectrum of the mth microphone channel at the tth frame, the fth time-frequency point, the initial value S m (0,f) is the |Y m (t,f)| 2 of the 0th frame, S m (t-1,f) represents the frequency domain smoothed power spectrum of the mth microphone channel at the t-1th frame, the fth time-frequency point, |Y m (t,f)| 2 represents the frequency domain signal of the mth microphone channel at the tth frame, the fth time-frequency point, α a represents the frequency domain power spectrum smoothing factor, and |·| represents the absolute value operation.

[0042] Then, according to the obtained frequency domain smoothed power spectrum of each frequency point, in each microphone channel, the frequency domain smoothed power spectrum of a certain frequency point of the current frame and the initial noise power value of the corresponding position frequency point of the last frame are compared in time sequence, and then the smaller value is taken, so as to obtain the initial noise power corresponding to a certain frequency point of the current frame. After the comparison operation of the current frequency point of the current frame is completed, the comparison operation of the corresponding position frequency point of the next frame is performed, until the comparison operation of all frequency points is completed, so as to obtain the initial noise power of each frequency point. The formula is as follows:

[0043] s m,min (t,f)=min{s m,min (t-1,f),S m (t,f)}

[0044] wherein, s m,min (t,f) represents the initial noise power of the mth microphone channel at the tth frame, the fth time-frequency point, the initial value s m,min (0,f) is the max(S m (t,f)) of the 0th frame, and min{·} represents the minimum value operation.

[0045] For example: assuming that X represents a speech signal and N represents a noise signal, and the noise signal remains unchanged as N, then the frequency domain signal of the silent frame is only the noise signal; if the next frame is a speech frame, then the frequency domain signal = X + N >= N, that is, the power value obtained after taking the minimum value of the signal power of the previous frame is still N, so that the initial noise power of each frequency point is tracked based on the minimum power value.

[0046] At this time, the initial noise power obtained is likely to contain the component of the voice instruction. Thus, the next step is to calculate the probability of the presence of voice in each frequency point based on the initial noise power to adjust the noise power of each frequency point.

[0047] The prior art method for calculating the probability of the presence of voice in each frequency point based on the initial noise power is to determine whether the real voice exists in each frequency point, and then calculate the probability of the presence of voice according to the real voice existence. The prior art formula is:

[0048]

[0049] wherein, indicates whether the voice exists in the mth microphone channel at the tth frame and the fth time-frequency point, and σ(f) indicates the threshold value of the fth frequency point. That is, when the ratio of the frequency domain smoothed power spectrum to the initial noise power exceeds a certain threshold, it is considered that the voice signal exists. In other words, S m (t,f) = X + N, wherein X is a clean voice signal, N is a noise signal, and when X+N / N exceeds the threshold value, it is considered that the voice signal exists, S m (t,f) = N, and when N / N does not exceed the threshold, it is considered that the voice signal does not exist. The problem of the prior art is that the threshold value is the same for different signal-to-noise ratios and there is no fixed measurable index. When the signal-to-noise ratio is low, X+N / N is likely to not satisfy the threshold value and thus is determined to be voiceless, but the actual voice exists.

[0050] And the embodiment of the present application modifies the threshold value by introducing the voice recognition confidence in the threshold decision of the prior art:

[0051]

[0052] wherein, indicates whether the voice exists in the mth microphone channel at the tth frame and the fth time-frequency point, and σ(f) indicates the threshold value of the fth frequency point, and α s (t-1) indicates the voice recognition confidence at the t-1th frame (the voice recognition confidence of the voice signal of the previous frame is used in the embodiment), the initial value α s (0) is 1, and β indicates the voice recognition confidence coefficient. When the ratio of the frequency domain smoothed power S m (t,f) of the mth microphone channel at the tth frame and the fth time-frequency point to the initial noise power s m,min (t,f) of the mth microphone channel at the tth frame and the fth time-frequency point exceeds the threshold value, it is determined that the voice exists, otherwise it is determined that the voice does not exist.

[0053] Speech recognition confidence also reflects the presence of speech to some extent. Therefore, it is hoped that the threshold value, βα, can be reduced through speech recognition confidence. s If the value is less than 1, the threshold value can be reasonably reduced by the confidence level of speech recognition, and the presence of speech can be determined. This improves the problem of inaccurate estimation of noise signals in low signal-to-noise ratio scenarios to a certain extent.

[0054] After obtaining the result of determining whether the speech exists, the probability of speech presence at each frequency point is calculated using the following formula:

[0055]

[0056] Where, p m (t,f) represents the probability of speech presence at the f-th time-frequency point in the t-th frame of the m-th microphone channel, p m (t-1,f) represents the probability of speech presence at the f-th time-frequency point in the (t-1)-th frame of the m-th microphone channel, α p This indicates that there is a probability smoothing factor in the speech.

[0057] Next, based on the obtained speech presence probability, update the noise smoothing factor for each frequency point of each channel:

[0058] α m (t,f)=α d +(1-α d )p m (t,f)

[0059] Where, α m (t,f) represents the noise smoothing factor of the m-th microphone channel at the f-th time-frequency point in frame t, α d This represents the noise smoothing factor coefficient.

[0060] Finally, based on the received signal's frequency domain power spectrum and noise smoothing factor, the noise power estimate for each frequency point of each microphone channel in the frequency domain signal is obtained. This allows for the estimation of the pure noise power, excluding speech command components, as much as possible.

[0061]

[0062] Among them, noise signal power estimate initial value For frame 0, |Y m (t,f)| 2 .

[0063] Specifically, in one embodiment, step two above includes the following steps:

[0064] Step five: in the current microphone channel of the speech signal to be enhanced at the current time, comparing the frequency domain smoothed power spectrum of the current frequency point with the initial noise power of the corresponding position frequency point in the speech signal at the last time;

[0065] Step six: taking the smaller value in the comparison result as the initial noise power of the current frequency point; wherein the frequency domain smoothed power spectrum of the current frequency point participating in comparison and the initial noise power of the corresponding position frequency point in the speech signal at the last time are reset in a preset period.

[0066] Specifically, in the embodiment, the simple mutation noise tracking is realized based on the time window and the parameter reset mechanism, so as to reduce the search complexity of the mutation noise.

[0067] Specifically, the embodiment of the application further provides a temporary power minimum value as an intermediate quantity for tracking the mutation noise. The specific implementation method is:

[0068] s m,min (t,f)=min{s m,min (t-1,f),S m (t,f)}

[0069] s m,tmp (t,f)=min{s m,tmp (t-1,f),S m (t,f)}

[0070] Wherein, s m,min (t,f) represents the initial noise power of the mth microphone channel at the tth frame and the fth time-frequency point, the initial value s m,min (0,f) is max(S m (t,f)) of the 0th frame, s m,tmp (t,f) represents the temporary minimum value of the power of the mth microphone channel at the tth frame and the fth time-frequency point, the initial value s m,tmp (0,f) is max(S m (t,f)) of the 0th frame, and min{·} represents the minimum value operation.

[0071] After performing L times of frequency domain power minimum value tracking, the initial noise power s m,min (t,f) of the mth microphone channel at the tth frame and the fth time-frequency point and the temporary minimum value s m,tmp (t,f) of the power of the mth microphone channel at the tth frame and the fth time-frequency point are reset:

[0072] s m,min (t,f)=min{s m,tmp (t-1,f),S m (t,f)}

[0073] s m,tmp (t,f)=S m (t,f)

[0074] wherein, L represents the length of time window (i.e. preset period). After the update, the frequency domain power minimum value s m,min (t,f) of the mth microphone channel at the tth frame and the fth time-frequency point is obtained.

[0075] For example, assuming that the noise signal power of the previous period is N1 and remains unchanged, the noise signal power of the next period is N2 and remains unchanged, and N2>N1, assuming that the current state is an ideal state "no voice instruction only noise signal", i.e. Sm(t,f) of the previous period is N1, Sm(t,f) of the next period is N1, if no time window is set for preset period resetting, no power temporary minimum value is needed, only comparison and tracking according to the method in step two is needed, but the comparison of each frequency point is always taking the minimum value in the frequency domain smoothed power spectrum, so only N1 can be tracked, and the larger amplitude noise N2 after mutation cannot be tracked. However, after setting the time window L, at the Lth resetting, the initial noise power can take the larger amplitude noise N2 after mutation through the power temporary minimum value, so as to track the mutated noise. In other words, assuming that the preset period is two frames, the noise power of the frequency domain signal of the previous period is 1, then s m,min (t,f) and s m,tmp (t,f) tracked noise power is 1, assuming that the noise power mutates to 2 at the third tracking, through the above resetting formula, s m,tmp (t,f) becomes 2, but s m,min (t,f) is still 1, then continue for one period, the fifth time, through the above resetting formula s m,min (t,f)=min{s m,tmp (t-1,f),S m (t,f)} resetting, s m,min (t,f) also becomes 2, i.e. the initial noise power is tracked after a delay of two periods. Through the method provided in the embodiment, there is no need to find the mutated power from the frequency domain signal after all the initial noise power is tracked, so there is no need to occupy the storage resource of the machine device to store the frequency domain signal. Therefore, the search complexity and cost of the mutated noise are reduced.

[0076] Specifically, in an embodiment, the step S103 specifically includes the following steps:

[0077] Step seven: calculating the noise covariance matrix of each frequency point based on the frequency domain signal and the noise power estimation value of each frequency point of each microphone channel.

[0078] Step eight: using the preset angle guide vector and the noise covariance matrix of each frequency point, the adaptive beam forming weight vector corresponding to each angle is calculated.

[0079] Step nine: using the adaptive beam forming weight vector corresponding to each angle, the frequency domain signal is decomposed to obtain the single path enhanced signal corresponding to each angle.

[0080] Step ten: the single path enhanced signals corresponding to each angle are fused to obtain the enhanced speech signal.

[0081] Specifically, in the embodiment, after obtaining the more accurate noise power estimation value through the above step S102, the adaptive beam forming algorithm is executed based on the accurate noise power estimation value, and then the signals of the angles where each user may appear are enhanced, and the speech enhancement effect with higher accuracy is realized. Specifically, it includes:

[0082] In the embodiment of the application, the preset angle refers to discretizing the area where the user to be observed in advance, for example, dividing 0°-180° into N angles at a fixed angle interval, considering that the user only appears in the N directions, and only performing adaptive beam forming on the N directions. The basic criterion of minimum variance distortionless response adaptive beam forming is to minimize the average power of the signal output after beam forming. Since the average power of the signal output is the sum of the average power of the speech signal and the average power of the noise signal, to minimize the average power of the output signal and to make the speech signal distortionless, in fact, it is to minimize the average power of the noise in the signal. Therefore, the noise covariance matrix of the adaptive beam forming should contain as few speech signal components as possible and contain as many noise signal components as possible. Therefore, in step S102, the noise power estimation value is corrected by the speech recognition confidence, so that the noise power estimation value contains as few speech powers as possible in the low signal-to-noise ratio environment, thereby reducing speech distortion.

[0083] In a specific embodiment, the phase of the speech signal frequency domain data of each channel and each frequency point of the frequency domain signal is first extracted:

[0084]

[0085] Wherein, Φ m (m,t,f) represents the phase of the mth microphone channel of the microphone array at the tth frame and the fth time-frequency point, Re(·) represents the real part operation, Im(·) represents the imaginary part operation, and arctan(·) represents the inverse tangent operation.

[0086] Then, using the current frame phase Φ m (m,t,f) and the noise power estimation value the noise signal frequency domain data of each channel and each frequency point is obtained:

[0087]

[0088] wherein, denotes the noise signal frequency domain data of the mth microphone channel of the microphone array at the tth frame, the fth time-frequency point, and j denotes the imaginary unit.

[0089] Secondly, the noise signal of the current frame is used to calculate the noise covariance matrix of each frequency point:

[0090]

[0091] wherein, R NN (t,f) denotes the noise covariance matrix of the microphone array at the tth frame, the fth time-frequency point, and a R denotes the noise covariance matrix smoothing factor, the initial value R NN (0,f) is the noise covariance matrix of the 0th frame. (·) H denotes the transpose conjugate operation.

[0092] Next, the steering vector generated by the preset angle and the noise covariance matrix are used to calculate the adaptive beamforming weight vector corresponding to each angle:

[0093]

[0094] wherein, ω n (t,f) denotes the adaptive beamforming weight vector corresponding to the n (n = 1, 2,..., N) preset angle at the tth frame, the fth time-frequency point, and θ n denotes the n preset angle, and a(θ n ,f) denotes the steering vector corresponding to the n preset angle and the fth time-frequency point.

[0095] Then, the adaptive beamforming weight vector and the frequency domain data of the current frame are used for frequency domain filtering to obtain the single-channel adaptive beamforming frequency domain enhancement signal corresponding to each angle:

[0096] Y′ n (t,f) = ω n (t,f) H Y(t,f)

[0097] wherein, Y′ n (t,f) denotes the single-channel adaptive beamforming frequency domain enhancement signal corresponding to the n preset angle at the tth frame, the fth time-frequency point.

[0098] ​Then, the individual single-path enhanced signals are fused, including but not limited to direct addition, weighted fusion and the like, to obtain a single-path enhanced frequency domain signal of adaptive beamforming. Finally, the single-path enhanced frequency domain signal is converted to the time domain through inverse Fourier transform to obtain an enhanced speech signal.

[0099] Specifically, in an embodiment, the above step ten specifically comprises the following steps:

[0100] Step eleven: inputting the single-path enhanced signals corresponding to the respective angles into a preset neural network model to output sound source position confidence degrees corresponding to the respective angles.

[0101] Step twelve: performing weighted fusion on the sound source position confidence degrees corresponding to the respective angles and the single-path enhanced signals corresponding to the respective angles to obtain an enhanced speech signal.

[0102] Specifically, although the enhanced signals of the respective angles in which the user may appear have been obtained through the above steps, the angle in which the user actually appears is unknown, i.e., the sound source position is not clear enough. In view of this technical problem, the embodiment of the present application proposes a scheme of optimizing the sound source position through a neural network model. The embodiment analyzes through the neural network the angle in which the user actually appears to obtain a more accurate sound source position, so that a weighted sum is performed based on the result output by the neural network to output a single-path signal, thereby further improving the speech enhancement effect. The specific method is as follows:

[0103] The single-path enhanced signals corresponding to the preset angles generated in the above step nine are input into a neural network model that has been pre-trained to obtain confidence degrees corresponding to the preset angles:

[0104] α n (t)=E(|Y n ′(t,f)| 2 )

[0105] wherein E(·) represents an angle classification neural network model, which can be realized through a recurrent neural network or a convolutional neural network and the like, and α n (t) represents a confidence degree corresponding to the nth preset angle, and the angle estimation result can be obtained through the maximum value of the confidence degrees corresponding to the preset angles.

[0106] Subsequently, a single-path enhanced frequency domain signal is obtained according to the single-path enhanced signals corresponding to the preset angles and the confidence degrees:

[0107]

[0108] Finally, the enhanced frequency domain signal is subjected to inverse Fourier transform to convert the enhanced speech signal in the time domain, thereby further improving the speech enhancement effect.

[0109] Specifically, asFigure 2 As shown in the figure, in an embodiment, before step S102, the method further comprises the following steps:

[0110] Step thirteen: judging whether the speech recognition confidence is greater than a preset threshold.

[0111] Step fourteen: if the speech recognition confidence is greater than the preset threshold, executing step S102.

[0112] Specifically, in the embodiment of the present application, when the speech recognition module score is greater than zero, it indicates that the user has a high probability of interacting with the device, and the device recognizes the preset wake-up word or related command word, so the obtained noise power estimation value is used to enhance the user voice, which can reduce the noise component in the user voice, thereby improving the recognition rate and improving the user experience; when the speech recognition module score is equal to zero, it indicates that the user is not interacting with the device, and the device does not recognize the preset wake-up word or related command word, so only the signal of the first channel of the microphone array is transmitted to the speech recognition module for continuous speech recognition, which can further reduce the operation amount. In a specific embodiment, according to the last frame speech recognition confidence α s (t-1), if adaptive beamforming is needed, then based on the current frame speech signal frequency domain data Y m (t,f) and the noise signal power estimation value adaptive beamforming is performed; if adaptive beamforming is not needed, then the first channel speech signal frequency domain data Y1(t,f) of the current frame is transmitted to the speech recognition module for speech recognition, so that the obtained current frame speech recognition confidence is used for processing of the next frame to be enhanced signal.

[0113] Specifically, in an embodiment, as Figure 3 shown, the speech enhancement method provided by the embodiment of the present application further comprises the following steps:

[0114] Step fifteen: performing speech recognition on the enhanced speech signal to obtain a voice command;

[0115] Step sixteen: updating the speech recognition confidence based on the recognized voice command; wherein the updated speech recognition confidence is used to participate in the speech enhancement process of the next time voice signal.

[0116] Specifically, in the embodiment, in combination with steps 13 to 14, the enhanced speech signal is extracted for feature parameters, sent to the recognition network for speech recognition, and the speech recognition confidence is returned to the noise power estimation module for participating in the speech enhancement process of the next moment of speech signal, thereby forming a speech enhancement closed loop, so that the embodiment performs full-automatic speech enhancement on the current signal through the speech recognition confidence of adjacent frames of the current speech signal, forms a speech enhancement semi-automatic system, and the speech recognition confidence of adjacent frames has strong reference in actual application scenarios, and the speech enhancement efficiency and accuracy are improved simultaneously through the embodiment.

[0117] In the embodiment, the single-channel frequency domain signal input to the speech recognition network can be an unenhanced signal or an enhanced signal. The extraction of feature parameters refers to extracting a set of parameters that can describe the characteristics of speech signals or data from the signal for speech recognition. The recognition network refers to a speech recognition system built by cascading a neural network and a hidden Markov model and using augmented speech. Speech data augmentation refers to first adding reverberation to clean audio to obtain reverberation audio, then calculating the signal-to-noise ratio coefficient according to the specified signal-to-noise ratio range, and then superimposing noise of the corresponding proportion on the reverberation audio to obtain noisy audio, and finally generating noisy audio with random amplitude coefficients according to the specified amplitude range, i.e., augmented speech. In a specific embodiment, first, the Fbank (Filter bank) feature parameters of the power spectrum of the current frame of single-channel frequency domain signal are extracted, then the feature parameters are sent to the pre-trained recognition network for recognition to obtain the speech recognition confidence of the current frame, and the speech recognition confidence is returned to the noise signal power estimation module, and finally, whether the user instruction is recognized according to the speech recognition confidence of the current frame is determined, i.e., when the speech recognition confidence exceeds the preset threshold, it is considered that the user instruction is recognized, and the corresponding adaptive beamforming operation is performed, and if the speech recognition confidence does not exceed the threshold, the adaptive beamforming is not performed.

[0118] Through the above steps, the technical scheme provided by the present application creatively introduces the speech recognition confidence for adaptive beamforming. The speech recognition confidence is the confidence that the current speech signal contains a real voice command based on the result output by the text recognition when the speech is text-recognized. First, the obtained to-be-enhanced speech signal is converted into a frequency domain signal to facilitate subsequent execution of the beamforming algorithm. In the adaptive beamforming algorithm, the noise power estimation value needs to be applied, that is, the noise power contained in the current signal is estimated. The embodiment of the present application adjusts the noise power estimation value by introducing the speech recognition confidence of the speech signal, so that a more accurate noise power estimation value is obtained. Finally, the adaptive beamforming algorithm is executed by using the frequency domain signal obtained in the first step and the noise power estimation value obtained in the second step, so that the effect of speech enhancement is improved, and a more reliable enhanced speech signal is obtained.

[0119] In addition, in the adaptive beamforming algorithm, the embodiment further estimates the real sound source position by using the pre-trained neural network model, obtains a plurality of sound source position confidences, and then performs adaptive beamforming by using the sound source position confidences, so that the accuracy of sound source positioning is improved, and the reliability of the enhanced speech signal is further improved. In addition, the embodiment takes the enhanced speech signal as a to-be-enhanced signal, and cyclically executes the speech enhancement algorithm, so that the reliability of the enhanced speech signal is further improved.

[0120] As shown in Figure 4 The embodiment further provides a speech enhancement device, and the device comprises:

[0121] The signal initialization module 101 is configured to obtain the speech recognition confidence and the to-be-enhanced speech signal, and convert the to-be-enhanced speech signal into a frequency domain signal. The speech recognition confidence is the confidence that the speech signal contains a user voice command. For details, refer to the related description of step S101 in the method embodiment, which will not be repeated here.

[0122] The noise power estimation module 102 is configured to calculate the noise power estimation value based on the speech recognition confidence and the frequency domain signal. For details, refer to the related description of step S102 in the method embodiment, which will not be repeated here.

[0123] The speech enhancement module 103 is configured to perform adaptive beamforming by using the frequency domain signal and the noise power estimation value to obtain an enhanced speech signal. For details, refer to the related description of step S103 in the method embodiment, which will not be repeated here.

[0124] The speech enhancement device provided by the embodiment of the present application is used to execute the speech enhancement method provided by the above embodiment, and the implementation manner and principle are the same. For details, refer to the related description of the method embodiment, which will not be repeated here.

[0125] Through the cooperation of the above various components, the technical scheme provided by the application creatively introduces voice recognition confidence for adaptive beamforming. The voice recognition confidence is the confidence that the current voice signal contains a real voice instruction based on the result output by the text recognition when the voice is recognized as text. First, the obtained voice signal to be enhanced is converted into a frequency domain signal to facilitate subsequent execution of the beamforming algorithm. In the adaptive beamforming algorithm, the noise power estimation value needs to be applied, that is, the noise power contained in the current signal is estimated. The embodiment of the application adjusts the noise power estimation value by introducing the voice recognition confidence of the voice signal, so as to obtain a more accurate noise power estimation value. Finally, the adaptive beamforming algorithm is executed by using the frequency domain signal obtained in the first step and the noise power estimation value obtained in the second step, so as to strengthen the effect of voice enhancement and obtain a more reliable enhanced voice signal.

[0126] In addition, in the adaptive beamforming algorithm, the embodiment further estimates the real sound source position by using the pre-trained neural network model to obtain a plurality of sound source position confidences, and then performs adaptive beamforming by using the sound source position confidences, so as to improve the accuracy of sound source positioning and further improve the reliability of the enhanced voice signal. In addition, the embodiment takes the enhanced voice signal as a signal to be enhanced, and repeatedly executes the voice enhancement algorithm to further improve the reliability of the enhanced voice signal.

[0127] Figure 5 An electronic device of an embodiment of the application is shown. The device includes a processor 901 and a memory 902, which can be connected by a bus or other means, Figure 5 The bus connection is taken as an example.

[0128] The processor 901 can be a central processing unit (CPU). The processor 901 can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, or a combination of the above-mentioned chips or components.

[0129] The memory 902, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the methods in the above method embodiments. The processor 901 performs various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 902, that is, implements the methods in the above method embodiments.

[0130] The memory 902 can include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function; and the data storage area can store data created by the processor 901 and the like. In addition, the memory 902 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory 902 can optionally include a memory disposed remotely with respect to the processor 901, which can be connected to the processor 901 through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0131] One or more modules are stored in the memory 902, and when executed by the processor 901, the methods in the above method embodiments are performed.

[0132] The above electronic device specific details can be understood in correspondence with the above method embodiments corresponding to the relevant description and effects, which will not be described here.

[0133] Those skilled in the art can understand that all or part of the processes in the above method embodiments can be completed by a computer program instructing related hardware, and the implemented program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD), etc. The storage medium can also include a combination of the above types of memories.

[0134] Although the embodiments of the present application are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A speech enhancement method, characterized by, The method comprises: acquiring a voice recognition confidence and a to-be-enhanced voice signal, and converting the to-be-enhanced voice signal into a frequency domain signal, wherein the voice recognition confidence is a confidence degree representing that a user voice instruction is contained in the voice signal; calculating a noise power estimation value based on the voice recognition confidence and the frequency domain signal, wherein the calculation of the noise power estimation value based on the voice recognition confidence and the frequency domain signal comprises: performing smoothing processing on each frequency point in each microphone channel of the frequency domain signal to obtain a frequency domain smoothed power spectrum of each frequency point in each microphone channel; tracking an initial noise power of each frequency point in each microphone channel based on the obtained frequency domain smoothed power spectrum; calculating a voice existence probability of each frequency point in each microphone channel according to the obtained frequency domain smoothed power spectrum, the initial noise power and the voice recognition confidence; the specific process is as follows: wherein, represents whether speech exists at the mth microphone channel in the tth frame, the fth time-frequency point, σ(f) represents a threshold value of the fth frequency point, and α s (t-1) represents a speech recognition confidence in the t-1th frame, the initial value α s (0) is 1, and β represents a speech recognition confidence coefficient; when the ratio of the frequency domain smoothed power S m (t,f) of the mth microphone channel in the tth frame, the fth time-frequency point to the initial noise power s m,min (t,f) of the mth microphone channel in the tth frame, the fth time-frequency point exceeds the threshold value, it is determined that speech exists, otherwise it is determined that speech does not exist; calculating a noise power estimation value of each frequency point in each microphone channel from the frequency domain signal based on the voice existence probability of each frequency point in each microphone channel; performing adaptive beamforming on the frequency domain signal and the noise power estimation value to obtain an enhanced voice signal.

2. The method of claim 1, wherein, Before the calculation of the noise power estimation value based on the voice recognition confidence and the frequency domain signal, the method further comprises: judging whether the voice recognition confidence is greater than a preset threshold; if the voice recognition confidence is greater than the preset threshold, performing the step of calculating the noise power estimation value based on the voice recognition confidence and the frequency domain signal.

3. The method of claim 2, wherein, The tracking of the initial noise power of each frequency point in each microphone channel based on the obtained frequency domain smoothed power spectrum comprises: comparing the frequency domain smoothed power spectrum of a current frequency point with an initial noise power of a corresponding position frequency point in a previous time voice signal in a current microphone channel of the to-be-enhanced voice signal at a current time; taking a smaller value in the comparison result as the initial noise power of the current frequency point; wherein the frequency domain smoothed power spectrum of the current frequency point and the initial noise power of the corresponding position frequency point in the previous time voice signal participating in the comparison are reset at a preset period.

4. The method of claim 2, wherein, The adaptive beamforming on the frequency domain signal and the noise power estimation value to obtain the enhanced voice signal comprises: calculating a noise covariance matrix of each frequency point based on the frequency domain signal and the noise power estimation value of each frequency point in each microphone channel; calculating an adaptive beamforming weight vector corresponding to each angle by using a plurality of preset angle steering vectors and the noise covariance matrix of each frequency point; decomposing the frequency domain signal by using the adaptive beamforming weight vector corresponding to each angle to obtain a single-path enhanced signal corresponding to each angle; fusing the single-path enhanced signal corresponding to each angle to obtain the enhanced voice signal.

5. The method of claim 4, wherein, The fusion of the single-path enhanced signal corresponding to each angle to obtain the enhanced voice signal comprises: inputting the single-path enhanced signal corresponding to each angle into a preset neural network model to output a sound source position confidence degree corresponding to each angle. The sound source position confidence corresponding to each angle and the single-path enhanced signal corresponding to each angle are weighted and fused to obtain the enhanced speech signal.

6. The method of claim 1, wherein, The method further comprises: performing speech recognition on the enhanced speech signal to obtain a voice instruction; updating the speech recognition confidence based on the recognized voice instruction; wherein the updated speech recognition confidence is used to participate in the voice enhancement process of the voice signal at the next time.

7. A speech enhancement device, characterized by The device comprises: a signal initialization module configured to obtain a speech recognition confidence and a voice signal to be enhanced, and convert the voice signal to be enhanced into a frequency domain signal, wherein the speech recognition confidence is a confidence degree indicating that the voice signal contains a voice instruction of a user; a noise power estimation module configured to calculate a noise power estimation value based on the speech recognition confidence and the frequency domain signal, wherein the calculation of the noise power estimation value based on the speech recognition confidence and the frequency domain signal comprises: performing smoothing processing on each frequency point in each microphone channel of the frequency domain signal to obtain a frequency domain smoothed power spectrum of each frequency point in each microphone channel; tracking an initial noise power of each frequency point in each microphone channel based on the obtained frequency domain smoothed power spectrum; calculating a voice existence probability of each frequency point in each microphone channel according to the obtained frequency domain smoothed power spectrum, the initial noise power, and the speech recognition confidence; and wherein, represents whether speech exists at the mth microphone channel in the tth frame, the fth time-frequency point, σ(f) represents a threshold value of the fth frequency point, and α s (t-1) represents a speech recognition confidence in the t-1th frame, the initial value α s (0) is 1, and β represents a speech recognition confidence coefficient; when the ratio of the frequency domain smoothed power S m (t,f) of the mth microphone channel in the tth frame, the fth time-frequency point to the initial noise power s m,min (t,f) of the mth microphone channel in the tth frame, the fth time-frequency point exceeds a threshold value, it is determined that speech exists, otherwise it is determined that speech does not exist; calculating a noise power estimation value of each frequency point in each microphone channel based on the voice existence probability of each frequency point in each microphone channel; a voice enhancement module configured to perform adaptive beamforming using the frequency domain signal and the noise power estimation value to obtain an enhanced speech signal.

8. An electronic device, comprising: comprise: a memory and a processor, which are communicatively connected to each other, and the memory stores computer instructions, and the processor executes the computer instructions to perform the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the computer to perform the method according to any one of claims 1-6.

Images