Speech enhancement evaluation method, speech enhancement evaluation model training method and device

Abstract

The present invention relates to a speech enhancement evaluation method, a training method and a device for a speech enhancement evaluation model. The speech enhancement evaluation method comprises: obtaining a noisy speech sample, wherein the noisy speech sample comprises an original speech sample and an original noise sample; inputting the noisy speech sample into a speech enhancement model to be tested to obtain enhanced noisy speech; inputting the enhanced noisy speech, the original speech sample and the original noise sample into the speech enhancement evaluation model to obtain an estimated enhanced speech and an estimated enhanced noise; and determining an evaluation value of the speech enhancement model to be tested according to the estimated enhanced speech and the estimated enhanced noise.

Description

Technical Field

[0001] The present disclosure relates to the field of speech enhancement technology, and in particular to a speech enhancement evaluation method, a training method and a device for a speech enhancement evaluation model. Background Art

[0002] The speech enhancement model is used to extract the purest original speech from noisy speech. However, in practical applications, any speech enhancement model will accidentally cut speech and / or residual noise, so an evaluation method is needed to evaluate the degree of accidentally cut speech and residual noise. To this end, the signal output by the speech enhancement model to be tested needs to be separated into two parts: speech component and residual noise component, so as to facilitate further performance analysis.

[0003] In the related art, there is an evaluation method based on SI-SNR (Scale-Invariant Source-to-Noise Ratio), which can separate the speech component and the residual noise component in the signal estimated by the speech enhancement model to be tested in the process of obtaining SI-SNR. However, this method has a disadvantage, that is, when the estimated signal passes through AGC (Automatic Gain Control), the speech component and the residual noise component separated by this method will produce a relatively large error. However, the speech enhancement model to be tested often includes an AGC module, and because the speech enhancement model to be tested is a whole black box and cannot be distributed for testing, the results that can be obtained are mostly signals after AGC, which makes this method unable to effectively measure the performance of the speech enhancement model to be tested. Summary of the invention

[0004] The present disclosure provides a speech enhancement evaluation method, a training method and a device for a speech enhancement evaluation model to at least solve the problem of how to effectively measure the performance of a speech enhancement model to be tested in the related art, and may not solve any of the above problems.

[0005] According to one aspect of the present disclosure, a training method for a speech enhancement evaluation model is provided, the training method comprising: obtaining an original speech sample and an original noise sample; performing enhancement processing on the original speech sample and the original noise sample respectively to obtain an enhanced speech sample and an enhanced noise sample; merging the enhanced speech sample and the enhanced noise sample to obtain an enhanced sample; inputting the enhanced sample, the original speech sample, and the original noise sample into the speech enhancement evaluation model to obtain at least one of an estimated enhanced speech and an estimated enhanced noise; determining a loss based on the estimated enhanced speech and the enhanced speech sample, and / or determining a loss based on the estimated enhanced noise and the enhanced noise sample; adjusting the parameters of the speech enhancement evaluation model based on the loss to train the speech enhancement evaluation model.

[0006] Optionally, the enhancing the original speech sample and the original noise sample respectively to obtain an enhanced speech sample and an enhanced noise sample comprises: acquiring gain data for simulating a speech enhancement operation; and based on the gain data, enhancing the original speech sample and the original noise sample respectively to obtain the enhanced speech sample and the enhanced noise sample.

[0007] Optionally, the gain data is the product of at least one complex gain, wherein, based on the gain data, the original speech sample and the original noise sample are enhanced respectively to obtain the enhanced speech sample and the enhanced noise sample, including: converting the original speech sample and the original noise sample from time domain signals to frequency domain signals; determining the product of the gain data and the original speech sample to obtain the enhanced speech sample; determining the product of the gain data and the original noise sample to obtain the enhanced noise sample.

[0008] Optionally, converting the original speech samples and the original noise samples from time domain signals to frequency domain signals includes: using short-time Fourier transform to convert the original speech samples and the original noise samples from time domain signals to multi-frame frequency domain signals.

[0009] Optionally, the step of inputting the enhanced sample, the original speech sample and the original noise sample into the speech enhancement evaluation model to obtain at least one of the estimated enhanced speech and the estimated enhanced noise comprises: inputting the enhanced sample, the original speech sample and the original noise sample into the speech enhancement evaluation model to obtain one of the estimated enhanced speech and the estimated enhanced noise; and removing one of the estimated enhanced speech and the estimated enhanced noise from the enhanced sample to obtain the other of the estimated enhanced speech and the estimated enhanced noise.

[0010] According to another aspect of the present disclosure, a speech enhancement evaluation method is provided for evaluating a speech enhancement model, the speech enhancement evaluation method comprising: obtaining a noisy speech sample, the noisy speech sample comprising an original speech sample and an original noise sample; inputting the noisy speech sample into a speech enhancement model to be tested to obtain enhanced noisy speech; inputting the enhanced noisy speech, the original speech sample and the original noise sample into a speech enhancement evaluation model to obtain estimated enhanced speech and estimated enhanced noise; and determining an evaluation value of the speech enhancement model to be tested based on the estimated enhanced speech and the estimated enhanced noise.

[0011] Optionally, the noisy speech sample is obtained by combining the original speech sample and the original noise sample.

[0012] Optionally, the step of inputting the enhanced noisy speech, the original speech sample, and the original noise sample into a speech enhancement evaluation model to obtain an estimated enhanced speech and an estimated enhanced noise comprises: inputting the enhanced noisy speech, the original speech sample, and the original noise sample into the speech enhancement evaluation model to obtain one of the estimated enhanced speech and the estimated enhanced noise; and removing one of the estimated enhanced speech and the estimated enhanced noise from the enhanced noisy speech to obtain the other of the estimated enhanced speech and the estimated enhanced noise.

[0013] Optionally, determining the evaluation value of the speech enhancement model to be tested based on the estimated enhanced speech and the estimated enhanced noise includes: determining the signal-to-noise ratio of the enhanced noisy speech based on the estimated enhanced speech and the estimated enhanced noise as the evaluation value of the speech enhancement model to be tested.

[0014] Optionally, the enhanced noisy speech, the estimated enhanced speech and the estimated enhanced noise all include multiple frames of frequency domain signals, and each frame of frequency domain signals includes complex signals at multiple center frequencies, wherein determining the signal-to-noise ratio of the enhanced noisy speech based on the estimated enhanced speech and the estimated enhanced noise includes: determining the sum of the amplitudes of all center frequencies of the estimated enhanced speech and the estimated enhanced noise in each frame respectively as the speech energy and noise energy of the enhanced noisy speech in the corresponding frame; determining the signal-to-noise ratio of the enhanced noisy speech in the corresponding frame according to the ratio of the speech energy and noise energy of the enhanced noisy speech in each frame; and determining the statistical value of the signal-to-noise ratio of the enhanced noisy speech in each frame as the signal-to-noise ratio of the enhanced noisy speech.

[0015] Optionally, the enhanced noisy speech, the estimated enhanced speech and the estimated enhanced noise are all time domain signals, wherein determining the signal-to-noise ratio of the enhanced noisy speech based on the estimated enhanced speech and the estimated enhanced noise includes: determining the arithmetic square root of the sum of the squares of the amplitude values ​​of the estimated enhanced speech and the estimated enhanced noise at each moment as the speech energy and the noise energy respectively; and determining the signal-to-noise ratio of the enhanced noisy speech based on the ratio of the speech energy to the noise energy.

[0016] Optionally, the speech enhancement evaluation model is trained using the training method according to the present disclosure.

[0017] According to another aspect of the present disclosure, a training device for a speech enhancement evaluation model is provided, the training device comprising: an acquisition unit, configured to acquire an original speech sample and an original noise sample; an enhancement unit, configured to perform enhancement processing on the original speech sample and the original noise sample respectively to obtain an enhanced speech sample and an enhanced noise sample; a merging unit, configured to merge the enhanced speech sample and the enhanced noise sample to obtain an enhanced sample; a separation unit, configured to input the enhanced sample, the original speech sample and the original noise sample into the speech enhancement evaluation model to obtain at least one of an estimated enhanced speech and an estimated enhanced noise; a calculation unit, configured to determine a loss according to the estimated enhanced speech and the enhanced speech sample, and / or determine a loss according to the estimated enhanced noise and the enhanced noise sample; a parameter adjustment unit, configured to adjust the parameters of the speech enhancement evaluation model based on the loss to train the speech enhancement evaluation model.

[0018] Optionally, the enhancement unit is further configured to: obtain gain data for simulating a speech enhancement operation; and based on the gain data, perform enhancement processing on the original speech sample and the original noise sample respectively to obtain the enhanced speech sample and the enhanced noise sample.

[0019] Optionally, the gain data is the product of at least one complex gain, and the enhancement unit is further configured to: convert the original speech sample and the original noise sample from time domain signals into frequency domain signals; determine the product of the gain data and the original speech sample to obtain the enhanced speech sample; determine the product of the gain data and the original noise sample to obtain the enhanced noise sample.

[0020] Optionally, the enhancement unit is further configured to: utilize short-time Fourier transform to convert the original speech sample and the original noise sample from time domain signals into multi-frame frequency domain signals.

[0021] Optionally, the separation unit is further configured to: input the enhanced sample, the original speech sample, and the original noise sample into the speech enhancement evaluation model to obtain one of the estimated enhanced speech and the estimated enhanced noise; remove one of the estimated enhanced speech and the estimated enhanced noise from the enhanced sample to obtain the other of the estimated enhanced speech and the estimated enhanced noise.

[0022] According to another aspect of the present disclosure, a speech enhancement evaluation device is provided for evaluating a speech enhancement model, the speech enhancement evaluation device comprising: an acquisition unit, configured to acquire a noisy speech sample, the noisy speech sample comprising an original speech sample and an original noise sample; an enhancement unit, configured to input the noisy speech sample into a speech enhancement model to be tested, to obtain enhanced noisy speech; a separation unit, configured to input the enhanced noisy speech, the original speech sample and the original noise sample into a speech enhancement evaluation model, to obtain an estimated enhanced speech and an estimated enhanced noise; and an evaluation unit, configured to determine an evaluation value of the speech enhancement model to be tested according to the estimated enhanced speech and the estimated enhanced noise.

[0023] Optionally, the noisy speech sample is obtained by combining the original speech sample and the original noise sample.

[0024] Optionally, the separation unit is further configured to: input the enhanced noisy speech, the original speech sample, and the original noise sample into the speech enhancement evaluation model to obtain one of the estimated enhanced speech and the estimated enhanced noise; remove one of the estimated enhanced speech and the estimated enhanced noise from the enhanced noisy speech to obtain the other of the estimated enhanced speech and the estimated enhanced noise.

[0025] Optionally, the evaluation unit is further configured to: determine a signal-to-noise ratio of the enhanced noisy speech according to the estimated enhanced speech and the estimated enhanced noise as an evaluation value of the speech enhancement model to be tested.

[0026] Optionally, the enhanced noisy speech, the estimated enhanced speech and the estimated enhanced noise all include multiple frames of frequency domain signals, each frame of frequency domain signals includes complex signals at multiple center frequencies, and the evaluation unit is further configured to: respectively determine the sum of the amplitudes of all center frequencies of the estimated enhanced speech and the estimated enhanced noise in each frame as the speech energy and noise energy of the enhanced noisy speech in the corresponding frame; determine the signal-to-noise ratio of the enhanced noisy speech in the corresponding frame according to the ratio of the speech energy and noise energy of the enhanced noisy speech in each frame; and determine the statistical value of the signal-to-noise ratio of the enhanced noisy speech in each frame as the signal-to-noise ratio of the enhanced noisy speech.

[0027] Optionally, the enhanced noisy speech, the estimated enhanced speech and the estimated enhanced noise are all time domain signals, and the evaluation unit is further configured to: determine the arithmetic square root of the sum of the squares of the amplitude values ​​of the estimated enhanced speech and the estimated enhanced noise at each moment as the speech energy and the noise energy respectively; and determine the signal-to-noise ratio of the enhanced noisy speech according to the ratio of the speech energy to the noise energy.

[0028] Optionally, the speech enhancement evaluation model is trained using the training method according to the present disclosure.

[0029] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to execute a training method for a speech enhancement evaluation model or a speech enhancement evaluation method according to the present disclosure.

[0030] According to another aspect of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the training method of the speech enhancement evaluation model or the speech enhancement evaluation method according to the present disclosure.

[0031] According to another aspect of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by at least one processor, implement the training method of the speech enhancement evaluation model or the speech enhancement evaluation method according to the present disclosure.

[0032] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0033] According to the speech enhancement evaluation method, training method and device of the speech enhancement evaluation model of the exemplary embodiment of the present disclosure, based on deep learning, a speech enhancement evaluation model is used that is only for test samples, has no requirements on complexity and real-time performance, and is given the original speech component and the original noise component. The enhanced noisy speech output by the speech enhancement model to be tested is separated from the speech and residual noise. This is equivalent to using a more capable model that knows the answer in advance to evaluate another model that is weaker and does not know the answer. Compared with the separation method based on SI-SNR in the related art, it can reduce the result error, thereby effectively measuring the performance of the speech enhancement model to be tested.

[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0036] Figure 1 is a flowchart illustrating a method for training a speech enhancement evaluation model according to an exemplary embodiment of the present disclosure.

[0037] Figure 2 Detailed description is given of the flowchart of a method for training a speech enhancement evaluation model according to an exemplary embodiment of the present disclosure.

[0038] Figure 3 Detailed description is a flow chart showing a speech enhancement evaluation method according to an exemplary embodiment of the present disclosure.

[0039] Figure 4 Detailed description is given of a flow chart of a speech enhancement evaluation method according to an exemplary embodiment of the present disclosure.

[0040] Figure 5 is a block diagram illustrating a training apparatus for a speech enhancement evaluation model according to an exemplary embodiment of the present disclosure.

[0041] Figure 6 Detailed description is a block diagram illustrating a speech enhancement evaluation apparatus according to an exemplary embodiment of the present disclosure.

[0042] Figure 7 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0044] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0045] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0046] The speech enhancement model is a model used to extract the purest original speech possible from noisy speech. The signal model of speech enhancement is:

[0047] y=(s+n)*rir*deq

[0048] Among them, y is the noisy speech input to the speech enhancement model, n is the noise, s is the speech signal, rir is the room reverberation, including direct sound (direct), early reflection (early), and late reverberation (late), deq is the frequency response curve of the acquisition device, and * is the convolution operator. The output of the speech enhancement model is:

[0049]

[0050] Among them, s' represents the speech clipping component introduced by the speech enhancement model, and n' represents the noise component that can be eliminated by the speech enhancement model. Therefore, (ss') is the speech component in the output signal, and (nn') is the residual noise component. dir+early The room reverberation components to be retained include the direct sound and the early reflections. In other words, the speech enhancement model is used to remove the late reverberation part of the room reverberation. In the actual model, the room reverberation can be retained or not.

[0051] However, in practical applications, any speech enhancement model may accidentally cut speech and / or leave residual noise, so an evaluation method is needed to assess the degree of accidentally cutting speech and residual noise. To this end, the signal output by the speech enhancement model to be tested needs to be separated into two parts: speech component and residual noise component. Extract (ss′)*rir dir+early *deq component and (nn′)*rir dir+early *deq component for further performance analysis.

[0052] There is an evaluation method based on SI-SNR in the related art, which can separate the speech component and residual noise component in the signal estimated by the speech enhancement model to be tested in the process of obtaining SI-SNR. However, this method has a disadvantage, that is, when the estimated signal passes through AGC, the speech component and residual noise component separated by this method will produce a relatively large error. However, the speech enhancement model to be tested often includes an AGC module, and because the speech enhancement model to be tested is a whole black box and cannot be distributed for testing, the results that can be obtained are mostly signals after AGC, which makes this method unable to effectively measure the performance of the speech enhancement model to be tested.

[0053] According to the speech enhancement evaluation scheme of the exemplary embodiment of the present disclosure, based on deep learning, the enhanced noisy speech output by the speech enhancement model to be tested (that is, the speech enhancement model whose performance is to be evaluated) is separated from the speech and residual noise through a speech enhancement evaluation model that is only for test samples and has no requirements for complexity and real-time performance and is given the original speech component and the original noise component. This is equivalent to using a model with stronger capabilities and knowing the answer in advance to evaluate another model with weaker capabilities and no knowledge of the answer. Compared with the separation method based on SI-SNR in the related art, it can reduce the result error, thereby effectively measuring the performance of the speech enhancement model to be tested.

[0054] Next, we will refer to Figures 1 to 5 A speech enhancement evaluation model training method, a speech enhancement evaluation method, a speech enhancement evaluation model training device, and a speech enhancement evaluation device according to exemplary embodiments of the present disclosure are specifically described.

[0055] Figure 1 is a flowchart illustrating a method for training a speech enhancement evaluation model according to an exemplary embodiment of the present disclosure. Figure 2 is a flowchart illustrating a method for training a speech enhancement evaluation model according to an exemplary embodiment of the present disclosure. It should be understood that the method for training a speech enhancement evaluation model according to an exemplary embodiment of the present disclosure can be implemented in a terminal device such as a smart phone, a tablet computer, a personal computer (PC), or can be implemented in a device such as a server.

[0056] Reference Figure 1 and Figure 2 , in step 101, original speech samples and original noise samples are obtained.

[0057] The noisy speech sample composed of the original speech sample and the original noise sample can be used as a test sample, which is equivalent to knowing the speech component and noise component in the noisy speech in advance, and can provide a reference for the component separation of the enhanced signal output by the speech enhancement model to be tested. It should be understood that by adjusting the energy of the original speech sample and the original noise sample, the signal-to-noise ratio of the noisy speech sample can be controlled. After the energy of the two is adjusted, the two are combined to obtain the noisy speech sample.

[0058] In step 102, the original speech sample and the original noise sample are enhanced to obtain an enhanced speech sample and an enhanced noise sample.

[0059] The enhancement process is used to simulate the speech enhancement model to be tested. By performing enhancement process on the original speech sample and the original noise sample respectively, the enhanced sample obtained by merging the enhanced speech sample and the enhanced noise sample (performed in step 103) can be regarded as the enhanced noisy speech output by the speech enhancement model to be tested (i.e., the aforementioned enhanced signal), and the training of the speech enhancement evaluation model can be realized based on this.

[0060] Optionally, step 102 may specifically include: obtaining gain data for simulating speech enhancement operation; based on the gain data, performing enhancement processing on the original speech sample and the original noise sample respectively, to obtain an enhanced speech sample and an enhanced noise sample. The gain data is used to reflect the effect of the enhancement processing on the data changes of the original speech sample and the original noise sample. By performing corresponding calculations on the original speech sample and the original noise sample based on the gain data, the enhancement processing can be achieved, and the calculated data are the enhanced speech sample and the enhanced noise sample. By configuring the gain data, different speech enhancement models can be simulated by adjusting the gain data, so that the enhanced sample obtained in step 103 is close to the enhanced noisy speech output by the speech enhancement model to be tested, which helps to improve the evaluation effect of the speech enhancement evaluation model obtained by training. Specifically, the speech enhancement model can perform reverberation, equalizer adjustment, speech shearing, noise residue, AGC and other operations on the noisy speech input therein, and can evaluate the operations involved in the speech enhancement model to be tested by comparing and analyzing the difference between the input signal and the output signal of the speech enhancement model to be tested, and then estimate the appropriate gain data, so as to customize different gain data for different speech enhancement models to be tested. Of course, gain data with a certain degree of universality can also be used to achieve a reasonable simulation effect, which helps to reduce the amount of calculation for determining the gain data. These are all implementation methods of the present disclosure.

[0061] Optionally, the gain data is the product of at least one complex gain. The operation of performing enhancement processing on the original speech sample and the original noise sample based on the gain data in step 102 to obtain the enhanced speech sample and the enhanced noise sample may specifically include: converting the original speech sample and the original noise sample from time domain signals to frequency domain signals to obtain the original speech sample and the original noise sample in complex form; determining the product of the gain data and the original speech sample to obtain the enhanced speech sample; determining the product of the gain data and the original noise sample to obtain the enhanced noise sample. By performing product calculation on the gain data, the original speech sample and the original noise sample, which are all in complex form, specifically dot product calculation (because the enhanced sample obtained also needs to be in complex form), the enhanced speech sample and the enhanced noise sample can be obtained, thereby achieving concise and reliable enhancement processing.

[0062] Optionally, the operation of converting the original speech sample and the original noise sample from the time domain signal to the frequency domain signal is specifically performed as follows: using the short-time Fourier transform (STFT, Short-Time Fourier Transform), the original speech sample and the original noise sample are converted from the time domain signal to a multi-frame frequency domain signal. The original speech sample and the original noise sample are usually non-stationary signals, and their frequencies will change with time. The Fourier transform is only used to convert the time domain signal into the frequency domain signal, and the time information therein cannot be extracted, that is, it is impossible to understand when each frequency appears specifically. By using the short-time Fourier transform, the original speech sample and the original noise sample can be first divided into multiple segments according to a certain time width, thereby forming a multi-frame time domain signal, and extracting the local information of the signal. Due to the short time span, each frame of the time domain signal can be considered to be approximately stable. Then, each frame of the approximately stable time domain signal is converted into a frequency domain signal, and a multi-frame frequency domain signal can be obtained, which can extract the time information and ensure the comprehensiveness of the information.

[0063] Specifically, if the original speech sample S and the original noise sample N of length T are S(t) and N(t) in the time domain, respectively, where t represents time, 0<t≤T, then after short-time Fourier transform, S(t) and N(t) can be expressed in the frequency domain as follows:

[0064] S(n,k)=STFT(S(t))

[0065] N(n,k)=STFT(N(t))

[0066] Where n is the frame sequence, 0<n≤N (N is the total number of frames); k is the center frequency sequence, 0k≤K (K is the total number of frequency bands). S(n,k) is the complex signal at the kth center frequency of the nth frame, consisting of amplitude and phase:

[0067] S(n,k)=|S(n,k)|·ej·φ(S(n,k))

[0068] When S(n,k) and N(n,k) are enhanced to simulate the output of a speech enhancement model to be tested, the process can be abstracted as follows:

[0069] S aug (n,k)=S(n,k)·G(n,k)

[0070]

[0071] Among them, G(n,j) is a gain data in complex form, including amplitude change and phase change, representing the total effect of several enhancement processing on the original signal, which is equal to the product of the complex gain data g(n,k) of these several enhancement processing. g(n,k) also includes amplitude change and phase change, representing the effect of a certain enhancement processing on the original signal.

[0072] In step 103, the enhanced speech sample and the enhanced noise sample are combined to obtain an enhanced sample.

[0073] In step 104, the enhanced sample, the original speech sample, and the original noise sample are input into a speech enhancement evaluation model to obtain at least one of an estimated enhanced speech and an estimated enhanced noise.

[0074] Step 104 is a step of executing the speech enhancement evaluation model. The speech enhancement evaluation model is used to separate the speech component and the residual noise component in the enhanced noisy speech output by the speech enhancement model to be tested, that is, to obtain the estimated enhanced speech and the estimated enhanced noise. In the model training stage here, the speech component and the residual noise component in the enhanced sample are separated.

[0075] Specifically, the speech enhancement evaluation model can be a neural network composed of any network layer such as CNN (Convolutional Neural Network) / RNN (Recurrent Neural Network) / DNN (Deep Neural Network), and there is no requirement for complexity and real-time performance. As an example, the speech enhancement evaluation model according to the exemplary embodiment of the present disclosure can be a network structure based on U-Net, and the model includes three encoding (encoder) blocks, four residual attention (RA, Residual-Attention) blocks and three decoding (decoder) blocks, and the four residual attention blocks are placed between the encoding block and the decoding block. Each encoding block includes a two-dimensional convolution (Conv2d) layer, a batch normalization (BN, Batch Normalization) layer and an exponential linear unit (ELU, Exponential Linear Unit) activation function. The only difference between each decoding block and the corresponding encoding block is that the two-dimensional convolution layer is replaced by a two-dimensional deconvolution (Conv2dTranspose) layer. The residual attention block mainly consists of two residual blocks and a separable self-attention (SSA) block. The self-attention in the separable self-attention block uses scaled dot-product self-attention.

[0076] Optionally, step 104 includes: inputting the enhanced sample, the original speech sample, and the original noise sample into the speech enhancement evaluation model to obtain one of the estimated enhanced speech and the estimated enhanced noise (in Figure 2 In the embodiment shown, an estimated enhanced speech can be obtained); one of the estimated enhanced speech and the estimated enhanced noise is removed from the enhanced sample to obtain the other of the estimated enhanced speech and the estimated enhanced noise. Since the enhanced noisy speech output by the speech enhancement model to be tested only has two components, namely speech and residual noise, the speech enhancement evaluation model can output only one of the components, and then obtain the other component by taking the difference, which can reduce the amount of calculation of the speech enhancement evaluation model and avoid the situation where the sum of the two components that may appear when the model calculates the two components at the same time is inconsistent with the signal before separation, thereby ensuring the reliability of the separation result.

[0077] It should be understood that in the application stage of the model, both the estimated enhanced speech and the estimated enhanced noise need to be obtained, while in the training stage of the model, only one or both can be obtained, depending on the loss determination scheme in step 105.

[0078] In step 105, a loss is determined based on the estimated enhanced speech and the enhanced speech sample, and / or a loss is determined based on the estimated enhanced noise and the enhanced noise sample.

[0079] When determining the loss, only the loss related to the speech component may be used, only the loss related to the residual noise component may be used, or both may be used simultaneously, which are all implementations of the present disclosure. For different implementations, corresponding components may be obtained in step 104 .

[0080] Alternatively, the mean absolute logarithm error (MALE) can be used as the loss function. In this case, the speech component-related loss MALE S The loss associated with the residual noise component MALE N Can be expressed as:

[0081] MALE s = abs(log e (abs(real(S aug )+imag(S aug )+1))-log e (abs(real(S aug ')+imag(S aug ')+1)))

[0082] MALE N = abs(log e (abs(real(N aug )+imag(N aug )+1))-log e (abs(real(N aug ')+imag(N aug ')+1)))

[0083] Among them, e is a natural constant, abs means taking the absolute value, real means taking the real part of the complex number, and imag means taking the imaginary part of the complex number. aug is the enhanced speech sample, S aug ' is the estimated enhanced speech, N aug is the enhanced noise sample, N aug ' is the estimated enhanced noise and satisfies:

[0084] N aug '=S aug +N aug -S aug '

[0085] It should be understood that the above loss function formula represents the loss at each center frequency of each frame, and the loss finally used may be the statistical value of the loss of all frames and all center frequencies.

[0086] In step 106, the parameters of the speech enhancement evaluation model are adjusted based on the loss to train the speech enhancement evaluation model. After training, the obtained speech enhancement evaluation model can be used to evaluate the performance of the speech enhancement model to be tested.

[0087] Figure 3 Detailed description is a flow chart showing a speech enhancement evaluation method according to an exemplary embodiment of the present disclosure. Figure 4 is a flow chart showing a method for speech enhancement evaluation according to an exemplary embodiment of the present disclosure. It should be understood that the method for speech enhancement evaluation according to an exemplary embodiment of the present disclosure can be implemented in a terminal device such as a smart phone, a tablet computer, a personal computer (PC), or in a device such as a server.

[0088] Reference Figure 3 and Figure 4 In step 301, a noisy speech sample is obtained, where the noisy speech sample includes an original speech sample and an original noise sample.

[0089] Specifically, the noisy speech sample is obtained by combining the original speech sample and the original noise sample, and can be used as a test sample, which is equivalent to knowing the speech component and the noise component in the noisy speech in advance, and can provide a reference for the component separation of the enhanced signal output by the speech enhancement model to be tested. It should be understood that by adjusting the energy of the original speech sample and the original noise sample, the signal-to-noise ratio of the noisy speech sample can be controlled. After the energy of the two is adjusted, the two are combined to obtain the noisy speech sample.

[0090] In step 302, the noisy speech sample is input into the speech enhancement model to be tested to obtain enhanced noisy speech. This step uses the noisy speech sample as the input of the speech enhancement model to be tested, and can obtain the output of the speech enhancement model to be tested as the target of subsequent component separation, so as to evaluate the performance of the speech enhancement model to be tested. It should be understood that only using noisy speech samples whose components are known in advance as input during the test, rather than randomly inputting noisy speech, can ensure the reliability of the subsequent component separation results and make the evaluation results reliable.

[0091] In step 303, the enhanced noisy speech, the original speech sample and the original noise sample are input into a speech enhancement evaluation model to obtain the estimated enhanced speech and the estimated enhanced noise.

[0092] Specifically, the speech enhancement evaluation model can be a neural network composed of any network layers such as CNN / RNN / DNN, and can be trained using the training method of the speech enhancement evaluation model according to the exemplary embodiment of the present disclosure. It should be understood that, unlike the training stage, only one component in the enhanced sample can be determined, in the application stage, both the speech and residual noise components of the enhanced noisy speech need to be obtained at the same time, that is, the estimated enhanced speech and the estimated enhanced noise are obtained at the same time.

[0093] Optionally, step 303 includes: inputting the enhanced noisy speech, the original speech sample, and the original noise sample into the speech enhancement evaluation model to obtain one of the estimated enhanced speech and the estimated enhanced noise (in Figure 4 In the embodiment shown, the estimated enhanced speech can be obtained); one of the estimated enhanced speech and the estimated enhanced noise is removed from the enhanced noisy speech to obtain the other of the estimated enhanced speech and the estimated enhanced noise. Since the enhanced noisy speech output by the speech enhancement model to be tested only has two components, namely speech and residual noise, the speech enhancement evaluation model can output only one of the components, and then obtain the other component by taking the difference, which can reduce the amount of calculation of the speech enhancement evaluation model and avoid the situation where the sum of the two components that may appear when the model calculates the two components at the same time is inconsistent with the signal before separation, thereby ensuring the reliability of the separation result.

[0094] In step 304, the evaluation value of the speech enhancement model to be tested is determined according to the estimated enhanced speech and the estimated enhanced noise. This step can evaluate the enhancement effect of the speech enhancement model to be tested by combining the separated speech component and the residual noise component.

[0095] Optionally, step 304 includes: determining the signal-to-noise ratio of the enhanced noisy speech based on the estimated enhanced speech and the estimated enhanced noise as an evaluation value of the speech enhancement model to be tested. That is, the performance of the speech enhancement model to be tested is evaluated based on the signal-to-noise ratio. If the signal-to-noise ratio is high, it indicates that the speech enhancement model to be tested can better enhance the speech input therein; on the contrary, if the signal-to-noise ratio is low, it indicates that the enhancement performance of the speech enhancement model to be tested is not good. As an example, a signal-to-noise ratio threshold can be configured as a reference for judging the signal-to-noise ratio. The signal-to-noise ratio (SNR, Source-to-Noise Ratio) can be expressed as:

[0096]

[0097] The energy of the corresponding signal is substituted into the formula, S aug ' is the estimated speech energy corresponding to the enhanced speech, N aug ' is the noise energy corresponding to the estimated enhanced noise.

[0098] Specifically, the enhanced noisy speech output by the speech enhancement model to be tested obtained in step 302 may be a multi-frame frequency domain signal, and each frame of the frequency domain signal includes a complex signal at multiple center frequencies, which is composed of amplitude and phase as described above. Accordingly, the original speech sample and the original noise sample may also be converted into a multi-frame frequency domain signal by using a short-time Fourier transform, and the estimated enhanced speech and estimated enhanced noise obtained in step 303 are also multi-frame frequency domain signals.

[0099] In some embodiments, optionally, step 304 includes: determining the sum of the amplitudes of all center frequencies of the estimated enhanced speech and the estimated enhanced noise in each frame as the speech energy and noise energy of the enhanced noisy speech in the corresponding frame; determining the signal-to-noise ratio of the enhanced noisy speech in the corresponding frame according to the ratio of the speech energy and noise energy of the enhanced noisy speech in each frame (for example, it can be determined using the above SNR formula); determining the statistical value of the signal-to-noise ratio of the enhanced noisy speech in each frame, such as the mean, median, mode, variance, standard deviation, as the signal-to-noise ratio of the enhanced noisy speech. This method can directly calculate the frequency domain signal to meet the calculation requirements of the signal-to-noise ratio.

[0100] In other embodiments, optionally, the enhanced noisy speech, the estimated enhanced speech and the estimated enhanced noise are all time domain signals. For example, the short-time inverse Fourier transform (iSTFT, Inverse Short-Time Fourier Transform) can be used to convert the multi-frame frequency domain signal into a time domain signal. At this time, step 304 includes: determining the speech energy and the noise energy according to the amplitude values ​​of the estimated enhanced speech and the estimated enhanced noise at each moment, for example, but not limited to, taking the square sum of the amplitude values ​​at each moment as the energy, or taking the arithmetic square root of the square sum as the energy. For the above SNR formula, the energy can be the arithmetic square root of the square sum; according to the ratio of the speech energy and the noise energy, determining the signal-to-noise ratio of the enhanced noisy speech (for example, it can be determined using the above SNR formula). This method can calculate a signal-to-noise ratio for the entire time domain signal by converting the frequency domain signal back to the time domain signal, without having to perform statistics on a large number of signal-to-noise ratios, and can also meet the calculation requirements of the signal-to-noise ratio.

[0101] Figure 5 is a block diagram showing a training device for a speech enhancement evaluation model according to an exemplary embodiment of the present disclosure. It should be understood that the training device for a speech enhancement evaluation model according to an exemplary embodiment of the present disclosure can be implemented in a terminal device such as a smart phone, a tablet computer, a personal computer (PC) in the form of software, hardware, or a combination of software and hardware, and can also be implemented in a device such as a server.

[0102] Reference Figure 5The training device 500 of the speech enhancement evaluation model includes an acquisition unit 501, an enhancement unit 502, a merging unit 503, a separation unit 504, a calculation unit 505, and a parameter adjustment unit 506.

[0103] The acquisition unit 501 may acquire original speech samples and original noise samples.

[0104] The noisy speech sample composed of the original speech sample and the original noise sample can be used as a test sample, which is equivalent to knowing the speech component and noise component in the noisy speech in advance, and can provide a reference for the component separation of the enhanced signal output by the speech enhancement model to be tested. It should be understood that by adjusting the energy of the original speech sample and the original noise sample, the signal-to-noise ratio of the noisy speech sample can be controlled. After the energy of the two is adjusted, the two are combined to obtain the noisy speech sample.

[0105] The enhancement unit 502 may perform enhancement processing on the original speech sample and the original noise sample respectively to obtain an enhanced speech sample and an enhanced noise sample.

[0106] The enhancement process is used to simulate the speech enhancement model to be tested. By performing enhancement process on the original speech sample and the original noise sample respectively, the enhanced sample obtained by merging the enhanced speech sample and the enhanced noise sample (performed by the merging unit 503) can be regarded as the enhanced noisy speech output by the speech enhancement model to be tested (i.e., the aforementioned enhanced signal), and the training of the speech enhancement evaluation model can be realized based on this.

[0107] Optionally, step 102 may specifically include: obtaining gain data for simulating speech enhancement operation; based on the gain data, performing enhancement processing on the original speech sample and the original noise sample respectively, to obtain an enhanced speech sample and an enhanced noise sample. The gain data is used to reflect the effect of the enhancement processing on the data changes of the original speech sample and the original noise sample. By performing corresponding calculations on the original speech sample and the original noise sample based on the gain data, the enhancement processing can be achieved, and the calculated data are the enhanced speech sample and the enhanced noise sample. By configuring the gain data, different speech enhancement models can be simulated by adjusting the gain data, so that the enhanced sample obtained by the merging unit 503 is close to the enhanced noisy speech output by the speech enhancement model to be tested, which helps to improve the evaluation effect of the speech enhancement evaluation model obtained by training. Specifically, the speech enhancement model can perform reverberation, equalizer adjustment, speech shearing, noise residue, AGC and other operations on the noisy speech input therein, and can evaluate the operations involved in the speech enhancement model to be tested by comparing and analyzing the difference between the input signal and the output signal of the speech enhancement model to be tested, and then estimate the appropriate gain data, so as to customize different gain data for different speech enhancement models to be tested. Of course, gain data with a certain degree of universality can also be used to achieve a reasonable simulation effect, which helps to reduce the amount of calculation for determining the gain data. These are all implementation methods of the present disclosure.

[0108] Optionally, the gain data is the product of at least one complex gain, and the operation of performing enhancement processing on the original speech sample and the original noise sample based on the gain data by the enhancement unit 502 to obtain the enhanced speech sample and the enhanced noise sample may specifically include: converting the original speech sample and the original noise sample from time domain signals to frequency domain signals to obtain the original speech sample and the original noise sample in complex form; determining the product of the gain data and the original speech sample to obtain the enhanced speech sample; determining the product of the gain data and the original noise sample to obtain the enhanced noise sample. By performing product calculation on the gain data, the original speech sample and the original noise sample, which are all in complex form, specifically dot product calculation (because the enhanced sample obtained also needs to be in complex form), the enhanced speech sample and the enhanced noise sample can be obtained, thereby achieving concise and reliable enhancement processing.

[0109] Optionally, the operation of converting the original speech sample and the original noise sample from the time domain signal to the frequency domain signal is specifically performed as follows: using the short-time Fourier transform (STFT, Short-Time Fourier Transform), the original speech sample and the original noise sample are converted from the time domain signal to a multi-frame frequency domain signal. The original speech sample and the original noise sample are usually non-stationary signals, and their frequencies will change with time. The Fourier transform is only used to convert the time domain signal into the frequency domain signal, and the time information therein cannot be extracted, that is, it is impossible to understand when each frequency appears specifically. By using the short-time Fourier transform, the original speech sample and the original noise sample can be first divided into multiple segments according to a certain time width, thereby forming a multi-frame time domain signal, and extracting the local information of the signal. Due to the short time span, each frame of the time domain signal can be considered to be approximately stable. Then, each frame of the approximately stable time domain signal is converted into a frequency domain signal, and a multi-frame frequency domain signal can be obtained, which can extract the time information and ensure the comprehensiveness of the information.

[0110] The merging unit 503 may merge the enhanced speech sample and the enhanced noise sample to obtain an enhanced sample.

[0111] The separation unit 504 may input the enhanced sample, the original speech sample, and the original noise sample into the speech enhancement evaluation model to obtain at least one of the estimated enhanced speech and the estimated enhanced noise.

[0112] The separation unit 504 performs the operation of the speech enhancement evaluation model. The speech enhancement evaluation model is used to separate the speech component and the residual noise component in the enhanced noisy speech output by the speech enhancement model to be tested, that is, to obtain the estimated enhanced speech and the estimated enhanced noise. In the model training stage here, the speech component and the residual noise component in the enhanced sample are separated.

[0113] Specifically, the speech enhancement evaluation model can be a neural network composed of any network layers such as CNN / RNN / DNN, with no requirements on complexity and real-time performance.

[0114] Optionally, the operations performed by the separation unit 504 include: inputting the enhanced sample, the original speech sample, and the original noise sample into the speech enhancement evaluation model to obtain one of the estimated enhanced speech and the estimated enhanced noise (for example, the estimated enhanced speech can be obtained); removing one of the estimated enhanced speech and the estimated enhanced noise from the enhanced sample to obtain the other of the estimated enhanced speech and the estimated enhanced noise. Since the enhanced noisy speech output by the speech enhancement model to be tested only has two components, speech and residual noise, the speech enhancement evaluation model can only output one of the components, and then obtain the other component by taking the difference, which can reduce the amount of calculation of the speech enhancement evaluation model and avoid the situation where the sum of the two components that may occur when the model calculates the two components at the same time is inconsistent with the signal before separation, thereby ensuring the reliability of the separation result.

[0115] It should be understood that in the application stage of the model, both the estimated enhanced speech and the estimated enhanced noise need to be obtained, while in the training stage of the model, only one or both can be obtained, depending on the loss determination scheme of the computing unit 505.

[0116] The calculation unit 505 may determine the loss according to the estimated enhanced speech and the enhanced speech sample, and / or determine the loss according to the estimated enhanced noise and the enhanced noise sample.

[0117] When determining the loss, only the loss related to the speech component may be used, only the loss related to the residual noise component may be used, or both may be used simultaneously, which are all implementations of the present disclosure. For different implementations, the separation unit 504 may obtain the corresponding components.

[0118] The parameter adjustment unit 506 can adjust the parameters of the speech enhancement evaluation model based on the loss to train the speech enhancement evaluation model. After training, the obtained speech enhancement evaluation model can be used to evaluate the performance of the speech enhancement model to be tested.

[0119] Figure 6 is a block diagram showing a speech enhancement evaluation device according to an exemplary embodiment of the present disclosure. It should be understood that the speech enhancement evaluation device according to the exemplary embodiment of the present disclosure can be implemented in a terminal device such as a smart phone, a tablet computer, a personal computer (PC) in the form of software, hardware, or a combination of software and hardware, and can also be implemented in a device such as a server.

[0120] Reference Figure 6 The speech enhancement and evaluation device 600 includes an acquisition unit 601, an enhancement unit 602, a separation unit 603, and an evaluation unit 604.

[0121] The acquisition unit 601 may acquire a noisy speech sample, where the noisy speech sample includes an original speech sample and an original noise sample.

[0122] Specifically, the noisy speech sample is obtained by combining the original speech sample and the original noise sample, and can be used as a test sample, which is equivalent to knowing the speech component and the noise component in the noisy speech in advance, and can provide a reference for the component separation of the enhanced signal output by the speech enhancement model to be tested. It should be understood that by adjusting the energy of the original speech sample and the original noise sample, the signal-to-noise ratio of the noisy speech sample can be controlled. After the energy of the two is adjusted, the two are combined to obtain the noisy speech sample.

[0123] The enhancement unit 602 may input the noisy speech sample into the speech enhancement model to be tested to obtain enhanced noisy speech. This step uses the noisy speech sample as the input of the speech enhancement model to be tested, and may obtain the output of the speech enhancement model to be tested, which is used as the target of subsequent component separation, so as to evaluate the performance of the speech enhancement model to be tested. It should be understood that during the test, only noisy speech samples whose components are known in advance are used as input, and no noisy speech is not randomly input, so as to ensure the reliability of the subsequent component separation results and make the evaluation results reliable.

[0124] The separation unit 603 may input the enhanced noisy speech, the original speech sample and the original noise sample into the speech enhancement evaluation model to obtain the estimated enhanced speech and the estimated enhanced noise.

[0125] Specifically, the speech enhancement evaluation model can be a neural network composed of any network layers such as CNN / RNN / DNN, and can be trained using the training method of the speech enhancement evaluation model according to the exemplary embodiment of the present disclosure. It should be understood that, unlike the training stage, only one component in the enhanced sample can be determined, in the application stage, both the speech and residual noise components of the enhanced noisy speech need to be obtained at the same time, that is, the estimated enhanced speech and the estimated enhanced noise are obtained at the same time.

[0126] Optionally, the operations performed by the separation unit 603 include: inputting the enhanced noisy speech, the original speech sample, and the original noise sample into the speech enhancement evaluation model to obtain one of the estimated enhanced speech and the estimated enhanced noise (for example, the estimated enhanced speech can be obtained); removing one of the estimated enhanced speech and the estimated enhanced noise from the enhanced noisy speech to obtain the other of the estimated enhanced speech and the estimated enhanced noise. Since the enhanced noisy speech output by the speech enhancement model to be tested only has two components, speech and residual noise, the speech enhancement evaluation model can only output one of the components, and then obtain the other component by taking the difference, which can reduce the amount of calculation of the speech enhancement evaluation model and avoid the situation where the sum of the two components that may occur when the model calculates the two components at the same time is inconsistent with the signal before separation, thereby ensuring the reliability of the separation result.

[0127] The evaluation unit 604 may determine the evaluation value of the speech enhancement model to be tested according to the estimated enhanced speech and the estimated enhanced noise. The evaluation unit 604 may evaluate the enhancement effect of the speech enhancement model to be tested in combination with the separated speech component and the residual noise component.

[0128] Optionally, the operation performed by the evaluation unit 604 includes: determining the signal-to-noise ratio of the enhanced noisy speech according to the estimated enhanced speech and the estimated enhanced noise as the evaluation value of the speech enhancement model to be tested. That is, the performance of the speech enhancement model to be tested is evaluated based on the signal-to-noise ratio. If the signal-to-noise ratio is high, it indicates that the speech enhancement model to be tested can better enhance the speech input therein; on the contrary, if the signal-to-noise ratio is low, it indicates that the enhancement performance of the speech enhancement model to be tested is not good. As an example, a signal-to-noise ratio threshold can be configured as a reference for judging the signal-to-noise ratio.

[0129] Specifically, the enhanced noisy speech output by the speech enhancement model to be tested obtained by the enhancement unit 602 may be a multi-frame frequency domain signal, each frame of the frequency domain signal includes a complex signal at multiple center frequencies, which is composed of amplitude and phase. Accordingly, the original speech sample and the original noise sample may also be converted into a multi-frame frequency domain signal by using a short-time Fourier transform, and the estimated enhanced speech and estimated enhanced noise obtained by the separation unit 603 are also multi-frame frequency domain signals.

[0130] In some embodiments, optionally, the operations performed by the evaluation unit 604 include: determining the sum of the amplitudes of all center frequencies of the estimated enhanced speech and the estimated enhanced noise in each frame as the speech energy and noise energy of the enhanced noisy speech in the corresponding frame; determining the signal-to-noise ratio of the enhanced noisy speech in the corresponding frame according to the ratio of the speech energy and noise energy of the enhanced noisy speech in each frame; determining the statistical value of the signal-to-noise ratio of the enhanced noisy speech in each frame, such as the mean, median, mode, variance, and standard deviation, as the signal-to-noise ratio of the enhanced noisy speech. This method can directly calculate the frequency domain signal to meet the calculation requirements of the signal-to-noise ratio.

[0131] In other embodiments, the enhanced noisy speech, the estimated enhanced speech and the estimated enhanced noise are all time domain signals, for example, short-time inverse Fourier transform can be used to convert the multi-frame frequency domain signals into time domain signals. At this time, the evaluation unit 604 performs the following operations: determining the speech energy and the noise energy according to the amplitude values ​​of the estimated enhanced speech and the estimated enhanced noise at each moment, for example, but not limited to, taking the square sum of the amplitude values ​​at each moment as the energy, or taking the arithmetic square root of the square sum as the energy; determining the signal-to-noise ratio of the enhanced noisy speech according to the ratio of the speech energy to the noise energy. This method can calculate a signal-to-noise ratio for the entire time domain signal by converting the frequency domain signal back to the time domain signal, without having to perform statistics on a large number of signal-to-noise ratios, and can also meet the calculation requirements of the signal-to-noise ratio.

[0132] Figure 7 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.

[0133] Reference Figure 7 The electronic device 700 includes at least one memory 701 and at least one processor 702, wherein the at least one memory 701 stores a set of computer executable instructions. When the computer executable instruction set is executed by the at least one processor 702, a training method for a speech enhancement evaluation model or a speech enhancement evaluation method according to an exemplary embodiment of the present disclosure is executed.

[0134] As an example, the electronic device 700 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instruction set. Here, the electronic device 700 is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above instruction (or instruction set) individually or in combination. The electronic device 700 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0135] In the electronic device 700, the processor 702 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0136] The processor 702 may execute instructions or codes stored in the memory 701, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.

[0137] The memory 701 may be integrated with the processor 702, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 701 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 701 and the processor 702 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 702 can read files stored in the memory.

[0138] In addition, the electronic device 700 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 700 may be connected to each other via a bus and / or a network.

[0139] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided. When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the training method of the speech enhancement evaluation model or the speech enhancement evaluation method according to the exemplary embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0140] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, which includes computer instructions. When the computer instructions are executed by at least one processor, the at least one processor is prompted to execute a training method for a speech enhancement evaluation model or a speech enhancement evaluation method according to an exemplary embodiment of the present disclosure.

[0141] According to the speech enhancement evaluation method, training method and device of the speech enhancement evaluation model of the exemplary embodiment of the present disclosure, based on deep learning, a speech enhancement evaluation model is used that is only for test samples, has no requirements on complexity and real-time performance, and is given the original speech component and the original noise component. The enhanced noisy speech output by the speech enhancement model to be tested is separated from the speech and residual noise. This is equivalent to using a more capable model that knows the answer in advance to evaluate another model that is weaker and does not know the answer. Compared with the separation method based on SI-SNR in the related art, it can reduce the result error, thereby effectively measuring the performance of the speech enhancement model to be tested.

[0142] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0143] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A speech enhancement evaluation method for evaluating a speech enhancement model, characterized in that: The speech enhancement evaluation method comprises: Acquire a noisy speech sample, wherein the noisy speech sample includes an original speech sample and an original noise sample; Inputting the noisy speech sample into the speech enhancement model to be tested to obtain the enhanced noisy speech output by the speech enhancement model to be tested; Inputting the enhanced noisy speech, the original speech sample and the original noise sample into a speech enhancement evaluation model to obtain an estimated enhanced speech and an estimated enhanced noise, wherein the estimated enhanced speech is a speech component separated from the enhanced noisy speech by the speech enhancement evaluation model, and the estimated enhanced noise is a residual noise component separated from the enhanced noisy speech by the speech enhancement evaluation model; An evaluation value of the speech enhancement model to be tested is determined according to the estimated enhanced speech and the estimated enhanced noise.

2. The speech enhancement evaluation method according to claim 1, characterized in that: The step of inputting the enhanced noisy speech, the original speech sample, and the original noise sample into a speech enhancement evaluation model to obtain an estimated enhanced speech and an estimated enhanced noise comprises: Inputting the enhanced noisy speech, the original speech sample, and the original noise sample into the speech enhancement evaluation model to obtain one of the estimated enhanced speech and the estimated enhanced noise; One of the estimated enhanced speech and the estimated enhanced noise is removed from the enhanced noisy speech to obtain the other of the estimated enhanced speech and the estimated enhanced noise.

3. The speech enhancement evaluation method according to claim 1, characterized in that: The step of determining the evaluation value of the speech enhancement model to be tested according to the estimated enhanced speech and the estimated enhanced noise includes: According to the estimated enhanced speech and the estimated enhanced noise, a signal-to-noise ratio of the enhanced noisy speech is determined as an evaluation value of the speech enhancement model to be tested.

4. The speech enhancement evaluation method according to claim 3, characterized in that: The enhanced noisy speech, the estimated enhanced speech, and the estimated enhanced noise all include multiple frames of frequency domain signals, each frame of frequency domain signals includes complex signals at multiple center frequencies, wherein determining the signal-to-noise ratio of the enhanced noisy speech according to the estimated enhanced speech and the estimated enhanced noise includes: Determine the sum of the amplitudes of all center frequencies of the estimated enhanced speech and the estimated enhanced noise in each frame respectively as the speech energy and noise energy of the enhanced noisy speech in the corresponding frame; Determining a signal-to-noise ratio of the enhanced noisy speech in a corresponding frame according to a ratio of speech energy to noise energy in each frame of the enhanced noisy speech; A statistical value of the signal-to-noise ratio of the enhanced noisy speech in each frame is determined as the signal-to-noise ratio of the enhanced noisy speech.

5. The speech enhancement evaluation method according to claim 3, characterized in that: The enhanced noisy speech, the estimated enhanced speech, and the estimated enhanced noise are all time domain signals, wherein determining the signal-to-noise ratio of the enhanced noisy speech according to the estimated enhanced speech and the estimated enhanced noise includes: Determine speech energy and noise energy according to the amplitude values ​​of the estimated enhanced speech and the estimated enhanced noise at each moment; The signal-to-noise ratio of the enhanced noisy speech is determined according to the ratio of the speech energy to the noise energy.

6. A method for training a speech enhancement evaluation model, characterized in that: The training method comprises: Obtaining original speech samples and original noise samples; Performing enhancement processing on the original speech sample and the original noise sample respectively to obtain an enhanced speech sample and an enhanced noise sample; Combining the enhanced speech sample and the enhanced noise sample to obtain an enhanced sample; Inputting the enhanced sample, the original speech sample, and the original noise sample into the speech enhancement evaluation model to obtain at least one of an estimated enhanced speech and an estimated enhanced noise; determining a loss based on the estimated enhanced speech and the enhanced speech sample, and / or determining a loss based on the estimated enhanced noise and the enhanced noise sample; The parameters of the speech enhancement evaluation model are adjusted based on the loss to train the speech enhancement evaluation model.

7. The training method according to claim 6, characterized in that: The step of performing enhancement processing on the original speech sample and the original noise sample respectively to obtain an enhanced speech sample and an enhanced noise sample comprises: obtaining gain data for simulating a speech enhancement operation; Based on the gain data, the original speech sample and the original noise sample are enhanced to obtain the enhanced speech sample and the enhanced noise sample.

8. The training method according to claim 7, characterized in that: The gain data is a product of at least one complex gain, wherein the enhancing process is performed on the original speech sample and the original noise sample based on the gain data to obtain the enhanced speech sample and the enhanced noise sample, respectively, including: Converting the original speech sample and the original noise sample from time domain signals to frequency domain signals; Determine the product of the gain data and the original speech sample to obtain the enhanced speech sample; The product of the gain data and the original noise sample is determined to obtain the enhanced noise sample.

9. The training method according to any one of claims 6 to 8, characterized in that: The step of inputting the enhanced sample, the original speech sample, and the original noise sample into the speech enhancement evaluation model to obtain at least one of an estimated enhanced speech and an estimated enhanced noise comprises: Inputting the enhanced sample, the original speech sample, and the original noise sample into the speech enhancement evaluation model to obtain one of the estimated enhanced speech and the estimated enhanced noise; One of the estimated enhanced speech and the estimated enhanced noise is removed from the enhanced sample to obtain the other of the estimated enhanced speech and the estimated enhanced noise.

10. A speech enhancement evaluation device for evaluating a speech enhancement model, characterized in that: The speech enhancement evaluation device comprises: An acquisition unit is configured to: acquire a noisy speech sample, wherein the noisy speech sample includes an original speech sample and an original noise sample; The enhancement unit is configured to: input the noisy speech sample into the speech enhancement model to be tested, and obtain the enhanced noisy speech output by the speech enhancement model to be tested; The separation unit is configured to: input the enhanced noisy speech, the original speech sample and the original noise sample into a speech enhancement evaluation model to obtain an estimated enhanced speech and an estimated enhanced noise, wherein the estimated enhanced speech is a speech component separated from the enhanced noisy speech by the speech enhancement evaluation model, and the estimated enhanced noise is a residual noise component separated from the enhanced noisy speech by the speech enhancement evaluation model; The evaluation unit is configured to determine an evaluation value of the speech enhancement model to be tested according to the estimated enhanced speech and the estimated enhanced noise.

11. The speech enhancement evaluation device according to claim 10, characterized in that: The separation unit is further configured to: Inputting the enhanced noisy speech, the original speech sample, and the original noise sample into the speech enhancement evaluation model to obtain one of the estimated enhanced speech and the estimated enhanced noise; One of the estimated enhanced speech and the estimated enhanced noise is removed from the enhanced noisy speech to obtain the other of the estimated enhanced speech and the estimated enhanced noise.

12. The speech enhancement evaluation device according to claim 10, characterized in that: The evaluation unit is further configured to: According to the estimated enhanced speech and the estimated enhanced noise, a signal-to-noise ratio of the enhanced noisy speech is determined as an evaluation value of the speech enhancement model to be tested.

13. The speech enhancement evaluation device according to claim 12, characterized in that: The enhanced noisy speech, the estimated enhanced speech and the estimated enhanced noise all include multiple frames of frequency domain signals, each frame of frequency domain signals includes complex signals at multiple center frequencies, and the evaluation unit is further configured as follows: Determine the sum of the amplitudes of all center frequencies of the estimated enhanced speech and the estimated enhanced noise in each frame respectively as the speech energy and noise energy of the enhanced noisy speech in the corresponding frame; Determining a signal-to-noise ratio of the enhanced noisy speech in a corresponding frame according to a ratio of speech energy to noise energy in each frame of the enhanced noisy speech; A statistical value of the signal-to-noise ratio of the enhanced noisy speech in each frame is determined as the signal-to-noise ratio of the enhanced noisy speech.

14. The speech enhancement evaluation device according to claim 12, characterized in that: The enhanced noisy speech, the estimated enhanced speech and the estimated enhanced noise are all time domain signals, and the evaluation unit is further configured to: Determine speech energy and noise energy according to the amplitude values ​​of the estimated enhanced speech and the estimated enhanced noise at each moment; The signal-to-noise ratio of the enhanced noisy speech is determined according to the ratio of the speech energy to the noise energy.

15. A training device for a speech enhancement evaluation model, characterized in that: The training device comprises: The acquisition unit is configured to: acquire an original speech sample and an original noise sample; The enhancement unit is configured to: perform enhancement processing on the original speech sample and the original noise sample respectively to obtain an enhanced speech sample and an enhanced noise sample; A merging unit, configured to: merge the enhanced speech sample and the enhanced noise sample to obtain an enhanced sample; The separation unit is configured to: input the enhanced sample, the original speech sample, and the original noise sample into the speech enhancement evaluation model to obtain at least one of the estimated enhanced speech and the estimated enhanced noise; a calculation unit configured to: determine a loss according to the estimated enhanced speech and the enhanced speech sample, and / or determine a loss according to the estimated enhanced noise and the enhanced noise sample; The parameter adjustment unit is configured to adjust the parameters of the speech enhancement evaluation model based on the loss to train the speech enhancement evaluation model.

16. The training device according to claim 15, characterized in that The enhancement unit is further configured to: obtaining gain data for simulating a speech enhancement operation; Based on the gain data, the original speech sample and the original noise sample are enhanced to obtain the enhanced speech sample and the enhanced noise sample.

17. The training device according to claim 16, characterized in that The gain data is a product of at least one complex gain, and the enhancement unit is further configured to: Converting the original speech sample and the original noise sample from time domain signals to frequency domain signals; Determine the product of the gain data and the original speech sample to obtain the enhanced speech sample; The product of the gain data and the original noise sample is determined to obtain the enhanced noise sample.

18. A training device according to any one of claims 15 to 17, characterized in that The separation unit is further configured to: Inputting the enhanced sample, the original speech sample, and the original noise sample into the speech enhancement evaluation model to obtain one of the estimated enhanced speech and the estimated enhanced noise; One of the estimated enhanced speech and the estimated enhanced noise is removed from the enhanced sample to obtain the other of the estimated enhanced speech and the estimated enhanced noise.

19. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the speech enhancement evaluation method as described in any one of claims 1 to 5 or the training method of the speech enhancement evaluation model as described in any one of claims 6 to 9.

20. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the speech enhancement evaluation method as described in any one of claims 1 to 5 or the training method of the speech enhancement evaluation model as described in any one of claims 6 to 9.

21. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by at least one processor, the speech enhancement evaluation method according to any one of claims 1 to 5 or the training method of the speech enhancement evaluation model according to any one of claims 6 to 9 is implemented.

Images