Speech enhancement method, apparatus and device

Abstract

The application discloses a speech enhancement method, device and equipment. The method estimates the noise mask of the complex spectrum of the noisy speech by a noise mask prediction model based on complex value calculation. The model uses complex operation to better model the correlation between the amplitude spectrum and the phase spectrum by using prior knowledge, avoids treating the real part and the imaginary part as two unrelated parts for operation respectively like a real value network, generates enhanced speech data of the noisy speech data according to the masking value of the complex spectrum and the complex spectrum of the noisy speech, so that the amplitude spectrum and the phase spectrum can be enhanced at the same time. Therefore, the hearing quality of the enhanced speech can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and specifically to speech enhancement methods and devices, noise mask prediction model processing methods and devices, video conferencing systems, video conferencing speech enhancement methods, telephone conferencing systems, telephone conferencing speech enhancement methods, online education systems, online education speech enhancement methods, live broadcast systems, live broadcast speech enhancement methods, speech recognition methods and systems, speech recognition text editing systems, user recognition systems, and electronic devices. Background Art

[0002] Speech enhancement (noise reduction) technology has been applied to scenarios in multiple fields, such as audio and video conferencing and online education in the field of real-time communication (RTC), and speech recognition and speaker recognition in the field of machine recognition. Its purpose is to suppress other noises besides the target speech, such as howling, mobile phone ringtones, keyboard sounds, and background voices.

[0003] With the application of deep learning technology, intelligent speech noise reduction has significant advantages over traditional signal processing noise reduction, particularly in model learning and handling non-stationary noise. Currently, a typical deep learning-based speech enhancement solution uses a non-end-to-end speech enhancement model to enhance the amplitude spectrum of speech based on manually designed speech features.

[0004] However, in the process of implementing the present invention, the inventors found that the above solution has at least the following problems: since a non-end-to-end speech enhancement model is adopted, artificially designed speech features are used, and the phase spectrum is not enhanced, the phase spectrum of noisy speech is still used, which affects the perceptual quality and intelligibility of the enhanced speech. Summary of the Invention

[0005] The present application provides a speech enhancement method to address the poor speech noise reduction performance issues of the prior art. The present application also provides a speech enhancement device, a noise mask prediction model processing method and device, a video conferencing system, a video conferencing speech enhancement method, a telephone conferencing system, a telephone conferencing speech enhancement method, an online education system, an online education speech enhancement method, a live broadcast system, a live broadcast speech enhancement method, a speech recognition method and system, a speech recognition text editing system, a user recognition system, and an electronic device.

[0006] This application provides a speech enhancement method, comprising:

[0007] Determining complex time-frequency domain feature data of the noisy speech data including an amplitude spectrum and a phase spectrum;

[0008] Determining a noise mask for the time-frequency domain feature data using a noise mask prediction model based on complex value calculation;

[0009] Enhanced speech data of the noisy speech data is generated according to the noise mask and the time-frequency domain feature data.

[0010] Optionally, determining the noise mask of the time-frequency domain feature data by using a noise mask prediction model based on complex value calculation includes:

[0011] Determining first depth feature data based on the time-frequency domain feature data by an encoder in the noise mask prediction model;

[0012] The noise mask is determined according to the first depth feature data by a decoder in a noise mask prediction model.

[0013] Optionally, the encoder includes multiple encoding layers, and the decoder includes multiple decoding layers;

[0014] The step of determining the noise mask of the time-frequency domain feature data by using a noise mask prediction model based on complex value calculation further includes:

[0015] Determining, by a first attention module corresponding to the coding layer in the noise mask prediction model, third depth feature data of at least one depth level based on the second depth feature data output by the coding layer; wherein the third depth feature data is residual input data of a decoding layer corresponding to the coding layer; and the third depth feature data includes depth feature information requiring special attention extracted from the second depth feature data;

[0016] The noise mask is determined by a decoder in a noise mask prediction model according to the first depth feature data and the third depth feature data.

[0017] Optionally, determining the noise mask according to the first depth feature data and the third depth feature data by a decoder in the noise mask prediction model includes:

[0018] Determining, by a second attention module before the decoding layer, second decoded data based on first decoded data output by a decoding layer preceding the decoding layer;

[0019] The first decoded data output by the decoding layer is determined through the decoding layer according to the second decoded data and the third depth feature data corresponding to the encoding layer corresponding to the decoding layer.

[0020] Optionally, the first attention module includes: a complex-valued channel attention module and a complex-valued spatial attention module;

[0021] The third depth feature data is determined by the following steps:

[0022] Determining, by the complex-valued channel attention module, fourth depth feature data of a plurality of complex-valued channels based on the second depth feature data output by the encoding layer; the fourth depth feature data includes depth feature information of different channels requiring special attention extracted from the second depth feature data;

[0023] The third depth feature data is determined based on the fourth depth feature data of the multiple channels through the complex-valued spatial attention module.

[0024] Optionally, the loss function of the noise mask prediction model includes: a joint time-frequency domain loss function to enhance the correlation between the amplitude spectrum and the phase spectrum.

[0025] Optionally, generating enhanced speech data of noisy speech data according to the noise mask and the time-frequency domain feature data includes:

[0026] Using the calculation results of the time-frequency domain feature data and the noise mask as time-frequency domain enhanced feature data;

[0027] The enhanced speech data is generated according to the time-frequency domain enhanced feature data.

[0028] Optionally, the time-frequency domain feature data includes: short-time Fourier transform (STFT) feature data;

[0029] Generating the enhanced speech data according to the time-frequency domain enhanced feature data includes:

[0030] The inverse short-time Fourier transform data of the time-frequency domain enhanced feature data is used as the enhanced speech data.

[0031] This application also provides a noise mask prediction model processing method, including:

[0032] Determine a training data set, the training data comprising: complex time-frequency domain feature data of noisy speech data including an amplitude spectrum and a phase spectrum, and a noise mask of the time-frequency domain feature data;

[0033] Construct the network structure of the noise mask prediction model based on complex-valued networks;

[0034] The network parameters of the noise mask prediction model are trained according to the training data set.

[0035] This application also provides a video conferencing system, including:

[0036] The first client is used to collect the video conference voice data and send the voice data to the server;

[0037] The server is configured to determine complex time-frequency domain feature data of the speech data, including an amplitude spectrum and a phase spectrum; determine a noise mask for the time-frequency domain feature data using a noise mask prediction model based on complex value calculation; generate enhanced speech data of the speech data based on the noise mask and the time-frequency domain feature data; and send the enhanced speech data to a second client;

[0038] The second client is used to play the enhanced voice data.

[0039] This application also provides a video conferencing voice enhancement method, comprising:

[0040] Collect video conference voice data;

[0041] Determining complex time-frequency domain feature data of the speech data including an amplitude spectrum and a phase spectrum;

[0042] Determining a noise mask for the time-frequency domain feature data using a noise mask prediction model based on a complex-valued computing network;

[0043] Enhanced voice data of the video conference voice data is generated according to the noise mask and the time-frequency domain feature data.

[0044] This application also provides a telephone conference system, including:

[0045] A first client, configured to collect conference call voice data and send the voice data to a server;

[0046] The server is configured to determine complex time-frequency domain feature data of the speech data, including an amplitude spectrum and a phase spectrum; determine a noise mask for the time-frequency domain feature data using a noise mask prediction model based on a complex-valued computing network; generate enhanced speech data of the speech data based on the noise mask and the time-frequency domain feature data; and send the enhanced speech data to a second client;

[0047] The second client is used to play the enhanced voice data.

[0048] This application also provides a method for enhancing voice in a telephone conference, comprising:

[0049] Collecting conference call voice data;

[0050] Determining complex time-frequency domain feature data of the speech data including an amplitude spectrum and a phase spectrum;

[0051] Determining a noise mask for the time-frequency domain feature data using a noise mask prediction model based on a complex-valued computing network;

[0052] Enhanced speech data of the telephone conference speech data is generated according to the noise mask and the time-frequency domain feature data.

[0053] This application also provides an online education system, including:

[0054] The first client is used to collect online education voice data and send the voice data to the server;

[0055] The server is configured to determine complex time-frequency domain feature data of the speech data, including an amplitude spectrum and a phase spectrum; determine a noise mask for the time-frequency domain feature data using a noise mask prediction model based on a complex-valued computing network; generate enhanced speech data of the speech data based on the noise mask and the time-frequency domain feature data; and send the enhanced speech data to a second client;

[0056] The second client is used to play the enhanced voice data.

[0057] This application also provides an online education speech enhancement method, comprising:

[0058] Collect online education voice data;

[0059] Determining complex time-frequency domain feature data of the speech data including an amplitude spectrum and a phase spectrum;

[0060] Determining a noise mask for the time-frequency domain feature data using a noise mask prediction model based on a complex-valued computing network;

[0061] Enhanced speech data of the online education speech data is generated according to the noise mask and the time-frequency domain feature data.

[0062] This application also provides a live broadcast system, including:

[0063] The first client is used to collect live voice data and send the voice data to the server;

[0064] The server is configured to determine complex time-frequency domain feature data of the speech data, including an amplitude spectrum and a phase spectrum; determine a noise mask for the time-frequency domain feature data using a noise mask prediction model based on a complex-valued computing network; generate enhanced speech data of the speech data based on the noise mask and the time-frequency domain feature data; and send the enhanced speech data to a second client;

[0065] The second client is used to play the enhanced voice data.

[0066] This application also provides a live voice enhancement method, comprising:

[0067] Collect live voice data;

[0068] Determining complex time-frequency domain feature data of the speech data including an amplitude spectrum and a phase spectrum;

[0069] Determining a noise mask for the time-frequency domain feature data using a noise mask prediction model based on a complex-valued computing network;

[0070] Enhanced speech data of the live speech data is generated according to the noise mask and the time-frequency domain feature data.

[0071] This application also provides a speech recognition system, comprising:

[0072] The client is used to collect voice data and send the voice data to the server;

[0073] The server is used to determine the complex time-frequency domain feature data of the speech data, including the amplitude spectrum and the phase spectrum; determine the noise mask of the time-frequency domain feature data through a noise mask prediction model based on a complex value computing network; generate enhanced speech data of the speech data based on the noise mask and the time-frequency domain feature data; and convert the enhanced speech data into text through a speech recognition model.

[0074] This application also provides a speech recognition method, comprising:

[0075] Collect voice data;

[0076] Determining complex time-frequency domain feature data of the speech data including an amplitude spectrum and a phase spectrum;

[0077] Determining a noise mask for the time-frequency domain feature data using a noise mask prediction model based on a complex-valued computing network;

[0078] generating enhanced speech data of the speech data according to the noise mask and the time-frequency domain feature data;

[0079] The enhanced speech data is converted into text through the speech recognition model.

[0080] This application also provides a speech recognition text editing system, including:

[0081] The client is used to collect voice data and send the voice data to the server; and edit the text of the voice data recognized by the server;

[0082] The server is used to determine the complex time-frequency domain feature data of the speech data, including the amplitude spectrum and the phase spectrum; determine the noise mask of the time-frequency domain feature data through a noise mask prediction model based on a complex value computing network; generate enhanced speech data of the speech data based on the noise mask and the time-frequency domain feature data; and convert the enhanced speech data into text through a speech recognition model.

[0083] This application also provides a user identification system, including:

[0084] The client is used to collect voice data and send the voice data to the server;

[0085] The server is used to determine the complex time-frequency domain feature data of the speech data, including the amplitude spectrum and the phase spectrum; determine the noise mask of the time-frequency domain feature data through a noise mask prediction model based on a complex value computing network; generate enhanced speech data of the speech data based on the noise mask and the time-frequency domain feature data; and determine the user information of the enhanced speech data through a user recognition model.

[0086] This application also provides a user identification method, including:

[0087] Collect voice data;

[0088] Determining complex time-frequency domain feature data of the speech data including an amplitude spectrum and a phase spectrum;

[0089] Determining a noise mask for the time-frequency domain feature data using a noise mask prediction model based on a complex-valued computing network;

[0090] generating enhanced speech data of the speech data according to the noise mask and the time-frequency domain feature data;

[0091] The user information of the enhanced voice data is determined through the user identification model.

[0092] The present application also provides a speech enhancement device, comprising:

[0093] A data conversion unit, configured to determine complex time-frequency domain feature data of the noisy speech data, including an amplitude spectrum and a phase spectrum;

[0094] a noise mask prediction unit, configured to determine a noise mask of the time-frequency domain feature data by using a noise mask prediction model based on complex value calculation;

[0095] The enhanced speech generating unit is configured to generate enhanced speech data of the noisy speech data according to the noise mask and the time-frequency domain feature data.

[0096] The present application also provides a noise mask prediction model processing device, comprising:

[0097] A training data determination unit is used to determine a training data set, wherein the training data includes: complex time-frequency domain feature data including an amplitude spectrum and a phase spectrum of the noisy speech data, and a noise mask of the time-frequency domain feature data;

[0098] A network construction unit, used to construct a network structure of a noise mask prediction model based on a complex-valued network;

[0099] A network training unit is used to train the network parameters of the noise mask prediction model according to the training data set.

[0100] The present application also provides an electronic device, comprising:

[0101] processor and memory;

[0102] The memory is used to store a program for implementing the above method. The device is powered on and the processor runs the program for implementing the method.

[0103] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned various methods.

[0104] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to perform the above methods.

[0105] Compared with the prior art, this application has the following advantages:

[0106] The speech enhancement method provided in the embodiments of the present application estimates the noise mask (masking value) of the time-frequency domain feature data (complex spectrum) of the noisy speech through a noise mask prediction model based on complex value calculation. The model uses complex operations to better model the correlation between the amplitude spectrum and the phase spectrum using prior knowledge, avoiding the situation where the real part and the imaginary part are treated as two unrelated parts and operated separately as in a real-valued network. Then, based on the masking value of the complex spectrum and the time-frequency domain feature data of the noisy speech, enhanced speech data of the noisy speech data is generated. In this way, the amplitude spectrum and the phase spectrum can be enhanced at the same time; therefore, the listening quality of the enhanced speech can be effectively improved.

[0107] The noise mask prediction model processing method provided in the embodiments of the present application constructs a noise mask prediction model based on complex value calculation. The model directly estimates the masking value of the complex spectrum of the noisy speech. Through complex operations, prior knowledge can be used to better model the correlation between the amplitude spectrum and the phase spectrum, avoiding the situation where the real part and the imaginary part are treated as two unrelated parts and operated separately as a real-valued network. Then, based on the masking value of the complex spectrum and the complex spectrum of the noisy speech, enhanced speech data of the noisy speech data is generated. In this way, the amplitude spectrum and the phase spectrum can be enhanced at the same time; therefore, the listening quality of the enhanced speech can be effectively improved.

[0108] The video conferencing system provided in an embodiment of the present application estimates the masking value of the complex spectrum of the video conferencing speech through a noise mask prediction model based on complex value calculation. The model uses complex operations to utilize prior knowledge to better model the correlation between the amplitude spectrum and the phase spectrum, avoiding the situation where the real part and the imaginary part are treated as two unrelated parts and operated separately like a real-valued network. Then, based on the masking value of the complex spectrum and the complex spectrum of the noisy speech, enhanced speech data of the noisy speech data is generated, thereby enhancing the amplitude spectrum and the phase spectrum at the same time; therefore, noise and howling can be effectively suppressed, thereby improving the enhanced video conferencing speech quality.

[0109] The telephone conferencing system provided in the embodiment of the present application estimates the masking value of the complex spectrum of the telephone conference speech through a noise mask prediction model based on complex value calculation. The model uses complex operations to utilize prior knowledge to better model the correlation between the amplitude spectrum and the phase spectrum, avoiding the real-valued network that treats the real part and the imaginary part as two unrelated parts and performs separate operations. Then, based on the masking value of the complex spectrum and the complex spectrum of the noisy speech, enhanced speech data of the noisy speech data is generated, so that the amplitude spectrum and the phase spectrum can be enhanced at the same time; therefore, noise and howling can be effectively suppressed, thereby improving the enhanced telephone conference speech quality.

[0110] The online education system provided in the embodiment of the present application estimates the masking value of the complex spectrum of the online teaching speech through a noise mask prediction model based on complex value calculation. The model uses complex operations to utilize prior knowledge to better model the correlation between the amplitude spectrum and the phase spectrum, avoiding the real-valued network that treats the real part and the imaginary part as two unrelated parts and performs separate operations. Then, based on the masking value of the complex spectrum and the complex spectrum of the noisy speech, enhanced speech data of the noisy speech data is generated, which can simultaneously enhance the amplitude spectrum and the phase spectrum; therefore, it can effectively suppress noise and howling, thereby improving the quality of the enhanced online teaching speech.

[0111] The live broadcast system provided in the embodiment of the present application estimates the masking value of the complex spectrum of the live broadcast voice through a noise mask prediction model based on complex value calculation. The model uses complex operations to make use of prior knowledge to better model the correlation between the amplitude spectrum and the phase spectrum, avoiding the real part and the imaginary part being treated as two unrelated parts and operated separately like a real-valued network. Then, based on the masking value of the complex spectrum and the complex spectrum of the noisy speech, enhanced speech data of the noisy speech data is generated, so that the amplitude spectrum and the phase spectrum can be enhanced at the same time; therefore, noise and howling can be effectively suppressed, thereby improving the quality of the enhanced live broadcast voice.

[0112] The speech recognition system provided in an embodiment of the present application estimates the masking value of the complex spectrum of noisy speech through a noise mask prediction model based on complex value calculations. The model uses complex operations to better model the correlation between the amplitude spectrum and the phase spectrum using prior knowledge, avoiding the situation where a real-valued network treats the real part and the imaginary part as two unrelated parts and performs separate operations. Enhanced speech data of the noisy speech data is then generated based on the masking value of the complex spectrum and the complex spectrum of the noisy speech. In this way, the amplitude spectrum and the phase spectrum can be enhanced simultaneously. The enhanced speech data is then converted into text through the speech recognition model. Therefore, noise and howling can be effectively suppressed, thereby improving speech quality and, in turn, speech recognition accuracy.

[0113] The speech recognition text editing system provided in the embodiments of the present application estimates the masking value of the complex spectrum of noisy speech through a noise mask prediction model calculated by complex values. The model uses complex operations to make use of prior knowledge to better model the correlation between the amplitude spectrum and the phase spectrum, avoiding the situation where the real part and the imaginary part are treated as two unrelated parts and operated separately like a real-valued network. Then, based on the masking value of the complex spectrum and the complex spectrum of the noisy speech, enhanced speech data of the noisy speech data is generated. In this way, the amplitude spectrum and the phase spectrum can be enhanced at the same time. The enhanced speech data is then converted into text through the speech recognition model, and the transcribed text is edited. Therefore, noise and howling can be effectively suppressed, thereby improving speech quality, thereby improving speech recognition accuracy, and thereby improving speech recognition text editing efficiency.

[0114] The user identification system provided in the embodiments of the present application estimates the masking value of the complex spectrum of noisy speech through a noise mask prediction model based on complex value calculations. The model uses complex operations to better model the correlation between the amplitude spectrum and the phase spectrum using prior knowledge, avoiding the situation where a real-valued network treats the real and imaginary parts as two unrelated parts and performs separate operations. Enhanced speech data of the noisy speech data is then generated based on the masking value of the complex spectrum and the complex spectrum of the noisy speech. This allows the amplitude spectrum and phase spectrum to be enhanced simultaneously, and user information of the enhanced speech data is then determined through a user identification model. This processing method enables an acoustic feature enhancement model to be obtained through an adversarial multi-task learning method based on self-supervised noise classification loss. The enhanced acoustic features of the noisy speech are determined using this model, which can avoid sensitivity to environmental noise when extracting the enhanced acoustic features. User identification processing is then performed based on the enhanced speech. Therefore, noise and howling can be effectively suppressed, thereby improving speech quality, thereby improving speech recognition accuracy, and thus improving speaker recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0115] Figure 1 Schematic diagram of an application scenario of an embodiment of the speech enhancement method provided in this application;

[0116] Figure 2 A flowchart of an embodiment of the speech enhancement method provided in this application;

[0117] Figure 3 A schematic diagram of a noise mask prediction model according to an embodiment of the speech enhancement method provided in the present application;

[0118] Figure 4 A schematic diagram of a noise mask prediction model with residual connection in an embodiment of the speech enhancement method provided by the present application;

[0119] Figure 5 The present application provides a flowchart of a complex-valued channel attention module and a spatial module of an embodiment of a speech enhancement method. DETAILED DESCRIPTION

[0120] The following description sets forth many specific details to facilitate a thorough understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of the present application. Therefore, the present application is not limited to the specific implementations disclosed below.

[0121] This application provides a speech enhancement method and apparatus, a speech enhancement model processing system, a noise mask prediction model method and apparatus, a video conferencing system, a video conferencing speech enhancement method, a telephone conferencing system, a telephone conferencing speech enhancement method, an online education system, an online education speech enhancement method, a live broadcast system, a live broadcast speech enhancement method, a speech recognition method and system, a speech recognition text editing system, a user identification system, and an electronic device. Each of these solutions is described in detail in the following embodiments.

[0122] First embodiment

[0123] The present application provides a speech enhancement method, the execution subject of which is a speech enhancement device, which is usually deployed on the server side, but is not limited to the server side, and can also be any device that can implement the speech enhancement method.

[0124] The method provided in the embodiment of the present application separates speech from background noise (environmental noise) and can be applied in a variety of speech processing systems, such as speech recognition systems, speaker recognition systems, speech recognition text editing systems, etc. Figure 1 , which is a schematic diagram of the usage scenario of an embodiment of the speech enhancement method of the present application. In this embodiment, the method is applied in a speech recognition text editing system. The system includes a server and a client. The server deploys a speech enhancement device, and collects user voice data through the client. Due to the presence of environmental noise, the voice data is noisy voice data; the client sends the noisy voice data to the server, and the server converts the voice data into text; accordingly, the server executes the method, performs speech enhancement processing on the noisy voice data through the speech enhancement device, that is, suppresses the environmental noise, and then performs speech recognition processing on the enhanced voice through the speech recognition model, and sends the recognized text back to the client so that the user can browse and edit the text.

[0125] Please refer to Figure 2 , which is a flow chart of an embodiment of the speech enhancement method of the present application. In this embodiment, the method may include the following steps:

[0126] Step S201: Determine complex time-frequency domain feature data of noisy speech data, including amplitude spectrum and phase spectrum.

[0127] The noisy speech data can be single-channel speech data collected by a microphone. The noisy speech data can be a one-dimensional array, the length of which is determined by the audio length and the sampling rate. For example, a sampling rate Fs of 16 kHz indicates that 16,000 points are sampled in one second. In this case, if the audio length is 10 seconds, the noisy speech data will contain 160,000 values, and the size of the values ​​generally represents the amplitude.

[0128] The time-frequency domain feature data is a representation of the speech signal decomposed in time and frequency. The time-frequency feature data is a complex spectrum, which may include an amplitude spectrum and a phase spectrum, and the complex spectrum is expressed in an imaginary part and a real part. The complex spectrum includes richer speech features, which makes the enhanced speech purer. Because noisy speech data contains environmental noise, the time-frequency domain feature data of this speech data is noisy time-frequency domain feature data.

[0129] In a specific implementation, the time-frequency feature data can be a spectrogram, that is, the time-frequency domain feature data can be the complex spectrum obtained by the short-time Fourier transform (STFT) of the speech waveform. STFT is a general tool for speech signal processing. It defines a time and frequency distribution class, which specifies the complex amplitude of any signal changing with time and frequency. The process of calculating the short-time Fourier transform is to divide a longer time signal into shorter segments (speech frames) of the same length, and calculate the Fourier transform, that is, the Fourier spectrum, on each shorter segment. In other words, STFT is to perform a discrete Fourier transform (DCT) on a series of windowed data. The function of the Fourier transform is to convert the time domain signal into a frequency domain signal. The spectrogram can be obtained by stacking the transformed frequency domain signal (spectrogram) of each frame in time.

[0130] After converting the noisy speech data into complex (including imaginary and real) time-frequency domain feature data including amplitude spectrum and phase spectrum, we can proceed to the next step, using the time-frequency domain feature data as input data of a noise mask prediction model based on complex value calculation, and determining the noise mask of the time-frequency domain feature data, that is, the masking value, through the noise mask prediction model.

[0131] Step S203: Determine the noise masks of the amplitude spectrum and the phase spectrum according to the time-frequency domain feature data using a noise mask prediction model based on complex value calculation.

[0132] The noise mask prediction model can adopt a neural network structure (complex network) based on complex number calculation. The input data of the noise mask prediction model is a complex value, that is, time-frequency domain feature data including an amplitude spectrum and a phase spectrum. In this embodiment, the complex value input data is divided into two parts, a real part and an imaginary part, which are respectively input into the complex value network for complex number operations. The noise mask of the amplitude spectrum and phase spectrum of the noisy speech is estimated by the noise mask prediction model based on the complex value network, and the output data obtained from the noise mask prediction model are the real part and imaginary part of the complex ideal ratio mask, respectively, which can improve the correlation between the amplitude spectrum and the phase spectrum.

[0133] The noise mask, also known as the masking value or mask value, is the target data that the noise mask prediction model needs to learn and output. It is a noise mask of the complex time-frequency domain feature data. The noise mask is a complex value consisting of real and imaginary parts. The noise mask defines the ratio of the clean speech signal to the noisy speech signal in the time-frequency domain. Based on the noise mask and the noisy speech, the clean speech can be reconstructed. The noise mask includes but is not limited to the Complex Ideal Ratio Mask.

[0134] The noise mask prediction model can be based on a Deep Complex U-net (DCUnet). The DCUnet-based noise mask prediction model includes an encoder and a decoder. The encoder can use a downsampling mechanism to extract high-order features, and the decoder can use upsampling to reconstruct the target map.

[0135] In one example, step S203 may include the following sub-steps: 1) determining first depth feature data based on time-frequency domain feature data through an encoder in a noise mask prediction model; 2) determining the noise mask based on the first depth feature data through a decoder in a noise mask prediction model.

[0136] Different from the manually designed time-frequency domain feature data, the first depth feature data is an abstract data representation obtained by processing the time-frequency domain feature data through weighted operation of an encoder in a noise mask prediction model pre-trained based on prior knowledge (model training data). The depth feature can be a high-order feature, also known as encoded data.

[0137] The encoder may include multiple encoding layers, and the decoder may include multiple decoding layers. The encoding layers and decoding layers may adopt complex-valued convolutional networks. In this case, the network structure of the noise mask prediction model includes a complex-valued convolutional encoder and a complex-valued convolutional decoder.

[0138] In practice, a long short-term memory (LSTM) layer can be added between the encoder and decoder to model temporal dependencies. This means that the encoding and decoding layers can employ LSTM neural networks. This approach combines the decoding and encoding networks with the LSTM network to implement a deep, complex-valued convolutional recurrent neural network. This allows for more accurate noise mask predictions based on contextual information (such as complex time-frequency feature data from adjacent frames).

[0139] After obtaining the complex-valued mask through the noise mask prediction module based on complex-valued calculation, the next step can be entered to perform noise reduction processing on the noisy speech.

[0140] Step S205: generating enhanced speech data of the noisy speech data according to the noise mask and the time-frequency domain feature data.

[0141] In this step, the masking value of the complex time-frequency domain feature data (including the amplitude spectrum and the phase spectrum) is applied to the time-frequency domain feature data (such as the spectrogram) of the noisy speech to obtain enhanced clean speech.

[0142] In one example, step S205 may include the following sub-steps: 1) using the result of the operation (e.g., product, n-fold product, etc.) of the time-frequency domain feature data and the noise mask as time-frequency domain enhanced feature data; and 2) generating the enhanced speech data based on the time-frequency domain enhanced feature data. The time-frequency domain enhanced feature data may be a spectrogram of enhanced clean speech. The enhanced speech data is obtained by converting the spectrogram of the clean speech into waveform speech.

[0143] In specific implementation, before using the calculation result of the time-frequency domain feature data and the noise mask as the time-frequency domain enhanced feature data, the noise mask may be normalized, which can further improve the accuracy of the time-frequency domain enhanced feature data.

[0144] Please see Figure 3 , which is a specific flow chart of an embodiment of the speech enhancement method of the present application. In this embodiment, the complex time-frequency domain feature data X including the amplitude spectrum and the phase spectrum of the noisy speech data x is first determined by the STFT data conversion unit; the noise mask of the amplitude spectrum and the phase spectrum of the noisy speech is estimated by a noise mask prediction model based on complex value calculation to obtain a complex masking value O; the real part and the imaginary part of the complex masking value are then normalized to obtain a normalized complex masking value M. In specific implementation, an activation function (such as a hyperbolic tangent tanh activation function) can be applied to limit the real part and the imaginary part of the complex masking value to between -1 and 1; then, a complex product operation is performed on the noisy speech spectrogram (time-frequency domain feature data X) and the complex masking (M) to obtain an estimated clean speech spectrogram, that is, the time-frequency domain enhanced feature data Y, thereby simultaneously enhancing the amplitude spectrum and the phase spectrum; finally, the waveform is converted into an estimated clean speech time domain waveform by an inverse short-time Fourier transform (ISTFT), that is, the enhanced speech data y.

[0145] The noise mask prediction model can be constructed using the following steps: 1) determining a training data set, wherein the training data includes time-frequency domain feature data of noisy speech data including an amplitude spectrum and a phase spectrum and a noise mask (masking value) of the amplitude spectrum and the phase spectrum; 2) constructing a network structure of the noise mask prediction model based on complex value calculation; 3) training the network parameters of the noise mask prediction model based on the training data set.

[0146] In one example, the loss function of the noise mask prediction model includes a joint time-frequency domain loss function, that is, combining the time-domain loss function and the time-frequency domain loss function to perform model optimization. This processing approach allows the noise mask prediction model to be trained based on the hybrid loss function to jointly optimize the complex-valued model in the time-frequency domain and the time domain. This can further enhance the correlation between the amplitude spectrum and the phase spectrum, thereby constructing a more accurate noise mask prediction model.

[0147] In specific implementation, the time domain loss function that can be optimized includes but is not limited to: Scale-Invariant Signal to Noise Ratio (SI-SNR), and the time-frequency domain loss function that can be optimized includes but is not limited to: mean square error of complex-valued masking estimation (i.e., noise mask).

[0148] The inventors of this application have discovered through experiments that determining the masking value of the complex spectrum using a noise mask prediction model based on a deep complex-valued convolutional network relies heavily on the expressive power of the complex-valued convolutional network. To improve the expressive power of the complex-valued convolutional network, in one example, step S203 may include the following sub-steps:

[0149] Step S2031: Determine first depth feature data based on time-frequency domain feature data through the encoder in the noise mask prediction model; and determine third depth feature data of at least one depth level based on the second depth feature data output by the encoding layer through the first attention module corresponding to the encoding layer in the noise mask prediction model.

[0150] In this embodiment, the encoder includes multiple encoding layers, and the first depth feature data is the data output by the last encoding layer of the encoder. For ease of description, the data output by the remaining encoding layers in this embodiment is referred to as the second depth feature data. Different encoding layers output second depth feature data at different depth levels. The depth level of the second depth feature data depends on the order in which the data is processed in the deep network.

[0151] The second depth feature data includes a part of feature data that is more important for noise mask prediction and another part of unimportant feature data. The second depth feature data output by different coding layers can be processed by different first attention modules to obtain third depth feature data of different depth levels. The third depth feature data is important feature data in the current prediction situation extracted from the second depth feature data through a neural network pre-trained based on prior knowledge. The third depth feature data is the residual input data of the decoding layer corresponding to the coding layer; the third depth feature data includes the depth feature information that needs to be focused on extracted from the second depth feature data.

[0152] The first attention module is responsible for checking the second depth feature information of different depth levels that is directly related to the target (i.e., noise mask) that the noise mask prediction model needs to learn, allowing useful information to pass through and blocking useless information, so as to effectively use information in limited operating resources. The first attention module is a mechanism that can effectively configure limited computing resources, thereby giving priority to more critical information features. In this case, the entire noise mask prediction model includes an encoder, a decoder, and an attention module, wherein the encoder and the decoder respectively contain multiple layers of deep network convolutional layers. The encoder abstracts data features layer by layer through a downsampling process, and the decoder reproduces signal data layer by layer through an upsampling process. The corresponding layers of the encoder and decoder facilitate the passage of effective information through a shortcut connection. An attention module is added to each shortcut connection to extract effective information and pass it to the corresponding decoder layer.

[0153] The first attention module used in this embodiment is an independent, lightweight, and universal module that can be easily embedded in any complex-valued convolutional layer and optimized together with the entire noise mask prediction model. In specific implementations, the first attention module can adopt a complex convolutional block attention module (CCBAM).

[0154] Please see Figure 4, which is a model diagram of the first attention module of the embodiment of the speech enhancement method of the present application. In this embodiment, the first attention module includes: a complex-valued channel attention module and a complex-valued spatial attention module; the third depth feature data can be determined by the following steps: 1) through the complex-valued channel attention module, according to the second depth feature data output by the encoding layer, the fourth depth feature data of multiple channels is determined; 2) through the complex-valued spatial attention module, according to the fourth depth feature data of the multiple channels, the third depth feature data is determined. Wherein, the fourth depth feature data includes the depth feature information of different channels that needs to be focused on extracted from the second depth feature data. By adopting this processing method, the attention mechanism based on the fusion of spatial and channel information is used in the complex-valued network modeling, which can further improve the speech enhancement performance. In addition, this attention module is a lightweight and universal module that can be easily embedded in the network structure.

[0155] It should be noted that the complex-valued channel attention module and the complex-valued spatial attention module of the first attention module can have different orders and the first attention module can also have different placement positions.

[0156] Step S2033: Determine the noise mask according to the first depth feature data and the third depth feature data through a decoder in a noise mask prediction model, where the decoder includes multiple decoding layers.

[0157] The third depth feature data may be used as residual input data of a decoding layer corresponding to the encoding layer. In a specific implementation, a residual connection may be used to transmit information (the third depth feature data) between each encoder layer and decoder layer.

[0158] The method provided in this embodiment constructs more informative features through the first attention module to improve the representation capability of the complex-valued convolutional network, thereby effectively improving the speech enhancement performance.

[0159] In one example, step S2033 may include the following sub-steps:

[0160] Step S20331: Determine second decoding data based on first decoding data output by a previous decoding layer of the decoding layer through a second attention module before the decoding layer.

[0161] In this example, the encoder abstracts data features layer by layer through a downsampling process, and the decoder reproduces signal data layer by layer through an upsampling process. In this embodiment, the first decoded data output by the decoding layer is input into the second attention module. The first decoded data includes a part of feature data that is more important for noise mask prediction and another part of unimportant feature data. After the first decoded data is processed by the second attention module, the important feature data in the current prediction situation can be extracted from the first decoded data, that is, the second decoded data. The first decoded data output by different decoding layers can be processed by different second attention modules to obtain second decoded data of different depth levels. The second decoded data. Includes feature information that needs to be focused on that is extracted from the first decoded data.

[0162] The second attention module is responsible for screening the first decoded data at different depth levels that are directly related to the target to be learned by the noise mask prediction model (i.e., the noise mask), allowing useful information to pass through and blocking useless information, effectively utilizing information within limited operating resources. The second attention module is a mechanism that effectively allocates limited computing resources, prioritizing more critical information features.

[0163] Step S20332: Determine, through the decoding layer, the first decoded data output by the decoding layer according to the second decoded data and the third depth feature data corresponding to the encoding layer corresponding to the decoding layer.

[0164] The second attention module can use the same network structure as the first attention module. Figure 4 As shown, in this embodiment, the complex-valued channel attention module and the complex-valued spatial attention module are connected in series and embedded in the residual connection and decoder, respectively. This processing method allows the first and second attention modules to construct more informative features, further improving the representation capability of the complex-valued convolutional network, thereby further improving speech enhancement performance.

[0165] In this embodiment, the determining of the fourth depth feature data of multiple channels according to the second depth feature data output by the encoding layer through the complex-valued channel attention module may include the following steps: 1.1) determining the first feature summary information according to the second depth feature data through the compression submodule in the complex-valued channel attention module; 1.2) adjusting the first network weight according to the first feature summary information through the excitation submodule in the complex-valued channel attention module; 1.3) determining the fourth depth feature data of multiple channels according to the adjusted first network weight through the attention submodule in the complex-valued channel attention module.

[0166] The first feature summary information includes summary information of the second depth feature data. In this embodiment, the second depth feature data is compressed by the compression submodule to obtain compressed data of the second depth feature data as the first feature summary information. Figure 5 As shown in a, in a specific implementation, the feature compression operation is performed by the compression submodule in a manner that: the information of the second depth feature data is summarized by using a pooling process (a pooling layer in the network).

[0167] The first network weight includes the weight of the attention submodule performing an attention operation on the second depth feature data, and the attention submodule outputs the fourth depth feature data.

[0168] Correspondingly, the determining of the third depth feature data according to the fourth depth feature data of the multiple channels through the complex-valued spatial attention module may include the following steps: 2.1) determining the second feature summary information according to the fourth depth feature data of the multiple channels through the compression submodule in the complex-valued spatial attention module; 2.2) adjusting the second network weight according to the second feature summary information through the excitation submodule in the complex-valued spatial attention module; 2.3) determining the third depth feature data according to the adjusted second network weight through the attention submodule in the complex-valued spatial attention module.

[0169] Similarly, the second feature summary information includes summary information of the fourth depth feature data. The second network weight includes a weight of the attention submodule performing an attention operation on the fourth depth feature data, and the attention submodule outputs the third depth feature data.

[0170] Please see Figure 5 , which is a schematic diagram of the model of the first attention module of the embodiment of the speech enhancement method of the present application. The attention module consists of two parts, Figure 5 a is the complex-valued channel attention module, Figure 5 b is a complex-valued spatial attention module. Both modules first perform a compression operation and then an excitation operation. The compression operation can summarize information through pooling, and the excitation operation uses the feature information summarized by the compression process to recalibrate the network weights.

[0171] As can be seen from the above embodiments, the speech enhancement method provided in the embodiments of the present application estimates the noise mask (masking value) of the time-frequency domain feature data (complex spectrum) of the noisy speech through a noise mask prediction model based on complex value calculation. The model uses complex operations to utilize prior knowledge to better model the correlation between the amplitude spectrum and the phase spectrum, avoiding the real part and the imaginary part being treated as two unrelated parts and operated separately like a real-valued network. Then, based on the masking value of the complex spectrum and the time-frequency domain feature data of the noisy speech, enhanced speech data of the noisy speech data is generated, so that the amplitude spectrum and the phase spectrum can be enhanced at the same time; therefore, the listening quality of the enhanced speech can be effectively improved.

[0172] Second embodiment

[0173] In the above-mentioned embodiments, a speech enhancement method is provided. Correspondingly, this application also provides a speech enhancement device. This device corresponds to the above-mentioned method embodiment. Since the device embodiment is substantially similar to the method embodiment, the description is relatively simple. For relevant details, please refer to the description of the method embodiment. The device embodiment described below is merely illustrative.

[0174] The present application further provides a speech enhancement device, comprising:

[0175] A data conversion unit, configured to determine complex time-frequency domain feature data of the noisy speech data, including an amplitude spectrum and a phase spectrum;

[0176] a noise mask prediction unit, configured to determine a noise mask of the time-frequency domain feature data by using a noise mask prediction model based on complex value calculation;

[0177] The enhanced speech generating unit is configured to generate enhanced speech data of the noisy speech data according to the noise mask and the time-frequency domain feature data.

[0178] Third embodiment

[0179] In the above-mentioned embodiments, a speech enhancement method is provided. Accordingly, this application also provides an electronic device. This device corresponds to the above-mentioned method embodiment. Since the device embodiment is substantially similar to the method embodiment, the description is relatively simple. For relevant details, please refer to the description of the method embodiment. The device embodiment described below is merely illustrative.

[0180] An electronic device of this embodiment includes: a processor and a memory; the memory is used to store a program for implementing a speech enhancement method. After the device is powered on and the program of the method is run by the processor, the device performs the following steps: determining complex time-frequency domain feature data of noisy speech data including an amplitude spectrum and a phase spectrum; determining a noise mask for the time-frequency domain feature data using a noise mask prediction model based on complex value calculation; and generating enhanced speech data for the noisy speech data based on the noise mask and the time-frequency domain feature data.

[0181] Fourth embodiment

[0182] Corresponding to the above-mentioned speech enhancement method, the present application also provides a speech enhancement model construction method, the execution subject of which includes but is not limited to: a server. The parts of this embodiment that are the same as the first embodiment are not repeated here, please refer to the corresponding parts in Example 1. The speech enhancement model construction method provided in the present application includes:

[0183] Step 1: Determine the training dataset.

[0184] The training data includes: complex time-frequency domain feature data of noisy speech data including an amplitude spectrum and a phase spectrum, and a noise mask of the time-frequency domain feature data.

[0185] The complex time-frequency domain feature data can be obtained by performing a short-time Fourier transform on the noisy speech data. The noise mask can be determined according to the ratio of the clean speech data to the noisy speech data.

[0186] Step 2: Construct the network structure of the noise mask prediction model based on the complex-valued network.

[0187] The noise mask prediction model can be based on a Deep Complex U-net (DCUnet). The DCUnet-based noise mask prediction model includes an encoder and a decoder. The encoder can use a downsampling mechanism to extract high-order features, and the decoder can use upsampling to reconstruct the target map.

[0188] In this embodiment, the encoder is used to determine first depth feature data based on time-frequency domain feature data; and the decoder is used to determine the noise mask based on the first depth feature data.

[0189] In practice, a long short-term memory (LSTM) layer can be added between the encoder and decoder to model temporal dependencies. This means that the encoding and decoding layers can employ LSTM neural networks. This approach combines the decoding and encoding networks with the LSTM network to implement a deep, complex-valued convolutional recurrent neural network. This allows for more accurate noise mask predictions based on contextual information (such as complex time-frequency feature data from adjacent frames).

[0190] In one example, the loss function of the noise mask prediction model includes a joint time-frequency domain loss function, that is, combining the time-domain loss function and the time-frequency domain loss function to perform model optimization. This processing approach allows the noise mask prediction model to be trained based on the hybrid loss function to jointly optimize the complex-valued model in the time-frequency domain and the time domain. This can further enhance the correlation between the amplitude spectrum and the phase spectrum, thereby constructing a more accurate noise mask prediction model.

[0191] In specific implementation, the time domain loss function that can be optimized includes but is not limited to: Scale-Invariant Signal to Noise Ratio (SI-SNR), and the time-frequency domain loss function that can be optimized includes but is not limited to: mean square error of complex-valued masking estimation (i.e., noise mask).

[0192] The inventors of the present application have discovered through experiments that determining the masking value of the complex spectrum by a noise mask prediction model based on a deep complex-valued convolutional network depends largely on the expression ability of the complex-valued convolutional network. In order to improve the expression ability of the complex-valued convolutional network, the encoder includes multiple encoding layers, and the decoder includes multiple decoding layers. In addition, the noise mask prediction model may also include a first attention module corresponding to the encoding layer, and the first attention module is used to determine the third depth feature data of at least one depth level based on the second depth feature data output by the encoding layer. Correspondingly, the decoder is used to determine the noise mask based on the first depth feature data and the third depth feature data. By adopting this model structure, more informative features are constructed through the first attention module to improve the expression ability of the complex-valued convolutional network, thereby effectively improving the accuracy of the model prediction of the noise mask.

[0193] Regarding the network structure of the model, please refer to the network structure description of the noise mask prediction model in Example 1, which will not be repeated here.

[0194] Step 3: Based on the training data set, train the network parameters of the noise mask prediction model.

[0195] This step is to use the complex time-frequency domain feature data in the training data as the input data of the model, and the noise mask (labeled value) in the training data as the output data of the model. The model outputs the estimated value of the noise mask of the time-frequency domain feature data, and calculates the loss value of the model prediction based on the difference between the estimated value of the noise mask and the labeled value of the noise mask. When the loss value reaches the model optimization target, the model training is stopped, the network parameters of the noise mask prediction model are stored, and the noise mask is predicted based on the network parameters in the online prediction stage.

[0196] As can be seen from the above embodiments, the noise mask prediction model processing method provided in the embodiments of the present application constructs a noise mask prediction model based on complex value calculation. The model directly estimates the masking value of the complex spectrum of the noisy speech. Through complex operations, prior knowledge can be used to better model the correlation between the amplitude spectrum and the phase spectrum, avoiding the real part and the imaginary part being treated as two unrelated parts and operated separately like a real-valued network. Then, based on the masking value of the complex spectrum and the complex spectrum of the noisy speech, enhanced speech data of the noisy speech data is generated, so that the amplitude spectrum and the phase spectrum can be enhanced at the same time; therefore, the listening quality of the enhanced speech can be effectively improved.

[0197] Fifth embodiment

[0198] In the above-mentioned embodiments, a noise mask prediction model processing method is provided. Correspondingly, this application also provides a noise mask prediction model processing device. This device corresponds to the above-mentioned method embodiment. Since the device embodiment is substantially similar to the method embodiment, the description is relatively simple. For relevant details, please refer to the description of the method embodiment. The device embodiment described below is merely illustrative.

[0199] The present application further provides a noise mask prediction model processing device, comprising:

[0200] A training data determination unit is used to determine a training data set, wherein the training data includes: complex time-frequency domain feature data including an amplitude spectrum and a phase spectrum of the noisy speech data, and a noise mask of the time-frequency domain feature data;

[0201] A network construction unit, used to construct a network structure of a noise mask prediction model based on a complex-valued network;

[0202] A network training unit is used to train the network parameters of the noise mask prediction model according to the training data set.

[0203] Sixth embodiment

[0204] In the above-mentioned embodiments, a noise mask prediction model processing method is provided. Correspondingly, this application also provides an electronic device. This device corresponds to the above-mentioned method embodiment. Since the device embodiment is substantially similar to the method embodiment, the description is relatively simple. For relevant details, please refer to the description of the method embodiment. The device embodiment described below is merely illustrative.

[0205] An electronic device of this embodiment includes: a processor and a memory; the memory is used to store a program for implementing a noise mask prediction model processing method. After the device is powered on and the program of the method is run through the processor, the following steps are performed: determining a training data set, the training data including: complex time-frequency domain feature data of noisy speech data including an amplitude spectrum and a phase spectrum, and a noise mask of the time-frequency domain feature data; constructing a network structure of a noise mask prediction model based on a complex-valued network; and training the network parameters of the noise mask prediction model based on the training data set.

[0206] Seventh embodiment

[0207] Corresponding to the above-mentioned speech enhancement method, the present application also provides a video conferencing system. The parts of this embodiment that are the same as those in the first embodiment are not repeated here, and please refer to the corresponding parts in the first embodiment. The video conferencing system provided by the present application includes: a first client, a server, and a second client.

[0208] Among them, the first client is used to collect video conference voice data and send the voice data to the server; the server is used to determine the complex time-frequency domain feature data of the voice data including the amplitude spectrum and the phase spectrum; determine the noise mask of the time-frequency domain feature data through a noise mask prediction model based on complex value calculation; generate enhanced voice data of the voice data based on the noise mask and the time-frequency domain feature data; send the enhanced voice data to the second client; the second client is used to play the enhanced voice data.

[0209] The client includes but is not limited to mobile communication devices, that is, commonly known as mobile phones or smart phones, as well as terminal devices such as personal computers, PADs, and iPads.

[0210] In this embodiment, multiple users conduct an online video conference through the DingTalk video conferencing system. The video conference voice data of the first client user can be collected through the first client of the video conference (such as the client of user A), and the voice data is sent to the video conferencing server; the video conferencing server first performs voice enhancement processing on the voice data to obtain enhanced voice data of the voice data; and then sends the enhanced voice data to the second client of the video conference (such as the clients of user B and user X), and the second client plays the enhanced voice data so that the second client user can hear the voice of the first client user after noise is removed.

[0211] It can be seen from the above embodiments that the video conferencing system provided by the embodiments of the present application estimates the masking value of the complex spectrum of the video conferencing speech through a noise mask prediction model based on complex value calculation. The model uses complex operations to utilize prior knowledge to better model the correlation between the amplitude spectrum and the phase spectrum, avoiding the real-valued network treating the real part and the imaginary part as two unrelated parts for separate calculations. Then, based on the masking value of the complex spectrum and the complex spectrum of the noisy speech, enhanced speech data of the noisy speech data is generated, so that the amplitude spectrum and the phase spectrum can be enhanced at the same time; therefore, noise and howling can be effectively suppressed, thereby improving the enhanced video conferencing speech quality.

[0212] Eighth embodiment

[0213] In the above-mentioned embodiments, a video conferencing system is provided. Correspondingly, this application also provides a method for enhancing speech during video conferencing. This method corresponds to the above-mentioned system embodiments. Since the method embodiments are generally similar to the system embodiments, their description is relatively brief. For relevant details, please refer to the description of the system embodiments. The method embodiments described below are merely illustrative.

[0214] This application also provides a video conferencing voice enhancement method, including:

[0215] Step 1: Collect video conference voice data.

[0216] In one example, the method is performed by a video conferencing server, which collects video conferencing voice data through a first client. For example, after collecting the video conferencing voice data, the first client sends the voice data to the server.

[0217] Step 2: Determine the complex time-frequency domain feature data of the speech data including the amplitude spectrum and the phase spectrum.

[0218] Step 3: Determine the noise mask of the time-frequency domain feature data through a noise mask prediction model based on a complex-valued computing network.

[0219] Step 4: Generate enhanced speech data of the video conference speech data based on the noise mask and the time-frequency domain feature data.

[0220] In one example, the method is performed by a video conferencing server. After the server generates enhanced voice data of the video conferencing voice data through the above steps, the server sends the enhanced voice data to the video conferencing client.

[0221] Ninth embodiment

[0222] Corresponding to the above-mentioned voice enhancement method, the present application also provides a telephone conference system. The parts of this embodiment that are the same as the first embodiment are not repeated here, please refer to the corresponding parts in the first embodiment. The video conferencing system provided by the present application includes: a first client, a server, and a second client.

[0223] Among them, the first client is used to collect telephone conference voice data and send the voice data to the server; the server is used to determine the complex time-frequency domain feature data of the voice data including the amplitude spectrum and the phase spectrum; determine the noise mask of the time-frequency domain feature data through a noise mask prediction model based on complex value calculation; generate enhanced voice data of the voice data based on the noise mask and the time-frequency domain feature data; send the enhanced voice data to the second client; the second client is used to play the enhanced voice data.

[0224] The client includes but is not limited to mobile communication devices, that is, commonly known as mobile phones or smart phones, as well as terminal devices such as personal computers, PADs, and iPads.

[0225] In this embodiment, multiple users conduct a telephone conference through a telephone conference system. The telephone conference voice data of the first client user can be collected through the first client of the telephone conference (such as the client of user A), and the voice data is sent to the telephone conference server; the telephone conference server first performs voice enhancement processing on the voice data to obtain enhanced voice data of the voice data; and then sends the enhanced voice data to the second client of the telephone conference (such as the client of user B and user X), and the second client plays the enhanced voice data so that the second client user can hear the voice of the first client user after noise is removed.

[0226] It can be seen from the above embodiments that the telephone conferencing system provided by the embodiments of the present application estimates the masking value of the complex spectrum of the telephone conference speech through a noise mask prediction model based on complex value calculation. The model uses complex operations to utilize prior knowledge to better model the correlation between the amplitude spectrum and the phase spectrum, avoiding the real-valued network treating the real part and the imaginary part as two unrelated parts for separate calculations. Then, based on the masking value of the complex spectrum and the complex spectrum of the noisy speech, enhanced speech data of the noisy speech data is generated, so that the amplitude spectrum and the phase spectrum can be enhanced at the same time; therefore, noise and howling can be effectively suppressed, thereby improving the enhanced video conferencing voice quality.

[0227] Tenth embodiment

[0228] In the above-mentioned embodiments, a telephone conferencing system is provided. Correspondingly, this application also provides a telephone conferencing voice enhancement method. This method corresponds to the above-mentioned system embodiment. Since the method embodiment is substantially similar to the system embodiment, the description is relatively simple. For relevant details, please refer to the description of the system embodiment. The method embodiment described below is merely illustrative.

[0229] The present application further provides a method for enhancing voice in a telephone conference, comprising:

[0230] Step 1: Collect conference call audio data.

[0231] In one example, the method is performed by a conference call server, which collects conference call voice data through a first client. For example, after collecting the conference call voice data, the first client sends the voice data to the server.

[0232] Step 2: Determine the complex time-frequency domain feature data of the speech data including the amplitude spectrum and the phase spectrum.

[0233] Step 3: Determine the noise mask of the time-frequency domain feature data through a noise mask prediction model based on a complex-valued computing network.

[0234] Step 4: Generate enhanced speech data of the video conference speech data based on the noise mask and the time-frequency domain feature data.

[0235] In one example, the method is performed by a teleconference server. After the server generates enhanced voice data of the teleconference voice data through the above steps, it sends the enhanced voice data to the teleconference client.

[0236] Eleventh embodiment

[0237] Corresponding to the above-mentioned speech enhancement method, the present application also provides an online education system. The parts of this embodiment that are the same as those in the first embodiment are not repeated here, please refer to the corresponding parts in the first embodiment. The online education system provided by the present application includes: a first client, a server, and a second client.

[0238] Among them, the first client is used to collect online education voice data and send the voice data to the server; the server is used to determine the complex time-frequency domain feature data of the voice data including the amplitude spectrum and the phase spectrum; determine the noise mask of the time-frequency domain feature data through a noise mask prediction model based on complex value calculation; generate enhanced voice data of the voice data based on the noise mask and the time-frequency domain feature data; send the enhanced voice data to the second client; the second client is used to play the enhanced voice data.

[0239] The client includes but is not limited to mobile communication devices, that is, commonly known as mobile phones or smart phones, as well as terminal devices such as personal computers, PADs, and iPads.

[0240] In this embodiment, a first client user conducts an online education activity through an online education system. The first client can collect the online education voice data of the first client user and send the voice data to the online education server. The online education server first performs voice enhancement processing on the voice data to obtain enhanced voice data of the voice data. The enhanced voice data is then sent to the second client of the online education (such as the client of user B and user X). The second client plays the enhanced voice data so that the second client user can hear the online education voice of the first client user after noise is removed.

[0241] It can be seen from the above embodiments that the online education system provided by the embodiments of the present application estimates the masking value of the complex spectrum of the online education speech through a noise mask prediction model based on complex value calculation. The model uses complex operations to utilize prior knowledge to better model the correlation between the amplitude spectrum and the phase spectrum, avoiding the real-valued network treating the real part and the imaginary part as two unrelated parts for separate calculations. Then, based on the masking value of the complex spectrum and the complex spectrum of the noisy speech, enhanced speech data of the noisy speech data is generated, so that the amplitude spectrum and the phase spectrum can be enhanced at the same time; therefore, noise and howling can be effectively suppressed, thereby improving the enhanced video conference voice quality.

[0242] Twelfth embodiment

[0243] In the above-mentioned embodiments, an online education system is provided. Correspondingly, this application also provides a method for online education speech enhancement. This method corresponds to the above-mentioned system embodiment. Since the method embodiment is substantially similar to the system embodiment, the description is relatively simple. For relevant details, please refer to the partial description of the system embodiment. The method embodiment described below is merely illustrative.

[0244] This application also provides an online education speech enhancement method, including:

[0245] Step 1: Collect online education voice data.

[0246] In one example, the execution subject of the method is an online education server, which collects online education voice data through a first client. For example, after the first client collects the online education voice data, it sends the voice data to the server.

[0247] Step 2: Determine the complex time-frequency domain feature data of the speech data including the amplitude spectrum and the phase spectrum.

[0248] Step 3: Determine the noise mask of the time-frequency domain feature data through a noise mask prediction model based on a complex-valued computing network.

[0249] Step 4: Generate enhanced speech data of the video conference speech data based on the noise mask and the time-frequency domain feature data.

[0250] In one example, the execution subject of the method is an online education server. After the server generates enhanced voice data of the telephone conference voice data through the above steps, the enhanced voice data is sent to the second client.

[0251] Thirteenth embodiment

[0252] Corresponding to the above-mentioned voice enhancement method, the present application also provides a live broadcast system. The parts of this embodiment that are the same as the first embodiment are not repeated here, please refer to the corresponding parts in embodiment 1. The online education system provided by the present application includes: a first client, a server, and a second client.

[0253] Among them, the first client is used to collect live voice data and send the voice data to the server; the server is used to determine the complex time-frequency domain feature data of the voice data including the amplitude spectrum and the phase spectrum; determine the noise mask of the time-frequency domain feature data through a noise mask prediction model based on complex value calculation; generate enhanced voice data of the voice data according to the noise mask and the time-frequency domain feature data; send the enhanced voice data to the second client; the second client is used to play the enhanced voice data.

[0254] The client includes but is not limited to mobile communication devices, that is, commonly known as mobile phones or smart phones, as well as terminal devices such as personal computers, PADs, and iPads.

[0255] In this embodiment, the anchor conducts live streaming and selling goods in the online live broadcast room. The anchor user's voice data can be collected through the first client and the voice data is sent to the live broadcast server. The live broadcast server first performs voice enhancement processing on the voice data to obtain enhanced voice data of the voice data; and then sends the enhanced voice data to the second client watching the live video. The second client plays the enhanced voice data, so that the users watching the live video can hear the anchor's voice after noise is removed.

[0256] It can be seen from the above embodiments that the live broadcast system provided by the embodiments of the present application estimates the masking value of the complex spectrum of the live broadcast voice through a noise mask prediction model based on complex value calculation. The model uses complex operations to utilize prior knowledge to better model the correlation between the amplitude spectrum and the phase spectrum, avoiding the real part and the imaginary part being treated as two unrelated parts and operated separately like a real-valued network. Then, based on the masking value of the complex spectrum and the complex spectrum of the noisy speech, enhanced speech data of the noisy speech data is generated, so that the amplitude spectrum and the phase spectrum can be enhanced at the same time; therefore, noise and howling can be effectively suppressed, thereby improving the enhanced video conference voice quality.

[0257] Fourteenth embodiment

[0258] In the above-mentioned embodiments, a live broadcast system is provided. Correspondingly, this application also provides a live broadcast voice enhancement method. This method corresponds to the above-mentioned system embodiment. Since the method embodiment is substantially similar to the system embodiment, the description is relatively simple. For relevant details, please refer to the description of the system embodiment. The method embodiment described below is merely illustrative.

[0259] This application also provides an online education speech enhancement method, including:

[0260] Step 1: Collect live audio data.

[0261] In one example, the execution subject of the method is a live broadcast server, which collects live broadcast voice data of the anchor user through a first client. After the first client collects the live broadcast voice data, it sends the voice data to the server.

[0262] Step 2: Determine the complex time-frequency domain feature data of the speech data including the amplitude spectrum and the phase spectrum.

[0263] Step 3: Determine the noise mask of the time-frequency domain feature data through a noise mask prediction model based on a complex-valued computing network.

[0264] Step 4: Generate enhanced speech data of the video conference speech data based on the noise mask and the time-frequency domain feature data.

[0265] In one example, the execution subject of the method is a live broadcast server. After the server generates enhanced voice data of the live broadcast voice data through the above steps, the enhanced voice data is sent to the second client.

[0266] Fifteenth embodiment

[0267] Corresponding to the above-mentioned speech enhancement method, the present application also provides a speech recognition system. The parts of this embodiment that are the same as those in the first embodiment are not repeated here, and please refer to the corresponding parts in the first embodiment. The speech recognition system provided by the present application includes: a client and a server.

[0268] Among them, the client is used to collect voice data and send the voice data to the server; the server is used to determine the complex time-frequency domain feature data of the voice data including the amplitude spectrum and the phase spectrum; determine the noise mask of the time-frequency domain feature data through a noise mask prediction model based on a complex value computing network; generate enhanced voice data of the voice data based on the noise mask and the time-frequency domain feature data; and convert the enhanced voice data into text through a voice recognition model.

[0269] Since the speech recognition model is a relatively mature existing technology, it will not be described in detail here.

[0270] As can be seen from the above embodiments, the speech recognition system provided by the embodiments of the present application estimates the masking value of the complex spectrum of the noisy speech through a speech processing model based on a complex-valued network. The model uses complex operations to utilize prior knowledge to better model the correlation between the amplitude spectrum and the phase spectrum, avoiding the real part and the imaginary part being treated as two unrelated parts and operated separately like a real-valued network. Then, based on the masking value of the complex spectrum and the complex spectrum of the noisy speech, enhanced speech data of the noisy speech data is generated. In this way, the amplitude spectrum and the phase spectrum can be enhanced at the same time, and then the enhanced speech data is converted into text through the speech recognition model; therefore, noise and howling can be effectively suppressed, thereby improving speech quality and further improving speech recognition accuracy.

[0271] Sixteenth embodiment

[0272] In the above-mentioned embodiments, a speech recognition system is provided. Correspondingly, this application also provides a speech recognition method. This method corresponds to the above-mentioned system embodiment. Since the method embodiment is substantially similar to the system embodiment, the description is relatively simple. For relevant details, please refer to the description of the system embodiment. The method embodiment described below is merely illustrative.

[0273] The present application further provides a speech recognition method, comprising:

[0274] Step 1: Collect voice data.

[0275] In one example, the method is performed by a speech recognition server, which collects user speech data through a client. After the client collects the speech data, it sends the speech data to the server.

[0276] Step 2: Determine the complex time-frequency domain feature data of the speech data including the amplitude spectrum and the phase spectrum.

[0277] Step 3: Determine the noise mask of the time-frequency domain feature data through a noise mask prediction model based on a complex-valued computing network.

[0278] Step 4: Generate enhanced speech data of the video conference speech data based on the noise mask and the time-frequency domain feature data.

[0279] Step 5: Convert the enhanced speech data into text through the speech recognition model.

[0280] Since speech recognition is a relatively mature existing technology, it will not be described in detail here.

[0281] In one example, the method is performed by a speech recognition server, which sends the speech transcription text to a client for the client user to view.

[0282] Seventeenth embodiment

[0283] Corresponding to the above-mentioned speech enhancement method, the present application also provides a speech recognition text editing system. The parts of this embodiment that are the same as the first embodiment are not repeated here, please refer to the corresponding parts in Example 1. The speech recognition text editing system provided by the present application includes: a client and a server.

[0284] Among them, the client is used to collect voice data and send the voice data to the server; and, edit the text of the voice data recognized by the server; the server is used to determine the complex time-frequency domain feature data of the voice data including the amplitude spectrum and the phase spectrum; determine the noise mask of the time-frequency domain feature data through a noise mask prediction model based on a complex value computing network; generate enhanced voice data of the voice data based on the noise mask and the time-frequency domain feature data; and convert the enhanced voice data into text through a voice recognition model.

[0285] Since speech recognition and editing of speech-transcribed text are relatively mature existing technologies, they will not be described in detail here.

[0286] As can be seen from the above embodiments, the speech recognition text editing system provided by the embodiments of the present application estimates the masking value of the complex spectrum of the noisy speech through a speech processing model based on a complex-valued network. The model uses complex operations to utilize prior knowledge to better model the correlation between the amplitude spectrum and the phase spectrum, avoiding the real part and the imaginary part being treated as two unrelated parts and operated separately like a real-valued network. Then, based on the masking value of the complex spectrum and the complex spectrum of the noisy speech, enhanced speech data of the noisy speech data is generated. In this way, the amplitude spectrum and the phase spectrum can be enhanced at the same time, and then the enhanced speech data is converted into text through the speech recognition model, and then the transcribed text is edited; therefore, noise and howling can be effectively suppressed, thereby improving speech quality, thereby improving speech recognition accuracy, and thereby improving speech recognition text editing efficiency.

[0287] Eighteenth embodiment

[0288] Corresponding to the above-mentioned speech enhancement method, the present application also provides a user identification system. The parts of this embodiment that are the same as those in the first embodiment are not repeated here, please refer to the corresponding parts in the first embodiment. The speech recognition text editing system provided by the present application includes: a client and a server.

[0289] Among them, the client is used to collect voice data and send the voice data to the server; the server is used to determine the complex time-frequency domain feature data of the voice data including the amplitude spectrum and the phase spectrum; determine the noise mask of the time-frequency domain feature data through a noise mask prediction model based on a complex value computing network; generate enhanced voice data of the voice data based on the noise mask and the time-frequency domain feature data; and determine the user information of the enhanced voice data through a user recognition model.

[0290] Since user identification is a relatively mature existing technology, it will not be described here in detail.

[0291] As can be seen from the above embodiments, the user identification system provided by the embodiments of the present application estimates the masking value of the complex spectrum of noisy speech through a speech processing model based on a complex-valued network. The model uses complex operations to utilize prior knowledge to better model the correlation between the amplitude spectrum and the phase spectrum, avoiding the real part and the imaginary part being treated as two unrelated parts and operated separately like a real-valued network. Then, based on the masking value of the complex spectrum and the complex spectrum of the noisy speech, enhanced speech data of the noisy speech data is generated. In this way, the amplitude spectrum and the phase spectrum can be enhanced at the same time, and then the user information of the enhanced speech data is determined through the user identification model; therefore, noise and howling can be effectively suppressed, thereby improving speech quality and further improving user identification accuracy.

[0292] Although the present application is disclosed as above with the preferred embodiments, it is not intended to limit the present application. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0293] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0294] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0295] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0296] 2. Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A speech enhancement method, characterized in that: include: Determining complex time-frequency domain feature data of the noisy speech data including an amplitude spectrum and a phase spectrum; Determining, by an encoder in a noise mask prediction model based on complex value calculation, first depth feature data based on time-frequency domain feature data, wherein the encoder includes a plurality of encoding layers; determining, by a first attention module in the noise mask prediction model corresponding to the encoding layer, third depth feature data of at least one depth level based on second depth feature data output by the encoding layer; wherein the third depth feature data is residual input data of a decoding layer corresponding to the encoding layer, wherein the decoder includes a plurality of decoding layers; Determining, by a decoder in a noise mask prediction model, a noise mask for the time-frequency domain feature data based on the first depth feature data and the third depth feature data; Enhanced speech data of the noisy speech data is generated according to the noise mask and the time-frequency domain feature data.

2. The method according to claim 1, characterized in that The step of determining the noise mask according to the first depth feature data and the third depth feature data by a decoder in the noise mask prediction model includes: Determining, by a second attention module before the decoding layer, second decoded data based on first decoded data output by a decoding layer preceding the decoding layer; The first decoded data output by the decoding layer is determined through the decoding layer according to the second decoded data and the third depth feature data corresponding to the encoding layer corresponding to the decoding layer.

3. The method according to claim 1, characterized in that The first attention module includes: a complex-valued channel attention module and a complex-valued spatial attention module; The third depth feature data is determined by the following steps: Determining, by the complex-valued channel attention module, fourth depth feature data of a plurality of complex-valued channels based on the second depth feature data output by the encoding layer; the fourth depth feature data includes depth feature information of different channels requiring special attention extracted from the second depth feature data; The third depth feature data is determined based on the fourth depth feature data of the multiple channels through the complex-valued spatial attention module.

4. The method according to claim 1, characterized in that The loss function of the noise mask prediction model includes: a time-frequency domain joint loss function to enhance the correlation between the amplitude spectrum and the phase spectrum.

5. The method according to claim 1, characterized in that The step of generating enhanced speech data of noisy speech data according to the noise mask and the time-frequency domain feature data comprises: Using the calculation results of the time-frequency domain feature data and the noise mask as time-frequency domain enhanced feature data; The enhanced speech data is generated according to the time-frequency domain enhanced feature data.

6. The method according to claim 5, characterized in that The time-frequency domain feature data includes: short-time Fourier transform (STFT) feature data; Generating the enhanced speech data according to the time-frequency domain enhanced feature data includes: The inverse short-time Fourier transform data of the time-frequency domain enhanced feature data is used as the enhanced speech data.

7. A noise mask prediction model processing method, characterized in that: include: Determine a training data set, the training data comprising: complex time-frequency domain feature data of noisy speech data including an amplitude spectrum and a phase spectrum, and a noise mask of the time-frequency domain feature data; Constructing a network structure of a noise mask prediction model based on a complex-valued network; the prediction model includes an encoder and a decoder, the encoder includes multiple encoding layers and a first attention module corresponding to the encoding layer; the encoder is used to determine first depth feature data based on the time-frequency domain feature data; the first attention module is used to determine third depth feature data of at least one depth level based on second depth feature data output by the corresponding encoding layer; wherein the third depth feature data is residual input data of a decoding layer corresponding to the encoding layer, and the decoder includes multiple decoding layers; the decoder is used to determine a noise mask for the time-frequency domain feature data based on the first depth feature data and the third depth feature data; The network parameters of the noise mask prediction model are trained according to the training data set.

8. A video conferencing system, characterized in that: include: The first client is used to collect the video conference voice data and send the voice data to the server; The server is used to determine the complex time-frequency domain feature data of the speech data including the amplitude spectrum and the phase spectrum; determine the first depth feature data according to the time-frequency domain feature data through an encoder in a noise mask prediction model based on complex value calculation, and the encoder includes multiple encoding layers; determine the third depth feature data of at least one depth level according to the second depth feature data output by the encoding layer through a first attention module corresponding to the encoding layer in the noise mask prediction model; wherein the third depth feature data is the residual input data of the decoding layer corresponding to the encoding layer, and the decoder includes multiple decoding layers; determine the noise mask of the time-frequency domain feature data according to the first depth feature data and the third depth feature data through the decoder in the noise mask prediction model; generate enhanced speech data of the speech data according to the noise mask and the time-frequency domain feature data; and send the enhanced speech data to the second client; The second client is used to play the enhanced voice data.

9. A method for enhancing speech in a video conference, characterized in that: include: Collect video conference voice data; Determining complex time-frequency domain feature data of the speech data including an amplitude spectrum and a phase spectrum; Determining, by an encoder in a noise mask prediction model based on a complex-valued computing network, first depth feature data based on time-frequency domain feature data, the encoder comprising a plurality of encoding layers; determining, by a first attention module in the noise mask prediction model corresponding to the encoding layer, third depth feature data of at least one depth level based on second depth feature data output by the encoding layer; wherein the third depth feature data is residual input data of a decoding layer corresponding to the encoding layer, the decoder comprising a plurality of decoding layers; determining, by a decoder in the noise mask prediction model, a noise mask for the time-frequency domain feature data based on the first depth feature data and the third depth feature data; Enhanced voice data of the video conference voice data is generated according to the noise mask and the time-frequency domain feature data.

10. A telephone conference system, characterized in that: include: A first client, configured to collect conference call voice data and send the voice data to a server; The server is used to determine the complex time-frequency domain feature data of the speech data including the amplitude spectrum and the phase spectrum; determine the first depth feature data according to the time-frequency domain feature data through an encoder in a noise mask prediction model based on a complex-valued computing network, and the encoder includes multiple encoding layers; determine the third depth feature data of at least one depth level according to the second depth feature data output by the encoding layer through a first attention module corresponding to the encoding layer in the noise mask prediction model; wherein the third depth feature data is the residual input data of the decoding layer corresponding to the encoding layer, and the decoder includes multiple decoding layers; determine the noise mask of the time-frequency domain feature data according to the first depth feature data and the third depth feature data through the decoder in the noise mask prediction model; generate enhanced speech data of the speech data according to the noise mask and the time-frequency domain feature data; and send the enhanced speech data to the second client; The second client is used to play the enhanced voice data.

11. A method for enhancing voice in a telephone conference, characterized in that: include: Collecting conference call voice data; Determining complex time-frequency domain feature data of the speech data including an amplitude spectrum and a phase spectrum; Determining, by an encoder in a noise mask prediction model based on a complex-valued computing network, first depth feature data based on time-frequency domain feature data, the encoder comprising a plurality of encoding layers; determining, by a first attention module in the noise mask prediction model corresponding to the encoding layer, third depth feature data of at least one depth level based on second depth feature data output by the encoding layer; wherein the third depth feature data is residual input data of a decoding layer corresponding to the encoding layer, the decoder comprising a plurality of decoding layers; determining, by a decoder in the noise mask prediction model, a noise mask for the time-frequency domain feature data based on the first depth feature data and the third depth feature data; Enhanced speech data of the telephone conference speech data is generated according to the noise mask and the time-frequency domain feature data.

12. An online education system, characterized in that: include: The first client is used to collect online education voice data and send the voice data to the server; The server is used to determine the complex time-frequency domain feature data of the speech data including the amplitude spectrum and the phase spectrum; determine the first depth feature data according to the time-frequency domain feature data through an encoder in a noise mask prediction model based on a complex-valued computing network, and the encoder includes multiple encoding layers; determine the third depth feature data of at least one depth level according to the second depth feature data output by the encoding layer through a first attention module corresponding to the encoding layer in the noise mask prediction model; wherein the third depth feature data is the residual input data of the decoding layer corresponding to the encoding layer, and the decoder includes multiple decoding layers; determine the noise mask of the time-frequency domain feature data according to the first depth feature data and the third depth feature data through the decoder in the noise mask prediction model; generate enhanced speech data of the speech data according to the noise mask and the time-frequency domain feature data; and send the enhanced speech data to the second client; The second client is used to play the enhanced voice data.

13. A method for enhancing online education speech, characterized in that: include: Collect online education voice data; Determining complex time-frequency domain feature data of the speech data including an amplitude spectrum and a phase spectrum; Determining, by an encoder in a noise mask prediction model based on a complex-valued computing network, first depth feature data based on time-frequency domain feature data, the encoder comprising a plurality of encoding layers; determining, by a first attention module in the noise mask prediction model corresponding to the encoding layer, third depth feature data of at least one depth level based on second depth feature data output by the encoding layer; wherein the third depth feature data is residual input data of a decoding layer corresponding to the encoding layer, the decoder comprising a plurality of decoding layers; determining, by a decoder in the noise mask prediction model, a noise mask for the time-frequency domain feature data based on the first depth feature data and the third depth feature data; Enhanced speech data of the online education speech data is generated according to the noise mask and the time-frequency domain feature data.

14. A live broadcast system, characterized in that: include: The first client is used to collect live voice data and send the voice data to the server; The server is used to determine the complex time-frequency domain feature data of the speech data including the amplitude spectrum and the phase spectrum; determine the first depth feature data according to the time-frequency domain feature data through an encoder in a noise mask prediction model based on a complex-valued computing network, and the encoder includes multiple encoding layers; determine the third depth feature data of at least one depth level according to the second depth feature data output by the encoding layer through a first attention module corresponding to the encoding layer in the noise mask prediction model; wherein the third depth feature data is the residual input data of the decoding layer corresponding to the encoding layer, and the decoder includes multiple decoding layers; determine the noise mask of the time-frequency domain feature data according to the first depth feature data and the third depth feature data through the decoder in the noise mask prediction model; generate enhanced speech data of the speech data according to the noise mask and the time-frequency domain feature data; and send the enhanced speech data to the second client; The second client is used to play the enhanced voice data.

15. A live speech enhancement method, characterized in that: include: Collect live voice data; Determining complex time-frequency domain feature data of the speech data including an amplitude spectrum and a phase spectrum; Determining, by an encoder in a noise mask prediction model based on a complex-valued computing network, first depth feature data based on time-frequency domain feature data, the encoder comprising a plurality of encoding layers; determining, by a first attention module in the noise mask prediction model corresponding to the encoding layer, third depth feature data of at least one depth level based on second depth feature data output by the encoding layer; wherein the third depth feature data is residual input data of a decoding layer corresponding to the encoding layer, the decoder comprising a plurality of decoding layers; determining, by a decoder in the noise mask prediction model, a noise mask for the time-frequency domain feature data based on the first depth feature data and the third depth feature data; Enhanced speech data of the live speech data is generated according to the noise mask and the time-frequency domain feature data.

16. A speech recognition system, characterized in that: include: The client is used to collect voice data and send the voice data to the server; A server is used to determine the complex time-frequency domain feature data of the speech data including the amplitude spectrum and the phase spectrum; determine first depth feature data according to the time-frequency domain feature data through an encoder in a noise mask prediction model based on a complex-valued computing network, and the encoder includes multiple encoding layers; determine third depth feature data of at least one depth level according to the second depth feature data output by the encoding layer through a first attention module corresponding to the encoding layer in the noise mask prediction model; wherein the third depth feature data is the residual input data of the decoding layer corresponding to the encoding layer, and the decoder includes multiple decoding layers; determine the noise mask of the time-frequency domain feature data according to the first depth feature data and the third depth feature data through the decoder in the noise mask prediction model; generate enhanced speech data of the speech data according to the noise mask and the time-frequency domain feature data; and convert the enhanced speech data into text through a speech recognition model.

17. A speech recognition method, characterized in that: include: Collect voice data; Determining complex time-frequency domain feature data of the speech data including an amplitude spectrum and a phase spectrum; Determining, by an encoder in a noise mask prediction model based on a complex-valued computing network, first depth feature data based on time-frequency domain feature data, the encoder comprising a plurality of encoding layers; determining, by a first attention module in the noise mask prediction model corresponding to the encoding layer, third depth feature data of at least one depth level based on second depth feature data output by the encoding layer; wherein the third depth feature data is residual input data of a decoding layer corresponding to the encoding layer, the decoder comprising a plurality of decoding layers; determining, by a decoder in the noise mask prediction model, a noise mask for the time-frequency domain feature data based on the first depth feature data and the third depth feature data; generating enhanced speech data of the speech data according to the noise mask and the time-frequency domain feature data; The enhanced speech data is converted into text through the speech recognition model.

18. A speech recognition text editing system, characterized in that: include: The client is used to collect voice data and send the voice data to the server; and edit the text of the voice data recognized by the server; A server is used to determine the complex time-frequency domain feature data of the speech data including the amplitude spectrum and the phase spectrum; determine first depth feature data according to the time-frequency domain feature data through an encoder in a noise mask prediction model based on a complex-valued computing network, and the encoder includes multiple encoding layers; determine third depth feature data of at least one depth level according to the second depth feature data output by the encoding layer through a first attention module corresponding to the encoding layer in the noise mask prediction model; wherein the third depth feature data is the residual input data of the decoding layer corresponding to the encoding layer, and the decoder includes multiple decoding layers; determine the noise mask of the time-frequency domain feature data according to the first depth feature data and the third depth feature data through the decoder in the noise mask prediction model; generate enhanced speech data of the speech data according to the noise mask and the time-frequency domain feature data; and convert the enhanced speech data into text through a speech recognition model.

19. A user identification system, characterized in that: include: The client is used to collect voice data and send the voice data to the server; The server is used to determine the complex time-frequency domain feature data of the speech data including the amplitude spectrum and the phase spectrum; determine the first depth feature data according to the time-frequency domain feature data through an encoder in a noise mask prediction model based on a complex-valued computing network, and the encoder includes multiple coding layers; determine the third depth feature data of at least one depth level according to the second depth feature data output by the coding layer through a first attention module corresponding to the coding layer in the noise mask prediction model; wherein the third depth feature data is the residual input data of the decoding layer corresponding to the coding layer, and the decoder includes multiple decoding layers; determine the noise mask of the time-frequency domain feature data according to the first depth feature data and the third depth feature data through the decoder in the noise mask prediction model; generate enhanced speech data of the speech data according to the noise mask and the time-frequency domain feature data; and determine user information of the enhanced speech data through a user recognition model.

20. A user identification method, characterized in that: include: Collect voice data; Determining complex time-frequency domain feature data of the speech data including an amplitude spectrum and a phase spectrum; Determining, by an encoder in a noise mask prediction model based on a complex-valued computing network, first depth feature data based on time-frequency domain feature data, the encoder comprising a plurality of encoding layers; determining, by a first attention module in the noise mask prediction model corresponding to the encoding layer, third depth feature data of at least one depth level based on second depth feature data output by the encoding layer; wherein the third depth feature data is residual input data of a decoding layer corresponding to the encoding layer, the decoder comprising a plurality of decoding layers; determining, by a decoder in the noise mask prediction model, a noise mask for the time-frequency domain feature data based on the first depth feature data and the third depth feature data; generating enhanced speech data of the speech data according to the noise mask and the time-frequency domain feature data; The user information of the enhanced voice data is determined through the user identification model.

21. A speech enhancement device, characterized in that: include: A data conversion unit, configured to determine complex time-frequency domain feature data of the noisy speech data, including an amplitude spectrum and a phase spectrum; A noise mask prediction unit is configured to determine, by an encoder in a noise mask prediction model based on complex value calculation, first depth feature data according to time-frequency domain feature data, wherein the encoder includes multiple encoding layers; determine, by a first attention module corresponding to the encoding layer in the noise mask prediction model, third depth feature data of at least one depth level according to second depth feature data output by the encoding layer; wherein the third depth feature data is residual input data of a decoding layer corresponding to the encoding layer, and the decoder includes multiple decoding layers; determine, by a decoder in the noise mask prediction model, a noise mask for the time-frequency domain feature data according to the first depth feature data and the third depth feature data; The enhanced speech generating unit is configured to generate enhanced speech data of the noisy speech data according to the noise mask and the time-frequency domain feature data.

22. A noise mask prediction model processing device, characterized in that: include: A training data determination unit is used to determine a training data set, wherein the training data includes: complex time-frequency domain feature data including an amplitude spectrum and a phase spectrum of the noisy speech data, and a noise mask of the time-frequency domain feature data; A network construction unit is configured to construct a network structure of a noise mask prediction model based on a complex-valued network; the prediction model includes an encoder and a decoder, the encoder includes multiple encoding layers and a first attention module corresponding to the encoding layer; the encoder is configured to determine first depth feature data based on the time-frequency domain feature data; the first attention module is configured to determine third depth feature data of at least one depth level based on second depth feature data output by the corresponding encoding layer; wherein the third depth feature data is residual input data of a decoding layer corresponding to the encoding layer, and the decoder includes multiple decoding layers; the decoder is configured to determine a noise mask for the time-frequency domain feature data based on the first depth feature data and the third depth feature data; A network training unit is used to train the network parameters of the noise mask prediction model according to the training data set.

23. An electronic device, characterized in that: include: processor and memory; A memory for storing a program for implementing the method according to any one of claims 1 to 7, 9, 11, 13, 15, 17, and 20, wherein the device is powered on and runs the program of the method through the processor.

Images