Speech enhancement model, electronic device, storage medium and related methods

Abstract

Embodiments of the present application provide a speech enhancement model, electronic device, storage medium, and related methods. The speech enhancement method includes: converting noisy speech data into time-frequency domain feature data; generating a masking value for the noisy speech data based on long-range correlations in the frequency direction of the time-frequency domain feature data; and generating enhanced speech data for the noisy speech data based on the masking value and the time-frequency domain feature data. This solution can improve the effectiveness of speech enhancement for speech signals.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of speech processing technology, and in particular to a speech enhancement model, electronic device, storage medium and related methods. Background Art

[0002] Speech enhancement is a speech processing technology that extracts useful speech signals from a noisy background, suppressing or reducing noise interference. Speech enhancement is widely used in real-time communication (RTC) scenarios such as audio and video conferencing and classroom delivery.

[0003] At present, when performing speech enhancement on speech data, the time series features of the speech data along the time direction and the short-term frequency features along the frequency direction are extracted. The speech data is enhanced based on the time series features and the short-term frequency features to suppress the noise in the speech data.

[0004] However, based on time series features and short-term frequency features, it is impossible to effectively identify noise and useful speech signals with large frequency differences in speech data, and thus it is impossible to filter out noise with large frequency differences from useful speech signals, resulting in poor speech enhancement effect on speech signals. Summary of the Invention

[0005] In view of this, embodiments of the present application provide a speech enhancement model, electronic device, storage medium and related methods to at least solve or alleviate the above-mentioned problems.

[0006] According to a first aspect of an embodiment of the present application, a speech enhancement method is provided, comprising: converting noisy speech data into time-frequency domain feature data; generating a masking value for the noisy speech data based on the long-range correlation of the time-frequency domain feature data in the frequency direction; and generating enhanced speech data for the noisy speech data based on the masking value and the time-frequency domain feature data.

[0007] According to a second aspect of an embodiment of the present application, a speech enhancement model is provided, comprising: an encoder, an attention module, a loop module and a decoder; the encoder is used to process time-frequency domain feature data corresponding to noisy speech data; the attention module is used to process the output data of the encoder; the loop module is used to process the output data of the encoder; the decoder is used to process the output data of the attention module and the loop module to obtain correlation feature data for characterizing the long-range correlation of the time-frequency domain feature data in the frequency direction.

[0008] According to a third aspect of an embodiment of the present application, a speech recognition method is provided, comprising: obtaining speech data to be processed, wherein the speech data to be processed includes noise, and the speech data to be processed includes one of the following: audio and video conference voice data, online education voice data, and network live broadcast voice data; converting the speech data to be processed into time-frequency domain feature data; generating a masking value for the noisy speech data based on the long-range correlation of the time-frequency domain feature data in the frequency direction; generating enhanced speech data for the speech data to be processed based on the masking value and the time-frequency domain feature data; performing speech recognition on the enhanced speech data to obtain a recognition result.

[0009] According to the fourth aspect of the embodiments of the present application, an electronic device is provided, including: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the speech enhancement method described in the first aspect above or operations corresponding to the speech recognition method described in the third aspect above.

[0010] According to the fifth aspect of the embodiments of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the speech enhancement method described in the first aspect or the speech recognition method described in the third aspect is implemented.

[0011] According to the sixth aspect of an embodiment of the present application, a computer program product is provided, comprising computer instructions, wherein the computer instructions instruct a computing device to perform operations corresponding to the speech enhancement method described in the first aspect above or operations corresponding to the speech recognition method described in the third aspect above.

[0012] It can be seen from the above technical solution that after the noisy speech data is converted into time-frequency domain feature data, due to the long-range correlation of the time-frequency domain feature data in the frequency direction, the correlation between the noise signal and the useful speech signal with a large frequency difference in the noisy speech data can be characterized. Therefore, based on the long-range correlation of the time-frequency domain feature data in the frequency direction, the noise signal and the useful speech signal with a large frequency difference in the time-frequency domain feature data can be determined, thereby generating a masking value of the noisy speech data, and generating enhanced speech data of the noisy speech data based on the masking value and the time-frequency domain feature data. The noise signal with a large frequency difference in the generated enhanced speech data is filtered out, thereby improving the effect of speech enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0014] Figure 1 is a schematic diagram of an exemplary system used in one embodiment of the present application;

[0015] Figure 2 is a flow chart of a speech enhancement method according to an embodiment of the present application;

[0016] Figure 3 is a schematic diagram of a speech enhancement model according to an embodiment of the present application;

[0017] Figure 4 is a schematic diagram of a speech enhancement model according to another embodiment of the present application;

[0018] Figure 5 is a schematic diagram of a first convolutional loop block according to an embodiment of the present application;

[0019] Figure 6 This is a flow chart of a model training method according to an embodiment of the present application;

[0020] Figure 7 is a schematic diagram of an auxiliary network according to an embodiment of the present application;

[0021] Figure 8 is a flow chart of a speech recognition method according to an embodiment of the present application;

[0022] Figure 9 is a schematic diagram of a speech enhancement device according to an embodiment of the present application;

[0023] Figure 10 It is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0025] In an embodiment of the present application, in order to improve the effect of speech enhancement, noisy speech data is converted into time-frequency domain feature data, and a masking value of the noisy speech data is generated based on the long-range correlation of the time-frequency domain feature data in the frequency direction, and then enhanced speech data of the noisy speech data is generated based on the masking value and the time-frequency domain feature data. The long-range correlation of the time-frequency domain feature data in the frequency direction can characterize the correlation between the noise signal and the useful speech signal with large frequency differences in the speech data. The long-range correlation in the frequency direction can effectively filter out the noise with large frequency differences from the useful speech signal. Therefore, generating a masking value based on the long-range correlation of the time-frequency domain feature data in the frequency direction, and then generating enhanced speech data based on the masking value can more effectively filter out the noise in the noisy speech data, thereby improving the effect of speech enhancement.

[0026] In specific implementation, the voice enhancement method provided by the embodiment of the present application can be used in a variety of application scenarios. For example, a certain cloud service system may provide a voice enhancement service, which can be implemented through the solution provided by the embodiment of the present application. Specifically, the above-mentioned cloud service system provides a voice enhancement model and provides users with a cloud voice enhancement interface. Multiple users can call the interface in their respective application systems. After receiving the call, the cloud service system runs the relevant processing program, implements voice enhancement through the voice enhancement model, and returns enhanced voice data. In addition, the voice enhancement method provided by the embodiment of the present application can also be used in localized devices. For example, the voice enhancement method provided by the embodiment of the present application can be implemented in local devices such as audio and video conferencing terminals and online education terminals.

[0027] Figure 1 An exemplary system for the speech enhancement method applicable to the embodiment of the present application is shown. Figure 1 As shown, the system 100 may include a server 102, a communication network 104 and / or one or more user devices 106. Figure 1 The example in the figure is for multiple user devices.

[0028] Server 102 can be any suitable server for storing information, data, programs, and / or any other suitable type of content. In some embodiments, server 102 can perform any suitable function. For example, in some embodiments, server 102 can be used for speech enhancement. As an alternative example, in some embodiments, server 102 can be used to perform speech enhancement using a speech enhancement model. As another example, in some embodiments, server 102 can be used to send speech enhancement results to a user device.

[0029] In some embodiments, the communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN) and / or any other suitable communication network. The user device 106 can be connected to the communication network 104 by one or more communication links (e.g., communication link 112), and the communication network 104 can be linked to the server 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between the user device 106 and the server 102, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link or any suitable combination of such links.

[0030] User device 106 may include any one or more user devices suitable for receiving and collecting voice data. In some embodiments, user device 106 may include any suitable type of device. For example, in some embodiments, user device 106 may include a mobile device, a tablet computer, a laptop computer, a desktop computer, a wearable computer, a game console, a media player, a vehicle entertainment system, and / or any other suitable type of user device.

[0031] Although server 102 is illustrated as one device, in some embodiments, any suitable number of devices may be used to perform the functions performed by server 102. For example, in some embodiments, multiple devices may be used to implement the functions performed by server 102. Alternatively, the functions of server 102 may be implemented using cloud services.

[0032] Speech Enhancement Methods

[0033] Based on the above system, an embodiment of the present application provides a speech enhancement method, which is described below through multiple embodiments.

[0034] Figure 2 This is a flow chart diagram of a speech enhancement method provided by an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0035] Step 201: Convert noisy speech data into time-frequency domain feature data.

[0036] Noisy speech data can be represented as x = y + z, where x represents the noisy speech data, y represents the clean speech signal, and z represents the noise signal. Noisy speech data describes the time-domain characteristics of the speech signal, specifically, how the amplitude of the speech signal changes over time. To enhance speech with noisy speech data, the time-frequency characteristics of the noisy speech signal are required. These characteristics can represent both the time and frequency domains of the noisy speech signal. To achieve this, the noisy speech data must be converted into time-frequency domain feature data, which indicates the strength of each frequency signal in the speech signal at different times.

[0037] Optionally, the noisy speech data is converted into time-frequency domain feature data through short-time Fourier transform (STFT).

[0038] Step 202: Generate a masking value for the noisy speech data based on the long-range correlation of the time-frequency domain feature data in the frequency direction.

[0039] The long-range correlation of the noisy speech data along the frequency axis can be captured from the time-frequency domain feature data, and a masking value of the noisy speech data is generated based on the captured long-range correlation. The masking value indicates the useful speech signal and the noisy speech signal in the time-frequency domain feature data.

[0040] The time-frequency domain feature data includes the long-range correlation of the noisy speech data along the frequency axis. The time-frequency domain feature data can be input into the speech enhancement model, and the long-range correlation of the noisy speech data along the frequency axis can be captured by the speech enhancement model. The useful speech signal and the noisy speech signal in the time-frequency domain feature data are determined based on the captured long-range correlation, thereby generating a masking value for indicating the useful speech signal and the noisy speech signal in the time-frequency domain feature data.

[0041] Step 203: Generate enhanced speech data of the noisy speech data according to the masking value and the time-frequency domain feature data.

[0042] Since the masking value of the time-frequency domain feature data can indicate the useful speech signal and the noisy speech signal in the time-frequency domain feature data, the noisy speech signal in the time-frequency domain feature data can be filtered out according to the masking value to obtain enhanced speech data mainly including the useful speech signal in the time-frequency domain feature data.

[0043] To facilitate subsequent speech recognition, speech playback, and other processing, the enhanced speech data in the time-frequency domain needs to be converted to the time domain. In an optional implementation, the enhanced speech data can be converted to the time domain by using an inverse short-time Fourier transform (ISTFT).

[0044] In an embodiment of the present application, after the noisy speech data is converted into time-frequency domain feature data, due to the long-range correlation of the time-frequency domain feature data in the frequency direction, the correlation between the noise signal and the useful speech signal with a large frequency difference in the noisy speech data can be characterized. Therefore, based on the long-range correlation of the time-frequency domain feature data in the frequency direction, the noise signal and the useful speech signal with a large frequency difference in the time-frequency domain feature data can be determined, thereby generating a masking value of the noisy speech data, and generating enhanced speech data of the noisy speech data based on the masking value and the time-frequency domain feature data. The noise signal with a large frequency difference in the generated enhanced speech data is filtered out, thereby improving the effect of speech enhancement.

[0045] In one possible implementation, the time-frequency domain feature data can be input into a speech enhancement model, and the speech enhancement model can be used to identify the noise signal and useful speech signal in the time-frequency domain feature data based on the long-range correlation of the time-frequency domain feature data in the frequency direction, and obtain correlation feature data that can indicate the noise signal and useful speech signal in the time-frequency domain feature data. The correlation feature data is then processed through an activation function to obtain a masking value.

[0046] When converting noisy speech data into time-frequency domain feature data, the noisy speech data can be converted into complex time-frequency domain feature data. The corresponding speech enhancement model is also based on a complex-valued neural network model. For example, the complex time-frequency domain feature data can be expressed as Among them, V is used to represent the time-frequency domain feature data, V r Used to characterize the real part of the time-frequency domain feature data, V i Used to characterize the imaginary part of the time-frequency domain feature data, j is the imaginary unit, j 2 =-1, It is used to represent the complex domain, C is used to represent the channel dimension of the time-frequency domain feature data, T is used to represent the frame dimension of the time-frequency domain feature data, and F is used to represent the frequency dimension of the time-frequency domain feature data.

[0047] Since the time-frequency domain feature data is in complex form, and the speech enhancement model used to process the time-frequency domain feature data is also a neural network model based on complex values, the correlation feature data output by the speech enhancement model is also in complex form. When the correlation feature data is processed by the activation function, the real part of the correlation feature data can be processed by the real part activation function, and the imaginary part of the correlation feature data can be processed by the imaginary part activation function, and the masking value is then formed by the outputs of the real part activation function and the imaginary part activation function.

[0048] Optionally, the real activation function and the imaginary activation function are both hyperbolic tangent functions (tanh). Of course, the real activation function and the imaginary activation function can also be other types of functions. The embodiment of the present application does not limit the types of the real activation function and the imaginary activation function.

[0049] In an embodiment of the present application, noisy speech data is converted into time-frequency domain feature data in complex form, and the time-frequency domain feature data is processed by a speech enhancement model based on a complex-valued neural network to obtain correlation feature data in complex form. The real and imaginary parts of the correlation feature data are then processed separately by an activation function to obtain a masking value. The noisy speech data is enhanced by the speech enhancement model of the complex-valued neural network. Based on the special operation method of complex numbers, the time-frequency domain feature data can be processed from multiple dimensions such as channels, frames, and frequencies of the time-frequency domain feature data to extract a high-level feature representation of the time-frequency domain feature data. Based on the high-level feature representation, the noisy speech data is speech enhanced to ensure the effect of speech enhancement.

[0050] In one possible implementation, a speech enhancement model for speech enhancement includes an encoder, an attention module, a loop module and an encoder. After the time-frequency domain feature data is input into the speech enhancement model, the time-frequency domain feature data is first processed by the encoder, and then the output data of the encoder is processed by the attention module and the loop module respectively. Then, the output data of the attention module and the output data of the loop module are processed by the encoder to obtain correlation feature data indicating the noise signal and the useful speech signal in the time-frequency domain feature data.

[0051] In an embodiment of the present application, the encoder-decoder is implemented by a convolutional neural network (CNN), and the loop module is implemented by a recurrent neural network (RNN). The encoder-decoder can extract high-level features of noisy speech data, and the loop module can model the long-term time dependency of noisy speech data. Therefore, the speech enhancement model with a convolutional loop structure can effectively filter out noise signals in the time-frequency domain feature data, thereby ensuring the effect of speech enhancement on noisy speech data.

[0052] After the encoder processes the time-frequency domain feature data, the attention module processes the encoder's output data, filtering out interference from it. The filtered data is then fed into the decoder for further processing. The attention module processes the encoder's output data based on the attention mechanism, filtering out interference from it. The decoder then processes the filtered data, enabling more accurate identification of noise and useful speech signals within the time-frequency domain feature data, further improving speech enhancement.

[0053] In one possible implementation, the encoder includes M first convolutional cyclic blocks, where M is a natural number greater than or equal to 2. The first convolutional cyclic block includes a first convolution unit for a first convolution process and a first cyclic unit for a first cyclic process. For the i-th first convolutional cyclic block in the encoder, the first convolution unit in the first convolutional cyclic block is used to perform a first convolution process on the input data, and the first cyclic unit in the first convolutional cyclic block is used to perform a first cyclic process on the result of the first convolution process to obtain output data of the first convolutional cyclic block, where 1≤i≤M. The output data of each first convolutional cyclic block is input into the attention module, and the output data of the M-th first convolutional cyclic block is input into the cyclic module.

[0054] In an embodiment of the present application, the encoder includes at least two first convolution cycle blocks, each of which includes a first convolution unit and a first circulation unit. The first convolution unit is used to perform a first convolution process on the data input to the first convolution cycle block, and the first circulation unit is used to perform a first circulation process on the result of the first convolution process. After each first convolution cycle block performs convolution cycle processing on the input data, the output data of each first convolution cycle block is input into the attention module, and the output data of the Mth first convolution cycle block is input into the circulation module. By performing convolution cycle processing on the time-frequency domain feature data through multiple first convolution cycle blocks, the high-level features of the time-frequency domain feature data can be more effectively and comprehensively extracted, and then the noisy speech data can be enhanced based on the extracted high-level features to ensure the effect of speech enhancement. The output data of each first convolution cycle block is input into the attention module, and the interference information is filtered by the attention module to ensure that the input to the decoder includes less interference information, ensuring that the decoder can more accurately identify the noise signal and useful speech information, and then generate correlation feature data that can accurately represent the noise signal and useful speech signal.

[0055] For each first convolution cycle block, after the first convolution unit performs convolution processing on the input data, the first cycle unit performs cycle processing on the output data of the first convolution unit. The first cycle unit can realize the cycle in the frequency direction, realize frequency recursion, and extract the long-range correlation of the time-frequency domain feature data along the frequency direction. Then, when the subsequent decoder processes the output data of the encoder, it can identify the noise signal and the useful speech signal based on the long-range correlation of the time-frequency domain feature data along the frequency direction, and generate correlation feature data that can accurately indicate the noise signal and the useful speech signal, thereby ensuring the effect of speech enhancement on noisy speech data.

[0056] In one possible implementation, when the value of i is 1, the input data of the i-th first convolution cycle block is the time-frequency domain feature data; when the position of i is 2 to M, the input data of the i-th first convolution cycle block is the output data of the i-1-th first convolution cycle block.

[0057] In an embodiment of the present application, the first first convolution cycle block uses time-frequency domain feature data as input data. Except for the first first convolution cycle block, the other first convolution cycle blocks use the output data of the previous first convolution cycle block as input data. Starting from the second first convolution cycle block, each first convolution cycle block performs convolution cycle processing on the output data of the previous first convolution cycle block, and inputs the convolution cycle processing result into the next first convolution cycle block, and the last (Mth) first convolution cycle block inputs the convolution cycle processing result into the loop module. The encoder includes multiple first convolution cycle blocks, and the latter first convolution cycle block uses the output data of the previous first convolution cycle block as input data. Through the convolution cycle processing of each first convolution cycle block, the high-level features in the time-frequency domain feature data can be accurately and comprehensively extracted, and the noisy speech data can be more accurately enhanced.

[0058] In one possible implementation, when using the attention module to process the output data of the encoder, the attention module can be used to process the output data of each first convolution cycle block respectively, and the output data of the attention module can be input into the decoder.

[0059] The encoder includes multiple first convolution cycle blocks. After each first convolution cycle block performs convolution cycle processing on the input data, the output data is input into the attention module. The attention module processes the output data of each first convolution cycle block respectively to filter out the interference information in the output data of each first convolution cycle block respectively. After the attention module processes the output data of each first convolution cycle block, the processing result is sent to the decoder. The decoder generates correlation feature data according to the output results of the attention module. Since the data received by the decoder includes less interference information, the noise signal and the useful speech signal in the time-frequency domain feature data can be more accurately identified according to the received data to generate correlation feature data that can accurately indicate the noise signal and the useful speech signal in the time-frequency domain feature data, thereby improving the effect of speech enhancement on noisy speech data.

[0060] In one possible implementation, the cyclic module includes one or more cyclic units, and the cyclic unit is implemented by a recurrent neural network. When the cyclic module includes multiple cyclic units, the first cyclic unit uses the output data of the Mth first convolution cyclic block as input data, and the other cyclic units except the first cyclic unit all use the output data of the previous cyclic unit as input data. By cyclically processing the output data of the Mth first convolution cyclic block by multiple cyclic units, the long-term time dependency of the time-frequency domain feature data can be effectively extracted, and then the decoder can generate correlation feature data based on the long-term time dependency of the time-frequency domain feature data, ensuring that the generated correlation feature data can accurately indicate the noise signal and useful speech signal in the time-frequency domain feature data, thereby improving the effect of speech enhancement on noisy speech data.

[0061] Optionally, the recurrent unit included in the recurrent module can be implemented using a feedforward sequential memory neural network (FSMN). When the frequency-domain feature data is in complex form and the speech enhancement model is based on a complex-valued neural network, the recurrent unit is a FSMN that can process complex values.

[0062] In one possible implementation, corresponding to the encoder including M first convolutional cyclic blocks, the decoder includes M second convolutional cyclic blocks, each of which includes a second convolution unit for second convolution processing and a second cyclic unit for second cyclic processing. When the decoder processes the output data of the attention module and the output data of the cyclic module to obtain correlation feature data, for the i-th second convolutional cyclic block among the M second convolutional cyclic blocks, the second convolution unit in the second convolutional cyclic block performs a second convolution processing on the second input data and the second input data, and the second cyclic unit in the second convolutional cyclic block performs a second cyclic processing on the result of the second convolution processing to obtain the output data of the second convolutional cyclic block. The output data of the M-th second convolutional cyclic block is the correlation feature data. If the value of i is 1, the first input data of the second convolution unit in the i-th second convolutional cyclic block is the output data after the attention module processes the output data of the M-th first convolutional cyclic block, and the second input data of the second convolution unit in the i-th second convolutional cyclic block is the output data of the cyclic module. If the value of i is 2~M, the first input data of the second convolution unit in the i-th second convolution cycle block is the output data after the attention module processes the output data of the M+1-i-th first convolution cycle block, and the second input data of the second convolution unit in the i-th second convolution cycle block is the output data of the i-1-th second convolution cycle block.

[0063] In an embodiment of the present application, the encoder and decoder have a symmetrical structure. The encoder includes M first convolution cycle blocks, and the decoder includes M second convolution cycle blocks. The output data of the i-th first convolution cycle block is processed by the attention module and input into the M+1-i-th second convolution cycle block as the first input data. The output data of the cycle module is input into the first second convolution cycle block as the second input data of the first second convolution cycle block. Starting from the second second convolution cycle block, each second convolution cycle block uses the output data of the previous second convolution cycle block as the second input data. The encoder and decoder adopt a symmetrical structure to ensure that the time-frequency domain feature data input to the speech enhancement model has the same data structure as the correlation feature data output by the speech enhancement model, thereby ensuring that the enhanced speech data of the noisy speech data can be obtained based on the correlation feature data output by the decoder, thereby ensuring the effect of speech noise reduction.

[0064] In one possible implementation, the first recurrent unit in the first convolutional recurrent block and the second recurrent unit in the second convolutional recurrent block can both be implemented by a feedforward sequence memory neural network. Because the feedforward sequence memory neural network can effectively perform frequency recursion to capture the long-range correlation of time-frequency domain feature data in the frequency direction, the encoder and decoder composed of the convolution unit and the feedforward sequence memory neural network can capture the local time-frequency structure of noisy speech data and the long-range correlation of the spectrum, thereby better distinguishing between noise signals and useful speech signals, thereby improving the effect of speech enhancement.

[0065] Speech Enhancement Model

[0066] The following is a detailed description of the speech enhancement model that can implement the speech enhancement method in the embodiment of the present application. Figure 3 Schematic diagram of a speech enhancement model provided in an embodiment of the present application. Figure 3 As shown, the speech enhancement model 31 includes an encoder 311, a decoder 312, an attention module 313, and a loop module 314. The encoder 311 is used to process the time-frequency domain feature data corresponding to the noisy speech data, the attention module 313 is used to process the output data of the encoder, the loop module 314 is used to process the output data of the encoder, and the decoder 312 is used to process the output data of the attention module 313 and the loop module 314 to obtain correlation feature data for characterizing the long-range correlation of the time-frequency domain feature data in the frequency direction.

[0067] It should be noted that the speech enhancement model 31 can be used in the speech enhancement method in the above embodiment to generate correlation feature data based on time-frequency domain feature data. For the data processing process of each module in the speech enhancement model 31, please refer to the description in the above method embodiment, which will not be repeated here.

[0068] Figure 4 Schematic diagram of another speech enhancement model provided in the embodiment of the present application. Figure 4 As shown, the encoder 311 includes M first convolutional cyclic blocks, the M first convolutional cyclic blocks are sequentially first convolutional cyclic blocks 3111 to first convolutional cyclic blocks 311M, and the i-th first convolutional cyclic block is a first convolutional cyclic block 311i. The decoder 312 includes M second convolutional cyclic blocks, the M second convolutional cyclic blocks are sequentially second convolutional cyclic blocks 3121 to second convolutional cyclic blocks 312M, and the i-th second convolutional cyclic block is a second convolutional cyclic block 312i, where M is a natural number greater than or equal to 2. The loop module 314 includes N complex feedforward sequence memory neural networks (CFSMNs) connected in series, where N is a natural number. When N is a natural number greater than or equal to 2, the N CFSMNs are CFSMN3141 to CFSMN314N, respectively. The i-th CFSMN is CFSMN314i. The first CFSMN3141 is connected to the M-th first convolution loop block, and the N-th CFSMN314N is connected to the first second convolution loop block 3121.

[0069] The noisy speech data x is converted into time-frequency domain feature data X in complex value form by short-time Fourier transform (STFT) 32. The time-frequency domain feature data X includes a real part X r and the imaginary part X i After the time-frequency domain feature data X is input into the speech enhancement model 31, the speech enhancement model 31 processes the time-frequency domain feature data X and outputs correlation feature data. The real part of the correlation feature data is input into the real part activation function (tanh) 331 to obtain the masking value. The real part of the correlation feature data is input into the imaginary part activation function (tanh) 332 to obtain the mask value The imaginary part of The real and imaginary parts of the complex masked value The time-frequency domain feature data X and the masking value Perform element-by-element complex multiplication to obtain enhanced speech data in the time-frequency domain The enhanced speech data in the time-frequency domain is transformed by inverse short-time Fourier transform (ISTFT)34 Enhanced speech data converted to time domain

[0070] Figure 5 Schematic diagram of a first convolution cycle block provided in an embodiment of the present application. Figure 5As shown, the first convolutional recurrent block 311i includes a convolution unit 501, a normalization unit 502, an activation function 503, and a recurrent unit 504. The convolution unit 501 is used to process a complex two-dimensional convolutional neural network. The activation function 503 is a LeakyReLU function. The recurrent unit 504 is implemented using a CFSMN (Complex Feedforward Sequential Memory Neural Network).

[0071] The input data of convolution unit 501 is a three-dimensional feature matrix Among them, V r Used to represent the real part, V i Used to represent the imaginary part, j is the imaginary unit, j 2 =-1, is used to represent the complex domain, C is used to represent the channel dimension, T is used to represent the frame dimension, and F is used to represent the frequency dimension. The convolution kernel of the convolution unit 501 is Among them, W r Used to represent the real part, W i is used to represent the imaginary part, C′ is used to represent the number of convolution kernels, and T′×F′ is used to represent the size of the convolution kernel. The output data of the convolution unit 501 can be expressed as The real part U of the output of convolution unit 501 r and the imaginary part U i It can be expressed as the following formula (1):

[0072] U r =V r *W r -V i *W i

[0073] U i =V r *W i +V i *W r (1)

[0074] Where * is used to represent convolution filtering of real numbers. Convolution unit 501 performs causal convolution with T′=2 in the time direction, stride equal to 1, and zero padding. In the frequency direction, F′=5, stride equal to 2, and no zero padding is required. This reduces the feature map corresponding to the time-frequency domain feature data in encoder 311 by half block by block.

[0075] The above processing process is applicable to the convolution unit 501 in each first convolution cycle block. In the convolution unit 501 included in each first convolution cycle block, the same C′ is used to ensure that each first convolution cycle block has a feature map with the same data, for example, C′=128.

[0076] The output data of the convolution unit 501 is processed by the normalization unit 502 and the activation function 503 and then input into the circulation unit 504. In the circulation unit 504, the data The same processing is performed on the real and imaginary parts of The real part of is described below, and the process of processing the imaginary part refers to the process of processing the real part.

[0077] The data The real part U r The dimension is converted from C×T×F″ to T×F″×C. For data U r ∈R T×F″×C The current frame t in the frequency sequence can be formed The frequency sequence is input into the real part cyclic unit in the cyclic unit 504 for processing. For the i-th component in the frequency sequence, the real part cyclic unit processes it using the following formulas (2) to (4):

[0078]

[0079]

[0080]

[0081] Among them, f i =f1, f2, ..., F", and t is omitted in the above formulas (2) to (4), δ represents the activation function ReLU, N L and N R Represent the look-back and look-ahead order of the i-th component, such as N L =20 and N L = 0, and the other parameters are model parameters of the cyclic unit 504. Since the cyclic unit 504 in the embodiment of the present application only includes one CFSMN, the value of l is 1. Equivalent to is the output of the current real recurrent unit and the input of the next real recurrent unit. For the first real recurrent unit, S r (t) is the output of the activation function 503. In other embodiments, l can be other positive integers, such as 2, 3, 4, etc.

[0082] Output of recurrent unit 504 It can be expressed as the following formula (5):

[0083] S out =FSMN r (S r )-FSMNi (S i )+j(FSMN r (S i )+FSMN i (S r )) (5)

[0084] Among them, FSMN r and FSMN i are used to represent the real part of the recurrent unit and the imaginary part of the recurrent unit 504, S r and S i They are used to represent the real and imaginary parts of the frequency series respectively.

[0085] The output of the recurrent unit 504 in the first convolutional recurrent block 311M After the input loop module 314, the data The real part of Where H = F "× C', in which the Based on Forming a time series . Then each CFSMN in the loop module 314 applies the above formula (2) to formula (4) to Q r , to fit the time dynamics. The output of the loop module 314 can refer to the above formula (5).

[0086] It should be noted that the first convolution loop block processes the real and imaginary parts of the input data in a similar manner, and the process of processing the input data by the loop module 314 is similar to the process of processing the input data by the loop unit in the first convolution loop block. Therefore, the embodiment of the present application only describes in detail the process of processing the real part of the input data by the first convolution loop block, the process of processing the imaginary part of the input data by the first convolution loop block, and the process of processing the input data by the loop module 314. They will not be repeated here. The specific process can refer to the description of the processing of the real part of the input data by the above-mentioned first convolution loop block.

[0087] It should also be noted that the structure of the decoder 312 is symmetrical with that of the encoder 311. Therefore, the process of processing the input data by the second convolution cycle block 312i in the decoder 312 is similar to the process of processing the input data by the first convolution cycle block 311i. Therefore, the data processing process of the second convolution cycle block 312i will not be described in detail. For the detailed process, please refer to the above description of the first convolution cycle block 311i.

[0088] Speech Enhancement Model Training Method

[0089] Regarding the above Figure 3The speech enhancement model shown, the embodiment of the present application also provides a method for training the speech enhancement model. Figure 6 This is a flow chart of a model training method provided in an embodiment of the present application, which is used to train the speech enhancement model in the above embodiment. Figure 6 As shown, the model training method includes the following steps:

[0090] Step 601: Input the time-frequency domain feature data of the training sample into the model to be trained.

[0091] The training samples can be formed by using clean speech data and noisy speech data. The training samples in the time domain are converted to the time-frequency domain to obtain time-frequency domain feature data of the training samples. For example, the training samples are converted to the time-frequency domain by using a short-time Fourier transform (SFTF) to obtain time-frequency domain feature data of the training samples.

[0092] Step 602: Determine a first loss value of the model to be trained.

[0093] After inputting the time-frequency domain feature data of the training sample into the to-be-trained model, an output result of the to-be-trained model is obtained, and a first loss value is obtained based on the output result of the to-be-trained model. The first loss value of the to-be-trained model is determined based on at least one of a signal-to-noise ratio loss (SI-SNR loss), a masking loss, and a real-imaginary spectrum loss.

[0094] In one possible implementation, the first loss value of the model to be trained can be expressed as formula (6):

[0095]

[0096] in, Used to characterize the first loss value, Used to characterize the signal-to-noise ratio loss, Used to characterize masking loss, Used to characterize the loss of real and imaginary spectrum. y is used to characterize the time domain feature data of clean speech in the training sample. It is used to represent the enhanced speech data in the time domain determined according to the output of the speech enhancement model, and M is used to represent the true masking value determined according to the clean speech data in the training sample. It is used to represent the masking value determined according to the output of the speech enhancement model, and Y is used to represent the time-frequency domain feature data of the clean speech in the training sample. It is used to represent the enhanced speech data in the time-frequency domain determined based on the correlation feature data output by the speech enhancement model. λ1 and λ2 are used to represent the preset coefficients.

[0097] Signal-to-noise ratio loss The formula can be Calculate, where ||.||2 represents the L2 norm, <.,.> represents dot product. Masking loss The formula can be Calculate the real and imaginary spectrum loss. The formula can be Perform calculations.

[0098] Step 603: Adjust the model parameters of the model to be trained according to the first loss value until the first loss value meets a preset first condition.

[0099] Each time the time-frequency domain feature data of a training sample is input to the model to be trained, a first loss value of the model to be trained is obtained, and it is determined whether the first loss value satisfies a preset first condition. If the first loss value satisfies the preset first condition, step 604 is executed. If the first loss value does not satisfy the preset first condition, the model parameters of the model to be trained are adjusted based on the first loss value, and step 601 is continued until the first loss value satisfies the preset first condition.

[0100] The first condition may be that the first loss value is less than a preset first threshold value, or that the first loss value no longer decreases, etc.

[0101] Step 604: Determine a second loss value of the model to be trained.

[0102] After the first loss value satisfies a preset first condition, a second loss value of the model to be trained is determined based on the output result of the model to be trained. The second loss value of the model to be trained is determined based on at least one of the first loss value, adversarial loss, and deep feature loss.

[0103] In one possible implementation, the second loss value of the model to be trained can be expressed as formula (7):

[0104]

[0105] in, Used to characterize the second loss value, Used to characterize adversarial loss, It is used to characterize the depth feature loss, and λ3 and λ4 are used to characterize the preset coefficients.

[0106] In order to determine the second loss value, an auxiliary network with the same model structure as the model to be trained is constructed. The structure of the auxiliary network is as follows: Figure 7 As shown, the auxiliary network includes an encoder 701, a recurrent module 702, a decoder 703, and a linear projection module 704. Adversarial loss and deep feature loss can be determined.

[0107] The encoder 701 and the linear projection module 704 included in the auxiliary network constitute the discriminator D, which is used to characterize the enhanced speech data in the time-frequency domain determined according to the correlation feature data output by the speech enhancement model. Input the discriminator D, and according to the output of the linear projection module 704, the adversarial loss can be determined Specifically, the formula Calculating adversarial loss Among them, E() is used to represent the expected value operation. The output result of the linear projection module 704 is used to characterize the time-frequency domain feature data Y and Input discriminator D, based on the output of encoder 701, determines the depth feature loss Specifically, the formula Calculating deep feature loss Among them L D It is used to represent the number of the first convolution cycle modules in the encoder 701, The first convolution cycle model used to characterize the encoder 701 is The output, D l (Y) is used to represent the output of the first convolutional loop model for Y in encoder 701.

[0108] In order to ensure that the discriminator D has sufficient spectral information, the encoder 701, the loop module 702 and the decoder 703 included in the auxiliary network constitute the reconstruction network C, and the loss value of the reconstruction network C is defined as The loss value of the discriminator D is defined as the following formula (8):

[0109]

[0110] Among them, E() is used to represent the expected value operation, and λ5 is used to represent the preset coefficient.

[0111] In the process of training the model to be trained, the reconstruction network C, the discriminator D and the model parameters of the model to be trained are trained alternately.

[0112] Step 605: Adjust the model parameters of the to-be-trained model according to the second loss value until the second loss value satisfies a preset second condition, thereby obtaining a speech enhancement model.

[0113] Each time the second loss value of the model to be trained is obtained, it is determined whether the second loss value satisfies a preset second condition. If the second loss value satisfies the preset second condition, training is completed, and a speech enhancement model is obtained. If the second loss value does not satisfy the preset second condition, the model parameters of the model to be trained are adjusted based on the second loss value, and step 601 is continued until the second loss value satisfies the preset second condition, completing model training and obtaining a speech enhancement model.

[0114] In an embodiment of the present application, the training process of the speech enhancement model is divided into two stages. In the first stage, the speech enhancement model is trained with a first loss value as a reference, and in the second stage, the speech enhancement model is trained with a second loss value as a reference. The first loss value is determined based on at least one of the signal-to-noise ratio loss, masking loss, and real and imaginary spectrum loss. The first loss value indicates the signal level loss of the speech signal itself. Training the speech enhancement model based on the first loss value can enable the speech enhancement model to converge quickly and improve the effect of model training. The second loss value is determined based on at least one of the first loss value, adversarial loss, and deep feature loss. Training the speech enhancement model based on the second loss value can improve the perception of the speech enhancement model and ensure the effect of speech enhancement by the speech enhancement model.

[0115] Speech Recognition Method

[0116] In view of the application scenario of the speech enhancement solution provided in the embodiment of the present application in speech recognition, the embodiment of the present application provides a speech recognition method, such as Figure 8 As shown, the speech recognition method includes the following steps:

[0117] Step 801: Acquire voice data to be processed, wherein the voice data to be processed includes noise, and the voice data to be processed includes one of the following: audio and video conference voice data, online education voice data, and network live broadcast voice data;

[0118] Step 802: converting the speech data to be processed into time-frequency domain feature data;

[0119] Step 803: Generate a masking value for the noisy speech data based on the long-range correlation of the time-frequency domain feature data in the frequency direction;

[0120] Step 804: Generate enhanced speech data of the speech data to be processed according to the masking value and the time-frequency domain feature data;

[0121] Step 805: Perform speech recognition on the enhanced speech data to obtain a recognition result.

[0122] It should be noted that Figure 8The embodiment shown is a specific application of the speech enhancement solution in the embodiment of the present application. The specific speech enhancement process can be found in the description of the aforementioned embodiment and will not be repeated here.

[0123] Speech enhancement device

[0124] Corresponding to the above method embodiment, Figure 9 A schematic diagram of a speech enhancement device is shown. Figure 9 As shown, the speech enhancement device includes:

[0125] A conversion unit 901 is used to convert the noisy speech data into time-frequency domain feature data;

[0126] A processing unit 902 is configured to generate a masking value for the noisy speech data based on the long-range correlation of the time-frequency domain feature data in the frequency direction;

[0127] The generating unit 903 is configured to generate enhanced speech data of the noisy speech data according to the masking value and the time-frequency domain feature data.

[0128] It should be noted that the speech enhancement device of this embodiment is used to implement the corresponding speech enhancement method in the aforementioned method embodiment and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0129] electronic devices

[0130] Figure 10 This is a schematic block diagram of an electronic device provided in an embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device. Figure 10 As shown, the electronic device may include: a processor 1002, a communications interface 1004, a memory 1006, and a communication bus 1008.

[0131] The processor 1002 , the communication interface 1004 , and the memory 1006 communicate with each other via a communication bus 1008 .

[0132] The communication interface 1004 is used to communicate with other electronic devices or servers.

[0133] The processor 1002 is configured to execute the program 1010, and specifically may execute the relevant steps in any of the aforementioned speech enhancement method embodiments.

[0134] Specifically, the program 1010 may include program codes, which include computer operation instructions.

[0135] The processor 1002 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0136] The memory 1006 is used to store the program 1010. The memory 1006 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0137] The program 1010 can be specifically used to enable the processor 1002 to execute the speech enhancement method in any of the aforementioned embodiments.

[0138] The specific implementation of each step in program 1010 can be found in the corresponding descriptions of the corresponding steps and units in any of the aforementioned speech enhancement method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding process descriptions in the aforementioned method embodiments, and will not be repeated here.

[0139] By using the electronic device of the embodiment of the present application, after the noisy speech data is converted into time-frequency domain feature data, due to the long-range correlation of the time-frequency domain feature data in the frequency direction, the correlation between the noise signal and the useful speech signal with a large frequency difference in the noisy speech data can be characterized. Therefore, based on the long-range correlation of the time-frequency domain feature data in the frequency direction, the noise signal and the useful speech signal with a large frequency difference in the time-frequency domain feature data can be determined, thereby generating a masking value for the noisy speech data, and generating enhanced speech data for the noisy speech data based on the masking value and the time-frequency domain feature data. The noise signal with a large frequency difference in the generated enhanced speech data is filtered out, thereby improving the effect of speech enhancement.

[0140] Computer storage media

[0141] This application also provides a computer-readable storage medium storing instructions for causing a machine to perform the speech enhancement method described herein. Specifically, a system or device equipped with a storage medium can be provided, wherein the storage medium stores software program code that implements the functions of any of the above-described embodiments, and a computer (or CPU or MPU) of the system or device can read and execute the program code stored in the storage medium.

[0142] In this case, the program code read from the storage medium itself can realize the function of any one of the above embodiments, so the program code and the storage medium storing the program code constitute part of this application.

[0143] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0144] Computer program product

[0145] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to perform any corresponding operation in the above-mentioned multiple method embodiments.

[0146] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0147] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or can be implemented as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or can be implemented as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown here.

[0148] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0149] The above implementation methods are only used to illustrate the embodiments of the present application, and are not intended to limit the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.

Claims

1. A speech enhancement method, comprising: Convert noisy speech data into time-frequency domain feature data; generating a masking value for the noisy speech data according to the long-range correlation of the time-frequency domain feature data in the frequency direction; generating enhanced speech data of the noisy speech data according to the masking value and the time-frequency domain feature data; Among them, the masking value of the noisy speech data is generated according to the long-range correlation of the time-frequency domain feature data in the frequency direction, including: inputting the time-frequency domain feature data into a speech enhancement model to obtain correlation feature data output by the speech enhancement model, wherein the correlation feature data can indicate the noise signal and the useful speech signal in the time-frequency domain feature data; and obtaining the masking value based on the correlation feature data.

2. The speech enhancement method according to claim 1, wherein: Obtaining the masking value based on the correlation feature data includes: The real part and the imaginary part of the correlation feature data are processed respectively by an activation function to obtain the masking value.

3. The speech enhancement method according to claim 2, wherein: The speech enhancement model includes an encoder, an attention module, a recurrent module and a decoder. Inputting the time-frequency domain feature data into a speech enhancement model to obtain correlation feature data output by the speech enhancement model includes: Processing the time-frequency domain feature data using an encoder; Using the attention module and the loop module to process the output data of the encoder respectively, wherein the attention module is used to filter interference information in the output data of the encoder; The decoder is used to process the output data of the attention module and the output data of the loop module to obtain the correlation feature data.

4. The speech enhancement method according to claim 3, wherein: The encoder includes M first convolution cycle blocks, where M is a natural number greater than or equal to 2, and the first convolution cycle block includes a first convolution unit for a first convolution process and a first cycle unit for a first cycle process. The processing of the time-frequency domain feature data using an encoder includes: For the i-th first convolution cycle block, perform a first convolution process on the input data using the first convolution unit in the first convolution cycle block, and perform a first loop process on the result of the first convolution process using the first loop unit in the first convolution cycle block to obtain output data of the first convolution cycle block, where 1≤i≤M; The output data of each first convolutional cycle block is input into the attention module, and the output data of the Mth first convolutional cycle block is input into the cycle module.

5. The speech enhancement method according to claim 4, wherein: If the value of i is 1, the input data of the i-th first convolution cycle block is the time-frequency domain feature data; If the value of i is 2 to M, the input data of the i-th first convolution cycle block is the output data of the i-1-th first convolution cycle block.

6. The speech enhancement method according to claim 4, wherein: Processing the output data of the encoder using the attention module includes: The attention module is used to process the output data of each of the first convolution cycle blocks respectively, and the output data of the attention module is input to the decoder.

7. The speech enhancement method according to claim 4, wherein: Processing the output data of the encoder using the loop module includes: The output data of the Mth first convolution cycle block is processed using the cycle module, and the output data of the cycle module is input to the decoder.

8. The speech enhancement method according to claim 4, wherein: The decoder includes M second convolution cycle blocks, each of which includes a second convolution unit for a second convolution process and a second cycle unit for a second cycle process. The using the decoder to process the output data of the attention module and the output data of the loop module to obtain the correlation feature data includes: For the i-th second convolution cycle block, use the second convolution unit in the second convolution cycle block to perform a second convolution process on the first input data and the second input data, and use the second convolution unit in the second convolution cycle block to perform a second loop process on the result of the second convolution process to obtain output data of the second convolution cycle block; wherein the output data of the M-th second convolution cycle block is the correlation feature data, If the value of i is 1, the first input data of the second convolution unit in the i-th second convolution cycle block is the output data after the attention module processes the output data of the M-th first convolution cycle block, and the second input data of the second convolution unit in the i-th second convolution cycle block is the output data of the cycle module; If the value of i is 2~M, the first input data of the second convolution unit in the i-th second convolution cycle block is the output data after the attention module processes the output data of the M+1-i-th first convolution cycle block, and the second input data of the second convolution unit in the i-th second convolution cycle block is the output data of the i-1-th second convolution cycle block.

9. A speech enhancement model comprising: Encoder, attention module, recurrent module and decoder; The encoder is used to process the time-frequency domain feature data corresponding to the noisy speech data; The attention module is used to process the output data of the encoder; The circulation module is used to process the output data of the encoder; The decoder is used to process the output data of the attention module and the loop module to obtain correlation feature data for characterizing the long-range correlation of the time-frequency domain feature data in the frequency direction.

10. The speech enhancement model according to claim 9, wherein: The speech enhancement model is trained in the following way: Input the time-frequency domain feature data of the training sample into the model to be trained; Determining a first loss value of the to-be-trained model, wherein the first loss value is determined according to at least one of a signal-to-noise ratio loss, a masking loss, and a real-imaginary part spectrum loss; Adjusting the model parameters of the to-be-trained model according to the first loss value until the first loss value satisfies a preset first condition; Determining a second loss value of the to-be-trained model, wherein the second loss value is determined according to at least one of the first loss value, adversarial loss, and deep feature loss; The model parameters of the model to be trained are adjusted according to the second loss value until the second loss value meets the preset second condition, thereby obtaining the speech enhancement model.

11. A speech recognition method, comprising: Acquire voice data to be processed, wherein the voice data to be processed includes noise, and the voice data to be processed includes one of the following: audio and video conference voice data, online education voice data, and network live broadcast voice data; Converting the speech data to be processed into time-frequency domain feature data; generating a masking value for the speech data to be processed according to the long-range correlation of the time-frequency domain feature data in the frequency direction; generating enhanced speech data of the speech data to be processed according to the masking value and the time-frequency domain feature data; Performing speech recognition on the enhanced speech data to obtain a recognition result; Among them, based on the long-range correlation of the time-frequency domain feature data in the frequency direction, the masking value of the speech data to be processed is generated, including: inputting the time-frequency domain feature data into the speech enhancement model to obtain the correlation feature data output by the speech enhancement model, wherein the correlation feature data can indicate the noise signal and the useful speech signal in the time-frequency domain feature data; and obtaining the masking value based on the correlation feature data.

12. An electronic device comprising: Processor, memory, communication interface and communication bus, the processor, memory and communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the speech enhancement method according to any one of claims 1 to 8 or an operation corresponding to the speech recognition method according to claim 11.

13. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the program implements the speech enhancement method according to any one of claims 1 to 8 or the speech recognition method according to claim 11.

14. A computer program product comprising computer instructions, wherein the computer instructions instruct a computing device to execute operations corresponding to the speech enhancement method according to any one of claims 1 to 8 or the speech recognition method according to claim 11.

Images