A speech enhancement method, apparatus, device, and medium based on neural networks.

Abstract

This application provides a speech enhancement method, apparatus, device, and medium based on neural networks. The method includes: inputting a first frequency domain feature corresponding to the target speech into a first neural network model for processing to obtain a first frequency band gain; inputting a second frequency domain feature corresponding to the target speech and the first frequency band gain into a second neural network model for processing to obtain a second frequency band gain; and performing calculations on the first frequency band gain, the second frequency band gain, and the target speech spectrum to obtain enhanced speech. By implementing the scheme of this application, the first and second frequency domain features corresponding to the target speech are input into two neural network models respectively, and then the frequency band gains output by the two neural network models and the target speech spectrum are calculated to obtain enhanced speech, which can effectively reduce the complexity of the network structure and improve the perceived quality of speech.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a method, apparatus, device and medium for speech enhancement based on neural networks. Background Technology

[0002] Speech enhancement aims to recover clean speech from noisy speech to improve communication efficiency between people or between machines and people. Currently, intelligent speech systems (such as smart homes and intelligent robots) primarily employ speech enhancement methods based on deep neural networks (DNNs). A typical approach involves inputting frequency domain features (such as logarithmic power spectrum (LPS), Mel-frequency cepstral coefficients (MFCC), and complex features) into a neural network, and then the neural network outputs a time-frequency mask (TFM) which is applied to the spectrum of the noisy speech to achieve speech enhancement. Applying DNN-based speech enhancement algorithms to embedded systems mainly considers: system time complexity (computational load) and space complexity (memory), and speech quality. To reduce system complexity, common approaches include: designing relatively simple network structures, reducing feature dimensions and the number of network nodes; and multi-stage noise reduction processing. To overcome speech distortion problems, common approaches include: incorporating multiple loss functions; outputting complex time-frequency masks; and using complex network structures to improve speech perception quality.

[0003] While the solutions described above improve the complexity and speech quality of speech enhancement systems to some extent, these solutions are contradictory and present the following problems in practical implementation: Designing simple network structures and reducing feature dimensions and the number of network nodes often leads to poor model performance and generalization capabilities; multi-stage denoising is effective in reducing model complexity, but it can easily exacerbate speech distortion. For example, if denoising in the previous stage damages the speech, subsequent denoising may not only fail to compensate for the distortion but could further damage it; the conventional loss function is Mean Square Error (MSE), which is prone to causing speech distortion at low signal-to-noise ratios, and even with the addition of other loss functions for collaborative training, this problem remains difficult to solve; while outputting complex frequency masks has certain advantages in improving speech intelligibility compared to conventional real number masks, it requires designing more reasonable and complex network structures. Summary of the Invention

[0004] The main objective of this application is to provide a speech enhancement method, apparatus, device, and medium based on neural networks, which can at least solve the problems of speech distortion and complex network structures in related technologies.

[0005] To achieve the above objectives, a first aspect of the present invention provides a speech enhancement method based on a neural network, the method comprising:

[0006] The first frequency domain feature corresponding to the target speech is input into the first neural network model for processing to obtain the first frequency band gain;

[0007] The second frequency domain feature corresponding to the target speech and the first frequency band gain are input into the second neural network model for processing to obtain the second frequency band gain; wherein, the second frequency domain feature is a complex feature, and the second frequency band gain includes a real part gain and an imaginary part gain;

[0008] The enhanced speech is obtained by performing calculations on the first frequency band gain, the second frequency band gain, and the target speech spectrum.

[0009] Furthermore, the second neural network model includes multiple sub-neural networks; each sub-neural network includes multiple first network layers; each first network layer includes multiple network sub-layers;

[0010] The step of inputting the second frequency domain features corresponding to the target speech and the first frequency band gain into the second neural network model to obtain the second frequency band gain includes:

[0011] Based on the frequency band information of the target speech spectrum, extract the second frequency domain features corresponding to each frequency band;

[0012] According to the preset feature input rules, each of the second frequency domain features and the first frequency band gain are respectively input into the corresponding sub-neural network in the second neural network model to obtain the third frequency band gain;

[0013] The gains of each of the third frequency bands are spliced ​​together to obtain the second frequency band gain.

[0014] Furthermore, the step of inputting each of the second frequency domain features and the first frequency band gain into the corresponding sub-neural network in the second neural network model to obtain the third frequency band gain includes:

[0015] The second frequency domain feature and the first frequency band gain are input into each sub-layer of the corresponding sub-neural network in the second neural network model to obtain the first intermediate parameter corresponding to the sub-layer of the network.

[0016] The first intermediate parameters output from the previous first network layer are concatenated and then input into the next first network layer to obtain the third frequency band gain.

[0017] Furthermore, the loss function of the first neural network model is expressed as:

[0018] L=αLMSE +(1-)L SPEECH

[0019]

[0020]

[0021] Where L is the loss function of the first neural network model, α is the control factor for speech preservation, and L MSE Let L be the mean squared error loss function. SPEECH The speech preservation loss function is defined as follows: t is the time of the target speech, f is the frequency of the target speech, T is the total number of time points, F is the total number of frequency points, X(t,f) is the time-frequency representation of the reference enhanced speech, G(t,f) is the time-frequency gain, and Y(t,f) is the time-frequency representation of the target speech.

[0022] Furthermore, the first neural network model includes multiple second network layers;

[0023] The step of inputting the first frequency domain feature corresponding to the target speech into the first neural network model to obtain the first frequency band gain includes:

[0024] The first frequency domain feature corresponding to the target speech is input into the second network layer of the first neural network model, and the second intermediate parameter is output.

[0025] The second intermediate parameters output from the previous second network layer are sequentially input into the next second network layer to obtain the first frequency band gain.

[0026] Further, the step of calculating the first frequency band gain, the second frequency band gain, and the target speech spectrum to obtain the enhanced speech includes:

[0027] The first frequency band gain and the target speech spectrum are calculated to obtain the first enhanced speech spectrum;

[0028] The second frequency band gain is multiplied by the first enhanced speech spectrum in the complex field to obtain the second enhanced speech spectrum;

[0029] The enhanced speech is obtained by performing an inverse Fourier transform on the spectrum of the second enhanced speech.

[0030] Furthermore, the step of calculating the first frequency band gain and the target speech spectrum to obtain the first enhanced speech spectrum includes:

[0031] Linear interpolation is performed on the first frequency band gain to obtain the fourth frequency band gain;

[0032] The fourth frequency band gain is multiplied by the target speech spectrum to obtain the first enhanced speech spectrum.

[0033] A second aspect of this application provides a speech enhancement device based on a neural network, comprising:

[0034] The first processing module is used to input the first frequency domain feature corresponding to the target speech into the first neural network model for processing to obtain the first frequency band gain;

[0035] The second processing module is used to input the second frequency domain feature corresponding to the target speech and the first frequency band gain into the second neural network model for processing to obtain the second frequency band gain; wherein, the second frequency domain feature is a complex feature; the second frequency band gain includes a real part gain and an imaginary part gain;

[0036] The calculation module is used to perform calculations on the first frequency band gain, the second frequency band gain, and the target speech spectrum to obtain enhanced speech.

[0037] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is used to execute a computer program stored in the memory, and when the processor executes the computer program, it implements the steps of the neural network-based speech enhancement method provided in the first aspect of this application.

[0038] The fourth aspect of this application provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the neural network-based speech enhancement method provided in the first aspect of this application.

[0039] As can be seen from the above, according to the neural network-based speech enhancement method, apparatus, device, and medium provided in this application, the first frequency domain feature corresponding to the target speech is input into a first neural network model for processing to obtain a first frequency band gain; the second frequency domain feature corresponding to the target speech and the first frequency band gain are input into a second neural network model for processing to obtain a second frequency band gain; wherein, the second frequency domain feature is a complex feature, and the second frequency band gain includes a real part gain and an imaginary part gain; the first frequency band gain, the second frequency band gain, and the target speech spectrum are calculated to obtain enhanced speech. Through the implementation of this application, the first frequency domain feature and the second frequency domain feature corresponding to the target speech are input into two neural network models respectively, and the frequency band gains output by the two neural network models and the target speech spectrum are calculated to obtain enhanced speech. The multi-stage enhancement processing and the output complex gain can effectively reduce the complexity of the network structure and improve the perceptual quality of speech. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A schematic diagram illustrating the basic process of the neural network-based speech enhancement method provided in the first embodiment of this application;

[0042] Figure 2 This is a schematic diagram of the structure of a second neural network model provided in the first embodiment of this application;

[0043] Figure 3 A detailed flowchart illustrating the neural network-based speech enhancement method provided in the second embodiment of this application;

[0044] Figure 4 A schematic diagram of a two-stage frequency division grouping speech enhancement network provided in the second embodiment of this application;

[0045] Figure 5 A schematic diagram of a neural network-based speech enhancement device provided in the third embodiment of this application;

[0046] Figure 6 This is a schematic diagram of the structure of an electronic device provided in the fourth embodiment of this application. Detailed Implementation

[0047] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0049] To address the issues of speech distortion and complex network structures in related technologies during speech enhancement, the first embodiment of this application provides a speech enhancement method based on neural networks, such as... Figure 1 This is a basic flowchart illustrating the neural network-based speech enhancement method provided in this embodiment. The neural network-based speech enhancement method includes the following steps:

[0050] Step 101: Input the first frequency domain feature corresponding to the target speech into the first neural network model for processing to obtain the first frequency band gain.

[0051] Specifically, in this embodiment, the system structure includes two deep neural network models, employing a two-stage speech enhancement approach. To reduce the complexity of the first stage, the extracted frequency domain features are low-dimensional F-bank features. The first neural network model is a low-complexity recurrent neural network (GRU network). The low-dimensional frequency domain features are input into the low-complexity neural network, and the output is the frequency band gain. F-bank stands for Filter Bank. Since the human ear's response to the sound spectrum is non-linear, F-bank is a front-end processing algorithm that processes audio in a manner similar to the human ear, improving speech recognition performance. The general steps for obtaining the fbank features of a speech signal are: pre-emphasis, framing, windowing, short-time Fourier transform (STFT), and mel filter filtering. The mel filter divides the frequency into a fixed number of frequency bands according to the characteristics of human auditory perception, such as 32 frequency bands in this embodiment, and then multiplies by a triangular window.

[0052] In some embodiments of this example, the first neural network model includes multiple second network layers; the step of inputting the first frequency domain feature corresponding to the target speech into the first neural network model to obtain the first frequency band gain includes: inputting the first frequency domain feature corresponding to the target speech into the second network layer of the first neural network model and outputting the second intermediate parameter; and sequentially inputting the second intermediate parameter output by the previous second network layer into the next second network layer to obtain the first frequency band gain.

[0053] Specifically, in this embodiment, the first neural network model is a low-complexity GRU network. This neural network model includes multiple network layers. After the frequency domain features are input into the neural network, each network layer processes the frequency domain features in turn. The output of the previous network layer will be used as the input of the next network layer until the last network layer has finished processing, and then outputs the frequency band gain.

[0054] Furthermore, in some embodiments of this example, the loss function of the first neural network model is expressed as:

[0055] L=αL MSE +(1-)L SPEECH

[0056]

[0057]

[0058] Where L is the loss function of the first neural network model, α is the control factor for speech preservation, and L MSE Let L be the mean squared error loss function. SPEECH The speech preservation loss function is defined as follows: t is the time of the target speech, f is the frequency of the target speech, T is the total number of time points, F is the total number of frequency points, X(t,f) is the time-frequency representation of the reference enhanced speech, which is obtained by Fourier transform of the waveform of the reference enhanced speech, i.e., the clean speech, and includes both time and frequency dimensions. G(t,f) is the time-frequency gain, which is output by the neural network model and multiplied by each time and frequency point of Y(t,f) to obtain the denoised speech (time-frequency domain). Y(t,f) is the time-frequency representation of the target speech, which is obtained by Fourier transform of the target speech, i.e., the noisy speech.

[0059] Specifically, in this embodiment, the use of a multi-stage noise reduction system introduces speech distortion, and the fidelity of the speech is highly dependent on the processing effect of the first stage. If the first stage damages speech components, the damage will be exacerbated in the second stage. In deep neural network systems based on time-frequency masks, the mainstream loss function is still MSE. Under the above MSE criterion, the training objective of the time-frequency gain ((t,f)) is the amplitude mask (SMM), that is, |(t,f)| / |9t,f)|. Under low signal-to-noise ratio conditions and in two-stage noise reduction network systems, the conventional MSE loss function is prone to speech distortion due to its fixed training objective.

[0060] To address the aforementioned issues, this embodiment designs a loss function suitable for two-stage network training. Based on the conventional MSE (Mean Sequence Estimation), a speech preservation loss function L is added. SPEECH However, if the speech preservation loss function is simply applied to the two-stage denoising network system, the training objective of G(t,f) will always be 1, meaning there will be no denoising function. For a two-stage training network, the problem of noise residue can be solved in the second stage. The key is how to prevent speech distortion in the first stage. Therefore, this embodiment provides an approach that combines the conventional MSE loss function and the speech preservation loss function module to ensure denoising is achieved without speech distortion as much as possible. A speech preservation control factor α is introduced, with a value ranging from 0 to 1. When α is 1, the loss function L applicable to the first stage degenerates into the MSE loss function. When α is 0, the first stage will have no denoising function (at which point there is no speech distortion). By controlling the value of α, the degree of denoising in the first stage can be balanced, while ensuring speech fidelity to a certain extent, thereby improving the robustness of the entire system.

[0061] Step 102: Input the second frequency domain features corresponding to the target speech and the first frequency band gain into the second neural network model for processing to obtain the second frequency band gain.

[0062] Specifically, in this embodiment, the second frequency domain feature is a complex feature, and the second frequency band gain includes real part gain and imaginary part gain. Currently, speech enhancement methods based on complex masks and complex network structures have been proven to effectively improve the perceptual quality and intelligibility of speech. Complex features generally include real and imaginary parts, amplitude combined with phase, logarithmic amplitude combined with phase difference between adjacent frequency points, etc. In this embodiment, the input feature of the neural network model in the second stage is a high-dimensional feature, which can optionally be a logarithmic power spectrum (LPS) feature. The general steps to obtain the LPS feature are: framing, windowing, short-time Fourier transform (STFT), squaring, and finally taking the logarithm to obtain the LPS.

[0063] In some embodiments of this example, the second neural network model includes multiple sub-neural networks; each sub-neural network includes multiple first network layers; each first network layer includes multiple network sub-layers; the step of inputting the second frequency domain features corresponding to the target speech and the first frequency band gain into the second neural network model to obtain the second frequency band gain includes: extracting the second frequency domain features corresponding to each frequency band based on the frequency band information of the target speech spectrum; inputting each second frequency domain feature and the first frequency band gain into the corresponding sub-neural network in the second neural network model according to a preset feature input rule to obtain the third frequency band gain; and concatenating the third frequency band gains to obtain the second frequency band gain.

[0064] Specifically, in this embodiment, considering the non-uniform distribution of speech harmonic components across the entire frequency band and the varying sensitivities of the human ear to different frequency bands, the second neural network model employs a network structure designed by dividing the network into groups according to frequency bands. For example... Figure 2 The diagram shows a schematic of a second neural network model provided in this embodiment. The frequency domain features corresponding to each frequency band are input into the corresponding sub-neural network, and each network layer is divided into different numbers of sub-layers. By dividing the frequency bands of noisy speech, extracting the frequency domain features of each band, and then inputting the extracted frequency domain features corresponding to each band into the corresponding sub-neural network according to the actual needs of the frequency band, the final frequency band gain is obtained by concatenating the frequency band gains output by each sub-neural network.

[0065] Furthermore, in some embodiments of this example, the step of inputting each second frequency domain feature and the first frequency band gain into the corresponding sub-neural network in the second neural network model to obtain the third frequency band gain includes: inputting the second frequency domain features and the first frequency band gain into each sub-layer of the corresponding sub-neural network in the second neural network model to obtain the first intermediate parameter corresponding to the sub-layer; and sequentially concatenating each first intermediate parameter output by the previous first network layer and inputting it into the next first network layer to obtain the third frequency band gain.

[0066] Specifically, in this embodiment, each sub-neural network in the second neural network model includes multiple network layers, and each network layer includes multiple network sub-layers. When the frequency domain features are input into the sub-neural network, the frequency domain features are processed in the network sub-layer of each network layer, and the intermediate parameters obtained from the processing are concatenated and used as the input of the next network layer, until the last network layer completes the processing and outputs the frequency band gain corresponding to the frequency domain features of the corresponding frequency band.

[0067] It should be understood that the framework of a conventional DNN-based speech enhancement system generally involves extracting features from noisy speech, inputting the obtained features into a neural network, and processing them through multiple network layers to obtain the enhanced speech spectrum. The higher the dimensionality of the input features and the more nodes in the network layers, the more accurate the system's predicted target (mask or amplitude spectrum, etc.) will be, and the better the speech enhancement effect will be. However, regardless of whether the network layer is a fully connected (FC) layer or a recurrent neural network (RNN) layer, the number of network parameters increases dramatically with the increase in the number of nodes. Using network layer grouping can solve this problem to some extent. Network layer grouping involves dividing a network layer into many sub-layers. For example, a linear layer or recurrent neural network layer with 128 input and 128 output nodes can be divided into two linear layers or recurrent neural network layers with 64 input and 64 output nodes, resulting in a Groups of 2; similarly, it can also be divided into four network layers with 32 input and 32 output nodes, resulting in a Groups of 4. The more groups, the fewer parameters are required. While this approach is effective in reducing complexity, it has the following drawbacks: grouping usually comes at the cost of performance, and the more groups there are, the more significant the performance drop becomes; fixed grouping patterns limit the flexibility of model design and make it difficult to resolve the contradiction between complexity and performance.

[0068] The frequency band grouping network design in this embodiment effectively addresses the aforementioned issues. Firstly, this network design is more flexible. For example, the frequency band division allows for different network designs and varying network layer group sizes for each frequency band. It also enables different frequency band resource allocation based on different application scenarios and user groups. For instance, if hearing-impaired patients are sensitive to 1000Hz-2000Hz but have severe hearing loss in other frequency bands, a special system structure can be adaptively designed for these individuals. Secondly, network parameter configuration can be more rational. Some frequency bands do not require a large number of parameters, but fixed grouping still allocates a proportional amount of parameters. For speech, most harmonic components are concentrated below 4kHz, requiring the network to learn this information effectively. The frequency band grouping network structure can allocate more network resources to low frequencies (e.g., larger network nodes, fewer groups) and relatively fewer resources to high frequencies (e.g., smaller network nodes, more groups), making the network configuration more rational without sacrificing performance.

[0069] Step 103: Calculate the first frequency band gain, the second frequency band gain, and the target speech spectrum to obtain the enhanced speech.

[0070] Specifically, in this embodiment, after performing corresponding calculations on the frequency band gain output by the two neural network models and the spectrum of the noisy speech, clean speech can be obtained.

[0071] In some embodiments of this example, the step of performing calculations on the first frequency band gain, the second frequency band gain, and the target speech spectrum to obtain enhanced speech includes: performing calculations on the first frequency band gain and the target speech spectrum to obtain a first enhanced speech spectrum; performing a complex domain multiplication operation on the second frequency band gain and the first enhanced speech spectrum to obtain a second enhanced speech spectrum; and performing an inverse Fourier transform on the second enhanced speech spectrum to obtain enhanced speech.

[0072] Specifically, in this embodiment, the bandwidth gain G output by the first neural network model B The enhanced speech spectrum X1(t,f) is obtained by operating on the noisy speech spectrum Y1(t,f) and the noisy speech spectrum Y1(t,f). The enhanced speech spectrum X1(t,f) is then compared with the complex gain G output by the second neural network model. r (t,f), G iMultiplication in the complex domain is performed on (t,f) to obtain the enhanced speech spectrum X2(t,f). Finally, an inverse Fourier transform is performed on the enhanced speech spectrum X2(t,f) to obtain the enhanced speech. The network output bandwidth gain in the first stage is multiplied with the noisy speech spectrum to achieve basic noise reduction. The network output bandwidth gain in the second stage is multiplied with the initially enhanced speech spectrum from the first stage to achieve further noise reduction. The advantage of this method is that it breaks down a large system into multiple small subsystems and optimizes the performance of each subsystem. This improves the system's noise suppression capability on the one hand, and reduces the system's complexity on the other.

[0073] Furthermore, in some embodiments of this example, the step of calculating the first frequency band gain and the target speech spectrum to obtain the first enhanced speech spectrum includes: performing linear interpolation on the first frequency band gain to obtain the fourth frequency band gain; and multiplying the fourth frequency band gain with the target speech spectrum to obtain the first enhanced speech spectrum.

[0074] Specifically, in this embodiment, the output bandwidth gain G of the first neural network model... B After (t,f), then the band gain G B Linear interpolation is performed on (t,f) to obtain the full-band gain G1(t,f). Finally, the full-band gain G1(t,f) is multiplied by the noisy speech spectrum Y1(t,f) to obtain the enhanced speech spectrum X19t,f).

[0075] It should be understood that the general process of current mainstream complex-domain DNN speech enhancement systems involves extracting complex features from noisy speech, inputting these features into a CNN-based encoder-decoder structure, outputting real and imaginary gain, and finally using complex-domain multiplication to obtain the enhanced speech spectrum. The CNN-based encoder-decoder structure typically uses skip connections for feature fusion, and sometimes recurrent neural networks are added to learn temporal information. However, in practical embedded systems, the above approach is difficult to implement, mainly due to the following problems: the method of complex feature input and complex mask output heavily relies on the network's learning ability, usually requiring a complex network design; although the shared weight characteristic of CNNs reduces the number of parameters, the computational complexity of reusing convolutional kernels is very high. Generally, reducing the dimensionality of input features and designing a simpler network structure can greatly reduce network complexity, but this also leads to performance issues. The two-stage noise reduction network designed in this embodiment can effectively solve these problems, and the two-stage noise reduction network has the following characteristics: extremely low complexity; can effectively improve speech quality and speech intelligibility; simple structure and flexible design; and the noise reduction depth of the system can be manually adjusted.

[0076] Based on the technical solution of the above embodiments of this application, a first frequency domain feature corresponding to the target speech is input into a first neural network model for processing to obtain a first frequency band gain; a second frequency domain feature corresponding to the target speech and the first frequency band gain are input into a second neural network model for processing to obtain a second frequency band gain; wherein, the second frequency domain feature is a complex feature, and the second frequency band gain includes a real part gain and an imaginary part gain; the first frequency band gain, the second frequency band gain, and the target speech spectrum are calculated to obtain enhanced speech. Through the implementation of the solution of this application, the first frequency domain feature and the second frequency domain feature corresponding to the target speech are input into two neural network models respectively, and the frequency band gains output by the two neural network models and the target speech spectrum are calculated to obtain enhanced speech. The multi-stage enhancement processing and the output complex gain can effectively reduce the complexity of the network structure and improve the perceptual quality of speech.

[0077] Figure 3 The method described in the second embodiment of this application is a refined neural network-based speech enhancement method, which includes:

[0078] Step 301: Input the first frequency domain feature corresponding to the target speech into the first neural network model to obtain the first frequency band gain.

[0079] Step 302: Extract the second frequency domain features corresponding to each frequency band based on the frequency band information of the target speech spectrum.

[0080] Step 303: Input the second frequency domain features and the first frequency band gain into each sub-layer of the corresponding sub-neural network in the second neural network model to obtain the intermediate parameters corresponding to the sub-layers.

[0081] Step 304: Sequentially concatenate the intermediate parameters output from the previous network layer in the sub-neural network and input them into the next network layer to obtain the second frequency band gain.

[0082] Step 305: Segment the gains of each second frequency band to obtain the gain of the third frequency band.

[0083] Step 306: Calculate the first frequency band gain and the target speech spectrum to obtain the first enhanced speech spectrum.

[0084] Step 307: Perform a complex domain multiplication operation between the third frequency band gain and the first enhanced speech spectrum to obtain the second enhanced speech spectrum.

[0085] Step 308: Perform an inverse Fourier transform on the spectrum of the second enhanced speech to obtain the enhanced speech.

[0086] Specifically, in this embodiment, the second neural network model adopts a network structure that combines frequency band division with network layer grouping. For example, under the condition that the speech frequency is 16kHz and the FFT number is 512, the network structure is as follows: Figure 4 As shown, the low-dimensional feature input to the first neural network model is Fbank, and the high-dimensional feature input to the second neural network model is LPS. After passing through a linear network layer and a GRU network layer, Fbank outputs a bandwidth gain G. B (t,f), the second neural network model adopts a frequency-division grouped network structure, and the input is the frequency band gain G. B (t,f) and LPS features, where the number of groups in both the linear network layer and the GRU network layer is 2, and the output of the second neural network model is the complex gain, i.e., the real part gain G. r (t,f) and the imaginary gain G i (t,f), where the number in parentheses represents the number of nodes.

[0087] It should be understood that the sequence number of each step in this embodiment does not imply the order in which the steps are executed. The execution order of each step should be determined by its function and internal logic, and should not constitute a unique limitation on the implementation process of this application embodiment.

[0088] Based on the above technical solution of the embodiments of this application, a first frequency domain feature corresponding to the target speech is input into a first neural network model to obtain a first frequency band gain; according to the frequency band information of the target speech spectrum, a second frequency domain feature corresponding to each frequency band is extracted; the second frequency domain feature is input into a second neural network model to obtain a third frequency band gain; wherein, the second neural network model adopts a frequency-grouping network design; the first frequency band gain and the target speech spectrum are calculated to obtain a first enhanced speech spectrum; the third frequency band gain and the first enhanced speech spectrum are multiplied in the complex domain to obtain a second enhanced speech spectrum; the second enhanced speech spectrum is subjected to an inverse Fourier transform to obtain enhanced speech. Through the implementation of the solution of this application, a multi-stage network structure combined with frequency-grouping is adopted, which can effectively reduce the complexity of the speech enhancement system based on deep neural networks while ensuring the noise reduction effect.

[0089] Figure 5 This application provides a neural network-based speech enhancement device according to a third embodiment. This neural network-based speech enhancement device can be applied to the aforementioned neural network-based speech enhancement method. For example... Figure 5 As shown, the neural network-based speech enhancement device mainly includes:

[0090] The first processing module 501 is used to input the first frequency domain feature corresponding to the target speech into the first neural network model for processing to obtain the first frequency band gain;

[0091] The second processing module 502 is used to input the second frequency domain features corresponding to the target speech and the first frequency band gain into the second neural network model for processing to obtain the second frequency band gain; wherein, the second frequency domain features are complex features, and the second frequency band gain includes real part gain and imaginary part gain;

[0092] The calculation module 503 is used to perform calculations on the first frequency band gain, the second frequency band gain, and the target speech spectrum to obtain enhanced speech.

[0093] In some embodiments of this example, the first processing module is specifically used to: input the first frequency domain feature corresponding to the target speech into the second network layer of the first neural network model and output the second intermediate parameter; and sequentially input the second intermediate parameter output by the previous second network layer into the next second network layer to obtain the first frequency band gain.

[0094] In some embodiments of this example, the second processing module is specifically used to: extract second frequency domain features corresponding to each frequency band based on the frequency band information of the target speech spectrum; input each second frequency domain feature and the first frequency band gain into the corresponding sub-neural network in the second neural network model according to the preset feature input rules to obtain the third frequency band gain; and concatenate each third frequency band gain to obtain the second frequency band gain.

[0095] Furthermore, in some embodiments of this example, the second processing module is also used to: input the second frequency domain feature and the first frequency band gain into each sub-layer of the corresponding sub-neural network in the second neural network model to obtain the first intermediate parameter corresponding to the sub-layer; and sequentially concatenate each first intermediate parameter output by the previous first network layer and input it into the next first network layer to obtain the third frequency band gain.

[0096] In some embodiments of this example, the computation module is specifically used to: perform operations on the first frequency band gain and the target speech spectrum to obtain a first enhanced speech spectrum; perform complex domain multiplication operations on the second frequency band gain and the first enhanced speech spectrum to obtain a second enhanced speech spectrum; and perform inverse Fourier transform on the second enhanced speech spectrum to obtain enhanced speech.

[0097] Furthermore, in some embodiments of this example, the arithmetic module is also used to: perform linear interpolation on the first frequency band gain to obtain the fourth frequency band gain; and multiply the fourth frequency band gain with the target speech spectrum to obtain the first enhanced speech spectrum.

[0098] It should be noted that the neural network-based speech enhancement methods in the foregoing embodiments can all be implemented based on the neural network-based speech enhancement device provided in this embodiment. Those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process of the neural network-based speech enhancement device described in this embodiment can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0099] Based on the technical solution of the above embodiments of this application, a first frequency domain feature corresponding to the target speech is input into a first neural network model for processing to obtain a first frequency band gain; a second frequency domain feature corresponding to the target speech and the first frequency band gain are input into a second neural network model for processing to obtain a second frequency band gain; wherein, the second frequency domain feature is a complex feature, and the second frequency band gain includes a real part gain and an imaginary part gain; the first frequency band gain, the second frequency band gain, and the target speech spectrum are calculated to obtain enhanced speech. Through the implementation of the solution of this application, the first frequency domain feature and the second frequency domain feature corresponding to the target speech are input into two neural network models respectively, and the frequency band gains output by the two neural network models and the target speech spectrum are calculated to obtain enhanced speech. The multi-stage enhancement processing and the output complex gain can effectively reduce the complexity of the network structure and improve the perceptual quality of speech.

[0100] Figure 6 An electronic device provided in the fourth embodiment of this application, which can be used to implement the neural network-based speech enhancement method in the foregoing embodiments, mainly includes:

[0101] The system includes a memory 601, a processor 602, and a computer program 603 stored on the memory 601 and executable on the processor 602. The memory 601 and the processor 602 are connected via communication. When the processor 602 executes the computer program 603, it implements the method described in Embodiment 1 or 2 above. The number of processors can be one or more.

[0102] The memory 601 can be a high-speed random access memory (RAM) or a non-volatile memory, such as a disk storage device. The memory 601 is used to store executable program code, and the processor 602 is coupled to the memory 601.

[0103] Furthermore, embodiments of this application also provide a computer-readable storage medium, which may be disposed in the aforementioned electronic device, and the computer-readable storage medium may be as described above. Figure 6 The memory in the illustrated embodiment.

[0104] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the neural network-based speech enhancement method described in the foregoing embodiments. Furthermore, the computer-readable storage medium can also be a USB flash drive, external hard drive, read-only memory (ROM), RAM, magnetic disk, or optical disk, or any other medium capable of storing program code.

[0105] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0106] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0107] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0108] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0109] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0110] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0111] The above is a description of the speech enhancement method, apparatus, device and medium based on neural networks provided in this application. For those skilled in the art, based on the ideas of the embodiments of this application, there will be changes in the specific implementation and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A speech enhancement method based on neural networks, characterized in that, include: The first frequency domain feature corresponding to the target speech is input into the first neural network model for processing to obtain the first frequency band gain; The second frequency domain feature corresponding to the target speech and the first frequency band gain are input into the second neural network model for processing to obtain the second frequency band gain; wherein, the second frequency domain feature is a complex feature, and the second frequency band gain includes a real part gain and an imaginary part gain; The enhanced speech is obtained by performing calculations on the first frequency band gain, the second frequency band gain, and the target speech spectrum. The second neural network model includes multiple sub-neural networks; each sub-neural network includes multiple first network layers; each first network layer includes multiple network sub-layers; the step of inputting the second frequency domain features corresponding to the target speech and the first frequency band gain into the second neural network model to obtain the second frequency band gain includes: extracting the second frequency domain features corresponding to each frequency band according to the frequency band information of the target speech spectrum; inputting each of the second frequency domain features and the first frequency band gain into the corresponding sub-neural network in the second neural network model according to a preset feature input rule to obtain the third frequency band gain; and concatenating the third frequency band gains to obtain the second frequency band gain.

2. The speech enhancement method based on neural networks according to claim 1, characterized in that, The step of inputting each of the second frequency domain features and the first frequency band gain into the corresponding sub-neural network in the second neural network model to obtain the third frequency band gain includes: The second frequency domain feature and the first frequency band gain are input into each sub-layer of the corresponding sub-neural network in the second neural network model to obtain the first intermediate parameter corresponding to the sub-layer of the network. The first intermediate parameters output from the previous first network layer are concatenated and then input into the next first network layer to obtain the third frequency band gain.

3. The speech enhancement method based on neural networks according to claim 1, characterized in that, The loss function of the first neural network model is expressed as: in, L Let be the loss function of the first neural network model. Control factors for speech preservation Let the mean squared error loss function be . Preserve loss function for speech. t The time of the target speech. f The frequency of the target speech. T This represents the total number of time points. F The total number of frequency points, For reference, the time-frequency representation of enhanced speech, For time-frequency gain, This is the time-frequency representation of the target speech.

4. The speech enhancement method based on neural networks according to claim 1, characterized in that, The first neural network model includes multiple second network layers; The step of inputting the first frequency domain feature corresponding to the target speech into the first neural network model to obtain the first frequency band gain includes: The first frequency domain feature corresponding to the target speech is input into the second network layer of the first neural network model, and the second intermediate parameter is output. The second intermediate parameters output from the previous second network layer are sequentially input into the next second network layer to obtain the first frequency band gain.

5. The speech enhancement method based on neural networks according to claim 1, characterized in that, The step of calculating the first frequency band gain, the second frequency band gain, and the target speech spectrum to obtain the enhanced speech includes: The first frequency band gain and the target speech spectrum are calculated to obtain the first enhanced speech spectrum; The second frequency band gain is multiplied by the first enhanced speech spectrum in the complex field to obtain the second enhanced speech spectrum; The enhanced speech is obtained by performing an inverse Fourier transform on the spectrum of the second enhanced speech.

6. The speech enhancement method based on neural networks according to claim 5, characterized in that, The step of calculating the first frequency band gain and the target speech spectrum to obtain the first enhanced speech spectrum includes: Linear interpolation is performed on the first frequency band gain to obtain the fourth frequency band gain; The fourth frequency band gain is multiplied by the target speech spectrum to obtain the first enhanced speech spectrum.

7. A speech enhancement device based on a neural network, characterized in that, include: The first processing module is used to input the first frequency domain feature corresponding to the target speech into the first neural network model for processing to obtain the first frequency band gain; The second processing module is used to input the second frequency domain features corresponding to the target speech and the first frequency band gain into a second neural network model for processing to obtain the second frequency band gain; wherein, the second frequency domain features are complex features, and the second frequency band gain includes real part gain and imaginary part gain; the second neural network model includes multiple sub-neural networks; each sub-neural network includes multiple first network layers; each first network layer includes multiple network sub-layers; the step of inputting the second frequency domain features corresponding to the target speech and the first frequency band gain into the second neural network model to obtain the second frequency band gain includes: extracting the second frequency domain features corresponding to each frequency band according to the frequency band information of the target speech spectrum; inputting each second frequency domain feature and the first frequency band gain into the corresponding sub-neural network in the second neural network model according to a preset feature input rule to obtain the third frequency band gain; and concatenating the third frequency band gains to obtain the second frequency band gain; The calculation module is used to perform calculations on the first frequency band gain, the second frequency band gain, and the target speech spectrum to obtain enhanced speech.

8. An electronic device, characterized in that, Includes memory and processor, of which: The processor is used to execute computer programs stored in the memory; When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images