Method, device, electronic device and computer-readable medium for speech noise reduction

Abstract

The embodiments of the present disclosure disclose a method, an apparatus, an electronic device, and a computer-readable medium for speech denoising. A specific implementation of the method includes: obtaining a target speech; preprocessing the target speech to obtain a Mel spectrum of the target speech; inputting the Mel spectrum into a pre-trained feature neural network to obtain an amplitude mask gain coefficient of the Mel spectrum; performing amplitude masking on the Mel spectrum according to the amplitude mask gain coefficient to obtain a denoised Mel spectrum; inputting the denoised Mel spectrum into a pre-trained neural network vocoder to obtain denoised speech. This implementation achieves a more lightweight denoising process for the target speech and obtains a better denoising effect.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the technical field of computer speech signal processing, and in particular to a method, device, electronic device, and computer-readable medium for speech noise reduction. Background Art

[0002] As communication technology matures, the volume of voice / video calls is increasing, and users are using more diverse terminals, such as personal computers, mobile phones, and edge devices such as microphones. However, due to the variety of environments in which users are located, the presence of various background noises can have a significant impact on the quality of voice calls, such as the sound of indoor air conditioner fans, keyboard tapping, and outdoor car sounds, bird calls, and so on.

[0003] In addition, when there are other people in the user's communication environment, the voices of other people will also interfere with the voice call. In order to hear the voice more clearly, it is necessary to perform noise reduction processing on the collected noisy voice to filter out background noise and other people's voices. Summary of the invention

[0004] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0005] Some embodiments of the present disclosure propose methods, devices, electronic devices, and computer-readable media for speech noise reduction to solve the technical problems mentioned in the above background technology section.

[0006] In a first aspect, some embodiments of the present disclosure provide a method for speech denoising, the method comprising: acquiring a target speech; preprocessing the target speech to obtain a Mel spectrum of the target speech; inputting the Mel spectrum into a pre-trained feature neural network to obtain an amplitude mask gain coefficient of the Mel spectrum; performing amplitude masking on the Mel spectrum according to the amplitude mask gain coefficient to obtain a denoised Mel spectrum; inputting the denoised Mel spectrum into a pre-trained neural network vocoder to obtain denoised speech.

[0007] In a second aspect, some embodiments of the present disclosure provide a speech denoising device, comprising: an acquisition unit, configured to acquire a target speech; a preprocessing unit, configured to preprocess the target speech to obtain a Mel spectrum of the target speech; a feature unit, configured to input the Mel spectrum into a pre-trained feature neural network to obtain an amplitude mask gain coefficient of the Mel spectrum; an amplitude masking unit, configured to perform amplitude masking on the Mel spectrum according to the amplitude masking gain coefficient to obtain a denoised Mel spectrum; and a generation unit, configured to input the denoised Mel spectrum into a pre-trained neural network vocoder to obtain denoised speech.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which network device includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation method in the first aspect.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0010] One of the above-mentioned embodiments of the present disclosure has the following beneficial effects: a target speech is acquired, and then the target speech is preprocessed to obtain a Mel spectrum of the target speech, and then the Mel spectrum is input into a pre-trained feature neural network to obtain an amplitude mask gain coefficient of the Mel spectrum, and then the Mel spectrum is amplitude masked according to the amplitude mask gain coefficient to obtain a denoised Mel spectrum, and then the denoised Mel spectrum is input into a pre-trained neural network vocoder to obtain a denoised speech, so that the background noise and other human voice interference are eliminated in the obtained denoised speech by processing the target speech, thereby achieving a lighter-weight denoising process for the target speech and obtaining a better denoising effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0012] Figure 1 is a schematic diagram of an application scenario of a speech noise reduction method according to some embodiments of the present disclosure;

[0013] Figure 2 is a flow chart of some embodiments of the speech noise reduction method according to the present disclosure;

[0014] Figure 3 is a schematic diagram of a characteristic neural network structure according to some embodiments of the present disclosure;

[0015] Figure 4 is a schematic diagram of a self-attention module structure according to some embodiments of the present disclosure;

[0016] Figure 5 is a schematic diagram of a convolution module structure according to some embodiments of the present disclosure;

[0017] Figure 6 is a schematic diagram of the structure of some embodiments of the speech noise reduction device according to the present disclosure;

[0018] Figure 7 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0021] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0022] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0023] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0024] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0025] Figure 1 It is a schematic diagram of an application scenario of the speech noise reduction method according to some embodiments of the present disclosure.

[0026] like Figure 1 As shown, first, the server 101 can obtain the target speech 102, and then pre-process the target speech 102 to obtain the Mel spectrum 103 of the target speech, and then input the Mel spectrum 103 into a pre-trained feature neural network to obtain an amplitude mask gain coefficient 104, and then perform amplitude masking on the Mel spectrum 103 according to the amplitude mask gain coefficient 104 to obtain a denoised Mel spectrum 105, and finally input the denoised Mel spectrum 105 into a pre-trained neural network vocoder to obtain a denoised speech 106.

[0027] It is understandable that the speech noise reduction method can be executed by a terminal device, or it can also be executed by a server 101. The execution subject of the above method can also include a device formed by integrating the above terminal device and the above server 101 through a network, or it can also be executed by various software programs. Among them, the terminal device can be various electronic devices with information processing capabilities, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers and desktop computers, etc. The execution subject can also be embodied as a server 101, software, etc. When the execution subject is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

[0028] It should be understood that Figure 1 The number of servers in the example is only for illustration purposes. Any number of servers may be provided as required.

[0029] Continue to refer Figure 2 , shows a process 200 of some embodiments of the speech noise reduction method according to the present disclosure. The speech noise reduction method comprises the following steps:

[0030] Step 201, obtaining target speech.

[0031] In some embodiments, the execution body of the speech noise reduction method (for example Figure 1 The server shown in the figure) can obtain the target voice through a wired connection or a wireless connection. Here, the target voice usually refers to the voice that needs to be noise-reduced.

[0032] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0033] Step 202: preprocess the target speech to obtain a mel spectrum of the target speech.

[0034] In some embodiments, based on the target speech obtained in step 201, the execution subject (eg Figure 1 The server shown in FIG. 1 may pre-process the target speech to obtain a mel-spectrogram of the target speech. Here, the pre-processing generally refers to a process of converting the target speech into a mel-spectrogram. There are many ways to implement the conversion of the target speech into a mel-spectrogram, which will not be described in detail here.

[0035] In some optional implementations of some embodiments, the execution subject may perform a frame operation on the target speech to obtain the frame-level speech of the target speech. Then, the frame-level speech is windowed and short-time Fourier transformed to obtain an amplitude spectrum of the frame-level speech spectrum. Here, the windowing process usually uses a Hanning window function.

[0036] After that, the amplitude spectrum is processed by a Mel filter to obtain a Mel energy spectrum. The Mel energy spectrum is subjected to energy normalization (Per-Channel Energy Normalization, PCEN) to obtain an energy normalized Mel energy spectrum. Here, the purpose of energy normalization is to obtain the cumulative average of each dimensional feature, and then divide the feature of the current frame by the average value to achieve normalization.

[0037] Then, the above Mel energy spectrum is concatenated with the above normalized Mel energy spectrum to obtain the Mel spectrum of the target speech.

[0038] In some optional implementations of some embodiments, the execution subject may perform energy normalization processing on the Mel energy spectrum according to the following formula: PCEN(t,f)=(E(t,f) / (M(t,f)+ε) α +δ) r -δ r , where PCEN(t, f) represents energy normalization for each frequency band channel; t represents a time domain sequence; f represents a frequency domain sequence; ε represents a constant (for example, ε can be 10 -8 ); α represents the energy normalization coefficient; δ and r represent the dynamic range coefficients; E(t,f) represents the Mel energy spectrum; M(t,f) represents the smoothed energy spectrum, which is determined according to the following formula: M(t,f) = (1-s)·M(t-1,f)+s·E(t,f), where s represents the smoothing coefficient.

[0039] In the preprocessing part, not only the Mel-spectrogram of speech is used, but also the trainable per-channel energy normalization of the Mel-spectrogram is added, automatic gain control and dynamic range control are performed on the Mel-spectrogram, and more past timeline information is added. At the same time, its trainable parameters also better adapt to the denoising neural network, which is beneficial to improving the overall speech denoising effect.

[0040] Step 203: input the Mel spectrum into a pre-trained feature neural network to obtain an amplitude mask gain coefficient of the Mel spectrum.

[0041] In some embodiments, the execution subject may input the mel spectrum into a pre-trained feature neural network to obtain the amplitude mask gain coefficient of the mel spectrum. Here, the feature neural network is usually used to characterize the corresponding relationship between the mel spectrum and the amplitude mask gain coefficient.

[0042] As an example, the above-mentioned feature neural network can be a correspondence table of sample Mel spectrum and sample amplitude mask gain coefficient integrated by researchers based on a large amount of data. The above-mentioned execution entity can determine the sample amplitude mask gain coefficient that is the same or similar to the Mel spectrum in the above-mentioned correspondence table, and determine the sample amplitude mask gain coefficient corresponding to the sample Mel spectrum as the amplitude mask gain coefficient.

[0043] In some optional implementations of some embodiments, the above-mentioned feature neural network includes a sequentially connected convolution module, a self-attention module, a gated recurrent unit module and / or a deconvolution module.

[0044] As an example, the above feature neural network can be as follows Figure 3 The neural network shown is composed of a convolution module 301, a self-attention module 302, a gated recurrent unit module 303 and a deconvolution module 304.

[0045] Specifically, Figure 4 As shown, the convolution module 301 is composed of 5 convolution units with the same structure but different parameters. Each convolution unit is composed of a point convolution layer 401 (PConv), a batch normalization layer 402 (BatchNorm), a linear rectifier function layer 403 (ReLU), a depth convolution layer 404 (DSConv), a batch normalization layer 405 (BatchNorm), and a linear rectifier function layer 406 (ReLU) in sequence. The output of the last ReLU of each convolution unit is sent to the next layer of the network and to the corresponding deconvolution unit of the deconvolution module 407 as partial input. The convolution kernel of the point convolution layer of each convolution unit is 1, and the step size is 1. The convolution kernels of the depth convolution layer of each convolution unit are 5, 3, 5, 3, 5, and the step sizes are 2, 1, 2, 1, 2, respectively.

[0046] The convolution module in the feature neural network is used to extract the compressed Mel spectrum features and reduce the dimension of the features. The convolution units with the same structure and different parameters reduce the loss of feature information while reducing the dimension. The convolution unit of the present invention uses point convolution and depth convolution, which greatly reduces the number of convolution parameters and the amount of computation compared to the one-step standard convolution, and achieves similar effects.

[0047] like Figure 5 As shown, the self-attention module 500 is composed of a multi-head self-attention layer 501 (MultiheadAttention), a layer normalization layer 502 (LayerNorm), a point convolution layer 503 (PConv), a batch normalization layer (BatchNorm) 504, and a linear rectifier function layer 505 (ReLU). The output of the convolution module is first sent to the multi-head self-attention layer, and then after passing through the layer normalization layer, it is added to the output of the convolution module and enters the point convolution layer. After being processed by the batch normalization layer and the linear rectifier function layer, it is sent to the next module.

[0048] The purpose of the self-attention module is to synthesize the Mel spectrum features extracted by the convolution module. The self-attention network is used instead of the recurrent neural network because the self-attention network is more capable of capturing global information, while the recurrent neural network pays more attention to local features and time domain relationships. The 8-head self-attention module used in the present invention has achieved better noise reduction effect with less parameters than the recurrent neural network.

[0049] The gated recurrent unit module consists of a gated recurrent unit layer (GRU), a point convolution layer (PConv), a batch normalization layer (BatchNorm), and a linear rectifier layer (ReLU). The output of the self-attention module is first sent to the gated recurrent unit layer, and then passes through the point convolution layer, batch normalization layer, and linear rectifier layer in sequence.

[0050] The gated recurrent unit is used to process the relationship between each previous and next speech frame and establish the information flow of the time axis. Here, a recurrent neural network is used instead of a self-attention network because the self-attention network cannot process the previous and next relationship of the time axis. For the choice of recurrent neural network, the use of a gated recurrent unit instead of a long short-term memory network can further reduce the amount of network calculation. However, due to the real-time requirements, a unidirectional gated recurrent network must be used here to form a causal system.

[0051] The deconvolution module 304 is composed of 5 deconvolution units with the same structure but different parameters. Each deconvolution unit is composed of a point convolution layer (PConv), a batch normalization layer (BatchNorm), a linear rectifier function layer (ReLU), a deconvolution layer (ConvT), a batch normalization layer (BatchNorm), and a linear rectifier function layer (ReLU) in sequence. The input of the first convolution layer of each deconvolution unit is received from both the upper layer of the network and the last linear rectifier function layer of the corresponding convolution unit of the convolution module, and the two parts are spliced ​​as the overall input. The convolution kernel of the point convolution layer of each deconvolution unit is 1, and the step size is 1. The convolution kernels of the deconvolution layers of each deconvolution unit are 5, 3, 5, 3, 5, and the step sizes are 2, 1, 2, 1, 2, respectively.

[0052] The deconvolution module restores the compressed and reduced mel spectrum to the size of the original mel spectrum to generate a mel spectrum mask. The function of the mel spectrum mask is to multiply the mel spectrum of the original noisy speech to obtain the denoised mel spectrum. The deconvolution module adopts a symmetrical structure with the convolution module and has 5 deconvolution units, which can be regarded as the inverse operation of the convolution module. The input of each deconvolution unit not only receives the output of the previous layer of the network, but also directly receives the output of the corresponding unit of the convolution module, which can make the gradient change more stable during network training and make it easier to train a good network model.

[0053] In some optional implementation methods of some embodiments, the above-mentioned execution entity can obtain a training sample set, wherein the above-mentioned training sample set includes a sample Mel spectrum and a sample amplitude mask gain coefficient corresponding to the above-mentioned sample speech; the above-mentioned sample Mel spectrum is input into the model to be trained to obtain the amplitude mask gain coefficient; the above-mentioned amplitude mask gain coefficient and the above-mentioned sample amplitude mask gain coefficient are compared according to a preset loss function to determine the loss value; in response to the above-mentioned loss value satisfying a preset condition, it is determined that the training of the above-mentioned model to be trained is completed, and the above-mentioned model to be trained is determined as a feature neural network.

[0054] The loss function of the feature neural network calculates the loss between the denoised Mel spectrum and the pure speech Mel spectrum, and uses a weighted multi-scale loss summation to cover a wider range of slices and give a greater weight to long slices. Due to the short-term stability of speech and the non-fixed duration of each pronunciation, this multi-scale loss weighted summation improves the denoising effect after network training and reduces speech distortion compared to a single loss function.

[0055] In some optional implementations of some embodiments, the above-mentioned execution entity may also adjust relevant parameters in the model to be trained in response to the above-mentioned loss value failing to meet preset conditions.

[0056] Step 204 , performing amplitude masking on the mel spectrum according to the amplitude masking gain coefficient to obtain a denoised mel spectrum.

[0057] In some embodiments, the execution subject may perform amplitude masking on the Mel spectrum according to the gain coefficient to obtain a de-noised Mel spectrum. Specifically, the amplitude masking process generally refers to the execution subject multiplying the Mel spectrum with the amplitude masking gain coefficient to obtain a de-noised Mel spectrum.

[0058] Step 205: input the denoised Mel-spectrogram to a pre-trained neural network vocoder to obtain denoised speech.

[0059] In some embodiments, the execution subject may input the denoised Mel spectrum into a pre-trained neural network vocoder to obtain denoised speech. Here, the neural network vocoder is usually used to characterize the correspondence between the denoised Mel spectrum and the denoised speech.

[0060] As an example, the above-mentioned neural network vocoder can be a correspondence table of sample denoised Mel-spectrograms and sample denoised speech integrated by researchers based on a large amount of data. The above-mentioned execution entity can determine the sample denoised Mel-spectrograms that are identical or similar to the denoised Mel-spectrograms in the above-mentioned correspondence table, and determine the sample denoised speech corresponding to the sample denoised Mel-spectrograms as the denoised speech.

[0061] As another example, the neural network vocoder can be based on Multi-band MelGAN technology, consisting of a generator and a discriminator. The generator is used to produce speech, and is mainly composed of convolution, upsampling, residual block, and activation function modules. The discriminator is used to identify the authenticity of speech, and is composed of full-band and self-contained multiple discriminator modules. During training, the generator and the discriminator take turns to cycle for adversarial training.

[0062] The neural network vocoder generates speech from the Mel spectrum. Compared with the traditional vocoder, the network automatically learns to complete the phase information, which greatly improves the quality of the generated speech. Since the noise reduction method of the present invention needs to run in real time, some large neural network vocoders may have better effects, but the generation speed is extremely slow and it is difficult to meet the real-time requirements. The neural network vocoder based on Multi-band MelGAN technology used in the present invention greatly improves the generation speed while ensuring the quality of the generated speech, making the overall speed better than the end-to-end speed.

[0063] The present invention is based on a denoising method based on deep learning. Instead of using the end-to-end denoising method that has been widely used, a two-stage denoising method is proposed. The denoising neural network in the first stage inputs the Mel spectrum and finally generates a Mel spectrum mask. Compared with the end-to-end network directly processing the frequency domain spectrum to generate the spectrum mask, the amount of calculation is greatly reduced by 4 to 8 times, and the Mel spectrum is more in line with human hearing, which is beneficial to improving the auditory effect after denoising. In order to restore the Mel spectrum to time domain speech with high quality, the present invention uses a neural network vocoder in the second stage. The neural network vocoder based on Multi-band MelGAN technology can not only generate high-quality speech, but also greatly improve the generation speed compared to large neural network vocoders. It only takes 3% of the speech time to run on the CPU, which can fully meet the real-time requirements of voice communication noise reduction.

[0064] The overall neural network is lightweight, using only a small number of network parameters and a very small amount of calculation, which is convenient for use on PC or ported to embedded terminals. It has the noise reduction effect that can only be achieved by large networks, and meets the real-time advantages that large networks do not have. It has low hardware resource usage and good noise reduction effect.

[0065] One of the above-mentioned embodiments of the present disclosure has the following beneficial effects: a target speech is acquired, and then the target speech is preprocessed to obtain a Mel spectrum of the target speech, and then the Mel spectrum is input into a pre-trained feature neural network to obtain an amplitude mask gain coefficient of the Mel spectrum, and then the Mel spectrum is amplitude masked according to the amplitude mask gain coefficient to obtain a denoised Mel spectrum, and then the denoised Mel spectrum is input into a pre-trained neural network vocoder to obtain a denoised speech, so that the background noise and other human voice interference are eliminated in the obtained denoised speech by processing the target speech, thereby achieving a lighter-weight denoising process for the target speech and obtaining a better denoising effect.

[0066] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a speech noise reduction device, and these device embodiments are Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0067] like Figure 6As shown, the speech denoising device 600 of some embodiments includes: an acquisition unit 601, a preprocessing unit 602, a feature unit 603, an amplitude mask unit 604 and a generation unit 605. The acquisition unit 601 is configured to acquire a target speech; the preprocessing unit 602 is configured to preprocess the target speech to obtain a Mel spectrum of the target speech; the feature unit 603 is configured to input the Mel spectrum into a pre-trained feature neural network to obtain an amplitude mask gain coefficient of the Mel spectrum; the amplitude mask unit 604 is configured to perform amplitude masking on the Mel spectrum according to the amplitude mask gain coefficient to obtain a denoised Mel spectrum; and the generation unit 605 is configured to input the denoised Mel spectrum into a pre-trained neural network vocoder to obtain denoised speech.

[0068] In some optional implementations of the embodiments, the preprocessing unit is further configured to: perform frame division operation on the target speech to obtain the frame-level speech of the target speech; perform windowing and short-time Fourier transform on the frame-level speech to obtain the amplitude spectrum of the frame-level speech spectrum; use a Mel filter to process the amplitude spectrum to obtain a Mel energy spectrum; perform energy normalization on the Mel energy spectrum to obtain an energy normalized Mel energy spectrum; and concatenate the Mel energy spectrum with the normalized Mel energy spectrum to obtain the Mel spectrum of the target speech.

[0069] In an optional implementation of some embodiments, the preprocessing unit is further configured to: perform energy normalization processing on the above Mel energy spectrum according to the following formula: PCEN(t,f)=(E(t,f) / (M(t,f)+ε)α+δ) r -δ r , where PCEN(t,f) represents energy normalization for each frequency band channel; t represents a time domain sequence; f represents a frequency domain sequence; ε represents a constant; α represents an energy normalization coefficient; δ and r represent dynamic range coefficients; E(t,f) represents Mel energy spectrum; M(t,f) represents a smoothed energy spectrum, which is determined according to the following formula: M(t,f) = (1-s)·M(t-1,f)+s·E(t,f), where s represents a smoothing coefficient.

[0070] In an optional implementation of some embodiments, the above-mentioned feature neural network includes a sequentially connected convolution module, a self-attention module, a gated recurrent unit module and / or a deconvolution module.

[0071] In an optional implementation of some embodiments, the feature neural network is trained according to the following steps: obtaining a training sample set, wherein the training sample set includes a sample Mel spectrum and a sample amplitude mask gain coefficient corresponding to the sample speech; inputting the sample Mel spectrum into the model to be trained to obtain the amplitude mask gain coefficient; comparing the amplitude mask gain coefficient with the sample amplitude mask gain coefficient according to a preset loss function to determine a loss value; in response to the loss value satisfying a preset condition, determining that the training of the model to be trained is completed, and determining the model to be trained as a feature neural network.

[0072] In an optional implementation of some embodiments, the speech noise reduction device further includes an adjustment unit configured to: in response to the loss value not satisfying a preset condition, adjust relevant parameters in the model to be trained.

[0073] It is understood that the units described in the device 600 are similar to those described in the reference Figure 2 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 600 and the units included therein, and will not be described in detail here.

[0074] One of the above-mentioned embodiments of the present disclosure has the following beneficial effects: a target speech is acquired, and then the target speech is preprocessed to obtain a Mel spectrum of the target speech, and then the Mel spectrum is input into a pre-trained feature neural network to obtain an amplitude mask gain coefficient of the Mel spectrum, and then the Mel spectrum is amplitude masked according to the amplitude mask gain coefficient to obtain a denoised Mel spectrum, and then the denoised Mel spectrum is input into a pre-trained neural network vocoder to obtain a denoised speech, so that the background noise and other human voice interference are eliminated in the obtained denoised speech by processing the target speech, thereby achieving a lighter-weight denoising process for the target speech and obtaining a better denoising effect.

[0075] Reference below Figure 7 , which shows an electronic device (eg, Figure 1 A schematic diagram of the structure of the server 700. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0076] like Figure 7As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0077] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0078] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0079] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0080] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0081] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains the target speech; pre-processes the target speech to obtain the Mel spectrum of the target speech; inputs the Mel spectrum into a pre-trained feature neural network to obtain the amplitude mask gain coefficient of the Mel spectrum; performs amplitude masking on the Mel spectrum according to the amplitude mask gain coefficient to obtain a denoised Mel spectrum; inputs the denoised Mel spectrum into a pre-trained neural network vocoder to obtain denoised speech.

[0082] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0083] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0084] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The units described may also be provided in a processor, for example, may be described as: a processor including an acquisition unit, a preprocessing unit, a feature unit, an amplitude mask unit, and a generation unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, a receiving unit may also be described as a "unit for acquiring target speech".

[0085] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0086] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A method for speech noise reduction, comprising: Get the target voice; Preprocessing the target speech to obtain a mel spectrum of the target speech; Inputting the Mel spectrum into a pre-trained feature neural network to obtain an amplitude mask gain coefficient of the Mel spectrum; Performing amplitude masking on the Mel spectrum according to the amplitude mask gain coefficient to obtain a denoised Mel spectrum, including: multiplying the Mel spectrum by the amplitude mask gain coefficient to obtain a denoised Mel spectrum; Inputting the denoised Mel-spectrogram into a pre-trained neural network vocoder to obtain denoised speech; The characteristic neural network is trained according to the following steps: Acquire a training sample set, wherein the training sample set includes a sample Mel spectrum and a sample amplitude mask gain coefficient corresponding to the sample speech; Inputting the sample Mel spectrum into the model to be trained to obtain the amplitude mask gain coefficient; Comparing the amplitude mask gain coefficient and the sample amplitude mask gain coefficient according to a preset loss function to determine a loss value; In response to the loss value satisfying a preset condition, it is determined that the training of the model to be trained is completed, and the model to be trained is determined to be a feature neural network.

2. The method according to claim 1, wherein: The preprocessing of the target speech to obtain a mel spectrum of the target speech includes: Performing a frame operation on the target speech to obtain frame-level speech of the target speech; Performing windowing and short-time Fourier transform on the frame-level speech to obtain an amplitude spectrum of the frame-level speech spectrum; Processing the amplitude spectrum using a Mel filter to obtain a Mel energy spectrum; Performing energy normalization processing on the Mel energy spectrum to obtain an energy normalized Mel energy spectrum; The Mel energy spectrum is concatenated with the normalized Mel energy spectrum to obtain the Mel spectrum of the target speech.

3. The method according to claim 2, wherein: The step of performing energy normalization processing on the Mel energy spectrum to obtain an energy normalized Mel energy spectrum includes: The Mel energy spectrum is energy normalized according to the following formula: PCEN(t,f)=(E(t,f) / (M(t,f)+ε) α +d) r -d r , Among them, P CEN (t, f) represents energy normalization for each frequency band channel; t represents the time domain series; f represents the frequency domain sequence; ε represents a constant; α represents the energy normalization coefficient; δ and r represent the dynamic range coefficient; E(t,f) represents the Mel energy spectrum; M(t,f) represents the smoothed energy spectrum, which is determined by the following formula: M(t,f)=(1-s)·M(t-1,f)+s·E(t,f), Where s represents the smoothing coefficient.

4. The method according to claim 1, wherein: The feature neural network includes a sequentially connected convolution module, a self-attention module, a gated recurrent unit module and / or a deconvolution module.

5. The method according to claim 1, wherein: The method further comprises: In response to the loss value not satisfying a preset condition, relevant parameters in the model to be trained are adjusted.

6. A device for speech noise reduction, comprising: An acquisition unit, configured to acquire a target speech; A preprocessing unit, configured to preprocess the target speech to obtain a mel spectrum of the target speech; The feature unit is configured to input the Mel spectrum into a pre-trained feature neural network to obtain an amplitude mask gain coefficient of the Mel spectrum; wherein the feature neural network is trained according to the following steps: Acquire a training sample set, wherein the training sample set includes a sample Mel spectrum and a sample amplitude mask gain coefficient corresponding to the sample speech; Inputting the sample Mel spectrum into the model to be trained to obtain the amplitude mask gain coefficient; Comparing the amplitude mask gain coefficient and the sample amplitude mask gain coefficient according to a preset loss function to determine a loss value; In response to the loss value satisfying a preset condition, determining that the training of the model to be trained is completed, and determining the model to be trained as a feature neural network; The amplitude masking unit is configured to perform amplitude masking on the Mel spectrum according to the amplitude masking gain coefficient to obtain a denoised Mel spectrum, including: multiplying the Mel spectrum by the amplitude masking gain coefficient to obtain a denoised Mel spectrum; The generating unit is configured to input the denoised Mel-spectrogram into a pre-trained neural network vocoder to obtain denoised speech.

7. The device according to claim 6, wherein: The feature neural network includes a sequentially connected convolution module, a self-attention module, a gated recurrent unit module and / or a deconvolution module.

8. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

9. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images