Voice noise reduction method, system and device based on time sequence convolutional network, and medium

Abstract

The invention discloses a voice noise reduction method, system and device based on a time sequence convolutional network, and a medium, and the method comprises the steps: carrying out the preprocessing of an original voice signal, and dividing the original voice signal into a plurality of time frames; determining amplitude spectrums, complex spectrums and phases of different frequency components; carrying out noise estimation on the amplitude spectrum, and carrying out initial noise suppression on the original voice to obtain an amplitude spectrum mask subjected to initial noise reduction; according to the amplitude spectrum mask and the phase of the preliminary noise reduction, obtaining preliminary noise reduction complex spectrums of different frequency components; estimating the complex spectrum of the first frequency component original voice and the complex spectrum noise of the first frequency component preliminary noise reduction to obtain a first complex spectrum of the first frequency component; and adding the first complex spectrum of the first frequency component and the complex spectrum of the first frequency component subjected to preliminary noise reduction, and then splicing with the complex spectrum of the second frequency component subjected to preliminary noise reduction to obtain a complex spectrum subjected to noise reduction. According to the method, the calculation amount of the noise reduction model can be reduced, and the noise reduction effect under the complex condition and the robustness under the sudden change scene are improved.

Description

Technical Field

[0001] The present application relates to the technical field of speech noise reduction, and in particular to a speech noise reduction method, system, device and medium based on a temporal convolutional network. Background Art

[0002] In the fields of voice communication, audio processing and speech recognition technology, speech noise reduction technology occupies a pivotal position. The existing noise reduction technology based on amplitude spectrum and complex spectrum captures voice signals through voice transmission and receiving devices, and uses a two-stage deep learning model to suppress noise on the signal. Combining the advantages of amplitude spectrum and complex spectrum, it can improve the noise reduction effect and voice restoration to a certain extent. However, although this method has improved the noise reduction performance, the computational complexity of its deep learning model has also increased, which undoubtedly limits the wide application of this method on some low-resource devices. For example, on digital signal processors (DSPs), headphones, IP phones and other devices, due to hardware resource limitations, traditional deep learning models are often difficult to deploy. Secondly, the existing two-stage noise reduction technology of amplitude spectrum and complex spectrum mostly uses GRU (Gated Recurrent Unit) operators in time series modeling. The GRU operator relies on too long historical information in the modeling process, and is not robust enough for sudden noise, large and small voice conversion, and noise voice conversion. Problems such as false cancellation and over-suppression often occur. Summary of the invention

[0003] The main purpose of this application is to overcome the shortcomings and deficiencies of the prior art and to provide a speech denoising method, system, device and medium based on a temporal convolutional network, which can reduce the amount of computation of the denoising model and improve the denoising effect of the denoising model under complex conditions and the robustness of the denoising model in sudden change scenarios.

[0004] In order to achieve the above objectives, this application adopts the following technical solutions:

[0005] In a first aspect, the present application provides a speech noise reduction method based on a temporal convolutional network, comprising the following steps:

[0006] Obtaining the original speech signal;

[0007] Preprocessing the original speech signal to obtain a preprocessed speech signal;

[0008] Dividing the preprocessed speech signal according to a preset time interval to obtain a plurality of short time frames;

[0009] Decomposing the multiple short-time frames into amplitude spectra of different frequency components, complex spectra of different frequency components and phase information of different frequency components by using short-time Fourier transform; wherein the different frequency components include a first frequency component and a second frequency component;

[0010] Performing noise estimation on the amplitude spectra of the different frequency components to obtain amplitude spectrum masks of the different frequency components;

[0011] Applying the amplitude spectrum masks of the different frequency components to the original speech signal, performing preliminary noise suppression on the original speech signal, and obtaining preliminary noise-reduced amplitude spectrum masks of the different frequency components;

[0012] Obtaining preliminary denoised complex spectra of different frequency components according to the preliminary denoised amplitude spectrum masks of the different frequency components and the phase information;

[0013] Noise estimation is performed based on the complex spectrum of the original speech of the first frequency component and the complex spectrum of the first frequency component after preliminary noise reduction, to obtain an estimated first complex spectrum of the first frequency component;

[0014] Adding the estimated first complex spectrum of the first frequency component to the preliminarily denoised complex spectrum of the first frequency component to obtain a second complex spectrum of the first frequency component;

[0015] splicing the second complex spectrum of the first frequency component and the complex spectrum of the second frequency component after preliminary noise reduction to obtain a complete complex spectrum of the original speech signal after noise reduction;

[0016] The complete denoised complex spectrum of the original speech signal is converted into a time domain signal output through short-time inverse Fourier transform.

[0017] As a preferred technical solution, the first frequency component includes 0-4KHZ;

[0018] The second frequency component includes 4-8KHZ.

[0019] As a preferred technical solution, the noise estimation is performed on the amplitude spectrum of the different frequency components to obtain the amplitude spectrum mask of the different frequency components, including:

[0020] The amplitude spectrum of the non-DC part of the amplitude spectrum of different frequency components is used as the input of the sub-band amplitude spectrum dimensionality reduction operation, and the reduced amplitude spectrum is processed by an N-layer amplitude spectrum encoder, an M-layer amplitude spectrum temporal convolution network and an N-layer amplitude spectrum decoder to obtain an amplitude spectrum mask of the same size as the original amplitude spectrum;

[0021] The input of the amplitude spectrum decoder is the residual sum of the output of the amplitude spectrum decoder of the next layer and the output of the amplitude spectrum encoder of the layer corresponding to the amplitude spectrum decoder of the current layer.

[0022] As a preferred technical solution, the frequency-band amplitude spectrum dimensionality reduction operation includes:

[0023] The 1x256 amplitude spectrum is divided into 2 kHz parts and converted into a 4x64 amplitude spectrum.

[0024] As a preferred technical solution, the noise estimation is performed according to the complex spectrum of the original speech of the first frequency component and the complex spectrum of the first frequency component after preliminary noise reduction to obtain the estimated first complex spectrum of the first frequency component, including:

[0025] The complex spectrum of the original speech of the first frequency component and the complex spectrum of the first frequency component after preliminary noise reduction are concatenated, and the concatenated result is used as the input of the encoder of the N-layer complex spectrum, and then processed by the temporal convolution network of the M-layer complex spectrum and the decoder of the N-layer complex spectrum in sequence to obtain the first complex spectrum of the first frequency component having the same size as the input;

[0026] The input of the complex spectrum decoder is the residual sum of the output of the decoder of the next layer of complex spectrum and the output of the encoder of the complex spectrum of the layer corresponding to the decoder of the current layer of complex spectrum.

[0027] As a preferred technical solution, N=4, M=1.

[0028] As a preferred technical solution, the temporal convolutional network includes point convolution, dilated depth convolution, LayerNrom layer and nonlinear activation layer.

[0029] In the second aspect, the present application provides a speech denoising system based on a time series convolutional network, which is applied to the speech denoising method based on a time series convolutional network, including a speech signal acquisition module, a preprocessing module, a division module, a decomposition module, a first noise estimation module, a noise suppression module, a denoising complex spectrum acquisition module, a second noise estimation module, a complex spectrum addition module, a denoising module and a denoising output module;

[0030] The voice signal acquisition module is used to acquire the original voice signal;

[0031] The preprocessing module is used to preprocess the original speech signal to obtain a preprocessed speech signal;

[0032] The division module is used to divide the preprocessed speech signal according to a preset time interval to obtain a plurality of short time frames;

[0033] The decomposition module is used to decompose the multiple short-time frames into amplitude spectra of different frequency components, complex spectra of different frequency components and phase information of different frequency components by short-time Fourier transform; wherein the different frequency components include a first frequency component and a second frequency component;

[0034] The first noise estimation module is used to perform noise estimation on the amplitude spectrum of the different frequency components to obtain amplitude spectrum masks of the different frequency components;

[0035] The noise suppression module is used to apply the amplitude spectrum masks of the different frequency components to the original speech signal, perform preliminary noise suppression on the original speech signal, and obtain the amplitude spectrum masks of the different frequency components for preliminary noise reduction;

[0036] The module for obtaining the denoised complex spectrum is used to obtain the preliminary denoised complex spectrum of different frequency components according to the preliminary denoised amplitude spectrum mask of the different frequency components and the phase information;

[0037] The second noise estimation module is used to perform noise estimation based on the complex spectrum of the original speech of the first frequency component and the complex spectrum of the first frequency component after preliminary noise reduction, so as to obtain an estimated first complex spectrum of the first frequency component;

[0038] The complex spectrum adding module is used to add the estimated first complex spectrum of the first frequency component and the complex spectrum of the first frequency component after preliminary noise reduction to obtain a second complex spectrum of the first frequency component;

[0039] The noise reduction module is used to splice the second complex spectrum of the first frequency component with the complex spectrum of the second frequency component after preliminary noise reduction to obtain a complete complex spectrum of the original speech signal after noise reduction;

[0040] The noise reduction output module is used to convert the complete noise-reduced complex spectrum of the original speech signal into a time domain signal output through short-time inverse Fourier transform.

[0041] In a third aspect, the present application provides an electronic device, the electronic device comprising:

[0042] at least one processor; and a memory communicatively coupled to the at least one processor;

[0043] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the speech denoising method based on the temporal convolutional network.

[0044] In a fourth aspect, the present application provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the speech denoising method based on a temporal convolutional network.

[0045] In summary, compared with the prior art, the effective effects brought about by the technical solution provided by this application include at least:

[0046] The present application proposes a speech denoising method based on a temporal convolutional network. Firstly, by adopting a denoising model structure with a two-stage operation mode of amplitude spectrum and complex spectrum, the denoising effect of the model under complex conditions is significantly improved, and the restoration ability and sound quality of human voice are significantly improved; then, by adopting the TCN (temporal convolutional network) operator for temporal modeling, the robustness of the model in mutation scenarios is significantly improved; secondly, by adopting the frequency band division of the input amplitude spectrum of the first stage, and the modeling of only the part below 4kHz of the complex spectrum of the second stage, the calculation amount of the model is effectively reduced, so that the model can run in real time on a platform with lower performance, and the whole system can be flexibly deployed in real-time conference communication equipment to better handle various complex environmental noises, thereby improving the application effect of denoised speech in voice communication, audio processing and speech recognition and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 A flowchart of a speech noise reduction method based on a temporal convolutional network provided in one embodiment of the present application;

[0049] Figure 2 A schematic diagram of the overall process of a speech noise reduction method based on a temporal convolutional network provided in one embodiment of the present application;

[0050] Figure 3 A block diagram of a speech noise reduction system based on a temporal convolutional network provided for one embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0052] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0053] Example:

[0054] See also Figure 1 and Figure 2 In one embodiment of the present application, a speech noise reduction method based on a temporal convolutional network is provided, comprising the following steps:

[0055] S1. Obtaining the original speech signal.

[0056] Furthermore, the voice signal of the present application is obtained from an acquisition unit, which may be composed of a microphone or a recording device and an ADC hardware chip. First, the original voice signal is collected from a voice source such as a microphone or a recording device, and then an ADC (analog-to-digital converter) hardware chip is used to convert the voice analog signal in the environment into a digital signal.

[0057] S2. Preprocess the original speech signal to obtain a preprocessed speech signal.

[0058] Furthermore, some preprocessing operations are performed on the original speech signal, and the preprocessing operations include removing DC offset, adjusting the sampling rate, etc., to ensure that the signal is within an appropriate range.

[0059] S3. Divide the preprocessed speech signal according to preset time intervals to obtain multiple short-time frames.

[0060] The preprocessed speech signal is divided according to preset time intervals to obtain multiple short-time frames, which is beneficial to subsequent speech signal processing.

[0061] S4. Decomposing the multiple short-time frames into amplitude spectra of different frequency components, complex spectra of different frequency components and phase information of different frequency components by using short-time Fourier transform; wherein the different frequency components include a first frequency component and a second frequency component.

[0062] Furthermore, a short-time Fourier transform (STFT) is used to decompose the multiple short-time frames into amplitude spectra of different frequency components, complex spectra of different frequency components and phase information of different frequency components, and further form amplitude spectra and complex spectra.

[0063] The first frequency component includes a 0-4KHZ part; the second frequency component includes a 4-8KHZ part.

[0064] S5. Perform noise estimation on the amplitude spectra of the different frequency components to obtain amplitude spectrum masks of the different frequency components.

[0065] Further, after short-time Fourier transform processing, the frequency domain amplitude spectrum of the noisy speech is extracted, wherein the size of the non-DC part is FFT_n / 2, wherein FFT_n represents the number of Fourier transform points of the short-time Fourier transform, and in the present application, 512 is preferably used as the value; subsequently, the amplitude spectrum of the non-DC part of the amplitude spectrum of different frequency components is used as the input of the first-stage Unet model structure (denoising model). Among them, the non-DC part refers to all frequency components except the DC component (component with a frequency of 0).

[0066] In the first stage of the Unet model structure, the amplitude spectrum of different frequency components is estimated by noise to obtain the amplitude spectrum mask of different frequency components. The steps include:

[0067] First, the amplitude spectrum of the non-DC part of the amplitude spectrum of different frequency components is subjected to a band-by-band amplitude spectrum dimensionality reduction operation. Specifically, the 1x256 amplitude spectrum is divided into 4x64 amplitude spectrums at every 2 kHz. The band-by-band dimensionality reduction operation helps to reduce the amount of data while retaining important frequency information.

[0068] Secondly, the reduced amplitude spectrum is processed by an N-layer amplitude spectrum encoder Encoder (Encoder1...Encoder4), an M-layer amplitude spectrum temporal convolutional network and an N-layer amplitude spectrum decoder Decoder (Decoder1...Decoder4), and finally an amplitude spectrum mask of the same size as the original amplitude spectrum is obtained; in the present application, N=4, M=1; the number of output channels of the amplitude spectrum encoder Encoder1 to the encoder Encoder4 is preferably 32; the number of output channels of the amplitude spectrum decoder Decoder1 and the decoder Decoder4 is preferably 32; the number of CNN channels of the amplitude spectrum TCN (temporal convolutional network) is preferably 32, and a total of 6 layers are stacked, that is, 6 TCNs are included, each TCN includes point convolution, dilated depth convolution, LayerNrom layer and nonlinear activation layer, and the 6 TCNs include 6 dilated depth convolutions, where the expansion rates of the 6 dilated depth convolutions are 1, 2, 4, 8, 16, and 32, respectively.

[0069] Furthermore, in this application, for the decoder of the amplitude spectrum i , whose input is defined as the decoder of the next level amplitude spectrumi+1 The output of the decoder of the current layer's amplitude spectrum corresponds to the encoder of the layer's amplitude spectrum. i+1 If i=4, the input of Decoder4 of the amplitude spectrum is defined as the residual sum of the output of the amplitude temporal convolutional network and the output of Encoder4 of the amplitude spectrum of the layer corresponding to the decoder of the amplitude spectrum of the current layer.

[0070] S6. Apply the amplitude spectrum masks of the different frequency components to the original speech signal, perform preliminary noise suppression on the original speech signal, and obtain preliminary noise-reduced amplitude spectrum masks of the different frequency components.

[0071] S7. Obtain preliminary denoised complex spectra of different frequency components according to the preliminary denoised amplitude spectrum masks of the different frequency components and the phase information.

[0072] S8. Perform noise estimation based on the complex spectrum of the original speech of the first frequency component and the complex spectrum of the first frequency component after preliminary noise reduction to obtain an estimated first complex spectrum of the first frequency component.

[0073] Further, in the second stage, in the Unet model structure of the second stage, the complex spectrum of the original speech of the first frequency component (0-4KHZ part) and the complex spectrum of the first frequency component (0-4KHZ part) with preliminary noise reduction are intercepted and spliced ​​as the input of the Unet model structure of the second stage; then, they are processed by the N-layer complex spectrum encoder Decoder, the M-layer complex spectrum temporal convolution network and the N-layer complex spectrum decoder Encoder in sequence to obtain the first complex spectrum of the first frequency component with the same size as the input. Where N=4, M=1. The number of output channels of the complex spectrum encoder Encoder1 to the complex spectrum encoder Encoder4 is preferably 32; the number of output channels of the complex spectrum decoder Decoder1 and the complex spectrum decoder Decoder4 is preferably 32; the number of channels of the convolutional neural network CNN of the complex spectrum temporal convolution network TCN is preferably 32, with a total of 6 layers stacked, that is, 6 TCNs, each TCN includes point convolution, dilated depth convolution, LayerNrom layer and nonlinear activation layer, and the 6 TCNs include 6 dilated depth convolutions, where the expansion rates of the 6 dilated depth convolutions are 1, 2, 4, 8, 16, and 32, respectively.

[0074] Furthermore, in this application, for the decoder of the complex spectrum i , whose input is defined as the decoder of the next layer of complex spectrum i+1 The output of the complex spectrum decoder of the current layer corresponds to the complex spectrum encoder Encoder of the corresponding layer i+1If i=4, the input of the complex spectrum Decoder4 is defined as the residual sum of the output of the complex temporal convolutional network and the output of the complex spectrum encoder Encoder4 of the layer corresponding to the current layer complex spectrum decoder.

[0075] Specifically, in the present application, the temporal convolution network of the amplitude spectrum in the first stage and the temporal convolution network of the complex spectrum in the second stage are composed of point convolution, dilated depth convolution, LayerNrom layer and nonlinear activation layer; wherein, the number of input channels and output channels of the point convolution are consistent with the output channels of the previous layer, and the number of groups is 1; and the number of input channels, output channels and groupings of the dilated depth convolution are consistent with the number of output channels of the point convolution, and the expansion rates of the dilated depth convolution are 1, 2, 4, 8, 16, and 32, respectively.

[0076] S9. Add the estimated first complex spectrum of the first frequency component and the complex spectrum of the first frequency component after preliminary noise reduction to obtain a second complex spectrum of the first frequency component.

[0077] Furthermore, the estimated first complex spectrum of the first frequency component is added to the complex spectrum of the first frequency component after preliminary noise reduction, and the amplitude and phase of this part are optimized and adjusted at the same time to obtain the second complex spectrum of the first frequency component (0-4KHZ) to improve the noise suppression effect and the degree of restoration of human voice.

[0078] S10. Concatenate the second complex spectrum of the first frequency component and the complex spectrum of the second frequency component after preliminary noise reduction to obtain a complete complex spectrum of the original speech signal after noise reduction.

[0079] The second complex spectrum of the first frequency component (0-4KHZ) is spliced ​​with the complex spectrum of the second frequency component (4-8KHZ) after preliminary noise reduction to obtain a complete noise-reduced complex spectrum of 0-8kHz (i.e., the complete noise-reduced complex spectrum of the original speech signal).

[0080] S11, converting the complete denoised complex spectrum of the original speech signal into a time domain signal output through inverse short-time Fourier transform.

[0081] The complete denoised complex spectrum of the original speech signal is converted into denoised speech in the time domain through inverse short-time Fourier transform (ISTFT).

[0082] Specifically, the voice output signal after noise reduction will be played through the terminal unit, and the terminal unit can be composed of any real-time conference communication equipment with network access.

[0083] In the first stage of the Unet model of the present application, the amplitude spectrum of the original speech after frequency band reduction is used for noise estimation to obtain an amplitude spectrum mask, and the amplitude spectrum mask is applied to the original speech to perform preliminary noise suppression on the original speech, and then converted into the complex spectrum of the speech after preliminary noise reduction. Then, in the second stage of the Unet model, the complex spectrum of the original speech and the 0-4kHz part of the complex spectrum of the first stage preliminary noise reduction are used for noise estimation to obtain an estimated 0-4kHz partial complex spectrum, and the estimated 0-4kHz part of the complex spectrum is added to the 0-4kHz part of the complex spectrum of the speech after preliminary noise reduction in the first stage, and the amplitude and phase of this part are optimized and adjusted at the same time to improve the noise suppression effect and the degree of restoration of the human voice, and then the complex spectrum of the 4-8kHz part of the speech after preliminary noise reduction in the first stage is spliced ​​to obtain the complete 0-8kHz complex spectrum after noise reduction, and finally output through the iSTFT time domain signal. The calculation can be performed by a computing unit composed of one unit or a computer chip with an operating system, which is used for the calculation of the noise reduction model in the above two stages; the output signal can be transmitted by a data transmission unit composed of a network system that can transmit data. The present application improves the noise reduction effect and robustness of the noise reduction model. The present application has higher adaptability at lower performance consumption, can be better deployed on low-resource devices, and can better handle various complex environmental noises, thereby improving the application effect of noise-reduced speech in voice communication, audio processing, and speech recognition.

[0084] It should be noted that, for the sake of convenience of description, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously.

[0085] Based on the same idea as the speech denoising method based on the time series convolutional network in the above-mentioned embodiment, the present application also provides a speech denoising system based on the time series convolutional network, which can be used to execute the above-mentioned speech denoising method based on the time series convolutional network. For ease of explanation, the structural schematic diagram of the embodiment of the speech denoising system based on the time series convolutional network only shows the parts related to the embodiment of the present application. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than the illustrated structure, or combine certain components, or arrange the components differently.

[0086] See also Figure 3In another embodiment of the present application, a speech denoising system based on a temporal convolutional network is provided, the system comprising a speech signal acquisition module 101, a preprocessing module 102, a division module 103, a decomposition module 104, a first noise estimation module 105, a noise suppression module 106, a denoising complex spectrum acquisition module 107, a second noise estimation module 108, a complex spectrum addition module 109, a denoising module 110 and a denoising output module 111;

[0087] The voice signal acquisition module 101 is used to acquire the original voice signal;

[0088] The preprocessing module 102 is used to preprocess the original speech signal to obtain a preprocessed speech signal;

[0089] The division module 103 is used to divide the pre-processed speech signal according to a preset time interval to obtain a plurality of short time frames;

[0090] The decomposition module 104 is used to decompose the multiple short-time frames into amplitude spectra of different frequency components, complex spectra of different frequency components and phase information of different frequency components by short-time Fourier transform; wherein the different frequency components include a first frequency component and a second frequency component;

[0091] The first noise estimation module 105 is used to perform noise estimation on the amplitude spectrum of the different frequency components to obtain amplitude spectrum masks of the different frequency components;

[0092] The noise suppression module 106 is used to apply the amplitude spectrum masks of the different frequency components to the original speech signal, perform preliminary noise suppression on the original speech signal, and obtain the amplitude spectrum masks of the different frequency components for preliminary noise reduction;

[0093] The module 107 for obtaining the denoised complex spectrum is used to obtain the preliminary denoised complex spectrum of the different frequency components according to the preliminary denoised amplitude spectrum mask of the different frequency components and the phase information;

[0094] The second noise estimation module 108 is used to perform noise estimation based on the complex spectrum of the original speech of the first frequency component and the complex spectrum of the first frequency component after preliminary noise reduction, so as to obtain an estimated first complex spectrum of the first frequency component;

[0095] The complex spectrum adding module 109 is used to add the estimated first complex spectrum of the first frequency component and the complex spectrum of the first frequency component after preliminary noise reduction to obtain a second complex spectrum of the first frequency component;

[0096] The noise reduction module 110 is used to splice the second complex spectrum of the first frequency component with the complex spectrum of the second frequency component after preliminary noise reduction to obtain a complete complex spectrum of the original speech signal after noise reduction;

[0097] The noise reduction output module 111 is used to convert the complete noise-reduced complex spectrum of the original speech signal into a time domain signal output through short-time inverse Fourier transform.

[0098] It should be noted that the speech denoising system based on the time series convolutional network of the present application corresponds one-to-one to the speech denoising method based on the time series convolutional network of the present application. The technical features and beneficial effects described in the above-mentioned embodiment of the speech denoising method based on the time series convolutional network are applicable to the embodiment of the speech denoising system based on the time series convolutional network. For specific contents, please refer to the description in the embodiment of the method of the present application. It will not be repeated here. This is hereby declared.

[0099] In addition, in the implementation of the speech noise reduction system based on the temporal convolutional network in the above-mentioned embodiment, the logical division of each program module is only an example. In actual applications, the above-mentioned functions can be assigned to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the speech noise reduction system based on the temporal convolutional network is divided into different program modules to complete all or part of the functions described above.

[0100] In another embodiment, an electronic device for implementing a speech noise reduction method based on a temporal convolutional network is provided, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor; when the processor executes the computer program, the speech noise reduction method based on a temporal convolutional network of any embodiment of the present application is implemented.

[0101] Exemplarily, in this embodiment, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present application. The one or more module elements may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the device.

[0102] The device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The device may include, but is not limited to, a processor and a memory.

[0103] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the device, and uses various interfaces and lines to connect various parts of the entire device;

[0104] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; in addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0105] Accordingly, the present application also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the speech noise reduction method based on the temporal convolutional network as described in any of the above embodiments.

[0106] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0107] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] The above embodiments are preferred implementation modes of the present application, but the implementation modes of the present application are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present application should be equivalent replacement methods and are included in the protection scope of the present application.

Claims

1. A speech denoising method based on a temporal convolutional network, characterized in that: The steps include: Obtaining the original speech signal; Preprocessing the original speech signal to obtain a preprocessed speech signal; Dividing the preprocessed speech signal according to a preset time interval to obtain a plurality of short time frames; Decomposing the multiple short-time frames into amplitude spectra of different frequency components, complex spectra of different frequency components and phase information of different frequency components by using short-time Fourier transform; wherein the different frequency components include a first frequency component and a second frequency component; Performing noise estimation on the amplitude spectra of the different frequency components to obtain amplitude spectrum masks of the different frequency components; Applying the amplitude spectrum masks of the different frequency components to the original speech signal, performing preliminary noise suppression on the original speech signal, and obtaining preliminary noise-reduced amplitude spectrum masks of the different frequency components; Obtaining complex spectra of the preliminary denoising of the different frequency components according to the amplitude spectrum masks of the preliminary denoising of the different frequency components and the phase information; Perform noise estimation based on the complex spectrum of the original speech of the first frequency component and the complex spectrum of the first frequency component after preliminary noise reduction to obtain an estimated first complex spectrum of the first frequency component; Adding the estimated first complex spectrum of the first frequency component and the preliminarily denoised complex spectrum of the first frequency component to obtain a second complex spectrum of the first frequency component; splicing the second complex spectrum of the first frequency component and the complex spectrum of the second frequency component after preliminary noise reduction to obtain a complete complex spectrum of the original speech signal after noise reduction; The complete denoised complex spectrum of the original speech signal is converted into a time domain signal output through short-time inverse Fourier transform.

2. According to claim 1, a speech noise reduction method based on a temporal convolutional network is characterized in that: The first frequency component includes 0-4KHZ; The second frequency component includes 4-8KHZ.

3. The speech denoising method based on a temporal convolutional network according to claim 1, characterized in that: The step of performing noise estimation on the amplitude spectra of the different frequency components to obtain amplitude spectrum masks of the different frequency components includes: The amplitude spectrum of the non-DC part of the amplitude spectrum of different frequency components is used as the input of the sub-band amplitude spectrum dimensionality reduction operation, and the reduced amplitude spectrum is processed by an N-layer amplitude spectrum encoder, an M-layer amplitude spectrum temporal convolution network and an N-layer amplitude spectrum decoder to obtain an amplitude spectrum mask of the same size as the original amplitude spectrum; The input of the amplitude spectrum decoder is the residual sum of the output of the amplitude spectrum decoder of the next layer and the output of the amplitude spectrum encoder of the layer corresponding to the amplitude spectrum decoder of the current layer.

4. The speech denoising method based on a temporal convolutional network according to claim 3 is characterized in that: The frequency-band amplitude spectrum dimension reduction operation comprises: The 1x256 amplitude spectrum is divided into 2 kHz parts and converted into a 4x64 amplitude spectrum.

5. The speech denoising method based on a temporal convolutional network according to claim 1, characterized in that: The step of performing noise estimation according to the complex spectrum of the original speech of the first frequency component and the complex spectrum of the first frequency component after preliminary noise reduction to obtain an estimated first complex spectrum of the first frequency component includes: The complex spectrum of the original speech of the first frequency component and the complex spectrum of the first frequency component after preliminary noise reduction are concatenated, and the concatenated result is used as the input of the encoder of the N-layer complex spectrum, and then processed by the temporal convolution network of the M-layer complex spectrum and the decoder of the N-layer complex spectrum in sequence to obtain the first complex spectrum of the first frequency component having the same size as the input; The input of the complex spectrum decoder is the residual sum of the output of the decoder of the next layer of complex spectrum and the output of the encoder of the complex spectrum of the layer corresponding to the decoder of the current layer of complex spectrum.

6. The speech denoising method based on a temporal convolutional network according to claim 3 or 5, characterized in that: Said N=4, M=1.

7. The speech denoising method based on a temporal convolutional network according to claim 3 or 5, characterized in that: The temporal convolutional network includes point convolution, dilated depth convolution, LayerNrom layer and nonlinear activation layer.

8. The speech noise reduction system based on the temporal convolutional network is characterized by: A speech denoising method based on a temporal convolutional network applied to any one of claims 1-7, comprising a speech signal acquisition module, a preprocessing module, a division module, a decomposition module, a first noise estimation module, a noise suppression module, a denoising complex spectrum acquisition module, a second noise estimation module, a complex spectrum addition module, a denoising module and a denoising output module; The voice signal acquisition module is used to acquire the original voice signal; The preprocessing module is used to preprocess the original speech signal to obtain a preprocessed speech signal; The division module is used to divide the preprocessed speech signal according to a preset time interval to obtain a plurality of short time frames; The decomposition module is used to decompose the multiple short-time frames into amplitude spectra of different frequency components, complex spectra of different frequency components and phase information of different frequency components by short-time Fourier transform; wherein the different frequency components include a first frequency component and a second frequency component; The first noise estimation module is used to perform noise estimation on the amplitude spectrum of the different frequency components to obtain amplitude spectrum masks of the different frequency components; The noise suppression module is used to apply the amplitude spectrum masks of the different frequency components to the original speech signal, perform preliminary noise suppression on the original speech signal, and obtain the amplitude spectrum masks of the different frequency components for preliminary noise reduction; The module for obtaining the denoised complex spectrum is used to obtain the preliminary denoised complex spectrum of different frequency components according to the preliminary denoised amplitude spectrum mask of the different frequency components and the phase information; The second noise estimation module is used to perform noise estimation based on the complex spectrum of the original speech of the first frequency component and the complex spectrum of the first frequency component after preliminary noise reduction, so as to obtain an estimated first complex spectrum of the first frequency component; The complex spectrum adding module is used to add the estimated first complex spectrum of the first frequency component and the complex spectrum of the first frequency component after preliminary noise reduction to obtain a second complex spectrum of the first frequency component; The noise reduction module is used to splice the second complex spectrum of the first frequency component with the complex spectrum of the second frequency component after preliminary noise reduction to obtain a complete complex spectrum of the original speech signal after noise reduction; The noise reduction output module is used to convert the complete noise-reduced complex spectrum of the original speech signal into a time domain signal output through short-time inverse Fourier transform.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the speech denoising method based on the temporal convolutional network as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the speech noise reduction method based on a temporal convolutional network described in any one of claims 1 to 7 is implemented.

Images