Speech denoising method, system, device and medium based on time sequence convolution network

Abstract

The application discloses a speech noise reduction method, system, device and medium based on a time sequence convolution network. The method comprises the following steps: preprocessing an original speech signal and dividing the original speech signal into multiple time frames; determining an amplitude spectrum, a complex spectrum and a phase of different frequency components; performing noise estimation on the amplitude spectrum and performing preliminary noise suppression on the original speech to obtain a preliminary noise-reduced amplitude spectrum mask; obtaining a preliminary noise-reduced complex spectrum of different frequency components according to the preliminary noise-reduced amplitude spectrum mask and the phase; performing noise estimation on the complex spectrum of the first frequency component original speech and the preliminary noise-reduced complex spectrum of the first frequency component to obtain a first complex spectrum of the first frequency component; adding the first complex spectrum of the first frequency component to the preliminary noise-reduced complex spectrum of the first frequency component, and then splicing the added result with the preliminary noise-reduced complex spectrum of the second frequency component to obtain a noise-reduced complex spectrum. The application can reduce the operation amount of the noise reduction model, improve the noise reduction effect under complex conditions and the robustness under a mutation scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of voice noise reduction, and in particular to a voice noise reduction method and system based on a time sequence convolution network, a device and a medium. BACKGROUND

[0002] In the field of voice communication, audio processing and voice recognition technology, voice noise reduction technology occupies a pivotal position. The existing noise reduction technology based on amplitude spectrum and complex spectrum captures voice signals through voice transmission and receiving equipment, and uses a two-stage deep learning model to suppress noise, combining the advantages of amplitude spectrum and complex spectrum, which can improve the noise reduction effect and voice restoration to a certain extent. However, although this method improves the noise reduction performance, the computational complexity of the deep learning model also increases, which undoubtedly limits the widespread application of this method on some low-resource devices, for example, on digital signal processors (DSP), earphones, IP phones and other devices, due to the limitation of hardware resources, the traditional deep learning model is often difficult to deploy. Secondly, the existing two-stage noise reduction technology based on amplitude spectrum and complex spectrum mostly uses GRU (Gated Recurrent Unit) operator for time sequence modeling. The GRU operator depends on too long historical information in the modeling process, and is not robust enough for sudden noise, large voice conversion and noise-voice conversion, and often has the problems of false elimination and over-inhibition. SUMMARY

[0003] The main purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide a voice noise reduction method and system based on a time sequence convolution network, which can reduce the computational complexity of the noise reduction model and improve the noise reduction effect of the noise reduction model under complex conditions and the robustness of the noise reduction model under sudden scene.

[0004] In order to achieve the above purpose, the following technical solutions are adopted in the present application:

[0005] In a first aspect, the present application provides a voice noise reduction method based on a time sequence convolution network, comprising the following steps:

[0006] Obtaining an original voice signal;

[0007] Pretreating the original voice signal to obtain a pretreated voice signal;

[0008] Dividing the pretreated voice signal according to a preset time interval to obtain a plurality of short-time frames;

[0009] Using short-time Fourier transform to decompose the plurality of short-time frames into amplitude spectrum of different frequency components, complex spectrum of different frequency components and phase information of different frequency components; wherein the different frequency components include first frequency components and second frequency components;

[0010] performing noise estimation on the amplitude spectrum of the different frequency components to obtain an amplitude spectrum mask of the different frequency components;

[0011] applying the amplitude spectrum mask of the different frequency components to the original speech signal to perform preliminary noise suppression on the original speech signal to obtain a preliminary noise-reduced amplitude spectrum mask of the different frequency components;

[0012] obtaining a preliminary noise-reduced complex spectrum of the different frequency components according to the preliminary noise-reduced amplitude spectrum mask of the different frequency components and the phase information;

[0013] performing noise estimation on the complex spectrum of the first frequency component original speech and the preliminary noise-reduced complex spectrum of the first frequency component to obtain an estimated first complex spectrum of the first frequency component;

[0014] adding the estimated first complex spectrum of the first frequency component to the preliminary noise-reduced complex spectrum of the first frequency component to obtain a second complex spectrum of the first frequency component;

[0015] splicing the second complex spectrum of the first frequency component and the preliminary noise-reduced complex spectrum of the second frequency component to obtain a complete noise-reduced complex spectrum of the original speech signal;

[0016] converting the complete noise-reduced complex spectrum of the original speech signal into a time-domain signal through an inverse short-time Fourier transform and outputting.

[0017] As a preferred technical solution, the first frequency component includes 0-4KHZ;

[0018] The second frequency component includes 4-8KHZ.

[0019] As a preferred technical solution, the noise estimation on the amplitude spectrum of the different frequency components to obtain an amplitude spectrum mask of the different frequency components includes:

[0020] The amplitude spectrum of the non-direct current part of the amplitude spectrum of the different frequency components is taken as the input of the frequency band division amplitude spectrum dimension reduction operation, and the dimension-reduced amplitude spectrum is processed through N layers of amplitude spectrum encoders, M layers of amplitude spectrum time sequence convolution networks and N layers of amplitude spectrum decoders to obtain an amplitude spectrum mask with the same size as the original amplitude spectrum.

[0021] The input of the amplitude spectrum decoder is the sum of the residual error of the output of the next layer of amplitude spectrum decoders and the output of the amplitude spectrum encoder of the corresponding layer of the current layer of amplitude spectrum decoders.

[0022] As a preferred technical solution, the frequency band division amplitude spectrum dimension reduction operation includes:

[0023] The 1x256 amplitude spectrum is divided by every 2kH, and a 4x64 amplitude spectrum is obtained by conversion.

[0024] As a preferred technical solution, the noise estimation is performed according to the complex spectrum of the first frequency component original speech and the complex spectrum of the first frequency component preliminary noise reduction, to obtain an estimated first frequency component first complex spectrum, and the method comprises the following steps:

[0025] The complex spectrum of the first frequency component original speech and the complex spectrum of the first frequency component preliminary noise reduction are spliced, and the spliced result is taken as the input of the encoder of the N-layer complex spectrum, and then sequentially processed by the M-layer complex spectrum time sequence convolution network and the N-layer complex spectrum decoder, to obtain the first frequency component first complex spectrum with the same size as the input;

[0026] The input of the complex spectrum decoder is the sum of the residual error between the output of the next layer complex spectrum decoder and the output of the complex spectrum encoder corresponding to the current layer complex spectrum decoder.

[0027] As a preferred technical solution, N=4 and M=1.

[0028] As a preferred technical solution, the time sequence convolution network comprises a point convolution, a dilated depth convolution, a LayerNorm layer and a nonlinear activation layer.

[0029] In a second aspect, the present application provides a speech noise reduction system based on a time sequence convolution network, which is applied to the speech noise reduction method based on the time sequence convolution network, and comprises an acquisition of a speech signal module, a preprocessing module, a division module, a decomposition module, a first noise estimation module, a noise suppression module, an acquisition of a noise reduction complex spectrum module, a second noise estimation module, a complex spectrum addition module, a noise reduction module and a noise reduction output module.

[0030] The acquisition of a speech signal module is configured to acquire an original speech signal.

[0031] The preprocessing module is configured to pre-process the original speech signal to obtain a pre-processed speech signal.

[0032] The division module is configured to divide the pre-processed speech signal according to a preset time interval to obtain a plurality of short-time frames.

[0033] The decomposition module is configured to decompose the plurality of short-time frames into amplitude spectra of different frequency components, complex spectra of different frequency components and phase information of different frequency components by using a short-time Fourier transform; wherein the different frequency components comprise a first frequency component and a second frequency component.

[0034] The first noise estimation module is configured to perform noise estimation on the amplitude spectrum of the different frequency components to obtain amplitude spectrum masks of the different frequency components.

[0035] The noise suppression module is configured to apply the amplitude spectrum masks of the different frequency components to the original speech signal to perform preliminary noise suppression on the original speech signal to obtain preliminary noise-reduced amplitude spectrum masks of the different frequency components.

[0036] The noise-reduced complex spectrum obtaining module is configured to obtain preliminary noise-reduced complex spectra of the different frequency components according to the preliminary noise-reduced amplitude spectrum masks of the different frequency components and the phase information.

[0037] The second noise estimation module is configured to perform noise estimation on the complex spectrum of the first frequency component of the original speech and the preliminary noise-reduced complex spectrum of the first frequency component to obtain an estimated first complex spectrum of the first frequency component.

[0038] The complex spectrum adding module is configured to add the estimated first complex spectrum of the first frequency component to the preliminary noise-reduced complex spectrum of the first frequency component to obtain a second complex spectrum of the first frequency component.

[0039] The noise-reducing module is configured to splice the second complex spectrum of the first frequency component with the preliminary noise-reduced complex spectrum of the second frequency component to obtain a complete noise-reduced complex spectrum of the original speech signal.

[0040] The noise-reduced output module is configured to convert the complete noise-reduced complex spectrum of the original speech signal into a time-domain signal through inverse short-time Fourier transform and output the time-domain signal.

[0041] In a third aspect, the present application provides an electronic device, which comprises:

[0042] at least one processor; and a memory connected to the at least one processor in communication;

[0043] The memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to execute the speech noise reduction method based on the time sequence convolution network.

[0044] In a fourth aspect, the present application provides a computer readable storage medium storing a program, and the program is executed by a processor to implement the speech noise reduction method based on the time sequence convolution network.

[0045] In summary, compared with the prior art, the technical scheme provided by the present application has at least the following effective effects:

[0046] The application provides a speech denoising method based on a time sequence convolution network. Firstly, a denoising model structure of a two-stage operation mode of an amplitude spectrum and a complex spectrum is adopted, so that the denoising effect of the model under complex conditions is significantly improved, and the restoration ability of human voice and the sound quality improvement are obvious. Then, a TCN (time sequence convolution network) operator is used for time sequence modeling, so that the robustness of the model under the mutation scene is significantly improved. Secondly, the input amplitude spectrum of the first stage is divided into frequency bands, and the complex spectrum of the second stage is only modeled below 4 kHz, so that the operation amount of the model is effectively reduced, the model can be run in real time on a low-performance platform, and the whole system can be flexibly deployed in real-time conference communication equipment, so as to better process various complex environmental noises, and improve the application effect of the denoised speech in the fields of speech communication, audio processing and speech recognition. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0048] Figure 1 The flow chart of the speech denoising method based on the time sequence convolution network provided by an embodiment of the application is shown in the figure.

[0049] Figure 2 The overall flowchart of the speech denoising method based on the time sequence convolution network provided by an embodiment of the application is shown in the figure.

[0050] Figure 3 The block diagram of the speech denoising system based on the time sequence convolution network provided by an embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0051] In order to make the person skilled in the art better understand the application scheme, the technical solutions in the embodiments of the application will be described clearly and completely in conjunction with the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0052] Reference to an "embodiment" in this application means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. As will be apparent to those of ordinary skill in the art, embodiments described in this application can be combined with one another.

[0053] Embodiments:

[0054] Reference is made to Figure 1 and Figure 2 In one embodiment of the application, a speech denoising method based on a timing convolution network is provided, comprising the following steps:

[0055] S1, obtaining an original speech signal.

[0056] Further, the speech signal of the application is obtained from a collection unit, which can be composed of a microphone or a recording device and an ADC hardware chip. First, the original speech signal is collected from the speech source such as a microphone or a recording device, and then an ADC (analog-to-digital converter) hardware chip is used to convert the speech analog signal in the environment into a digital signal.

[0057] S2, pre-processing the original speech signal to obtain a pre-processed speech signal.

[0058] Further, some pre-processing operations are performed on the original speech signal, which include removing DC offset, adjusting sampling rate, etc., to ensure that the signal is within a proper range.

[0059] S3, dividing the pre-processed speech signal according to a preset time interval to obtain a plurality of short-time frames.

[0060] The pre-processed speech signal is divided according to a preset time interval to obtain a plurality of short-time frames, which is beneficial for subsequent processing of the speech signal.

[0061] S4, using short-time Fourier transform to decompose the plurality of short-time frames into amplitude spectrum of different frequency components, complex spectrum of different frequency components, and phase information of different frequency components; wherein the different frequency components include first frequency components and second frequency components.

[0062] Further, the plurality of short-time frames are decomposed into amplitude spectrum of different frequency components, complex spectrum of different frequency components, and phase information of different frequency components by using short-time Fourier transform (STFT, Short-Time Fourier Transform), further forming amplitude spectrum and complex spectrum.

[0063] The first frequency component includes a 0-4KHZ part; and the second frequency component includes a 4-8KHZ part.

[0064] S5, performing noise estimation on the amplitude spectrum of the different frequency components to obtain an amplitude spectrum mask of the different frequency components.

[0065] Further, after the short-time Fourier transform processing, the frequency domain amplitude spectrum of the noisy speech is extracted, wherein the size of the non-direct current part is FFT_n / 2, wherein FFT_n represents the Fourier transform point number of the short-time Fourier transform, and in the present application, 512 is preferably used as the value; subsequently, the amplitude spectrum of the non-direct current part of the amplitude spectrum of the different frequency components is taken as the input of the first-stage Unet model structure (denoising model).

[0066] In the first-stage Unet model structure, noise estimation is performed on the amplitude spectrum of the different frequency components to obtain an amplitude spectrum mask of the different frequency components, and the steps include:

[0067] First, the amplitude spectrum of the non-direct current part of the amplitude spectrum of the different frequency components is subjected to a frequency band amplitude spectrum dimension reduction operation, which specifically is to divide the 1x256 amplitude spectrum every 2kH into a 4x64 amplitude spectrum, and the frequency band amplitude spectrum dimension reduction operation helps to reduce the data amount while retaining important frequency information;

[0068] Secondly, the dimension-reduced amplitude spectrum is processed through N layers of amplitude spectrum encoders Encoder (Encoder1…Encoder4), M layers of amplitude spectrum time sequence convolution networks, and N layers of amplitude spectrum decoders Decoder (Decoder1…Decoder4), and finally an amplitude spectrum mask with the same size as the original amplitude spectrum is obtained; in the present application, N=4 and M=1; the output channel numbers of the amplitude spectrum encoders Encoder1 to Encoder4 are all preferably 32; the output channel numbers of the amplitude spectrum decoders Decoder1 and Decoder4 are all preferably 32; the CNN channel number of the amplitude spectrum TCN (time sequence convolution network) is preferably 32, and a total of 6 layers are stacked, i.e., 6 TCNs are included, each of which includes a point convolution, a dilated depth convolution, a LayerNorm layer, and a nonlinear activation layer, and the 6 dilated depth convolutions included in the 6 TCNs have expansion rates of 1, 2, 4, 8, 16, and 32, respectively.

[0069] Further, in the present application, for the amplitude spectrum decoder Decoder i , the input is defined as the next layer of the amplitude spectrum decoder Decoderi+1 The output corresponds to the encoder of the amplitude spectrum of the layer corresponding to the decoder of the current layer's amplitude spectrum. i+1 The input to the amplitude spectrum Decoder4 is defined as the sum of the residuals of the output of the amplitude temporal convolutional network and the output of the amplitude spectrum encoder Encoder4 of the corresponding layer of the current layer's amplitude spectrum decoder.

[0070] S6. Apply the amplitude spectrum mask of the different frequency components to the original speech signal to perform preliminary noise suppression on the original speech signal, and obtain the amplitude spectrum mask of the preliminary noise reduction of the different frequency components.

[0071] S7. Based on the amplitude spectrum mask of the initial noise reduction of the different frequency components and the phase information, the complex spectrum of the initial noise reduction of the different frequency components is obtained.

[0072] S8. Based on the complex spectrum of the original speech of the first frequency component and the complex spectrum of the first frequency component after preliminary noise reduction, noise estimation is performed to obtain the estimated first complex spectrum of the first frequency component.

[0073] Further, in the second stage, the complex spectrum of the original speech (0-4kHz portion) and the initially denoised complex spectrum of the first frequency component (0-4kHz portion) are concatenated as the input to the second-stage Unet model structure. Then, it is processed sequentially through an N-layer complex spectrum encoder, an M-layer complex spectrum temporal convolutional network, and an N-layer complex spectrum decoder to obtain the first complex spectrum of the first frequency component with the same size as the input. Here, N=4 and M=1. The number of output channels for the complex spectrum encoders Encoder1 to Encoder4 is preferably 32; the number of output channels for the complex spectrum decoders Decoder1 and Decoder4 is preferably 32; the number of channels for the convolutional neural network (CNN) of the complex spectrum temporal convolutional network (TCN) is preferably 32, and it is stacked in a total of 6 layers, i.e., it contains 6 TCNs. Each TCN includes point convolution, dilated depthwise convolution, LayerNrom layer and nonlinear activation layer. The 6 TCNs contain 6 dilated depthwise convolutions, and the dilation rates of the 6 dilated depthwise convolutions are 1, 2, 4, 8, 16 and 32, respectively.

[0074] Furthermore, in this application, for the decoder of complex spectra... i Its input is defined as the decoder of the next layer's complex spectrum. i+1 The output corresponds to the encoder of the complex spectrum of the layer corresponding to the decoder of the current layer's complex spectrum. i+1If i = 4, the input of Decoder4 of the complex spectrum is defined as the sum of the residual of the output of the complex time convolution network and the output of the encoder Encoder4 of the corresponding layer of the decoder of the complex spectrum of the current layer of the complex spectrum.

[0075] Specifically, in the present application, the time convolution network of the amplitude spectrum in the first stage and the time convolution network of the complex spectrum in the second stage are composed of point convolution, dilated depth convolution, LayerNorm layer and nonlinear activation layer; wherein the input channel number, the output channel number and the group number of the point convolution are consistent with the output channel of the previous layer, and the group number is 1; and the input channel number, the output channel number and the group number of the dilated depth convolution are consistent with the output channel number of the point convolution, and the expansion rates of the dilated depth convolution are 1, 2, 4, 8, 16 and 32 respectively.

[0076] S9, adding the estimated first frequency component first complex spectrum and the first frequency component preliminary noise reduction complex spectrum to obtain the first frequency component second complex spectrum.

[0077] Further, the estimated first frequency component first complex spectrum is added to the first frequency component preliminary noise reduction complex spectrum, and the amplitude and phase of this part are simultaneously optimized and adjusted to obtain the first frequency component (0-4KHZ) second complex spectrum, so as to improve the noise suppression effect and the degree of human voice restoration.

[0078] S10, splicing the first frequency component second complex spectrum and the second frequency component preliminary noise reduction complex spectrum to obtain the complete noise reduction complex spectrum of the original speech signal.

[0079] The first frequency component (0-4KHZ) second complex spectrum is spliced with the second frequency component (4-8KHZ) preliminary noise reduction complex spectrum to obtain the complete noise reduction complex spectrum (i.e. the complete noise reduction complex spectrum of the original speech signal) of 0-8kHz.

[0080] S11, converting the complete noise reduction complex spectrum of the original speech signal into a time domain signal output through inverse short-time Fourier transform.

[0081] The complete noise reduction complex spectrum of the original speech signal is converted into a time domain noise reduction speech through inverse short-time Fourier transform (ISTFT, Inverse Short-Time Fourier Transform).

[0082] Specifically, the noise reduction speech output signal will be played through a terminal unit, and the terminal unit can be composed of any real-time conference communication equipment with network access.

[0083] The application uses the amplitude spectrum of the original speech after frequency band dimension reduction for noise estimation in the first stage Unet model, obtains an amplitude spectrum mask, and applies the amplitude spectrum mask to the original speech to perform preliminary noise suppression on the original speech, and then converts the original speech into a complex spectrum of the preliminary noise-reduced speech. Then, in the second stage Unet model, the complex spectrum of the original speech and the 0-4 kHz part of the complex spectrum of the preliminary noise-reduced speech in the first stage are used for noise estimation to obtain an estimated 0-4 kHz part complex spectrum, and the estimated 0-4 kHz part complex spectrum and the 0-4 kHz part complex spectrum of the preliminary noise-reduced speech in the first stage are added to simultaneously optimize and adjust the amplitude and phase of the part, improve the noise suppression effect and the human voice restoration degree, then splice the 4-8 kHz part complex spectrum of the preliminary noise-reduced speech in the first stage to obtain a 0-8 kHz complete noise-reduced complex spectrum, and finally output the iSTFT time domain signal. The calculation can be performed by a single unit or a computer chip set with an operating system, which is used for calculation of the noise reduction model in the above two stages; the output signal can be transmitted by a data transmission unit composed of a network system that can transmit data to transmit the calculated data. The application improves the noise reduction effect and robustness of the noise reduction model, and has higher adaptability under lower performance consumption, can be better deployed on low-resource devices, and can better handle various complex environmental noise, thereby improving the application effect of noise-reduced speech in the fields of voice communication, audio processing and voice recognition.

[0084] It should be noted that, for the foregoing method embodiments, in order to facilitate description, they are all described as a series of action combinations, but those skilled in the art should know that the application is not limited by the action sequence described, because according to the application, certain steps can be performed in other sequences or simultaneously.

[0085] Based on the same idea as the speech noise reduction method based on the timing convolution network in the above embodiment, the application also provides a speech noise reduction system based on the timing convolution network, which can be used to execute the speech noise reduction method based on the timing convolution network. For the convenience of description, only the part related to the embodiment of the application is shown in the structure diagram of the speech noise reduction system based on the timing convolution network, and those skilled in the art can understand that the structure diagram does not constitute a limitation on the device, and can include more or fewer components than the diagram, or combine certain components, or different component arrangements.

[0086] Please refer to Figure 3In another embodiment of the present application, a speech noise reduction system based on a time convolution network is provided, which comprises an original speech signal acquisition module 101, a preprocessing module 102, a division module 103, a decomposition module 104, a first noise estimation module 105, a noise suppression module 106, a preliminary noise reduction complex spectrum acquisition module 107, a second noise estimation module 108, a complex spectrum addition module 109, a noise reduction module 110, and a noise reduction output module 111.

[0087] The original speech signal acquisition module 101 is configured to acquire an original speech signal.

[0088] The preprocessing module 102 is configured to preprocess the original speech signal to obtain a preprocessed speech signal.

[0089] The division module 103 is configured to divide the preprocessed speech signal according to a preset time interval to obtain a plurality of short-time frames.

[0090] The decomposition module 104 is configured to decompose the plurality of short-time frames into amplitude spectra of different frequency components, complex spectra of different frequency components, and phase information of different frequency components by using a short-time Fourier transform; wherein the different frequency components include first frequency components and second frequency components.

[0091] The first noise estimation module 105 is configured to perform noise estimation on the amplitude spectra of different frequency components to obtain amplitude spectrum masks of different frequency components.

[0092] The noise suppression module 106 is configured to apply the amplitude spectrum masks of different frequency components to the original speech signal to perform preliminary noise suppression on the original speech signal to obtain preliminary noise reduction amplitude spectrum masks of different frequency components.

[0093] The preliminary noise reduction complex spectrum acquisition module 107 is configured to obtain preliminary noise reduction complex spectra of different frequency components according to the preliminary noise reduction amplitude spectrum masks of different frequency components and the phase information.

[0094] The second noise estimation module 108 is configured to perform noise estimation on the complex spectrum of the first frequency component original speech and the preliminary noise reduction complex spectrum of the first frequency component to obtain an estimated first frequency component first complex spectrum.

[0095] The complex spectrum addition module 109 is configured to add the estimated first frequency component first complex spectrum and the preliminary noise reduction complex spectrum of the first frequency component to obtain a first frequency component second complex spectrum.

[0096] The noise reduction module 110 is configured to splice the first frequency component second complex spectrum and the second frequency component preliminary noise-reduced complex spectrum to obtain a complete noise-reduced complex spectrum of the original speech signal.

[0097] The noise reduction output module 111 is configured to convert the complete noise-reduced complex spectrum of the original speech signal into a time-domain signal through inverse short-time Fourier transform and output the time-domain signal.

[0098] It should be noted that the speech noise reduction system based on the time sequence convolution network of the present application corresponds to the speech noise reduction method based on the time sequence convolution network of the present application. The technical features and advantages described in the embodiment of the speech noise reduction method based on the time sequence convolution network are applicable to the embodiment of the speech noise reduction system based on the time sequence convolution network. For specific content, refer to the description in the method embodiment of the present application. Here, no further description is given, and this is hereby declared.

[0099] In addition, in the embodiment of the speech noise reduction system based on the time sequence convolution network of the above embodiment, the logical division of each program module is only an example. In actual application, the above functions can be completed by different program modules according to needs, for example, considering the configuration requirements of the corresponding hardware or the convenience of software implementation. That is, the internal structure of the speech noise reduction system based on the time sequence convolution network is divided into different program modules to complete all or part of the functions described above.

[0100] In another embodiment, an electronic device implementing the speech noise reduction method based on the time sequence convolution network is provided, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor; when the processor executes the computer program, the speech noise reduction method based on the time sequence convolution network of any embodiment of the present application is implemented.

[0101] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present application. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the device.

[0102] The device can be a desktop computer, a notebook, a palm computer, and a cloud server, etc. The device can include, but is not limited to, a processor and a memory.

[0103] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor or the like, and the processor is a control center of the device, and connects all parts of the device through various interfaces and lines;

[0104] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the device by running or executing the computer program and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function, etc.; in addition, the memory can include a high-speed random access memory, and can also include a nonvolatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory device.

[0105] Correspondingly, the application further provides a computer readable storage medium, which comprises a stored computer program, wherein when the computer program runs, the device where the computer readable storage medium is located performs the voice noise reduction method based on a timing convolution network as any one of the above embodiments.

[0106] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware. The program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0107] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0108] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations and simplifications made without departing from the spirit and principle of the present application shall be equivalent replacement methods and shall be included in the protection scope of the present application.

Claims

1. A speech denoising method based on temporal convolutional networks, characterized in that, Includes the following steps: Acquire the raw speech signal; The original speech signal is preprocessed to obtain the preprocessed speech signal; The preprocessed speech signal is divided into multiple short frames according to a preset time interval; The multiple short-time frames are decomposed into amplitude spectra of different frequency components, complex spectra of different frequency components, and phase information of different frequency components using short-time Fourier transform; wherein, the different frequency components include a first frequency component and a second frequency component. Noise estimation is performed on the amplitude spectra of the different frequency components to obtain the amplitude spectrum masks of the different frequency components; The amplitude spectrum masks of the different frequency components are applied to the original speech signal to perform preliminary noise suppression on the original speech signal, resulting in amplitude spectrum masks for preliminary noise reduction of the different frequency components. Based on the amplitude spectrum mask of the initial noise reduction of the different frequency components and the phase information, the complex spectrum of the initial noise reduction of the different frequency components is obtained. Noise estimation is performed based on the complex spectrum of the original speech of the first frequency component and the complex spectrum of the first frequency component after preliminary noise reduction, to obtain the estimated first complex spectrum of the first frequency component; The estimated first complex spectrum of the first frequency component is added to the complex spectrum of the first frequency component after preliminary noise reduction to obtain the second complex spectrum of the first frequency component; The second complex spectrum of the first frequency component is spliced ​​with the complex spectrum of the second frequency component after preliminary noise reduction to obtain the complete denoised complex spectrum of the original speech signal. The complex spectrum of the original speech signal after complete noise reduction is converted into a time-domain signal output through short-time inverse Fourier transform.

2. The speech denoising method based on temporal convolutional networks according to claim 1, characterized in that, The first frequency component includes 0-4kHz; The second frequency component includes 4-8kHz.

3. The speech denoising method based on temporal convolutional networks according to claim 1, characterized in that, The step of performing noise estimation on the amplitude spectra of the different frequency components to obtain amplitude spectrum masks for the different frequency components includes: The amplitude spectrum of the non-DC portion of the amplitude spectrum of different frequency components is used as the input of the frequency band amplitude spectrum dimensionality reduction operation. The dimensionality-reduced amplitude spectrum is processed by an N-layer amplitude spectrum encoder, an M-layer amplitude spectrum temporal convolutional network, and an N-layer amplitude spectrum decoder to obtain an amplitude spectrum mask of the same size as the original amplitude spectrum. The input to the amplitude spectrum decoder is the sum of the residuals between the output of the next layer amplitude spectrum decoder and the output of the corresponding layer amplitude spectrum encoder of the current layer amplitude spectrum decoder.

4. The speech denoising method based on temporal convolutional networks according to claim 3, characterized in that, The frequency band amplitude spectrum dimensionality reduction operation includes: The amplitude spectrum of 1x256 is divided into 2kHz segments to obtain an amplitude spectrum of 4x64.

5. The speech denoising method based on temporal convolutional networks according to claim 1, characterized in that, The step of performing noise estimation based on the complex spectrum of the original speech of the first frequency component and the complex spectrum of the first frequency component after preliminary noise reduction, to obtain the estimated first complex spectrum of the first frequency component, includes: The complex spectrum of the original speech of the first frequency component and the complex spectrum of the first frequency component after preliminary noise reduction are concatenated. The concatenated result is used as the input of the encoder of the N-layer complex spectrum, and then processed by the temporal convolutional network of the M-layer complex spectrum and the decoder of the N-layer complex spectrum in sequence to obtain the first complex spectrum of the first frequency component with the same size as the input. The input to the decoder of the complex spectrum is the sum of the residuals between the output of the decoder of the next layer of the complex spectrum and the output of the encoder of the complex spectrum of the corresponding layer of the decoder of the current layer of the complex spectrum.

6. The speech denoising method based on temporal convolutional networks according to claim 3 or 5, characterized in that, The N=4, M=1.

7. The speech denoising method based on temporal convolutional networks according to claim 3 or 5, characterized in that, The temporal convolutional network includes point convolutions, dilated depthwise convolutions, LayerNrom layers, and nonlinear activation layers.

8. A speech denoising system based on temporal convolutional networks, characterized in that, The speech denoising method based on temporal convolutional networks applied to any one of claims 1-7 includes a speech signal acquisition module, a preprocessing module, a partitioning module, a decomposition module, a first noise estimation module, a noise suppression module, a denoising complex spectrum acquisition module, a second noise estimation module, a complex spectrum addition module, a denoising module, and a denoising output module. The audio signal acquisition module is used to acquire the original audio signal; The preprocessing module is used to preprocess the original speech signal to obtain a preprocessed speech signal; The segmentation module is used to segment the preprocessed speech signal according to a preset time interval to obtain multiple short frames; The decomposition module is used to decompose the multiple short-time frames into amplitude spectra of different frequency components, complex spectra of different frequency components, and phase information of different frequency components using short-time Fourier transform; wherein, the different frequency components include a first frequency component and a second frequency component. The first noise estimation module is used to perform noise estimation on the amplitude spectrum of the different frequency components to obtain the amplitude spectrum mask of the different frequency components. The noise suppression module is used to apply the amplitude spectrum mask of the different frequency components to the original speech signal to perform preliminary noise suppression on the original speech signal and obtain the amplitude spectrum mask of the preliminary noise reduction of the different frequency components. The module for obtaining the denoised complex spectrum is used to obtain the complex spectrum of the preliminary denoising for different frequency components based on the amplitude spectrum mask of the preliminary denoising for different frequency components and the phase information. The second noise estimation module is used to perform noise estimation based on the complex spectrum of the original speech of the first frequency component and the complex spectrum of the first frequency component after preliminary noise reduction, so as to obtain the estimated first complex spectrum of the first frequency component. The complex spectrum addition module is used to add the estimated first complex spectrum of the first frequency component to the complex spectrum of the first frequency component after preliminary noise reduction to obtain the second complex spectrum of the first frequency component. The noise reduction module is used to splice the second complex spectrum of the first frequency component with the complex spectrum of the second frequency component after preliminary noise reduction to obtain the complete noise-reduced complex spectrum of the original speech signal. The noise reduction output module is used to convert the complete noise-reduced complex spectrum of the original speech signal into a time-domain signal output through short-time inverse Fourier transform.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to said at least one processor; The memory stores computer program instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the speech denoising method based on temporal convolutional networks as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the speech denoising method based on temporal convolutional networks as described in any one of claims 1-7.

Images