Speech noise reduction method, device, equipment and readable storage medium

Abstract

The present invention discloses a speech noise reduction method, apparatus, device, and computer-readable storage medium. The method comprises: obtaining spectral data of multiple signal frames, performing voice activity detection on each of the signal frames to obtain voice activity detection information for each of the signal frames; inputting the spectral data into a full-band model to extract full-band spectral information corresponding to each of the signal frames; and inputting the full-band spectral information and the voice activity detection information into a sub-band model for prediction to obtain clean speech corresponding to each of the signal frames. The present invention implements a deep learning speech noise reduction solution that integrates voice activity detection information, enabling the model to better focus on utilizing speech frame information for noise reduction processing, thereby improving the speech noise reduction effect.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a speech noise reduction method, apparatus, device, and computer-readable storage medium. Background Art

[0002] Single-channel (mono) speech noise reduction methods aim to improve speech quality and intelligibility by removing background noise from a single-channel noisy audio signal. However, traditional signal processing methods often have poor results for non-stationary or bursty noise. Furthermore, traditional speech noise reduction methods rely on pre-defined assumptions, but these assumptions can be irrational. Furthermore, traditional speech enhancement relies on parameter settings and manual experience. With the development of deep learning, more and more people are turning to deep learning to solve speech noise reduction problems. Summary of the Invention

[0003] The main purpose of the present invention is to provide a speech noise reduction method, device, equipment and computer-readable storage medium, aiming to realize a deep learning speech noise reduction solution that integrates voice activity detection information and improve the speech noise reduction effect.

[0004] To achieve the above object, the present invention provides a method for reducing speech noise, the method comprising the following steps:

[0005] Acquiring spectrum data of a plurality of signal frames, and performing voice activity detection on each of the signal frames to obtain voice activity detection information of each of the signal frames;

[0006] Inputting the spectrum data into a full-band model to extract full-band spectrum information corresponding to each signal frame;

[0007] The full-band spectrum information and the voice activity detection information are input into a sub-band model for prediction to obtain clean speech corresponding to each signal frame.

[0008] Optionally, the step of acquiring spectrum data of a plurality of signal frames includes:

[0009] Acquire amplitude spectrum data, real spectrum data, and imaginary spectrum data of multiple signal frames;

[0010] The step of inputting the spectrum data into a full-band model to extract the full-band spectrum information corresponding to each signal frame comprises:

[0011] The amplitude spectrum data, the real spectrum data and the imaginary spectrum data are input into a full-band model to extract the full-band amplitude spectrum information, the full-band real spectrum information and the full-band imaginary spectrum information corresponding to each signal frame.

[0012] Optionally, the step of inputting the amplitude spectrum data, the real spectrum data and the imaginary spectrum data into a full-band model to extract the full-band amplitude spectrum information, the full-band real spectrum information and the full-band imaginary spectrum information corresponding to each signal frame includes:

[0013] The amplitude spectrum data, the real spectrum data and the imaginary spectrum data are respectively input into the amplitude spectrum weighting module, the real spectrum weighting module and the imaginary spectrum weighting module in the full-band model for frequency band weighting, so as to obtain weighted amplitude spectrum data, weighted real spectrum data and weighted imaginary spectrum data corresponding to each signal frame;

[0014] The weighted amplitude spectrum data, the weighted real spectrum data and the weighted imaginary spectrum data are respectively input into the amplitude full-band extractor, the real full-band extractor and the imaginary full-band extractor in the full-band model to extract the full-band amplitude spectrum information, the full-band real spectrum information and the full-band imaginary spectrum information corresponding to each signal frame.

[0015] Optionally, the step of inputting the full-band spectrum information and the voice activity detection information into a sub-band model for prediction to obtain clean speech corresponding to each signal frame includes:

[0016] For any target signal frame in each of the signal frames, and for any target frequency, splicing the full-band information corresponding to the target frequency in the full-band amplitude spectrum information corresponding to the target signal frame, the full-band information corresponding to the target frequency in the full-band real spectrum information corresponding to the target signal frame, the full-band information corresponding to the target frequency in the full-band imaginary spectrum information corresponding to the target signal frame, and the voice activity detection information corresponding to the target signal frame to obtain a sub-band model input sequence of the target frequency corresponding to the target signal frame;

[0017] The sub-band model input sequence of each frequency corresponding to each signal frame is input into the sub-band model for prediction to obtain the clean speech corresponding to each signal frame.

[0018] Optionally, the step of inputting the sub-band model input sequence of each frequency corresponding to each signal frame into the sub-band model for prediction to obtain the clean speech corresponding to each signal frame includes:

[0019] splicing the weighted data of the target frequency in the weighted amplitude spectrum data corresponding to the target signal frame with the weighted data of multiple frequencies adjacent to the target frequency to obtain sub-band data of the target frequency corresponding to the target signal frame;

[0020] The sub-band data of the target frequency corresponding to the target signal frame and the sub-band model input sequence are spliced ​​and then input into the sub-band model for prediction to obtain the clean speech corresponding to the target signal frame.

[0021] Optionally, the step of inputting the sub-band model input sequence of each frequency corresponding to each signal frame into the sub-band model for prediction to obtain the clean speech corresponding to each signal frame includes:

[0022] Inputting the sub-band model sequence of each frequency corresponding to each signal frame into the sub-band model for prediction, to obtain a complex ideal ratio mask corresponding to each signal frame;

[0023] The clean speech corresponding to each of the signal frames is obtained by calculation according to the complex ideal ratio mask and the spectrum data.

[0024] Optionally, the step of acquiring spectrum data of a plurality of signal frames includes:

[0025] Obtaining an audio signal to be denoised;

[0026] Performing frame processing on the audio signal to obtain multiple signal frames;

[0027] Each of the signal frames is converted from the time domain to the frequency domain to obtain spectrum data.

[0028] To achieve the above object, the present invention further provides a speech noise reduction device, comprising:

[0029] an acquisition module, configured to acquire spectrum data of a plurality of signal frames, and perform voice activity detection on each of the signal frames to obtain voice activity detection information of each of the signal frames;

[0030] An extraction module, configured to input the spectrum data into a full-band model to extract full-band spectrum information corresponding to each signal frame;

[0031] The prediction module is used to input the full-band spectrum information and the voice activity detection information into a sub-band model for prediction to obtain the clean speech corresponding to each signal frame.

[0032] To achieve the above-mentioned objectives, the present invention further provides a speech noise reduction device, comprising: a memory, a processor, and a speech noise reduction program stored in the memory and executable on the processor, wherein the speech noise reduction program, when executed by the processor, implements the steps of the speech noise reduction method described above.

[0033] In addition, to achieve the above objectives, the present invention also proposes a computer-readable storage medium, on which a speech noise reduction program is stored. When the speech noise reduction program is executed by a processor, the steps of the speech noise reduction method described above are implemented.

[0034] In an embodiment of the present invention, spectral data of multiple signal frames is obtained, and voice activity detection is performed on each of the signal frames to obtain voice activity detection information for each signal frame; the spectral data is input into a full-band model to extract the full-band spectral information corresponding to each signal frame; and the full-band spectral information and the voice activity detection information are input into a sub-band model for prediction to obtain the clean speech corresponding to each signal frame. This embodiment of the present invention implements a deep learning speech noise reduction solution that integrates voice activity detection information, enabling the model to better focus on utilizing speech frame information for noise reduction processing, thereby improving the speech noise reduction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of an embodiment of a method for reducing speech noise according to the present invention;

[0036] Figure 2 This is a schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention.

[0037] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0038] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0039] Reference Figure 1 , Figure 1 FIG. 4 is a flow chart of an embodiment of a method for reducing speech noise according to the present invention.

[0040] The embodiment of the present invention provides an embodiment of a method for speech noise reduction. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown here. In this embodiment, the execution subject of the speech noise reduction method can be a headset, a smart phone, a personal computer, a server and other devices, which are not limited in this embodiment. In this embodiment, for ease of description, each embodiment is described with the noise reduction device as the execution subject. In this embodiment, the speech noise reduction method includes the following steps:

[0041] Step S10, acquiring spectrum data of a plurality of signal frames, and performing voice activity detection on each of the signal frames to obtain voice activity detection information of each of the signal frames;

[0042] The signal frame can be obtained by framing the audio signal, and the length of each signal frame is not limited in this embodiment. In this embodiment, there is no restriction on the method of obtaining the audio signal and the scenario of performing voice noise reduction on the audio signal. For example, in a feasible implementation, the noise reduction device can collect the audio signal through a microphone, frame the audio signal in real time, and then perform real-time voice noise reduction on each signal frame. The spectrum data can be converted from the time domain to the frequency domain to obtain spectrum data in the frequency domain. The spectrum data can specifically be amplitude spectrum data, or can also include real spectrum data or imaginary spectrum data, which is not limited in this embodiment.

[0043] Voice activity detection (VAD), also known as speech activity detection or speech detection, is a technology used in speech processing to detect the presence of speech signals. The specific implementation of voice activity detection is not limited in this embodiment. The voice activity detection information of a signal frame may be, but is not limited to, information indicating whether the signal frame is a speech frame.

[0044] Step S20, inputting the spectrum data into a full-band model to extract full-band spectrum information corresponding to each signal frame;

[0045] The full-band model is a model used to extract full-band spectrum information based on full-band spectrum data. Full-band spectrum information includes global frequency domain feature information corresponding to each frequency in the full band. Assuming there are T signal frames and F frequencies, the spectrum data can be expressed as X∈R F×T , the full-band spectrum information can be expressed as Ψ∈R F×T .

[0046] In this embodiment, the specific structure of the full-band model is not limited. For example, it can be implemented using LSTM (Long Short-Term Memory).

[0047] Step S30: Input the full-band spectrum information and the voice activity detection information into a sub-band model for prediction to obtain clean speech corresponding to each signal frame.

[0048] The subband model is used to extract subband spectral information based on the subband's spectral data. In this embodiment, by replacing the input subband spectral data with the input subband's global frequency-domain feature information and voice activity detection information, the subband model can, on the one hand, make predictions based on the global frequency-domain feature information extracted by the full-band model, and on the other hand, focus on using speech frame information for prediction, thereby improving the speech noise reduction effect.

[0049] In this embodiment, there is no limitation on the specific structure of the sub-band model, and for example, LSTM can be used to implement it.

[0050] It should be noted that the full-band model and the sub-band model can be obtained in advance through joint training of a training data set. The training data set may include multiple bands of noisy frequency signals and clean speech signals corresponding to each band of noisy frequency data. The clean speech signal can be used as supervision data, and the parameters of the full-band model and the sub-band model can be trained using a supervised training method. When speech noise reduction is required, the trained model is used to perform speech noise reduction.

[0051] In one feasible implementation, for any one of the signal frames and any one of the frequencies, the noise reduction device can concatenate the full-band information corresponding to the frequency of the signal frame (i.e., full-band frequency domain feature information) with the voice activity detection information of the signal frame to obtain the sub-band model input sequence of the frequency of the signal frame, and input the sub-band model input sequence of each frequency of each signal frame into the sub-band model for prediction to obtain the clean speech corresponding to each signal frame. For example, assuming there are T signal frames and F frequencies, the voice activity detection information for the t-th signal frame can be expressed as Ψ s t ∈R, for the fth frequency of the tth signal frame, its full-band information can be expressed as Ψ m t，f ∈R, then the spliced ​​t-th signal frame in the f-th frequency sub-band model input sequence can be expressed as Ψ n t，f ∈R 2 The sub-band model input sequence of the f-th frequency of T signal frames can be used as an independent input sequence. A total of F input sequences can be input into the sub-band model for prediction to obtain the clean spectrum data of T signal frames at each frequency. Then, through frequency domain to time domain conversion, the clean spectrum data is converted into the clean speech signal corresponding to each signal frame.

[0052] In this embodiment, spectral data from multiple signal frames is acquired, and voice activity detection is performed on each signal frame to obtain voice activity detection information for each signal frame. This spectral data is then input into a full-band model to extract full-band spectral information corresponding to each signal frame. This full-band spectral information and the voice activity detection information are then input into a sub-band model for prediction to obtain clean speech corresponding to each signal frame. This embodiment implements a deep learning speech noise reduction solution that integrates voice activity detection information, enabling the model to better focus on utilizing speech frame information for noise reduction processing, thereby improving the speech noise reduction effect.

[0053] Furthermore, based on the above-mentioned first embodiment, a second embodiment of the speech noise reduction method of the present invention is proposed. In this embodiment, step S10 includes:

[0054] Step S101, acquiring amplitude spectrum data, real spectrum data, and imaginary spectrum data of a plurality of signal frames;

[0055] The amplitude spectrum data, real spectrum data and imaginary spectrum data can be obtained by converting the signal frame from the time domain to the frequency domain, for example, by short-time Fourier transform. Assuming there are T signal frames and F frequencies, the amplitude spectrum data, real spectrum data and imaginary spectrum data can be expressed as X m ∈R F×T 、X r ∈R F×T 、X i ∈R F×T By obtaining amplitude spectrum data, real spectrum data, and imaginary spectrum data, it is possible to utilize both amplitude information and phase information, thereby further improving the noise reduction effect of speech noise reduction.

[0056] The step S20 includes:

[0057] Step S201: input the amplitude spectrum data, the real spectrum data and the imaginary spectrum data into a full-band model, and extract the full-band amplitude spectrum information, full-band real spectrum information and full-band imaginary spectrum information corresponding to each signal frame.

[0058] In a specific embodiment, a full-band extractor for three types of spectrum data can be set in the full-band model, and the noise reduction device can input the amplitude spectrum data, real spectrum data and imaginary spectrum data into the corresponding full-band extractor respectively, and extract the full-band amplitude spectrum information, full-band real spectrum information and full-band imaginary spectrum information corresponding to each signal frame. For example, the full-band amplitude spectrum information, the full-band real spectrum information and the full-band imaginary spectrum information can be expressed as Ψ m ∈R F×T , Ψ r ∈R F×T , Ψ i∈R F×T The specific implementation of the full-band extractor is not limited in this embodiment, and for example, it can be implemented by using a TNC (Temporal Convolutional Network).

[0059] Furthermore, in a feasible implementation, the full-band amplitude spectrum information, full-band real spectrum information, full-band imaginary spectrum information, and voice activity detection information of each signal frame can be spliced ​​together and then input into a sub-band model for prediction to obtain clean speech corresponding to each signal frame. For example, for any signal frame in each signal frame and any frequency in each frequency, the noise reduction device can splice the full-band information corresponding to the frequency in the full-band amplitude spectrum information of the signal frame (i.e., full-band frequency domain feature information), the full-band information corresponding to the frequency in the full-band real spectrum information of the signal frame (i.e., full-band frequency domain feature information), and the full-band information corresponding to the frequency in the full-band imaginary spectrum information of the signal frame (i.e., full-band frequency domain feature information) with the voice activity detection information of the signal frame to obtain a sub-band model input sequence for the frequency of the signal frame, and input the sub-band model input sequence for each frequency of each signal frame into the sub-band model for prediction to obtain clean speech corresponding to each signal frame. For example, assuming there are T signal frames and F frequencies, the voice activity detection information for the t-th signal frame can be expressed as Ψ s t ∈R, for the fth frequency of the tth signal frame, its full-band information in the full-band real spectrum information can be expressed as Ψ r t，f ∈R, the full-band information in the full-band imaginary spectrum can be expressed as Ψ i t，f ∈R, the full-band information in the full-band amplitude spectrum can be expressed as Ψ m t，f ∈R, then the sub-band model input sequence of the spliced ​​t-th signal frame at the f-th frequency can be expressed as Ψ n t，f ∈R 4 The sub-band model input sequence of the f-th frequency of T signal frames can be used as an independent input sequence. A total of F input sequences can be input into the sub-band model for prediction to obtain the clean spectrum data of T signal frames at each frequency. Then, through frequency domain to time domain conversion, the clean spectrum data is converted into the clean speech signal corresponding to each signal frame.

[0060] Furthermore, in a feasible implementation manner, step S201 includes:

[0061] Step S2011, inputting the amplitude spectrum data, the real spectrum data and the imaginary spectrum data into the amplitude spectrum weighting module, the real spectrum weighting module and the imaginary spectrum weighting module in the full-band model respectively for frequency band weighting, to obtain weighted amplitude spectrum data, weighted real spectrum data and weighted imaginary spectrum data corresponding to each signal frame;

[0062] Step S2012, input the weighted amplitude spectrum data, the weighted real spectrum data and the weighted imaginary spectrum data into the amplitude full-band extractor, the real full-band extractor and the imaginary full-band extractor in the full-band model respectively, and extract the full-band amplitude spectrum information, full-band real spectrum information and full-band imaginary spectrum information corresponding to each signal frame.

[0063] Since different frequencies have different effects on speech noise reduction, in this embodiment, by weighting the amplitude spectrum data, real spectrum data and imaginary spectrum data for each frequency in the full-band model for the amplitude spectrum weighting module, the real spectrum weighting module and the imaginary spectrum weighting module, the weights corresponding to different frequencies may be different, so that the amplitude, real number and imaginary number data of the frequencies that have a deeper impact on speech noise reduction can provide more information for predicting clean speech. Assuming there are T signal frames and F frequencies, the weighted amplitude spectrum data, weighted real number spectrum data and weighted imaginary number spectrum data can be expressed as

[0064] The specific implementation of the amplitude spectrum weighting module, the real spectrum weighting module, and the imaginary spectrum weighting module is not limited in this embodiment. For example, in one feasible implementation, several parallel one-dimensional depth convolutions can be used with different kernel sizes, k1 (small), k2 (medium), and k3 (large), to extract different time-scale features for each frequency along the time axis; the convolution is followed by an average pooling layer and a rectified linear unit (ReLU) activation function to generate different time-scale features; then a fully connected layer is used to fuse the three features into a fused feature. Finally, two fully connected layers are stacked to learn weights from the fused features and use the weights to weight the spectral data.

[0065] Furthermore, in a feasible implementation manner, step S30 includes:

[0066] Step S301: for any target signal frame in each of the signal frames, and for any target frequency, splicing the full-band information corresponding to the target frequency in the full-band amplitude spectrum information corresponding to the target signal frame, the full-band information corresponding to the target frequency in the full-band real spectrum information corresponding to the target signal frame, the full-band information corresponding to the target frequency in the full-band imaginary spectrum information corresponding to the target signal frame, and the voice activity detection information corresponding to the target signal frame to obtain a sub-band model input sequence of the target frequency corresponding to the target signal frame;

[0067] Step S302: Input the sub-band model input sequence of each frequency corresponding to each signal frame into the sub-band model for prediction to obtain the clean speech corresponding to each signal frame.

[0068] Any signal frame in each signal frame is called a target signal frame, and any frequency in each frequency is called a target frequency to distinguish them. For example, assuming there are T signal frames and F frequencies, the voice activity detection information for the tth signal frame can be expressed as Ψ s t ∈R, for the fth frequency (target frequency) of the tth signal frame (target signal frame), its full-band information in the full-band real spectrum information can be expressed as Ψ r t，f ∈R, the full-band information in the full-band imaginary spectrum can be expressed as Ψ i t，f ∈R, the full-band information in the full-band amplitude spectrum can be expressed as Ψ m t，f ∈R, then the sub-band model input sequence of the spliced ​​t-th signal frame at the f-th frequency can be expressed as Ψ n t，f ∈R 4 The sub-band model input sequence of the f-th frequency of T signal frames can be used as an independent input sequence. A total of F input sequences can be input into the sub-band model for prediction to obtain the clean spectrum data of T signal frames at each frequency. Then, through frequency domain to time domain conversion, the clean spectrum data is converted into the clean speech signal corresponding to each signal frame.

[0069] Furthermore, in a feasible implementation manner, step S302 includes:

[0070] Step S3021, concatenating the weighted data of the target frequency in the weighted amplitude spectrum data corresponding to the target signal frame with the weighted data of multiple frequencies adjacent to the target frequency to obtain sub-band data of the target frequency corresponding to the target signal frame;

[0071] The weighted amplitude spectrum data includes the weighted data corresponding to each frequency (that is, the full-band amplitude characteristic information after weighting of each frequency). The number of multiple frequencies adjacent to the target frequency can be set as needed and is not limited here. Assume that N frequencies are selected on the left and right sides of the target frequency (f), and the weighted data of the target frequency and the weighted data of these 2N frequencies are spliced ​​to obtain the sub-band data of the target frequency corresponding to the target signal frame, for example, expressed as Ψ O t，f ∈R 2N+1 .

[0072] Step S3022: splice the sub-band data of the target frequency corresponding to the target signal frame and the sub-band model input sequence, and input the concatenated data into the sub-band model for prediction to obtain clean speech corresponding to the target signal frame.

[0073] For example, the sub-band model input sequence of the t-th signal frame at the f-th frequency is expressed as Ψ n t，f ∈R 4 , the sub-band data of the fth frequency of the tth signal frame is expressed as Ψ O t，f ∈R 2N+1 , then, after splicing, it can be expressed as Ψ c t，f ∈R (2N +5) , the splicing result of the fth frequency of T signal frames can be used as an independent input sequence. A total of F input sequences can be input into the sub-band model for prediction to obtain the clean spectrum data of T signal frames at each frequency. Then, through the conversion from frequency domain to time domain, the clean spectrum data can be converted into the clean speech signal corresponding to each signal frame.

[0074] Furthermore, in a feasible implementation manner, step S302 includes:

[0075] Step S3023, inputting the sub-band model input sequence of each frequency corresponding to each signal frame into the sub-band model for prediction, to obtain a complex ideal ratio mask corresponding to each signal frame;

[0076] Step S3024: Calculate the clean speech corresponding to each signal frame according to the complex ideal ratio mask and the spectrum data.

[0077] Given the complex spectrum Y of the noisy signal t,f , the complex spectrum S of the clean speech t,f , there is the following relationship:

[0078] S t,f =Mt,f *Y t,f

[0079] Among them, M t,f is the complex ideal ratio mask, * represents complex multiplication, S t,f 、M t,f 、Y t,f are all complex numbers. In this embodiment, the complex ideal ratio mask is used as the prediction target, and the sub-band model of each frequency corresponding to each signal frame is input into the sub-band model for prediction (or after being spliced ​​with the sub-band data), and the complex ideal ratio mask corresponding to each signal frame is obtained. Then, the clean speech is calculated based on the complex ideal ratio mask and the spectrum data, that is, the clean speech is obtained by formula S t,f =M t,f *Y t,f Calculate the clean speech.

[0080] In addition, an embodiment of the present invention further provides a speech noise reduction device, the device comprising:

[0081] an acquisition module, configured to acquire spectrum data of a plurality of signal frames, and perform voice activity detection on each of the signal frames to obtain voice activity detection information of each of the signal frames;

[0082] An extraction module, configured to input the spectrum data into a full-band model to extract full-band spectrum information corresponding to each signal frame;

[0083] The prediction module is used to input the full-band spectrum information and the voice activity detection information into a sub-band model for prediction to obtain the clean speech corresponding to each signal frame.

[0084] Furthermore, the acquisition module is further configured to:

[0085] Acquire amplitude spectrum data, real spectrum data, and imaginary spectrum data of multiple signal frames;

[0086] The extraction module is further configured to:

[0087] The amplitude spectrum data, the real spectrum data and the imaginary spectrum data are input into a full-band model to extract the full-band amplitude spectrum information, the full-band real spectrum information and the full-band imaginary spectrum information corresponding to each signal frame.

[0088] Furthermore, the extraction module is further configured to:

[0089] The amplitude spectrum data, the real spectrum data and the imaginary spectrum data are respectively input into the amplitude spectrum weighting module, the real spectrum weighting module and the imaginary spectrum weighting module in the full-band model for frequency band weighting, so as to obtain weighted amplitude spectrum data, weighted real spectrum data and weighted imaginary spectrum data corresponding to each signal frame;

[0090] The weighted amplitude spectrum data, the weighted real spectrum data and the weighted imaginary spectrum data are respectively input into the amplitude full-band extractor, the real full-band extractor and the imaginary full-band extractor in the full-band model to extract the full-band amplitude spectrum information, the full-band real spectrum information and the full-band imaginary spectrum information corresponding to each signal frame.

[0091] Furthermore, the prediction module is further configured to:

[0092] For any target signal frame in each of the signal frames, and for any target frequency, splicing the full-band information corresponding to the target frequency in the full-band amplitude spectrum information corresponding to the target signal frame, the full-band information corresponding to the target frequency in the full-band real spectrum information corresponding to the target signal frame, the full-band information corresponding to the target frequency in the full-band imaginary spectrum information corresponding to the target signal frame, and the voice activity detection information corresponding to the target signal frame to obtain a sub-band model input sequence of the target frequency corresponding to the target signal frame;

[0093] The sub-band model input sequence of each frequency corresponding to each signal frame is input into the sub-band model for prediction to obtain the clean speech corresponding to each signal frame.

[0094] Furthermore, the prediction module is further configured to:

[0095] splicing the weighted data of the target frequency in the weighted amplitude spectrum data corresponding to the target signal frame with the weighted data of multiple frequencies adjacent to the target frequency to obtain sub-band data of the target frequency corresponding to the target signal frame;

[0096] The sub-band data of the target frequency corresponding to the target signal frame and the sub-band model input sequence are spliced ​​and then input into the sub-band model for prediction to obtain the clean speech corresponding to the target signal frame.

[0097] Furthermore, the prediction module is further configured to:

[0098] Inputting the sub-band model sequence of each frequency corresponding to each signal frame into the sub-band model for prediction, to obtain a complex ideal ratio mask corresponding to each signal frame;

[0099] The clean speech corresponding to each of the signal frames is obtained by calculation according to the complex ideal ratio mask and the spectrum data.

[0100] Furthermore, the acquisition module is further configured to:

[0101] Obtaining an audio signal to be denoised;

[0102] Performing frame processing on the audio signal to obtain multiple signal frames;

[0103] Each of the signal frames is converted from the time domain to the frequency domain to obtain spectrum data.

[0104] In addition, the embodiment of the present invention also proposes a speech noise reduction device, such as Figure 2 As shown, Figure 2 It should be noted that the speech noise reduction device of the embodiment of the present invention can be a headset, a smart phone, a personal computer, a server, etc., and is not specifically limited here.

[0105] like Figure 2 As shown, the speech noise reduction device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0106] Those skilled in the art will understand that Figure 2 The device structure shown in the figure does not constitute a limitation on the speech noise reduction device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0107] like Figure 2 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a voice noise reduction program. The operating system is a program that manages and controls the hardware and software resources of the device and supports the operation of the voice noise reduction program and other software or programs. Figure 2 In the device shown, the user interface 1003 is mainly used to communicate data with the client; the network interface 1004 is mainly used to establish a communication connection with the server; and the processor 1001 can be used to call the speech noise reduction program stored in the memory 1005 and perform the following operations:

[0108] Acquiring spectrum data of a plurality of signal frames, and performing voice activity detection on each of the signal frames to obtain voice activity detection information of each of the signal frames;

[0109] Inputting the spectrum data into a full-band model to extract full-band spectrum information corresponding to each signal frame;

[0110] The full-band spectrum information and the voice activity detection information are input into a sub-band model for prediction to obtain clean speech corresponding to each signal frame.

[0111] Furthermore, the operation of obtaining spectrum data of multiple signal frames includes:

[0112] Acquire amplitude spectrum data, real spectrum data, and imaginary spectrum data of multiple signal frames;

[0113] The operation of inputting the spectrum data into the full-band model to extract the full-band spectrum information corresponding to each signal frame includes:

[0114] The amplitude spectrum data, the real spectrum data and the imaginary spectrum data are input into a full-band model to extract the full-band amplitude spectrum information, the full-band real spectrum information and the full-band imaginary spectrum information corresponding to each signal frame.

[0115] Furthermore, the operation of inputting the amplitude spectrum data, the real spectrum data and the imaginary spectrum data into a full-band model to extract the full-band amplitude spectrum information, the full-band real spectrum information and the full-band imaginary spectrum information corresponding to each signal frame includes:

[0116] The amplitude spectrum data, the real spectrum data and the imaginary spectrum data are respectively input into the amplitude spectrum weighting module, the real spectrum weighting module and the imaginary spectrum weighting module in the full-band model for frequency band weighting, so as to obtain weighted amplitude spectrum data, weighted real spectrum data and weighted imaginary spectrum data corresponding to each signal frame;

[0117] The weighted amplitude spectrum data, the weighted real spectrum data and the weighted imaginary spectrum data are respectively input into the amplitude full-band extractor, the real full-band extractor and the imaginary full-band extractor in the full-band model to extract the full-band amplitude spectrum information, the full-band real spectrum information and the full-band imaginary spectrum information corresponding to each signal frame.

[0118] Furthermore, the operation of inputting the full-band spectrum information and the voice activity detection information into a sub-band model for prediction to obtain clean speech corresponding to each signal frame includes:

[0119] For any target signal frame in each of the signal frames, and for any target frequency, splicing the full-band information corresponding to the target frequency in the full-band amplitude spectrum information corresponding to the target signal frame, the full-band information corresponding to the target frequency in the full-band real spectrum information corresponding to the target signal frame, the full-band information corresponding to the target frequency in the full-band imaginary spectrum information corresponding to the target signal frame, and the voice activity detection information corresponding to the target signal frame to obtain a sub-band model input sequence of the target frequency corresponding to the target signal frame;

[0120] The sub-band model input sequence of each frequency corresponding to each signal frame is input into the sub-band model for prediction to obtain the clean speech corresponding to each signal frame.

[0121] Furthermore, the operation of inputting the sub-band model input sequence of each frequency corresponding to each signal frame into the sub-band model for prediction to obtain the clean speech corresponding to each signal frame includes:

[0122] splicing the weighted data of the target frequency in the weighted amplitude spectrum data corresponding to the target signal frame with the weighted data of multiple frequencies adjacent to the target frequency to obtain sub-band data of the target frequency corresponding to the target signal frame;

[0123] The sub-band data of the target frequency corresponding to the target signal frame and the sub-band model input sequence are spliced ​​and then input into the sub-band model for prediction to obtain the clean speech corresponding to the target signal frame.

[0124] Furthermore, the operation of inputting the sub-band model input sequence of each frequency corresponding to each signal frame into the sub-band model for prediction to obtain the clean speech corresponding to each signal frame includes:

[0125] Inputting the sub-band model sequence of each frequency corresponding to each signal frame into the sub-band model for prediction, to obtain a complex ideal ratio mask corresponding to each signal frame;

[0126] The clean speech corresponding to each of the signal frames is obtained by calculation according to the complex ideal ratio mask and the spectrum data.

[0127] Furthermore, the operation of obtaining spectrum data of multiple signal frames includes:

[0128] Obtaining an audio signal to be denoised;

[0129] Performing frame processing on the audio signal to obtain a plurality of signal frames;

[0130] Each of the signal frames is converted from the time domain to the frequency domain to obtain spectrum data.

[0131] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a speech noise reduction program is stored. When the speech noise reduction program is executed by a processor, the steps of the speech noise reduction method described below are implemented.

[0132] The various embodiments of the speech noise reduction device and the computer-readable storage medium of the present invention can refer to the various embodiments of the speech noise reduction method of the present invention, and will not be repeated here.

[0133] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0134] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0135] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0136] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A speech noise reduction method, characterized in that: The method comprises the following steps: Acquiring spectrum data of a plurality of signal frames, and performing voice activity detection on each of the signal frames to obtain voice activity detection information of each of the signal frames; Inputting the spectrum data into a full-band model to extract full-band spectrum information corresponding to each signal frame, wherein the full-band spectrum information includes global frequency domain feature information corresponding to each frequency in the full band; Inputting the full-band spectrum information and the voice activity detection information into a sub-band model for prediction to obtain clean speech corresponding to each signal frame; The step of inputting the full-band spectrum information and the voice activity detection information into a sub-band model for prediction to obtain clean speech corresponding to each signal frame comprises: For any target signal frame in each of the signal frames and any target frequency in each of the frequencies, concatenate the full-band frequency domain feature information corresponding to the target frequency of the target signal frame with the voice activity detection information of the target signal frame to obtain a sub-band model input sequence of the target frequency corresponding to the target signal frame; The sub-band model input sequence of each frequency corresponding to each signal frame is input into the sub-band model for prediction to obtain the clean speech corresponding to each signal frame.

2. The speech noise reduction method according to claim 1, wherein: The step of obtaining spectrum data of a plurality of signal frames comprises: Acquire amplitude spectrum data, real spectrum data, and imaginary spectrum data of multiple signal frames; The step of inputting the spectrum data into a full-band model to extract the full-band spectrum information corresponding to each signal frame comprises: The amplitude spectrum data, the real spectrum data and the imaginary spectrum data are input into a full-band model to extract the full-band amplitude spectrum information, the full-band real spectrum information and the full-band imaginary spectrum information corresponding to each signal frame.

3. The speech noise reduction method according to claim 2, wherein: The step of inputting the amplitude spectrum data, the real spectrum data and the imaginary spectrum data into a full-band model to extract the full-band amplitude spectrum information, the full-band real spectrum information and the full-band imaginary spectrum information corresponding to each signal frame comprises: The amplitude spectrum data, the real spectrum data and the imaginary spectrum data are respectively input into the amplitude spectrum weighting module, the real spectrum weighting module and the imaginary spectrum weighting module in the full-band model for frequency band weighting, so as to obtain weighted amplitude spectrum data, weighted real spectrum data and weighted imaginary spectrum data corresponding to each signal frame; The weighted amplitude spectrum data, the weighted real spectrum data and the weighted imaginary spectrum data are respectively input into the amplitude full-band extractor, the real full-band extractor and the imaginary full-band extractor in the full-band model to extract the full-band amplitude spectrum information, the full-band real spectrum information and the full-band imaginary spectrum information corresponding to each signal frame.

4. The speech noise reduction method according to claim 3, wherein: The step of inputting the full-band spectrum information and the voice activity detection information into a sub-band model for prediction to obtain clean speech corresponding to each signal frame comprises: For any target signal frame in each of the signal frames, and for any target frequency, splicing the full-band information corresponding to the target frequency in the full-band amplitude spectrum information corresponding to the target signal frame, the full-band information corresponding to the target frequency in the full-band real spectrum information corresponding to the target signal frame, the full-band information corresponding to the target frequency in the full-band imaginary spectrum information corresponding to the target signal frame, and the voice activity detection information corresponding to the target signal frame to obtain a sub-band model input sequence of the target frequency corresponding to the target signal frame; The sub-band model input sequence of each frequency corresponding to each signal frame is input into the sub-band model for prediction to obtain the clean speech corresponding to each signal frame.

5. The speech noise reduction method according to claim 4, wherein: The step of inputting the sub-band model of each frequency corresponding to each signal frame into a sub-band model for prediction to obtain clean speech corresponding to each signal frame includes: splicing the weighted data of the target frequency in the weighted amplitude spectrum data corresponding to the target signal frame with the weighted data of multiple frequencies adjacent to the target frequency to obtain sub-band data of the target frequency corresponding to the target signal frame; The sub-band data of the target frequency corresponding to the target signal frame and the sub-band model input sequence are spliced ​​and then input into the sub-band model for prediction to obtain the clean speech corresponding to the target signal frame.

6. The method for reducing speech noise according to claim 4, wherein: The step of inputting the sub-band model of each frequency corresponding to each signal frame into a sub-band model for prediction to obtain clean speech corresponding to each signal frame includes: Inputting the sub-band model sequence of each frequency corresponding to each signal frame into the sub-band model for prediction, to obtain a complex ideal ratio mask corresponding to each signal frame; The clean speech corresponding to each of the signal frames is obtained by calculation according to the complex ideal ratio mask and the spectrum data.

7. The speech noise reduction method according to any one of claims 1 to 6, wherein: The step of obtaining spectrum data of a plurality of signal frames comprises: Obtaining an audio signal to be denoised; Performing frame processing on the audio signal to obtain multiple signal frames; Each of the signal frames is converted from the time domain to the frequency domain to obtain spectrum data.

8. A speech noise reduction device, characterized in that: The device comprises: an acquisition module, configured to acquire spectrum data of a plurality of signal frames, and perform voice activity detection on each of the signal frames to obtain voice activity detection information of each of the signal frames; An extraction module is configured to input the spectrum data into a full-band model to extract full-band spectrum information corresponding to each signal frame, wherein the full-band spectrum information includes global frequency domain feature information corresponding to each frequency in the full band; A prediction module, configured to input the full-band spectrum information and the voice activity detection information into a sub-band model for prediction to obtain clean speech corresponding to each signal frame; The prediction module is also used to: For any target signal frame in each of the signal frames and any target frequency in each of the frequencies, concatenate the full-band frequency domain feature information corresponding to the target frequency of the target signal frame with the voice activity detection information of the target signal frame to obtain a sub-band model input sequence of the target frequency corresponding to the target signal frame; The sub-band model input sequence of each frequency corresponding to each signal frame is input into the sub-band model for prediction to obtain the clean speech corresponding to each signal frame.

9. A speech noise reduction device, characterized in that: The speech noise reduction device includes: a memory, a processor, and a speech noise reduction program stored in the memory and executable on the processor. When the speech noise reduction program is executed by the processor, the steps of the speech noise reduction method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a speech noise reduction program, which, when executed by a processor, implements the steps of the speech noise reduction method according to any one of claims 1 to 7.

Images