Multi-channel speech enhancement method based on time-frequency-null masking and computer equipment

Abstract

The invention belongs to the field of multi-channel voice communication, and discloses a multi-channel voice enhancement method based on time-frequency-null masking and computer equipment, and the method comprises the steps: obtaining the time domain data of a multi-channel noisy voice, carrying out the frequency domain conversion, and obtaining the time-frequency domain data of the multi-channel noisy voice through combining a time index in the time domain data; converting the time-frequency domain data of the multi-channel noisy voice into time-frequency-space domain data based on the transformation matrix satisfying the space base; inputting the time-frequency spatial domain data of the multi-channel noisy voice into the 3D U-Net model to obtain a time-frequency spatial domain mask value output by the 3D U-Net model; performing masking operation on the time-frequency space-domain data of the multi-channel noisy voice by using the time-frequency space-domain mask value to obtain the time-frequency space-domain data of the denoised target voice; and recovering the time-frequency-space domain data of the target voice after noise reduction to the time domain, and outputting the time domain data of the target voice after noise reduction to complete multi-channel voice enhancement. The performance of the speech enhancement method in the complex environment is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of multi-channel voice communication, and specifically relates to a multi-channel voice enhancement method and computer equipment based on time-frequency-space masking. When the signal collected and transmitted by a microphone array is polluted by a large amount of background noise, the signal processing technology of the present invention is used to achieve the purpose of voice enhancement, improve the intelligibility of voice, improve the voice listening experience, and improve communication efficiency. Background Art

[0002] In the application of microphone arrays, multi-channel speech enhancement technology has stronger robustness than single-channel speech processing. Microphone arrays can provide spatial diversity. With the help of beamforming technology in the field of array signal processing, the target speech signal can be separated from the input audio signal contaminated by noise. The so-called beamforming is to design appropriate weights for different channel speech to achieve the purpose of enhancing the target speech beam while suppressing the interference beam. Common beamforming algorithms are divided into two categories. The first category: fixed-weight beamforming, which compensates for the phase difference caused by the time difference of sound waves reaching different microphones to achieve the purpose of speech enhancement. It mainly includes delay and sum algorithm (DAS), weighted delay sum algorithm (Weighted DAS), etc. The performance of this type of algorithm is linearly related to the number of array microphones, and the fixed parameters of the filter are not suitable for complex and changeable actual scenarios. The second category: adaptive beamforming algorithm. Classic algorithms include: Linearly Constrained Minimum Variance Criteria Algorithm (LCMV) and Generalized Side-lobe Canceller (GSC). The two algorithms are proven to be equivalent. The GSC algorithm performs well in coherent noise fields, but when there are multiple interference sources, interference sources come from multiple directions, or the number of array microphones is insufficient, the GSC algorithm will cause the desired signal to leak and the noise cancellation ability to be insufficient.

[0003] Considering that the single-channel speech enhancement method has a good elimination effect on incoherent noise, the conventional multi-channel speech enhancement method will perform post-filtering after beamforming, that is, a single-channel noise reduction algorithm is used at the output of the beamforming to further filter out the residual noise. Common traditional single-channel noise reduction schemes make steady-state assumptions on the noise model or rely on prior knowledge of the signal-to-noise ratio, such as spectral subtraction, statistical model-based algorithms, subspace-based algorithms, etc. In view of this, more deep learning models are applied to speech enhancement technology, such as convolutional neural networks (CNN), recurrent neural networks, autoencoders, etc. Among them, a typical deep learning-based algorithm is time-frequency masking. Starting from the spectrogram, with the characteristics that the speech of interest is sparse or nearly sparse in both the time domain and the frequency domain, a reasonable training target is set, the neural network is trained, the output time-frequency mask and the input time-frequency data are Hadamard products, and finally restored to the time domain audio. Related studies have shown that the GSC algorithm post-time-frequency masking scheme can provide better speech enhancement effects, but in the case of many interference sources, insufficient beam resolution will still lead to performance degradation. Summary of the invention

[0004] One of the purposes of the present invention is to provide a multi-channel speech enhancement method based on time-frequency-space masking, which effectively improves the performance of the speech enhancement method in a complex environment.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A multi-channel speech enhancement method based on time-frequency-space masking, the multi-channel speech enhancement method based on time-frequency-space masking comprising:

[0007] Acquire time domain data of multi-channel noisy speech, perform frequency domain conversion on the time domain data of the multi-channel noisy speech, and obtain time-frequency domain data of the multi-channel noisy speech in combination with the time index in the time domain data;

[0008] Based on a transformation matrix that satisfies a spatial basis, the time-frequency domain data of the multi-channel noisy speech is converted into the time-frequency spatial domain data of the multi-channel noisy speech;

[0009] Inputting the time-frequency-spatial domain data of the multi-channel noisy speech into a 3D U-Net model to obtain a time-frequency-spatial domain mask value output by the 3D U-Net model;

[0010] Using the time-frequency-spatial-domain mask value to perform a masking operation on the time-frequency-spatial-domain data of the multi-channel noisy speech to obtain the time-frequency-spatial-domain data of the target speech after noise reduction;

[0011] The time-frequency spatial domain data of the target speech after noise reduction is restored to the time domain, and the time domain data of the target speech after noise reduction is output to complete multi-channel speech enhancement.

[0012] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution, but are merely further supplements or preferences. Under the premise that there are no technical or logical contradictions, each optional method can be combined with the above-mentioned overall solution separately, and multiple optional methods can also be combined.

[0013] Preferably, the step of performing frequency domain conversion on the time domain data of the multi-channel noisy speech and obtaining the time-frequency domain data of the multi-channel noisy speech in combination with the time index in the time domain data comprises:

[0014] Performing frame division and windowing processing on the time domain data of the multi-channel noisy speech;

[0015] Perform short-time Fourier transform on each frame of data, and combine the channel, frequency point and time to obtain the time-frequency domain data of multi-channel noisy speech, which is recorded as ,in is the channel index, is the frequency index, Index for the moment.

[0016] Preferably, the step of converting the time-frequency domain data of the multi-channel noisy speech into the time-frequency spatial domain data of the multi-channel noisy speech based on the transformation matrix satisfying the spatial basis comprises:

[0017] Using a 3D tensor Characterize the time-frequency domain data of multi-channel noisy speech, satisfying , and apply the transformation matrix that satisfies the spatial basis to the three-dimensional tensor Above, as follows:

[0018] ;

[0019] In the formula, is the time-frequency-spatial data of multi-channel noisy speech. represents the mode-1 product of a tensor, To satisfy the spatial basis and size The transformation matrix, is the total number of channels.

[0020] Preferably, when the 3D U-Net model is trained, the complex-valued ideal ratio masking value after hyperbolic tangent compression is used as the training target, and the formula is as follows:

[0021] ;

[0022] ;

[0023] ;

[0024] ;

[0025] In the formula, is the complex ideal ratio masking value after hyperbolic tangent compression corresponding to the real part of the time-frequency spatial domain data of the multi-channel noisy speech, is the complex ideal ratio masking value after hyperbolic tangent compression corresponding to the imaginary part of the time-frequency and spatial domain data of the multi-channel noisy speech, is the default value, is the steepness, is the real part simplified parameter, is the simplified parameter of the imaginary part, is the real part of the time-frequency-spatial domain data of multi-channel noisy speech in network training, is the imaginary part of the time-frequency-spatial domain data of multi-channel noisy speech in network training, is the real part of the time-frequency spatial domain data of the clean speech in network training, It is the imaginary part of the time-frequency spatial domain data of the clean speech in network training.

[0026] Preferably, when the 3D U-Net model is trained, the loss function calculation formula is as follows:

[0027] ;

[0028] ;

[0029] ;

[0030] In the formula, represents the loss value, represents the mean square error, Represents the time-frequency spatial domain data of clean speech in training, represents the time-frequency spatial domain data of the target speech after noise reduction estimated during training, represents the weighted source-distortion ratio loss, represents the time domain data of multi-channel noisy speech in training, represents the time domain data of clean speech in training, represents the time domain data of the target speech after noise reduction estimated during training, represents the weighted weight, Indicates the distortion ratio loss.

[0031] Preferably, the step of performing a masking operation on the time-frequency-spatial-domain data of the multi-channel noisy speech using the time-frequency-spatial-domain mask value to obtain the time-frequency-spatial-domain data of the target speech after noise reduction includes:

[0032] ;

[0033] In the formula, Represents the time-frequency spatial domain data of the target speech after noise reduction, Represents the time-frequency spatial domain mask value output by the 3D U-Net model, represents the tensor product, Represents the time-frequency and spatial domain data of multi-channel noisy speech.

[0034] Preferably, the step of restoring the time-frequency spatial domain data of the target speech after noise reduction to the time domain and outputting the time domain data of the target speech after noise reduction comprises:

[0035] ;

[0036] In the formula, Represents the three-dimensional tensor of the target speech after noise reduction, Represents the time-frequency spatial domain data of the target speech after noise reduction, represents the mode-1 product of a tensor, It means that it satisfies the spatial basis and has a size of The transformation matrix, represents the mode-2 product of a tensor, It means that it satisfies the spatial basis and has a size of The transformation matrix, represents conjugate transpose;

[0037] Shape it to The three-dimensional tensor Expand to , then The array is the time domain data of the target speech after noise reduction ,in, is the total number of channels, is the total number of frequency points, is the frame length, The total number of moments.

[0038] The present invention provides a multi-channel speech enhancement method based on time-frequency-space masking, which abandons the pre-beamforming algorithm, uses a neural network to learn the spatial domain characteristics of speech signals, replaces the pre-beamforming algorithm, and expands the time-frequency two-dimensional masking scheme to a time-frequency-space three-dimensional masking scheme, thereby achieving a better beamforming effect, thereby improving the performance of the speech enhancement algorithm in a more complex environment.

[0039] A second object of the present invention is to provide a computer device, comprising a processor and a memory storing a plurality of computer instructions, wherein the computer instructions, when executed by the processor, implement the steps of the multi-channel speech enhancement method based on time-frequency-space masking. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1A flow chart of a multi-channel speech enhancement method based on time-frequency-space masking of the present invention;

[0041] Figure 2 This is a schematic diagram of the data flow of the 3D U-Net model in Example 1 of the present invention;

[0042] Figure 3 Schematic diagram of data flow of the 3D U-Net model in Example 2 of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0045] Example 1

[0046] In the field of multi-channel speech enhancement, the target speech signal has sparse characteristics not only in the time-frequency domain, but also in the space domain. To overcome the defects of the prior art, this embodiment provides a multi-channel speech enhancement method based on time-frequency-space masking.

[0047] Consider a scenario where a uniform linear microphone array collects the target speech. The number of microphones is , the spacing is The collected noisy frequency duration is seconds, the sampling rate is , that is, the number of sampling points is Without loss of generality, we assume that the target sound source is in the far field of the microphone array. The speech data collected by the microphone array is recorded as , is a Assume that The clean speech signal emitted by the sound source, is a noise signal, then It can be expressed as:

[0048] ;

[0049] The microphone array responds The size is The vector of the microphone spacing , wavelength, and the angle at which the transmitted sound wave reaches the array. Represents the time domain data of the clean speech signal collected by the microphone array, is transposed.

[0050] The existing beamforming post-time-frequency masking method is divided into two steps: ① Perform traditional beamforming processing, such as using the GSC algorithm, to output a single-channel signal, and further obtain the frequency domain information of the single-channel signal, that is, corresponding to the spectrogram. The size of each time-frequency unit data represents the time-frequency relationship between clean speech and noise. ② Use the single-channel time-frequency data after beamforming as the input of the neural network model, set appropriate training targets, such as the complex ideal ratio mask (cIRM) value, and train the neural network model CNN to obtain the speech features in the noisy speech spectrum. This solution can have a good suppression effect on incoherent noise when the number of microphone array elements is sufficient, but when the number of microphones is small, the spatial resolution is far from enough, resulting in insufficient beamforming effect, which in turn affects the speech enhancement performance.

[0051] In order to accurately obtain spatial domain features, this embodiment uses a deep neural network to learn the three-dimensional features of the time-frequency and spatial domains and predict the time-frequency and spatial masks. Figure 1 The specific details are as follows:

[0052] Step S1, obtaining time domain data of multi-channel noisy speech, performing frequency domain conversion on the time domain data of the multi-channel noisy speech, and obtaining time-frequency domain data of the multi-channel noisy speech in combination with a time index in the time domain data.

[0053] Step S1.1: Frame and window the time domain data of multi-channel noisy speech and set the frame length to , the frame shift is , the total number of frames is recorded as In this embodiment, Take the length of The Hamming window is used to perform windowing processing on the framed data.

[0054] Step S1.2: Perform short-time Fourier transform on each frame of data, and combine the channel, frequency point and time to obtain the time-frequency domain data of multi-channel noisy speech, which is recorded as ,in is the channel index, is the frequency index, is the time index (that is, the frame index).

[0055] Step S2: based on a transformation matrix satisfying a spatial basis, converting the time-frequency domain data of the multi-channel noisy speech into the time-frequency spatial domain data of the multi-channel noisy speech.

[0056] Using a 3D tensor Characterize the time-frequency domain data of multi-channel noisy speech, satisfying , design the transformation matrix , acting on the array dimensions, intending to transform the time-frequency domain data into the time-frequency-space domain. Apply the transformation matrix that satisfies the spatial basis to the three-dimensional tensor Above, as follows:

[0057] ;

[0058] In the formula, is the time-frequency-spatial data of multi-channel noisy speech. represents the mode-1 product (matrix-tensor product) of tensors, that is, Left-multiply the transformation matrix on one dimension . To satisfy the spatial basis and size The transformation matrix, is the total number of channels. Take this as an example to illustrate.

[0059] This embodiment adopts a relatively simple The form is The DFT (Discrete Fourier Transform) matrix of is called the time-frequency-spatial information of the multi-channel noisy speech. Can be decomposed into two parts ,in and The time-frequency and spatial domain data corresponding to clean speech and noise information respectively. The time-frequency and spatial domain information is complex valued. When used as input to the neural network model, their real and imaginary parts will be separated, that is, Reorganize to size Tensor array, this embodiment uses Take this as an example to illustrate.

[0060] Step S3: input the time-frequency-spatial domain data of the multi-channel noisy speech into the 3D U-Net model to obtain the time-frequency-spatial domain mask value output by the 3D U-Net model.

[0061] ① Network structure: such as Figure 2As shown, analogous to the 3D image processing method, this embodiment adopts a 3D U-Net model structure, and feature extraction and output are completed by a convolutional neural network. The network adopts a jump connection method to effectively transmit feature information while better avoiding the gradient vanishing problem during back propagation, thereby improving network performance. The 3D U-Net model structure includes mirrored encoder and decoder convolutional neural network layers. Both the encoder and the decoder are composed of four convolution blocks. Each convolution block of the encoder has 2 three-dimensional convolutional layers (Conv) and 1 three-dimensional maximum pooling layer (Max-Pool). Each convolution block of the decoder contains 1 three-dimensional upsampling layer (Up-Sample) and 2 three-dimensional convolutional layers. Each convolution layer is followed by a batch normalization layer (BN) and a ReLU activation layer. The encoder downsamples the time, frequency, and spatial dimensions to reduce resolution, but increases the number of channels to extract high-level semantic information. The decoder reduces the number of channels and increases data details.

[0062] ② Network learning target: This embodiment uses the complex ideal ratio masking value as the training target, which includes the amplitude and phase characteristics of the speech signal. The standard cIRM calculation formula is:

[0063] ;

[0064] in, is the noisy spectrum of multi-channel speech, is the clean speech spectrum, subscript Indicates taking the real part, subscript In order to facilitate network training, this embodiment performs hyperbolic tangent compression cIRM, that is:

[0065] ;

[0066] ;

[0067] ;

[0068] ;

[0069] In the formula, is the complex ideal ratio masking value after hyperbolic tangent compression corresponding to the real part of the time-frequency spatial domain data of the multi-channel noisy speech, is the complex ideal ratio masking value after hyperbolic tangent compression corresponding to the imaginary part of the time-frequency and spatial domain data of the multi-channel noisy speech, is a preset value, which is 10 in this embodiment. is the steepness, and in this embodiment, the value is 0.1. is the real part simplified parameter, is the simplified parameter of the imaginary part, is the real part of the time-frequency-spatial domain data of multi-channel noisy speech in network training, is the imaginary part of the time-frequency-spatial domain data of multi-channel noisy speech in network training, is the real part of the time-frequency spatial domain data of the clean speech in network training, It is the imaginary part of the time-frequency spatial domain data of the clean speech in network training.

[0070] In this embodiment, the cIRM value is compressed to Similarly, in this embodiment, the spectrum data is replaced by the time-frequency spatial domain information data. , by inputting the 3D U-Net model, a three-dimensional time-frequency space mask value can be obtained, which is the model training target.

[0071] ③ Loss function: The loss function set in this embodiment includes two parts: the first part is to estimate the time-frequency-space information of the speech Time-frequency-spatial information with clean speech The second part is the weighted source-to-distortion ratio loss (weighted Source-to-Distortion Ratio, w-SDR), which is specifically characterized as:

[0072] ;

[0073] in , is the distortion ratio loss, expressed as:

[0074] ;

[0075] In the formula, represents the loss value, represents the mean square error, Represents the time-frequency spatial domain data of clean speech in training, represents the time-frequency spatial domain data of the target speech after noise reduction estimated during training, represents the weighted source-distortion ratio loss, represents the time domain data of multi-channel noisy speech in training, represents the time domain data of clean speech in training, represents the time domain data of the target speech after noise reduction estimated during training, represents the weighted weight, represents the distortion ratio loss, where clean speech is used as the label in training.

[0076] Step S4: masking the time-frequency-spatial domain data of the multi-channel noisy speech using the time-frequency-spatial domain mask value to obtain the time-frequency-spatial domain data of the target speech after noise reduction.

[0077] The predicted time-frequency spatial mask value is obtained from the output of the 3D-UNet model , you can get the time-frequency spatial domain data of the target speech after noise reduction:

[0078] ;

[0079] In the formula, Represents the time-frequency spatial domain data of the target speech after noise reduction (the shape is ), Represents the time-frequency spatial domain mask value output by the 3DU-Net model, represents the tensor product, Represents the time-frequency and spatial domain data of multi-channel noisy speech.

[0080] Step S5, restore the time-frequency spatial domain data of the target speech after noise reduction to the time domain, output the time domain data of the target speech after noise reduction, and complete the multi-channel speech enhancement. Finally, the time domain target speech is restored as follows:

[0081] ;

[0082] In the formula, Represents the three-dimensional tensor of the target speech after noise reduction, Represents the time-frequency spatial domain data of the target speech after noise reduction (the shape is ), represents the mode-1 product of a tensor, It means that it satisfies the spatial basis and has a size of The transformation matrix, represents the mode-2 product of a tensor, It means that it satisfies the spatial basis and has a size of The transformation matrix, represents the conjugate transpose.

[0083] Shape it to The three-dimensional tensor Expand to , then The array is the time domain data of the target speech after noise reduction ,in, is the total number of channels, is the total number of frequency points, is the frame length, is the total time, , , .

[0084] Example 2

[0085] Due to the symmetry of the amplitude spectrum, this embodiment provides a method for accelerating the calculation speed and reducing the calculation time based on the first embodiment. When processing data, only the data of the first half of the frequency points of the amplitude spectrum need to be taken for calculation, and the number of frequency points taken is recorded as , and restore it after denoising. The specific steps are as follows:

[0086] Step S1, obtaining time domain data of multi-channel noisy speech, performing frequency domain conversion on the time domain data of the multi-channel noisy speech, and obtaining time-frequency domain data of the multi-channel noisy speech in combination with a time index in the time domain data.

[0087] Step S1.1: Frame and window the time domain data of multi-channel noisy speech and set the frame length to , the frame shift is , the total number of frames is recorded as In this embodiment, Take the length of The Hamming window is used to perform windowing processing on the framed data.

[0088] Step S1.2: Perform short-time Fourier transform on each frame of data, and combine the channel, frequency point and time to obtain the time-frequency domain data of multi-channel noisy speech, which is recorded as ,in is the channel index, is the frequency index, is the time index (that is, the frame index).

[0089] Different from Example 1, due to the symmetry of the amplitude spectrum, this embodiment only needs to take the data of the first half of the frequency points of the amplitude spectrum for calculation, and the number of frequency points taken is recorded as .

[0090] Step S2: based on a transformation matrix satisfying a spatial basis, converting the time-frequency domain data of the multi-channel noisy speech into the time-frequency spatial domain data of the multi-channel noisy speech.

[0091] Using a 3D tensor Characterize the time-frequency domain data of multi-channel noisy speech, satisfying , design the transformation matrix , acting on the array dimensions, intending to transform the time-frequency domain data into the time-frequency-space domain. Apply the transformation matrix that satisfies the spatial basis to the three-dimensional tensor Above, as follows:

[0092] ;

[0093] In the formula, is the time-frequency-spatial data of multi-channel noisy speech. represents the mode-1 product (matrix-tensor product) of tensors, that is, Left-multiply the transformation matrix on one dimension . To satisfy the spatial basis and size The transformation matrix, is the total number of channels. Take this as an example to illustrate.

[0094] This embodiment adopts a relatively simple The form is The DFT (Discrete Fourier Transform) matrix of is called the time-frequency-spatial information of the multi-channel noisy speech. Can be decomposed into two parts ,in and The time-frequency and spatial domain data corresponding to clean speech and noise information respectively. The time-frequency and spatial domain information is complex valued. When used as input to the neural network model, their real and imaginary parts will be separated, that is, Reorganize to size Tensor array, this embodiment uses Take this as an example to illustrate.

[0095] Step S3: input the time-frequency-spatial domain data of the multi-channel noisy speech into the 3D U-Net model to obtain the time-frequency-spatial domain mask value output by the 3D U-Net model.

[0096] ① Network structure: Analogous to 3D image processing methods, this embodiment adopts a 3D U-Net model structure, and feature extraction and output are completed by convolutional neural networks. The network adopts a jump connection method to effectively transmit feature information while better avoiding the problem of gradient disappearance during back propagation, thereby improving network performance. The 3D U-Net model structure includes mirrored encoder and decoder convolutional neural network layers. Both the encoder and the decoder are composed of four convolution blocks. Each convolution block of the encoder has 2 three-dimensional convolutional layers (Conv) and 1 three-dimensional maximum pooling layer (Max-Pool). Each convolution block of the decoder contains 1 three-dimensional upsampling layer (Up-Sample) and 2 three-dimensional convolutional layers. Each convolution layer is followed by a batch normalization layer (BN) and a ReLU activation layer. The encoder downsamples the time, frequency, and spatial dimensions to reduce resolution, but increases the number of channels to extract high-level semantic information. The decoder will reduce the number of channels and increase data details.

[0097] ② Network learning target: This embodiment uses the complex ideal ratio masking value as the training target, which includes the amplitude and phase characteristics of the speech signal. The standard cIRM calculation formula is:

[0098] ;

[0099] in, is the noisy spectrum of multi-channel speech, is the clean speech spectrum, subscript Indicates taking the real part, subscript In order to facilitate network training, this embodiment performs hyperbolic tangent compression cIRM, that is:

[0100] ;

[0101] ;

[0102] ;

[0103] ;

[0104] In the formula, is the complex ideal ratio masking value after hyperbolic tangent compression corresponding to the real part of the time-frequency spatial domain data of the multi-channel noisy speech, is the complex ideal ratio masking value after hyperbolic tangent compression corresponding to the imaginary part of the time-frequency and spatial domain data of the multi-channel noisy speech, is a preset value, which is 10 in this embodiment. is the steepness, and in this embodiment, the value is 0.1. is the real part simplified parameter, is the simplified parameter of the imaginary part, is the real part of the time-frequency-spatial domain data of multi-channel noisy speech in network training, is the imaginary part of the time-frequency-spatial domain data of multi-channel noisy speech in network training, is the real part of the time-frequency spatial domain data of the clean speech in network training, It is the imaginary part of the time-frequency spatial domain data of the clean speech in network training.

[0105] In this embodiment, the cIRM value is compressed to Similarly, in this embodiment, the spectrum data is replaced by the time-frequency spatial domain information data. , by inputting the 3D U-Net model, a three-dimensional time-frequency space mask value can be obtained, which is the model training target. Figure 3 The 3D U-Net model structure and data flow diagram are shown. For example, when reconstructing the speech signal later, an inverse transform is required to restore the original cIRM value.

[0106] ③ Loss function: The loss function set in this embodiment includes two parts: the first part is to estimate the time-frequency-space information of the speech Time-frequency-spatial information with clean speech The second part is the weighted source-to-distortion ratio loss (weighted Source-to-Distortion Ratio, w-SDR), which is specifically characterized as:

[0107] ;

[0108] in , is the distortion ratio loss, expressed as:

[0109] ;

[0110] In the formula, represents the loss value, represents the mean square error, Represents the time-frequency spatial domain data of clean speech in training, represents the time-frequency spatial domain data of the target speech after noise reduction estimated during training, represents the weighted source-distortion ratio loss, represents the time domain data of multi-channel noisy speech in training, represents the time domain data of clean speech in training, represents the time domain data of the target speech after noise reduction estimated during training, represents the weighted weight, represents the distortion ratio loss, where clean speech is used as the label in training.

[0111] Step S4: masking the time-frequency-spatial domain data of the multi-channel noisy speech using the time-frequency-spatial domain mask value to obtain the time-frequency-spatial domain data of the target speech after noise reduction.

[0112] The predicted time-frequency spatial mask value is obtained from the output of the 3D-UNet model , you can get the time-frequency spatial domain data of the target speech after noise reduction:

[0113] ;

[0114] In the formula, represents the time-frequency-spatial data of the target speech after noise reduction (the time-frequency-spatial data of the target speech after noise reduction before completion), Represents the time-frequency spatial domain mask value output by the 3D U-Net model, represents the tensor product, Represents the time-frequency and spatial domain data of multi-channel noisy speech.

[0115] Step S5: restore the time-frequency spatial domain data of the target speech after noise reduction to the time domain, output the time domain data of the target speech after noise reduction, and complete multi-channel speech enhancement.

[0116] Since only half of the data is taken during calculation, the time-frequency spatial domain data of the target speech after noise reduction is currently half of the data, so the other half of the frequency domain information needs to be completed first. The shape of the time-frequency spatial domain data of the target speech after noise reduction before completion is , then the shape of the time-frequency spatial domain data of the denoised target speech after data symmetry completion is For example, if the number of FFT points in the frequency domain transform is 512, then after completion The size is . Finally, the time domain target speech is restored:

[0117] ;

[0118] In the formula, Represents the three-dimensional tensor of the target speech after noise reduction, represents the time-frequency and spatial data of the target speech after noise reduction (the time-frequency and spatial data of the target speech after noise reduction after completion), represents the mode-1 product of a tensor, It means that it satisfies the spatial basis and has a size of The transformation matrix, represents the mode-2 product of a tensor, It means that it satisfies the spatial basis and has a size of The transformation matrix, represents the conjugate transpose.

[0119] Shape it to The three-dimensional tensor Expand to , then The array is the time domain data of the target speech after noise reduction ,in, is the total number of channels, is the total number of frequency points, is the frame length, is the total time, , , .

[0120] Example 3

[0121] In another embodiment, the present invention further provides a computer device, including a processor and a memory storing a plurality of computer instructions, wherein the computer instructions, when executed by the processor, implement the steps of the multi-channel speech enhancement method based on time-frequency-space masking.

[0122] The memory and the processor are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The memory stores a computer program that can be run on the processor, and the processor implements the method of the present invention by running the computer program stored in the memory.

[0123] The memory may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory is used to store a program, and the processor executes the program after receiving an execution instruction.

[0124] The processor may be an integrated circuit chip with data processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0125] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A multi-channel speech enhancement method based on time-frequency-space masking, characterized in that: The multi-channel speech enhancement method based on time-frequency-space masking comprises: Acquire time domain data of multi-channel noisy speech, perform frequency domain conversion on the time domain data of the multi-channel noisy speech, and obtain time-frequency domain data of the multi-channel noisy speech in combination with the time index in the time domain data; Based on a transformation matrix that satisfies a spatial basis, the time-frequency domain data of the multi-channel noisy speech is converted into the time-frequency spatial domain data of the multi-channel noisy speech; Inputting the time-frequency-spatial domain data of the multi-channel noisy speech into a 3D U-Net model to obtain a time-frequency-spatial domain mask value output by the 3D U-Net model; Using the time-frequency-spatial-domain mask value to perform a masking operation on the time-frequency-spatial-domain data of the multi-channel noisy speech to obtain the time-frequency-spatial-domain data of the target speech after noise reduction; The time-frequency spatial domain data of the target speech after noise reduction is restored to the time domain, and the time domain data of the target speech after noise reduction is output to complete multi-channel speech enhancement.

2. The multi-channel speech enhancement method based on time-frequency-space masking according to claim 1, characterized in that: The step of performing frequency domain conversion on the time domain data of the multi-channel noisy speech and obtaining the time-frequency domain data of the multi-channel noisy speech in combination with the time index in the time domain data comprises: Performing frame division and windowing processing on the time domain data of the multi-channel noisy speech; Perform short-time Fourier transform on each frame of data, and combine the channel, frequency point and time to obtain the time-frequency domain data of multi-channel noisy speech, which is recorded as ,in is the channel index, is the frequency index, Index for the moment.

3. The multi-channel speech enhancement method based on time-frequency-space masking according to claim 2 is characterized in that: The method of converting the time-frequency domain data of the multi-channel noisy speech into the time-frequency spatial domain data of the multi-channel noisy speech based on the transformation matrix satisfying the spatial basis includes: Using a 3D tensor Characterize the time-frequency domain data of multi-channel noisy speech, satisfying , and apply the transformation matrix that satisfies the spatial basis to the three-dimensional tensor Above, as follows: ； In the formula, is the time-frequency-spatial data of multi-channel noisy speech. represents the mode-1 product of a tensor, To satisfy the spatial basis and size The transformation matrix, is the total number of channels.

4. The multi-channel speech enhancement method based on time-frequency-space masking according to claim 1 is characterized in that: When the 3D U-Net model is trained, the complex ideal ratio mask value after hyperbolic tangent compression is used as the training target, and the formula is as follows: ； ； ； ； In the formula, is the complex ideal ratio masking value after hyperbolic tangent compression corresponding to the real part of the time-frequency spatial domain data of the multi-channel noisy speech, is the complex ideal ratio masking value after hyperbolic tangent compression corresponding to the imaginary part of the time-frequency and spatial domain data of the multi-channel noisy speech, is the default value, is the steepness, is the real part simplified parameter, is the simplified parameter of the imaginary part, is the real part of the time-frequency-spatial domain data of multi-channel noisy speech in network training, is the imaginary part of the time-frequency-spatial domain data of multi-channel noisy speech in network training, is the real part of the time-frequency spatial domain data of the clean speech in network training, It is the imaginary part of the time-frequency spatial domain data of the clean speech in network training.

5. The multi-channel speech enhancement method based on time-frequency-space masking according to claim 1, characterized in that: When the 3D U-Net model is trained, the loss function calculation formula is as follows: ； ； ； In the formula, represents the loss value, represents the mean square error, Represents the time-frequency spatial domain data of clean speech in training, represents the time-frequency spatial domain data of the target speech after noise reduction estimated during training, represents the weighted source-distortion ratio loss, represents the time domain data of multi-channel noisy speech in training, represents the time domain data of clean speech in training, represents the time domain data of the target speech after noise reduction estimated during training, represents the weighted weight, Indicates the distortion ratio loss.

6. The multi-channel speech enhancement method based on time-frequency-space masking according to claim 1, characterized in that: The step of performing a masking operation on the time-frequency-spatial-domain data of the multi-channel noisy speech using the time-frequency-spatial-domain mask value to obtain the time-frequency-spatial-domain data of the target speech after noise reduction includes: ； In the formula, Represents the time-frequency spatial domain data of the target speech after noise reduction, Represents the time-frequency spatial domain mask value output by the 3D U-Net model, represents the tensor product, Represents the time-frequency and spatial domain data of multi-channel noisy speech.

7. The multi-channel speech enhancement method based on time-frequency-space masking according to claim 1, characterized in that: The step of restoring the time-frequency spatial domain data of the target speech after noise reduction to the time domain and outputting the time domain data of the target speech after noise reduction includes: ； In the formula, Represents the three-dimensional tensor of the target speech after noise reduction, Represents the time-frequency spatial domain data of the target speech after noise reduction, represents the mode-1 product of a tensor, It means that it satisfies the spatial basis and has a size of The transformation matrix, represents the mode-2 product of a tensor, It means that it satisfies the spatial basis and has a size of The transformation matrix, represents conjugate transpose; Shape it to The three-dimensional tensor Expand to , then The array is the time domain data of the target speech after noise reduction ,in, is the total number of channels, is the total number of frequency points, is the frame length, The total number of moments.

8. A computer device comprising a processor and a memory storing a plurality of computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the multi-channel speech enhancement method based on time-frequency-space masking described in any one of claims 1 to 7 are implemented.

Images