Multi-channel speech enhancement method based on fusion spatial features

Abstract

The invention discloses a multi-channel speech enhancement method based on fusion spatial features, which comprises the following steps of: firstly, respectively extracting multi-channel complex spectrum features and fusion spatial features from noisy multi-channel speech; inputting the extracted multi-channel complex spectrum features and fusion spatial features into a deep neural network for processing, setting a processing target as ideal complex value ratio masking, and constructing a deep reconvolution cyclic speech enhancement network based on a reconvolution encoder and decoder structure; and processing the multi-channel complex spectrum features and the fused spatial features by using a deep re-convolution cyclic speech enhancement network, and outputting predicted complex value ratio masking. According to the method, spectral information, direction features, inter-channel features and correlation coefficient features are fully integrated, and spatial modeling is performed on a dynamic sound source, so that the ability of the network to understand and process voice signals is far better than that of a traditional method, complex and changeable voice scenes can be effectively dealt with, and the method is suitable for popularization and application. And the speech enhancement effect in a static scene and a dynamic scene is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of speech enhancement, and in particular to a multi-channel speech enhancement method based on fusion of spatial features. Background Art

[0002] Thanks to the development of deep learning, the latest progress of data-driven models in speech enhancement tasks has shown superior performance compared to traditional machine learning methods and signal processing methods. These supervised learning methods are becoming mainstream because they can mine effective information from massive noisy-clean speech data pairs, thereby learning powerful noise suppression capabilities (especially for non-stationary noise). Compared with single-channel speech enhancement, multi-channel speech enhancement can capture spectral and spatial information from the observed noise signal, thereby improving the quality of enhanced speech.

[0003] As a typical scheme, beamforming (or spatial filtering) suppresses noise by applying a linear spatial filter to a noisy multichannel signal, thereby exploiting spatial information. In mask-based beamforming, the role of DNNs is to provide more accurate estimates of the spatial statistics of speech and noise for various data-dependent beamformers (such as GEV or MVDR). Recently, more attention has been focused on using full neural networks to implicitly perform spatial filtering, which is called a "full neural beamformer". Such networks can exploit spectral and spatial information to perform speech enhancement for fixed sound sources in non-specific directions. However, in dynamic scenes where the sound source moves, the performance of previous multi-channel speech enhancement methods has significantly degraded due to their limited ability to track the sound source and mismatched training data.

[0004] From the above description, how to make full use of spatial information to perform spatial modeling of dynamic sound sources is the key to improving the performance of multi-channel speech enhancement in dynamic scenes and is a problem that needs to be solved currently.

[0005] Therefore, a multi-channel speech enhancement method based on fusion of spatial features is proposed. Summary of the invention

[0006] The purpose of the present invention is to provide a multi-channel speech enhancement method based on fusion of spatial features to solve the problems raised in the above background technology.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: a multi-channel speech enhancement method based on fusion spatial features, comprising the following steps:

[0008] Step S1, extracting multi-channel complex spectrum features and fused spatial features from noisy multi-channel speech respectively, and then inputting the extracted multi-channel complex spectrum features and fused spatial features into a deep neural network for processing and setting the processing target as an ideal complex value ratio masking;

[0009] Step S2, constructing a deep complex convolutional cyclic speech enhancement network based on a complex convolutional encoder-decoder structure, and then using the deep complex convolutional cyclic speech enhancement network to process multi-channel complex spectrum features and fused spatial features, and output a predicted complex value ratio mask;

[0010] Step S3, using a gradient descent method to process the deep complex convolutional recurrent speech enhancement network and shorten the distance between the predicted complex valued ratio mask and the ideal complex valued ratio mask to obtain a processed predicted complex valued ratio mask;

[0011] Step S4, using a deep complex convolutional recurrent speech enhancement network to reconstruct the enhanced speech waveform according to the processed predicted complex value ratio masking to obtain an enhanced speech waveform.

[0012] The aforementioned multi-channel speech enhancement method based on fused spatial features, step S1, extracting multi-channel complex spectrum features and fused spatial features from noisy multi-channel speech respectively, and then inputting the extracted multi-channel complex spectrum features and fused spatial features into a deep neural network for processing and setting the processing target as an ideal complex value ratio masking;

[0013] Step S11: extract directional features, as shown in formula (1):

[0014]

[0015] Where M represents the number of candidate azimuths, θ i represents the azimuth angle to which the feature points, p represents the index of the microphone pair, and t and f are the subscripts of the time and frequency domain axes respectively;

[0016] Step S12: extract inter-channel features, as shown in formula (2):

[0017]

[0018] Where p = (p1, p2) represents the index of the microphone pair, represents the spectral coefficients of the signals from microphones p1 and p2 at time t and frequency f and The phase difference;

[0019] Step S13: extract normalized cross-correlation features, as shown in formula (3):

[0020]

[0021] Where * represents the complex conjugate operation of the matrix.

[0022] The fusion spatial features include directional features, inter-channel features and normalized cross-correlation features.

[0023] Among them, the inter-channel features include inter-channel phase features and inter-channel level features.

[0024] The aforementioned multi-channel speech enhancement method based on fused spatial features, step S2, constructs a deep complex convolutional cyclic speech enhancement network based on a complex convolutional encoder-decoder structure, and then uses the deep complex convolutional cyclic speech enhancement network to process the multi-channel complex spectrum features and fused spatial features, and outputs a predicted complex value ratio mask. The specific steps are as follows:

[0025] Step S21, grouping the directional features based on the angle, classifying the directional features within the range of ±45° of each microphone angle into a group, and recording them as directional features

[0026] Step S22, calculating the channel weighting coefficient based on the group-based weighting module, performing weighted processing on the multi-channel complex spectrum features to obtain weighted combined features;

[0027] Step S221: group the directional features Through the same feature extraction and compression path f, stacking and compression are performed to obtain the channel weighting coefficients

[0028] Step S222, weighting the combined input features of the real part and the imaginary part of the multi-channel complex spectrum and the amplitude spectrum to obtain a weighted combined feature As shown in formula (4):

[0029]

[0030] in, represents the combined input features of the real and imaginary parts of the multi-channel complex spectrum and the amplitude spectrum, c = 1, ..., C and They respectively represent the number of input channels and the short-time Fourier coefficients of the grouped directional features of the input in each channel and each time frame.

[0031] Step S23, inputting the inter-channel features and the normalized cross-correlation features into a dense feature encoder module, and outputting high-order fused spatial features;

[0032] Step S24: Input the output high-order fused spatial features into the stacked frequency-time LSTM block to capture the correlation along the frequency and time axes, extract basic information from the feature stream, and output the predicted complex-valued ratio mask.

[0033] The aforementioned multi-channel speech enhancement method based on fusion spatial features, step S3, uses the gradient descent method to process the deep complex convolutional cyclic speech enhancement network and narrow the distance between the predicted complex value ratio mask and the ideal complex value ratio mask to obtain the processed predicted complex value ratio mask, the specific steps are as follows,

[0034] Step S31, using a combination of two losses to perform model training to obtain a combined loss of the two losses;

[0035] Step S311, introducing multi-resolution spectrum loss L mrstft , the multi-resolution short-time Fourier transform loss is the sum of the short-time Fourier transform losses with different analysis parameters, as shown in formula (5);

[0036]

[0037] Among them, R is 3, indicating that there are three short-time Fourier transform losses L with different analysis parameters. STFT , that is, the frame lengths are {1024, 512, 256}, and the frame shifts are {512, 256, 128};

[0038] Step S312, introduce PMSQE loss L pmsqe , as shown in formula (6);

[0039] L DATFCN =αL pmsqe +(1-α)L mrstft , (6)

[0040] Among them, α is 0.6;

[0041] In step S32, the obtained combined loss is processed using the Adam optimization algorithm, the gradient of the combined loss function is calculated, the network coefficients are updated, and the processed predicted complex value ratio mask is obtained.

[0042] The aforementioned multi-channel speech enhancement method based on fusion spatial features, step S4, uses a deep complex convolutional cyclic speech enhancement network to reconstruct the enhanced speech waveform according to the processed predicted complex value ratio masking to obtain the enhanced speech waveform, and the specific steps are as follows:

[0043] Step S41: multiply the processed predicted complex-valued ratio mask by the spectrum of the target speech and the noise signal element by element to reconstruct the clean speech spectrum, as shown in formula (7):

[0044] s=IFFT[(M r Y r -M i Y i )+i(M r Yi +M i Y r )], (7)

[0045] Where s represents the time domain waveform of the desired speech, r represents the real part, i represents the imaginary part, M represents the complex ratio mask, S represents the spectrum of the target speech, and Y represents the spectrum of the noise signal;

[0046] Step S42: restore the time domain waveform using inverse Fourier transform, and synthesize the enhanced speech waveform using overlap-add algorithm.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The aforementioned multi-channel speech enhancement method based on fusion spatial features,

[0049] The present invention is based on a multi-channel speech enhancement method that integrates spatial features, and proposes a direction-aware time-frequency convolutional network with integrated spatial features for multi-channel speech enhancement. The network is a deep complex convolutional recurrent speech enhancement network based on a complex convolutional encoder-decoder structure, which makes full use of spectral information, directional features, inter-channel features, and correlation coefficient features, captures the correlation of these features along the frequency and time axes, extracts basic information from the feature stream, and finally outputs a predicted complex value ratio mask.

[0050] The present invention fully integrates spectral information, directional features, inter-channel features and correlation coefficient features, and performs spatial modeling of dynamic sound sources, which can deeply explore the intrinsic connections between these features in frequency and time dimensions, thereby extracting more accurate and representative basic information. This enables the network to understand and process speech signals far beyond traditional methods, effectively cope with complex and changeable speech scenes, and effectively improve the speech enhancement effect in static and dynamic scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is the overall flow chart of the present invention;

[0052] Figure 2 Schematic diagram of the training phase and enhancement phase of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with the accompanying drawings.

[0054] See also Figure 1-2 The present invention provides a multi-channel speech enhancement method based on fusion spatial features, comprising the following steps:

[0055] Step S1, extracting multi-channel complex spectrum features and fused spatial features from noisy multi-channel speech respectively, and then inputting the extracted multi-channel complex spectrum features and fused spatial features into a deep neural network for processing and setting the processing target as an ideal complex value ratio masking;

[0056] Step S11: extract directional features, as shown in formula (1):

[0057]

[0058] Where M represents the number of candidate azimuths, θ i represents the azimuth angle to which the feature points, p represents the index of the microphone pair, and t and f are the subscripts of the time and frequency domain axes respectively;

[0059] Step S12: extract inter-channel features, as shown in formula (2):

[0060]

[0061] Where p = (p1, p2) represents the index of the microphone pair, represents the spectral coefficients of the signals from microphones p1 and p2 at time t and frequency f and The phase difference;

[0062] Step S13: extract normalized cross-correlation features, as shown in formula (3):

[0063]

[0064] Where * represents the complex conjugate operation of the matrix.

[0065] The fusion spatial features include directional features, inter-channel features and normalized cross-correlation features.

[0066] Among them, the inter-channel features include inter-channel phase features and inter-channel level features.

[0067] Step S2, constructing a deep complex convolutional cyclic speech enhancement network based on a complex convolutional encoder-decoder structure, and then using the deep complex convolutional cyclic speech enhancement network to process multi-channel complex spectrum features and fused spatial features, and output a predicted complex value ratio mask;

[0068] Step S2, constructing a deep complex convolutional cyclic speech enhancement network based on a complex convolutional encoder-decoder structure, and then using the deep complex convolutional cyclic speech enhancement network to process multi-channel complex spectrum features and fused spatial features, and output predicted complex value ratio masking, the specific steps are as follows:

[0069] Step S21, grouping the directional features based on the angle, classifying the directional features within the range of ±45° of each microphone angle into a group, and recording them as directional features

[0070] Step S22, calculating the channel weighting coefficient based on the group-based weighting module, performing weighted processing on the multi-channel complex spectrum features to obtain weighted combined features;

[0071] Step S221: group the directional features Through the same feature extraction and compression path f, stacking and compression are performed to obtain the channel weighting coefficients

[0072] Step S222, weighting the combined input features of the real part and the imaginary part of the multi-channel complex spectrum and the amplitude spectrum to obtain a weighted combined feature As shown in formula (4):

[0073]

[0074] in, represents the combined input features of the real and imaginary parts of the multi-channel complex spectrum and the amplitude spectrum, c = 1, ..., C and They respectively represent the number of input channels and the short-time Fourier coefficients of the grouped directional features of the input in each channel and each time frame.

[0075] Step S23, inputting the inter-channel features and the normalized cross-correlation features into a dense feature encoder module, and outputting high-order fused spatial features;

[0076] Step S24: Input the output high-order fused spatial features into the stacked frequency-time LSTM block to capture the correlation along the frequency and time axes, extract basic information from the feature stream, and output the predicted complex-valued ratio mask.

[0077] Step S3, using a gradient descent method to process the deep complex convolutional recurrent speech enhancement network and shorten the distance between the predicted complex valued ratio mask and the ideal complex valued ratio mask to obtain a processed predicted complex valued ratio mask;

[0078] Step S31, using a combination of two losses to perform model training to obtain a combined loss of the two losses;

[0079] Step S311, introducing multi-resolution spectrum loss L mrstft , the multi-resolution short-time Fourier transform loss is the sum of the short-time Fourier transform losses with different analysis parameters, as shown in formula (5);

[0080]

[0081] Among them, R is 3, indicating that there are three short-time Fourier transform losses L with different analysis parameters. STFT , that is, the frame lengths are {1024, 512, 256}, and the frame shifts are {512, 256, 128};

[0082] Step S312, introduce PMSQE loss L pmsqe , as shown in formula (6);

[0083] L DATFCN =aL pmsqe +(1-α)L mrstft , (6)

[0084] Among them, α is 0.6;

[0085] In step S32, the obtained combined loss is processed using the Adam optimization algorithm, the gradient of the combined loss function is calculated, the network coefficients are updated, and the processed predicted complex value ratio mask is obtained.

[0086] Step S4, using a deep complex convolutional recurrent speech enhancement network to reconstruct the enhanced speech waveform according to the processed predicted complex value ratio masking to obtain an enhanced speech waveform.

[0087] Step S41: multiply the processed predicted complex-valued ratio mask by the spectrum of the target speech and the noise signal element by element to reconstruct the clean speech spectrum, as shown in formula (7):

[0088] s=IFFT[(M r Y r -M i Y i )+i(M r Y i +M i Y r )], (7)

[0089] Where s represents the time domain waveform of the desired speech, r represents the real part, i represents the imaginary part, M represents the complex ratio mask, S represents the spectrum of the target speech, and Y represents the spectrum of the noise signal;

[0090] Step S42: restore the time domain waveform using inverse Fourier transform, and synthesize the enhanced speech waveform using overlap-add algorithm.

[0091] In order to better illustrate the use effect of the present invention, a specific embodiment of the present invention is introduced below.

[0092] When generating simulation data, we randomly sample a target speech from the Deep Noise Suppression (DNS) dataset

[16] and randomly sample 2 to 3 noise signals from the DNS dataset, one of which is used as directional interference noise and the rest are used to generate background noise through an isotropic noise generator.

[0093] All mixed signals were 16 s long and resampled to 16 kHz, and the simulated room sizes ranged from 3 × 3 × 3.5 m to 6 × 8 × 4 m.

[0094] The speaking position is within the range of 0.2 to 2 meters from the microphone array, and the minimum distance from the wall is 0.5 meters. The room reverberation T60 time range is between 0.05 and 0.4 seconds, and the signal-to-noise ratio of speech signals and noise signals is set between -10 and 15 dB.

[0095] We first set a starting coordinate for the sound source, and then the position of the sound source will move randomly every 0.2 seconds.

[0096] Based on the constructed database, the performance comparison of the proposed method with the prior art methods in the scenarios with moving sound sources or fixed sound sources in non-specific directions is shown in the following table.

[0097]

[0098]

[0099] As can be seen from the above table, in the performance evaluation experiment of the multi-channel speech enhancement method, a variety of evaluation indicators were used for testing in two scenarios: mobile sound source and fixed sound source in non-specific direction, including PESQ-NB (perceptual evaluation of narrowband speech quality), PESQ-WB (perceptual evaluation of wideband speech quality), STOI (short-time objective intelligibility) and SDR (signal-to-distortion ratio).

[0100] Based on the experimental results in the two scenarios, it can be seen that the proposed DATFCN method is superior to the current most advanced results in terms of speech quality perception evaluation indicators (PESQ-NB and PESQ-WB), short-term objective intelligibility index (STOI) and signal-to-distortion ratio index (SDR) in the presence of fixed and moving sound sources. This fully demonstrates the excellent performance and significant advantages of the method in the field of multi-channel speech enhancement, and provides a more effective technical means to solve the problem of speech enhancement in complex sound source scenarios.

[0101] While the invention has been shown and described, it will be appreciated by those skilled in the art that many changes, modifications, substitutions and variations may be made thereto without departing from the principles and spirit of the invention, the scope of the invention being defined by the appended claims and their equivalents.

Claims

1. A multi-channel speech enhancement method based on fusion spatial features, characterized in that: The following steps are involved: Step S1, extracting multi-channel complex spectrum features and fused spatial features from noisy multi-channel speech respectively, and then inputting the extracted multi-channel complex spectrum features and fused spatial features into a deep neural network for processing and setting the processing target as an ideal complex value ratio masking; Step S2, constructing a deep complex convolutional cyclic speech enhancement network based on a complex convolutional encoder-decoder structure, and then using the deep complex convolutional cyclic speech enhancement network to process multi-channel complex spectrum features and fused spatial features, and output a predicted complex value ratio mask; Step S3, using a gradient descent method to process the deep complex convolutional recurrent speech enhancement network and shorten the distance between the predicted complex valued ratio mask and the ideal complex valued ratio mask to obtain a processed predicted complex valued ratio mask; Step S4, using a deep complex convolutional recurrent speech enhancement network to reconstruct the enhanced speech waveform according to the processed predicted complex value ratio masking to obtain an enhanced speech waveform.

2. The multi-channel speech enhancement method based on fusion spatial features according to claim 1, characterized in that: The fusion spatial features include directional features, inter-channel features and normalized cross-correlation features. Among them, the inter-channel features include inter-channel phase features and inter-channel level features.

3. The multi-channel speech enhancement method based on fusion spatial features according to claim 2 is characterized in that: Step S1, extracting multi-channel complex spectrum features and fused spatial features from noisy multi-channel speech respectively, and then inputting the extracted multi-channel complex spectrum features and fused spatial features into a deep neural network for processing and setting the processing target as an ideal complex value ratio masking. The specific steps are as follows: Step S11: extract directional features, as shown in formula (1): Where M represents the number of candidate azimuths, θ i represents the azimuth angle to which the feature points, p represents the index of the microphone pair, and t and f are the subscripts of the time and frequency domain axes respectively; Step S12: extract inter-channel features, as shown in formula (2): Where p = (p1, p2) represents the index of the microphone pair, represents the spectral coefficients of the signals from microphones p1 and p2 at time t and frequency f and The phase difference; Step S13: extract normalized cross-correlation features, as shown in formula (3): Where * represents the complex conjugate operation of the matrix.

4. The multi-channel speech enhancement method based on fusion spatial features according to claim 1, characterized in that: Step S2, constructing a deep complex convolutional cyclic speech enhancement network based on a complex convolutional encoder-decoder structure, and then using the deep complex convolutional cyclic speech enhancement network to process multi-channel complex spectrum features and fused spatial features, and output predicted complex value ratio masking, the specific steps are as follows: Step S21, grouping the directional features based on the angle, classifying the directional features within the range of ±45° of each microphone angle into a group, and recording them as directional features Step S22, calculating the channel weighting coefficient based on the group-based weighting module, performing weighted processing on the multi-channel complex spectrum features to obtain weighted combined features; Step S23, inputting the inter-channel features and the normalized cross-correlation features into a dense feature encoder module, and outputting high-order fused spatial features; Step S24: Input the output high-order fused spatial features into the stacked frequency-time LSTM block to capture the correlation along the frequency and time axes, extract basic information from the feature stream, and output the predicted complex-valued ratio mask.

5. The multi-channel speech enhancement method based on fusion spatial features according to claim 4 is characterized in that: Step S22, calculating the channel weighting coefficient based on the group weighting module, performing weighted processing on the multi-channel complex spectrum features, and obtaining the weighted combined features, the specific steps are as follows; Step S221: group the directional features Through the same feature extraction and compression path f, stacking and compression are performed to obtain the channel weighting coefficients Step S222, weighting the combined input features of the real part and the imaginary part of the multi-channel complex spectrum and the amplitude spectrum to obtain a weighted combined feature As shown in formula (4): in, represents the combined input features of the real and imaginary parts of the multi-channel complex spectrum and the amplitude spectrum, c = 1, ..., C and They respectively represent the number of input channels and the short-time Fourier coefficients of the grouped directional features of the input in each channel and each time frame.

6. The multi-channel speech enhancement method based on fusion of spatial features according to claim 4 is characterized in that: The dense feature encoder module includes two convolution blocks, and a dilated dense block is located between the two convolution blocks. The dilated dense block is composed of four layers of 2D dilated convolutions, and the dilation rates of the four layers of dilated convolutions in the dilated dense block are set to 1, 2, 4 and 8 respectively.

7. The multi-channel speech enhancement method based on fusion spatial features according to claim 1, characterized in that: Step S3, using a gradient descent method to process the deep complex convolutional recurrent speech enhancement network and shorten the distance between the predicted complex valued ratio mask and the ideal complex valued ratio mask to obtain a processed predicted complex valued ratio mask; Step S31, using a combination of two losses to perform model training to obtain a combined loss of the two losses; Step S311, introducing multi-resolution spectrum loss L mrstft , the multi-resolution short-time Fourier transform loss is the sum of the short-time Fourier transform losses with different analysis parameters, as shown in formula (5); Among them, R is 3, indicating that there are three short-time Fourier transform losses L with different analysis parameters. STFT , that is, the frame lengths are {1024, 512, 256}, and the frame shifts are {512, 256, 128}; Step S312, introduce PMSQE loss L pmsqe , as shown in formula (6); L DATFCN =αL pmsqe +(1-α)L mrstft , (6) Among them, α is 0.6; In step S32, the obtained combined loss is processed using the Adam optimization algorithm, the gradient of the combined loss function is calculated, the network coefficients are updated, and the processed predicted complex value ratio mask is obtained.

8. The multi-channel speech enhancement method based on fusion spatial features according to claim 1, characterized in that: Step S4, using a deep complex convolutional cyclic speech enhancement network to reconstruct the enhanced speech waveform according to the processed predicted complex value ratio masking to obtain an enhanced speech waveform, the specific steps are as follows: Step S41: multiply the processed predicted complex valued ratio mask by the spectrum of the target speech and the noise signal element by element to reconstruct the clean speech spectrum, as shown in formula (7): s=IFFT[(M r Y r -M i Y i )+i(M r Y i +M i Y r )], (7) Where s represents the time domain waveform of the desired speech, r represents the real part, i represents the imaginary part, M represents the complex ratio mask, S represents the spectrum of the target speech, and Y represents the spectrum of the noise signal; Step S42: using inverse Fourier transform to restore the time domain waveform, and using overlap-add algorithm to synthesize the enhanced speech waveform.

Images