A speech enhancement method based on spherical microphone array and deep neural network

Abstract

The present invention discloses a speech enhancement method based on a spherical microphone array and a deep neural network, which specifically includes the following steps: Step 1: microphone array pre-processing stage; Step 2: training a speech model; Step 3: DNN speech enhancement stage; The present invention takes acoustic analysis to speech enhancement as a route, and uses a spherical microphone array to generate a spatial filter to minimize noise power. In view of the problem that single-channel speech enhancement has unsatisfactory speech enhancement effects in actual noise environments and traditional beamforming algorithms, a deep neural network is used to learn the complex nonlinear relationship between the noise and speech signals after pre-processing, classify the speech signal and the noise signal, and enhance the speech signal. Compared with a single beamforming algorithm and a deep neural network speech enhancement, the method of the present invention has a more obvious improvement effect, is also applicable in a variety of noise environments, and has strong robustness.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech signal processing, and in particular relates to a speech enhancement method based on a spherical microphone array and a deep neural network. Background Art

[0002] Speech recognition has become a hot topic in the field of automation due to its simplicity and convenience. It can free people from complicated memories and complete corresponding operations with just one machine-recognizable command. Therefore, it is widely used in systems with human-computer interaction such as car audio equipment, mobile phones and home appliances, which greatly facilitates people's production and life. In the past decade, with the continuous increase in the accumulation of big data corpus and the deepening of machine learning research, speech recognition technology has made great strides. Although traditional single-channel speech recognition has achieved a high recognition rate, it still needs to be further optimized according to actual needs, especially under low signal-to-noise ratio and reverberation conditions, the speech recognition rate cannot be guaranteed, and there is a problem of unsatisfactory speech enhancement effect. Summary of the invention

[0003] The purpose of the present invention is to provide a speech enhancement method based on a spherical microphone array and a deep neural network, which solves the problem that the recognition rate of current speech recognition under low signal-to-noise ratio and reverberation conditions is not high enough, and the user experience needs to be further improved and optimized.

[0004] The technical solution adopted by the present invention is:

[0005] A speech enhancement method based on a spherical microphone array and a deep neural network specifically comprises the following steps:

[0006] Step 1: Microphone array pre-processing stage: The multi-channel speech is picked up by a spherical microphone array. The beam formed by the array can locate the target speech, suppress the noise signal, and thus extract the target speech;

[0007] Step 2: Train the speech model; divide the noise signal and the target speech signal remaining after the microphone array pre-processing into two groups of feature samples, input them into the input end of the DNN for training, and use the DNN forward propagation and back propagation algorithms to obtain the network model;

[0008] Step 3: DNN speech enhancement stage: Input speech, obtain feature vector after microphone array processing, then input feature vector into the network model obtained in step 2 for enhancement processing, and finally use waveform reconstruction method to synthesize pure speech with the output feature vector.

[0009] The present invention is also characterized in that:

[0010] The spherical microphone array is a three-dimensional array. The spherical microphone array can obtain the horizontal direction angle and vertical azimuth angle of the signal, and can also obtain three-dimensional information about the distance between the sound source and the microphone array reference point.

[0011] In step 1, a linear constrained minimum variance beamforming LCMV structure is used. As the output signal changes, the weight coefficient of the filter is dynamically adjusted to adapt to the variable noise, and the residual noise is processed according to the DNN. First, the signal received by each element of the microphone array is delayed compensated and transformed into the frequency domain by DFT. Then, the frequency domain is divided into sub-bands, and the narrowband LCVM algorithm is applied in each sub-band. Finally, the result is inversely transformed back to the time domain by IDFT.

[0012] In step 1, the microphone array is composed of multiple array elements, and the delay compensation characteristic can be used to obtain x n (t), represents a signal with consistent timing, and then samples it to obtain the following formula (1):

[0013] x(n)=1 M s(n)+v(n) (1);

[0014] Among them, x n (t) is the signal vector after sampling, and its value is x(n)=[x1(n)x2(n)...x M (n)] T , 1 M =[11...1] T ; s(n) is the expected signal, v(n) = [v1(n)v2(n)...v M (n)] T is the noise vector, and the received signal is divided into S segments, each segment contains N sampling points. DFT is performed on it, and equation (2) is obtained:

[0015] x(k,s)=[x1(k,s)x2(k,s)...x M (K,S)] T (2);

[0016] From equation (1) and equation (2), we can get equation (3):

[0017] x(k,s)=1 M s(k,s)+v(k,s) (3);

[0018] Where s(k,s) is the expected output signal v(k,s)=[v1(k,s)v2(k,s)...v M (k,s)] T , perform N narrowband LCMV beamforming in the sth segment.

[0019] The spherical microphone array uses a spherical microphone array with a diameter of 8.4 cm, and the spherical microphone array is set to be spherical and equidistantly distributed.

[0020] The beneficial effect of the present invention is that the present invention is a speech enhancement method based on a spherical microphone array and a deep neural network. In the process of pitch detection, the first step is to analyze and process the input speech signal according to the linear prediction method to obtain the prediction coefficient. The linear prediction processing process can well remove the resonance peak effect. The second step is to use the OM-LSA algorithm to denoise the speech signal, and then use the Wiener filter algorithm to further filter out the residual noise. The third step is to use the main extension method to detect the pitch of the denoised speech. This method can still accurately detect the pitch period in a low signal-to-noise ratio and different noise environments, and the detection accuracy rate reaches more than 82%, which has strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of a spherical microphone array topology structure of a speech enhancement method based on a spherical microphone array and a deep neural network of the present invention;

[0022] Figure 2 It is a schematic diagram of an adaptive beamformer array element structure involving speech data in an embodiment of a speech enhancement method based on a spherical microphone array and a deep neural network of the present invention;

[0023] Figure 3 It is an algorithm framework diagram of a speech enhancement method based on a spherical microphone array and a deep neural network of the present invention;

[0024] Figure 4 It is a schematic diagram of the comparison of LSD scores of a speech enhancement method based on a spherical microphone array and a deep neural network and the detection results of an original algorithm according to the present invention;

[0025] Figure 5 It is a schematic diagram of the comparison of PESQ scores of a speech enhancement method based on a spherical microphone array and a deep neural network in the present invention and a speech enhancement result of an original algorithm; DETAILED DESCRIPTION

[0026] The following is a detailed description of a speech enhancement method based on a spherical microphone array and a deep neural network of the present invention in conjunction with the accompanying drawings and specific implementation methods.

[0027] The present invention provides a speech enhancement method based on a spherical microphone array and a deep neural network. The method uses a spherical microphone array to generate a spatial filter to minimize noise power, taking acoustic analysis to speech enhancement as the route. The complex nonlinear relationship between the noise and speech signals after the previous processing is further learned through deep neural networks (DNN), and the speech signal and the noise signal are classified to achieve the enhancement of the speech signal. In addition, the method proposed by the present invention is also applicable in a variety of noise environments, and has the characteristics of strong robustness and good implementation effect.

[0028] See attached Figure 1 , a schematic diagram of the topological structure of the spherical microphone array of the present invention. The spherical microphone array is a new type of three-dimensional array. Compared with one-dimensional and two-dimensional arrays, it can not only obtain the horizontal direction angle and vertical azimuth angle of the signal, but also obtain three-dimensional information such as the distance between the sound source and the microphone array reference point. In addition, it is also helpful to generate a spatial filter, and the symmetrical and rotatable structure is more conducive to using the beamforming algorithm to determine the location of the sound source.

[0029] See attached Figure 2 , a schematic diagram of the adaptive beamformer array element structure of the present invention. Beamforming is a manifestation of the selection of space by sound source signals. The microphone array can obtain the position information from the sound source signal, and can weaken the directional noise, retain the desired speech signal, and realize spatial filtering. The present invention adopts a linearly constrained minimum variance beamforming (LCMV) structure. As the output signal changes, the weight coefficient of the filter is dynamically adjusted to adapt to the variable noise, and the residual noise is processed according to the DNN. First, the signal received by each element of the microphone array is delayed compensated, and DFT is performed to the frequency domain. Then, the sub-bands are divided in the frequency domain, and the narrowband LCVM algorithm is applied in each sub-band. Finally, the result is inversely transformed back to the time domain by IDFT, which greatly reduces the computational complexity.

[0030] Attached Figure 2 Where y1(t) is the signal received by the array element. The actual microphone array is often composed of multiple array elements. The delay compensation characteristic can be used to obtain x n (t), represents a signal with consistent timing, and then samples it to obtain formula (1):

[0031] x(n)=1 M s(n)+v(n) (1);

[0032] where x n (t) is the signal vector after sampling, and its value is x(n)=[x1(n)x2(n)...x M(n)] T , 1 M =[11...1] T , s(n) is the expected signal, v(n)=[v1(n)v2(n)...v M (n)] T is the noise vector, and the received signal is divided into S segments, each segment contains N sampling points. DFT is performed on it, and equation (2) is obtained:

[0033] x(k,s)=[x1(k,s)x2(k,s)...x M (K,S)] T (2);

[0034] From equation (1) and equation (2), we can get equation (3):

[0035] x(k,s)=1 M s(k,s)+v(k,s) (3);

[0036] Where s(k,s) is the expected output signal v(k,s)=[v1(k,s)v2(k,s)...v M (k,s)] T , perform N narrowband LCMV beamforming in the sth segment.

[0037] See attached Figure 3 , the algorithm framework diagram of the present invention. The algorithm can be divided into three main parts: microphone array pre-processing stage, speech model training stage, and DNN speech enhancement stage.

[0038] In the pre-processing stage of the microphone array, the multi-channel speech is picked up by the spherical microphone array. The beam formed by the array can locate the target speech more accurately, and the noise signal is suppressed, so as to extract the target speech more accurately. The present invention uses a spherical microphone array with a diameter of 8.4 cm, and sets the array to be spherical and equidistantly distributed. It can receive speech information from any direction with the same speech input effect. Its beam pattern is not affected by the observation direction, and its recognition accuracy is comparable to or even better than that of a large linear array with an aperture of 126 cm.

[0039] During the speech model training phase, after preliminary processing by the microphone array, a portion of the signals irrelevant to the target signal are processed away, and the remaining noise signal and the target speech signal are divided into two groups of feature samples, which are input into the input end of the DNN for training respectively, and the optimal network model is obtained using the DNN forward propagation and back propagation algorithms.

[0040] In the DNN speech enhancement stage, real speech is input and processed by the microphone array to obtain a feature vector, which is then input into the optimized network model for enhancement processing. Finally, the output feature vector is synthesized into pure speech using the waveform reconstruction method to achieve speech enhancement function.

[0041] See attached Figure 4 , the speech LSD score diagram of the present invention. In order to better illustrate the reliability of the method proposed by the present invention, the attached Figure 4 The method proposed in the present invention is compared with the non-negative matrix factorization algorithm (NMF) for noise-free classification and the single DNN speech enhancement algorithm under different noise backgrounds. The log spectral distance (LSD) is defined as shown in formula (4):

[0042]

[0043] Where M represents the frame length, L{S(l,m)} represents the logarithmic spectrum within a certain dynamic range, and the smaller the LSD value, the smaller the degree of speech distortion. It is an objective evaluation indicator of speech quality and can reflect the difference between the reconstructed signal and the original signal.

[0044] See attached Figure 5 , PESQ score diagram of speech enhancement of the present invention. Perceptual Evaluation of Speech Quality (PESQ) is performed on different algorithms. PESQ is used for end-to-end measurement in field and simulated networks. Environmental noise and noise processing are evaluated by PESQ. Under the condition that the PESQ score range is between -0.5 and 4.5, the higher the score, the better the speech quality. The present invention evaluates the quality of unprocessed speech signals, minimum variance distortionless response (MVDR) beamforming algorithm and the algorithm of the present invention. It can be seen that the PESQ scores of the algorithm of the present invention are all greater than 2.0 under various noise backgrounds, with an average score of 2.74. The PESQ scores of the single MVDR beamforming algorithm are mostly below 2.0, with an average score of 1.86.

[0045] It can be seen that the speech enhancement method based on spherical microphone array and deep neural network of the present invention has a certain improvement in performance compared with the MVDR beamforming algorithm. In addition, based on the two indicators and speech enhancement results, the spherical microphone array used in the present invention is used as a first-step speech enhancement, combined with DNN noise speech classification for the second-step speech enhancement, combined with the current mainstream trend of using neural networks for speech recognition, using DNN for secondary speech enhancement can not only effectively remove the impact of noise, but also show strong robustness for a variety of noise environments.

Claims

1. A speech enhancement method based on a spherical microphone array and a deep neural network, characterized in that: The specific steps include: Step 1: Microphone array pre-processing stage: The multi-channel speech is picked up by a spherical microphone array. The beam formed by the array can locate the target speech, suppress the noise signal, and thus extract the target speech; In step 1, a linear constrained minimum variance beamforming LCMV structure is used. As the output signal changes, the weight coefficient of the filter is dynamically adjusted to adapt to the variable noise, and the residual noise is processed according to the DNN. First, the signal received by each element of the microphone array is delayed compensated, and DFT is transformed to the frequency domain. Then, the sub-bands are divided in the frequency domain, and the narrowband LCVM algorithm is applied in each sub-band. Finally, the obtained result is inversely transformed back to the time domain by IDFT. In step 1, the microphone array is composed of multiple elements. The delay compensation characteristic can be used to obtain , which represents a signal with consistent timing, and then samples it to obtain the following formula (1): （1）； in, is the sampled signal vector, and its value is , ; is the expected signal, is the noise vector, and the received signal is divided into S segments, each segment contains N sampling points. DFT is performed on it, and equation (2) is obtained: （2）； From equation (1) and equation (2), we can get equation (3): （3）； in The expected output signal , perform N narrowband LCMV beamforming in the sth segment; Step 2: Train the speech model; divide the noise signal and the target speech signal remaining after the microphone array pre-processing into two groups of feature samples, input them into the input end of the DNN for training, and use the DNN forward propagation and back propagation algorithms to obtain the network model; Step 3: DNN speech enhancement stage: Input speech, obtain feature vector after microphone array processing, then input feature vector into the network model obtained in step 2 for enhancement processing, and finally use waveform reconstruction method to synthesize pure speech with the output feature vector.

2. The method for speech enhancement based on a spherical microphone array and a deep neural network according to claim 1, characterized in that: The spherical microphone array is a three-dimensional array, which can obtain the horizontal direction angle and vertical azimuth angle of the signal, and can also obtain three-dimensional information of the distance between the sound source and the microphone array reference point.

3. The method for speech enhancement based on a spherical microphone array and a deep neural network according to claim 2, characterized in that: The spherical microphone array uses a spherical microphone array with a diameter of 8.4 cm, and the spherical microphone array is set to be spherical and equidistantly distributed.

Images