A deep learning optimized beam characteristic microphone array speech enhancement method

Abstract

The application provides a microphone array speech enhancement method for optimizing beam characteristics through deep learning, which comprises a model training stage and a speech enhancement stage. In the model training stage, a speech enhancement model is obtained based on a first noisy double-channel speech signal, FB coefficients obtained through a traditional fixed beam forming algorithm, and STFT eigenvalues of a single-channel pure speech signal controlled by a weight. In the speech enhancement stage, a second noisy double-channel speech signal is convolved with FB coefficients optimized and output through the speech enhancement model, and a beam forming enhanced speech signal is obtained through weighted summation. Through the introduction of a deep learning network, the beam characteristics of the microphone array are optimized, the beam pointing characteristics are better than those of a traditional method, and low complexity engineering implementation is supported.

Description

Technical Field

[0001] This application relates to the field of intelligent speech, and in particular to a microphone array speech enhancement method that optimizes beam characteristics using deep learning. Background Technology

[0002] Voice has become an important entry point for human-computer interaction in the era of the Internet of Things, and microphone array technology is widely used in the field of intelligent voice enhancement. Among the traditional microphone array voice enhancement algorithms, fixed beamforming algorithms such as filtering accumulation algorithms are insufficient in enhancing voice signals when the number of microphones is small. Although the minimum variance distortionless response algorithm can suppress environmental noise well, the algorithm has high complexity and poor reverberation suppression.

[0003] In recent years, deep learning-based microphone array speech enhancement algorithms have been widely studied. Deep learning networks can achieve good speech enhancement performance by constructing deep learning models to learn the nonlinear mapping between noisy and clean speech signals in a dataset. Luo et al. proposed an end-to-end multi-channel speech enhancement algorithm—a filter-and-sum network with Tramsform-average-concatenate (FaSNet-TAC) connected to a transform-average module. The FaSNet network is a neural network beamformer for temporal filter-and-sum beamforming. It directly estimates the filter coefficients of the temporal beamformer without specifying the number and arrangement of microphones. By estimating the filter coefficients of the temporal beamformer, the filter coefficients are convolved with their corresponding channel signals, and the summation is performed to form the output of the beamformer, thus achieving speech enhancement.

[0004] Lei Xie et al. proposed the Deep Complex Convolution Recurrent Network (DCCRN) for speech enhancement. The DCCRN model uses a complex network for complex-valued spectrum modeling. Under the constraints of complex multiplication rules, DCCRN can achieve better performance in PESQ and MOS than other algorithms with similar model parameter configurations.

[0005] The aforementioned methods of integrating microphone arrays into the network for speech enhancement can effectively suppress noise signals and improve speech signal quality. However, due to the large number of model parameters, these methods are typically deployed on servers or in the cloud. In engineering applications, speech signals collected by miniaturized terminals such as wearable devices based on small embedded platforms are transmitted to servers or the cloud via data transmission. After model enhancement, an enhanced speech signal is obtained, and subsequent tasks such as speech recognition are performed on the enhanced signal. This approach requires a network connection as a prerequisite, making it difficult to implement on small embedded platforms such as wearable devices that lack network connectivity. Summary of the Invention

[0006] To address the aforementioned technical problems, this application provides a microphone array speech enhancement method based on deep learning to optimize beam characteristics, comprising a model training phase and a speech enhancement phase, wherein the model training phase includes the following steps:

[0007] S1. Extract GCC feature values ​​from the first noisy dual-channel speech signal and use them as input to the model to obtain the GCC feature vector;

[0008] S2. Construct a spatial optimization deep learning network framework with the GCC feature vector as input and train the network to obtain the FB coefficients of the optimized fixed beamforming algorithm.

[0009] S3. Convolve the FB coefficients of the optimized fixed beamforming algorithm with the first noisy dual-channel speech signal, and sum them by weight to obtain a single-channel enhanced speech signal.

[0010] S4. The FB coefficients obtained by the traditional fixed beamforming algorithm and the STFT feature values ​​of the single-channel clean speech signal controlled by weights are proportionally fused to construct the model training target.

[0011] S5. Calculate the loss function of the FB coefficients obtained by the traditional fixed beamforming algorithm, the FB coefficients of the optimized fixed beamforming algorithm, the STFT feature value of the single-channel enhanced speech signal, and the STFT feature value of the single-channel clean speech signal, perform gradient backpropagation and model training to obtain the speech enhancement model.

[0012] The speech enhancement stage includes the following steps: convolving the second noisy dual-channel speech signal with the FB coefficients optimized by the speech enhancement model, and then weighting and summing them to obtain the beamforming enhanced speech signal.

[0013] In some specific embodiments, step S1 constructs a spatially optimized deep learning network framework with the GCC feature vector as input for network training, for array elements M i and array element M jThe received signal is set as follows:

[0014] x i =a i s(t-τ i )+n i (t)

[0015] x j =a j s(t-τ j )+n j (t)

[0016] Where s(t) represents the sound source signal, a i a j Let τ be the attenuation coefficient of the corresponding array element, τ be the corresponding time delay, and n(t) represent the environmental noise collected by the array element. Then, the cross-correlation function between the array elements can be expressed as:

[0017]

[0018] Among them G xixj (ω) represents the cross-power spectrum of the signals between array elements, X i (ω) and X j (ω) represents x i (t) and x j The Fourier transform of (t), (·) * Denotes taking the complex conjugate, where This represents the weighting function in the frequency domain.

[0019] In some specific embodiments, the network framework of step S2 consists of an input layer, a hidden layer, and an output layer, wherein the hidden layer includes at least one LSTM network layer and one fully connected layer.

[0020] In some specific embodiments, step S2 specifically includes the following steps:

[0021] S11. Input the GCC feature vector from the input layer into the hidden layer;

[0022] S12. The hidden layer is non-linearly activated based on the Tanh function, and the non-linearly activated GCC feature vector is input into the output layer.

[0023] S13. Obtain the FB coefficients of the optimized fixed beamforming algorithm through linear processing of the output layer. Nonlinear activation based on the Tanh function can avoid the problem of positive and negative values ​​in the FB coefficients.

[0024] In some specific embodiments, in step S4, constructing the model training objective, obtaining the STFT feature value of the weighted single-channel clean speech signal includes attenuating the STFT feature of the single-channel clean speech signal with a preset angle deviation as the weight for optimizing the beam, constructing the spatial transmission response, and calculating the angle deviation using the following formula:

[0025] Δθ≡|θ target -θ signal |

[0026] Where θ target Where θ represents the desired direction. signal This represents the actual direction of the sound source signal.

[0027] In some specific embodiments, the weight decays to 1 when the angular deviation is 0 degrees; to 0.5 when the angular deviation is 30 degrees; and to 0.001 when the angular deviation does not include 0 or 30 degrees. This ensures that the deep learning-optimized beam of the network model can enhance the speech signal in the desired direction and suppress interference signals in other directions, achieving the expected spatial transmission response.

[0028] In some specific embodiments, step S5 includes minimizing the loss function using gradient descent, wherein the backpropagation of the gradient includes updating the parameters of the network model using a backpropagation algorithm.

[0029] In some specific embodiments, the model training phase is executed on the server or computer, while the speech enhancement phase is executed on an embedded device.

[0030] According to a second aspect of the present invention, a microphone array speech enhancement system for optimizing beam characteristics using deep learning is proposed, comprising a computer system and an embedded system. The computer system includes a feature extraction module, a network training module, a weighted summation module, a training target generation module, and a loss function calculation module. The embedded system includes a speech enhancement module.

[0031] The feature extraction module is configured to extract GCC feature values ​​from the first noisy dual-channel speech signal as input to the model to obtain a GCC feature vector.

[0032] The network training module is configured to construct a spatially optimized deep learning network framework with the GCC feature vector as input for network training, and obtain the optimized FB coefficients of the fixed beamforming algorithm.

[0033] The weighted summation module is configured to convolve the FB coefficients of the optimized fixed beamforming algorithm with the first noisy dual-channel speech signal and then sum them to obtain a single-channel enhanced speech signal.

[0034] The training target generation module is configured to proportionally fuse the FB coefficients obtained by the traditional fixed beamforming algorithm and the STFT feature values ​​of the weighted single-channel clean speech signal to construct the model training target.

[0035] The model generation module is configured to calculate the loss function of the FB coefficients obtained by the conventional fixed beamforming algorithm and the FB coefficients of the optimized fixed beamforming algorithm, as well as the STFT feature values ​​of the single-channel enhanced speech signal and the STFT feature values ​​of the single-channel clean speech signal, and to perform gradient backpropagation and model training to obtain a speech enhancement model.

[0036] In some specific embodiments, the embedded system includes a speech enhancement module configured to convolve a second noisy dual-channel speech signal and FB coefficients optimized by the speech enhancement model, and then sum them by weights to obtain a beamforming enhanced speech signal.

[0037] This invention proposes a microphone array speech enhancement method based on deep learning to optimize beam characteristics. Its technical advantages are as follows:

[0038] The beam characteristics are optimized using deep learning, and the main differences compared to existing deep network-optimized microphone array beamformers are as follows:

[0039] First, the input and output of the network differ. Existing neural network beamformers are either combined with a back-end recognition segment, and the output of the trained beamformer is the feature of the corresponding acoustic model, without needing to reconstruct the feature values ​​to obtain the enhanced speech signal; or they use a separate neural network beamformer, such as the FaSNet-TAC model, which concatenates the NCC feature values ​​with the feature values ​​of the encoded speech signal as the network input, and the network output is the enhanced speech time-domain signal. The proposed method takes generalized cross-correlation feature values ​​as its model input and outputs a time-domain signal.

[0040] Secondly, existing neural network beamformers require the use of trained deep network models for enhancement processing, which requires a large amount of storage and computation, and online support is required. The method disclosed in this invention only uses the fixed beamforming (FB) coefficients output by the model optimization for speech enhancement, and network optimization is implemented in the form of microphone array beamforming coefficients for enhancement. The computational complexity is low and can support low-complexity embedded processing on the terminal side. Attached Figure Description

[0041] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of this application. Other embodiments and many anticipated advantages of these embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0042] Figure 1 A flowchart of a microphone array speech enhancement method for optimizing beam characteristics using deep learning according to an embodiment of the present invention is shown;

[0043] Figure 2 A specific structural diagram of a microphone array speech enhancement method for optimizing beam characteristics using deep learning according to an embodiment of the present invention is shown.

[0044] Figure 3 A room impulse response diagram with a reverberation time of 0.6 s is shown according to a specific embodiment of the present invention;

[0045] Figure 4 A network framework diagram of a microphone array speech enhancement method for optimizing beam characteristics using deep learning according to a specific embodiment of the present invention is shown.

[0046] Figure 5 A schematic diagram of the experimental environment setup according to a specific embodiment of the present invention is shown;

[0047] Figure 6 The convergence curve of the training error of a speech enhancement model according to a specific embodiment of the present invention is shown;

[0048] Figure 7 A framework diagram of a microphone array speech enhancement system with deep learning-optimized beam characteristics according to a specific embodiment of the present invention is shown.

[0049] Figure 8 A framework diagram of the computer system and embedded system included in the microphone array speech enhancement system for optimizing beam characteristics using deep learning according to a specific embodiment of the present invention is shown. Detailed Implementation

[0050] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0051] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0052] A microphone array speech enhancement method based on deep learning for optimizing beam characteristics, according to one embodiment of this application. Figure 1 A flowchart of a microphone array speech enhancement method for optimizing beam characteristics using deep learning according to an embodiment of the present invention is shown. Figure 2 A specific structural diagram of a microphone array speech enhancement method for optimizing beam characteristics using deep learning according to an embodiment of the present invention is shown, as follows: Figure 1 and Figure 2 As shown, the method includes a model training phase and a speech enhancement phase, wherein the model training phase includes the following steps:

[0053] S1. Extract GCC feature values ​​from the first noisy dual-channel speech signal and use them as input to the model to obtain the GCC feature vector;

[0054] S2. Construct a spatial optimization deep learning network framework with the GCC feature vector as input and train the network to obtain the FB coefficients of the optimized fixed beamforming algorithm.

[0055] S3. Convolve the FB coefficients of the optimized fixed beamforming algorithm with the first noisy dual-channel speech signal, and sum them by weight to obtain a single-channel enhanced speech signal.

[0056] S4. The FB coefficients obtained by the traditional fixed beamforming algorithm and the STFT feature values ​​of the single-channel clean speech signal controlled by weights are proportionally fused to construct the model training target.

[0057] S5. Calculate the loss function of the FB coefficients obtained by the traditional fixed beamforming algorithm, the FB coefficients of the optimized fixed beamforming algorithm, the STFT feature value of the single-channel enhanced speech signal, and the STFT feature value of the single-channel clean speech signal, perform gradient backpropagation and model training to obtain the speech enhancement model.

[0058] The speech enhancement stage includes the following steps: convolving the second noisy dual-channel speech signal with the FB coefficients optimized by the speech enhancement model, and then weighting and summing them to obtain the beamforming enhanced speech signal.

[0059] In some specific embodiments, the model training phase is executed on the server and / or computer, while the speech enhancement phase is executed on an embedded device. By pre-training the model on the server or computer and then deploying the trained speech enhancement model on a low-complexity embedded device to perform the speech enhancement phase, the problem of having to deploy the optimized model on a server or cloud due to the large number of model parameters, which requires a network connection, is avoided in engineering applications. This avoids the difficulty of transmitting speech signals collected by small terminals such as wearable devices based on small-sized embedded platforms to a server or cloud for model enhancement, followed by speech recognition and other subsequent work, which is difficult to implement on small-sized embedded platforms such as wearable devices that do not have network access.

[0060] In some specific embodiments, the training phase requires the pre-setting of a training set to obtain a speech enhancement model through training with a first noisy dual-channel speech signal and a weighted single-channel clean speech signal. The data corpus used in the training set consists of corpora of Chinese and English languages, as shown in Table 1.

[0061] Table 1 Dataset

[0062] THCHS-30 train 10000 25 THCHS-30 dev 893 2.23 THCHS-30 test 2495 6.25 TIMIT train 4620 3.2 TIMIT test 1600 0.81

[0063] The Chinese corpus is the THCHS-30 dataset, an open-source Chinese speech database released by the Center for Speech and Language Technology (CSLT) at Tsinghua University. This dataset consists of single-channel signals recorded in a quiet office environment, totaling 33.48 hours in duration, with a sampling rate of 16 kHz. The English corpus is the TIMIT dataset, which is a single-channel speech signal with a sampling rate of 16 kHz, containing 6300 sentences spoken by 630 individuals from eight major dialect regions in the United States, each speaking 10 given sentences. To simulate real-world noisy environments, different types of noise were superimposed on the signals to simulate various noisy conditions, as shown in Table 2.

[0064] Table 2 Noise Library

[0065]

[0066]

[0067] The noise library consists of the NoiseX-92 noise library and the MUSAN corpus noise dataset. In this embodiment, the microphone array is a linear dual-microphone array with a spacing of 28mm, resulting in a small array size. The plane containing the array is divided into 7 angles, spaced 30° apart. The reverberation tool RIR_generator is used to simulate the dual-channel room impulse response (RIR) at different incident angles from 0° to 180°. RIRs with different reflection intensities and reverberation times are set. This paper sets two RIRs with reflection intensities of 0.6 and 0.8, and reverberation times of 0.3s, 0.4s, 0.5s, and 0.6s, for a total of 8 RIRs. The RIR with a reverberation time of 0.6s and reflection intensities of 0.6 and 0.8 is shown below. Figure 3 As shown, the number of simulated sound sources is 1.

[0068] The train and dev subsets from the THCHS-30 dataset and the train subset from the TIMIT dataset were used to construct the model training dataset. White noise and the MUSAN noise library from the noise library were used as the noise library for the training set, resulting in a total of 930 noise environments. Taking the generation of a simulated signal at a 0° angle as an example, the single-channel signal from the dataset was convolved with the dual-channel RIR signal to obtain a reverberant dual-channel speech signal. Then, it was randomly superimposed with noise signals from the noise library to generate dual-channel reverberant and noisy speech signals with SNRs of 0dB, 5dB, 10dB, and 15dB. Regardless of the type of noise signal, a total of 32 reverberant and noisy scene types were generated. The reverberation environment and SNR settings were the same for different angles of the same signal. The sampling rate of the signal was 16kHz, and a total of 427.42 hours of dual-channel reverberant and noisy speech signals at 7 angles were generated.

[0069] In some specific embodiments, step S1 constructs a spatially optimized deep learning network framework with the GCC feature vector as input for network training, for array elements M i and array element M j The received signal is set as follows:

[0070] x i =a i s(t-τ i )+n i (t)

[0071] x j =a j s(t-τ j )+n j (t)

[0072] Where s(t) represents the sound source signal, a i a jLet τ be the attenuation coefficient of the corresponding array element, τ be the corresponding time delay, and n(t) represent the environmental noise collected by the array element. Then, the cross-correlation function between the array elements can be expressed as:

[0073]

[0074] Among them G xixj (ω) represents the cross-power spectrum of the signals between array elements, X i (ω) and X j (ω) represents x i (t) and x j The Fourier transform of (t), (·) * Denotes taking the complex conjugate, where This represents the weighting function in the frequency domain. This embodiment uses the GCC of the PHAT weighting function.

[0075] In some specific embodiments, Figure 4 A network framework diagram of a microphone array speech enhancement method for optimizing beam characteristics using deep learning according to a specific embodiment of the present invention is shown. (Refer to...) Figure 2 The network framework in step S2 consists of an input layer, a hidden layer, and an output layer. The hidden layer includes at least one LSTM network layer and one fully connected layer, forming the main structure of the network. In this embodiment, the model structure is set to four LSTM layers, one fully connected layer, and one convolutional operation layer. The first LSTM layer has 5 neurons (GCC feature values ​​as the data dimension), the second and third LSTM layers have 128 neurons each, and the fourth and fifth LSTM layers have 256 neurons each. The network output FB coefficients are convolved with the noisy dual-channel speech signal in the convolutional operation layer to obtain an enhanced speech signal. The FB coefficients are the convolution kernels of the convolutional layer, and the noisy dual-channel speech signal is the input data.

[0076] In some specific embodiments, step S2 specifically includes the following steps:

[0077] S11. Input the GCC feature vector from the input layer into the hidden layer;

[0078] S12. The hidden layer is non-linearly activated based on the Tanh function, and the non-linearly activated GCC feature vector is input into the output layer.

[0079] S13. Obtain the FB coefficients of the optimized fixed beamforming algorithm through linear processing of the output layer. Nonlinear activation based on the Tanh function can avoid the problem of positive and negative values ​​in the FB coefficients.

[0080] In some specific embodiments, in step S4, which constructs the network training objective, the model optimizes the algorithm at the signal level. Therefore, the method disclosed in this application uses STFT features to construct a pair of STFT features for noisy dual-channel speech signal enhancement and STFT features for clean single-channel speech signal, as part of the network training objective. To enable the deep learning optimization beam of the network model to enhance the speech signal in the desired direction and suppress interference signals in other directions, achieving the expected spatial transmission response, the acquisition of the STFT feature value of the weighted clean single-channel speech signal includes attenuating the STFT feature of the clean single-channel speech signal by using a preset angle deviation as the weight for beam optimization, constructing the spatial transmission response, and calculating the angle deviation using the following formula:

[0081] Δθ≡|θ target -θ signal |

[0082] Where θ target Where θ represents the desired direction. signal This represents the actual direction of the sound source signal.

[0083] In some specific embodiments, the weight decays to 1 when the angular deviation is 0 degrees; to 0.5 when the angular deviation is 30 degrees; and to 0.001 when the angular deviation does not include 0 or 30 degrees. This ensures that the deep learning-optimized beam of the network model can enhance the speech signal in the desired direction and suppress interference signals in other directions, achieving the expected spatial transmission response.

[0084] In some specific embodiments, the speech enhancement stage of this embodiment is tested using test hardware and a test set. A binary linear array with an element spacing of 28mm is used, and the front-end acquisition uses an SPV1142LR5H-1 MEMS analog patch microphone. A batch of recorded dual-channel speech signals with reverberation were acquired in a reverberation speech laboratory. The room dimensions of the reverberation laboratory are 6.5×5.7×2.7m, and the reverberation time is 0.4s.

[0085] Experimental environment setup such as Figure 5As shown, a portion of the Aishell-3 high-fidelity Chinese speech database dataset was resampled to a 16kHz speech signal and used as the original single-channel speech for transcription. Noise signals from a noise library were extracted as noise sources. A Marshall Kilburn portable Bluetooth speaker was used to play the single-channel speech signal, while a dual-microphone array (as described in this embodiment) was used to acquire dual-channel speech signals. The distance between the sound source and the array was 1.5m, and both the sound source and the array were 1m above the ground. The sound source playback volume was controlled between 65dB and 75dB. During transcription, the array was rotated to change its receiving direction, obtaining dual-channel reverberant recorded speech signals at seven angles, each 30° apart, ultimately yielding 760 transcribed dual-channel speech signals. Following this experimental setup, the noise set for transcription was adjusted to obtain dual-channel noise signals at seven angles, ultimately yielding 94 transcribed noise signals, including various noises such as bell tolls, factory noise, pink noise, and rain sounds. To evaluate the algorithm's enhancement effect under noise interference from different directions in a noisy environment, a transcribed noise signal was randomly superimposed on the recorded signal with a transcription direction different from the speech signal direction, generating noisy speech signals with reverberation of -5dB, 0dB, and 5dB. The dataset used for model training did not include SNR signals of -5dB and 0dB, so this SNR was set for the algorithm's SNR speech signal generalization test.

[0086] In some specific embodiments, in the loss function calculation of step S5, this embodiment uses the MSE loss function, which is a type of deep learning loss function. The loss function is as follows:

[0087]

[0088] Where w1 and w2 are the fusion weights for fusing the two training objectives, y FSB Let y be the FB coefficient of the FSB algorithm. model_i Let y be the FB coefficients of the training model optimized for sample i. STFT_clean_i Let y be the STFT feature value corresponding to the single-channel clean speech signal of sample i. STFT_model_i The STFT feature value of the enhanced speech signal is obtained by weighted averaging of the vectors used to optimize the model and the convolution of sample i.

[0089] In step S5, after obtaining the loss function, the gradient descent method is used to minimize the loss function, and the backpropagation algorithm is used to update the parameters of the network model.

[0090] After debugging and verification, the hyperparameters for model training in this embodiment were set as optimal as follows: 300 training epochs, a batch size of 512, an exponentially decaying learning rate with an initial learning rate of 0.001 and a decay rate of 0.98, and network parameters initialized to follow a normal distribution with a mean of 0 and a variance of 0.01. Based on these parameter settings, model training was performed, training 7 sets of FB coefficients with different beam directions. The model training loss value for the FB coefficients aligned to a 180° angle enhancement varies with the number of epochs as shown below. Figure 6 As shown, the initial loss value is relatively large. As the model is trained, the training error gradually decreases, and the model's fit to the data improves.

[0091] As a performance comparison method for testing in this embodiment, two traditional microphone array enhancement methods, FSB and MVDR, and two deep learning microphone array enhancement methods, DCCRN and FaSNet-TAC, are used as comparison methods for this embodiment of the invention to compare the performance of enhanced speech quality and recognition rate.

[0092] Table 3 below shows the MOS-NET scores of different speech enhancement algorithms under different SNRs. Compared with the original noisy speech enhancement algorithm, the MVDR algorithm suppresses noise energy and improves the SNR value of the speech signal. Therefore, it improves the MOS-NET score of the speech signal by 0.06 under low SNR conditions. However, the MOS-NET score decreases slightly at 0dB and 5dB. The FSB algorithm and FaSNet-TAC model suppress noise energy and improve the SNR value of the speech signal, thus improving the speech listening quality and increasing the MOS-NET score. However, the improvement is not as great as that of the DCCRN model and DLOB model. The DCCRN model has strong noise suppression, so the SNR value is significantly improved. However, the loss of high-frequency speech signals reduces the listening quality of the speech. At SNRs of 0dB and 5dB, the MOS-NET score is not as good as that of the DLOB model in this embodiment.

[0093] Table 3. MOS-NET scores of different speech enhancement algorithms

[0094]

[0095] To evaluate the recognition performance of the enhanced speech, an end-to-end model was adopted, employing a hybrid connectivity temporal classification and attention mechanism architecture. The recognition rate of the enhanced speech was tested using speech recognition software that had undergone multi-task learning and was fully trained.

[0096] Table 4 shows the improvement in recognition rate of the five speech enhancement algorithms compared to the original noisy speech signal under three low signal-to-noise ratio (SNR) conditions: -5dB, 0dB, and 5dB. As can be seen from Table 4, the SNR of the original noisy speech signal decreases with increasing noise intensity, and the recognition rate continuously declines, reaching 80.9% under the -5dB SNR condition. Five speech enhancement algorithms showed varying degrees of improvement in recognition rate under different SNR conditions. While the DCCRN model suppressed most noise and improved the MOS-NET score, the lack of signal in the speech signal frequency band led to a deterioration in the speech recognition rate. The FaSNet-TAC model also suppressed noise in noisy speech signals and improved the SNR, but the presence of speech signal distortion resulted in a decrease in recognition rate. The recognition rate of speech signals processed by the MVDR algorithm decreased slightly, which is related to the varying suppression effects of the MVDR algorithm on noise at different angles; the suppression effect was poor at some noise incidence angles. The FSB algorithm and the DLOB model both improved the recognition rate, with the SONET model showing the best improvement, increasing the recognition rate by 6.47%, 2.67%, and 2% at -5dB, 0dB, and 5dB SNRs, respectively. The DLOB model disclosed in this invention can improve both signal quality and speech recognition rate in a simulated interactive environment, demonstrating good application performance.

[0097] Table 4. Recognition rates of different speech enhancement algorithms

[0098]

[0099] As another aspect, a microphone array speech enhancement system with deep learning-optimized beam characteristics according to an embodiment of this application, Figure 7 A framework diagram of a microphone array speech enhancement system based on deep learning for optimizing beam characteristics, according to a specific embodiment of the present invention, is shown. Figure 7 As shown, the system includes a computer system 100 and an embedded system 200. Figure 8 A framework diagram of the computer system and embedded system included in a microphone array speech enhancement system for optimizing beam characteristics using deep learning according to a specific embodiment of the present invention is shown, as follows: Figure 8 As shown, the computer system 100 includes a feature extraction module, a network training module, a weighted summation module, a training target generation module, and a loss function calculation module; the embedded system includes a speech enhancement module.

[0100] The feature extraction module is configured to extract GCC feature values ​​from the first noisy dual-channel speech signal as input to the model to obtain the GCC feature vector;

[0101] The network training module is configured to construct a spatially optimized deep learning network framework with the GCC feature vector as input for network training, and obtain the FB coefficients of the optimized fixed beamforming algorithm.

[0102] The weighted summation module is configured to convolve the FB coefficients of the optimized fixed beamforming algorithm with the first noisy dual-channel speech signal and sum them to obtain a single-channel enhanced speech signal.

[0103] The training target generation module is configured to proportionally fuse the FB coefficients obtained by the traditional fixed beamforming algorithm and the STFT feature values ​​of the weighted single-channel clean speech signal to construct the model training target.

[0104] The model generation module is configured to calculate the loss function of the FB coefficients obtained by the conventional fixed beamforming algorithm and the FB coefficients of the optimized fixed beamforming algorithm, as well as the STFT feature values ​​of the single-channel enhanced speech signal and the STFT feature values ​​of the single-channel clean speech signal, perform gradient backpropagation and model training to obtain a speech enhancement model.

[0105] In some specific embodiments, the embedded system 200 includes a speech enhancement module configured to convolve a second noisy dual-channel speech signal and FB coefficients optimized by the speech enhancement model, and then sum them by weights to obtain a beamforming enhanced speech signal.

[0106] The technology of this invention can be implemented by a wide variety of devices or apparatuses, including desktop computers, laptop computers, embedded systems, tablet computers, set-top boxes, handsets (e.g., so-called smartphones), televisions, cameras, display devices, digital media players, video game consoles, etc. Furthermore, such technology can be implemented using integrated circuits (ICs) or a set of ICs (e.g., chipsets). A device configured to implement the technology of this invention may comprise any of the aforementioned devices, and in some cases, may be formed by a combination of hardware, software, and firmware. The description of various modules in this invention may be intended to emphasize functional aspects of a device configured to implement the disclosed technology, but does not necessarily require implementation by different hardware units.

[0107] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A microphone array speech enhancement method for optimizing beam characteristics using deep learning, characterized in that, It includes a model training phase and a speech enhancement phase, wherein the model training phase includes the following steps: S1. Extract GCC feature values ​​from the first noisy dual-channel speech signal and use them as input to the model to obtain the GCC feature vector; S2. Construct a spatial optimization deep learning network framework with the GCC feature vector as input and train the network to obtain the FB coefficients of the optimized fixed beamforming algorithm. S3. Convolve the FB coefficients of the optimized fixed beamforming algorithm with the first noisy dual-channel speech signal, and sum them by weight to obtain a single-channel enhanced speech signal; wherein, the FB coefficients are the convolution kernels of the convolution operation layer, and the first noisy dual-channel speech signal is the input data of the convolution operation layer; S4. The FB coefficients obtained by the traditional fixed beamforming algorithm and the STFT feature values ​​of the single-channel clean speech signal controlled by weights are proportionally fused to construct the model training target. S5. Calculate the loss function of the FB coefficients obtained by the traditional fixed beamforming algorithm, the FB coefficients of the optimized fixed beamforming algorithm, the STFT feature value of the single-channel enhanced speech signal, and the STFT feature value of the single-channel clean speech signal. The loss function is the MSE loss function, expressed as: in To fuse the corresponding fusion weights of the two training objectives, The FB coefficients of the FSB algorithm, For the sample The corresponding FB coefficients for optimizing the trained model. For the sample The STFT characteristic value corresponding to a single-channel clean speech signal. Optimize vectors for the model. STFT feature values ​​of the enhanced speech signal obtained by convolution weighted averaging; Gradient backpropagation and model training are performed based on the loss function to obtain a speech enhancement model; The speech enhancement stage includes the following steps: convolving the second noisy dual-channel speech signal with the FB coefficients optimized by the speech enhancement model, and then weighting and summing them to obtain the beamforming enhanced speech signal.

2. The microphone array speech enhancement method for optimizing beam characteristics using deep learning according to claim 1, characterized in that, Step S1 involves constructing a spatially optimized deep learning network framework using the GCC feature vector as input for network training, for each array element. Harmony Formation The received signal is set as follows: in Indicates the sound source signal. This represents the attenuation coefficient of the corresponding array element. For the corresponding delay time, If we represent the environmental noise collected by the array elements, then the cross-correlation function between the array elements can be expressed as: in The cross-power spectrum of signals between array elements. and They are respectively and Fourier transform, Denotes taking the complex conjugate, where This represents the weighting function in the frequency domain.

3. The microphone array speech enhancement method for optimizing beam characteristics using deep learning according to claim 1, characterized in that, The network framework in step S2 consists of an input layer, a hidden layer, and an output layer. The hidden layer includes at least one LSTM network layer and one fully connected layer.

4. The microphone array speech enhancement method for optimizing beam characteristics using deep learning according to claim 3, characterized in that, Step S2 specifically includes the following steps: S11. Input the GCC feature vector from the input layer into the hidden layer; S12. The hidden layer is non-linearly activated based on the Tanh function, and the non-linearly activated GCC feature vector is input into the output layer. S13. Obtain the FB coefficients of the optimized fixed beamforming algorithm through linear processing of the output layer.

5. The microphone array speech enhancement method for optimizing beam characteristics using deep learning according to claim 1, characterized in that, In step S4, which constructs the model training objective, obtaining the STFT feature value of the weighted single-channel clean speech signal includes attenuating the STFT feature of the single-channel clean speech signal by using a preset angle deviation as the weight for optimizing the beam, constructing the spatial transmission response, and calculating the angle deviation using the following formula: in For the desired direction, This represents the actual direction of the sound source signal.

6. The microphone array speech enhancement method for optimizing beam characteristics using deep learning according to claim 5, characterized in that, The preset angle deviation includes When the angular deviation is 0 degrees, the weight decays to 1. When the angular deviation is 30 degrees, the weight decays to 0.

5. When the angular deviation does not include 0 and 30 degrees, the weight decays to 0.

001.

7. The microphone array speech enhancement method for optimizing beam characteristics using deep learning according to claim 1, characterized in that, Step S5 includes minimizing the loss function using gradient descent, and the backpropagation of the gradient includes updating the parameters of the network model using the backpropagation algorithm.

8. The microphone array speech enhancement method for optimizing beam characteristics using deep learning according to claim 1, characterized in that, The model training phase is executed on the server or computer, while the speech enhancement phase is executed on an embedded device.

9. A microphone array speech enhancement system for optimizing beam characteristics using deep learning, characterized in that, It includes a computer system and an embedded system. The computer system includes a feature extraction module, a network training module, a weighted summation module, a training target generation module, and a loss function calculation module. The embedded system includes a speech enhancement module. The feature extraction module is configured to extract GCC feature values ​​from the first noisy dual-channel speech signal as input to the model to obtain a GCC feature vector. The network training module is configured to construct a spatially optimized deep learning network framework with the GCC feature vector as input for network training, and obtain the optimized FB coefficients of the fixed beamforming algorithm. The weighted summation module is configured to convolve the FB coefficients of the optimized fixed beamforming algorithm and the first noisy dual-channel speech signal, and then sum them to obtain a single-channel enhanced speech signal; wherein, the FB coefficients are the convolution kernels of the convolution operation layer, and the first noisy dual-channel speech signal is the input data of the convolution operation layer. The training target generation module is configured to proportionally fuse the FB coefficients obtained by the traditional fixed beamforming algorithm and the STFT feature values ​​of the weighted single-channel clean speech signal to construct the model training target. The model generation module is configured to calculate a loss function for the FB coefficients obtained by the traditional fixed beamforming algorithm and the FB coefficients of the optimized fixed beamforming algorithm, as well as the STFT feature values ​​of the single-channel enhanced speech signal and the STFT feature values ​​of the single-channel clean speech signal. The loss function is the MSE loss function, expressed as: in To fuse the corresponding fusion weights of the two training objectives, The FB coefficients of the FSB algorithm, For the sample The corresponding FB coefficients for optimizing the trained model. For the sample The STFT characteristic value corresponding to a single-channel clean speech signal. Optimize vectors for the model. STFT feature values ​​of the enhanced speech signal obtained by convolution weighted averaging; Gradient backpropagation and model training are performed based on the loss function to obtain a speech enhancement model.

10. A microphone array speech enhancement system for optimizing beam characteristics using deep learning according to claim 9, characterized in that, The embedded system includes a speech enhancement module configured to convolve the second noisy dual-channel speech signal and the FB coefficients optimized by the speech enhancement model, and then sum them by weights to obtain the beamforming enhanced speech signal.

Images