A microphone array speech enhancement system and method based on multi-task network

Abstract

The present invention discloses a microphone array speech enhancement system and method based on a multi-task network. The system comprises a speech preprocessing module, a multi-task network module, a multi-task loss statistics module, a network weight calculation module, and a speech reconstruction module. The speech preprocessing module obtains array speech, reference echo speech, and target speech of each task as input speech and performs preprocessing; the multi-task network module performs dereverberation, echo cancellation, and noise reduction tasks for each channel of the array speech, and outputs multi-channel speech fusion as enhanced speech; the multi-task loss statistics module is used to calculate the loss value of each task in the multi-task network module and calculate the total loss of the network; the network weight calculation module calculates the gradient based on the total loss of the network, back-propagates the gradient, and calculates the weight of the updated network; and the speech reconstruction module completes the mapping from frequency domain features to time domain speech to obtain enhanced clean speech.

Description

Technical Field

[0001] The present invention relates to the technical field of speech enhancement, and in particular to a microphone array speech enhancement system and method based on a multi-task network. Background Art

[0002] Speech enhancement based on microphone arrays is one of the effective methods for suppressing interference in voice communication systems. Existing microphone array speech enhancement technologies can be mainly divided into two categories: traditional enhancement technologies and enhancement technologies based on deep neural networks. Traditional enhancement technologies usually design corresponding filters for the relevant characteristics of echo cancellation, spatial reverberation, and environmental noise. They have low requirements for hardware computing power and can achieve good real-time performance. However, traditional enhancement technologies still have the following shortcomings: (1) they cannot effectively eliminate nonlinear noise components, resulting in nonlinear echo residues in the output speech; (2) they have poor ability to suppress non-stationary burst noise, resulting in low output speech quality.

[0003] In recent years, thanks to the rapid improvement of computer hardware computing power, deep neural network technology has been widely used. Compared with traditional enhancement technology, enhancement technology based on deep neural network does not require noise estimation and has stronger adaptability to non-stationary noise. In current research, an integrated deep neural network is usually designed to simultaneously complete echo cancellation, reverberation and denoising. The integrated network model can achieve relatively good results in a relatively single interference environment, but these networks still have the following shortcomings: (1) When multiple types of interference coexist, the training difficulty of the network is greatly increased, and the network is difficult to learn the mapping relationship between input features and labels; (2) When the network training target is single, it is easy to fall into local minimum, thus failing to achieve a good enhancement effect; (3) In the testing phase, the generalization performance is insufficient for different noise reduction tasks. Summary of the Invention

[0004] The purpose of the present invention is to address the shortcomings of existing microphone array speech enhancement technologies based on deep neural networks and to provide a microphone array speech enhancement system and method based on a multi-task network. This system and method addresses the shortcomings of existing network systems, such as difficulty in training and susceptibility to local minima. By refining the structure of the deep neural network according to the functions of the speech enhancement system, a multi-task speech enhancement network model consisting of an echo cancellation subnetwork, a dereverberation subnetwork, and a noise reduction subnetwork is provided. This effectively reduces the difficulty of network training, compensates for the shortcomings of a single network objective, and significantly improves the effect of speech enhancement. The present invention can be widely used in online voice conferencing, car phones, mobile call terminals, and other occasions.

[0005] The first object of the present invention can be achieved by adopting the following technical solutions:

[0006] A multi-task based microphone array speech enhancement system consists of a speech preprocessing module, a multi-task network module, a multi-task loss statistics module, a network weight calculation module and a speech reconstruction module. Among them, the speech preprocessing module is connected to the multi-task network module and the multi-task loss statistics module respectively. This module obtains the array speech, reference echo speech and each task target speech as input speech, and preprocesses these input speech. The preprocessing work includes normalization of the speech signal and extraction of the logarithmic amplitude spectrum of each channel speech and the reference echo speech; the multi-task network module is connected to the speech preprocessing module, the multi-task loss statistics module, the network weight calculation module and the speech reconstruction module respectively, completes the dereverberation, echo cancellation and noise reduction tasks of each channel of the array speech, and outputs the multi-channel speech fusion as enhanced speech; the multi-task loss statistics module is connected to the multi-task network module and the network weight calculation module respectively, calculates the loss value of each task in the multi-task network module, and counts the total loss of the network; the network weight calculation module is connected to the multi-task network module and the multi-task loss statistics module respectively, calculates the gradient according to the total loss of the network, backpropagates the gradient, and calculates the weight of the updated network; the speech reconstruction module is connected to the multi-task network module to complete the mapping from frequency domain features to time domain speech, and obtains enhanced clean speech.

[0007] Furthermore, the multi-task network module includes M parallel channel branch networks and a channel fusion sub-network respectively connected to the above-mentioned M channel branch networks, each of the above-mentioned channel branch networks includes a dereverberation sub-network, an echo cancellation sub-network and a noise reduction sub-network connected in sequence, wherein M is the same as the number of array elements of the microphone array in the speech preprocessing module, the dereverberation sub-network in each channel branch network is connected to the echo cancellation sub-network, the multi-task loss statistics module and the speech preprocessing module to complete the reverberation removal task of the channel; the echo cancellation sub-network in each channel branch network is connected to the multi-task loss statistics module and the speech preprocessing module to complete the reverberation removal task of the channel; The sound cancellation subnetwork is connected to the dereverberation subnetwork, the noise reduction subnetwork, the speech preprocessing module, and the multi-task loss statistics module, and uses the reference echo signal to eliminate the echo component of the channel. The noise reduction subnetwork in each channel branch network is connected to the echo cancellation subnetwork, the channel fusion subnetwork, and the multi-task loss statistics module to eliminate the noise component in the channel. The channel fusion subnetwork is connected to the noise reduction subnetworks in the M channel branch networks, the multi-task loss statistics module, and the network weight calculation module, and uses the spatial information contained in each channel of the array speech to perform secondary enhancement. In the multi-task network module, each channel branch network is used to enhance the channel signals of the array speech, and the enhanced output is passed through the channel fusion subnetwork in parallel to obtain the enhanced single-channel speech. Each of the above channel branch networks utilizes different functions in the speech enhancement system to divide the network into subnetworks for dereverberation, echo cancellation, and noise reduction tasks, making the subnetwork tasks clearer and the network easier to train.

[0008] Furthermore, the multi-task loss statistics module is composed of a dereverberation loss statistics module, an echo cancellation loss statistics module, a noise reduction loss statistics module, a channel fusion loss statistics module and a total loss statistics module. The dereverberation loss statistics module is respectively connected to the speech preprocessing module, the dereverberation subnetwork of each channel and the total loss statistics module, and uses the output of the dereverberation subnetwork and the dereverberation label to calculate the loss value of the dereverberation subnetwork of each channel; the echo cancellation loss statistics module is respectively connected to the speech preprocessing module, the echo cancellation subnetwork of each channel and the total loss statistics module, and uses the output of the echo cancellation subnetwork and the dereverberation and echo label to calculate the loss value of the echo cancellation subnetwork of each channel; the noise reduction loss statistics module is respectively connected to the speech preprocessing module, the noise reduction subnetwork of each channel and the total loss statistics module. The statistics module is connected to the above-mentioned preprocessing module, and the loss value of each channel noise reduction sub-network is calculated using the output of the noise reduction sub-network and the noise-free label; the channel fusion loss statistics module is respectively connected to the above-mentioned speech preprocessing module, the above-mentioned channel fusion sub-network and the total loss statistics module, and the loss value of the channel fusion sub-network is calculated using the output of the channel fusion sub-network and the clean speech label; the total loss statistics module is respectively connected to the dereverberation loss statistics module, the echo cancellation loss statistics module, the noise reduction loss statistics module, the channel fusion loss statistics module and the above-mentioned network weight calculation module, receives the loss value of each task loss statistics module, and calculates the total loss value of the network. The multi-task loss statistics module includes the loss statistics modules of each sub-network mentioned above, which calculates the target loss for the output of each sub-network and obtains the loss value of the total network by statistics. The multi-task loss statistics module enables each sub-network to adjust the network parameters according to the preset task during training.

[0009] Another object of the present invention can be achieved by adopting the following technical solutions:

[0010] A microphone array speech enhancement method based on a multi-task network, the speech enhancement method comprising the following steps:

[0011] S1. Construct an array speech training set and preprocess the speech to obtain the input features of each channel and the labels of the dereverberation task, echo cancellation task, noise reduction task, and fusion task;

[0012] S2. Input the features obtained from the preprocessing of each channel into the multi-task network module. After passing through the dereverberation sub-network, echo cancellation sub-network, and noise reduction sub-network corresponding to each channel, they are fused in the channel fusion sub-network. From the output ends of the above sub-networks, the dereverberated target output, the dereverberated and echo-free target output, the noise-free target output, and the fused clean speech target output of each channel are obtained respectively.

[0013] S3. Input the network output and labels of each task into the multi-task loss statistics module, calculate the loss value of the echo cancellation sub-network, the loss value of the reverberation sub-network, the loss value of the noise reduction sub-network, and the loss value of the channel fusion sub-network in each channel, and calculate the total network loss based on the above loss values;

[0014] S4. Calculate the gradient of the total network loss and use backpropagation to update the network weights.

[0015] S5. Repeat steps S2, S3, and S4 until the total loss of the network is lower than a preset threshold, and save the network parameters at this time;

[0016] S6. Preprocess the test speech, calculate the input features of each channel, input the speech features to be enhanced into the multi-task network saved in step S5 to obtain enhanced speech features, and input the enhanced speech features into the speech reconstruction module to obtain enhanced speech.

[0017] Furthermore, the construction process of the array speech training set in step S1 is as follows:

[0018] S1.1. Constructing a noisy array speech and corresponding dereverberated array speech, dereverberated and echo-free array speech, and noise-free array speech:

[0019] The noisy array speech is x(n)=[x1(n),x2(n),...,x m (n),...,x M (n)] T ,m∈[1,M], where

[0020]

[0021] x m (n) is the noisy speech signal received by the mth array element, M is the number of array elements in the microphone array, s′(n) is the source speech component containing reverberation, d(n) is the echo component containing reverberation, v(n) is the ambient noise, s(n) is the source speech, e(n) is the reference echo speech, τ m (θ) is the time delay between array element m and the reference element when the signal from a sound source with a direction angle of θ is received. * represents the convolution operation. h(n) is the multipath propagation channel parameter in the room. ser is the randomly distributed signal-to-echo ratio. snr is the randomly distributed signal-to-noise ratio. The distribution of noisy speech in the training set should cover as many noise types and interference intensities as possible, so that the network can learn as many noisy speech features as possible and avoid overfitting.

[0022] The dedeverberated array speech is

[0023] y dereb(n)=[y 1_dereb (n),y 2_dereb (n),...,y m_dereb (n),...,y M_dereb (n)] T ,m∈[1,M], where

[0024] y m_dereb (n) = s(n-τ m (θ))+e(n-τ m (θ))+v(n) (2)

[0025] The array speech that has been de-reverberated and de-echoed is

[0026] y decho (n)=[y 1_decho (n),y 2_decho (n),...,y m_decho (n),...,y M_decho (n)] T ,m∈[1,M], where

[0027] y m_decho (n) = s(n-τ m (θ))+v(n) (3)

[0028] The noise-free array speech is

[0029] y enh (n)=[y 1_enh (n),v 2_enh (n),...,y m_enh (n),...,y M_enh (n)] T ,m∈[1,M], where

[0030] y m_enh (n) = s(n-τ m (θ)) (4)

[0031] Among them, y m_dereb (n) is the de-reverberated speech of the m-th array element, y m_enh (n) is the de-reverberated and de-echoed speech of the m-th array element, y m_enh (n) is the noise-free speech of the mth array element. The above three speech structures will be used to subsequently produce labels for each sub-network in the vocal tract branch network.

[0032] S1.2. Extract the input features of the network: the speech signal x received by the array element m m (n) Perform short-time Fourier transform to obtain the logarithmic amplitude spectrum X of each frame m(i,k), k is the discrete frequency number, i∈[0,N] is the frame number, N is the total number of frames, the i-th frame is combined with the logarithmic amplitude spectrum of the previous l frames to obtain the network input feature of the i-th frame of the m-th channel The reference echo speech e(n) undergoes the same processing to obtain the logarithmic amplitude spectrum E(i,k) of the reference echo. The purpose of expanding the input by l frames is to allow the network to obtain more information from past time frames, thereby helping the network to recover the speech signal of the current i-th frame;

[0033] S1.3, construct the labels of each task of the mth channel: m_dereb (n), y m_decho (n), y m_enh Perform short-time Fourier transform on the (n) and s(n) speech to obtain the amplitude spectrum of each frame And S(i,k), k is the discrete frequency number, i∈[0,N] is the frame number, N is the total number of frames, and the target ideal ratio mask (Ideal Ratio Mask, IRM) is used as the label of each task. IRM is calculated using the following formula, Where T m (i, k) is the expected amplitude spectrum of the target speech, β is the adjustable scale factor, M m (i,k)=exp(X m (i,k)) is the amplitude spectrum of the noisy speech, k is the discrete frequency number, i is the frame number, and the above obtained and Replace T in formula (5) m (i,k), get the label of the de-reverberated Tags with reverb and echo removed and denoised labels In addition, S(i,k) is logarithmically transformed to obtain the logarithmic amplitude spectrum label S of the clean speech lms .

[0034] Furthermore, in step S2, the process of inputting the features into the multi-task network module to obtain the target output of each task sub-network is as follows:

[0035] S2.1、 Input the dereverberation subnetwork of the mth channel and obtain the IRM output of the desired dereverberation-free speech Using IRM as the network's objective can simultaneously utilize information from both noisy and clean speech, allowing the network to learn more features.

[0036] S2.2, and E(i,k) are input into the echo cancellation subnetwork of the mth channel to obtain the IRM output of the desired speech with removed reverberation and echo. The echo cancellation subnetwork takes as input the noisy speech and the reference signal. The network uses the amplitude spectrum features E(i,k) of the reference echo signal to eliminate the echo component in the noisy speech.

[0037] S2.3, the noise reduction sub-network of the mth channel contains two output layers, one output layer outputs the logarithmic amplitude spectrum of the noise-reduced speech, and the other output layer outputs the IRM of the noise-reduced speech. Input the denoising subnetwork and get the logarithmic amplitude spectrum output and IRM output The denoising sub-network uses a two-input layer structure, which enables the network to extract data features from different dimensions and obtain different outputs, providing higher-dimensional input for the subsequent channel fusion sub-network.

[0038] S2.4. Constructing the Input of the Channel Fusion Subnetwork

[0039] in

[0040]

[0041] Use one The convolution layer of the layer is used as the channel fusion sub-network, M is the total number of array elements, d is the width of the convolution kernel, for the input X fuse_in Its dimension can be expressed as X fuse_in ∈R 2×M×k , convolve the matrix along the M-dimensional direction to obtain the monophonic logarithmic amplitude spectrum after the channel fusion subnetwork enhancement The convolutional structure of the channel fusion subnetwork can focus on learning the feature information between channels, and fuse the outputs of each channel into a single channel while maintaining the original enhancement results of each channel as much as possible to complete the secondary enhancement.

[0042] Furthermore, the process of obtaining the total loss value by the multi-task loss statistics module in step S3 is as follows:

[0043] S3.1, the output of the dereverberation sub-network and IRM tags for dedeverberated speech The total loss value L of each channel dereverberation sub-network is calculated by the following formula dereb The loss importance of the dereverberation sub-network of each channel is the same, so a direct accumulation method is used with the same weight;

[0044]

[0045] S3.2, the output of the echo cancellation sub-network and tags for voices with de-reverberation and echo removed The echo cancellation sub-network loss value L of each channel is calculated by the following formuladecho ,Here, each channel echo cancellation sub-network uses the same loss weight and is directly accumulated;

[0046]

[0047] S3.3. Output the denoising sub-network and noise-free labels The loss value L of each channel noise reduction sub-network is calculated by the following formula: enh ,Here, each channel noise reduction sub-network uses the same loss weight and is directly accumulated;

[0048]

[0049] S3.4. Output of the channel fusion sub-network and the logarithmic magnitude spectrum S of the clean speech lms The loss L of the channel fusion sub-network is calculated by the following formula fuse The Log-Cosh function has a high tolerance for outliers in the data, and its gradient decreases near the minimum value. Unlike the mean square error loss, which always uses the same loss, it may cause the minimum value to be missed.

[0050]

[0051] S3.5, the loss value L of each task module dereb 、L decho 、L enh and L fuse The total network loss is calculated as follows. No weight parameters are added to each sub-network, and the target loss weights are the same, treating each task as equally important:

[0052] loss = L dereb +L decho +L enh +L fuse (11).

[0053] Furthermore, in step S6, the enhanced speech features are reconstructed into enhanced speech using the following method:

[0054] The logarithmic magnitude spectrum of the channel fusion sub-network output Convert to linear magnitude spectrum Then with x m Phase spectrum of (n) Multiplication to obtain the complex representation of the enhanced speech Will The inverse short-time Fourier transform and overlap-add are performed to obtain the time domain waveform of the enhanced speech. The overlap-add method can be used to restore the speech frames that were framed for short-time Fourier transform during feature extraction.

[0055] The present invention has the following advantages and effects compared to the prior art:

[0056] (1) The present invention adopts multiple enhanced task sub-networks connected in series in the channel branch network to perform targeted noise reduction on the various noise components contained in the noise, alleviating the problem that the single target network model is difficult to fit and train when facing multiple types of noise, thereby having stronger noise reduction performance.

[0057] (2) The present invention makes full use of the correlation between different noise reduction tasks and designs the echo cancellation task objectives, dereverberation objectives, noise reduction objectives and channel fusion objectives. Through the interaction between different tasks, the problem that the single task network tends to fall into the local minimum during the back propagation of the gradient is avoided.

[0058] (3) The present invention adopts a channel branch network to independently process each channel of the array speech, and adopts a channel fusion sub-network to perform secondary enhancement on the enhanced features of each channel. It can achieve speech enhancement without obtaining prior information about the direction of the speaker's speech wave and noise, thus overcoming the shortcomings of traditional microphone array performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0060] Figure 1 1 is a structural block diagram of a microphone array speech enhancement system based on a multi-task network disclosed in an embodiment of the present invention;

[0061] Figure 2 This is a network structure diagram of a multi-task network module in an embodiment of the present invention;

[0062] Figure 3 1 is a flow chart of training and testing of an array speech enhancement method using a multi-task network according to an embodiment of the present invention;

[0063] Figure 4 Schematic diagram of generating noisy speech in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0065] Example 1

[0066] This embodiment discloses a microphone array speech enhancement system based on a multi-task network. The system structure is as follows: Figure 1 As shown in the figure, the system consists of a speech preprocessing module, a multi-task network module, a multi-task loss statistics module, a network weight calculation module and a speech reconstruction module. The speech preprocessing module is connected to the multi-task network module and the multi-task loss statistics module. This module obtains the array speech, reference echo speech and each task target speech as input speech, and preprocesses these input speech. The preprocessing work includes normalizing the speech signal and extracting the logarithmic amplitude spectrum of each channel speech and the reference echo speech; the multi-task network module is connected to the speech preprocessing module, the multi-task loss statistics module and the network weight calculation module to complete the dereverberation, echo cancellation and noise reduction tasks of each channel of the array speech, and outputs the multi-channel speech fusion as enhanced speech; the multi-task loss statistics is connected to the multi-task network module and the network weight calculation module to calculate the loss value of each task in the multi-task network module and calculate the total loss of the network; the network weight calculation module is connected to the multi-task network module and the multi-task loss statistics module to calculate the gradient according to the total loss of the network, backpropagate the gradient, and calculate the weight of the updated network; the speech reconstruction module is connected to the multi-task network module to complete the mapping from frequency domain features to time domain speech, and obtain enhanced clean speech.

[0067] In this embodiment, the network structure of the multi-task network module is as follows: Figure 2As shown in the figure, it includes 4 channel branch networks and 1 channel fusion subnetwork. The above channel fusion subnetwork is built by a 3-layer fully convolutional neural network. Each channel branch network is composed of a dereverberation subnetwork, an echo cancellation subnetwork, and a noise reduction subnetwork. The 4 channel branch networks correspond to the 4 array elements of the microphone array. The dereverberation subnetwork is built by a 2-layer unidirectional long short-term memory network (Long short-term memory, LSTM) and a fully connected output layer. The echo cancellation subnetwork is built by a 2-layer bidirectional LSTM and a fully connected output layer. The noise reduction subnetwork is built by a 2-layer unidirectional LSTM and two fully connected output layers. The dereverberation subnetwork in each channel branch network is connected to the echo cancellation subnetwork, the above-mentioned multi-task loss statistics module and the above-mentioned speech preprocessing module to complete the reverberation removal task of the channel; the echo cancellation subnetwork in each channel branch network is connected to the dereverberation subnetwork, the noise reduction subnetwork, the above-mentioned speech preprocessing module and the above-mentioned multi-task loss statistics module, and uses the reference echo signal to eliminate the echo component of the channel; the noise reduction subnetwork in each channel branch network is connected to the echo cancellation subnetwork, the channel fusion subnetwork and the above-mentioned multi-task loss statistics module to remove the noise component in the channel; the multi-channel fusion subnetwork is connected to the noise reduction subnetworks in the 4 channel branch networks, the above-mentioned multi-task loss statistics module and the above-mentioned network weight calculation module, and uses the spatial information contained in each channel of the array speech for secondary enhancement.

[0068] The above-mentioned multi-task loss statistics module is composed of a dereverberation loss statistics module, an echo cancellation loss statistics module, a noise reduction loss statistics module, a channel fusion loss statistics module and a total loss statistics module. The dereverberation loss statistics module is connected to the above-mentioned speech preprocessing module, the above-mentioned dereverberation sub-networks of each channel and the total loss statistics module, and uses the output of the dereverberation sub-network and the dereverberation label to calculate the loss value of the dereverberation sub-network of each channel; the echo cancellation loss statistics module is connected to the above-mentioned speech preprocessing module, the above-mentioned echo cancellation sub-networks of each channel and the total loss statistics module, and uses the output of the echo cancellation sub-network and the dereverberation and echo label to calculate the loss value of the echo cancellation sub-network of each channel; the noise reduction loss statistics module is connected to the above-mentioned speech preprocessing module, the above-mentioned noise reduction sub-networks of each channel and the total loss statistics module. The blocks are connected, and the output of the denoising sub-network and the noise-free label are used to calculate the loss value of each channel denoising sub-network; the channel fusion loss statistics module is connected with the above-mentioned speech preprocessing module, the above-mentioned channel fusion sub-network and the total loss statistics module, and the output of the channel fusion sub-network and the clean speech label are used to calculate the loss value of the channel fusion sub-network; the total loss statistics module is connected with the dereverberation loss statistics module, the echo cancellation loss statistics module, the denoising loss statistics module, the channel fusion loss statistics module and the above-mentioned network weight calculation module, receives the loss value of each task loss statistics module, and counts the total loss value of the network.

[0069] Example 2

[0070] Based on the above embodiment, a microphone array speech enhancement system based on a multi-task network is disclosed. This embodiment further discloses a microphone array speech enhancement method based on a multi-task network. The method uses the following steps to complete training and testing. The training and testing process is as follows: Figure 3 As shown:

[0071] S1. Construct an array speech training set and preprocess the speech to obtain the input features of each channel and the labels of the dereverberation task, echo cancellation task, noise reduction task, and fusion task. The process is as follows:

[0072] S1.1. Constructing a noisy array speech and corresponding dereverberated array speech, dereverberated and echo-free array speech, and noise-free array speech:

[0073] The noisy array speech is x(n)=[x1(n),x2(n),...,x m (n),...,x M (n)] T ,m∈[1,M], where M=4 is the total number of array elements. The generation of noisy speech is as follows Figure 4 As shown, x m (n) is the noisy speech signal received by the mth array element,

[0074]

[0075] s′(n) is the source speech component including reverberation, d(n) is the echo component including reverberation, v(n) is the ambient noise, s(n) is the source speech, e(n) is the reference echo speech, τ m (θ) is the time delay relative to the reference element when array element m receives the signal generated by the sound source with a direction angle of θ. * represents the convolution operation. The signal-to-echo ratio ser∈[-10,10] is randomly distributed, and the signal-to-noise ratio snr∈[-5,15] is randomly distributed. h(n) is the multipath propagation channel parameter in the room, which is determined by the reverberation time, room size, sound source, and array position. In this embodiment, the reverberation time is randomly selected from 0.2s / 0.3s / 0.4s / 0.5s / 0.6s. The room size is randomly selected from 5 / 7 / 9 / 11 / 13m in length, 4 / 6 / 8 / 10m in width, and 3m in height. Five array positions are randomly selected in the room.

[0076] The dedeverberated array speech is

[0077] y dereb (n)=[y 1_dereb (n),y 2_dereb (n),...,y m_dereb(n),...,y M_dereb (n)] T ,m∈[1,M], where

[0078] y m_dereb (n) = s(n-τ m (θ))+e(n-τ m (θ))+v(n) (2)

[0079] The array speech that has been de-reverberated and de-echoed is

[0080] y decho (n)=[y 1_decho (n),y 2_decho (n),...,y m_decho (n),...,y M_decho (n)] T ,m∈[1,M], where

[0081] y m_decho (n) = s(n-τ m (θ))+v(n) (3)

[0082] The noise-free array speech is y enh (n)=[y 1_enh (n),v 2_enh (n),...,y m_enh (n),...,y M_enh (n)] T ,m∈[1,M], where

[0083] y m_enh (n) = s(n-τ m (θ)) (4)

[0084] S1.2. Extract the input features of the network: the speech signal x received by the array element m m (n) Perform short-time Fourier transform to obtain the logarithmic amplitude spectrum X of each frame m (i,k), k is the discrete frequency number, i∈[0,N] is the frame number, N is the total number of frames, the i-th frame is combined with the logarithmic amplitude spectrum of the previous l frames to obtain the network input feature of the i-th frame of the m-th channel After the reference echo speech e(n) is processed in the same way, the logarithmic amplitude spectrum E(i,k) of the reference echo is obtained. m (n) and e(n) are divided into frames with a frame length of 256 and a frame shift of 192, and are windowed with a Hamming window function before short-time Fourier transform. The discrete frequency sequence is k∈[0,128], and the number of frames in the logarithmic amplitude spectrum is l=4.

[0085] S1.3, construct the labels of each task of the mth channel:m_dereb (n), y m_decho (n), y m_enh Perform short-time Fourier transform on the (n) and s(n) speech to obtain the amplitude spectrum of each frame And S(i,k), k is the discrete frequency number, i∈[0,N] is the frame number, and N is the total number of frames. The target ideal ratio mask (Ideal Ratio Mask, IRM) is used as the label of each task. IRM is calculated using the following formula,

[0086]

[0087] Where T m (i, k) is the expected amplitude spectrum of the target speech, and the adjustable scale factor β = 0.5, M m (i,k)=exp(X m (i,k)) is the amplitude spectrum of the noisy speech, and the above obtained and Replace T in formula (5) m (i,k), get the label of the de-reverberated Tags with reverb and echo removed and denoised labels In addition, S(i,k) is logarithmically transformed to obtain the logarithmic amplitude spectrum label S of the clean speech lms .

[0088] S2. Input the features obtained from the preprocessing of each channel into the multi-task network module. In this example, four channel branch networks are used, and the parameters of each branch network are independent. After the features of each channel pass through the dereverberation sub-network, echo cancellation sub-network, and noise reduction sub-network corresponding to each channel, they are fused in the channel fusion sub-network. From the output ends of the above sub-networks, the dereverberated target output, the dereverberated and echo-free target output, the noise-free target output, and the fused clean speech target output of each channel are obtained respectively. The process is as follows:

[0089] S2.1, the reverberation sub-network uses a two-layer unidirectional LSTM network with an input dimension of 129 and a hidden layer width of 512. The activation function of the network output layer uses the Softmax function. Input the dereverberation subnetwork of the mth channel and obtain the IRM output of the desired dereverberation-free speech

[0090] S2.2, the echo cancellation sub-network uses a two-layer bidirectional LSTM network with an input dimension of 258 and a hidden layer width of 512. The activation function of the network output layer uses the Softmax function. and E(i,k) are input into the echo cancellation subnetwork of the mth channel to obtain the IRM output of the desired speech with removed reverberation and echo.

[0091] S2.3, the denoising sub-network uses a two-layer unidirectional LSTM network with an input dimension of 129 and a hidden layer width of 512. The denoising sub-network of the mth channel contains two output layers. One output layer uses the ReLu activation function to output the logarithmic amplitude spectrum of the denoised speech, and the other output layer uses the softmax activation function to output the IRM of the denoised speech. Input the denoising subnetwork and get the logarithmic amplitude spectrum output and IRM output

[0092] S2.4. Constructing the Input of the Channel Fusion Subnetwork

[0093] in

[0094]

[0095] A 3-layer convolutional layer is used as the channel fusion subnetwork, where the convolution kernel size of each layer is 2×9, 2×19, and 2×129, and the input and output dimensions of each layer are (1,16), (16,16), and (16,1). For the input X fuse_in Its dimension can be expressed as X fuse_in ∈R 2×4×129 , convolve the matrix along the second dimension, i.e., the channel direction, to obtain the monophonic logarithmic amplitude spectrum after the channel fusion subnetwork enhancement

[0096] S3. Input the network output and labels of each task into the multi-task loss statistics module, calculate the loss value of the echo cancellation sub-network, the loss value of the reverberation sub-network, the loss value of the noise reduction sub-network, and the loss value of the channel fusion sub-network in each channel, and calculate the total network loss based on the above loss values; the process is as follows:

[0097] S3.1, the output of the dereverberation sub-network and IRM tags for dedeverberated speech The total loss value L of each channel dereverberation sub-network is calculated by the following formula dereb ;

[0098]

[0099] S3.2, the output of the echo cancellation sub-network and tags for voices with de-reverberation and echo removed The echo cancellation sub-network loss value L of each channel is calculated by the following formula decho;

[0100]

[0101] S3.3: Output the denoising sub-network and noise-free labels The loss value L of each channel noise reduction sub-network is calculated by the following formula: enh ;

[0102]

[0103] S3.4. Output of the channel fusion sub-network and the logarithmic magnitude spectrum S of the clean speech lms The loss L of the channel fusion sub-network is calculated by the following formula fuse ;

[0104]

[0105] S3.5, the loss value L of each task module dereb 、L decho 、L enh and L fuse The total loss of the network is calculated as follows:

[0106] loss = L dereb +L decho +L enh +L fuse (11)

[0107] S4. Calculate the gradient of the total loss obtained in S3 and use the Adam optimizer to calculate and update the network weights.

[0108] S5, the training batch size is 1000, the learning rate is 0.001, the total number of training rounds is 200, and steps S2, S3, and S4 are repeated in each round until the total loss of the network is lower than the preset threshold, and the network parameters at this time are saved;

[0109] S6. Preprocess the test speech, calculate the input features of each channel, input the speech features to be enhanced into the multi-task network saved in step S5 to obtain enhanced speech features, and input the enhanced speech features into the speech reconstruction module to obtain enhanced speech. The specific reconstruction process is as follows;

[0110] The logarithmic magnitude spectrum of the channel fusion sub-network output Convert to linear magnitude spectrum Take the speech phase spectrum e of the first channel jθ and Multiplication to obtain the complex representation of the enhanced speech Will Perform inverse short-time Fourier transform and overlap-add to obtain the time domain waveform of the enhanced speech.

[0111] Example 3

[0112] Based on the above embodiment, a microphone array speech enhancement system based on a multi-task network is disclosed. This embodiment further discloses a microphone array speech enhancement method based on a multi-task network. The method uses the following steps to complete training and testing. The training and testing process is as follows: Figure 3 As shown:

[0113] S1. Construct an array speech training set and preprocess the speech to obtain the input features of each channel and the labels of the dereverberation task, echo cancellation task, noise reduction task, and fusion task. The process is as follows:

[0114] S1.1. Constructing a noisy array speech and corresponding dereverberated array speech, dereverberated and echo-free array speech, and noise-free array speech:

[0115] The noisy array speech is x(n)=[x1(n),x2(n),...,x m (n),...,x M (n)] T ,m∈[1,M], where M=4 is the total number of array elements. The generation of noisy speech is as follows Figure 4 As shown, x m (n) is the noisy speech signal received by the mth array element,

[0116]

[0117] s′(n) is the source speech component including reverberation, d(n) is the echo component including reverberation, v(n) is the ambient noise, s(n) is the source speech, e(n) is the reference echo speech, τ m (θ) is the time delay relative to the reference element when array element m receives the signal generated by the sound source with a direction angle of θ. * represents the convolution operation. The signal-to-echo ratio ser∈[-10,10] is randomly distributed, and the signal-to-noise ratio snr∈[-5,15] is randomly distributed. h(n) is the multipath propagation channel parameter in the room, which is determined by the reverberation time, room size, sound source, and array position. In this embodiment, the reverberation time is randomly selected from 0.2s / 0.3s / 0.4s / 0.5s / 0.6s. The room size is randomly selected from 5 / 7 / 9 / 11 / 13m in length, 4 / 6 / 8 / 10m in width, and 3m in height. Five array positions are randomly selected in the room.

[0118] The dedeverberated array speech is

[0119] y dereb (n)=[y 1_dereb(n),y 2_dereb (n),...,y m_dereb (n),...,y M_dereb (n)] T ,m∈[1,M], where

[0120] y m_dereb (n) = s(n-τ m (θ))+e(n-τ m (θ))+v(n) (2)

[0121] The array speech that has been de-reverberated and de-echoed is

[0122] y decho (n)=[y 1_decho (n),y 2_decho (n),...,y m_decho (n),...,y M_decho (n)] T ,m∈[1,M], where

[0123] y m_decho (n) = s(n-τ m (θ))+v(n) (3)

[0124] The noise-free array speech is y enh (n)=[y 1_enh (n),v 2_enh (n),...,y m_enh (n),...,y M_enh (n)] T ,m∈[1,M], where

[0125] y m_enh (n) = s(n-τ m (θ)) (4)

[0126] S1.2. Extract the input features of the network: the speech signal x received by the array element m m (n) Perform short-time Fourier transform to obtain the logarithmic amplitude spectrum X of each frame m (i,k), k is the discrete frequency number, i∈[0,N] is the frame number, N is the total number of frames, the i-th frame is combined with the logarithmic amplitude spectrum of the previous l frames to obtain the network input feature of the i-th frame of the m-th channel After the reference echo speech e(n) is processed in the same way, the logarithmic amplitude spectrum E(i,k) of the reference echo is obtained. m (n) and e(n) are divided into frames with a frame length of 256 and a frame shift of 192, and are windowed with a Hamming window function before short-time Fourier transform. The discrete frequency sequence is k∈[0,128], and the number of frames in the logarithmic amplitude spectrum is l=4.

[0127] S1.3, construct the labels of each task of the mth channel: m_dereb (n), y m_decho (n), y m_enh Perform short-time Fourier transform on the (n) and s(n) speech to obtain the amplitude spectrum of each frame And S(i,k), k is the discrete frequency number, i∈[0,N] is the frame number, and N is the total number of frames. The target ideal ratio mask (Ideal Ratio Mask, IRM) is used as the label of each task. IRM is calculated using the following formula,

[0128]

[0129] Where T m (i, k) is the expected amplitude spectrum of the target speech, and the adjustable scale factor β = 0.5, M m (i,k)=exp(X m (i,k)) is the amplitude spectrum of the noisy speech, and the above obtained and Replace T in formula (5) m (i,k), get the label of the de-reverberated Tags with reverb and echo removed and denoised labels In addition, S(i,k) is logarithmically transformed to obtain the logarithmic amplitude spectrum label S of the clean speech lms .

[0130] S2. Input the features obtained from the preprocessing of each channel into the multi-task network module. In this example, a channel branch network is used, and the four channels share the branch network parameters. The features of each channel are serially input into the branch network, and then pass through the dereverberation sub-network, echo cancellation sub-network, and noise reduction sub-network in sequence. The serial output is fused in the channel fusion sub-network. The dereverberation target output, dereverberation and echo removal target output, noise-free target output, and fused clean speech target output of each channel are obtained from the output ends of the above sub-networks respectively. The process is as follows:

[0131] S2.1, the reverberation sub-network uses a two-layer unidirectional LSTM network with an input dimension of 129 and a hidden layer width of 512. The activation function of the network output layer uses the Softmax function. Input the dereverberation subnetwork of the mth channel and obtain the IRM output of the desired dereverberation-free speech

[0132] S2.2, the echo cancellation sub-network uses a two-layer bidirectional LSTM network with an input dimension of 258 and a hidden layer width of 512. The activation function of the network output layer uses the Softmax function. and E(i,k) are input into the echo cancellation sub-network to obtain the IRM output of the desired speech with reverberation and echo removed

[0133] S2.3, the denoising sub-network uses a two-layer unidirectional LSTM network with an input dimension of 129 and a hidden layer width of 512. The denoising sub-network contains two output layers. One output layer uses the ReLu activation function to output the logarithmic amplitude spectrum of the denoised speech, and the other output layer uses the softmax activation function to output the IRM of the denoised speech. Input the denoising subnetwork and get the logarithmic amplitude spectrum output and IRM output

[0134] S2.4. Constructing the Input of the Channel Fusion Subnetwork in,

[0135] A 3-layer convolutional layer is used as the channel fusion subnetwork, where the convolution kernel size of each layer is 2×9, 2×19, and 2×129, and the input and output dimensions of each layer are (1,16), (16,16), and (16,1). For the input X fuse_in Its dimension can be expressed as X fuse_in ∈R 2×4×129 , convolve the matrix along the second dimension, i.e., the channel direction, to obtain the monophonic logarithmic amplitude spectrum after the channel fusion subnetwork enhancement

[0136] S3. Input the network output and labels of each task into the multi-task loss statistics module, calculate the loss value of the echo cancellation sub-network, the loss value of the reverberation sub-network, the loss value of the noise reduction sub-network, and the loss value of the channel fusion sub-network in each channel, and calculate the total network loss based on the above loss values; the process is as follows:

[0137] S3.1, the output of the dereverberation sub-network and IRM tags for dedeverberated speech The total loss value L of each channel dereverberation sub-network is calculated by the following formula dereb ;

[0138]

[0139] S3.2, the output of the echo cancellation sub-network and tags for voices with de-reverberation and echo removed The echo cancellation sub-network loss value L of each channel is calculated by the following formula decho ;

[0140]

[0141] S3.3: Output the denoising sub-network and noise-free labels The loss value L of each channel noise reduction sub-network is calculated by the following formula: enh ;

[0142]

[0143] S3.4. Output of the channel fusion sub-network and the logarithmic magnitude spectrum S of the clean speech lms The loss L of the channel fusion sub-network is calculated by the following formula fuse ;

[0144]

[0145] S3.5, the loss value L of each task module dereb 、L decho 、L enh and L fuse The total loss of the network is calculated as follows:

[0146] loss = L dereb +L decho +L enh +L fuse (11)

[0147] S4. Calculate the gradient of the total loss obtained in S3 and use the Adam optimizer to calculate and update the network weights.

[0148] S5. The training batch size is 1000, the learning rate is 0.001, and the total number of training rounds is 200. Steps S2, S3, and S4 are repeated in each round until the total loss of the network is lower than the preset threshold. The one-channel branch network is expanded to a four-channel branch network. The four channel branch networks share the same parameters, and the network parameters at this time are saved.

[0149] S6. Preprocess the test speech, calculate the input features of each channel, input the speech features to be enhanced into the multi-task network saved in step S5 to obtain enhanced speech features, and input the enhanced speech features into the speech reconstruction module to obtain enhanced speech. The specific reconstruction process is as follows;

[0150] The logarithmic magnitude spectrum of the channel fusion sub-network output Convert to linear magnitude spectrum Take the speech phase spectrum e of the first channeljθ and Multiplication to obtain the complex representation of the enhanced speech Will Perform inverse short-time Fourier transform and overlap-add to obtain the time domain waveform of the enhanced speech.

[0151] In order to verify the technical effect of the present invention, the above-mentioned Examples 2 and 3 are compared with the existing baseline system (see: Westhausen NL, Meyer B T. Acoustic echo cancellation with the dual-signal transformation LSTM network [C] / / ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021: 7138-7142.), and the evaluation index uses the speech quality perception evaluation PESQ. The source speech and reference echo speech of the training set are selected from the TIMIT training set, and the additive noise is Babble noise. The source speech and reference echo of the test set are selected from the TIMIT test set. The tests were completed at signal-to-noise ratios of -3dB, 2dB, 7dB, and 12dB and signal-to-echo ratios of -2dB, 3dB, and 8dB respectively.

[0152] The baseline system test results were obtained using the pre-trained model provided in the original paper. The comparative results are shown in Tables 1 and 2. It can be seen that under different signal-to-noise ratios, the implementation of this patent achieves an average improvement of 0.6 in PESQ over noisy speech and an average improvement of 0.2 over the baseline system PESQ. Under different echo signal ratios, the implementation of this patent achieves an average improvement of 25% to 50% in PESQ over noisy speech and an average improvement of approximately 11% over the baseline system PESQ.

[0153] Table 1. Comparison of PESQ under different signal-to-noise ratio conditions

[0154] -3dB 2dB 7dB 12dB Noisy speech 1.358 1.445 1.517 1.561 Baseline system 1.529 1.828 1.986 2.058 This embodiment 2 1.868 2.013 2.122 2.186 This embodiment 3 1.853 1.991 2.104 2.170

[0155] Table 2. PESQ comparison under different signal-echo ratio conditions

[0156] -2dB 3dB 8dB Noisy speech 1.263 1.477 1.671 Baseline system 1.703 1.869 1.979 This embodiment 2 1.862 2.067 2.213 This embodiment 3 1.852 2.048 2.189

[0157] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A microphone array speech enhancement system based on a multi-task network, characterized in that: The microphone array speech enhancement system includes a speech preprocessing module, a multi-task network module, a multi-task loss statistics module, a network weight calculation module and a speech reconstruction module, wherein: The speech preprocessing module is connected to the multi-task network module and the multi-task loss statistics module respectively, and is used to obtain the array speech, the reference echo speech and the target speech of each task as input speech, and preprocess the input speech, wherein the preprocessing includes normalizing the speech signal and extracting the logarithmic amplitude spectrum of each channel speech and the reference echo speech; The multi-task network module is respectively connected to the speech preprocessing module, the multi-task loss statistics module, the network weight calculation module and the speech reconstruction module to complete the dereverberation, echo cancellation and noise reduction tasks of each channel of the array speech, and output the multi-channel speech fusion as the enhanced speech; the multi-task network module includes M parallel channel branch networks and 1 channel fusion sub-network, each of the above channel branch networks includes a dereverberation sub-network, an echo cancellation sub-network and a noise reduction sub-network connected in sequence, where M is the same as the number of array elements of the microphone array in the speech preprocessing module, the dereverberation sub-network and the echo cancellation sub-network in each channel branch network, the multi-task loss statistics module and the The speech preprocessing module is connected to complete the reverberation removal task of the channel; the echo cancellation subnetwork in each channel branch network is connected to the dereverberation subnetwork, the noise reduction subnetwork, the speech preprocessing module and the multi-task loss statistics module, and uses the reference echo signal to eliminate the echo component of the channel; the noise reduction subnetwork in each channel branch network is connected to the echo cancellation subnetwork, the channel fusion subnetwork and the multi-task loss statistics module to eliminate the noise component in the channel; the channel fusion subnetwork is respectively connected to the noise reduction subnetworks in the M channel branch networks, the multi-task loss statistics module and the network weight calculation module, and uses the spatial information contained in each channel of the array speech for secondary enhancement; The multi-task loss statistics module is connected to the multi-task network module and the network weight calculation module respectively, calculates the loss value of each task in the multi-task network module, and counts the total loss of the network; The network weight calculation module is connected to the multi-task network module and the multi-task loss statistics module respectively, calculates the gradient according to the total loss of the network, backpropagates the gradient, and calculates the weight of the updated network; The speech reconstruction module is connected to the multi-task network module to complete the mapping from frequency domain features to time domain speech, and obtain enhanced clean speech.

2. A microphone array speech enhancement system based on a multi-task network according to claim 1, characterized in that: The multi-task loss statistics module includes a dereverberation loss statistics module, an echo cancellation loss statistics module, a noise reduction loss statistics module, a channel fusion loss statistics module and a total loss statistics module, wherein the dereverberation loss statistics module is respectively connected to the above-mentioned speech preprocessing module, the dereverberation subnetwork of each channel and the total loss statistics module, and uses the output of the dereverberation subnetwork and the dereverberation label to calculate the loss value of the dereverberation subnetwork of each channel; the echo cancellation loss statistics module is respectively connected to the above-mentioned speech preprocessing module, the echo cancellation subnetwork of each channel and the total loss statistics module, and uses the output of the echo cancellation subnetwork and the dereverberation and echo label to calculate the loss value of the echo cancellation subnetwork of each channel; the noise reduction loss statistics module is respectively connected to the above-mentioned speech preprocessing module, the echo cancellation subnetwork of each channel and the total loss statistics module, and uses the output of the echo cancellation subnetwork and the dereverberation and echo label to calculate the loss value of the echo cancellation subnetwork of each channel; The calculation module is respectively connected to the above-mentioned speech preprocessing module, the denoising subnetwork of each channel and the total loss statistics module, and uses the output of the denoising subnetwork and the noise-free label to calculate the loss value of each channel denoising subnetwork; the channel fusion loss statistics module is respectively connected to the above-mentioned speech preprocessing module, the above-mentioned channel fusion subnetwork and the total loss statistics module, and uses the output of the channel fusion subnetwork and the clean speech label to calculate the loss value of the channel fusion subnetwork; the total loss statistics module is respectively connected to the dereverberation loss statistics module, the echo cancellation loss statistics module, the denoising loss statistics module, the channel fusion loss statistics module and the above-mentioned network weight calculation module, receives the loss value of each task loss statistics module, and calculates the total loss value of the network.

3. A speech enhancement method for a microphone array speech enhancement system based on a multi-task network according to claim 1 or 2, characterized in that: The speech enhancement method comprises the following steps: S1. Construct an array speech training set and preprocess the speech to obtain the input features of each channel and the labels of the dereverberation task, echo cancellation task, noise reduction task, and fusion task; S2. Input the features obtained from the preprocessing of each channel into the multi-task network module. After passing through the dereverberation sub-network, echo cancellation sub-network, and noise reduction sub-network corresponding to each channel, they are fused in the channel fusion sub-network. From the output ends of the above sub-networks, the dereverberated target output, the dereverberated and echo-free target output, the noise-free target output, and the fused clean speech target output of each channel are obtained respectively. S3. Input the network output and labels of each task into the multi-task loss statistics module, calculate the loss value of the echo cancellation sub-network, the loss value of the reverberation sub-network, the loss value of the noise reduction sub-network, and the loss value of the channel fusion sub-network in each channel, and calculate the total network loss based on the above loss values; S4. Calculate the gradient of the total network loss and use backpropagation to update the network weights. S5. Repeat steps S2, S3, and S4 until the total loss of the network is lower than a preset threshold, and save the network parameters at this time; S6. Preprocess the test speech, calculate the input features of each channel, input the speech features to be enhanced into the multi-task network saved in step S5 to obtain enhanced speech features, and input the enhanced speech features into the speech reconstruction module to obtain enhanced speech.

4. The method for speech enhancement using a microphone array based on a multi-task network according to claim 3, wherein: The construction process of the array speech training set in step S1 is as follows: S1.

1. Constructing a noisy array speech and corresponding dereverberated array speech, dereverberated and echo-free array speech, and noise-free array speech: The noisy array speech is x(n)=[x1(n),x2(n),...,x m (n),...,x M (n)] T ,m∈[1,M], where x m (n) is the noisy speech signal received by the mth array element, M is the number of array elements in the microphone array, s′(n) is the source speech component containing reverberation, d(n) is the echo component containing reverberation, v(n) is the ambient noise, s(n) is the source speech, e(n) is the reference echo speech, τ m (θ) is the time delay of the signal generated by the sound source with a direction angle of θ when the array element m receives it relative to the reference array element, * represents the convolution operation, h(n) is the multipath propagation channel parameter in the room, ser is the randomly distributed signal-to-echo ratio, and snr is the randomly distributed signal-to-noise ratio; The dedeverberated array speech is y dereb (n) = [y 1_dereb (n), y 2_dereb (n),..., y m_dereb (n),..., y M_dereb (n)] T , m ∈ [1, M], where y m_dereb (n)=s(n-τ m (θ))+e(n-τ m (θ))+v(n) (2) The array speech that has been de-reverberated and de-echoed is y decho (n) = [y 1_decho (n), y 2_decho (n),..., y m_decho (n),..., y M_decho (n)] T , m ∈ [1, M], where y m_decho (n)=s(n-τ m (θ))+v(n) (3) The noise-free array speech is y enh (n) = [y 1_enh (n), y 2_enh (n),..., y m_enh (n),..., y M_enh (n)] T , m ∈ [1, M], where y m_enh (n)=s(n-τ m (i)) (4) Among them, y m_dereb (n) is the de-reverberated speech of the m-th array element, y m_enh (n) is the de-reverberated and de-echoed speech of the m-th array element, y m_enh (n) is the noise-free speech of the m-th array element; S1.

2. Extract the input features of the network: the speech signal x received by the array element m m (n) Perform short-time Fourier transform to obtain the logarithmic amplitude spectrum X of each frame m (i,k), k is the discrete frequency number, i∈[0,N] is the frame number, N is the total number of frames, the i-th frame is combined with the logarithmic amplitude spectrum of the previous l frames to obtain the network input feature of the i-th frame of the m-th channel The reference echo speech e(n) is processed in the same way to obtain the logarithmic amplitude spectrum E(i,k) of the reference echo; S1.3, construct the labels of each task of the mth channel: m_dereb (n), y m_decho (n), y m_enh Perform short-time Fourier transform on the (n) and s(n) speech to obtain the amplitude spectrum of each frame and S(i,k), the target ideal ratio mask IRM is used as the label of each task, and IRM is calculated using the following formula, Among them, T m (i, k) is the expected amplitude spectrum of the target speech, β is the adjustable scale factor, M m (i,k)=exp(X m (i,k)) is the amplitude spectrum of the noisy speech, and the above obtained and Replace T in formula (5) m (i,k), get the label of the de-reverberated Tags with reverb and echo removed and denoised labels In addition, S(i,k) is logarithmically transformed to obtain the logarithmic amplitude spectrum label S of the clean speech lms .

5. The method for speech enhancement using a microphone array based on a multi-task network according to claim 4, wherein: The process of inputting the features into the multi-task network module in step S2 to obtain the target output of each task sub-network is as follows: S2.1、 Input the dereverberation subnetwork of the mth channel and obtain the IRM output of the desired dereverberation-free speech S2.2, and E(i,k) are input into the echo cancellation subnetwork of the mth channel to obtain the IRM output of the desired speech with removed reverberation and echo. S2.3, the noise reduction sub-network of the mth channel contains two output layers, one output layer outputs the logarithmic amplitude spectrum of the noise-reduced speech, and the other output layer outputs the IRM of the noise-reduced speech. Input the denoising subnetwork and get the logarithmic amplitude spectrum output and IRM output S2.

4. Constructing the Input of the Channel Fusion Subnetwork in Use one The convolution layer of the layer is used as the channel fusion sub-network, M is the total number of array elements, d is the width of the convolution kernel, for the input X fuse_in Its dimension can be expressed as X fuse_in ∈R 2×M×k , convolve the matrix along the M-dimensional direction to obtain the monophonic logarithmic amplitude spectrum after the channel fusion subnetwork enhancement 6. The method for speech enhancement using a microphone array based on a multi-task network according to claim 5, wherein: The process of obtaining the total loss value by the multi-task loss statistics module in step S3 is as follows: S3.1, the output of the dereverberation sub-network and IRM tags for dedeverberated speech The total loss value L of each channel dereverberation sub-network is calculated by the following formula dereb ; S3.2, the output of the echo cancellation sub-network and tags for voices with de-reverberation and echo removed The echo cancellation sub-network loss value L of each channel is calculated by the following formula decho ; S3.

3. Output the denoising sub-network and noise-free labels The loss value L of each channel noise reduction sub-network is calculated by the following formula: enh ; S3.

4. Output of the channel fusion sub-network and the logarithmic magnitude spectrum S of the clean speech lms The loss L of the channel fusion sub-network is calculated by the following formula fuse ; S3.5, the loss value L of each task module dereb 、L decho 、L enh and L fuse The total loss of the network is calculated as follows: loss=L dereb +L decho +L enh +L fuse (11)。 7. The method for speech enhancement using a microphone array based on a multi-task network according to claim 5, wherein: In step S6, the enhanced speech features are reconstructed into enhanced speech using the following method: The logarithmic magnitude spectrum of the channel fusion sub-network output Convert to linear magnitude spectrum Then with x m Phase spectrum of (n) Multiplication to obtain the complex representation of the enhanced speech Will Perform inverse short-time Fourier transform and overlap-add to obtain the time domain waveform of the enhanced speech.

Images