A beamforming method based on complex-gated recurrent unit

Abstract

The application discloses a beam forming method based on a complex gating recurrent unit, and comprises the following steps: (1) performing complex masking prediction; the number of predicted complex masks corresponds to the number of speakers; (2) performing element multiplication on the complex mask and a complex mixed audio feature to obtain a speaker complex audio feature, performing matrix multiplication on the speaker complex audio feature and a conjugate transpose of the speaker complex audio feature, performing summation along a time dimension after normalization to obtain a complex power spectrum density matrix; (3) the complex power spectrum density matrix corresponds to the audio of each speaker; (4) constructing a complex gating recurrent unit network, inputting a spliced complex matrix feature into the complex gating recurrent unit network, fully utilizing the characteristics of the complex network, and predicting a weighting coefficient of beam forming; after the weighting coefficient of beam forming and the complex mixed audio feature are subjected to matrix multiplication, more accurate speaker complex audio features can be obtained.

Description

Technical Field

[0001] This invention belongs to the field of speech separation, specifically involving a beamforming method based on complex gated cyclic units. It mainly uses deep learning combined with signal processing knowledge to improve the signal-to-distortion ratio (SDR), objective speech quality assessment (PESQ), short-time objective intelligibility (STOI), and other indicators of multi-channel speaker-separated speech, providing cleaner audio for speech recognition and other modules. Background Technology

[0002] Speech separation aims to isolate the audio of each speaker from mixed audio, serving as a front-end technology to enable machines to recognize the voices of multiple speakers even in complex acoustic environments. In noisy acoustic environments, such as a classic cocktail party setting, people can accurately focus on and understand the voices of certain speakers while ignoring background noise, reverberation, and the voices of other speakers. For machines, speech separation is a prerequisite for achieving natural interaction with humans in complex scenarios. Early research largely focused on auditory scenes and signal processing, such as beamforming and auditory scene analysis, involving complex signal processing algorithms and processes. In recent years, with the development of deep learning, neural network architectures have replaced some traditional signal processing methods, leading to rapid advancements in speech separation technology. In the multi-channel domain, methods combining deep learning with signal processing have been extensively studied. [1] In particular, the application of beamforming in the field of deep learning, such as Minimum Variance Distortionless Response (MVDR), has greatly improved the performance of multi-channel multi-speaker speech separation.

[0003] However, using beamforming in the field of deep learning still requires many mathematical operations. For example, calculating the weighting coefficients for beamforming requires inverting the power spectral density matrix, which often leads to numerical instability. [2] In addition, while MVDR filtering reduces distortion, significant residual interference remains. Generally, MVDR operates at the sentence level, a coarse approach that often ignores inter-frame correlation information, making it less than optimal. Frame-level beamforming, which calculates beamforming weighting coefficients for each frame instead of using a single coefficient to beam the entire speech, achieves frame-by-frame separation and outperforms traditional MVDR methods. Recently, researchers in the multimodal domain have combined audio and video features, using a gated recurrent unit-based beamforming method to improve the performance of target speaker extraction in video. [3 , 4This invention suggests that beamforming methods based on gated cyclic units have broad application prospects and can be applied to multi-channel, multi-speaker speech separation scenarios. However, a drawback is that the input to beamforming based on gated cyclic units is a feature concatenated from real and imaginary parts; this simple concatenation is not conducive to the full utilization of the features. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and address the problems of numerical instability, severe interference residue, and insufficient utilization of complex features in multi-channel multi-speaker speech separation. This invention provides a beamforming method based on complex gated cyclic units, aiming to improve the overall performance of multi-channel multi-speaker speech separation under reverberation conditions by improving the prediction method of beamforming weighting coefficients, and striving to explore its own characteristics to promote the development of the speech separation field.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A beamforming method based on complex gated cyclic units includes the following steps:

[0007] (1) Perform complex masking prediction; the mixed audio is transformed into complex mixed audio features through short-time Fourier transform, and the logarithmic power spectrum is calculated using the real and imaginary parts of the complex mixed audio features. The smallest unit of the complex mixed audio features and the logarithmic power spectrum is the time-frequency unit, also known as a frame; the logarithmic power spectrum is input into a bidirectional long short-time memory network to predict complex masking. [5] Since the mixed audio consists of the audio of several speakers, in order to extract the audio of each speaker, the number of predicted complex masks corresponds to the number of speakers;

[0008] (2) The speaker's complex audio features are obtained by element-wise multiplication of the complex masking and complex mixed audio features. The speaker's complex audio features are then matrix-multiplied with their conjugate transpose, standardized, and summed along the time dimension to obtain the complex power spectral density matrix. Depth filtering is considered during element-wise multiplication. [6] The calculated speaker complex audio features for the current frame are numerically equal to the sum of the complex mixed audio features for the current frame and the products of the complex masking of several frames before and after, thus achieving a many-to-one mapping.

[0009] (3) There are several complex power spectral density matrices, each corresponding to the audio of a speaker. In order to separate the audio of one speaker, the complex power spectral density matrix corresponding to the audio of one speaker is regarded as the target, which is the speech power spectral density matrix. The complex power spectral density matrices corresponding to the audio of all other speakers are summed and regarded as the interference power spectral density matrix. The real parts of the speech power spectral density matrix and the interference power spectral density matrix are concatenated with the real parts and the imaginary parts are concatenated with the imaginary parts to obtain the complex matrix features.

[0010] (4) Construct a complex gated recurrent unit network and input the spliced ​​complex matrix features into the complex gated recurrent unit network. Make full use of the characteristics of the complex network to predict the weighting coefficients of beamforming. After matrix multiplication of the weighting coefficients of beamforming and the complex mixed audio features, more accurate speaker complex audio features can be obtained. Perform inverse Fourier transform on the speaker complex audio features to obtain the separated speaker audio. The beamforming method based on complex gated recurrent units uses the scale-invariant signal-to-noise ratio as the loss function.

[0011] Furthermore, the specific process of depth filtering in step (2) is as follows: For the current frame of complex mixed audio features, take K frames before and after the time dimension and L frames before and after the frequency dimension, for a total of (2K+1)*(2L+1) frames. The complex masking is taken in the same way. After the element-wise product of (2K+1)*(2L+1) frames is completed, the sum is obtained, which is the current frame of the speaker's complex audio features.

[0012] Furthermore, the input and output of the complex gated cyclic unit network are both complex numbers. After the concatenated complex matrix features are input into the complex gated cyclic unit network, the real and imaginary parts of the complex matrix features are fully interacted, and the output prediction results are used as weighting coefficients for beamforming.

[0013] Furthermore, a weighted coefficient is used to separate the audio of a speaker. In order to separate the audio of each speaker, steps (3) and (4) in weight requirement 1 need to be repeated. The number of weighted coefficients for predicting beamforming is equal to the number of speakers.

[0014] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0015] Previous methods for multi-channel, multi-speaker speech separation using deep learning have mostly employed complex masking combined with sentence-level minimum variance distortionless response (MVDR) beamforming, which carries significant risks. This invention not only utilizes deep-filtered complex masking to fully leverage the accuracy of complex power spectral density matrix calculations across frames, but also employs gated recurrent units to directly predict frame-level beamforming weighting coefficients. This avoids complex mathematical calculations, improves numerical stability, and suppresses residual interference from the MVDR method. Furthermore, the use of complex gated recurrent units in network construction, compared to simply using gated recurrent units for real-valued feature processing and alternating prediction of the real and imaginary parts of the complex beamforming weighting coefficients, further enhances performance.

[0016] Furthermore, this invention combines deep learning and signal processing knowledge, providing a new perspective for advancing existing multi-channel speech separation schemes. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the baseline method in the prior art;

[0018] Figure 2 The graph of depth filtering shows that L and K are both 1, indicating that one frame is taken forward and one frame is taken in the time and frequency dimensions.

[0019] Figure 3a This is a schematic diagram of the gated loop unit structure. Figure 3b A schematic diagram of a complex gated loop unit structure;

[0020] Figure 4 This is an overall schematic diagram of the invention, showing the main content of the experiment, which is omitted here. Figure 1 Some specific steps are shown as a whole module;

[0021] Figure 5a and Figure 5b These are the results obtained by decoding the same mixed audio using a conventional baseline method and the method of this invention, respectively, with the shorter audio segment being selected in each case. Detailed Implementation

[0022] The following description and proof of the invention are based on the experimental process and results. Building upon current common methods for multi-channel, multi-speaker speech separation, this invention focuses on improving the enhancement potential of masking prediction and enhancing beamforming accuracy. It applies depth filtering technology to complex masking; replaces the mathematical calculations in minimum variance distortionless response beamforming with complex-gated recurrent neural units, ensuring that all intermediate results rely on the nonlinearity of the neural network, avoiding complex calculations such as large matrix inversions; simultaneously, the complex-gated recurrent neural units maintain the complex characteristics during feature processing, directly predicting the weighting coefficients for frame-level beamforming. The key technical points are divided into the following three parts:

[0023] (1) Complex masking prediction network

[0024] In the field of speech separation, one of the most classic methods is masking-based approaches. This method is highly effective for both single-channel and multi-channel multi-speaker speech separation, as well as for time-domain and time-frequency domain solutions. Essentially, this method predicts a filter that extracts the target audio features from a complex mixture of audio features. For single-channel speech separation, after obtaining the mask, the separated audio can be obtained by directly multiplying the bitwise pairs.

[0025]

[0026] Where t and f represent time and frequency. The audio is the separated speaker audio. M(t,f) usually refers to masking, ⊙ refers to element-wise multiplication, and X(t,f) is the complex mixed audio feature. For complex masking, the numerical type of the mask is the complex domain. After performing a short-time Fourier transform on the multi-channel mixed audio, the logarithmic power spectrum (LPS) is further calculated as the input to the complex masking prediction network. The formula for calculating the logarithmic power spectrum is as follows:

[0027] LPS = Log(Sqrt(real) 2 +imag 2 (2)

[0028] Here, `real` and `imag` represent the real and imaginary parts of the complex mixed audio feature after the short-time Fourier transform. This feature is applicable to both single-channel and multi-channel audio computation methods. The multi-channel audio feature has an additional dimension, the size of which is equal to the number of array microphones. To conform to the network input format, the channel dimension and batch dimension are superimposed, making the dimension of the multi-channel logarithmic power spectrum feature consistent with that of the single-channel feature, allowing the network to directly predict the complex masking.

[0029] Details of the entire complex masking prediction network and Figure 1 The demonstrated structures are largely consistent. A Bidirectional Long Short-Term Memory (BLSTM) network is used as the basic framework, with three BLSTM layers, a fully connected layer, and a ReLU activation function layer sequentially connected. The fully connected layer maps the frequency dimension of the complex mask to 257*2 dimensions, where 257 represents the frequency dimension of the original complex mixed audio features, and 2 signifies that the mask uses complex numbers. By splitting and combining the dimensions into a complex tensor, the complex mask is obtained. In contrast, the baseline method predicts a real mask, and the fully connected layer maps the frequency dimension of the real mask to 257 dimensions. The complex mask is multiplied by the input complex mixed audio features to extract the speaker's complex audio features for further calculation of the complex power spectral density matrix.

[0030] Complex masking involves predicting the real and imaginary parts separately. Multiplying the mask by the complex-domain audio directly yields the speaker's complex audio features, each reflecting the numerical characteristics of its respective part. However, multiplying a mask from the real domain with the complex audio essentially involves data broadcasting. Multiplying the real mask by the real and imaginary parts of the mixed complex audio features separately can easily lead to mismatch issues. Therefore, complex masking has become increasingly popular in recent years, and its performance is slightly better than real masking.

[0031] Further depth filtering is used for many-to-one mapping. The mapping method is as follows: Figure 2As shown, the fully connected layer of the complex masking prediction network maps the frequency dimension of the complex mask to 257*2*(2L+1)*(2K+1), and decomposes it into (257, 2, 2L+1, 2K+1). This is feasible because the complex mask has no physical meaning; it only has a filtering effect when multiplied by the corresponding complex mixed audio features. Therefore, for the depth filtering method to work, it is only necessary to extend the complex mixed audio features by two dimensions, 2L+1 and 2K+1, where L and K represent the forward and backward extensions of the time and frequency dimensions of the current frame, respectively. When both L and K are 1, it is... Figure 2 The mapping method is shown. After calculating the product of the pairs of elements, the summation is performed along the dimensions of (2L+1) and (2K+1) to obtain the current frame of the speaker's complex audio features.

[0032] (2) Calculation of power spectral density matrix

[0033] In the field of multi-channel speech separation, simple masking filtering, similar to that used in single-channel methods, results in severe distortion. Beamforming is essential to improve sound quality. The calculation of the power spectral density matrix is ​​for the derivation of the beamforming weighting coefficients. In the baseline method, MVDR beamforming is used as the beamforming algorithm. The following is a brief introduction to the process of the traditional MVDR beamforming algorithm.

[0034] MVDR beamforming is an adaptive beamforming algorithm based on the maximum signal-to-interference-plus-noise ratio (SINR) criterion. This algorithm adaptively minimizes the power of the microphone array output in the desired direction while maximizing the SINR, thus suppressing noise and interference. MVDR has wide applications in the military field, often used in underwater acoustic wireless communication technology, enabling communication between surface ships and submarines. The MVDR algorithm employs a sampling matrix inversion algorithm commonly used in adaptive beamforming, which has a fast convergence speed under certain SINR conditions. The multi-channel audio signal can be expressed in matrix form as follows:

[0035] x(n)=αs(n-τ)+v(n) (3)

[0036] Where x, α, s, τ, and v represent the audio received by the array, the steering vector, the audio emitted by the sound source, the propagation delay, and the interference, respectively, and n is the time sampling point. * indicates the convolution operation. Performing a Fourier transform on the above equation converts the time-domain signal to the time-frequency domain:

[0037] X(t,f)=αS(t,f)+V(t,f) (4)

[0038] α is the array's steering vector, reflecting the microphone array's sensitivity to direction; this vector has a specific value:

[0039]

[0040] Where ω represents the angular frequency, M represents the number of microphones, e is the exponent, and j represents the imaginary part. The meanings of the other elements are as described above. αS(t,f) is expressed using a single parameter. The beamformed signal can then be represented in the time-frequency domain as:

[0041]

[0042] Among them W i (f) represents the weighting coefficient of the i-th microphone at frequency f, which is W when written as a matrix. H represents the conjugate transpose, and Y(t,f) is the enhanced single-channel audio. The key to MVDR beamforming is calculating the power spectral density matrix of the signal. The power spectral density matrix of the output signal is:

[0043] φ=E(YY H ) = W H E(XX H )W (7)

[0044] Where Y and X represent the enhanced signal and the original output signal of the microphone array, respectively. W represents the weighting coefficients, φ represents the covariance matrix, and E usually refers to the expected value.

[0045] For a signal S(t,f) in the frequency domain, the power spectral density matrix of the signal S(t,f) is generally calculated using the following formula, assuming that the noise and audio are uncorrelated:

[0046] φ SS =E[S(t,f)S(t,f)] (8)

[0047] Therefore, for the received audio, the audio emitted by the sound source, and the interference, the following conditions must be met:

[0048] φ XX =φ SS +φ VV (9)

[0049] φ XX φ SS and φ VV These correspond to the power spectral density matrices of the received audio, the audio emitted by the sound source, and the interference, respectively. The power spectral density matrices of the input and output audio determine the signal-to-interference-plus-noise ratio (SNR). The MVDR method aims to minimize the power of the output audio to obtain the optimal weighting coefficient prediction. The output power spectrum is determined by the above formula. During the optimization process, it is crucial to avoid making the weighting coefficients zero, i.e., to ensure that the signal is not distorted in the desired direction.

[0050] W H α=1 (10)

[0051] Solve the optimal problem under these constraints, that is, find the minimum weighting coefficient in Equation 7 under the above conditions:

[0052] min W H φ XX W st W H α=1 (11)

[0053] Thus, by solving the constrained optimization problem, the adaptive weighting coefficients for MVDR beamforming can be obtained as follows:

[0054]

[0055] This coefficient can vary according to the power spectral density matrix of the interference, thus the MVDR algorithm can adaptively maximize the SINR of the microphone array output in the desired direction, achieving the best effect. When the number of array elements in the microphone array decreases, or in a high signal-to-noise ratio environment, the desired signal and interference often exhibit significant coherence, which greatly affects the performance of the MVDR algorithm. As can be seen from Equation 12, beamforming requires inverting a large matrix, which is inherently unstable. This is why this invention uses a neural network instead of mathematical calculations.

[0056] In deep learning-based joint MVDR methods, the formula for calculating the power spectral density matrix is ​​similar to that described above. Taking real-number masking as an example, combined with complex-number mixed audio features, the formula for calculating the sentence-level power spectral density matrix is ​​as follows:

[0057]

[0058] The denominator uses real-number masking as a standardization method, T is the total length of the time dimension, and the remaining notation is defined as described above. This refers to the speaker's complex audio features obtained by multiplying masked and complex mixed audio features. Using a sentence-level complex power spectral density matrix, the calculated beamforming weighting coefficients are also sentence-level, which is rather crude. Furthermore, the lack of a temporal dimension makes the features unsuitable for input into complex gated recurrent units, hindering the learning of temporal features. Therefore, calculating a frame-level power spectral density matrix based on the above formula is necessary. Compared to Equation 13, this calculation formula does not require summation in the temporal dimension:

[0059]

[0060] The power spectral density matrix of the speech and the power spectral density matrix of the interference are calculated separately. The feature input to the complex gated recurrent unit is the complex matrix feature obtained by concatenating the two.

[0061] (3) Complex gated cyclic unit beamforming

[0062] The entire training process of the baseline method is as follows: Figure 1 As shown. Figure 3b The structure of a complex gated cyclic unit (GRU) is shown, mainly composed of two GRUs: a real GRU and an imaginary GRU. The structure of a single GRU is as follows: Figure 3a As shown, the Gated Recurrent Unit (GRU) is a variant of the Long Short-Term Memory (LSTM) network. It is computationally less complex than LSTM and delivers impressive results. It can address the long dependency problem in recurrent neural networks. Unlike LSTM, which uses three gates (input, forget, and output) to control input, memory, and output, GRU only requires updating and resetting two gates to retain important features. Furthermore, it has fewer parameters than LSTM and is faster to train.

[0063] For complex networks, experiments have shown that they perform slightly better than real network structures in the enhancement domain. Therefore, this invention constructs a complex gated cyclic unit network. Two gated cyclic units are used in the entire complex network, and the entire beamforming weighting coefficient calculation process can be summarized by the following formulas:

[0064] R r =GRU r (Real)

[0065] I r =GRU r (Imag)

[0066] R i =GRU i (Real)

[0067] I i =GRU i (Imag)

[0068] Out real =R r -I i

[0069] Out imag =I r +R i (15)

[0070] Where Real and Immag are the real and imaginary parts of the characteristics of a complex matrix, GRU r For real part GRU, GRU i For the imaginary part GRU, and R r I r R i and I i All four are intermediate products of the network; when combined, they ultimately yield the predicted real part Out. real and the imaginary part Outimag The beamforming weighting coefficients are used to form the beamforming weighting coefficients. The activation function chosen is PReLU, an improved ReLU function that is more flexible in handling negative gradients. Matrix multiplication is performed using the beamforming weighting coefficients and complex mixed audio features to obtain more accurate separated single-channel audio features. An inverse Fourier transform is then performed to complete one beamforming process. The first channel is selected as the reference channel during beamforming; therefore, the first channel of the label is also selected when calculating the scale-invariant signal-to-noise ratio (SISNR). The entire network uses a serial pipeline structure, with the complex masking prediction network and the complex gated recurrent unit beamforming trained simultaneously. The final framework diagram of this invention is shown below. Figure 4 As shown.

[0071] The data used in this invention comprises 28,000 audio samples, with a training set:validation set:test set ratio of 20:5:3. The mixed audio was generated using code simulation. Clean, single-channel audio was provided during simulation, with the sound source from the Wall Street Journal dataset wsj0. First, a single-channel mixed audio sample, wsj0-2mix, was generated. This step does not require spatial or array information, and its simulation process is relatively simple. Then, a picture-based room impulse response generator (RIR) was used to spatialize the wsj0-2mix dataset. This method sets the coordinates of the microphone array, the coordinates of the two speakers, and the location coordinates of the reverberation generated by the picture method within a virtual room. The final dataset includes a clean multi-channel dataset and a reverberated multi-channel dataset, both being eight-channel mixed audio samples from two speakers at a sampling rate of 8kHz. To highlight the complexity of the experimental environment, this invention primarily uses the reverberated version of the data. Multi-channel two-speaker speech separation was achieved under reverberation conditions. The presence of reverberation affects separation performance; research shows that reverberation-free scenarios perform approximately 20% better than reverberated scenarios. Clean, multi-channel audio data was primarily used for preliminary experiments to observe the impact of reverberation on multi-channel speech separation. It's worth noting that dereverberation methods such as weighted prediction errors (WPE) were not introduced in this experiment. In addition, data cleaning was performed to ensure that all audio files were at least one second long.

[0072] In the experiment, since the validation set and training set have the same speaker, the validation set is also called the Closed Condition (CC), indicating that the speaker's method is known during decoding. The speaker in the test set is different from that in the training set, and is therefore called the Open Condition (OC), meaning that the speaker is unknown during testing. The performance of the invention under the closed and open conditions is tested, with the decoding result of the open condition set serving as the primary observation.

[0073] The training parameters for the baseline method are shown in Table 1. Two NVIDIA 3090 GPUs were used for training and decoding, with each GPU having 24GB of VRAM. The baseline method is based on ESPNet, an open-source speech tool that integrates cutting-edge technologies in the field, including multi-channel speech separation schemes, namely a framework combining masking and MVDR beamforming. The final method of this invention is also integrated into ESPNet, and the design of the new method is achieved through improvements to the lowest-level code of this tool.

[0074] Table 1. Structure and training parameters of the baseline method

[0075]

[0076] The improved baseline is the basis of this invention. Due to the increased complexity of the network, the required GPU memory also increases dramatically. The parameters of this invention are shown in Table 2. The training parameters remain consistent with those of the baseline method, but the dimensions of the fully connected layers in the complex masking prediction network are significantly different from those of the baseline method, mainly performing complex number and mapping operations. The complex gated recurrent unit network consists of real part GRU units and imaginary part GRU units, with each GRU node set to 300. Since the complex gated recurrent unit is time-sensitive, the features of the complex matrix need to undergo dimensional transformation, superimposing the batch dimension and frequency dimension while keeping the time dimension unchanged. Only after superimposing all remaining dimensions can the data be input into the complex gated recurrent unit, where nonlinear operations are performed on the new dimension created by superimposing all remaining dimensions.

[0077] Table 2 Network parameters and training parameters of the method of the present invention

[0078]

[0079]

[0080] First, there is a confirmatory experiment. Figure 1 The performance of separating clean and reverberant audio datasets was tested using the baseline method. Training and decoding were performed on different datasets, and preliminary experiments verified that, with consistent methods and parameters, separating reverberant mixed audio is significantly more difficult than separating clean audio. As shown in Table 3, in some key metrics such as PESQ, STOI, and SDR, the performance of reverberant data is far inferior to that of clean data. Considering real-life scenarios, reverberant data has greater research value and room for improvement, as truly clean speech is almost nonexistent in real-world environments.

[0081] Table 3 Comparison of reverberation data and clean data under the baseline method.

[0082]

[0083] Based on the baseline method, this invention was trained using reverberant data. Experimental results are shown in Table 4. The results demonstrate that this invention significantly improves the separation performance of reverberant mixed audio. All three evaluation metrics are improved compared to the baseline, regardless of whether the set is closed or open. Taking the open set as an example, PESQ, STOI, and SDR are improved by 32.08%, 12.20%, and 31.54% respectively, showing a very significant improvement. To further observe the quality of the separated audio, spectrograms of the same mixed audio separated under both the baseline and this invention's method were selected. The original mixed audio consists of a long audio and a short audio. The mixed audio sometimes ends with only one person speaking; therefore, the separated short audio was selected for observation. Theoretically, the latter half of the separated short audio should be entirely silent, and this was indeed the case. Observation Figure 5a and Figure 5b It was found that the decoded audio obtained by the baseline method contained unexpected spectrogram information in the latter half, indicating incomplete separation and residual interference. Playing the audio confirmed this conclusion. In contrast, the audio decoded by this invention exhibited excellent suppression in the latter half, filtering out almost all long audio segments, resulting in clearer data recovery, which is in line with expectations.

[0084] Table 4. Performance of the baseline method and the present invention on reverberation data.

[0085]

[0086]

[0087] References:

[0088] [1] R.Gu, S.-X.Zhang, and et al., Multi-modal multi-channel target speech separation, IEEE Journal of Selected Topics in Signal Processing, vol.14, no.3, pp.530–541, 2020.

[0089] [2]X.Mestre and M.A.Lagunas,On diagonal loading for minimum variancebeamformers,in Proceedings of the 3rd IEEE International Symposium on SignalProcessing and Information Technology,2003,pp.459–462.

[0090] [3]Zhuohuang Zhang,Yong Xu,Meng Yu,Shi-Xiong Zhang,Lianwu Chen,DongYu,ADL-MVDR:All deep learning MVDR beamformer for target speech separation,icassp,2020

[0091] [4]Yong Xu,Zhuohuang Zhang,Meng Yu,Shi-Xiong Zhang,Lianwu Chen,DongYu,G eneralized RNN Beamformer for Target Speech Separation,arXiv:2101.01280,2021

[0092] [5]D.S.Williamson,Y.Wang,and D.Wang,Complex ratio masking formonaural speech separation,IEEE TASLP,vol.24,no.3,pp.483–492,2015.

[0093] [6]W.Mack and E.A.Habets,Deep filtering:Signal extraction andreconstruction using complex time-frequency filters,IEEE Signal ProcessingLetters,vol.27,pp.61–65,2019.

[0094] This invention is not limited to the embodiments described above. The above description of specific embodiments is intended to illustrate and explain the technical solutions of this invention. The specific embodiments described above are merely illustrative and not restrictive. Without departing from the spirit and scope of the claims, those skilled in the art can make many specific modifications based on the teachings of this invention, and these modifications all fall within the scope of protection of this invention.

Claims

1. A beamforming method based on complex-gated cyclic units, characterized in that, Includes the following steps: (1) Perform complex masking prediction; the mixed audio is transformed into complex mixed audio features through short-time Fourier transform, and the logarithmic power spectrum is calculated using the real and imaginary parts of the complex mixed audio features. The smallest unit of the complex mixed audio features and the logarithmic power spectrum is the time-frequency unit, also known as a frame; the logarithmic power spectrum is input into the bidirectional long short-time memory network to predict complex masking. Since the mixed audio consists of the audio of several speakers, in order to extract the audio of each speaker, the number of predicted complex masks corresponds to the number of speakers. (2) The complex audio features of the speaker are obtained by multiplying the complex masking and complex mixed audio features element by element. The complex audio features of the speaker are then multiplied by their own conjugate transpose, and after standardization, the complex power spectral density matrix is ​​obtained by summing along the time dimension. When multiplying the elements, depth filtering is considered: the current frame of the calculated complex audio features of the speaker is numerically equal to the sum of the complex mixed audio features multiplied by the complex masking in the current frame and several frames before and after, respectively, to achieve a many-to-one mapping. (3) There are several complex power spectral density matrices, each corresponding to the audio of a speaker. In order to separate the audio of one speaker, the complex power spectral density matrix corresponding to the audio of one speaker is regarded as the target, which is the speech power spectral density matrix. The complex power spectral density matrices corresponding to the audio of all other speakers are summed and regarded as the interference power spectral density matrix. The real parts of the speech power spectral density matrix and the interference power spectral density matrix are concatenated with the real parts and the imaginary parts are concatenated with the imaginary parts to obtain the complex matrix features. (4) Construct a complex gated cyclic unit network, input the spliced ​​complex matrix features into the complex gated cyclic unit network, make full use of the characteristics of the complex network to predict the beamforming weighting coefficients; after matrix multiplication of the beamforming weighting coefficients and the complex mixed audio features, more accurate speaker complex audio features can be obtained. Perform inverse Fourier transform on the speaker complex audio features to obtain the separated speaker audio. The beamforming method based on complex gated cyclic units uses the scale-invariant signal-to-noise ratio as the loss function.

2. The beamforming method based on complex gated cyclic units according to claim 1, characterized in that, The specific process of depth filtering in step (2) is as follows: For the current frame of complex mixed audio features, take K frames before and after the time dimension and L frames before and after the frequency dimension, for a total of (2K+1)*(2L+1) frames. The complex masking is taken in the same way. After the element-wise product of (2K+1)*(2L+1) frames is completed, the sum is obtained, which is the current frame of the speaker's complex audio features.

3. The beamforming method based on complex gated cyclic units according to claim 1, characterized in that, The input and output of the complex gated cyclic unit network are both complex numbers. After the concatenated complex matrix features are input into the complex gated cyclic unit network, the real and imaginary parts of the complex matrix features are fully interacted, and the output prediction results are used as the weighting coefficients for beamforming.

4. The beamforming method based on complex gated cyclic units according to claim 1, characterized in that, A weighted coefficient is used to separate the audio of a speaker. In order to separate the audio of each speaker, steps (3) and (4) in claim 1 need to be repeated. The number of weighted coefficients for predicting beamforming is equal to the number of speakers.

Images