Beamforming method and system based on neural network filter with bidirectional GRU structure

Abstract

The present invention provides a beamforming method and system based on a neural network filter with a bidirectional GRU structure, comprising the steps of: S1: obtaining microphone array signals, dividing the microphone array signals into reference microphone signals and residual microphone signals; S2: constructing a reference channel processing module and a residual channel processing module; S3: inputting the reference microphone signal and the residual microphone signal into the reference channel processing module, and outputting a noise-reduced reference microphone signal; S4: inputting the noise-reduced reference microphone signal and the residual microphone signal into the residual channel processing module, and outputting a beamforming signal. The present invention is superior to baseline models and common traditional beamformers in terms of signal-to-noise ratio, and is more suitable for shipborne small microphone array signal enhancement tasks; a neural network filter including a bidirectional GRU module is used to replace a traditional filter, and the bidirectional GRU module strengthens the ability to perceive and model time series signals, thereby obtaining a better noise reduction effect.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a beamforming method and system for a neural network filter based on a bidirectional GRU structure. Background Art

[0002] During the voyage of a ship, the ship's machinery and equipment will generate relatively strong noise, and the small cabin of the ship will produce reverberation, which will interfere with the microphone array for audible sound monitoring. In order to remove the reverberation and noise in the audio signal as much as possible, the microphone array often uses beamforming technology as a means of gaining the target signal of the microphone array. Beamforming is a powerful microphone array processing technology that aims to extract the target signal in a specific direction and reduce the influence of noise and reverberation from multi-channel signals. As deep learning methods gradually dominate many audio processing tasks, beamformers based on deep learning have also been proven to be effective.

[0003] Most neural network beamformers fall roughly into three categories.

[0004] Mask-based beamforming estimates the filter and sum (FaS) beamforming filter in the frequency domain by estimating the time-frequency mask of the target source. This method uses the time-frequency mask to calculate the spatial covariance characteristics required for the optimal weights of the beamformer, such as minimum variance distortionless response and generalized eigenvalue beamformer.

[0005] Regression-based methods implicitly incorporate beamforming into the neural network without explicitly generating beamforming filters. This class of methods makes the assumption that the beamforming operation is implicitly included in the mapping function defined by the model. In this framework, the input channels are directly passed to the neural network (usually a convolutional neural network), and the training goal is to learn the mapping function between multi-channel input and target sources.

[0006] Based on the filtering method, a set of beamforming filters are learned using prior data to perform filtered sum beamforming on the target signal in the time domain or frequency domain. FaS beamforming applies beamforming filters to each channel and then adds them together to generate a single channel output, where the filters can be fixed or adaptive.

[0007] In traditional beamforming research, neural network beamformers combined with frequency domain features are usually designed and applied to tasks such as automatic speech recognition and sound source localization. Time domain neural network beamforming methods are less common than frequency domain neural network beamformers. Although frequency domain beamformers are known to be more robust and effective than time domain beamformers in various tasks. However, while frequency domain beamformers improve performance, they require higher frequency resolution and longer input signal lengths, which often leads to excessive time delays. For example, masking-based beamforming methods rely on the efficiency of the masking estimation network, and their performance is usually reduced in real-time application scenarios. Summary of the invention

[0008] In order to solve the above technical problems, the present invention provides a beamforming method of a neural network filter based on a bidirectional GRU structure, comprising the steps of:

[0009] S1: Acquire a microphone array signal, and divide the microphone array signal into a reference microphone signal and a remaining microphone signal;

[0010] S2: construct reference channel processing module and remaining channel processing module;

[0011] S3: input the reference microphone signal and the remaining microphone signal into the reference channel processing module, and output the reference microphone signal after noise reduction;

[0012] S4: Input the noise-reduced reference microphone signal and the remaining microphone signal into the remaining channel processing module, and output the beamforming signal.

[0013] Preferably, step S1 specifically comprises:

[0014] The microphone array signal includes N microphone signals. A reference microphone signal x is extracted from the microphone array signal. 1 , the remaining microphone signals are taken as the residual microphone signals x i , i is the number of the microphone, i=2,...,N.

[0015] Preferred:

[0016] The reference channel processing module includes: a normalization module, an average pooling module, a neural network filter and a convolution module, and the normalization module, the average pooling module, the neural network filter and the convolution module are connected in sequence;

[0017] The residual channel processing module includes: a normalization module, an average pooling module, a neural network filter, a convolution module and an addition module, and the normalization module, the average pooling module, the neural network filter, the convolution module and the addition module are connected in sequence.

[0018] Preferred:

[0019] The neural network filter includes: a cropping and splicing module, a first bidirectional GRU module, a second bidirectional GRU module and an overlap-addition module, and the cropping and splicing module, the first bidirectional GRU module, the second bidirectional GRU module and the overlap-addition module are connected in sequence;

[0020] The bidirectional GRU module includes: a GRU layer, a linear connection layer, a normalization layer and an addition layer, and the GRU layer, the linear connection layer, the normalization layer and the addition layer are connected in sequence.

[0021] Preferably, step S3 specifically comprises:

[0022] S31: The reference microphone signal x 1 Input the normalization module to obtain the reference feature vector R of the reference microphone signal 1 ; The remaining microphone signal x i Input normalization module and convert x 1 As the context window, calculate the remaining feature vector f i , i is the number of the microphone, i=2,...,N;

[0023] S32: The remaining eigenvector f i Input the average pooling module to calculate the remaining average feature vector The calculation formula is:

[0024]

[0025] S33: Reference feature vector R 1 and the remaining mean eigenvector After splicing, it is input into the neural network filter to generate the beamforming filter h 1 , the calculation formula is: H 1 (·) is the first mapping function;

[0026] S34: Set the beamforming filter h 1 With the reference microphone signal x 1 Channel The input convolution module performs convolution operation and outputs the reference microphone signal after noise reduction. The calculation formula is: is the convolution operator.

[0027] Preferably, step S4 is specifically:

[0028] S41: The reference microphone signal after noise reduction Input the normalization module to obtain the reference feature vector R of the reference microphone signal after noise reduction y ; The remaining microphone signal x iInput normalization module and convert x i As the context window, calculate the remaining feature vector g i , i is the number of the microphone, i=2,...,N;

[0029] S42: Reference feature vector R y and the remaining eigenvector g i After splicing, it is input into the neural network filter to generate the beamforming filter h i , the calculation formula is: i =H 2 ([R y ,g i ]), H 2 (·) is the second mapping function;

[0030] S43: Set the beamforming filter h i and the reference microphone signal after noise reduction Channel The input convolution module performs convolution operation and the output result is combined with the reference microphone signal after noise reduction. Add together to obtain the beamforming signal The calculation formula is: is the convolution operator.

[0031] A beamforming system based on a neural network filter of a bidirectional GRU structure comprises the following steps:

[0032] A microphone signal division module is used to obtain a microphone array signal and divide the microphone array signal into: a reference microphone signal and a remaining microphone signal;

[0033] A channel processing building module, used for building a reference channel processing module and a remaining channel processing module;

[0034] A noise reduction module, used for inputting the reference microphone signal and the residual microphone signal into the reference channel processing module, and outputting the reference microphone signal after noise reduction;

[0035] The beamforming signal acquisition module is used to input the noise-reduced reference microphone signal and the residual microphone signal into the residual channel processing module and output the beamforming signal.

[0036] The present invention has the following beneficial effects:

[0037] 1. Perform two noise reduction operations on the microphone signal through the reference channel processing module and the remaining channel processing module to remove the reverberation and noise in the audio signal collected by the microphone array as much as possible;

[0038] 2. The reference microphone signal is de-noised by the reference channel processing module, making the beamforming signal more accurate. The residual channel processing module considers the addition of the filtered outputs of all channels of the residual microphone signal in the task of extracting the beam, making the beamforming more comprehensive and stable. It is superior to the baseline model and the common traditional beamformer in terms of signal-to-noise ratio, and is more suitable for the signal enhancement task of shipborne small microphone arrays.

[0039] 3. Use a neural network filter containing a bidirectional GRU module instead of a traditional filter. The bidirectional GRU module enhances the ability to perceive and model time series signals, thereby achieving better noise reduction effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;

[0041] Figure 2 It is the workflow diagram of the reference channel processing module;

[0042] Figure 3 The workflow diagram for the remaining channel processing module;

[0043] Figure 4 This is the workflow diagram of the neural network filter;

[0044] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0046] Reference Figure 1 The present invention provides a beamforming method for a neural network filter based on a bidirectional GRU structure, comprising the steps of:

[0047] S1: Acquire a microphone array signal, and divide the microphone array signal into a reference microphone signal and a remaining microphone signal;

[0048] S2: construct reference channel processing module and remaining channel processing module;

[0049] S3: input the reference microphone signal and the remaining microphone signal into the reference channel processing module, and output the reference microphone signal after noise reduction;

[0050] S4: Input the noise-reduced reference microphone signal and the remaining microphone signal into the remaining channel processing module, and output the beamforming signal.

[0051] Furthermore, step S1 is specifically as follows:

[0052] The microphone array signal includes N microphone signals. A reference microphone signal x is extracted from the microphone array signal. 1 , the remaining microphone signals are taken as the residual microphone signals x i , i is the number of the microphone, i=2,...,N.

[0053] Specifically, the time-domain filter summation beamforming problem can be defined as estimating a set of time-domain filters for a microphone array with two or more microphones, so that the sum of the filtered signals of all microphones provides the best estimate of the target signal in the selected reference microphone. First, the signal x at each microphone is i Split into frames of L sampling points:

[0054]

[0055] Where t is the frame number, i is the microphone number, and L is the length. This operation can be interpreted as selecting microphone signals with a length of L from number tL to number tL+L-1. The value of

[0056] To account for the time difference of arrival (TDOA) of the signal of interest at different microphones, a filter summation operation is applied to a context window around the t-th frame for each microphone to generate the beamformed output at the t-th frame:

[0057]

[0058] in is the beamforming signal of the tth frame, is the context window used by microphone i, R is a set of real numbers, is the beamforming filter to be learned for microphone i, and the operator represents the convolution operation. Since the direction of the sound source is always unknown, a context window is used to ensure that the model can capture the cross-microphone delay of ±L samples.

[0059] For further reference, Figure 2-3 :

[0060] The reference channel processing module includes: a normalization module, an average pooling module, a neural network filter and a convolution module, and the normalization module, the average pooling module, the neural network filter and the convolution module are connected in sequence;

[0061] The residual channel processing module includes: a normalization module, an average pooling module, a neural network filter, a convolution module and an addition module, and the normalization module, the average pooling module, the neural network filter, the convolution module and the addition module are connected in sequence.

[0062] For further reference, Figure 4 :

[0063] The neural network filter includes: a cropping and splicing module, a first bidirectional GRU module, a second bidirectional GRU module and an overlap-addition module, and the cropping and splicing module, the first bidirectional GRU module, the second bidirectional GRU module and the overlap-addition module are connected in sequence;

[0064] The bidirectional GRU module includes: a GRU layer, a linear connection layer, a normalization layer and an addition layer, and the GRU layer, the linear connection layer, the normalization layer and the addition layer are connected in sequence.

[0065] Specifically, the neural network filter adopts a processing structure similar to that of DPRNN (Dual-Path Recurrent Neural Network). Taking the reference channel processing module as an example: first, the input Segment along the time direction, and combine the segmented S blocks into a synthetic feature vector through the cropping and splicing module Each block is 2P long (the first and last blocks are zero-padded);

[0066] The neural network filter is designed using a residual structure. The sum of the input and output of each bidirectional GRU module is used as the input of the next bidirectional GRU module; the synthetic feature vector In the process of transferring between two bidirectional GRU modules, the time dimension and channel dimension in the vector are transposed. The transposed vector As the input of the second bidirectional GRU module; the two bidirectional GRU modules examine the context information in the time direction and the channel direction of the cropped stack respectively, and finally organize it into a sequential output through overlap addition

[0067] The bidirectional GRU module connects a linear connection layer, i.e. a fully connected layer, after the GRU layer to ensure the consistency of the length and number of channels between the input and output, and normalizes the output of the linear connection layer, making the network easier to train.

[0068] A training set and a test set containing different background noises and target signals are used to construct the established neural network filter. The training method of the filter summation beamforming method model based on the bidirectional recurrent neural network filter also includes: retraining the acoustic signal filter model after changing different application scenarios; after changing the application scenario, transfer learning is performed on the model to adapt to changes in different application scenarios, so that the model can cover different application scenarios and improve the generalization of the filter model.

[0069] Furthermore, step S3 is specifically as follows:

[0070] S31: The reference microphone signal x 1 Input the normalization module to obtain the reference feature vector R of the reference microphone signal 1 ; The remaining microphone signal x i Input normalization module and convert x 1 As the context window, calculate the remaining feature vector f i , i is the number of the microphone, i=2,...,N;

[0071] Specifically, in the first stage of beamforming, the reference channel processing module randomly selects a reference microphone from the microphone array and applies the beamforming filter to the signal in this channel; the normalization module uses the frame-level normalized cross-correlation (NCC) as the inter-channel feature; 1 Acts as a context window for the signal in the reference microphone; is the corresponding center frame of the other microphones with the same frame number, and then calculate and The NCC feature between , in this example defined as the cosine similarity:

[0072]

[0073] The subscript j is the element number in the vector. is the vector f i The elements in is the residual feature vector (i.e., the cosine similarity between the reference microphone and microphone i), and the NCC feature contains the time difference information and content-related information of the signal of interest in the reference microphone and other microphones;

[0074] S32: The remaining eigenvector f i Input the average pooling module to calculate the remaining average feature vector The calculation formula is:

[0075]

[0076] S33: Reference feature vector R 1 and the remaining mean eigenvector After splicing, it is input into the neural network filter to generate the beamforming filter h 1 , the calculation formula is: H 1 (·) is the first mapping function;

[0077] Specifically, for the characteristics of a specific channel, Apply a linear layer on top to generate a K-dimensional embedding This can be interpreted as multiplying the matrix by the weight matrix:

[0078] R 1 =x 1 U

[0079] in is the weight matrix;

[0080] S34: Set the beamforming filter h 1 With the reference microphone signal x 1 Channel The input convolution module performs convolution operation and outputs the reference microphone signal after noise reduction. The calculation formula is: is the convolution operator.

[0081] Furthermore, step S4 is specifically as follows:

[0082] S41: The reference microphone signal after noise reduction Input the normalization module to obtain the reference feature vector R of the reference microphone signal after noise reduction y ; The remaining microphone signal x i Input normalization module and convert x i As the context window, calculate the remaining feature vector g i , i is the number of the microphone, i=2,...,N;

[0083] Specifically, the remaining channel processing module uses the output from the reference channel processing module as a clue and applies the above-mentioned similar procedure to all the remaining microphones; for the context window The microphone i, between which Calculate the NCC feature between:

[0084]

[0085] The subscript j is the element number in the vector. is the vector g i Elements in

[0086] S42: Reference feature vector R y and the remaining eigenvector g i After splicing, it is input into the neural network filter to generate the beamforming filter h i , the calculation formula is: i =H 2 ([R y ,g i ]), H 2(·) is the second mapping function;

[0087] S43: Set the beamforming filter h i and the reference microphone signal after noise reduction Channel The input convolution module performs convolution operation and the output result is then combined with the reference microphone signal after noise reduction. Add together to obtain the beamforming signal The calculation formula is: is the convolution operator.

[0088] The present invention provides a beamforming system of a neural network filter based on a bidirectional GRU structure, comprising the steps of:

[0089] A microphone signal division module is used to obtain a microphone array signal and divide the microphone array signal into: a reference microphone signal and a remaining microphone signal;

[0090] A channel processing building module, used for building a reference channel processing module and a remaining channel processing module;

[0091] A noise reduction module, used for inputting the reference microphone signal and the residual microphone signal into the reference channel processing module, and outputting the reference microphone signal after noise reduction;

[0092] The beamforming signal acquisition module is used to input the noise-reduced reference microphone signal and the residual microphone signal into the residual channel processing module and output the beamforming signal.

[0093] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0094] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order and these words may be interpreted as identifiers.

[0095] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A beamforming method for a neural network filter based on a bidirectional GRU structure, characterized in that: Includes steps: S1: Acquire a microphone array signal, and divide the microphone array signal into a reference microphone signal and a remaining microphone signal; S2: construct reference channel processing module and remaining channel processing module; S3: input the reference microphone signal and the remaining microphone signal into the reference channel processing module, and output the reference microphone signal after noise reduction; S4: inputting the noise-reduced reference microphone signal and the remaining microphone signal into the remaining channel processing module, and outputting a beamforming signal; The reference channel processing module includes: a normalization module, an average pooling module, a neural network filter and a convolution module, and the normalization module, the average pooling module, the neural network filter and the convolution module are connected in sequence; The residual channel processing module includes: a normalization module, an average pooling module, a neural network filter, a convolution module and an addition module, and the normalization module, the average pooling module, the neural network filter, the convolution module and the addition module are connected in sequence; Step S3 is specifically as follows: S31: The reference microphone signal x 1 Input the normalization module to obtain the reference feature vector R of the reference microphone signal 1 ; The remaining microphone signal x i Input normalization module and convert x 1 As the context window, calculate the remaining feature vector f i , i is the number of the microphone, i=2,...,N; S32: The remaining eigenvector f i Input the average pooling module to calculate the remaining average feature vector The calculation formula is: S33: Reference feature vector R 1 and the remaining mean eigenvector After splicing, it is input into the neural network filter to generate the beamforming filter h 1 , the calculation formula is: H1(·) is the first mapping function; S34: Set the beamforming filter h 1 With the reference microphone signal x 1 Channel The input convolution module performs convolution operation and outputs the reference microphone signal after noise reduction. The calculation formula is: is the convolution operator; Step S4 is specifically as follows: S41: The reference microphone signal after noise reduction Input the normalization module to obtain the reference feature vector R of the reference microphone signal after noise reduction y ; The remaining microphone signal x i Input normalization module and convert x i As the context window, calculate the remaining feature vector g i , i is the number of the microphone, i=2,...,N; S42: The reference feature vector R y and the remaining eigenvector g i After splicing, the neural network filter is input to generate the beamforming filter h, and the calculation formula is: h = H2([R y ,g i ]), H2(·) is the second mapping function; S43: Set the beamforming filter h i and the reference microphone signal after noise reduction Channel The input convolution module performs convolution operation and the output result is then combined with the reference microphone signal after noise reduction. Add together to obtain the beamforming signal The calculation formula is: is the convolution operator.

2. The beamforming method of the neural network filter based on the bidirectional GRU structure according to claim 1 is characterized in that: Step S1 is specifically as follows: The microphone array signal includes N microphone signals. A reference microphone signal x is extracted from the microphone array signal. 1 , the remaining microphone signals are taken as the residual microphone signals x i , i is the number of the microphone, i=2,...,N.

3. The beamforming method of the neural network filter based on the bidirectional GRU structure according to claim 1 is characterized in that: The neural network filter includes: a cropping and splicing module, a first bidirectional GRU module, a second bidirectional GRU module and an overlap-addition module, and the cropping and splicing module, the first bidirectional GRU module, the second bidirectional GRU module and the overlap-addition module are connected in sequence; The bidirectional GRU module includes: a GRU layer, a linear connection layer, a normalization layer and an addition layer, and the GRU layer, the linear connection layer, the normalization layer and the addition layer are connected in sequence.

4. A beamforming system for a neural network filter based on a bidirectional GRU structure, used to implement the beamforming method for a neural network filter based on a bidirectional GRU structure as described in any one of claims 1 to 3, characterized in that: include: A microphone signal division module is used to obtain a microphone array signal and divide the microphone array signal into: a reference microphone signal and a remaining microphone signal; A channel processing building module, used for building a reference channel processing module and a remaining channel processing module; A noise reduction module, used for inputting the reference microphone signal and the residual microphone signal into the reference channel processing module, and outputting the reference microphone signal after noise reduction; The beamforming signal acquisition module is used to input the noise-reduced reference microphone signal and the residual microphone signal into the residual channel processing module and output the beamforming signal.

Images