A multi-channel speech enhancement method using microphone array pickup

Abstract

The present invention discloses a multi-channel speech enhancement method using a microphone array to pick up sound, the method comprising using multiple single-channel microphones to pick up sound in multiple channels and pre-processing the sound; based on the selection of a reference microphone, using NCC and MCS feature extraction and joint averaging to reduce the signal difference between channels, and designing a filter corresponding to the reference microphone; performing NCC and MCS feature extraction on other single-channel microphones, and designing filters corresponding to other single-channel reference microphones one by one; combining the enhanced speech signals of all single-channel microphones and performing mean processing to obtain the final enhanced speech signal. The multi-channel speech enhancement method provided by the present invention uses NCC and MCS feature extraction and joint averaging to reduce the signal difference between channels, and has a good noise reduction effect on near- and far-field noisy speech in indoor space.

Description

Technical Field

[0001] The present invention relates to the technical field of speech noise reduction, and in particular to a multi-channel speech enhancement method using a microphone array to pick up sound. Background Art

[0002] In real production and life scenarios, various complex noise interferences will seriously affect the communication quality of people using voice devices or the recognition performance of voice recognition systems. It is necessary to improve the clarity and intelligibility of voice through front-end processing of voice signals in order to improve system performance. Generally speaking, multiple microphones are used to form a certain form of array structure for multi-channel sound pickup. In addition to the time domain and frequency domain characteristics of the voice signal, more spatial feature information can be obtained than that of a single microphone, so as to design a higher-dimensional voice enhancement algorithm. Most traditional microphone array-based voice enhancement algorithms use beamforming (BF) methods. The algorithm collects the spatial information of the signal, improves the voice gain in the target direction, and enhances the target signal. Signals in other directions are considered as interference signals. The beamforming algorithm is generally divided into two stages. In the first stage, the signals received by each microphone are synchronously processed. A microphone is selected as a reference microphone, and the relative delay from the voice signal to each microphone is calculated. The signals of each microphone channel are synchronized in time through delay compensation. In the second stage, the weights of each microphone channel are determined. Different channels are given different weights to adapt to the problem scenario, and the weighted sum and average of each channel signal are used to obtain the final enhanced speech. The beamforming algorithm based on signal processing has several shortcomings:

[0003] 1. The algorithm is based on some mathematical assumptions and noise spectrum estimation, which only conform to the characteristics of the scene to a certain extent, making the system's noise reduction performance limited;

[0004] 2. This type of method cannot enhance useful speech signals outside the beam and cannot meet the requirements of full-scene sound pickup;

[0005] 3. Due to the different distances between the speaker and the microphone in real scenes, it is difficult to ensure the uniform distribution of feature energy of the speaker's voice signal. In a sentence, some segments have loud sounds while others have soft sounds. The non-stable voice state also makes the system's voice enhancement effect unstable. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention provides a multi-channel speech enhancement method using a microphone array to pick up sound, and the technical solution is as follows:

[0007] The present invention provides a multi-channel speech enhancement method using a microphone array to pick up sound, comprising the following steps:

[0008] S1, using multiple single-channel microphones to pick up the voice in multiple channels and perform pre-processing;

[0009] S2. Select one of the single-channel microphones as a reference microphone. Based on the selection of the reference microphone, perform normalization processing on the multi-channel sound pickup signal, use NCC and MCS feature extraction and joint averaging to reduce the signal difference between channels, and then design a filter corresponding to the reference microphone through a multi-head attention network. Use the filter to perform a convolution operation on the sound pickup signal of the reference microphone to obtain an enhanced speech signal after filtering by the reference microphone.

[0010] S3, combining the enhanced speech signal after filtering by the reference microphone, performing NCC and MCS feature extraction on other single-channel microphones, and designing filters corresponding to other single-channel reference microphones through a multi-head attention network, and other single-channel microphones perform convolution operations on their respective pickup signals through corresponding filters to obtain enhanced speech signals after filtering by other single-channel microphones respectively;

[0011] S4. Combine the enhanced speech signals of all single-channel microphones and perform average processing to obtain a final enhanced speech signal.

[0012] Further, in step S1, the preprocessing includes the following steps:

[0013] S101, padding the voice data with zeros before and after;

[0014] S102, performing frame processing on the zero-filled voice data,

[0015]

[0016] Where M is the frame length, K is the frame shift, Z is the number of frames, t is the frame index value, and i is the index of the microphone. represents the speech vector of single-channel microphone i at frame t;

[0017] S103, adding a context voice window to the framed voice data,

[0018]

[0019] Where W is the context window size, is the speech vector containing the context of microphone i at frame t;

[0020] S104, add the framed voice data x of all channels to an array with a shape of [N, M, Z], and add the voice data of the context window of all channels Append to another array of shape [N,2W+M,Z], where N is the number of channels.

[0021] Furthermore, in step S2, NCC and MCS feature extraction are performed on the speech data with the context window added, and average pooling operations are performed respectively to serve as part of the input data in the multi-head attention network.

[0022] Furthermore, the results of MCS feature extraction and average pooling operation are subjected to one-dimensional convolution and GroupNorm operation to obtain another part of input data in the multi-head attention network.

[0023] Furthermore, the design of the filter corresponding to the reference microphone comprises the following steps:

[0024] S201, the input data is input into the multi-head attention network output, a jump connection is used to add the output result to the input data, and a LayerNorm regularization operation is performed;

[0025] S202, subjecting the result of step S201 to a one-dimensional convolution process with a Prelu activation function;

[0026] S203, the result of step S202 is first transformed into a shape, and then processed by BiLSTM and Prelu activation functions;

[0027] S204, input the result of step S203 to the Linear layer and perform shape transformation;

[0028] S205, using a skip connection to add the result of step S201 to the result of step S204, and performing a LayerNorm regularization operation;

[0029] S206, performing a shape transformation on the result of step S205, and then performing a one-dimensional convolution operation on the transformed result and the input data in step S201 using a skip connection, and processing using the Prelu activation function;

[0030] S207: Input the results of step S206 into one-dimensional convolutions with sigmoid and Tanh activation functions respectively, and transform the shapes after multiplying them to generate a filter corresponding to the reference microphone.

[0031] Furthermore, in step S3, NCC feature extraction is performed on the speech data with the context window added in combination with the enhanced speech signal after filtering by the reference microphone, and MCS feature extraction and average pooling operation are performed on the framed speech data corresponding to other single-channel microphones to serve as part of the input data in the multi-head attention network.

[0032] Furthermore, one-dimensional convolution and GroupNorm operations are performed on the speech data with added context windows corresponding to other single-channel microphones to obtain another part of input data in the multi-head attention network.

[0033] Furthermore, the design of the filter corresponding to other single-channel microphones includes the following steps:

[0034] S301, the input data is transformed in shape and then input into the multi-head attention network output, a jump connection is used to add the output result to the input data, and a LayerNorm regularization operation is performed;

[0035] S302, after the result of step S301 is transformed in shape, it is subjected to a one-dimensional convolution process with a Prelu activation function;

[0036] S303, after the result of step S302 is transformed in shape, it is processed by BiLSTM and Prelu activation function;

[0037] S304, input the result of step S303 to the Linear layer for processing;

[0038] S305, using a skip connection to add the result of step S301 to the result of step S304, and performing a LayerNorm regularization operation;

[0039] S306, performing a shape transformation on the result of step S305, and then performing a one-dimensional convolution operation on the transformed result and the input data in step S301 using a skip connection, and processing using the Prelu activation function;

[0040] S307: Input the results of step S306 into one-dimensional convolutions with sigmoid and Tanh activation functions respectively, and transform the shapes after multiplying them to generate filters corresponding to other single-channel microphones.

[0041] Further, a multi-channel speech enhancement model is constructed by using steps S2-S4, and the multi-channel speech enhancement model is trained. Before step S1, training data needs to be generated as the object of the single-channel microphone pickup, and then steps S1-S4 are performed, wherein generating training data includes the following steps:

[0042] P1, randomly select an audio from the clean speech dataset and the noise dataset with the same sampling rate;

[0043] P2. Split the selected clean speech and noise audio into audio files of the same length;

[0044] P3. Set the signal-to-noise ratio according to the noise intensity of the noise reduction scene;

[0045] P4. Set the size of the noise reduction space and the position of the single-channel microphone, speaker, and noise source, and use the gpuRIR toolkit to simulate sound reverberation.

[0046] Furthermore, when training the multi-channel speech enhancement model, a scale-invariant signal-to-noise ratio is used as a loss function to perform model parameter adjustment calculation to obtain an optimized multi-channel speech enhancement model.

[0047] The beneficial effects brought by the technical solution provided by the present invention are as follows:

[0048] (1) It has a good noise reduction effect on near- and far-field noisy speech in indoor spaces;

[0049] (2) The problem of uneven energy distribution of the received speech signal is solved, and the accuracy of the received signal is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 1 is a schematic diagram of the first stage of speech enhancement in the multi-channel speech enhancement method provided by an embodiment of the present invention;

[0052] Figure 2 Schematic diagram of the ACG-Net module structure of the multi-channel speech enhancement method provided by an embodiment of the present invention;

[0053] Figure 3 1 is a schematic diagram of the second stage of speech enhancement in the multi-channel speech enhancement method provided by an embodiment of the present invention;

[0054] Figure 4 is a schematic diagram of the microphone distribution structure in the multi-channel speech enhancement method provided by an embodiment of the present invention;

[0055] Figure 5 It is a schematic diagram of the MCS feature extraction process in the multi-channel speech enhancement method provided by an embodiment of the present invention;

[0056] Figure 6 It is a schematic diagram of the model training and reasoning process in the multi-channel speech enhancement method provided by an embodiment of the present invention;

[0057] Figure 7It is a speech waveform diagram before noise reduction of different microphones in the multi-channel speech enhancement method provided by an embodiment of the present invention;

[0058] Figure 8 is a speech waveform diagram after noise reduction of different microphones in the multi-channel speech enhancement method provided by an embodiment of the present invention;

[0059] Fig. 9 is a spectrum diagram of different microphones before noise reduction in the multi-channel speech enhancement method provided by an embodiment of the present invention;

[0060] Fig.10 It is a spectrum diagram after noise reduction of different microphones in the multi-channel speech enhancement method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to enable those skilled in the art to better understand the scheme of the present invention, and to more clearly understand the purpose, technical scheme and advantages of the present invention, the technical scheme in the embodiment of the present invention is clearly and completely described below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be noted that the implementation methods not shown or described in the accompanying drawings are forms known to those skilled in the art. In addition, although demonstrations of parameters containing specific values ​​may be provided herein, it should be understood that the parameters do not need to be exactly equal to the corresponding values, but may be approximated to the corresponding values ​​within an acceptable error tolerance or design constraint. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work should belong to the scope of protection of the present invention. In addition, the terms "including" and "having" in the specification and claims of the present invention and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment containing a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0062] In one embodiment of the present invention, a multi-channel speech enhancement method using a microphone array to pick up sound is provided, comprising the following steps:

[0063] S1, using multiple single-channel microphones to pick up the voice in multiple channels and perform pre-processing;

[0064] Among them, the preprocessing includes the following steps:

[0065] S101, padding the voice data with zeros before and after;

[0066] S102, performing frame processing on the zero-filled voice data,

[0067]

[0068] Where M is the frame length, K is the frame shift, Z is the number of frames, t is the frame index value, and i is the index of the microphone. represents the speech vector of single-channel microphone i at frame t;

[0069] S103, adding a context voice window to the framed voice data,

[0070]

[0071] Where W is the context window size, is the speech vector containing the context of microphone i at frame t;

[0072] S104, add the framed voice data x of all channels to an array with a shape of [N, M, Z], and add the voice data of the context window of all channels Append to another array of shape [N,2W+M,Z], where N is the number of channels.

[0073] S2, see Figure 1 , select one of the single-channel microphones as a reference microphone, based on the selection of the reference microphone, perform normalization processing on the multi-channel sound pickup signal, use NCC and MCS feature extraction and joint averaging to reduce the signal difference between channels, and then design a filter corresponding to the reference microphone through a multi-head attention network, and use the filter to perform a convolution operation on the sound pickup signal of the reference microphone to obtain an enhanced speech signal after filtering by the reference microphone as the first stage of speech enhancement;

[0074] Specifically, the speech data with the context window added is subjected to NCC and MCS feature extraction and average pooling operations respectively, to serve as part of the input data in the multi-head attention network. The result of the MCS feature extraction and average pooling operation is subjected to one-dimensional convolution and GroupNorm operation to obtain another part of the input data in the multi-head attention network.

[0075] Among them, see Figure 2 , the design of the filter corresponding to the reference microphone comprises the following steps:

[0076] S201, the input data is input into the multi-head attention network output, a jump connection is used to add the output result to the input data, and a LayerNorm regularization operation is performed;

[0077] S202, subjecting the result of step S201 to a one-dimensional convolution process with a Prelu activation function;

[0078] S203, the result of step S202 is first transformed into a shape, and then processed by BiLSTM and Prelu activation functions;

[0079] S204, input the result of step S203 to the Linear layer and perform shape transformation;

[0080] S205, using a skip connection to add the result of step S201 to the result of step S204, and performing a LayerNorm regularization operation;

[0081] S206, performing a shape transformation on the result of step S205, and then performing a one-dimensional convolution operation on the transformed result and the input data in step S201 using a skip connection, and processing using the Prelu activation function;

[0082] S207: Input the results of step S206 into one-dimensional convolutions with sigmoid and Tanh activation functions respectively, and transform the shapes after multiplying them to generate a filter corresponding to the reference microphone.

[0083] S3, see Figure 3 , combined with the enhanced speech signal filtered by the reference microphone, two features of NCC and MCS are extracted for other single-channel microphones, and filters corresponding to other single-channel reference microphones are designed through a multi-head attention network. Other single-channel microphones perform convolution operations on their respective picked-up signals through corresponding filters to obtain enhanced speech signals filtered by other single-channel microphones, respectively, as the second stage of speech enhancement;

[0084] Specifically, the NCC feature extraction is performed on the speech data with the added context window combined with the enhanced speech signal after filtering by the reference microphone, and the MCS feature extraction and average pooling operation are performed on the framed speech data corresponding to other single-channel microphones to serve as part of the input data in the multi-head attention network. The speech data with the added context window corresponding to other single-channel microphones is subjected to one-dimensional convolution and GroupNorm operation to obtain another part of the input data in the multi-head attention network.

[0085] The design of the filter corresponding to other single-channel microphones includes the following steps:

[0086] S301, the input data is transformed in shape and then input into the multi-head attention network output, a jump connection is used to add the output result to the input data, and a LayerNorm regularization operation is performed;

[0087] S302, after the result of step S301 is transformed in shape, it is subjected to a one-dimensional convolution process with a Prelu activation function;

[0088] S303, after the result of step S302 is transformed in shape, it is processed by BiLSTM and Prelu activation function;

[0089] S304, input the result of step S303 to the Linear layer for processing;

[0090] S305, using a skip connection to add the result of step S301 to the result of step S304, and performing a LayerNorm regularization operation;

[0091] S306, performing a shape transformation on the result of step S305, and then performing a one-dimensional convolution operation on the transformed result and the input data in step S301 using a skip connection, and processing using the Prelu activation function;

[0092] S307: Input the results of step S306 into one-dimensional convolutions with sigmoid and Tanh activation functions respectively, and transform the shapes after multiplying them to generate filters corresponding to other single-channel microphones.

[0093] S4. Combine the enhanced speech signals of all single-channel microphones and perform average processing to obtain a final enhanced speech signal.

[0094] A multi-channel speech enhancement model is constructed using steps S2-S4, and the multi-channel speech enhancement model is trained. Before step S1, training data needs to be generated as the object of sound pickup by the single-channel microphone, and then steps S1-S4 are executed. When the multi-channel speech enhancement model is trained, the scale-invariant signal-to-noise ratio is used as the loss function to perform model parameter adjustment calculation to obtain an optimized multi-channel speech enhancement model.

[0095] Generating training data includes the following steps:

[0096] P1, randomly select an audio from the clean speech dataset and the noise dataset with the same sampling rate;

[0097] P2. Split the selected clean speech and noise audio into audio files of the same length;

[0098] P3. Set the signal-to-noise ratio according to the noise intensity of the noise reduction scene;

[0099] P4. Set the size of the noise reduction space and the position of the single-channel microphone, speaker, and noise source, and use the gpuRIR toolkit to simulate sound reverberation.

[0100] This embodiment uses a two-stage enhancement method based on a deep learning framework to design a multi-channel speech noise reduction algorithm. It balances the energy distribution of the received signal through normalized calculation, organically combines multiple channel feature extraction methods, fully utilizes the spatial features between microphone arrays, and jointly models the multi-head attention mechanism network and the sequence network to improve the feature extraction ability of the enhancement model for data, so as to better train the multi-channel array filter. Finally, based on the summation of multi-channel filtering, a pure speech signal is obtained.

[0101] The following is a further explanation using a simulated indoor meeting scenario as a specific example:

[0102] Step 1 generates training data to simulate the multi-channel speech enhancement problem in an indoor meeting scenario.

[0103] Step 1.1 randomly select an audio from the clean speech dataset and the noise dataset, both of which have a sampling rate of 16K Hz. The sampling rate of commonly used datasets is 16K Hz.

[0104] Step 1.2: Divide the selected clean speech and noise into audio files of length 4s.

[0105] Step 1.3 sets the signal-to-noise ratio according to the noise intensity of the denoising scene, in the range of [-15db, 0db], to increase the coverage of the training data.

[0106] Step 1.4 Set the size of the noise reduction space and the locations of the distributed microphones, speakers, and noise sources. Use the gpuRIR toolkit to simulate sound reverberation. The settings of the distributed microphone array are as follows: Figure 4 shown. Figure 4 The middle circle indicates the speaker's position, the triangle indicates the noise source's position, the rectangle of 10 meters long and 5 meters wide is the size of the conference room, the rectangle in the middle of 5 meters long and 1 meter wide is a conference table, the five-pointed star on the conference table indicates a single-channel microphone, and the four five-pointed stars indicate a four-microphone array structure distributed in a ring.

[0107] Step 1.5 adds the noise to the speaker's clean speech according to different signal-to-noise ratios to generate noisy speech.

[0108] Step 2: Data preprocessing

[0109] Step 2.1 pads the noisy speech data with zeros at the front and back, with 32 zeros at the front and back, to prevent the first and last features of the speech data from being lost.

[0110] Step 2.2 performs frame processing on the zero-filled speech data, setting the frame length M to 64 sampling points and the frame shift K to 32 sampling points.

[0111]

[0112] Where t is the frame index value, i is the index of the microphone, represents the speech vector of microphone i at frame t, and Z is the frame number.

[0113] Step 2.3 adds a context speech window to the framed speech.

[0114]

[0115] Where W is the context window size, is the speech vector of microphone i at frame t containing context to reduce the delay error between channels.

[0116] Step 2.4 Add the framed speech data x of all channels to an array with a shape of [N, M, Z], and add contextual speech data of all channels Append to another array of shape [N, 2W+M, Z], where N is the number of channels.

[0117] Step 3: The first stage of multi-channel speech enhancement

[0118] Step 3.1 NCC-based channel feature extraction

[0119] Step 3.1.1 Perform normalized cross-correlation channel feature extraction

[0120]

[0121] in, is the cosine similarity between the reference microphone and microphone i, is the reference microphone signal.

[0122] Step 3.1.2: Perform an average pooling operation on the result of step 3.1.1

[0123]

[0124] in, Normalized cross-correlation after taking the mean.

[0125] Step 3.2 Inter-channel feature extraction based on MCS

[0126] Step 3.2.1 See Figure 5 , perform multi-channel convolution and channel feature extraction

[0127]

[0128] in, The size of the two-dimensional convolution kernel is [N, K], the step size is set to K / 2, and the number of convolution kernels is set to E.

[0129] Step 3.2.2 Perform an average pooling operation on the result of step 3.2.1

[0130]

[0131] in,

[0132] Step 3.3 Pre-processing module design (Pre-Net)

[0133] Step 3.3.1 to step 3.2.2 and Perform shape transformation

[0134]

[0135]

[0136] in,

[0137] Step 3.3.2: out ref Perform a one-dimensional convolution operation, a GroupNorm operation, and then a shape transformation

[0138] m t =GroupNorm(Conv1d(out ref ))

[0139]

[0140] in, is the speech feature vector at frame number t, the convolution kernel size of the one-dimensional convolution is 2W+M, and the number of output channels is M.

[0141] Step 3.4 Filter Design (ACG-Net)

[0142] Step 3.4.1 and Conduct a joint

[0143]

[0144] in, represents the combined speech vector at frame number t,

[0145] Represents the combined speech vector.

[0146] Step 3.4.2 Transform the shape of the combined vector and input it into the one-dimensional convolution

[0147]

[0148]

[0149] in The convolution kernel size of one-dimensional convolution is 1, and the number of output channels is M.

[0150] Step 3.4.3 inputs out2 into the multi-head attention network, uses skip connections, and performs LayerNorm regularization.

[0151] out3=LayerNorm(MultiHeadAtt(out2)+out2)

[0152] in, The output dimension of MultiHeadAtt is set to M and the head value is 4.

[0153] Step 3.4.4: The result of step 3.4.3 is subjected to a one-dimensional convolution with the Prelu activation function.

[0154] out4 = Prelu(Conv1d(out3))

[0155] in, The convolution kernel size of one-dimensional convolution is 1, and the number of output channels is M / 2.

[0156] Step 3.4.5: Then transform the result of step 3.4.4 into a new shape, and then pass it through BiLSTM and Prelu activation functions.

[0157]

[0158]

[0159] in The output dimension of BiLSTM is set to 2M.

[0160] Step 3.4.6: Input the result of step 3.4.5 into the Linear layer and perform shape transformation.

[0161] out6 = reshape(Linear(out5))

[0162] in, The output dimension of the Linear layer is M.

[0163] Step 3.4.7 uses a skip connection to add out3 and out6 and perform LayerNorm regularization.

[0164] out7=LayerNorm(out3+out6)

[0165] in,

[0166] Step 3.4.8 performs a shape transformation on the result of step 3.4.7, and then performs a shape transformation on out2 and Using skip connections

[0167]

[0168]

[0169] in,

[0170] Step 3.4.9 performs a one-dimensional convolution operation on the result of step 3.4.8 and uses the Prelu activation function

[0171] out9 = Prelu(Conv1d(out8))

[0172] in, The convolution kernel size of one-dimensional convolution is 1, and the output channel is 2W+1.

[0173] Step 3.4.10 inputs out9 into the one-dimensional convolution with sigmoid and Tanh activation functions respectively and multiplies them.

[0174] h=Tanh(Conv1d(out9))⊙Sigmoid(Conv1d(out9))

[0175] in, ⊙ is the Hadamard product, the convolution kernel size of the one-dimensional convolution is 1, and the number of output channels C is 2W+1.

[0176] Step 3.5 Convolution summation to generate pure speech

[0177] Step 3.5.1 Transform h

[0178] Filter1 = reshape(h)

[0179] in, is the generated beamforming filter.

[0180] Step 3.5.2 uses the beamforming filter generated in step 3.5.1 to filter the speech from the reference microphone. Perform convolution operation,

[0181]

[0182] in, Represents the enhanced speech signal after filtering of the reference microphone.

[0183] Step 4: Multi-channel speech enhancement stage 2

[0184] Step 4.1 NCC-based channel feature extraction

[0185] Step 4.1.1 Perform normalized cross-correlation channel feature extraction

[0186]

[0187] in, is the cosine similarity calculation result of the reference microphone signal after noise reduction and the signal of microphone i, is the speech signal of microphone i.

[0188] Step 4.2 Inter-channel feature extraction based on multi-channel convolution

[0189] Step 4.2.1 Get the input signal x t The other microphone signals are transformed

[0190]

[0191] in,

[0192] Step 4.2.2 Perform shape transformation on the output of step 3

[0193] Ref t = reshape(R t )

[0194] in,

[0195] Step 4.2.3 Combine Ref with otherx

[0196] Con t =concat([Ref t ,otherx t ])

[0197] in, is the combined speech vector.

[0198] Step 4.2.4 Perform MCS channel feature extraction (two-dimensional convolution)

[0199] g2 t =Conv2d(Con t )

[0200] in, The size of the two-dimensional convolution kernel is [N, G], the step size is set to G / 2, and the number of convolution kernels is set to E.

[0201] Step 4.2.5 performs an average pooling operation on step 4.2.4.

[0202]

[0203] in,

[0204] Step 4.2.6 performs shape transformation on the result of step 4.2.5 and then performs dimension expansion.

[0205]

[0206]

[0207] in,

[0208] Step 4.3 Pre-processing module design (Pre-Net)

[0209] Step 4.3.1 Get input signal Average the other microphone signals and change the shape

[0210]

[0211] in,

[0212] Step 4.3.2: Cotherx t Perform a one-dimensional convolution operation and a GroupNorm operation.

[0213] cm t =GroupNorm(Conv1d(Cotherx t ))

[0214]

[0215] in, is the speech feature vector at frame number t, the convolution kernel size of the one-dimensional convolution is 2W+M, and the number of output channels is M.

[0216] Step 4.4 Filter Design (ACG-Net)

[0217] Step 4.4.1 f t Conduct a joint

[0218]

[0219] in,

[0220] Step 4.4.2: Transform the shape of the combined vector and input it into the one-dimensional convolution.

[0221]

[0222]

[0223] in The convolution kernel size of one-dimensional convolution is 1, and the number of output channels is M.

[0224] Step 4.4.3. 10 After the shape is changed, it is input into the multi-head attention network, using skip connections and performing LayerNorm regularization.

[0225]

[0226]

[0227] in, The output dimension of MultiHeadAtt is set to M and the head value is 4.

[0228] Step 4.4.4 will be out 11 First, it undergoes shape transformation, and then undergoes a one-dimensional convolution with a Prelu activation function.

[0229]

[0230]

[0231] in, The convolution kernel size of one-dimensional convolution is 1, and the number of output channels is M / 2.

[0232] Step 4.4.5: Then transform the result of step 4.4.4, and then pass it through BiLSTM and Prelu activation functions.

[0233]

[0234]

[0235] in The output dimension of BiLSTM is set to M.

[0236] Step 4.4.6: Input the result of step 4.4.5 to the Linear layer for processing.

[0237] out14 =Linear(out 13 )

[0238] in, The output dimension of the Linear layer is M.

[0239] Step 4.4.7 uses skip connection to connect out 11 without 14 Add and perform LayerNorm regularization operation

[0240] out 15 =LayerNorm(out 11 +out 14 )

[0241] in,

[0242] Step 4.4.8 performs a shape transformation on the result of step 4.4.7, and then performs a shape transformation on the out 10 and Using skip connection processing,

[0243]

[0244]

[0245] in,

[0246] Step 4.4.9: out 16 Perform a one-dimensional convolution operation and use the Prelu activation function.

[0247] out 17 =Prelu(Conv1d(out 16 ))

[0248] in, The convolution kernel size of one-dimensional convolution is 1, and the output channel is 2W+1.

[0249] Step 4.4.10 will be out 17 Input into the one-dimensional convolution with sigmoid and Tanh activation functions respectively and multiply them.

[0250] h2=Tanh(Conv1d(out 17 ))⊙Sigmoid(Conv1d(out 17 ))

[0251] in, ⊙ is the Hadamard product, the convolution kernel size of one-dimensional convolution is 1, and the number of output channels is 2W+1.

[0252] Step 4.5 Convolution and summation

[0253] Step 4.5.1 Transform the shape of h2.

[0254] Filter = reshape(h2)

[0255] in, is the generated beamforming filter.

[0256] Step 4.5.2 uses the beamforming filter generated in step 4.5.1 to filter the corresponding speech Perform convolution operation,

[0257]

[0258] in, represents the filtered speech signal of microphone i.

[0259] Step 4.6 Generate pure speech

[0260] Step 4.6.1: Enhance the speech signal Combined with R, the dimension is changed and the previously padded zeros are removed to restore the original speech length.

[0261]

[0262] in, Represents the speech enhancement signal restored to its original length for all microphones.

[0263] Step 4.6.2 calculates the average value of the results of step 4.6.1.

[0264]

[0265] in, Represents the final output speech enhancement signal.

[0266] Step 5 Loss function design

[0267] Step 5.1 Use scale-invariant signal-to-noise ratio (SI-SNR) as the loss function

[0268]

[0269] e noise =ys target

[0270]

[0271] Among them, y, They are the noise-reduced speech and the pure speech respectively.

[0272] Step 6 Hyperparameter Setting

[0273] The head of the multi-head attention network is set to 4, the convolution kernel size used by MSC is 64, the number of convolution kernels is 16, the step size is 2, and the dilation number is 2. In the training phase, 150 epochs are used for training, and the model with the best performance in the validation set is saved. In addition, during training, if the model loss value of the last 10 epochs does not decrease on the validation set, the training will be automatically stopped. Adam is used as the loss function optimizer, and to prevent gradient explosion, the L2 norm of 5 is used for gradient clipping. For the learning rate, the dynamic strategy warmup is used to adjust it during the training process. It can warm up the model with a smaller learning rate at the beginning of training to increase the stability of the model, and then gradually reduce it every 3 epochs at a decay rate of 0.98.

[0274]

[0275]

[0276] Among them, lr refers to the learning rate, n is the number of training steps, a1, a2, n_warmups, d model is a hyperparameter. In the experiment, we set a1 to 0.2, a2 ​​to 1e-3, n_warmups to 4000, and d model is 64.

[0277] After the training is completed, a multi-channel speech enhancement model with determined parameters is obtained as an optimized model.

[0278] Step 7: Model Inference

[0279] Use noisy speech as model input and pass the data through steps 2-4 to generate enhanced speech. For details, see Figure 6 .

[0280] The speech waveform and spectrum diagram of this embodiment before and after multi-channel speech enhancement are shown in Figure 2. Figures 7 to 10 Before processing, the noise energy is high, and it is difficult to distinguish the speech component and the noise component in the time domain. After algorithm processing, the noise component is eliminated more thoroughly, the silent segment has tended to be a straight line, and there is also obvious noise cancellation on the spectrum diagram. The part with the lowest noise energy corresponds to the straight line part of the time domain waveform.

[0281] This embodiment is based on the selection of a single-channel reference microphone signal, normalizes the multi-channel pickup signal, uses NCC and MCS to extract and jointly average the signal difference between channels, and designs the ACG-Net filter through a multi-head attention mechanism, and uses convolution calculation to obtain the reference microphone noise-reduced speech signal. By calculating the inter-channel features, a filter corresponding to the microphone is generated through the ACG-Net network. After convolution filtering, the result is averaged to obtain the final noise-reduced speech. The advantage of this method is that the input data is processed in a standardized manner, and a two-stage form is used to strengthen the extraction of inter-channel features. The channel features of microphones in different channels are calculated at each stage, and the far and near field features of the space are fully balanced, which better solves the problem of uneven speech energy in actual scenes. Experimental results show that this method can achieve a good noise reduction effect on far and near field noisy speech in indoor spaces.

[0282] The multi-channel speech enhancement method provided by the present invention designs a multi-channel speech enhancement algorithm based on a deep learning framework. Compared with the traditional method, there is no need to assume conditions for algorithm design, and it is data-driven to better utilize spatial features. For the multi-channel signal picked up in the microphone array, the speech signal data is first transformed so that its energy distribution is within the energy distribution range of the speech data used in the training set, and the mean and variance are normalized to solve the problem of uneven energy distribution of the received speech signal, avoid the received signal affecting the model effect, and reduce the model performance. In the algorithm framework of deep learning, a two-stage method is designed for multi-channel speech enhancement. In the first stage, a reference microphone is selected, and multiple inter-channel feature extraction methods are integrated to perform speech enhancement on the reference microphone; in the second stage, the enhanced reference microphone signal is used to extract inter-channel features for other channels, and the signals of other channels are enhanced, and finally the enhanced signals are averaged and pooled to form an enhanced speech signal.

[0283] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-channel speech enhancement method using a microphone array to pick up sound, characterized in that: The following steps are involved: S1, using multiple single-channel microphones to pick up the voice in multiple channels and perform pre-processing; S2. Select one of the single-channel microphones as a reference microphone. Based on the selection of the reference microphone, perform normalization processing on the multi-channel sound pickup signal, use NCC and MCS feature extraction and joint averaging to reduce the signal difference between channels, and then design a filter corresponding to the reference microphone through a multi-head attention network. Use the filter to perform a convolution operation on the sound pickup signal of the reference microphone to obtain an enhanced speech signal after filtering by the reference microphone. S3, combining the enhanced speech signal after filtering by the reference microphone, performing NCC and MCS feature extraction on other single-channel microphones, and designing filters corresponding to other single-channel reference microphones through a multi-head attention network, and other single-channel microphones perform convolution operations on their respective pickup signals through corresponding filters to obtain enhanced speech signals after filtering by other single-channel microphones respectively; S4, combining the enhanced speech signals of all single-channel microphones and performing mean processing to obtain a final enhanced speech signal; In step S1, the preprocessing includes the following steps: S101, padding the voice data with zeros before and after; S102, performing frame processing on the zero-filled voice data, Where M is the frame length, K is the frame shift, Z is the number of frames, t is the frame index value, and i is the index of the microphone. represents the speech vector of single-channel microphone i at frame t, x i Represents the framed speech data corresponding to the i-th single-channel microphone; S103, adding a context voice window to the framed voice data, Where W is the context window size, is the speech vector containing the context of microphone i at frame t; S104, add the framed voice data x of all channels to an array with a shape of [N, M, Z], and add the voice data of the context window of all channels Add to another array with shape [N,2W+M,Z], where N is the number of channels; The design of the filter corresponding to the reference microphone comprises the following steps: S201, input data to the multi-head attention network output, use a jump connection to add the output result to the input data, and perform a LayerNorm regularization operation; S202, subjecting the result of step S201 to a one-dimensional convolution process with a Prelu activation function; S203, the result of step S202 is first transformed into a shape, and then processed by BiLSTM and Prelu activation functions; S204, input the result of step S203 to the Linear layer and perform shape transformation; S205, using a skip connection to add the result of step S201 to the result of step S204, and performing a LayerNorm regularization operation; S206, performing a shape transformation on the result of step S205, and then performing a one-dimensional convolution operation on the transformed result and the input data in step S201 using a skip connection, and processing using the Prelu activation function; S207, inputting the results of step S206 into one-dimensional convolutions with sigmoid and Tanh activation functions respectively, and transforming the shapes after multiplying them to generate a filter corresponding to the reference microphone; In step S3, NCC feature extraction is performed on the speech data with the context window added and the enhanced speech signal after filtering by the reference microphone, and MCS feature extraction and average pooling operation are performed on the framed speech data corresponding to other single-channel microphones to serve as part of the input data in the multi-head attention network; The speech data with the added context window corresponding to other single-channel microphones is subjected to one-dimensional convolution and GroupNorm operation to obtain another part of input data in the multi-head attention network.

2. The multi-channel speech enhancement method according to claim 1, characterized in that: In step S2, NCC and MCS feature extraction are performed on the speech data with the context window added, and average pooling operations are performed respectively to serve as part of the input data in the multi-head attention network.

3. The multi-channel speech enhancement method according to claim 2, characterized in that: The results of MCS feature extraction and average pooling operation are subjected to one-dimensional convolution and GroupNorm operation to obtain another part of input data in the multi-head attention network.

4. The multi-channel speech enhancement method according to claim 1, characterized in that: The design of the filter corresponding to other single-channel microphones includes the following steps: S301, the input data is transformed in shape and then input into the multi-head attention network output, a jump connection is used to add the output result to the input data, and a LayerNorm regularization operation is performed; S302, after the result of step S301 is transformed in shape, it is subjected to a one-dimensional convolution process with a Prelu activation function; S303, after the result of step S302 is transformed in shape, it is processed by BiLSTM and Prelu activation function; S304, input the result of step S303 to the Linear layer for processing; S305, using a skip connection to add the result of step S301 to the result of step S304, and performing a LayerNorm regularization operation; S306, performing a shape transformation on the result of step S305, and then performing a one-dimensional convolution operation on the transformed result and the input data in step S301 using a skip connection, and processing using the Prelu activation function; S307: Input the results of step S306 into one-dimensional convolutions with sigmoid and Tanh activation functions respectively, and transform the shapes after multiplying them to generate filters corresponding to other single-channel microphones.

5. The multi-channel speech enhancement method according to claim 1, characterized in that: A multi-channel speech enhancement model is constructed by using steps S2-S4, and the multi-channel speech enhancement model is trained. Before step S1, training data needs to be generated as the object of the single-channel microphone pickup, and then steps S1-S4 are performed, wherein generating training data includes the following steps: P1, randomly select an audio from the clean speech dataset and the noise dataset with the same sampling rate; P2. Split the selected clean speech and noise audio into audio files of the same length; P3. Set the signal-to-noise ratio according to the noise intensity of the noise reduction scene; P4. Set the size of the noise reduction space and the position of the single-channel microphone, speaker, and noise source, and use the gpuRIR toolkit to simulate sound reverberation.

6. The multi-channel speech enhancement method according to claim 5, characterized in that: When training the multi-channel speech enhancement model, the scale-invariant signal-to-noise ratio is used as the loss function to perform model parameter adjustment calculation to obtain an optimized multi-channel speech enhancement model.

Images