Super-long-distance multi-channel speech enhancement method based on generative filtering summation

Abstract

The invention provides an ultra-long-distance multi-channel speech enhancement method based on generative filtering summation, and belongs to the technical field of speech signal processing. Multi-channel feature fusion is realized through a united strategy of a normalized cross correlation coefficient and a signal difference feature, and the weight of a time domain filter is optimized to enhance the quality of a target voice signal in combination with the two-stage design of a generative filtering summation network. In the first stage, a reference channel is preliminarily enhanced, and in the second stage, multi-channel weighted summation is utilized to further optimize an output signal. According to the method, the scale-invariant signal-to-noise ratio (SI-SNR) is adopted as an optimization target, and the signal-to-noise ratio and the voice definition can be remarkably improved in a complex noise environment. The method provided by the invention is suitable for complex scenes of ultra-long-distance voice signal acquisition and enhancement.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech signal processing, and in particular, relates to a method for ultra-long-distance multi-channel speech enhancement based on generative filter summation. Background Art

[0002] As an important carrier for information transmission, speech signals have wide applications in many fields, such as human-computer interaction, communication control, security monitoring, etc. However, with the diversification of application scenarios, especially under the demand for ultra-long-distance speech acquisition, traditional sound pickup devices and speech enhancement methods face many challenges.

[0003] In an ultra-long-distance environment, the propagation of speech signals will encounter serious attenuation problems. During the propagation process, sound waves will cause energy loss due to geometric attenuation, absorption attenuation, and scattering attenuation. Specifically, as the propagation distance increases, the sound pressure level of the sound wave drops sharply, and the speech signal far from the sound source becomes very weak and is difficult to be effectively captured by traditional microphones. When the distance from the source reaches dozens of meters, the signal-to-noise ratio of the speech signal may drop to -30 dB or even lower, which makes it difficult to obtain sufficiently clear speech data even when using ordinary microphone devices for signal acquisition.

[0004] To enhance such low signal-to-noise ratio long-distance speech signals, researchers have proposed various methods, such as using hardware devices such as high-sensitivity microphones, microphone arrays, and acoustic amplifiers. However, these devices rely on complex hardware designs and precise configurations, and there are still deficiencies in terms of cost, volume, and actual application effects. For example, although using radar sensors or optical laser microphones can improve the ability of long-distance acquisition, these devices are often costly and have limited effects in specific environments (such as environments without reflectors).

[0005] In addition, most traditional speech enhancement algorithms are based on the processing of single-channel or near-field environment speech signals. Facing the extremely low signal-to-noise ratio of ultra-long-distance speech signals, they usually cannot achieve significant effects. Current common speech enhancement technologies, such as algorithms based on Wiener filtering, deep neural networks (DNN), and convolutional neural networks (CNN), perform unstably in ultra-long-distance scenarios. Especially in complex environments with multiple noise source interferences, the robustness and effectiveness of traditional algorithms are poor.

[0006] In summary, the existing technologies have multiple deficiencies in the field of ultra-long-distance speech enhancement, including problems such as too low signal-to-noise ratio and unstable performance of traditional enhancement algorithms. Therefore, there is an urgent need for a new type of ultra-long-distance speech enhancement method that can simultaneously solve the technical bottlenecks in the acquisition and enhancement processes. Summary of the Invention

[0007] In view of this, the present invention aims to overcome the deficiencies of the above problems in the prior art, and proposes a super-long-distance multi-channel speech enhancement method based on generative filter summation, which is applicable to complex scenarios of super-long-distance speech signal acquisition and enhancement.

[0008] To achieve the above object, the technical solution of the present invention is realized as follows:

[0009] The first aspect of the present invention provides a super-long-distance multi-channel speech enhancement method based on generative filter summation, including the following steps:

[0010] Step 1: Calculate the normalized cross-correlation coefficient of the speech signals collected by multiple microphones. By performing differential operations on the signals of adjacent microphones, extract the differential features of the noise signal, fuse the normalized cross-correlation coefficient with the signal differential features, and extract the joint feature representation through a deep learning network;

[0011] Step 2: Select the reference channel as the target enhancement channel, generate a time-domain filter through a deep learning network, and perform preliminary enhancement on the reference signal;

[0012] Step 3: Optimize the filter weights and generate the final enhanced signal through multi-channel filter summation;

[0013] Step 4: Use the scale-invariant signal-to-noise ratio as the training target, directly optimize the time-domain signal quality, and use the stochastic gradient descent algorithm to perform iterative updates of the network parameters.

[0014] Further, in the step 1, the normalized cross-correlation coefficient is calculated by using the cosine similarity.

[0015] Further, in the step 1, the extraction of the differential features of the noise signal includes:

[0016] First, obtain the time-domain differential signal, and then perform Fourier transform on the differential signal to obtain:

[0017] Diff = DFT{y(t)};

[0018] where Diff is the signal differential feature, y(t) = n(t) - n(t - τ0) is the difference in the time domain of the signals received by two microphones, is the time delay of the noise signal arriving at the two microphones in the propagation direction. Among them, the included angle between the acoustic wave signal emitted by the noise source and the microphone array is α, fs is the sampling frequency, and c is the speed of sound.

[0019] Further, in step 2, the time-domain filter adopts an architecture combining multi-head attention mechanism and BiLSTM. The time-domain filter inputs the fused feature vector x obtained from the upper layer into the multi-head attention network, and combines the original features and the processed features in a skip connection manner, and performs LayerNorm regularization operation to obtain out1:

[0020] out1 = LayerNorm(Multiheadatt(x)+x);

[0021] Input the processed model feature out1 into a one-dimensional convolutional layer with Prelu as the activation function to reduce the dimension of the channels and obtain out2:

[0022] out2 = Prelu(Conv1d(out1));

[0023] Input out2 into a BiLSTM with Prelu as the activation function, and utilize the advantage of BiLSTM in representing time series to obtain out3;

[0024] out3 = Prelu(BiLSTM(out2));

[0025] Use a fully connected layer to perform a dimension reduction operation on the output result of BiLSTM, and the output after dimension reduction is out4;

[0026] out4 = Linear(out3);

[0027] In a skip connection manner, add out2 and out4, and perform LayerNorm regularization operation to obtain the finally generated beamforming filter out5;

[0028] out5 = LayerNorm(out2+out4).

[0029] Further, step 3 specifically includes:

[0030] Use an iterative update method to optimize the filter weights to ensure the consistency of each channel signal in space and time;

[0031] Introduce a multi-channel collaborative training mechanism to enable the generated filter to better adapt to the characteristics of different channel signals;

[0032] Apply the optimized filters to all channel signals respectively to generate enhanced multi-channel signals;

[0033] Through weighted summation operation, synthesize the enhancement results of different channels and output the final enhanced signal.

[0034] The second aspect of the present invention provides a long - distance multi - channel voice enhancement device based on generative filter summation, including:

[0035] A multi - channel voice signal feature fusion module, which is used to calculate the normalized cross - correlation coefficient of the voice signals collected by multiple microphones, extract the differential features of the noise signal by performing differential operations on the signals of adjacent microphones, fuse the normalized cross - correlation coefficient with the signal differential features, and extract the joint feature representation through a deep learning network;

[0036] A filter generation module, which is used to select a reference channel as the target enhancement channel, generate a time - domain filter through a deep learning network, and perform preliminary enhancement on the reference signal;

[0037] A filter optimization module, which is used to optimize the filter weights and generate the final enhanced signal through multi - channel filter summation;

[0038] A training and optimization module, which is used to use the scale - invariant signal - to - noise ratio as the training target, directly optimize the time - domain signal quality, and perform iterative updates of the network parameters using the stochastic gradient descent algorithm.

[0039] The third aspect of the present invention provides an electronic device, including a processor and a memory communicatively connected to the processor and used to store the executable instructions of the processor. The processor is used to execute the above - mentioned long - distance multi - channel voice enhancement method based on generative filter summation.

[0040] The fourth aspect of the present invention provides a computer - readable storage medium storing a computer program, and when the computer program is executed by a processor, it implements the above - mentioned long - distance multi - channel voice enhancement method based on generative filter summation.

[0041] Compared with the prior art, the long - distance multi - channel voice enhancement method based on generative filter summation of the present invention has the following advantages:

[0042] Through the two - stage design of multi - channel signal feature extraction and fusion and the generative filter summation framework, the present invention can significantly improve the quality and intelligibility of long - distance voice signals in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0044] Figure 1 It is the overall architecture diagram of the generative filter summation network of the present invention.

[0045] Figure 2Schematic diagram of the multi-channel feature fusion method of the present invention.

[0046] Figure 3 Structural diagram of the filter generation module of the present invention. Detailed implementation manners

[0047] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments may be combined with each other.

[0048] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "transverse", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first", "second", etc. may explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise specified, the meaning of "plurality" is two or more.

[0049] In the description of the present invention, it should be noted that, unless otherwise clearly defined and limited, the terms "mounted", "connected", "coupled" should be understood in a broad sense, for example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection, an electrical connection; it may be directly connected, or indirectly connected through an intermediate medium, and it may be the internal communication of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0050] The present invention will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0051] The present invention provides a super-long-distance multi-channel voice enhancement method based on a generative filter summation network. Figure 1 It is the overall architecture diagram of the generative filter summation network, showing the filter summation network structures of the first stage and the second stage. The filter production modules in the figure are all the same. The specific steps are as follows:

[0052] The first step: Multi-channel feature calculation and fusion

[0053] 1. Calculation of the normalized cross-correlation coefficient

[0054] The normalized cross - correlation coefficient extracts the spatial features of the target speech by measuring the temporal synchronization and directionality between multi - channel signals. To calculate the features between channels more conveniently, the present invention specializes the normalized cross - correlation coefficient to cosine similarity (CS). Specifically, as Figure 2 shown, this process first normalizes each pair of channel signals to eliminate the influence of signal amplitude differences, and then calculates the correlation features between the signals. Suppose there are two vectors in a D - dimensional space, which are X=(x1, x2,..., x D ), Y=(y1, y2,..., y D ), then the cosine similarity is obtained as follows:

[0055]

[0056] where θ represents the angle between vectors X and Y. D is the dimension of the space, and cosθ ∈ [- 1,1].

[0057] By normalizing the signals, the influence of the signal intensity differences between different channels is significantly reduced, making the correlation features more stable. This method is particularly suitable for ultra - long - distance scenarios, can effectively extract the directional information of the target speech, and provides reliable support for the subsequent filter generation. The specific method includes:

[0058] 1) Normalize each pair of channel signals to eliminate the influence of signal amplitude differences;

[0059] 2) Calculate the normalized cross - correlation function to obtain the correlation features between the signals;

[0060] 3) Select the features related to the directionality of the target speech as the key input for subsequent processing.

[0061] 2. Signal differential feature extraction

[0062] Based on the design of the differential microphone array, the present invention estimates the spatial characteristics of the noise signal by calculating the differences between adjacent channel signals. The differential features can effectively suppress the noise in non - target directions and enhance the directionality of the target speech signal. The extraction method of the differential features is as Figure 2 shown. The differential features are obtained by first acquiring the time - domain differential signal and then performing a Fourier transform on the differential signal. That is:

[0063] Diff = DFT{y(t)};

[0064] where Diff is the signal differential feature adopted by the present invention, and y(t)=n(t)-n(t - τ0) is the difference in the time domain between the signals received by two microphones, that is, the differential signal, τ is the time delay for the noise signal to reach the two microphones in the propagation direction. Among them, the angle between the acoustic wave signal emitted by the noise source and the microphone array is α, fs is the sampling frequency, and c is the speed of sound (c = 340 m / s).

[0065] The differential feature can effectively capture the spatial characteristics of environmental noise and suppress background noise in a multi-noise source scenario. Based on the design of the differential microphone array, this method can not only maintain the integrity of the target speech but also effectively eliminate the interference noise in non-target directions.

[0066] 3. Feature Fusion

[0067] The present invention inputs the normalized cross-correlation coefficient and differential features into the feature fusion module, and extracts the joint feature representation through a deep learning network. The fused features retain the key spatial information of the multi-channel signals and at the same time have strong anti-noise interference ability. The joint fusion strategy of the normalized cross-correlation coefficient and the signal differential features makes full use of the spatio-temporal information of the multi-channel signals to generate a unified feature representation. The fused feature vector is as Figure 2 shown, which can effectively support the subsequent filter generation process. It not only has high sensitivity to the target speech but also can significantly enhance the adaptability to complex noise scenarios, providing high-quality input for the subsequent operations of the generative filtering summation framework.

[0068] Step 2: First-stage filter generation

[0069] Select the reference channel as the target enhancement channel, use a multi-layer convolutional neural network (CNN) to extract the time-domain features, generate the filter weights through the feature calculation and fusion module, and perform convolution operations with the reference channel signal to output the preliminarily enhanced reference signal.

[0070] Figure 3 shows the structure of the filter generation module. The present invention adopts an architecture combining the multi-head attention mechanism and BiLSTM (Bidirectional Long Short-Term Memory Network) in the filter generation module:

[0071] a. The multi-head attention mechanism can simultaneously focus on different parts of the input sequence and capture various features in the speech signal. This mechanism can not only understand the information in the signal more comprehensively, including local details and global context, but also improve the robustness of the system and adapt to various acoustic scenarios.

[0072] b. BiLSTM effectively models the long-distance dependence relationship in the speech signal by comprehensively considering the context information of the time series. Its bidirectional structure can capture the front and back temporal characteristics of the signal, and at the same time alleviates the problem of gradient disappearance, providing a better feature representation.

[0073] This module first inputs the fused feature vector x obtained from the upper layer into the multi-head attention network, and combines the original features and the processed features using skip connections to increase the reusability of the model features. Then, a LayerNorm regularization operation is performed to improve the operation speed of the model, and thus out1 is obtained:

[0074] out1 = LayerNorm(Multiheadatt(x) + x);

[0075] The processed model feature out1 is input into a one-dimensional convolutional layer with Prelu as the activation function to reduce the dimension of the channels, reduce the computational load of the subsequent module, and also increase the fusion of information between channels. After such processing, out2 is obtained:

[0076] out2 = Prelu(Conv1d(out1));

[0077] Next, out2 is input into a BiLSTM with Prelu as the activation function, and out3 is obtained by taking advantage of the superiority of BiLSTM in representing time series.

[0078] out3 = Prelu(BiLSTM(out2));

[0079] Since the output dimension of BiLSTM is twice the input dimension, for the convenience of subsequent processing, a fully connected layer is also needed to reduce the dimension of the output result of BiLSTM, and the output after dimension reduction is out4.

[0080] out4 = Linear(out3).

[0081] Finally, in the same way of using skip connections, out2 and out4 are added together, and a LayerNorm regularization operation is performed to obtain the finally generated beamforming filter out5.

[0082] out5 = LayerNorm(out2 + out4).

[0083] Step 3: Optimization of the filter and filtering summation in the second stage

[0084] On the basis of the enhancement in the first stage, the second stage further optimizes the filter weights and generates the final enhanced signal through multi-channel filtering summation.

[0085] 1. Generation of the optimized filter

[0086] 1) Use the iterative update method to optimize the filter weights to ensure the consistency of the signals in each channel in space and time;

[0087] 2) Introduce a multi-channel collaborative training mechanism to enable the generated filters to better adapt to the characteristics of different channel signals.

[0088] 2. Filter summation processing

[0089] 1) As Figure 1 and Figure 3 shown, apply the optimized filters to all channel signals respectively to generate enhanced multi-channel signals;

[0090] 2) Through weighted summation operation, synthesize the enhancement results of different channels and output the final enhanced signal.

[0091] Fourth step: Training and optimization

[0092] The present invention uses scale-invariant signal-to-noise ratio (SI-SNR) as the training objective. By directly optimizing the time-domain signal quality and using the stochastic gradient descent (SGD) algorithm for iterative update of network parameters, it ensures that the enhanced signal retains a higher auditory perception effect.

[0093] The second aspect of the present invention provides a super-long-distance multi-channel speech enhancement device based on generative filter summation, including:

[0094] A multi-channel speech signal feature fusion module, which is used to calculate the normalized cross-correlation coefficient of the speech signals collected by multiple microphones, extract the differential features of the noise signal by performing differential operations on the signals of adjacent microphones, perform feature fusion on the normalized cross-correlation coefficient and the signal differential features, and extract joint feature representations through a deep learning network;

[0095] A filter generation module, which is used to select a reference channel as the target enhancement channel, generate a time-domain filter through a deep learning network, and perform preliminary enhancement on the reference signal;

[0096] A filter optimization module, which is used to optimize the filter weights and generate the final enhanced signal through multi-channel filter summation;

[0097] A training and optimization module, which is used to use scale-invariant signal-to-noise ratio as the training objective, directly optimize the time-domain signal quality, and use the stochastic gradient descent algorithm for iterative update of network parameters.

[0098] The third aspect of the present invention provides an electronic device, including a processor and a memory communicatively connected to the processor and used to store instructions executable by the processor. The processor is used to execute the above-mentioned super-long-distance multi-channel speech enhancement method based on generative filter summation.

[0099] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which when executed by a processor implements the above-mentioned ultra-long-distance multi-channel voice enhancement method based on generative filter summation.

[0100] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for ultra-long-distance multi-channel speech enhancement based on generative filtering summation, characterized in that: It includes the following steps: Step 1: Calculate the normalized cross-correlation coefficients of the speech signals collected by multiple microphones. By performing differential operations on the signals of adjacent microphones, extract the differential features of the noise signals. Perform feature fusion on the normalized cross-correlation coefficients and the signal differential features, and extract the joint feature representation through a deep learning network; Step 2: Select the reference channel as the target enhancement channel, generate a time-domain filter through a deep learning network, and perform preliminary enhancement on the reference signal; Step 3: Optimize the filter weights, and generate the final enhanced signal through multi-channel filtering and summation; Step 4: Use the scale-invariant signal-to-noise ratio as the training objective. By directly optimizing the time-domain signal quality, use the stochastic gradient descent algorithm to perform iterative updates of the network parameters.

2. The ultra-long-distance multi-channel speech enhancement method based on generative filter summation according to claim 1, characterized in that: In the above Step 1, the cosine similarity is used to calculate the normalized cross-correlation coefficients.

3. A method for ultra-long-distance multi-channel speech enhancement based on generative filtering summation according to claim 1, characterized in that: In the above Step 1, the extraction of the differential features of the noise signals includes: First obtain the time-domain differential signal, and then perform Fourier transform on the differential signal to obtain: Diff = DFT{y(t)}; Among them, Diff is the signal differential feature, y(t) = n(t) - n(t - τ0) is the difference in the time domain of the signals received by the two microphones, τ0 is the time delay for the noise signal to reach the two microphones in the propagation direction, α is the angle between the acoustic wave signal emitted by the noise source and the microphone array, fs is the sampling frequency, and c is the speed of sound.

4. A super-long-distance multi-channel speech enhancement method based on generative filter summation according to claim 1, characterized in that: In the above Step 2, the time-domain filter adopts an architecture combining the multi-head attention mechanism and BiLSTM. The time-domain filter inputs the fused feature vector x obtained from the upper layer into the multi-head attention network, and uses the skip connection method to combine the original features and the processed features, and perform the LayerNorm regularization operation to obtain out1: out1 = LayerNorm(Multiheadatt(x) + x); Input the processed model feature out1 into a one-dimensional convolutional layer with Prelu as the activation function to reduce the dimension of the channels and obtain out2: out2 = Prelu(Conv1d(out1)); Input out2 into a BiLSTM with Prelu as the activation function, and utilize the advantage of BiLSTM in representing time series to obtain out3; out3 = Prelu(BiLSTM(out2)); Use a fully connected layer to perform a dimension reduction operation on the output result of BiLSTM, and the output after dimension reduction is out4; out4 = Linear(out3); Adopt the skip connection method, add out2 and out4, and perform the LayerNorm regularization operation to obtain the finally generated beamforming filter out5; out5 = LayerNorm(out2 + out4).

5. A method for ultra-long-distance multi-channel speech enhancement based on generative filtering summation according to claim 1, characterized in that: The above Step 3 specifically includes: Use the iterative update method to optimize the filter weights to ensure the consistency of each channel signal in space and time; Introduce a multi-channel collaborative training mechanism to enable the generated filter to better adapt to the characteristics of different channel signals; Apply the optimized filters to all channel signals respectively to generate enhanced multi-channel signals; Through weighted summation operation, synthesize the enhancement results of different channels and output the final enhanced signal.

6. A super-long-distance multi-channel voice enhancement device based on generative filter summation, characterized in that: It includes: The multi-channel speech signal feature fusion module is used to calculate the normalized cross-correlation coefficient of the speech signals collected by multiple microphones. By performing differential operations on the signals of adjacent microphones, the differential features of the noise signals are extracted. The normalized cross-correlation coefficient and the signal differential features are subjected to feature fusion, and the joint feature representation is extracted through a deep learning network; The filter generation module is used to select a reference channel as the target enhancement channel, generate a time-domain filter through a deep learning network, and perform preliminary enhancement on the reference signal; The filter optimization module is used to optimize the filter weights and generate the final enhanced signal through multi-channel filtering summation; The training and optimization module is used to adopt the scale-invariant signal-to-noise ratio as the training target, directly optimize the time-domain signal quality, and use the stochastic gradient descent algorithm to perform iterative updates of the network parameters.

7. An electronic device, comprising a processor and a memory communicatively connected to the processor and configured to store executable instructions of the processor, wherein: The processor is used to execute a super-long-distance multi-channel speech enhancement method based on generative filtering summation according to any one of claims 1-5 above.

8. A computer-readable storage medium stores a computer program, characterized in that: When the computer program is executed by the processor, it implements a super-long-distance multi-channel speech enhancement method based on generative filtering summation according to any one of claims 1-5.

Images