Multichannel voice signal enhancement method

By using covariance matrix and sparse technology in multi-channel speech signal processing, combined with deep learning networks and traditional methods, voice enhancement in complex environments is achieved, the problems of adaptability and computing resource consumption are solved, and noise is improved Inhibition and generalization ability.

CN119049497APending Publication Date: 2024-11-29PEOPLE CN CO LTD
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411084674.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The existing multi-channel speech enhancement technology is insufficiently adaptable in complex acoustic environments, and it is difficult to balance between maintaining speech naturalness and noise suppression, which consumes a lot of computing resources, and lacks generalization capabilities in low signal-to-noise ratio environments.

Method used

By framing the multi-channel voice signals collected by the microphone array, the covariance matrix and the global noise covariance matrix are calculated, the weight matrix is ​​generated and sparse, and the FusionRNNoise network and MCRA-OMLSA method are combined to achieve dual enhancement to realize the voice signal Beamforming and noise suppression.

Benefits of technology

While ensuring voice quality, effectively suppress background noise, reduce computing resource consumption, and improve generalization capabilities in complex noise environments and low signal-to-noise ratio environments. It is suitable for low-resource platforms, with real-time and good voice enhancement effects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119049497A_ABST
    Figure CN119049497A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-channel voice signal enhancement method, and belongs to the technical field of multi-channel voice enhancement. According to the method, the problems that the background noise is difficult to be effectively suppressed and the consumption of computing resources is difficult to be reduced while the voice enhancement effect is ensured, and the generalization ability is insufficient in a complex noise environment and a low signal-to-noise ratio environment in an existing method are solved. According to the method, the voice segment and the noise segment in the signal are determined through the instantaneous energy, the voice signal after first enhancement is obtained through beam forming, the Fusion RNNoise network is suitable for noise segment signal enhancement, and the MCRA-OMLSA method is suitable for signal segment signal enhancement, so that the voice signal after first enhancement is obtained. Therefore, in combination with a FusionRNNoise network and an MCRA-OMLSA method, the voice signal after the first enhancement is enhanced for the second time, and a finally enhanced voice signal is obtained. The method can be applied to the field of multi-channel speech enhancement.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of multi-channel speech enhancement, and in particular relates to a multi-channel speech signal enhancement method. Background Art

[0002] Multi-channel speech enhancement technology has been widely used in voice communication, speech recognition and other fields. However, there are still many challenges in achieving efficient multi-channel speech enhancement. First, the problem of insufficient adaptability is particularly evident in complex acoustic environments. Noise characteristics and room impulse responses may change rapidly, and existing methods often find it difficult to adapt to these changes in a timely manner, resulting in unstable enhancement effects. For example, the performance of the MVDR method will significantly degrade when the noise covariance matrix is ​​estimated inaccurately. Second, existing methods have difficulties in the trade-off between speech quality and noise suppression, and it is difficult to effectively suppress background noise while maintaining the naturalness and intelligibility of speech. Excessive noise suppression may cause speech distortion, while insufficient noise suppression will affect speech quality. In addition, the multipath effect is particularly prominent in indoor environments. When speech signals reach the microphone array through multiple paths, reflections and reverberations will occur. These multipath effects will interfere with the performance of beamforming and reduce the effect of speech enhancement.

[0003] The amount of computing resources often limits the application of speech enhancement technology in large-scale speech information processing systems and embedded systems. Although large-scale speech information processing systems have a lot of computing resources, the focus is on the subsequent speech content analysis and recognition. The computing resources that can be used for speech enhancement in the preprocessing stage are limited. The processor, memory and energy supply of embedded platforms are all greatly limited. These factors limit the complexity and model size of the speech enhancement algorithms that can be used, and a trade-off needs to be made between performance and computing resource consumption. Secondly, the problem of insufficient generalization ability also needs attention. Many existing methods perform well in specific noise environments, but when faced with unseen noise types or acoustic environments, the performance may drop sharply. Finally, the spectrum estimation bias is particularly obvious under low signal-to-noise ratio conditions. Traditional methods such as spectral subtraction are prone to spectrum estimation bias, resulting in speech distortion or music noise.

[0004] In summary, it is difficult for existing methods to effectively suppress background noise and reduce the consumption of computing resources while ensuring the speech enhancement effect. The generalization ability is insufficient when facing complex noise environments and low signal-to-noise ratio environments. Therefore, a multi-channel speech enhancement method with low computing resource consumption, good background noise suppression effect, strong adaptability and good speech quality is proposed, which has important value and wide application scenarios. Summary of the invention

[0005] The purpose of the present invention is to propose a multi-channel speech signal enhancement method to solve the problems that existing methods are difficult to effectively suppress background noise and reduce the consumption of computing resources while ensuring the speech enhancement effect, and the generalization ability is insufficient when facing complex noise environments and low signal-to-noise ratio environments.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: a multi-channel speech signal enhancement method, the method specifically comprising the following steps:

[0007] Step 1: Frame the multi-channel speech signal collected by the microphone array, and then determine the speech segment and noise segment in the signal according to the instantaneous energy of each frame of the speech signal;

[0008] Step 2: Calculate the covariance matrix R of each channel speech signal xx , based on the covariance matrix R xx And the determination result of step 1 calculates the global noise covariance matrix;

[0009] Step 3: Calculate the weight matrix w according to the global noise covariance matrix, calculate the weight corresponding to each frequency point according to the weight matrix w, and normalize the weight vector composed of the weights corresponding to each frequency point to obtain a normalized weight vector;

[0010] The normalized weight vector is then sparsed to obtain a sparse weight vector, and beamforming is then performed based on the sparse weight vector to obtain a speech signal after the first enhancement;

[0011] Step 4: After Fourier transforming the speech signal after the first enhancement, a frequency domain signal corresponding to the speech signal after the first enhancement is obtained, and then according to the determination result in step 1, the speech short and noise segments in the frequency domain signal are obtained;

[0012] Step 5: Use the FusionRNNoise network to perform a second enhancement on the noise segment obtained in step 4, use the MCRA-OMLSA method to perform a second enhancement on the speech segment obtained in step 4, and use the noise segment and speech segment after the second enhancement to form the final enhanced speech signal.

[0013] Furthermore, in the step 1, the multi-channel speech signal collected by the microphone array is framed, and then the speech segment and the noise segment are determined according to the instantaneous energy of each frame of the speech signal; specifically:

[0014] Step 1: Perform Fourier transform on the speech signals of each channel to obtain the frequency domain signals corresponding to each channel;

[0015] Step 1 and 2, respectively, segment the frequency domain signal of each channel, use the first segment signal of each channel to form the first frame signal, use the second segment signal of each channel to form the second frame signal, and similarly, obtain each frame signal in sequence;

[0016] Step 13: Calculate the instantaneous energy of each frame signal respectively, and compare the instantaneous energy of each frame signal with the threshold;

[0017] If the instantaneous energy of the current frame signal is greater than or equal to the threshold, the current frame signal is a speech segment signal;

[0018] If the instantaneous energy of the current frame signal is less than the threshold, the current frame signal is a noise segment signal.

[0019] Furthermore, the instantaneous energy of each frame signal is calculated separately, specifically:

[0020] Take any frame signal as an example:

[0021]

[0022] Among them, E n Indicates the instantaneous energy of the signal of the nth channel contained in the current frame signal, x n (m) represents the value of the mth sampling point of the nth channel contained in the current frame signal, |·| represents the absolute value, and M represents the total number of sampling points in each channel of the current frame signal;

[0023] Then the instantaneous energy of the current frame signal is:

[0024]

[0025] Where N is the total number of channels of the microphone array to collect speech signals.

[0026] Furthermore, the covariance matrix R of the speech signals of each channel is xx for:

[0027]

[0028] Among them, Y n is the frequency domain signal corresponding to the nth channel, Y n H Yes n The transpose of .

[0029] Furthermore, the covariance matrix R xx The global noise covariance matrix is ​​calculated based on the judgment result of step 1, and an adaptive filtering method is used.

[0030] Furthermore, the weight matrix w is calculated according to the global noise covariance matrix, specifically:

[0031]

[0032] Among them, R nn is the global noise covariance matrix, the superscript -1 represents the inverse of the matrix, p s is the guiding vector.

[0033] Furthermore, the weight corresponding to each frequency point is calculated according to the weight matrix w, specifically:

[0034] Calculate the L1 norm of the i-th frequency point:

[0035]

[0036] Among them, w n,i is the element in the nth row and ith column of the weight matrix w, w n,i is a complex number, ||x i || is the L1 norm of the i-th frequency point;

[0037] The L1 norm of the i-th frequency point is used as the weight corresponding to the i-th frequency point.

[0038] Furthermore, the normalized weight vector is sparsely processed to obtain a sparse weight vector; specifically:

[0039] For the i-th element w in the normalized weight vector i ′, w i The coefficient of the real part of ′ is denoted as a, and w i The coefficient of the imaginary part of ′ is recorded as b, and then we calculate The value of

[0040] like The value of is greater than the regularization parameter λ, then The value of is multiplied by the reduction factor, and the multiplication result is used as the i-th element in the sparse weight vector. If If the value of is less than or equal to the regularization parameter λ, the i-th element in the sparsified weight vector is set to 0;

[0041] Similarly, each element in the normalized weight vector is processed separately to obtain a sparse weight vector.

[0042] Furthermore, the working process of the FusionRNNoise network is as follows:

[0043] Step 51: Use the full-band spectrum feature extraction module of the FusionRNNoise network to extract the full-band spectrum feature h0, and then extract the sub-band spectrum feature h1 based on the full-band spectrum feature h0 and the sub-band spectrum feature extraction module;

[0044] Step 52: Use the spatial fusion module to fuse the full-band spectrum feature h0 and the sub-band spectrum feature h1, and then pass the fusion result through the first GRU unit, the second GRU unit and the Linear layer in sequence, and output the enhanced signal through the Linear layer.

[0045] Furthermore, the working process of the spatial fusion module is as follows:

[0046] Perform an addition operation on the full-band spectrum feature h0 and the sub-band spectrum feature h1, use the addition result as the input of the MS-CAM unit, multiply the output of the MS-CAM unit with h0 to obtain the multiplication result a1, and multiply the output of the MS-CAM unit with h1 to obtain the multiplication result a2;

[0047] Then, a1 and a2 are added, and the addition result is used as the output of the spatial fusion module;

[0048] The working process of the MS-CAM unit is:

[0049] The input of the MS-CAM unit is input to two branches in parallel, and then the outputs of the two branches are added, the addition result is passed through the Sigmoid activation function, and the output of the Sigmoid activation function is multiplied with the input of the MS-CAM unit, and the multiplication result is used as the output of the MS-CAM unit; wherein:

[0050] The first branch includes the global average pooling layer, Point-wise Conv layer, Relu layer, Point-wise Conv layer and BN layer in sequence;

[0051] The second branch includes Point-wise Conv layer, Relu layer, Point-wise Conv layer and BN layer in sequence.

[0052] The beneficial effects of the present invention are:

[0053] The present invention first determines the speech segment and noise segment in the signal by instantaneous energy, and then obtains the speech signal after the first enhancement by beamforming. Since the FusionRNNoise network is suitable for noise segment signal enhancement, and the MCRA-OMLSA method is suitable for signal segment signal enhancement, the FusionRNNoise network and the MCRA-OMLSA method are combined to perform a second enhancement on the speech signal after the first enhancement, which can effectively suppress background noise while ensuring the speech enhancement effect, and has a strong generalization ability when facing complex noise environments and low signal-to-noise ratio environments. Moreover, the computational complexity of the method of the present invention is low, thereby reducing the consumption of computing resources, and is suitable for low-resource platforms. The method of the present invention can significantly improve speech quality and intelligibility, and at the same time has strong real-time performance, which can meet real-time processing requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a structural diagram of the FusionRNNoise network of the present invention;

[0055] Figure 2 A structural diagram of the MS-CAM unit used in the spatial fusion module of the present invention;

[0056] D×T×F, D×1×1, D / r×T×F and D / r×1×1 all represent the dimensions of the output. Taking D×T×F as an example, the first element D is the feature dimension, the second element T is the time dimension, and the third element F is the frequency dimension.

[0057] Figure 3 It is a structural diagram of the spatial fusion module of the present invention. DETAILED DESCRIPTION

[0058] Specific implementation method 1: A multi-channel speech signal enhancement method described in this implementation method specifically includes the following steps:

[0059] Step 1: Frame the multi-channel speech signal collected by the microphone array, and then determine the speech segment and noise segment in the signal according to the instantaneous energy of each frame of the speech signal;

[0060] Step 2: Calculate the covariance matrix R of each channel speech signal xx , based on the covariance matrix R xx And the determination result of step 1 calculates the global noise covariance matrix;

[0061] Step 3: Calculate the weight matrix w according to the global noise covariance matrix, calculate the weight corresponding to each frequency point according to the weight matrix w, and normalize the weight vector composed of the weights corresponding to each frequency point to obtain a normalized weight vector;

[0062] The normalized weight vector is then sparsed to obtain a sparse weight vector, and beamforming is then performed based on the sparse weight vector to obtain a speech signal after the first enhancement;

[0063] Step 4: After Fourier transforming the speech signal after the first enhancement, a frequency domain signal corresponding to the speech signal after the first enhancement is obtained, and then according to the determination result in step 1, the speech short and noise segments in the frequency domain signal are obtained;

[0064] Step 5: Use the FusionRNNoise network to perform a second enhancement on the noise segment obtained in step 4, use the MCRA-OMLSA method to perform a second enhancement on the speech segment obtained in step 4, and use the noise segment and speech segment after the second enhancement to form the final enhanced speech signal.

[0065] Since the suppression effect of the MCRA-OMLSA method is worse than that of deep learning under low signal-to-noise ratio conditions, and the deep learning method will cause the loss of some speech components, the FusionRNNoise network is used for denoising in frames with low probability of speech existence to increase the proportion of the deep learning method in the noise reduction signal. In frames with high probability of speech existence, the MCRA-OMLSA method is used for denoising. By fusing the MCRA-OMLSA method and the FusionRNNoise network, the noise suppression effect and speech quality can be balanced.

[0066] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that, in step 1, the multi-channel speech signal collected by the microphone array is framed, and then the speech segment and the noise segment are determined according to the instantaneous energy of each frame of the speech signal; specifically:

[0067] Step 1: Perform Fourier transform on the speech signals of each channel to obtain the frequency domain signals corresponding to each channel;

[0068] Step 1 and 2, respectively, segment the frequency domain signal of each channel, use the first segment signal of each channel to form the first frame signal, use the second segment signal of each channel to form the second frame signal, and similarly, obtain each frame signal in sequence;

[0069] When the frequency domain signal of each channel is processed in segments, the number of sampling points included in each segment of the signal is M, that is, the data from the 0th sampling point to the M-1th sampling point in the channel is taken as the first segment of the signal, and the data from the Mth sampling point to the 2M-1th sampling point in the channel is taken as the second segment of the signal, and so on, each channel is divided into segments of the signal;

[0070] Step 13: Calculate the instantaneous energy of each frame signal respectively, and compare the instantaneous energy of each frame signal with a threshold value (which can be set according to actual conditions);

[0071] If the instantaneous energy of the current frame signal is greater than or equal to the threshold, the current frame signal is a speech segment signal;

[0072] If the instantaneous energy of the current frame signal is less than the threshold, the current frame signal is a noise segment signal.

[0073] The other steps and parameters are the same as those in the first embodiment.

[0074] Specific implementation method three: This implementation method is different from specific implementation method one or two in that the instantaneous energy of each frame signal is calculated separately, specifically:

[0075] Take any frame signal as an example:

[0076]

[0077] Among them, E n Indicates the instantaneous energy of the signal of the nth channel contained in the current frame signal, x n (m) represents the value of the mth sampling point of the nth channel contained in the current frame signal, |·| represents the absolute value, and M represents the total number of sampling points in each channel of the current frame signal;

[0078] Then the instantaneous energy of the current frame signal is:

[0079]

[0080] Where N is the total number of channels of the microphone array to collect speech signals.

[0081] The other steps and parameters are the same as those in the first or second embodiment.

[0082] Specific implementation method 4: This implementation method is different from any one of the specific implementation methods 1 to 3 in that the covariance matrix R of the speech signal of each channel is xx for:

[0083]

[0084] Among them, Y n is the frequency domain signal corresponding to the nth channel, Y n H Yes n The transpose of .

[0085] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0086] Specific implementation method 5: This implementation method is different from the specific implementation methods 1 to 4 in that the covariance matrix R xx The global noise covariance matrix is ​​calculated based on the judgment result of step 1, and an adaptive filtering method (LMS) is adopted.

[0087] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0088] The adaptive filtering method can automatically adjust the parameters of the filter according to the characteristics of the input signal to adapt to the changing environment and signal conditions. Different iteration rates are set for noise segments and speech segments. In the first 100 iterations, the global covariance is updated for both noise segments and speech segments. After that, the covariance is only updated in the noise segment, and the global covariance is returned between speech segments. This method can improve the accuracy of noise covariance estimation and reduce the amount of covariance calculation.

[0089] Specific implementation method 6: This implementation method is different from any one of specific implementation methods 1 to 5 in that the weight matrix w is calculated according to the global noise covariance matrix, specifically:

[0090]

[0091] Among them, R nn is the global noise covariance matrix, the superscript -1 represents the inverse of the matrix, p s is the guiding vector.

[0092] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0093] Specific implementation method 7: This implementation method is different from any one of specific implementation methods 1 to 6 in that the weight corresponding to each frequency point is calculated according to the weight matrix w, specifically:

[0094] Calculate the L1 norm of the i-th frequency point:

[0095]

[0096] Among them, w n,i is the element in the nth row and ith column of the weight matrix w, w n,i is a complex number, ||x i || is the L1 norm of the i-th frequency point;

[0097] The L1 norm of the i-th frequency point is used as the weight corresponding to the i-th frequency point.

[0098] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0099] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that the normalized weight vector is sparsely processed to obtain a sparsely processed weight vector; specifically:

[0100] For the i-th element w in the normalized weight vector i ′, w i The coefficient of the real part of ′ is denoted as a, and w i The coefficient of the imaginary part of ′ is recorded as b, and then we calculate The value of

[0101] like The value of is greater than the regularization parameter λ (the specific value can be set according to the actual situation), then The value of is multiplied by the reduction factor (usually set to 0.01), and the multiplication result is used as the i-th element in the sparse weight vector. If the value of is less than or equal to the regularization parameter λ, the i-th element in the sparsified weight vector is set to 0;

[0102] Similarly, each element in the normalized weight vector is processed separately to obtain a sparse weight vector.

[0103] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0104] Specific implementation method nine: Combination Figure 1 This embodiment is different from the first to eighth embodiments in that the working process of the FusionRNNoise network is as follows:

[0105] Step 51: Use the full-band spectrum feature extraction module of the FusionRNNoise network to extract the full-band spectrum feature h0, and then extract the sub-band spectrum feature h1 based on the full-band spectrum feature h0 and the sub-band spectrum feature extraction module;

[0106] Step 52: Use the spatial fusion module to fuse the full-band spectrum feature h0 and the sub-band spectrum feature h1, and then pass the fusion result through the first GRU unit (gated recurrent unit), the second GRU unit and the Linear layer in sequence, and output the enhanced signal through the Linear layer.

[0107] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0108] The full-band spectrum features and sub-band spectrum features are extracted using existing feature extraction networks, such as convolutional neural networks such as CNN.

[0109] Specific implementation method ten: Combination Figure 2 and Figure 3This embodiment is different from the first to ninth embodiments in that the working process of the space fusion module is as follows:

[0110] Perform an addition operation on the full-band spectrum feature h0 and the sub-band spectrum feature h1, use the addition result as the input of the MS-CAM unit, multiply the output of the MS-CAM unit with h0 to obtain the multiplication result a1, and multiply the output of the MS-CAM unit with h1 to obtain the multiplication result a2;

[0111] Then, a1 and a2 are added, and the addition result is used as the output of the spatial fusion module;

[0112] The working process of the MS-CAM unit is:

[0113] The input of the MS-CAM unit is input to two branches in parallel, and then the outputs of the two branches are added, the addition result is passed through the Sigmoid activation function, and the output of the Sigmoid activation function is multiplied with the input of the MS-CAM unit, and the multiplication result is used as the output of the MS-CAM unit; wherein:

[0114] The first branch includes the global average pooling layer, Point-wise Conv layer, Relu layer, Point-wise Conv layer and BN layer in sequence;

[0115] The second branch includes Point-wise Conv layer, Relu layer, Point-wise Conv layer and BN layer in sequence.

[0116] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0117] The present invention has been successfully deployed in multiple low-resource platforms such as Raspberry Pi 4B, with good enhancement effect and real-time performance, and meets the usage restrictions of memory and CPU.

[0118] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A multi-channel speech signal enhancement method, characterized in that: The method specifically comprises the following steps: Step 1: Frame the multi-channel speech signal collected by the microphone array, and then determine the speech segment and noise segment in the signal according to the instantaneous energy of each frame of the speech signal; Step 2: Calculate the covariance matrix R of each channel speech signal xx , based on the covariance matrix R xx And the determination result of step 1 calculates the global noise covariance matrix; Step 3: Calculate the weight matrix w according to the global noise covariance matrix, calculate the weight corresponding to each frequency point according to the weight matrix w, and normalize the weight vector composed of the weights corresponding to each frequency point to obtain a normalized weight vector; The normalized weight vector is then sparsed to obtain a sparse weight vector, and beamforming is then performed based on the sparse weight vector to obtain a speech signal after the first enhancement; Step 4: After Fourier transforming the speech signal after the first enhancement, a frequency domain signal corresponding to the speech signal after the first enhancement is obtained, and then according to the determination result in step 1, the speech short and noise segments in the frequency domain signal are obtained; Step 5: Use the FusionRNNoise network to perform a second enhancement on the noise segment obtained in step 4, use the MCRA-OMLSA method to perform a second enhancement on the speech segment obtained in step 4, and use the noise segment and speech segment after the second enhancement to form the final enhanced speech signal.

2. A multi-channel speech signal enhancement method according to claim 1, characterized in that: In the step 1, the multi-channel speech signal collected by the microphone array is framed, and then the speech segment and the noise segment are determined according to the instantaneous energy of each frame of the speech signal; Specifically: Step 1: Perform Fourier transform on the speech signals of each channel to obtain the frequency domain signals corresponding to each channel; Step 1 and 2, respectively, segment the frequency domain signal of each channel, use the first segment signal of each channel to form the first frame signal, use the second segment signal of each channel to form the second frame signal, and similarly, obtain each frame signal in sequence; Step 13: Calculate the instantaneous energy of each frame signal respectively, and compare the instantaneous energy of each frame signal with the threshold; If the instantaneous energy of the current frame signal is greater than or equal to the threshold, the current frame signal is a speech segment signal; If the instantaneous energy of the current frame signal is less than the threshold, the current frame signal is a noise segment signal.

3. A multi-channel speech signal enhancement method according to claim 2, characterized in that: The instantaneous energy of each frame signal is calculated separately, specifically: Take any frame signal as an example: Among them, E n Indicates the instantaneous energy of the signal of the nth channel contained in the current frame signal, x n (m) represents the value of the mth sampling point of the nth channel contained in the current frame signal, |·| represents the absolute value, and M represents the total number of sampling points in each channel of the current frame signal; Then the instantaneous energy of the current frame signal is: Where N is the total number of channels of the microphone array to collect speech signals.

4. A multi-channel speech signal enhancement method according to claim 2, characterized in that: The covariance matrix R of the speech signals of each channel xx for: Among them, Y n is the frequency domain signal corresponding to the nth channel, Yes n The transpose of .

5. The multi-channel speech signal enhancement method according to claim 1, characterized in that: The covariance matrix R xx The global noise covariance matrix is ​​calculated based on the judgment result of step 1, and an adaptive filtering method is adopted.

6. A multi-channel speech signal enhancement method according to claim 1, characterized in that: The weight matrix w is calculated according to the global noise covariance matrix, specifically: Among them, R nn is the global noise covariance matrix, the superscript -1 represents the inverse of the matrix, p s is the guiding vector.

7. A multi-channel speech signal enhancement method according to claim 6, characterized in that: The weight corresponding to each frequency point is calculated according to the weight matrix w, specifically: Calculate the L1 norm of the i-th frequency point: Among them, w n,i is the element in the nth row and ith column of the weight matrix w, w n,i is a complex number, ||x i || is the L1 norm of the i-th frequency point; The L1 norm of the i-th frequency point is used as the weight corresponding to the i-th frequency point.

8. A multi-channel speech signal enhancement method according to claim 7, characterized in that: The normalized weight vector is sparsely processed to obtain a sparse weight vector; specifically: For the i-th element w in the normalized weight vector i ′, w i The coefficient of the real part of ′ is denoted as a, and w i The coefficient of the imaginary part of ′ is recorded as b, and then we calculate The value of like The value of is greater than the regularization parameter λ, then The value of is multiplied by the reduction factor, and the multiplication result is used as the i-th element in the sparse weight vector. If If the value of is less than or equal to the regularization parameter λ, the i-th element in the sparsified weight vector is set to 0; Similarly, each element in the normalized weight vector is processed separately to obtain a sparse weight vector.

9. The multi-channel speech signal enhancement method according to claim 1, characterized in that: The working process of the FusionRNNoise network is as follows: Step 51: Use the full-band spectrum feature extraction module of the FusionRNNoise network to extract the full-band spectrum feature h0, and then extract the sub-band spectrum feature h1 based on the full-band spectrum feature h0 and the sub-band spectrum feature extraction module; Step 52: Use the spatial fusion module to fuse the full-band spectrum feature h0 and the sub-band spectrum feature h1, and then pass the fusion result through the first GRU unit, the second GRU unit and the Linear layer in sequence, and output the enhanced signal through the Linear layer.

10. A multi-channel speech signal enhancement method according to claim 9, characterized in that: The working process of the spatial fusion module is as follows: Perform an addition operation on the full-band spectrum feature h0 and the sub-band spectrum feature h1, use the addition result as the input of the MS-CAM unit, multiply the output of the MS-CAM unit with h0 to obtain the multiplication result a1, and multiply the output of the MS-CAM unit with h1 to obtain the multiplication result a2; Then, a1 and a2 are added, and the addition result is used as the output of the spatial fusion module; The working process of the MS-CAM unit is: The input of the MS-CAM unit is input to two branches in parallel, and then the outputs of the two branches are added, the addition result is passed through the Sigmoid activation function, and the output of the Sigmoid activation function is multiplied with the input of the MS-CAM unit, and the multiplication result is used as the output of the MS-CAM unit; wherein: The first branch includes the global average pooling layer, Point-wise Conv layer, Relu layer, Point-wiseConv layer and BN layer in sequence; The second branch includes Point-wise Conv layer, Relu layer, Point-wise Conv layer and BN layer in sequence.

Citation Information

Cited By

  • Multi-channel digital audio signal processing method, equipment and medium

    CN119811352A

  • Speech enhancement method and device based on full-band and sub-band fusion, terminal and medium

    CN121054024A

  • Speech enhancement method and device based on full-band and sub-band fusion, terminal and medium

    CN121054024B