Multi-channel speech enhancement method based on multi-scale spatial information and spectrum feature fusion

Abstract

The invention discloses a multi-channel speech enhancement method based on multi-scale spatial information and spectrum feature fusion, which comprises the following steps: recombining different spectrum components according to spectrum features, and extracting in-channel, inter-channel and full-channel feature modes; the features are fused, and unified depth features are created; a local feature extraction module is introduced to enhance the feature weight of the current frame, and features under different scales are fused through a feature attention mechanism; a decomposition attention mechanism is introduced, and encoder and decoder outputs are fused in a plurality of decomposition spaces, so that detail features can also be used by a depth module. According to the method, spatial features and spectrum features are combined, and unified feature expression is created by using a feature fusion method. The attention module adaptively learns modes contained in the spatial features and utilizes the modes instead of fitting directional features with physical meanings, so that the method can adapt to different scenes more flexibly, and has a good application prospect.

Description

Technical Field

[0001] The invention relates to a multi-channel speech enhancement method based on the fusion of multi-scale spatial information and spectrum features, and belongs to the technical field of speech enhancement. Background Art

[0002] Speech enhancement aims to improve the accuracy and quality of noise-contaminated speech and is an indispensable and important component in many applications such as hearing aids, smart terminal devices, and video conferencing systems. In these applications, speech is inevitably affected by environmental noise, room reverberation, and other factors. In addition, with the development of automatic speech recognition and large language models, the importance of speech enhancement as a front-end basic module has also steadily increased, because high-quality speech signals are a prerequisite for accurate prediction in these applications. According to the number of recording microphones, speech enhancement can be divided into two categories: single-channel speech enhancement and multi-channel speech enhancement. Single-channel speech enhancement mainly extracts clear speech from a single noisy speech channel to remove background noise. Limited by the number of microphones, single-channel speech enhancement can only use spectral information, which limits its performance to a certain extent. Due to the lack of spatial information, single-channel models often have difficulty in effectively distinguishing the target speech from background noise, especially in complex acoustic environments. Multi-channel speech enhancement can use the spatial information captured from multiple microphone recordings, and multi-channel speech enhancement performs better than single-channel speech enhancement in terms of speech clarity and quality. Multi-channel speech enhancement uses spatial filtering to enhance speech signals from a specific direction while suppressing noise from other directions, which can better separate speech from noise, especially in complex environments. Nevertheless, classical spatial filtering also has many limitations. Classical spatial filtering is usually based on linear operations and cannot effectively handle nonlinear features or complex feature patterns. In some cases, linear filtering may lead to the loss of important information. At the same time, different filters and parameter settings may lead to completely different effects. How to choose the appropriate parameters often requires experience and trial and error, which may increase the complexity of processing.

[0003] Recently, with the significant progress made by deep learning technology in the field of single-channel speech enhancement, some researchers have proposed methods that combine deep neural networks with traditional beamforming, which are called neural beamforming methods. The core idea driving these methods is to replace traditional algorithms with deep neural network models to estimate components used in beamformers or subsequent filtering techniques, such as the posterior probability of speech presence, interaural phase difference, and the amplitude spectrum of interfering microphone signals. Neural beamformers generally have lower complexity and are also improvements over traditional methods; however, challenges such as linearization problems and the ability to handle non-stationary noise remain to be overcome. In addition, when combined with deep neural networks for training, matrix operations such as inversion of matrices sometimes become numerically unstable. For this reason, more researchers have begun to focus on data-driven deep learning methods. Data-driven deep learning methods can make full use of the model's nonlinear fitting and time series modeling capabilities and are more suitable for multi-channel speech enhancement. The channel-attention dense CA Dense U-net model implements beamforming processing by introducing a channel attention mechanism, allowing the model to perform nonlinear beamforming. The dense frequency-time attention network DeFT-AN integrates three different types of modules to process feature patterns in spatial, frequency, and time domain dimensions respectively. The SpatialNet model processes time-frequency features independently. These methods show that how to process spatial information is crucial to distinguishing speech from noise, because speech is usually directional and spatially coherent, while noise is usually diffuse or has low spatial correlation. These methods use independent modules to process spatial and spectral information without fusing the two. Although this design effectively extracts each feature, it may limit the flexibility and overall performance of the model when processing complex signals. Without effective integration, the model may not be able to fully utilize the relationship between space and spectrum, thus affecting the effect of multi-channel speech enhancement. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a multi-channel speech enhancement method based on the fusion of multi-scale spatial information and spectral features. By extracting and fusing the spatial information and spectral features of different channel combinations, the advancedness of the single-channel speech enhancement model is extended to the multi-channel field, thereby better separating the target speech from the background noise.

[0005] The present invention adopts the following technical solutions to solve the above technical problems:

[0006] The multi-channel speech enhancement method based on the fusion of multi-scale spatial information and spectral features includes the following steps:

[0007] Step 1: for a multi-channel microphone noisy spectrum, extract spatial features within the channel, between channels and all channels respectively by using a grouped convolution method, wherein the spatial features include spatial information and spectral features;

[0008] Step 2: A hierarchical feature fusion strategy is used to fuse the intra-channel, inter-channel, and full-channel spatial features extracted in step 1 to obtain a unified deep feature expression;

[0009] Step 3: construct a multi-scale spatial aggregation encoder and a multi-scale decoder, wherein a recurrent neural network is connected between the multi-scale spatial aggregation encoder and the multi-scale decoder, and the unified deep feature expression obtained in step 2 is sent to the multi-scale spatial aggregation encoder for encoding and feature mapping; the output of the multi-scale spatial aggregation encoder is sent to the recurrent neural network to extract feature patterns with frames as time steps, and perform noise suppression and speech enhancement; the output of the recurrent neural network is sent to the multi-scale decoder for signal recovery and reconstruction;

[0010] Step 4: Fuse the output of the multi-scale spatial aggregation encoder and the output of the multi-scale decoder, and enlarge the fused deep features and map them to the spectrum domain to obtain an enhanced speech signal spectrum.

[0011] As a preferred embodiment of the present invention, the specific process of step 1 is as follows:

[0012] Step 1.1: For the microphone noise spectrum of M channels, extract the spatial features in each channel respectively. Each channel corresponds to a set of convolutions. Then the spatial features X of M channels are inner It is expressed as:

[0013]

[0014] Among them, δ and Respectively represent the PReLU activation function and group normalization operation; X (m) (t,k) represents the spatial features of the mth channel, represents a real number, T and F represent the number of frames and the number of frequency points respectively; Φ m represents the parameters of the convolution kernel of the mth channel, m = 1, ..., M, * represents the convolution operation; (t, k) represents the index of the frame and frequency point, t = 1, ..., T, k = 1, ..., F;

[0015] Step 1.2, for the microphone noisy spectrum of the M channel, combine the real parts of all channel spectra together, and combine the imaginary parts of the channel spectra together, respectively, and extract spatial features from the real and imaginary parts. Each combination corresponds to a set of convolutions, expressed as:

[0016]

[0017] Among them, X outer (r / i) represents the spatial characteristics of the combination of the real and imaginary parts of the M channel,

[0018]

[0019] Θ r / i Represents the convolution kernel parameters corresponding to the real and imaginary combined convolution, Respectively represent the real part of the 1st and Mth channel spectra, Respectively represent the imaginary parts of the 1st and Mth channel spectra;

[0020] Step 1.3: For the microphone noisy spectrum of the M channel, a set of convolutions is used to extract spatial features. The convolution kernel receptive field includes all microphone channels, and the full channel spatial feature X is obtained. full .

[0021] As a preferred solution of the present invention, the specific process of step 2 is as follows:

[0022] Step 2.1: Use the feature attention fusion method to fuse the spatial features within and between channels to obtain the fused features; specifically:

[0023] Add the intra-channel spatial features and inter-channel spatial features point-to-point:

[0024]

[0025] Among them, Z is the feature after point-to-point addition, and C represents the number of channels of the intermediate feature;

[0026] Calculate the soft selection fusion weight W:

[0027]

[0028] Where σ represents the Sigmoid operator, σ(x) = 1 / 1 + e -x , operator Indicates that the global feature expression is calculated along the subband, represents point-to-point addition, PWC represents point-to-point convolution with a convolution kernel size of (1,1), Represents the normalization operation of the layer, θ1 and θ2 represent The corresponding parameters of the first and second layers of PWC, φ1 and φ2 represent The corresponding parameters of the first and second layers of PWC;

[0029] According to the soft selection fusion weight W, the spatial features within and between channels are fused:

[0030] X1=W⊙X inner +(IW)⊙X outer

[0031] Among them, X1 represents the fused feature, ⊙ represents the point-to-point multiplication operation, and I represents the all-one vector;

[0032] Step 2.2, based on the time-domain attention mechanism, the fused features obtained in step 2.1 are fused with the full-channel spatial features to obtain a unified deep feature expression; specifically:

[0033] Using each frequency component in X1 as the key value, X full The frequency components in all channels are used as query values ​​and action values ​​to generate a spatially aggregated attention map across frames and channels;

[0034] For frequency point k, calculate the key value of attention weight The query value is pass Calculate the weight matrix A of frequency point k k ,in, The masking matrix combines the weight matrices of all frequency points And act on The spatial information and spectral features are aggregated to obtain a unified deep feature expression, that is,

[0035]

[0036] Here, @ represents matrix multiplication operation.

[0037] As a preferred solution of the present invention, the specific process of step 3 is as follows:

[0038] Step 3.1, construct a multi-scale spatial aggregation encoder, including several layers of multi-scale spatial aggregation modules. The input X1 of the first layer of multi-scale spatial aggregation module is the unified deep feature expression obtained in step 2, and the input X i is the cumulative combination of the regression outputs of all previous layers, expressed as in, They represent the output results of the i-1th and 1st-layer multi-scale spatial aggregation modules respectively;

[0039] The i-th layer multi-scale spatial aggregation module takes the input X i Perform dilated convolution operation to obtain the global feature X global ; At the same time, the i-th layer multi-scale spatial aggregation module inputs X i Perform convolution operation to obtain local features X local ;

[0040]

[0041] The feature attention fusion method is used to global and X local After fusion, we get:

[0042]

[0043] Two cascaded conformer modules are used to aggregate spatial information and spectral features from the time dimension and spectrum dimension respectively. That is, the time domain dimension conformer module is used to Reshape format, and calculate the attention weight of each frequency component in the time dimension The frequency domain dimension conformer module is used to Reshape format, and calculate the attention weights based on the frequency component correlation matrix Both the time domain and frequency domain dimension conformer modules apply the calculated attention weights to Thus, the output of the i-th layer multi-scale spatial aggregation module is obtained

[0044] Step 3.2, construct a multi-scale decoder, including several layers of multi-scale decoding modules. The input of the first layer of multi-scale decoding module is the output Y1 of the recurrent neural network, and the input Y i is the cumulative combination of the regression outputs of all previous layers, expressed as in, Respectively represent the output results of the i-1th and 1st layer multi-scale decoding modules;

[0045] The i-th layer multi-scale decoding module takes input Y i Perform dilated convolution operation to obtain the global feature Y global ; At the same time, the i-th layer multi-scale decoding module inputs Y i Perform convolution operation to obtain local feature Y local ;

[0046]

[0047] The feature attention fusion method is used to global and Y local Fusion is performed to obtain the output of the i-th layer multi-scale decoding module

[0048]

[0049] As a preferred solution of the present invention, the specific process of step 4 is as follows:

[0050] Step 4.1, use parallel linear transformation to map the output features of the multi-scale spatial aggregation encoder to H factorized subspaces, and the h-th subspace feature It is expressed as:

[0051]

[0052] Among them, h∈[1,2,…,H], C is the dimension of each subspace, that is, the dimension of the channel, X e represents the output features of the multi-scale spatial aggregation encoder, W e Represents the feature transformation matrix corresponding to the multi-scale spatial aggregation encoder;

[0053] Step 4.2, map the output features of the multi-scale decoder to the same subspace as the multi-scale spatial aggregation encoder, and use the normalized exponential function operator softmax to calculate the weight of each subspace It is expressed as:

[0054]

[0055] Among them, X d represents the output features of the multi-scale decoder, W d Represents the feature transformation matrix corresponding to the multi-scale decoder;

[0056] Step 4.3, the weights obtained by the multi-scale decoder in different subspaces are applied to the spatial features corresponding to the space, and the sum is obtained to obtain the enhanced speech signal features, which is expressed as:

[0057]

[0058] Among them, X ffa Represents the enhanced speech signal features.

[0059] A computer device comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the steps of the multi-channel speech enhancement method based on the fusion of multi-scale spatial information and spectral features are implemented.

[0060] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the multi-channel speech enhancement method based on the fusion of multi-scale spatial information and spectral features.

[0061] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0062] 1. The method designed by the present invention aims to capture spatial information and spectral features of different scales at the same time, and enhances the processing ability of multi-channel speech signals through an effective fusion mechanism. This design not only improves the performance of the model in complex environments, but also better adapts to the directionality and coherence of speech, thereby achieving a more superior speech enhancement effect.

[0063] 2. The present invention fully extracts and aggregates implicit spatial information across channels, frequencies and frames, and captures the cross-frame dynamic characteristics of spectral features, and aggregates spectral and spatial information based on dynamic features to create a unified deep feature expression.

[0064] 3. The present invention constructs a multi-scale spatial aggregation encoder and a multi-scale decoder, and enhances the weight of the current frame spectrum through local processing and feature attention fusion methods, thereby alleviating the problem of weakening of local features in the long-term receptive field, and further integrates the deep feature expressions of different time scales by utilizing the inherent relationship within the spectrum.

[0065] 4. The present invention connects the outputs of the encoder and decoder based on the decomposed attention skip connection module, which solves the problem of continuous loss of details of features in the forward process of the model and further improves the overall performance and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a flow chart of a multi-channel speech enhancement method based on the fusion of multi-scale spatial information and spectral features of the present invention;

[0067] Figure 2 It is a network structure diagram of the multi-channel speech enhancement model of the present invention;

[0068] Figure 3 1 is a comparison diagram of a noisy speech spectrum and a speech spectrum enhanced by the method of the present invention; wherein (a) is a noisy speech spectrum, (b) is a pure speech spectrum, and (c) is a speech spectrum enhanced by the method of the present invention. DETAILED DESCRIPTION

[0069] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0070] like Figure 1 As shown, the multi-channel speech enhancement method based on the fusion of multi-scale spatial information and spectral features of the present invention comprises the following steps:

[0071] (1) The input multi-channel noisy speech spectrum is sent to a multi-channel spatial spectrum feature extractor to extract spatial information from the intra-channel, inter-channel and all-channel receptive fields respectively;

[0072] (2) The three outputs extracted in step (1) are fed into a multi-layer spatial fusion module, which is divided into two stages of feature fusion, namely, feature attention fusion and T-conformer multi-head attention fusion:

[0073] 1) The feature attention module receives deep features of spatial information from within and between channels, calculates soft selection weights based on their global and local information, and fuses two paths of spatial information from different receptive fields while ensuring the dynamic characteristics of the features.

[0074] 2) Taking the output of the feature attention module and the full-channel features as input, the conformer module is used, the output of the feature attention module is used as the key, the full-channel features are used as the query and value, and the causal mask is introduced to aggregate the spatial information and spectral features to generate a unified deep feature expression.

[0075] (3) The unified deep feature expression generated by the multi-layer spatial fusion module is sent to the multi-scale spatial aggregation encoder for encoding and feature mapping;

[0076] (4) The output of the encoder is fed into a recurrent neural network to extract feature patterns with frames as time steps, and perform noise suppression and extraction;

[0077] (5) Send the output of the recurrent neural network to the multi-scale decoder for signal recovery and reconstruction;

[0078] (6) The outputs of the encoder and decoder are fused using decomposed attention, the lost details are introduced, the deep features are enlarged and mapped to the spectral domain, the signal features are reconstructed, and the enhanced speech signal spectrum is output.

[0079] Figure 2 It is a network structure diagram of the multi-channel speech model of the present invention, which is mainly described as follows:

[0080] In step (A), a multi-channel spatial-spectral feature extractor is constructed to extract spatial information from different channel combinations, thereby capturing spatial patterns across channels.

[0081] In a multi-channel speech spectrum, spatial information is hidden between channels and between spectra. In the present invention, group convolution is used to extract spatial patterns within channels, between channels, and across channels. The specific steps are as follows:

[0082] (A1) For intra-channel spatial information, assuming there are M-channel microphone inputs, the spectral characteristics of the m-th channel can be expressed as The extraction process can be expressed as:

[0083]

[0084] Among them, δ and Represent the PReLU activation function and group normalization operation respectively; Φ m represents the parameters of the convolution kernel of the mth channel. * represents the convolution operation. And t and k represent the index of the frame and frequency point.

[0085] (A2) For inter-channel spatial information, the real and imaginary parts of all microphone spectra are first combined into two groups, namely Then two different sets of convolution modules are used to extract spatial features, which can be expressed as:

[0086]

[0087] Among them, Θ r / i Represents the convolution kernel parameters corresponding to the two sets of convolutions.

[0088] (A3) For the full channel spatial information X full , calculated in a similar way to the intra-channel spatial information extraction, except that in this stage, there is only one set of convolution operations, and the convolution kernel receptive field includes all microphone channels.

[0089] Step (B) constructs a spatial feature fusion strategy based on multi-channel spatial spectrum feature output, adopts a hierarchical fusion mechanism, gradually integrates spatial information and spectrum features, and creates a unified deep feature expression.

[0090] This paper adopts a hierarchical feature fusion strategy to reduce the complexity of learning. The fusion process is divided into two steps. First, the feature attention method is used to merge the inter-channel and intra-channel features. Then, a narrowband processing module based on the conformer structure is proposed to integrate it with the full channel output. The specific steps are as follows:

[0091] (B1) The feature attention fusion method is used to fuse the intra-channel and inter-channel spatial information. These features contain spatial patterns of different scales. In order to allow the fused features to retain the original numerical dynamic range and distribution characteristics, the present invention adopts a soft selection method. Given that the feature after point-to-point addition is Then the soft selection fusion weight can be expressed as:

[0092]

[0093] Where σ represents the Sigmoid operator, σ(x) = 1 / 1 + e -x , operator Indicates that the global feature expression is calculated along the subband; and the module with θ / φ as parameter It can be represented as two layers of cascaded convolution:

[0094]

[0095] Among them, PWC is a point-to-point convolution with a convolution kernel size of (1,1). Indicates layer standardization. The parameter subscript indicates the number of layers, such as one of the modules The parameter is represented by θ, θ1 and θ2 represent the parameters of the first and second layers of PWC respectively. After calculating the soft selection weights, the overall fusion process can be expressed as:

[0096] X1=W⊙X inner +(IW)⊙X outer

[0097] Among them, ⊙ represents a point-to-point multiplication operation, and I represents an all-one vector.

[0098] (B2) Use the conformer attention module to further integrate the output X1 of step (B1) and the full channel feature X full Fusion. Use narrowband to capture the dynamic characteristics of each frequency component, using each frequency component in X1 as the key value, X full The frequency components in all channels are used as query values ​​and action values ​​to generate a spatial aggregation attention map across frames and channels, and the two inputs are aggregated. For frequency point k, the key value for calculating the attention weight can be expressed as The query value is pass Calculate the weight matrix of frequency point k, where is a masking matrix used to control the causal characteristics of the model, and C represents the number of channels of the intermediate features. The feature matrices of all frequency points are combined And act on The spatial information and spectral features are aggregated to obtain a unified feature expression, namely:

[0099]

[0100] Here, @ represents matrix multiplication operation.

[0101] Step (C) constructs a multi-scale spatial aggregation encoder and a multi-scale decoder, introduces local convolution based on the densely connected convolutional network, enhances the resolution of the current frame spectrum feature, and thus constructs a multi-scale codec. In order to introduce the spatial aggregation strategy in the multi-scale encoder, the present invention first uses the feature attention mechanism to fuse the feature expressions at different time scales, and then fuses the spatial features in the channel dimension through narrowband and full-band methods. The specific steps are as follows:

[0102] (C1) In each dilated convolution step of the dense convolution encoder, a PWC convolution is added as a local feature processing, and the deep feature expressions of different scales are fused through the feature attention mechanism, namely, the cross-frame global features and the intra-frame cross-band spectral pattern features. Given the input feature The input of the i-th layer is the cumulative combination of the outputs of all previous regressions, expressed as in represents the output result of the feature attention fusion module at the i-1th layer, which integrates the global feature X global and local features X local , X global and X local The extraction process can be expressed as:

[0103]

[0104] (C2) Use the feature attention fusion method proposed in step (B1) to fuse the global feature X global and local features X local , which can be expressed as in This is the output of the feature attention fusion module of the i-th layer of the multi-scale codec. If there are subsequent iterations, it will be combined with the input It is fed into the i+1th layer, otherwise it is used as the output of the multi-scale encoder and decoder.

[0105] (C3) In order to aggregate spatial information and spectral features in the encoding stage, the output of each layer of the encoder Add a post-processing module, namely the time-domain conformer, which uses two cascaded conformer modules to aggregate spatial information and spectral features from the time dimension and spectral dimension respectively. The time-domain dimension conformer block reshapes the input representation into And calculate the attention weight of each frequency component in the time dimension In contrast, the frequency domain dimension conformer reshapes the input into the format And calculate the attention weight according to the frequency component correlation matrix Both T-conformer and F-conformer apply the calculated attention weights to the input features to obtain a unified deep feature expression after fusion.

[0106] Step (D) constructs a decomposed attention fusion connection module to connect the multi-scale spatial aggregation encoder and the multi-scale decoder to solve the problem of continuous loss of details when features are transmitted in the model.

[0107] (D1) introduces the factored attention method to integrate the deep representations of the encoder and decoder. Given the encoder output X e and decoder output X d , the present invention uses parallel linear transformation to map the output features of the encoder to H factorized subspaces, and the hth subspace feature can be expressed as:

[0108]

[0109] Among them, h∈[1,2,…,H], C is the dimension of each subspace, that is, the dimension of the channel. e Represents the feature conversion matrix corresponding to the encoder, which is the variable learned by the model.

[0110] (D2) The decoder output feature X d Mapped to the same subspace as the encoder, and use the normalized exponential function operator softmax to calculate the weight of each subspace, expressed as:

[0111]

[0112] Among them, W d Represents the feature conversion matrix corresponding to the decoder, which is the variable learned by the model.

[0113] (D3) The weights obtained by the decoder in different subspaces are applied to the decoder features corresponding to the space, and the sum is obtained to obtain the fusion feature. The process can be expressed as:

[0114]

[0115] Table 1 shows the advancement of the proposed method on the 4-channel L3DAS dataset. From the performance indicators, the proposed multi-channel speech enhancement method based on the fusion of multi-scale spatial information and spectral features has significantly improved multiple speech quality perception indicators and is superior to the currently popular multi-channel speech enhancement methods.

[0116] Table 1 Algorithm performance comparison

[0117] Model Parameter magnitude M Complexity FLOPS / G PESQ STOI WER(%) Unproc - - 1.21 0.62 34.10 Baseline 5.52 32.15 1.63 0.86 19.18 SEUSpeech 2.18 16.57 2.62 0.92 5.44 DeFT-AN 2.67 37.99 3.09 0.95 4.95 McNet 1.85 27.96 2.74 0.94 4.01 Model of the present invention 3.10 29.75 3.37 0.96 3.63

[0118] Figure 3 1 is a comparison diagram of a noisy speech spectrum and a speech spectrum enhanced by the method of the present invention; wherein (a) is a noisy speech spectrum, (b) is a pure speech spectrum, and (c) is a speech spectrum enhanced by the method of the present invention.

[0119] Based on the same inventive concept, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the aforementioned multi-channel speech enhancement method based on the fusion of multi-scale spatial information and spectral features are implemented.

[0120] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the aforementioned multi-channel speech enhancement method based on the fusion of multi-scale spatial information and spectral features.

[0121] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0125] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A multi-channel speech enhancement method based on the fusion of multi-scale spatial information and spectral features, characterized in that: The steps include: Step 1: for a multi-channel microphone noisy spectrum, extract spatial features within the channel, between channels and all channels respectively by using a grouped convolution method, wherein the spatial features include spatial information and spectral features; Step 2: A hierarchical feature fusion strategy is used to fuse the intra-channel, inter-channel, and full-channel spatial features extracted in step 1 to obtain a unified deep feature expression; Step 3: construct a multi-scale spatial aggregation encoder and a multi-scale decoder, wherein a recurrent neural network is connected between the multi-scale spatial aggregation encoder and the multi-scale decoder, and the unified deep feature expression obtained in step 2 is sent to the multi-scale spatial aggregation encoder for encoding and feature mapping; the output of the multi-scale spatial aggregation encoder is sent to the recurrent neural network to extract feature patterns with frames as time steps, and perform noise suppression and speech enhancement; The output of the recurrent neural network is fed into a multi-scale decoder for signal recovery and reconstruction; Step 4: Fuse the output of the multi-scale spatial aggregation encoder and the output of the multi-scale decoder, and enlarge the fused deep features and map them to the spectrum domain to obtain an enhanced speech signal spectrum.

2. The multi-channel speech enhancement method based on the fusion of multi-scale spatial information and spectral features according to claim 1 is characterized in that: The specific process of step 1 is as follows: Step 1.1: For the microphone noise spectrum of M channels, extract the spatial features in each channel respectively. Each channel corresponds to a set of convolutions. Then the spatial features X of M channels are inner It is expressed as: Among them, δ and Respectively represent the PReLU activation function and group normalization operation; X (m) (t,k) represents the spatial features of the mth channel, represents a real number, T and F represent the number of frames and the number of frequency points respectively; Φ m represents the parameters of the convolution kernel of the mth channel, m = 1, ..., M, * represents the convolution operation; (t, k) represents the index of the frame and frequency point, t = 1, ..., T, k = 1, ..., F; Step 1.2, for the microphone noisy spectrum of the M channel, combine the real parts of all channel spectra together, and combine the imaginary parts of the channel spectra together, respectively, and extract spatial features from the real and imaginary parts. Each combination corresponds to a set of convolutions, expressed as: Among them, X outer (r / i) represents the spatial characteristics of the combination of the real and imaginary parts of the M channel, Θ r / i Represents the convolution kernel parameters corresponding to the real and imaginary combined convolution, Respectively represent the real part of the 1st and Mth channel spectra, Respectively represent the imaginary parts of the 1st and Mth channel spectra; Step 1.3: For the microphone noisy spectrum of the M channel, a set of convolutions is used to extract spatial features. The convolution kernel receptive field includes all microphone channels, and the full channel spatial feature X is obtained. full .

3. The multi-channel speech enhancement method based on the fusion of multi-scale spatial information and spectral features according to claim 2 is characterized in that: The specific process of step 2 is as follows: Step 2.1: Use the feature attention fusion method to fuse the spatial features within and between channels to obtain the fused features; specifically: Add the intra-channel spatial features and inter-channel spatial features point-to-point: Among them, Z is the feature after point-to-point addition, and C represents the number of channels of the intermediate feature; Calculate the soft selection fusion weight W: Where σ represents the Sigmoid operator, σ(x) = 1 / 1 + e -x , operator Indicates that the global feature expression is calculated along the subband, represents point-to-point addition, PWC represents point-to-point convolution with a convolution kernel size of (1,1), Represents the normalization operation of the layer, θ1 and θ2 represent The corresponding parameters of the first and second layers of PWC, φ1 and φ2 represent The corresponding parameters of the first and second layers of PWC; According to the soft selection fusion weight W, the spatial features within and between channels are fused: X1=W⊙X inner +(IW)⊙X outer Among them, X1 represents the fused feature, ⊙ represents the point-to-point multiplication operation, and I represents the all-one vector; Step 2.2, based on the time-domain attention mechanism, the fused features obtained in step 2.1 are fused with the full-channel spatial features to obtain a unified deep feature expression; specifically: Using each frequency component in X1 as the key value, X full The frequency components in all channels are used as query values ​​and action values ​​to generate a spatially aggregated attention map across frames and channels; For frequency point k, calculate the key value of attention weight The query value is pass Calculate the weight matrix A of frequency point k k ,in, The masking matrix combines the weight matrices of all frequency points And act on The spatial information and spectral features are aggregated to obtain a unified deep feature expression, that is, Here, @ represents matrix multiplication operation.

4. The multi-channel speech enhancement method based on the fusion of multi-scale spatial information and spectral features according to claim 3 is characterized in that: The specific process of step 3 is as follows: Step 3.1, construct a multi-scale spatial aggregation encoder, including several layers of multi-scale spatial aggregation modules. The input X1 of the first layer of multi-scale spatial aggregation module is the unified deep feature expression obtained in step 2, and the input X i is the cumulative combination of the regression outputs of all previous layers, expressed as in, They represent the output results of the i-1th and 1st-layer multi-scale spatial aggregation modules respectively; The i-th layer multi-scale spatial aggregation module takes the input X i Perform dilated convolution operation to obtain the global feature X global ; At the same time, the i-th layer multi-scale spatial aggregation module inputs X i Perform convolution operation to obtain local features X local ; The feature attention fusion method is used to global and X local After fusion, we get: Two cascaded conformer modules are used to aggregate spatial information and spectral features from the time dimension and spectrum dimension respectively. That is, the time domain dimension conformer module is used to Reshape format, and calculate the attention weight of each frequency component in the time dimension The frequency domain dimension conformer module is used to Reshape format, and calculate the attention weights based on the frequency component correlation matrix Both the time domain and frequency domain dimension conformer modules apply the calculated attention weights to Thus, the output of the i-th layer multi-scale spatial aggregation module is obtained Step 3.2, construct a multi-scale decoder, including several layers of multi-scale decoding modules. The input of the first layer of multi-scale decoding module is the output Y1 of the recurrent neural network, and the input Y i is the cumulative combination of the regression outputs of all previous layers, expressed as in, Respectively represent the output results of the i-1th and 1st layer multi-scale decoding modules; The i-th layer multi-scale decoding module takes input Y i Perform dilated convolution operation to obtain the global feature Y global ; At the same time, the i-th layer multi-scale decoding module inputs Y i Perform convolution operation to obtain local feature Y local ; The feature attention fusion method is used to global and Y local Fusion is performed to obtain the output of the i-th layer multi-scale decoding module 5. The multi-channel speech enhancement method based on the fusion of multi-scale spatial information and spectral features according to claim 4 is characterized in that: The specific process of step 4 is as follows: Step 4.1, use parallel linear transformation to map the output features of the multi-scale spatial aggregation encoder to H factorized subspaces, and the h-th subspace feature It is expressed as: Among them, h∈[1,2,…,H], C is the dimension of each subspace, that is, the dimension of the channel, X e represents the output features of the multi-scale spatial aggregation encoder, W e Represents the feature transformation matrix corresponding to the multi-scale spatial aggregation encoder; Step 4.2, map the output features of the multi-scale decoder to the same subspace as the multi-scale spatial aggregation encoder, and use the normalized exponential function operator softmax to calculate the weight of each subspace It is expressed as: Among them, X d represents the output features of the multi-scale decoder, W d Represents the feature transformation matrix corresponding to the multi-scale decoder; Step 4.3, the weights obtained by the multi-scale decoder in different subspaces are applied to the spatial features corresponding to the space, and the sum is obtained to obtain the enhanced speech signal features, which is expressed as: Among them, X ffa Represents the enhanced speech signal features.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the steps of the multi-channel speech enhancement method based on the fusion of multi-scale spatial information and spectral features as described in any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the multi-channel speech enhancement method based on the fusion of multi-scale spatial information and spectral features as described in any one of claims 1 to 5 are implemented.

Images