Speech enhancement method based on dynamic convolution and narrowband conformer

Abstract

The present application relates to the technical field of speech processing, and particularly relates to a speech enhancement method based on dynamic convolution and narrowband Conformer, which comprises a training stage and a test stage and can realize high-quality speech enhancement.The speech enhancement model proposed in the present application is composed of a generator and a discriminator, the narrowband Conformer network is used in the generator to improve the extraction ability of the model to speech spectrum information, and the dynamic convolution is further used to replace the traditional convolution, so that the parameter quantity and the calculation quantity of the model are greatly reduced, the noise reduction effect is improved, and the operation efficiency of the algorithm and the stability and reliability of the model are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and specifically to a speech enhancement method based on dynamic convolution and narrowband Conformer. Background Technology

[0002] In real-world environments, speech signals are inevitably affected by various noises, such as conversations from other people, television programs, restaurant sounds, and subway noise. Besides environmental noise, reverberation caused by sound wave attenuation and delayed reflections in enclosed spaces (such as living rooms and conference rooms) also affects the perceived quality of speech. Researchers have summarized the factors affecting speech quality in real-world scenarios into three main aspects: environmental noise, room reverberation, and interference from other speakers. The goal of speech enhancement research is to eliminate these three aspects. Speech enhancement technology refers to the use of speech signal processing algorithms to recover the target speech from noise-interfered speech, thereby improving the perceived quality and intelligibility of the target speech.

[0003] Common speech enhancement algorithms can be divided into two main categories: traditional speech enhancement algorithms and machine learning-based speech enhancement algorithms. Over the decades, several classic speech enhancement algorithms have emerged in the field of traditional speech enhancement algorithms. In 1979, Bol et al. proposed the classic spectral subtraction method, and subsequently, Lim et al. proposed the Wiener filtering method to address a series of problems with spectral subtraction. In 1985, Ephraim et al. proposed a minimum variance error estimation method based on the logarithmic spectrum. In 1991, Dendrinos et al. proposed applying singular value decomposition to the signal subspace, but this method also suffers from some speech distortion and musical noise problems. To address this issue, in 2000, Mittal et al. proposed a speech enhancement algorithm based on the noisy Hotelling transform matrix.

[0004] With the improvement of computer hardware capabilities and the continuous expansion of training datasets, machine learning-based speech enhancement algorithms have attracted much attention from researchers. In 1992, Ephraim et al. proposed using the basic idea of ​​Hidden Markov Models in the field of speech recognition to perform maximum a posteriori signal estimation and minimum mean square error on noisy and clean speech to achieve speech enhancement, achieving better speech enhancement results than spectral subtraction. In 2008, Wilson et al. proposed an algorithm based on nonnegative matrix factorization. In 2013, Lu et al. successfully applied deep neural networks to the field of speech enhancement based on the ideas of deep belief networks and denoising autoencoders. With the rise of convolutional neural networks, the computation in their networks can be executed in parallel, and their unique weight sharing and local perception characteristics can effectively capture some local details in speech signals. In 2017, Pascualet et al. proposed the SEGAN model, verifying the feasibility of temporal speech enhancement based on generative adversarial networks. In 2018, Kim et al. proposed the MDPhD model, which alleviated the modeling difficulties of temporal speech enhancement methods to some extent.

[0005] Thanks to the rapid development of the field of speech enhancement, the methods mentioned above have improved the perceptual quality and intelligibility of speech enhancement to a certain extent. However, the methods mentioned above still have the following problems: the computational load caused by long input is too large, which reduces the efficiency of the model. On the other hand, the model's ability to extract speech feature information needs to be improved. Therefore, how to reduce the number of computational parameters and further improve the efficiency of the model, while improving the model's ability to extract speech feature information, has become a research hotspot and difficulty in this field. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, such as excessively large number of computational parameters, low model efficiency, and excessive time consumption, this invention discloses a speech enhancement method based on dynamic convolution and narrowband Conformer. This method can enhance the network's representation ability, effectively extract local and global contextual feature information, improve the model's ability to learn deep features, and thus obtain higher speech perception quality and intelligibility.

[0007] The technical solution adopted in this invention is as follows:

[0008] A speech enhancement method based on dynamic convolution and narrowband Conformer includes a training phase and a testing phase. The training phase includes the following steps:

[0009] Step 1: Obtain a large amount of training data. The training data consists of data from multiple speakers, including noisy speech and clean speech.

[0010] Step 2: Preprocess the original noisy speech and original clean speech in the training corpus to obtain noisy speech x and clean speech s of fixed length;

[0011] Step 3: Input the preprocessed fixed-length noisy speech x and clean speech s into the enhancement network for training. The enhancement network consists of a generator G and a discriminator D. The generator G consists of an encoding network, a separation network, a narrowband Conformer network, and a decoding network. The narrowband Conformer network is built between the separation network and the decoding network. The speaker's high-dimensional intermediate vector is sequentially input into the narrowband Conformer network and the decoding network. The discriminator D consists of an encoding network, a convolutional neural network, and an evaluation network. The encoding network of the discriminator D shares weights with the encoding network of the generator G.

[0012] Step 4: During the training of the augmented network, minimize the loss function of the generator and the loss function of the discriminator, set the hyperparameters of the augmented network to minimize the objective function, and continue this process until the set number of iterations is reached, thereby obtaining the trained augmented network, which is called the DyConv-NBC TSEGAN network.

[0013] The testing phase includes the following steps:

[0014] Step 5: Select original noisy speech with different noise levels as the corpus to be enhanced, preprocess the original noisy speech in the corpus to be enhanced, and obtain noisy speech x′ of fixed length;

[0015] Step 6: Input the fixed-length noisy speech x′ obtained after the above preprocessing into the enhancement network trained in Step 4 to obtain the enhanced speech.

[0016] In the above technical solution, a narrowband Conformer network is constructed between the separation network and the decoding network of the generator G.

[0017] A further improvement to the present invention is that the training process in steps 3 and 4 is as follows:

[0018] (1) Preprocess the original noisy speech to obtain a noisy speech x of fixed length. Input the noisy speech x of fixed length into the encoding network of the generator G to extract features from the noisy speech and obtain a high-dimensional intermediate vector U(x).

[0019] (2) Input the high-dimensional intermediate vector U(x) obtained above into the separation network of the generator G to estimate the latent feature mask, thereby obtaining the latent feature mask F(U(x)) corresponding to each noisy speech;

[0020] (3) Input the latent feature mask F(U(x)) obtained above into the decoding network of the generator G for training. During the training process, minimize the loss function of the generator G to obtain the enhanced speech.

[0021] (4) The enhanced speech obtained above The clean speech s is input into the discriminator D for training, and the loss function of the discriminator D is minimized.

[0022] (5) Repeat (1)-(4) until the set number of iterations is reached, so as to obtain the trained DyConv-NBC TSEGAN network.

[0023] A further improvement to the present invention is that the input process in step 6 is as follows:

[0024] (1) Preprocess the original noisy speech to obtain a noisy speech x of fixed length. Input the noisy speech x of fixed length into the encoding network of the generator G to extract features from the noisy speech and obtain a high-dimensional intermediate vector U(x).

[0025] (2) Input the high-dimensional intermediate vector U(x) obtained above into the separation network of the generator G to estimate the latent feature mask, thereby obtaining the latent feature mask F(U(x)) corresponding to each noisy speech;

[0026] (3) Input the latent feature mask F(U(x)) obtained above into the decoding network of the generator G for training. During the training process, minimize the loss function of the generator G to obtain the enhanced speech.

[0027] In this invention, the metric evaluation Q(·,s) is:

[0028]

[0029] Where SI-SNR(·,s) represents the SI-SNR loss, i.e., the scale-invariant signal-to-noise ratio; β represents the parameter for adjusting the sensitivity range of tanh(·), which is set to 100 in the experiment. SI-SNR(·,s)∈R is used to obtain the metric Q(·,s)∈[-1,1] using tanh(·), where Q(·,s)=1 and Q(·,s)=-1 represent the best and worst cases, respectively, corresponding to the best and worst SI-SNR values.

[0030] In this invention, the SI-SNR loss is:

[0031]

[0032] in, This represents the scaling factor, and s represents the clean speech. Indicates enhanced speech, Indicates signal power. It represents the square of the 2-norm.

[0033] In this invention, the loss function of the generator G is:

[0034]

[0035] Where λ represents L p Norm Constraints The penalty parameter, ||·|1, represents the 1-norm, D M (·,s)∈[-1,1] represents the discriminator D, and q represents the target evaluation score that the generator is expected to achieve through the discriminator. Setting it to 1 will obtain the best evaluation Q(s,s).

[0036] In this invention, the loss function of the discriminator D is:

[0037]

[0038] Among them, D M (·,s)∈[-1,1] denotes the discriminator D; Q(s,s) and These represent clean speech (s) and enhanced speech, respectively. The assessment score.

[0039] A further improvement of this invention is that the encoding network consists of a single one-dimensional convolutional module, wherein the one-dimensional convolutional module includes a convolutional layer, a PReLU activation guidance layer, and a normalization layer:

[0040] The formula for the PReLU activation guidance is expressed as follows:

[0041]

[0042] Where δ∈R represents a trainable scalar used to control the negative slope of the rectifier;

[0043] In a non-causal configuration, the normalization layer is a global normalization, and its formula is expressed as:

[0044]

[0045]

[0046]

[0047] Where F∈R N×T Let E[·] denote the feature, E[·] denote the expectation of the feature, and ⊙ denote the product of corresponding points, where γ and τ ∈ R. N×1The parameter represents the trainable parameter, where N and T represent the dimension; ∈ represents a very small value used to maintain numerical stability.

[0048] In the causal configuration, the normalization layer is the cumulative layer normalization, and its formula is expressed as:

[0049]

[0050]

[0051]

[0052] Among them, f k ∈R N×1 Let f be the k-th frame of feature F. t≤k ∈R N×k Represents the features [f1, f2, ..., f] corresponding to frame k. k ], γ, τ∈R N×1 represents the trainable parameters applied to all frames, ⊙ represents the product of corresponding points, and N and T represent the dimensions.

[0053] A further improvement of the present invention is that the separation network consists of M one-dimensional convolutional modules, a PReLU activation guidance layer, and a normalization layer. The one-dimensional convolutional module includes a convolutional layer, an activation guidance layer, and a normalization layer. The activation guidance layer includes PReLU activation guidance and Sigmoid activation guidance.

[0054] The Sigmoid activation guidance function is expressed as follows:

[0055]

[0056] Where 'a' represents the input to the activation guidance layer. represents the hyperparameter, set to 0.1, and exp represents the exponential function with the natural constant e as the base.

[0057] The narrowband Conformer network is integrated between the generator separation network and the decoder network. It consists of a convolutional layer, an average pooling layer, an activation guidance layer, and three fully connected layers. The convolutional module consists of a multi-head self-attention network with relative position encoding, an L2 group feedforward network with convolutional layers, layer normalization, and a dropout layer.

[0058] A further improvement of the present invention is that the output of the Multi-Head Self-Attention with Relative Positional Encoding (RPSA) module is:

[0059]

[0060] Where, x i-1 Let represent the input of the i-th improved Conformer network, LayerNorm represent the layer normalization, H1 is the number of hidden units, and T represents the dimension.

[0061] The enhanced voice It can be characterized as:

[0062]

[0063] Where x represents noisy speech, U(·), V(·), and F(·) represent the encoder, decoder, and feature mask function, respectively, and ⊙ represents the multiplication of corresponding points.

[0064] The formula for activating the SiLU guidance layer is:

[0065]

[0066] Where 'a' represents the input to the activation guidance layer. represents the hyperparameter, and exp represents the exponential function with the natural constant e as the base.

[0067] The discriminator D consists of an encoding module, a convolutional neural network module, and an evaluation module. The convolutional neural network module includes a 2D-CNN module, which consists of dynamic convolution based on lightweight convolution and weighted adaptive instance normalization WAdaIN.

[0068] The formula for the lightweight convolution with respect to the i-th element in the sequence and the output dimension m is:

[0069]

[0070] Where X, H, d, W∈R d×k These represent the feature matrix, number of channels, dimension, and weight, respectively. ":" indicates that all elements in the specified row / column are retrieved. DepthwiseConv represents depthwise convolution, and softmax represents the normalized exponential function.

[0071] The formula for normalizing the normalized exponential function softmax is:

[0072]

[0073] Where r represents the time dimension, l represents the number of channels, and Y∈R l×r The weights in the time dimension r are represented by exp, which represents an exponential function with the natural constant e as its base.

[0074] The output of the lightweight convolution is:

[0075]

[0076] Where, X∈R b×t×c Let b, t, and c be the feature matrix, representing the batch size, segment length, and number of channels, respectively; K ∈ R k×h The kernel is denoted as , and k and h represent the kernel size and number of heads, respectively. Based on lightweight convolution, dynamic convolution introduces an additional kernel generation mechanism to generate kernels from the input features X, so that the shape of the kernel K′ of dynamic convolution becomes [b,t,k,h].

[0077] The output of the dynamic convolution is:

[0078]

[0079] Where K′∈R b×t×k×h Let be the kernel of the dynamic convolution generated by the feature matrix X through the activation guidance layer and the linear layer, where b, t, c, h, and k represent the batch size, segment length, number of channels, number of heads, and kernel size, respectively.

[0080] The kernel K′ of the dynamic convolution is:

[0081] K′=[GLU(XW1+b1)]×W2+b2

[0082] Where GLU represents the gating activation function, W1∈R c×(2×c) b1∈R 2×c W2∈R c×(k×h) b2∈R k×h These represent the training parameters, where c, h, and k represent the number of channels, the number of heads, and the kernel size, respectively.

[0083] To further explain, the formula for the adaptive convolution kernel W′ of the weighted adaptive instance normalized WAdaIN is:

[0084] W′=θ*W

[0085] Where x″ represents the input of the weighted adaptive instance normalization WAdaIN layer, θ represents the affine parameters generated from the target speaker embedding s″ through a linear layer, and the target speaker embedding s″ represents the speaker information extracted from the convolutional neural network layer preceding the weighted adaptive instance normalization WAdaIN layer.

[0086] The beneficial effects of this invention are as follows: Compared with existing technologies, this invention introduces a narrowband Conformer network into the separation and decoding networks. The narrowband Conformer network automatically utilizes narrowband information to perform speech separation, combining the attention mechanism with Convolutional Neural Networks (CNNs). It leverages the local modeling capabilities of CNNs to obtain local features of the sequence while simultaneously increasing the network depth. The Conformer uses many efficient regularization techniques to optimize the network, making the model easier to train and converge. The narrowband Conformer network utilizes the global modeling capabilities of the attention mechanism and the local modeling capabilities of CNNs, leveraging the advantages of both to improve the model's ability to extract speech feature information, thus significantly improving the perceptual quality and intelligibility of enhanced speech. Furthermore, this invention replaces traditional convolution with dynamic convolution based on lightweight convolution, introducing it into the discriminator. Dynamic convolution has a kernel that changes with the time step, improving the ability to model local dynamic information while reducing the number of parameters, thus solving the problem of excessive computation caused by long inputs. In summary, this invention not only improves the noise reduction effect but also effectively enhances the algorithm's operating efficiency and the model's stability and reliability. Attached Figure Description

[0087] Figure 1 This is a schematic diagram of the principle of the model described in the embodiment of the present invention.

[0088] Figure 2 This is a network structure diagram of the generator in the model described in this embodiment of the invention.

[0089] Figure 3 This is a network structure diagram of the discriminator in the model described in the embodiments of the present invention.

[0090] Figure 4 This is a diagram of the network structure of dynamic convolution in the model described in this embodiment of the invention.

[0091] Figure 5 This is a network structure diagram of the narrowband Conformer in the model described in the embodiment of the present invention. Detailed Implementation

[0092] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0093] Examples, such as Figure 1As shown, a speech enhancement method based on dynamic convolution and narrowband Conformer includes a training phase and a testing phase. The training phase is used to obtain the parameters and loss function required for speech enhancement, while the testing phase is used to enhance noisy speech.

[0094] The training phase first preprocesses the original noisy and clean speech to obtain noisy and clean speech of fixed length. Then, the noisy and clean speech of fixed length are fed into the neural network model to learn the mapping relationship between the noisy and clean speech. After the training phase, the trained model is saved. In the testing phase, the noisy speech is first preprocessed, then fed into the trained model for speech enhancement, and finally the enhanced speech is output.

[0095] To further explain, the training phase includes the following steps:

[0096] Step 1: Obtain training data. The training data comes from the VoiceBank-DEMAND corpus, which contains clean speech and its corresponding pre-mixed noisy speech. In our experiment, the clean speech samples used in the training phase consisted of 11,572 voice samples from 28 speakers (14 males and 14 females) at a sampling frequency of 48 kHz. The clean speech samples used in the testing phase consisted of 824 voice samples from 2 speakers (1 male and 1 female) at the same sampling frequency. The noisy speech samples used in the training phase consisted of 11,572 voice samples from 28 speakers (14 males and 14 females) and 824 voice samples from 2 speakers (1 male and 1 female). The noisy speech samples used in the training phase were composed of clean speech mixed with 10 types of noise at signal-to-noise ratios (SNRs) of 0 dB, 5 dB, 10 dB, and 15 dB. The noisy speech samples used in the testing phase were composed of clean speech mixed with 5 types of noise at SNRs of 2.5 dB, 7.5 dB, 12.5 dB, and 17.5 dB.

[0097] Step 2: Resample the noisy speech dataset and the clean speech dataset to 16kHz. The training set consists of 11,572 pairs of clean and noisy speech, and the test set consists of 824 pairs of clean and noisy speech.

[0098] Step 3: The speech enhancement method based on dynamic convolution and narrowband Conformer in this embodiment uses Generative Adversarial Networks (GANs) as a benchmark. By improving the GAN structure, the effectiveness of the enhancement network is enhanced. The GAN benchmark model consists of two parts: a generator G that generates the real spectrum and a discriminator D that determines whether the input spectrum is a real spectrum feature or a generated spectrum feature.

[0099] The objective function of the DyConv-NBC TSEGAN network in this embodiment is expressed as:

[0100]

[0101] in, The loss function for the generator:

[0102]

[0103] Where λ represents L p Norm Constraints The penalty parameter, ||·|1, represents the 1-norm, D M (·,s)∈[-1,1] represents the discriminator D, and q represents the target evaluation score that the generator is expected to achieve through the discriminator. Setting it to 1 will obtain the best evaluation Q(s,s).

[0104] Loss function of the discriminator:

[0105]

[0106] Among them, D M (·,s)∈[-1,1] denotes the discriminator D; Q(s,s) and These represent clean speech (s) and enhanced speech, respectively. The assessment score.

[0107] Step 4: Input the noisy speech x and clean speech s extracted in Step 2 into the generator for training. Train the generator until the set number of iterations is reached, so that the generator's loss function... To make it as small as possible, thereby enhancing speech.

[0108] The generator consists of an encoding network, a separation network, a narrowband Conformer network, and a decoding network. The encoding network comprises one one-dimensional convolutional module, which includes convolution, a PReLU activation guidance layer, and a normalization layer. The separation network consists of M one-dimensional convolutional modules, PReLU activation guidance layers, and normalization layers. The one-dimensional convolutional modules include convolution, activation guidance layers, and normalization layers; the activation guidance layers include PReLU activation guidance and sigmoid activation guidance. The narrowband Conformer network, located between the generator's separation and decoding networks, consists of one convolutional module, an average pooling layer, an activation guidance layer, and three fully connected layers. The convolutional module comprises a multi-head self-attention network with relative position encoding, an L2-group feedforward network with convolutional layers, layer normalization, and a dropout layer. The decoding network consists of one one-dimensional convolutional module, which includes convolution, a PReLU activation guidance layer, and a normalization layer.

[0109] Step 5: The enhanced speech obtained in Step 4 The clean speech s is input into the discriminator D to train the discriminator, so that the loss function of the discriminator is... As small as possible;

[0110] The discriminator D consists of an encoding module, a convolutional neural network module, and an evaluation module. The convolutional neural network module includes a 2D-CNN module, which consists of a dynamic convolutional network based on lightweight convolution and a weighted adaptive instance normalization WAdaIN.

[0111] The loss function of the discriminator is:

[0112]

[0113] Among them, D M (·,s) denotes the discriminator D; Q(s,s) and They represent the evaluation of clean speech s and enhanced speech, respectively. The evaluation score; the Q(·,s)∈[-1,1] is calculated based on SI-SNR(·,s) such that D M (·,s)∈[-1,1];

[0114] Step 6: Repeat steps 4-5 to minimize the objective function until the set number of iterations is reached. During training, minimize the loss of the augmented network until the set number of iterations is reached, thus obtaining the trained augmented network. In this experiment, the number of iterations is set to 200,000.

[0115] To further explain, the testing phase includes the following steps:

[0116] Step 1: Select original noisy speech with different noise levels as the corpus to be enhanced, preprocess the original noisy speech in the corpus to be enhanced, and obtain noisy speech x′ of fixed length;

[0117] Step 2: Input the fixed-length noisy speech x′ obtained after the above preprocessing into the enhancement network trained in Step 6 to obtain the enhanced speech.

[0118] The above description is an exemplary embodiment of the present invention and does not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A speech enhancement method based on dynamic convolution and narrowband Conformer, characterized in that, It includes a training phase and a testing phase, wherein the training phase includes the following steps: Step 1: Obtain a large amount of training data. The training data consists of data from multiple speakers, including noisy speech and clean speech. Step 2: Preprocess the original noisy speech and original clean speech in the training corpus to obtain noisy speech of fixed length. and pure voice ; Step 3: Process the preprocessed fixed-length noisy speech. Pure voice The input is fed into the augmentation network for training; Step 4: Set the hyperparameters of the augmented network to minimize the objective function until the set number of iterations is reached, thereby obtaining the trained augmented network; The testing phase includes the following steps: Step 5: Select original noisy speech samples with different levels of noise as the corpus to be enhanced, and preprocess the original noisy speech samples in the corpus to be enhanced to obtain noisy speech samples of a fixed length. ; Step 6: Process the preprocessed fixed-length noisy speech. Input the enhanced network trained in step 4 into the enhanced speech; In step 3, the speech enhancement network includes a generator G and a discriminator D. The generator G consists of an encoding network, a splitting network, a narrowband Conformer network, and a decoding network, wherein the narrowband Conformer network is constructed between the splitting network and the decoding network. The speaker's high-dimensional intermediate vector is sequentially input into the narrowband Conformer network and the decoding network. The discriminator D consists of an encoding network, a convolutional neural network, and an evaluation network, and the encoding network of the discriminator D shares weights with the encoding network of the generator G.

2. The speech enhancement method based on dynamic convolution and narrowband Conformer according to claim 1, characterized in that, The training process in step 3 is as follows: (1) Preprocess the original noisy speech to obtain a fixed length of noisy speech. Noisy speech of fixed length The input to the encoder network of generator G is used to extract features from the noisy speech, resulting in a high-dimensional intermediate vector. ; (2) The high-dimensional intermediate vector obtained above The latent feature masks are estimated in the separation network input to the generator G, thereby obtaining the latent feature masks corresponding to each noisy speech. ; (3) The latent feature mask obtained above The input is fed into the decoding network of the generator G for training. During training, the loss function of the generator G is minimized to obtain the enhanced speech. ; (4) The enhanced speech obtained above Pure voice They are input together into the discriminator D for training, minimizing the loss function of the discriminator D; (5) Repeat (1)-(4) until the set number of iterations is reached, so as to obtain the trained DyConv-NBCTSEGAN network.

3. The speech enhancement method based on dynamic convolution and narrowband Conformer according to claim 2, characterized in that, The input process in step 6 is as follows: (1) Preprocess the original noisy speech to obtain a fixed length of noisy speech. Noisy speech of fixed length The input is fed into the encoding network of the generator G for feature extraction, resulting in a high-dimensional intermediate vector. ; (2) Input the high-dimensional intermediate vector obtained above into the separation network of the generator G to estimate the latent feature mask, thereby obtaining the latent feature mask corresponding to each noisy speech. ; (3) The latent feature mask obtained above The input is fed into the decoding network of the generator G for training. During training, the loss function of the generator G is minimized to obtain the enhanced speech. .

4. The speech enhancement method based on dynamic convolution and narrowband Conformer according to claim 3, characterized in that, The discriminator D consists of a two-dimensional convolutional neural network module (2D-CNN), a locked encoding module that shares weights with the encoding network in the generator G, and a metric evaluation module, wherein the metric evaluation module introduces an evaluation metric. This guides the optimization of the discriminator G and calculates the discriminator's loss. The evaluation score of the discriminator D is used to limit the discriminator's score and is not directly used as the loss function. The metric evaluation for: ， in, express Loss, i.e., scale-invariant signal-to-noise ratio, Indicates adjustment Parameters of the sensitivity range use Measure and evaluate ,in, and These represent the best and worst scenarios, respectively, corresponding to the optimal and worst-case situations. value; The The loss is: ， in, Indicates the scaling factor. Indicates pure speech. Indicates enhanced speech, Indicates signal power. It represents the square of the 2-norm.

5. The speech enhancement method based on dynamic convolution and narrowband Conformer according to claim 4, characterized in that, The loss function of the generator G is: ， in, express Norm Constraints The penalty parameter, Describing the 1-norm, Discriminator D, This represents the target evaluation score that the generator is expected to achieve through the discriminator. Setting it to 1 will yield the best evaluation. ; The loss function of the discriminator D is: ， in, Discriminator D, and These represent pure speech. and enhanced voice The assessment score.

6. The speech enhancement method based on dynamic convolution and narrowband Conformer according to claim 5, characterized in that, The generator G consists of an encoding network, a splitting network, a narrowband Conformer network, and a decoding network. The encoding network comprises a one-dimensional convolutional module, which includes convolutional layers, ... Activate the guidance layer and the normalization layer; The The formula for activating the guidance is expressed as follows: ， in, This represents a trainable scalar used to control the negative slope of the rectifier; In a non-causal configuration, the normalization layer is a global normalization, and its formula is expressed as: ， ， ， in, Indicates features, The expectation of the feature. This indicates that corresponding points are multiplied. Indicates trainable parameters, Indicates dimension; It represents a very small value, used to maintain numerical stability; In the causal configuration, the normalization layer is the cumulative layer normalization, and its formula is expressed as: ， ， ， in, Representation of features The frame, Indicates corresponding to Frame characteristics , This represents the trainable parameters applied to all frames. This indicates that corresponding points are multiplied. Indicates dimension; The separation network consists of M one-dimensional convolutional modules, an activation guidance layer, and a normalization layer. The one-dimensional convolutional modules include convolutional layers, Activation guidance and normalization layer, the activation guidance layer includes Activation guide and Activation instructions; The The activation guidance function is expressed as: ， in, This indicates the input to activate the guidance layer. This represents a hyperparameter, set to 0.

1. This represents an exponential function with the natural constant e as its base. The narrowband Conformer network, fused between the generator separation network and the decoder network, consists of one convolutional layer, an average pooling layer, an activation guidance layer, and three fully connected layers. The convolutional module comprises a multi-head self-attention network with relative position encoding. The group consists of a feedforward network with convolutional layers, layer normalization, and Layer composition; The output of the Multi-Head Self-Attention with Relative Positional Encoding (RPSA) module is: ， in, Indicates the first The input to an improved Conformer network, Representation layer normalization, Indicates the number of hidden units. Indicates dimension.

7. The speech enhancement method based on dynamic convolution and narrowband Conformer according to claim 6, characterized in that, The enhanced voice Characterized as: ， in, Indicates noisy speech. These represent the encoder, decoder, and feature mask function, respectively. This indicates that corresponding points are multiplied.

8. The speech enhancement method based on dynamic convolution and narrowband Conformer according to claim 7, characterized in that, The feedforward network with convolutional layers consists of 1D group convolutions, normalization layers, and... Activation guidance layer components; The The formula for activating the guidance layer is: , in, This indicates the input to activate the guidance layer. Indicates hyperparameters, This represents an exponential function with the natural constant e as its base.

9. The speech enhancement method based on dynamic convolution and narrowband Conformer according to claim 8, characterized in that, The discriminator D consists of an encoding module, a convolutional neural network module, and an evaluation module. The convolutional neural network module includes a 2D-CNN module, which consists of dynamic convolution based on lightweight convolution and weighted adaptive instance normalization WAdaIN. The lightweight convolution is for the first... Each element and output dimension The formula is: ， in, These represent the feature matrix, number of channels, dimension, and weights, respectively. ":" indicates retrieving all elements from the specified row / column. Represents depthwise convolution. This represents the normalized exponential function; The normalized exponential function The normalization formula is: ， in, Indicates the time dimension. Indicates the number of channels. Representing the time dimension Weight on, This represents an exponential function with the natural constant e as its base. The output of the lightweight convolution is: ， in, Represents the characteristic matrix, These represent batch size, segment length, and number of channels, respectively. Represents the convolution kernel. These represent kernel size and number of heads, respectively. Based on lightweight convolution, dynamic convolution introduces an additional kernel generation mechanism, drawing from input features... Generate kernels to make the kernels of dynamic convolutions... The shape becomes ; The output of the dynamic convolution is: ， in, Representation of the characteristic matrix By activating the kernels of dynamically generated convolutions in the guiding layer and linear layer, These represent batch size, segment length, number of channels, number of headers, and core size, respectively. The kernel of the dynamic convolution for: ， in, This represents the activation function of the gating mechanism. , , , These represent the training parameters, These represent the number of channels, the number of headers, and the core size, respectively. The adaptive convolution kernel of the weighted adaptive instance normalized WAdaIN The formula is: ， in, This represents the input to the weighted adaptive instance normalization WAdaIN layer. This indicates embedding from the target speaker. Target speaker embedding using affine parameters generated by a linear layer. This represents the speaker information extracted by the convolutional neural network layer preceding the weighted adaptive instance normalization WAdaIN layer.

Images