Speech enhancement method based on multi-modal fusion deep learning

Abstract

The invention provides a speech enhancement model based on an encoder-decoder structure and fused with a generative adversarial network, and aims to solve the problem of speech quality improvement in a complex acoustic environment. The experiment is based on an AI-SHELL3 clean corpus data set, and after large noise and large reverberation interference are added, the model is adopted for processing. The result shows that the PESQ (objective evaluation index of speech quality) of the speech enhanced by the model is relatively improved by 1.26, which indicates that the designed model can effectively suppress strong noise interference and reduce reverberation influence, significantly improve the speech quality under severe acoustic conditions, and improve the speech quality. The effectiveness of the combination of the encoder-decoder and the generative adversarial network in the speech enhancement task is verified, and an efficient solution is provided for speech processing in a high-interference environment.

Description

Technical Field

[0001] This invention relates to the field of audio signal processing and speech enhancement technology, and in particular to a speech enhancement method based on multimodal fusion deep learning. Background Technology

[0002] In real-life scenarios, voice signals captured by devices often suffer from dual quality losses: on the one hand, they are affected by noise interference from surrounding objects and self-reflection, resulting in reverberation distortion; on the other hand, narrowband communication devices only retain low-frequency components during transmission, leading to bandwidth limitations. These two problems severely degrade the performance of automatic speech recognition, hearing aids, and telecommunications systems. Against this backdrop, speech enhancement (SE) technology, with its core objective of improving the intelligibility and perceptual quality of distorted speech, has become a key means to solve these problems.

[0003] Traditional end-to-end architectures utilize neural networks to directly learn the mapping from distorted waveforms to the original waveforms (such as WaveNet). During modeling, amplitude and phase information are implicitly recovered, eliminating the need for explicit phase definition. While this approach avoids the direct challenges of phase modeling, it suffers from quality bottlenecks and is less efficient at generating high-resolution waveforms.

[0004] Another type of method focuses on short-time complex spectrum enhancement, implicitly recovering amplitude and phase in the complex domain. Another type employs a multi-stage decoupling strategy, first explicitly enhancing the amplitude spectrum, and then indirectly optimizing the phase through complex spectrum refinement. While these methods place greater emphasis on phase utilization than time-domain methods, they still haven't overcome the limitations of "implicit optimization." Phase accuracy depends on the indirect correlation between the amplitude spectrum and the complex spectrum, making it difficult to independently guarantee phase continuity and accuracy.

[0005] With the continuous development of deep learning technology, scholars have proposed the Transformer encoder-decoder architecture in recent years (Reference: A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, ANGomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol.30, 2017.). Previous methods defined loss functions only in the time or complex domain, leading to a serious “amplitude-phase implicit compensation” problem. When phase prediction is inaccurate, the model adjusts the amplitude spectrum to compensate for phase distortion, resulting in the destruction of the harmonic structure of the amplitude and a reduction in speech quality. Although subsequent research has alleviated this problem by introducing additional amplitude domain loss, the lack of explicit modeling and optimization of the phase means that core issues such as phase discontinuity and prediction bias remain unresolved, becoming a key bottleneck restricting the improvement of SE performance. (Reference: Z.-Q. Wang, G. Wichern, and J. Le Roux, “On the compensation between magnitude and phase inspeech separation,” IEEE Signal Processing Letters, vol. 28, pp. 2018–2022, 2021.). Furthermore, some researchers have employed parallel network structures to predict clean speech signals. (Reference: Y. Ai and Z.-H. Ling, “Neural speechphase prediction based on parallel estimation architecture and anti-wrapping losses,” in Proc. ICASSP, 2023.).

[0006] Therefore, deep learning-based speech enhancement methods face a significant challenge in recovering high-quality speech signals in complex noisy environments. Summary of the Invention

[0007] This invention primarily addresses the issue of insufficient input information in speech enhancement models when recovering distorted signals under conditions of high noise and reverberation. The proposed method achieves significant advantages through modular design: the encoder employs a concatenated structure of convolutional blocks and extended DenseNet for efficient speech feature extraction; the fusion module integrates multimodal information from speech and pinyin through a splicing strategy, enhancing semantic robustness; the amplitude decoder combines sub-pixel deconvolution with a task-adaptive activation function to accurately recover the amplitude spectrum; and the phase decoder explicitly models the phase through a parallel estimation structure, solving the entanglement problem. The overall architecture is flexibly adaptable to various scenarios, effectively improving the clarity and naturalness of speech in complex environments.

[0008] This invention discloses a speech enhancement method based on multimodal fusion deep learning, which includes:

[0009] S1: Obtain the original speech signal to be enhanced and the corresponding pinyin text information;

[0010] S2: Perform time-frequency conversion on the preprocessed original speech signal to obtain speech spectrum features; extract features from the pinyin text information to obtain pinyin sequence features;

[0011] S3: Input the speech spectrum features into the speech encoder and process them through a multi-layer neural network to obtain the speech coding features; input the pinyin sequence features into the pinyin encoder and process them through a sequence modeling network to obtain the pinyin coding features;

[0012] S4: Perform dimensional matching and feature fusion on speech coding features and pinyin coding features to obtain multimodal fusion features;

[0013] S5: Input the multimodal fusion features into the amplitude decoder, and process them through convolutional and deconvolutional networks to obtain the amplitude spectrum of the enhanced speech; at the same time, input the multimodal fusion features into the phase decoder, and process them through recurrent neural networks and attention mechanisms to obtain the phase information of the enhanced speech;

[0014] S6: Perform time-frequency inverse conversion based on the amplitude spectrum and phase information of the enhanced speech to obtain the time-domain enhanced speech signal and output it.

[0015] 2. The speech enhancement method based on multimodal fusion deep learning according to claim 1, wherein step 2 includes:

[0016] The preprocessed speech signal is subjected to short-time Fourier transform to obtain a spectrum matrix containing speech time-frequency features. Simultaneously, the pinyin signal is processed by pinyin encoding to convert the pinyin text into a corresponding pinyin symbol sequence. At the same time, the pinyin symbol sequence is embedded and encoded using the torch.embedding method to obtain pinyin embedding features. The speech embedding features and the pinyin embedding features have the same feature dimension and are used for subsequent feature fusion processing.

[0017] 3. The speech enhancement method based on multimodal fusion deep learning according to claim 1 or 2, characterized in that the structure of the speech encoder includes a cascaded structure of a convolutional block, an extended DenseNet, and another convolutional block, specifically configured as follows:

[0018] The speech encoder converts the input features into a TF-domain representation with C channels, T time dimensions, and F' = F / 2 frequency dimensions. Each convolutional block contains a 2D convolutional layer, an instance normalization layer, and a parameter-corrected linear unit (PReLU) activation layer. The 2D convolutional layer is used to extract the local spatial correlation of the input features, the instance normalization layer is used to standardize the feature distribution to accelerate model convergence, and the PReLU activation layer is used to introduce nonlinear transformations to enhance feature expressiveness. DenseNet is located between two convolutional blocks and achieves the fusion and reuse of features at different levels through dense connections, enhancing feature propagation ability and alleviating the gradient vanishing problem. At the same time, the bottleneck layer reduces the feature dimension to reduce computational complexity.

[0019] Through the sequential processing of the above cascaded structure, the deep encoding of the input features is completed, providing a speech feature representation with rich semantic information for subsequent feature fusion.

[0020] 4. The speech enhancement method based on multimodal fusion deep learning according to any one of claims 1 to 3, characterized in that the processing procedure of the feature fusion module includes:

[0021] Obtain the speech features X output by the speech encoder A The pinyin feature X output by the pinyin encoder L ; for speech features X A And Pinyin Features X L Dimension adaptation is performed by converting both features to the same feature dimension through linear transformation to ensure the feasibility of the concatenation operation; vector concatenation is used to concatenate the speech feature X. A And Pinyin Features X L The fusion process involves directly concatenating the two feature vectors along the feature dimension to obtain the fused feature Z. {A,L} .

[0022] 5. The speech enhancement method based on multimodal fusion deep learning according to any one of claims 1 to 4, characterized in that the processing procedure of the amplitude mask decoder includes:

[0023] The amplitude mask decoder comprises a cascaded structure of dilated DenseNet, deconvolutional blocks, and a mask estimation architecture:

[0024] Dilated DenseNet is used to perform deep feature extraction and fusion on the input TF domain representation, thereby enhancing the expressive power of the features;

[0025] The deconvolution block consists of a two-dimensional subpixel convolutional layer, an instance normalization layer, and a PReLU activation layer. The two-dimensional subpixel convolutional layer is used to upsample the frequency dimension of the TF domain representation back to F, which avoids artifacts compared to transposed convolution. The instance normalization layer is used to standardize the feature distribution, and the PReLU activation layer is used to introduce nonlinear transformation.

[0026] The mask estimation structure includes a two-dimensional convolutional layer and an activation function. The two-dimensional convolutional layer reduces the number of output channels of the deconvolutional block to 1, and the activation function is used to estimate the amplitude mask. The predicted compressed amplitude mask is multiplied by the compressed distortion amplitude spectrum to obtain the enhanced compressed amplitude spectrum (X). m ) c ;

[0027] The boundary magnitude mask is estimated using a learnable sigmoid (LSigmoid) function, calculated as follows:

[0028]

[0029] Where β is set to 2.0, α∈R F These are trainable frequency-dependent parameters that enable the model to adaptively adjust the shape of the activation function across frequency bands;

[0030] 6. The speech enhancement method based on multimodal fusion deep learning according to any one of claims 1 to 5, characterized in that the processing procedure of the phase decoder includes:

[0031] Obtain the TF domain representation after feature fusion, and directly predict the enhanced packaged phase spectrum X using a phase decoder. p To address the phase entanglement problem, the phase decoder and amplitude mask decoder employ a similar architecture, including a cascaded structure of extended DenseNet, deconvolutional blocks, and parallel estimation structures.

[0032] The DenseNet extension is used for deep feature extraction and fusion of the input TF domain representation, enhancing the expressive power of the features;

[0033] The convolutional block consists of a two-dimensional subpixel convolutional layer, an instance normalization layer, and a PReLU activation layer. The two-dimensional subpixel convolutional layer is used to upsample the frequency dimension of the TF domain representation back to the original frequency dimension to avoid artifacts. The instance normalization layer is used to standardize the feature distribution, and the PReLU activation layer is used to introduce nonlinear transformation.

[0034] The parallel estimation structure consists of two parallel two-dimensional convolutional layers that output pseudo-real and pseudo-imaginary components, respectively. These components are then activated using a bivariate arctangent (Arctan2) function, ultimately yielding the enhanced wrapped phase spectrum X. p Then, a short-time Fourier transform is performed on both to obtain the enhanced speech signal.

[0035] Because of the adoption of the above technical solution, the present invention has the following advantages:

[0036] The model innovatively combines speech spectral features with pinyin text features, leveraging prior linguistic information from the text modality (such as pronunciation rules and semantic logic) to provide "semantic anchors" for speech recognition in noisy environments. Compared to traditional models that rely solely on acoustic signals, this fusion strategy effectively combats information loss caused by high noise, reverberation, and bandwidth limitations. For example, in noisy in-vehicle or industrial environments, even with severely interfered speech signals, pinyin features can still assist the model in locating speech semantic boundaries. The experimental result of a 1.26 PESQ improvement also confirms the significant improvement in speech quality brought about by this strategy.

[0037] 2. To address the long-neglected issue of phase distortion, the model designs an independent phase decoder. By estimating the structure in parallel, it outputs pseudo-real and imaginary components, and then directly predicts the enhanced wrapped phase spectrum using a bivariate arctangent function, effectively solving the problems of phase entanglement and unstructured representation. This design overcomes the limitations of implicit phase recovery in the time or TF domains. Traditional methods often compensate for phase errors through amplitude, leading to harmonic structure disruption. This model, however, reduces the amplitude-phase compensation effect through explicit phase optimization, making the reconstructed speech closer to the original signal in terms of temporal continuity and harmonic structure.

[0038] 3. The model modules adopt a highly adaptable design: the encoder achieves efficient feature extraction by cascading convolutional blocks and extended DenseNet, avoiding redundant computation; the amplitude decoder combines subpixel deconvolution with task-adaptive activation functions (LSigmoid for denoising / dereverberation and PReLU for bandwidth expansion) to flexibly meet different enhancement requirements; compared with single denoising, dereverberation and bandwidth expansion task models, it significantly reduces the development cost of multi-scenario deployment.

[0039] 4. The technical details of the model in key stages enhance its engineering applicability: Feature dimension adaptation and time alignment (zero padding / interpolation) before fusion ensure the effectiveness of multimodal data collaboration, providing a highly practical solution for speech enhancement technology. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0041] Figure 1 This is a model diagram of the speech enhancement method based on multimodal fusion deep learning of the present invention;

[0042] Figure 2 This is a diagram of the encoder structure of the speech enhancement method based on multimodal fusion deep learning of the present invention;

[0043] Figure 3 This is a decoder structure diagram of the speech enhancement method based on multimodal fusion deep learning of the present invention;

[0044] Figure 4 This is a spectral comparison diagram of enhanced speech, noisy speech, and clean speech in an embodiment of the present invention; Detailed Implementation

[0045] The present invention will be further described in conjunction with the accompanying drawings and embodiments. The described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.

[0046] Figure 1 This is a model diagram of a speech enhancement method based on multimodal fusion deep learning. The model flow is as follows: First, a short-time Fourier transform is performed on the input audio waveform to obtain amplitude and phase information. These two are then concatenated and input into an audio encoder. Simultaneously, the input text is encoded by a pinyin encoder. Next, the outputs of the audio encoder and the pinyin encoder are fed into a deep multimodal fusion module. The fusion results are input into an amplitude decoder and a phase decoder, respectively. The output of the amplitude decoder is processed with the original amplitude information, and then combined with the output of the phase decoder. Finally, through an inverse short-time Fourier transform (iSTFT), the enhanced audio waveform is generated.

[0047] Figure 2This is a model structure diagram of the speech encoder and pinyin encoder in a multimodal fusion deep learning speech enhancement method. In this structure, the encoder module can be divided into three parts: the left and right sides are symmetrical, each consisting of a 2D convolutional network (Conv2d), an instance normalization network (Instance Norm), and a parameterized corrected linear unit (PReLU); the core part in the middle is a dilated dense network (DenseNet). Through this combination, the system performs operations such as feature extraction, normalization, nonlinear transformation, and deep dense feature fusion to complete the encoding task.

[0048] Figure 3 This is a model structure diagram of the decoder in a multimodal fusion deep learning speech enhancement method. The module begins with a DenseNet network to extract features from the input. Then, it sequentially passes through convolutions to perform pixel rearrangement and other operations to improve resolution; instance normalization to normalize features and enhance model stability; and parameterized linear units to introduce non-linear activation and increase model expressiveness. Finally, it connects to Sigmoid or Arctan2 to complete the final activation operations. The overall process involves a series of transformations and optimizations of the features.

[0049] Figure 4 This is a comparison of the spectrograms of enhanced speech, noisy speech, and clean speech in this invention. Noisy represents the spectrogram of noisy speech, Enhanced represents the spectrogram of enhanced speech, and Clean represents the spectrogram of clean speech.

[0050] The present invention will be further described below with reference to embodiments:

[0051] 1. A speech enhancement method based on multimodal fusion deep learning, characterized in that it includes:

[0052] S1: Obtain the original speech signal to be enhanced and the corresponding pinyin text information;

[0053] S2: Perform time-frequency conversion on the preprocessed original speech signal to obtain speech spectrum features; extract features from the pinyin text information to obtain pinyin sequence features;

[0054] S3: Input the speech spectrum features into the speech encoder and process them through a multi-layer neural network to obtain the speech coding features; input the pinyin sequence features into the pinyin encoder and process them through a sequence modeling network to obtain the pinyin coding features;

[0055] S4: Perform dimensional matching and feature fusion on speech coding features and pinyin coding features to obtain multimodal fusion features;

[0056] S5: Input the multimodal fusion features into the amplitude decoder, and process them through convolutional and deconvolutional networks to obtain the amplitude spectrum of the enhanced speech; at the same time, input the multimodal fusion features into the phase decoder, and process them through recurrent neural networks and attention mechanisms to obtain the phase information of the enhanced speech;

[0057] S6: Perform time-frequency inverse conversion based on the amplitude spectrum and phase information of the enhanced speech to obtain the time-domain enhanced speech signal and output it.

[0058] 2. The speech enhancement method based on multimodal fusion deep learning according to claim 1, wherein step 2 includes:

[0059] The preprocessed speech signal is subjected to short-time Fourier transform to obtain a spectrum matrix containing speech time-frequency features. Simultaneously, the pinyin signal is processed by pinyin encoding to convert the pinyin text into a corresponding pinyin symbol sequence. At the same time, the pinyin symbol sequence is embedded and encoded using the torch.embedding method to obtain pinyin embedding features. The speech embedding features and the pinyin embedding features have the same feature dimension and are used for subsequent feature fusion processing.

[0060] 3. The speech enhancement method based on multimodal fusion deep learning according to claim 1 or 2, characterized in that the structure of the speech encoder includes a cascaded structure of a convolutional block, an extended DenseNet, and another convolutional block, specifically configured as follows:

[0061] The speech encoder converts the input features into a TF-domain representation with C channels, T time dimensions, and F' = F / 2 frequency dimensions. Each convolutional block contains a 2D convolutional layer, an instance normalization layer, and a parameter-corrected linear unit (PReLU) activation layer. The 2D convolutional layer is used to extract the local spatial correlation of the input features, the instance normalization layer is used to standardize the feature distribution to accelerate model convergence, and the PReLU activation layer is used to introduce nonlinear transformations to enhance feature expressiveness. DenseNet is located between two convolutional blocks and achieves the fusion and reuse of features at different levels through dense connections, enhancing feature propagation ability and alleviating the gradient vanishing problem. At the same time, the bottleneck layer reduces the feature dimension to reduce computational complexity.

[0062] Through the sequential processing of the above cascaded structure, the deep encoding of the input features is completed, providing a speech feature representation with rich semantic information for subsequent feature fusion.

[0063] 4. The speech enhancement method based on multimodal fusion deep learning according to any one of claims 1 to 3, characterized in that the processing procedure of the feature fusion module includes:

[0064] Obtain the speech features X output by the speech encoder AThe pinyin feature X output by the pinyin encoder L ; for speech features X A And Pinyin Features X L Dimension adaptation is performed by converting both features to the same feature dimension through linear transformation to ensure the feasibility of the concatenation operation; vector concatenation is used to concatenate the speech feature X. A And Pinyin Features X L The fusion process involves directly concatenating the two feature vectors along the feature dimension to obtain the fused feature Z. {A,L} .

[0065] 5. The speech enhancement method based on multimodal fusion deep learning according to any one of claims 1 to 4, characterized in that the processing procedure of the amplitude mask decoder includes:

[0066] The amplitude mask decoder comprises a cascaded structure of dilated DenseNet, deconvolutional blocks, and a mask estimation architecture:

[0067] Dilated DenseNet is used to perform deep feature extraction and fusion on the input TF domain representation, thereby enhancing the expressive power of the features;

[0068] The deconvolution block consists of a two-dimensional subpixel convolutional layer, an instance normalization layer, and a PReLU activation layer. The two-dimensional subpixel convolutional layer is used to upsample the frequency dimension of the TF domain representation back to F, which avoids artifacts compared to transposed convolution. The instance normalization layer is used to standardize the feature distribution, and the PReLU activation layer is used to introduce nonlinear transformation.

[0069] The mask estimation structure includes a two-dimensional convolutional layer and an activation function. The two-dimensional convolutional layer reduces the number of output channels of the deconvolutional block to 1, and the activation function is used to estimate the amplitude mask. The predicted compressed amplitude mask is multiplied by the compressed distortion amplitude spectrum to obtain the enhanced compressed amplitude spectrum (X). m ) c ;

[0070] The boundary magnitude mask is estimated using a learnable sigmoid (LSigmoid) function, calculated as follows:

[0071]

[0072] Where β is set to 2.0, α∈R F These are trainable frequency-dependent parameters that enable the model to adaptively adjust the shape of the activation function across frequency bands;

[0073] 6. The speech enhancement method based on multimodal fusion deep learning according to any one of claims 1 to 5, characterized in that the processing procedure of the phase decoder includes:

[0074] Obtain the TF domain representation after feature fusion, and directly predict the enhanced packaged phase spectrum X using a phase decoder. p To address the phase entanglement problem, the phase decoder and amplitude mask decoder employ a similar architecture, including a cascaded structure of extended DenseNet, deconvolutional blocks, and parallel estimation structures.

[0075] The DenseNet extension is used for deep feature extraction and fusion of the input TF domain representation, enhancing the expressive power of the features;

[0076] The convolutional block consists of a two-dimensional subpixel convolutional layer, an instance normalization layer, and a PReLU activation layer. The two-dimensional subpixel convolutional layer is used to upsample the frequency dimension of the TF domain representation back to the original frequency dimension to avoid artifacts. The instance normalization layer is used to standardize the feature distribution, and the PReLU activation layer is used to introduce nonlinear transformation.

[0077] The parallel estimation structure consists of two parallel two-dimensional convolutional layers that output pseudo-real and pseudo-imaginary components, respectively. These components are then activated using a bivariate arctangent (Arctan2) function, ultimately yielding the enhanced wrapped phase spectrum X. p Then, a short-time Fourier transform is performed on both to obtain the enhanced speech signal.

[0078] The relevant parameter settings for deep learning are shown in Table 1.

[0079] Table 1 Simulation parameter settings

[0080] parameter Reference setting value STFT window length 400 STFT FFT points 400 STFT hop_size 100 Learning rate α 0.0005 Learning rate decay coefficient γ 0.99 batch_size 2 Training steps 500k

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A speech enhancement method based on multi-modal fusion deep learning, characterized in that, Comprise: S1: obtaining the original speech signal to be enhanced and the corresponding pinyin text information; S2: performing time-frequency conversion on the preprocessed original speech signal to obtain speech spectrum features; extracting features from the pinyin text information to obtain pinyin sequence features; S3: inputting the speech spectrum features into the speech encoder to obtain speech coding features through multi-layer neural network processing; input the pinyin sequence features into the pinyin encoder, and obtain the pinyin coding features through the sequence modeling network processing; S4: Dimension matching and feature fusion are performed on the speech coding features and the pinyin coding features to obtain multi-modal fusion features; S5: input the multi-modal fusion features into the amplitude decoder, and obtain the amplitude spectrum of the enhanced speech through convolution and deconvolution network processing; at the same time, input the multi-modal fusion features into the phase decoder, and obtain the phase information of the enhanced speech through the recurrent neural network and the attention mechanism; S6: Based on the amplitude spectrum and phase information of the enhanced speech, time-frequency inverse conversion is performed to obtain the time-domain enhanced speech signal and output. 2.The multi-modal fusion deep learning based speech enhancement method according to claim 1, characterized in that, The step 2 comprises: The preprocessed speech signal is processed by short-time Fourier transform to obtain a frequency spectrum matrix containing speech time-frequency features; at the same time, the pinyin signal is processed by pinyin coding to convert the pinyin text into a corresponding pinyin symbol sequence; at the same time, the torch.embedding method is used to embed the pinyin symbol sequence to obtain pinyin embedding features; the speech embedding features and the pinyin embedding features have the same feature dimension, which are used for subsequent feature fusion processing. 3.The multi-modal fusion deep learning based speech enhancement method according to claim 1 or 2, characterized in that, The structure of the speech encoder includes a cascade structure of convolution blocks, an extended DenseNet and another convolution block, and the specific configuration is as follows: The speech encoder converts the input features into a TF domain representation with C channels, T time dimensions and F'=F / 2 frequency dimensions; each convolution block includes a 2D convolution layer, an instance normalization layer and a PReLU activation layer, wherein the 2D convolution layer is used to extract the local spatial correlation of the input features, the instance normalization layer is used to standardize the feature distribution to speed up the model convergence, and the PReLU activation layer is used to introduce a nonlinear transformation to enhance the feature expression ability; The DenseNet is located between the two convolution blocks, which realizes the fusion and reuse of features of different levels through dense connection, enhances the feature propagation ability and relieves the gradient vanishing problem, and reduces the feature dimension through the bottleneck layer to reduce the calculation complexity; Through the above cascade structure, the input features are deeply encoded to provide speech features with rich semantic information for subsequent feature fusion.

4. The multi-modal fusion deep learning based speech enhancement method according to any one of claims 1 to 3, characterized in that, The processing process of the feature fusion module comprises: Obtaining the speech features X output by the speech encoder A and the pinyin features X output by the pinyin encoder L ; performing dimension adaptation processing on the speech features X A and the pinyin features X L , converting both into the same feature dimension through linear transformation to ensure the feasibility of the splicing operation; fusing the speech features X A and the pinyin features X L in a vector splicing manner, specifically, directly splicing the two feature vectors in the feature dimension direction to obtain fused features Z {A,L} .

5. The multi-modal fusion deep learning based speech enhancement method according to any one of claims 1 to 4, characterized in that, The processing process of the amplitude mask decoder comprises: The amplitude mask decoder comprises a cascade structure of dilated DenseNet, deconvolution block and mask estimation architecture: The dilated DenseNet is used for deep feature extraction and fusion of the input TF domain representation to enhance the feature expression ability; The deconvolution block is composed of a two-dimensional sub-pixel convolution layer, an instance normalization layer and a PReLU activation layer, wherein the two-dimensional sub-pixel convolution layer is used for up-sampling the frequency dimension of the TF domain representation back to F, which can avoid artifacts compared with transpose convolution, the instance normalization layer is used for normalizing the feature distribution, and the PReLU activation layer is used for introducing a nonlinear transformation; The mask estimation structure comprises a two-dimensional convolution layer for reducing the output channel number of the deconvolution block to 1 and an activation function for realizing the estimation of the amplitude mask, multiplying the predicted compressed amplitude mask with the compressed distorted amplitude spectrum to obtain an enhanced compressed amplitude spectrum (X m ) c ; The boundary amplitude mask is estimated using a learnable sigmoid (LSigmoid) function, and the calculation formula is: where β is set to 2.0, a e R F are trainable frequency-dependent parameters that enable the model to adaptively adjust the shape of the activation function across the frequency band.

6. The multi-modal fusion deep learning based speech enhancement method according to any one of claims 1 to 5, characterized in that, The processing procedure of the phase decoder comprises: The TF domain representation after feature fusion is obtained, and an enhanced wrapped phase spectrum X is directly predicted by a phase decoder p to solve the phase wrapping problem; the phase decoder and the amplitude mask decoder adopt a similar architecture, including an extended DenseNet, a deconvolution block, and a cascade structure of a parallel estimation structure: The extended DenseNet is used for deep feature extraction and fusion of the input TF domain representation, and the expression capability of the features is enhanced; The convolution block is composed of a two-dimensional sub-pixel convolution layer, an instance normalization layer and a PReLU activation layer, wherein the two-dimensional sub-pixel convolution layer is used for up-sampling the frequency dimension of the TF domain representation back to the original frequency dimension, which can avoid artifacts, the instance normalization layer is used for normalizing the feature distribution, and the PReLU activation layer is used for introducing a nonlinear transformation; The parallel estimation structure includes two two-dimensional convolution layers arranged in parallel, respectively outputting pseudo real components and pseudo imaginary components, and then the two components are activated by using a bi-variable arctangent (Arctan2) function to finally obtain an enhanced wrapped phase spectrum X p Then, short-time Fourier transform is performed on both to obtain an enhanced speech signal.

7. A computer program product comprising a computer program which, when run on a computer, causes the computer to perform the method steps of any one of claims 1 to 6.

Images