Voice noise reduction method based on deep spanning feature extraction and feature cross fusion

Abstract

The invention particularly relates to a voice noise reduction method based on deep spanning feature extraction and feature cross fusion, which comprises the following steps: a double-flow network structure comprising an amplitude feature extraction branch and a phase feature extraction branch is constructed, the amplitude feature extraction branch adopts a multi-scale coding-decoding structure and a time sequence convolutional network for joint modeling, and the phase feature extraction branch adopts a multi-scale coding-decoding structure; local amplitude details are extracted to be dependent on long-range time; the phase feature extraction branch uses a hierarchical decoding structure for recovering multi-level phase structure features. Moreover, a cross-flow feature fusion mechanism is designed between double flows, so that the amplitude features and the phase features are dynamically complementary in the deep network. According to the method, acoustic physical priori and data driving characteristics are combined, noise reduction can be accurately carried out under complex background noise, and the performance of a voice noise reduction task in a non-stationary noise environment is effectively improved; therefore, problems of insufficient time-frequency correlation modeling, insufficient amplitude and phase feature utilization and the like of the existing speech enhancement technology in a complex noise scene can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech signal processing, in particular to a speech noise reduction method based on deep cross-over feature extraction and feature cross-fusion. BACKGROUND

[0002] In the field of speech signal processing, single-channel speech enhancement aims to extract clear speech components from noisy speech signals collected by a single microphone and suppress environmental noise interference. Single-channel speech enhancement technology can effectively reduce background noise, echo and other interference, helping the system to more accurately capture user voice commands.

[0003] Traditional time-frequency domain (T-F) speech enhancement algorithms use a large amount of unsupervised data for feature extraction in the preprocessing stage, decompose the speech signal into time-frequency atoms through time-frequency transformation, and learn the statistical characteristics of noise and speech. In the subsequent task adaptation stage, specific time-frequency masks or filtering rules are designed, combined with a small amount of labeled clean speech and noise mixed data, to adjust the parameters of the model to adapt to the noise reduction needs in specific scenarios. Although traditional methods have advantages in computational efficiency and real-time performance, they still face challenges such as insufficient generalization ability in complex noise environments or sparse data scenarios, and difficulty in parameter tuning.

[0004] Adaptive template learning re-expresses the speech enhancement task as a resolvable template form, allowing traditional algorithms to directly use the time-frequency features extracted in the preprocessing stage without complex manual parameter tuning. However, adaptive template learning models perform well in the field of speech enhancement, but have inherent defects such as insufficient cross-task generalization ability and strong dependence on template design. When faced with complex noise environments with large sample sizes, diverse noise types, and unknown noise sources, the model is highly sensitive to the coverage range of pre-defined noise templates and data quality, and the cascading effect of noise estimation errors easily occurs, resulting in suboptimal performance in complex noise scenarios with multiple factors coupled.

[0005] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The present application provides a speech noise reduction method based on deep cross-over feature extraction and feature cross-fusion, a computer-readable storage medium, and a computer program product, which can effectively overcome the defects in the prior art.

[0007] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0008] According to a first aspect of the present application, a speech noise reduction method based on deep cross-feature extraction and feature cross-fusion is provided, the method comprising: performing short-time Fourier transform on the to-be-processed speech signal to obtain an input amplitude stream and an input phase stream; modeling the input amplitude stream and the input phase stream based on a dual-stream network model to generate a target amplitude estimation stream and a target phase estimation stream corresponding to the to-be-processed speech signal; wherein the target dual-stream network model comprises an amplitude feature extraction branch and a phase feature extraction branch, the amplitude feature extraction branch is used to extract multi-scale amplitude features of the input amplitude stream, and the phase feature extraction branch is used to perform hierarchical coding on the input phase stream to extract phase structure features of the input phase stream; generating time-frequency estimation information based on the target amplitude estimation stream and the target phase estimation stream; and performing inverse short-time Fourier transform on the time-frequency estimation information in combination with a sampling rate to obtain a target speech signal; wherein the target speech signal is the to-be-processed speech signal after noise reduction.

[0009] In some example embodiments, the modeling of the input amplitude stream and the input phase stream based on the dual-stream network model to generate the target amplitude estimation stream and the target phase estimation stream corresponding to the to-be-processed speech signal comprises: modeling the input amplitude stream in time sequence using the amplitude feature extraction branch to obtain multi-scale amplitude features; and performing hierarchical coding on the input phase stream using the phase feature extraction branch to obtain phase structure features; performing cross-stream feature fusion on the multi-scale amplitude features and the phase structure features to obtain cross-stream fusion features; performing feature mapping on the cross-stream fusion features based on the amplitude mapping branch and the phase mapping branch to generate an amplitude estimation stream and a phase estimation stream.

[0010] In some example embodiments, the modeling of the input amplitude stream in time sequence using the amplitude feature extraction branch to obtain multi-scale amplitude features comprises: inputting the input amplitude stream in time sequence into a first multi-level coding layer, pre-processing the input amplitude stream using a first coding pre-processing module of the first multi-level coding layer to obtain pre-processed amplitude features; at the same time, performing multi-scale feature extraction on the pre-processed amplitude features using each level coding layer in a first U-shaped network structure, and fusing the extracted amplitude features with the pre-processed amplitude features to obtain first amplitude coding features; and saving the amplitude coding features corresponding to each level coding layer; inputting the first amplitude coding features into a time sequence convolution structure, modeling the context information of the first amplitude coding features in the time dimension through a space-time convolution structure to obtain amplitude enhancement features; The amplitude enhancement feature and the amplitude coding feature corresponding to each coding layer are reconstructed and fused by using the multi-level decoding layer for step-by-step up-sampling, to obtain multi-scale amplitude features; wherein the multi-scale amplitude features include local amplitude features and global time structure features.

[0011] In some example embodiments, the step of reconstructing and fusing the amplitude enhancement feature and the amplitude coding feature corresponding to each coding layer by using the multi-level decoding layer for step-by-step up-sampling to obtain multi-scale amplitude features comprises: performing up-sampling processing on the amplitude enhancement feature by using the first decoding layer of the first multi-level decoding layer to obtain first amplitude decoding features corresponding to the first decoding layer; performing same-scale skip connection on the first amplitude decoding features and the saved second amplitude coding features corresponding to the first coding layer to obtain fused second amplitude decoding features; taking the second amplitude decoding features as inputs of the next decoding layer, and sequentially performing up-sampling and same-scale skip connection processing to recover time-frequency resolution and integrate amplitude decoding features of different scales step by step, to obtain multi-scale amplitude features.

[0012] In some example embodiments, the step of encoding the phase stream by using the phase feature extraction branch to obtain phase structure features comprises: inputting the phase stream in time sequence into a second encoding preprocessing module of the first coding layer of the second multi-level coding layer, and performing gating filtering, amplitude normalization and nonlinear transformation on the input features by using the second encoding preprocessing module to obtain preprocessed phase features; performing multi-scale feature extraction on the preprocessed phase features by using a second U-shaped network structure to obtain phase semantic representations; performing feature fusion on the phase semantic representations and the preprocessed phase features to obtain phase structure features.

[0013] In some example embodiments, the step of performing cross-stream feature fusion on the multi-scale amplitude features and the phase structure features to obtain cross-stream fusion features comprises: performing feature alignment processing on the multi-scale amplitude features output by the amplitude feature extraction branch and the phase structure features output by the phase feature extraction branch; superimposing the aligned multi-scale amplitude features and the phase structure features to obtain cross-stream fusion features.

[0014] In some example embodiments, the amplitude mapping branch comprises a first convolutional layer, a bidirectional long short-term memory network layer and a multi-layer fully connected layer, the phase mapping branch comprises a second convolutional layer and a normalization layer, and the cross-stream fusion features are mapped by using the amplitude mapping branch and the phase mapping branch respectively to generate amplitude estimation streams and phase estimation streams, which comprises: The amplitude initial feature is obtained by performing channel compression and linear mapping on the cross-flow fusion feature by using a first convolutional layer; The time-series enhanced amplitude feature is obtained by performing cross-time frame context association modeling on the amplitude initial feature by using an input bidirectional long short-term memory network; The amplitude estimation flow is generated by performing nonlinear mapping and dimension transformation on the time-series enhanced amplitude feature based on a plurality of full connection layers; The phase initial feature is obtained by performing channel compression and linear mapping on the cross-flow fusion feature by using a second convolutional layer; The phase estimation flow is generated by performing normalization processing on the phase initial feature based on a normalization layer.

[0015] In some example embodiments, the method further comprises: mixing the noise data set and the speech data set to generate training data for training the initial dual-flow network model; and performing short-time Fourier transform on the mixed speech signal and the reference speech signal in the training data set respectively to generate corresponding initial amplitude flow, initial phase flow, and reference amplitude flow, reference phase flow; The initial amplitude estimation flow and the initial phase estimation flow are generated by performing spatio-temporal feature modeling on the initial amplitude flow and the initial phase flow by using the initial dual-flow network model; The joint loss is calculated according to the initial amplitude estimation flow, the initial phase estimation flow, and the reference amplitude flow, reference phase flow; the model parameters of the initial dual-flow network model are updated by using the joint loss and an optimizer to obtain a target dual-flow network model for speech denoising; wherein the joint loss includes an amplitude loss and a phase loss, the amplitude loss is determined based on the initial amplitude estimation flow and the reference amplitude flow, and the phase loss is determined based on the initial phase estimation flow and the reference phase flow.

[0016] According to a second aspect of the present application, a computer-readable storage medium is provided, which includes a stored executable program, wherein the executable program controls a device where the storage medium is located to perform the above-mentioned speech denoising method based on deep cross-over feature extraction and feature cross-fusion when the executable program is running.

[0017] According to a third aspect of the present application, a computer program product is provided, which includes a computer program, the computer program implements the above-mentioned speech denoising method based on deep cross-over feature extraction and feature cross-fusion when the computer program is executed by a processor.

[0018] According to a fourth aspect of the present application, an electronic device is provided, which includes: a processor; and a memory for storing executable instructions of the processor; The processor is configured to implement the above-mentioned speech noise reduction method based on deep cross-feature extraction and cross-feature fusion by executing the executable instructions.

[0019] The speech noise reduction method based on deep cross-feature extraction and cross-feature fusion provided by the embodiment of the application takes a short-time Fourier transform spectrogram as input, and constructs a dual-flow network structure including an amplitude feature extraction branch and a phase feature extraction branch. The amplitude feature extraction branch adopts a multi-scale coding-decoding structure and a time sequence convolution network for joint modeling to extract local amplitude details and long-range time dependence. The phase feature extraction branch uses a hierarchical decoding structure to restore multi-level phase structure features. A cross-flow feature fusion mechanism is designed between the dual flows to enable dynamic complementation of the amplitude features and the phase features in the deep network. Based on the above structure, the dual-flow network structure combines acoustic physical priors and data-driven features, can more accurately distinguish between speech and noise patterns in a complex background noise, effectively improves the performance in a non-stationary noise environment in the speech noise reduction task, and thus solves the problems of insufficient time-frequency correlation modeling and insufficient utilization of amplitude and phase features in the existing speech enhancement technology in a complex noise scene.

[0020] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the application. BRIEF DESCRIPTION OF DRAWINGS

[0021] The drawings incorporated into the specification and forming part of the specification, show embodiments consistent with the application, and together with the specification, serve to explain the principles of the application. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained from these drawings without creative labor for those skilled in the art.

[0022] Figure 1 The flowchart of the speech noise reduction method based on deep cross-feature extraction and cross-feature fusion according to an exemplary embodiment of the application is schematically shown; Figure 2 The target dual-flow network model architecture diagram of the speech noise reduction method based on deep cross-feature extraction and cross-feature fusion according to an exemplary embodiment of the application is schematically shown; Figure 3A The U-Net architecture diagram of the speech noise reduction method based on deep cross-feature extraction and cross-feature fusion according to an exemplary embodiment of the application is schematically shown. 2 -Encoder architecture diagram; Figure 3B The S-TCN architecture diagram of the speech noise reduction method based on deep cross-feature extraction and cross-feature fusion according to an exemplary embodiment of the application is schematically shown. Figure 3C Fig. 1 schematically shows a UML diagram of a speech denoising method based on deep cross-feature extraction and fusion according to an example embodiment of the present application. 2 Fig. 2 schematically shows a decoder architecture diagram according to an example embodiment of the present application. Figure 4 Fig. 3 schematically shows a block diagram of an electronic device according to an example embodiment of the present application. DETAILED DESCRIPTION

[0023] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations can be implemented in any

[0024] In addition, the drawings are to be regarded as being schematic and are not necessarily drawn to scale. Like reference numerals are used to convey like elements throughout certain of the figures. Descriptions of features, structures, or characteristics can be presented in terms of functional descriptions and / or structural descriptions. The descriptions and / or drawings can be one implementation. Other implementations can be described in terms of other functional descriptions and / or structural descriptions.

[0025] Single-channel speech enhancement aims to extract clean speech components from noisy speech signals collected by a single microphone, and suppress environmental noise interference. The core is to convert the noisy speech input into high-quality speech output through algorithm, so as to improve the performance of speech communication or speech recognition system. For example, in the scene of intelligent voice assistant, single-channel speech enhancement technology can effectively reduce background noise, echo and other interference, and help the system to capture user voice instructions more accurately. This task essentially belongs to the problem of signal separation, which can be realized through traditional filtering algorithm or deep learning model. In practical application, it is necessary to design enhancement algorithm according to specific noise scene, and to train model and optimize parameters through a large number of noisy / clean speech data pairs.

[0026] In the prior art, the traditional time-frequency domain (T-F) speech enhancement algorithm technology has been an important method in the field of noise suppression. In the preprocessing stage, the technology uses a large amount of unsupervised data (such as untagged speech and noise mixed signals) for feature extraction, and decomposes the speech signal into time-frequency atoms through time-frequency transformation such as short-time Fourier transform, and learns the statistical characteristics of noise and speech. These features can be expressed as amplitude spectrum, phase spectrum and noise energy distribution, etc. Priori knowledge. In the subsequent task adaptation stage, the traditional method adjusts the parameters of the enhancement model by designing specific time-frequency masks or filtering rules, combining a small amount of clean speech and noise mixed data with labels, to adapt to the noise reduction requirements in specific scenarios.

[0027] Although the traditional method has advantages in computational efficiency and real-time performance, it still faces challenges such as insufficient generalization ability in complex noise environments or sparse data scenarios, and difficulty in parameter tuning. These problems mainly arise from the strong assumptions of traditional methods on noise statistical characteristics and the dependence on artificially designed rules. Especially in non-stationary noise or low signal-to-noise ratio scenarios, fixed rule enhancement algorithms are difficult to achieve ideal results.

[0028] To solve this problem, adaptive template learning has emerged. This method re-expresses the speech enhancement task as an analyzable template form (such as a dynamic time-frequency mask template or a noise power estimation template), allowing the traditional algorithm to directly use the time-frequency features extracted in the preprocessing stage without complex manual parameter tuning. The core idea is to design task-specific template rules (such as adaptive noise threshold templates) to guide the algorithm to process time-frequency masking in a data-driven manner, thereby effectively improving the adaptability of the algorithm to unknown noise environments without significantly increasing computational complexity. Adaptive template learning has shown significant advantages in multiple speech enhancement tasks, especially in data-scarce or few-sample scenarios, significantly reducing the dependence on manually annotated data and enhancing the generalization ability of the algorithm.

[0029] For example, the time-frequency domain speech enhancement model based on adaptive template learning (Adaptive Spectro-Temporal Enhancement, ASTE) can significantly improve the speech intelligibility and time-frequency feature separation accuracy in noisy environments. Based on the traditional time-frequency masking framework, ASTE optimizes the adaptive coefficient parameters of the template and reconstructs the enhancement process with frequency domain constraints. By only iteratively adjusting the adaptive coefficient parameters of the template, the ASTE model can accurately coordinate the noise suppression intensity and the speech component preservation ratio of the two core indicators. Experiments show that ASTE outperforms traditional spectral subtraction, wavelet denoising and variational Bayesian algorithm in subjective intelligibility score and objective stoi / PESQ indicators, showing significant technical advantages.

[0030] The adaptive template learning model performs well in the field of speech enhancement, but has inherent defects such as insufficient cross-task generalization ability and strong dependence on template design. When facing a scene with large sample size and complex noise environment (multiple noise types, including unknown noise sources), due to the high sensitivity of the model to the coverage range of the pre-defined noise template and the quality of the data, and the cascading effect caused by the noise estimation error, the performance of the model in dealing with complex noise scenes with multiple factors is not as expected.

[0031] In view of the shortcomings and deficiencies of the prior art, a speech denoising method based on deep cross-feature extraction and feature cross-fusion is provided in the example embodiment. Referring to Figure 1 As shown, specifically can include: Step S10, performing short-time Fourier transform on the to-be-processed speech signal to obtain an input amplitude stream and an input phase stream; Step S12, modeling the input amplitude stream and the input phase stream based on a dual-stream network model to generate a target amplitude estimation stream and a target phase estimation stream corresponding to the to-be-processed speech signal; wherein the target dual-stream network model includes an amplitude feature extraction branch and a phase feature extraction branch, the amplitude feature extraction branch is used to extract multi-scale amplitude features of the input amplitude stream, and the phase feature extraction branch is used to perform hierarchical coding on the input phase stream to extract phase structure features of the input phase stream; Step S14, generating time-frequency estimation information based on the target amplitude estimation stream and the target phase estimation stream; and performing inverse short-time Fourier transform on the time-frequency estimation information in combination with the sampling rate to obtain a target speech signal; wherein the target speech signal is the to-be-processed speech signal after denoising.

[0032] Based on the above steps S10 to S14, by modeling the amplitude estimation and the phase estimation of the to-be-processed speech signal in the time-frequency domain based on the dual-stream network model, the fine recovery of the time-frequency structure of the speech in the complex noise environment is realized, the multi-type background noise can be effectively suppressed, and the naturalness and continuity of the speech signal can be maintained, so that a target speech signal with high fidelity is obtained. In the following, the steps of the speech denoising method based on deep cross-feature extraction and feature cross-fusion in the example embodiment will be described in more detail in combination with the drawings and examples.

[0033] For example, in step S12, the dual-stream network model is used to model the input amplitude stream and the input phase stream based on the dual-stream network model to generate a target amplitude estimation stream and a target phase estimation stream corresponding to the to-be-processed speech signal, which includes: Step S121, modeling the input amplitude stream in time sequence by using the amplitude feature extraction branch to obtain multi-scale amplitude features; and performing hierarchical coding on the input phase stream by using the phase feature extraction branch to obtain phase structure features; Step S122: Perform cross-current feature fusion on the multi-scale amplitude features and phase structure features to obtain cross-current fused features; Step S123: Based on the amplitude mapping branch and the phase mapping branch, feature mapping is performed on the cross-flow fusion features to generate the amplitude estimation stream and the phase estimation stream.

[0034] For details, please refer to Figure 2 As shown, in the amplitude feature extraction branch, the amplitude stream extracted from the Short-Time Fourier Transform (STFT) spectrum is passed through a multi-stage encoder U containing a U-Net module. 2 -Encoder, Temporal Convolutional Network S-TCN, and decoder U 2 - The Decoder performs time-series modeling on the input amplitude stream to obtain multi-scale amplitude features. In the phase feature extraction branch, the phase stream extracted from the STFT spectrum is passed through a multi-stage encoder U containing a U-Net module. 2 -Encoder performs hierarchical encoding on the input phase stream to obtain phase structure features.

[0035] Next, cross-current feature fusion is performed on the multi-scale amplitude features and phase structure features to obtain cross-current fused features. Before performing cross-current feature fusion, it is necessary to calculate the number of feature vectors of the third axis and the length and width of the feature vectors in the multi-scale amplitude features and phase structure features to ensure that the dimensions of the multi-scale amplitude features and phase structure features are consistent. Finally, feature mapping is performed on the cross-stream fusion features based on the amplitude mapping branch and the phase mapping branch, respectively. Specifically, in the amplitude mapping branch, the cross-stream fusion features are processed sequentially through a 1x1 convolutional layer, a Bidirectional Long Short-Term Memory (BiLSTM) layer, and fully connected layers (FC600, FC600, FC541) for feature mapping, and the output is the amplitude estimation stream. In the phase mapping branch, the cross-stream fusion features are processed sequentially through 1x1 convolution and batch normalization, and the output is the phase estimation stream.

[0036] For example, in step S121, the step of using the amplitude feature extraction branch to perform time-series modeling on the input amplitude stream to obtain multi-scale amplitude features includes: Step S201: The input amplitude stream is input into the first multi-level coding layer in chronological order. The first coding preprocessing module of the first multi-level coding layer is used to preprocess the input amplitude stream to obtain the preprocessed amplitude features. At the same time, the coding layers of the first U-shaped network structure are used to extract multi-scale features from the preprocessed amplitude features, and the extracted amplitude features and the preprocessed amplitude features are fused to obtain the first amplitude coding features. And, the amplitude coding features corresponding to each coding layer are saved. In step S201, the first multi-level coding layer is a U²-Encoder structure.

[0037] The first U-shaped network structure mentioned above is the U-Net block in the U²-Encoder.

[0038] For details, please refer to Figure 3A As shown, the input amplitude stream is input into the first multi-level coding layer in chronological order; wherein, the first multi-level coding layer includes a first coding preprocessing module and a first U-shaped network structure, which are composed of a two-dimensional gated linear unit (2D-GLU), a normalization layer (Norm), and a parameterized modified linear unit (PReLU) in sequence.

[0039] Specifically, the first preprocessing module performs gated filtering, amplitude normalization, and nonlinear transformation on the input amplitude stream to obtain preprocessed amplitude features. Then, each coding layer in the first U-shaped network structure performs downsampling convolution operations on the preprocessed amplitude features, progressively reducing the time-frequency resolution of the amplitude stream to extract multi-scale amplitude coding features that encompass both local and global data at different scales. Finally, the output of the first U-shaped network structure is fused with the preprocessed amplitude features to obtain the first amplitude coding feature. Simultaneously, the amplitude coding features corresponding to each coding layer in the first U-shaped network structure are saved for subsequent skip connections.

[0040] Step S202: Input the first amplitude coding feature into the temporal convolutional structure, and model the context information of the first amplitude coding feature in the time dimension through the spatiotemporal convolutional structure to obtain the amplitude enhancement feature; For details, please refer to Figure 3BAs shown, the spatio-temporal convolution structure is composed of multiple stacked temporal convolution modules (S-TCM) in series. The input end of each S-TCM is connected to the output end of the previous module, forming a deep stacked serial structure. The first amplitude encoded feature input enters the first S-TCM from the left side of the figure, and after the local temporal feature is extracted by the module, it is output to the next S-TCM. Multiple S-TCMs sequentially perform layer-by-layer convolution processing on the input features in the time dimension to model the context information between consecutive time frames with different time receptive fields, thereby capturing long temporal dependencies across multiple time frames. Finally, the amplitude enhanced feature corresponding to the first amplitude encoded feature is output in the S-TCM on the far right as the input of the subsequent decoding operation.

[0041] Specifically, the first amplitude encoded feature is input into a temporal convolution structure composed of multiple stacked temporal convolution modules; the temporal convolution structure models the context information between consecutive time frames by applying dilated convolution, gated convolution or causal convolution in the time dimension. Each S-TCM extracts temporal dependency features in a multi-scale time window with an increasing dilation factor, thereby capturing long temporal dependencies across multiple time frames and generating amplitude enhanced features containing speech evolution trends and dynamic change patterns.

[0042] Step S203, using a multi-level decoding layer to perform level-by-level upsampling reconstruction and jump connection fusion processing on the amplitude enhanced feature and the amplitude encoded feature corresponding to each level of the encoding layer, to obtain a multi-scale amplitude feature; wherein the multi-scale amplitude feature contains local amplitude features and global time structure features.

[0043] For example, in step S203, the multi-level decoding layer is used to perform level-by-level upsampling reconstruction and jump connection fusion processing on the amplitude enhanced feature and the amplitude encoded feature corresponding to each level of the encoding layer to obtain a multi-scale amplitude feature, including: Step S301, after pre-processing the amplitude enhanced feature using a decoding preprocessing module in the first multi-level decoding layer, performing upsampling processing on the amplitude enhanced feature based on the first decoding layer to obtain the first amplitude decoding feature corresponding to the first decoding layer; Specifically, referring to Figure 3C As shown, the first multi-level decoding layer includes a decoding preprocessing module composed of a two-dimensional inverse gated linear unit (2D-DeGLU), a normalization layer (Norm) and a parametric rectified linear unit (PReLU), and an internal U-Net block.

[0044] Specifically, the input amplitude enhancement features are first processed by a decoding preprocessing module through inverse gated filtering, amplitude normalization, and nonlinear transformation to obtain preprocessed amplitude enhancement features. Then, the preprocessed amplitude enhancement features are upsampled and convolved using the decoding layers within the U-Net block to gradually restore the time-frequency resolution. Finally, the output of the U-Net block is fused with the preprocessed amplitude enhancement features to obtain the first amplitude decoding feature.

[0045] Step S302: Perform a same-scale skip connection between the first amplitude decoding feature and the second amplitude coding feature corresponding to the saved first-level coding layer to obtain the fused second amplitude decoding feature. Step S303: The second amplitude decoding feature is used as the input of the next level decoding layer. Upsampling and same-scale skip connection processing are performed sequentially to restore the time-frequency resolution and integrate the amplitude decoding features of different scales to obtain multi-scale amplitude features.

[0046] For example, in step S121, the hierarchical encoding of the phase stream using the phase feature extraction branch to obtain phase structure features includes: Step S401: Input the phase stream into the second coding preprocessing module of the first coding layer of the second multi-level coding layer in chronological order. Use the second coding preprocessing module to perform gated filtering, amplitude normalization and nonlinear transformation on the input features to obtain the preprocessed phase features. Step S402: Using the second U-shaped network structure, multi-scale feature extraction is performed on the preprocessed phase features to obtain the phase semantic representation; Step S403: The phase semantic representation and the preprocessed phase features are fused to obtain the phase structure features.

[0047] For details, please refer to Figure 3A As shown, the second multi-level coding layer has the same structure as the first multi-level coding layer, but the model parameters are different. The input phase stream is input into the second multi-level coding layer in chronological order; wherein, the second multi-level coding layer includes a second coding preprocessing module and a second U-shaped network structure, which are composed of a second gated linear unit (2D-GLU), a normalization layer (Norm), and a parameterized corrected linear unit (PReLU) in sequence.

[0048] Specifically, first, the second encoding preprocessing module is used to perform gating filtering, amplitude normalization and nonlinear transformation on the input phase stream to obtain preprocessed phase features. Then, each level of the encoding layer in the second U-shaped network structure is used to perform down-sampling convolution operation on the preprocessed phase features, gradually reducing the time-frequency resolution of the phase stream, so as to extract multi-scale phase semantic representation with local and global characteristics at different scales. Finally, the output of the second U-shaped network structure is fused with the preprocessed phase features to obtain phase structure features.

[0049] For example, in step S122, the multi-scale amplitude features and the phase structure features are cross-flow feature fused to obtain cross-flow fused features, including: Step S501, the multi-scale amplitude features output by the amplitude feature extraction branch are aligned with the phase structure features output by the phase feature extraction branch; Step S502, the aligned multi-scale amplitude features and the phase structure features are superimposed to obtain cross-flow fused features.

[0050] Specifically, before the multi-scale amplitude features and the phase structure features are cross-fused and superimposed, the number of feature vectors and the length and width of the feature vectors of the third axis of the two need to be calculated, and the two are aligned to ensure that the multi-scale amplitude features and the phase structure features have consistent dimensions. The obtained multi-scale amplitude features and phase structure features with the same dimensions are linearly superimposed on the third axis (depth direction) of the tensor to realize the cross-fusion and superposition of the multi-scale amplitude features and the phase structure features, and the specific process is as follows:

[0051] Wherein, and respectively represent the multi-scale amplitude features and the phase structure features after alignment.

[0052] For example, in step S123, the amplitude mapping branch includes a first convolution layer, a bidirectional long short-term memory network layer and a multi-layer fully connected layer, and the phase mapping branch includes a second convolution layer and a normalization layer. The cross-flow fused features are respectively mapped by the amplitude mapping branch and the phase mapping branch to generate amplitude estimation flow and phase estimation flow, including: Step S601, using the first convolution layer to perform channel compression and linear mapping on the cross-flow fused features to obtain amplitude initial features; Step S602, using the input bidirectional long short-term memory network to model the context association across time frames for the amplitude initial features to obtain time series enhanced amplitude features; Step S603, based on the multi-layer fully connected layer, the time series enhanced amplitude features are nonlinearly mapped and dimensionally transformed to generate amplitude estimation flow; Specifically, referenceFigure 2 As shown, the cross-stream fusion features are first input into the first convolutional layer (Conv 1*1). The first convolutional layer performs a 1×1 linear convolution operation to compress and linearly map the input features, remove redundant channel information, and reconstruct a compact feature representation to obtain the initial amplitude features. This step ensures that the feature dimension received by the subsequent time series modeling module is consistent with the sequence structure.

[0053] Then, a bidirectional long short-term memory (BiLSTM) network is used to model the initial amplitude features in the time dimension. BiLSTM extracts contextual dependencies simultaneously along both forward and backward time-series paths, capturing dynamic structural information of speech across multiple time frames, thereby generating temporally enhanced amplitude features that reflect speech evolution trends and cross-frame correlations. This step enables the multi-scale amplitude features output by the decoder to achieve stronger temporal continuity and contextual consistency.

[0054] Finally, the temporal-enhanced amplitude features are input into a multilayer fully connected network. Through layer-by-layer nonlinear mapping and dimensionality transformation, the input features are gradually projected onto an amplitude estimation space with a shape consistent with the amplitude spectrum, ultimately generating an amplitude estimation stream. The multilayer fully connected layers include continuous high-dimensional to low-dimensional mapping structures, which can effectively integrate global temporal information and recover the amplitude estimate in the frequency dimension.

[0055] Step S604: The second convolutional layer is used to perform channel compression and linear mapping on the cross-flow fusion features to obtain the initial phase features; Step S605: Normalize the initial phase features based on the normalization layer to generate a phase estimation stream.

[0056] For details, please refer to Figure 2 As shown, firstly, a second convolutional layer (Conv 1*1) is used to perform channel compression and linear mapping on the cross-stream fusion features to reduce the number of channels in the phase branch input and generate compact initial phase features. The second convolutional layer employs a 1×1 convolutional structure, which enables linear combination and channel reconstruction of the input features without changing the time-frequency resolution.

[0057] Then, the initial phase features are normalized using a normalization layer to ensure that the phase features of different time frames and frequency channels maintain a uniform numerical range and distribution, thereby generating a phase estimation stream. The normalization layer is used to suppress phase amplitude differences between different channels, improving the stability and consistency of phase estimation.

[0058] For example, the method further includes: Step S701, mixing the noise data set and the speech data set to generate training data for training the initial dual-stream network model; and performing short-time Fourier transform on the mixed speech signal and the reference speech signal in the training data set respectively to generate corresponding initial amplitude flow, initial phase flow and reference amplitude flow, reference phase flow; Specifically, first, the short-time Fourier transform (STFT) spectrogram used for input model training is standardized. The speech enhancement data set used for model training is constructed in an AudioSet and AVSpeech cross-mixing manner. The noise data set AudioSet is a large-scale audio event data set, which contains more than 2,000,000 audio segments and 527 audio event categories, such as barking, traffic noise, music types, etc. The noise data set AVSpeech is a large-scale audio event data set, which contains more than 100,000 speech data and can cover diversified speakers and scenes.

[0059] First, extract the human voice audio segment Speech in the event audio segment Noice, mix them, and record the mixed noise Mixed and the clear speech Speech against <Mixed, Speech>. Then, select the parameters such as window function, window length, frame shift, and frequency range for STFT, and perform STFT transformation on the mixed noise Mixed and the clear speech Speech respectively. Finally, extract the initial amplitude flow and the initial phase flow from the STFT transformation corresponding to the mixed noise, and extract the reference amplitude flow and the reference phase flow from the STFT transformation corresponding to the clear.

[0060] Step S702, using the initial dual-stream network model to model the initial amplitude flow and the initial phase flow in time and space to generate the initial amplitude estimation flow and the initial phase estimation flow; In step S702, the initial dual-stream network model has the same model structure as the target dual-stream network model. The initial dual-stream network model is trained using the training data set, and when the training end condition is met, the target dual-stream network model for speech denoising is obtained. Reference Figure 2 As shown, the initial amplitude flow and the initial phase flow obtained based on the training set are input into the initial dual-stream network model to obtain the corresponding initial amplitude estimation flow and the initial phase estimation flow.

[0061] Step S703, calculating a joint loss according to the initial amplitude estimation flow, the initial phase estimation flow, and the reference amplitude flow, the reference phase flow; updating the model parameters of the initial dual-stream network model using the joint loss and an optimizer to obtain the target dual-stream network model for speech denoising; wherein the joint loss includes an amplitude loss and a phase loss, the amplitude loss is determined based on the initial amplitude estimation flow and the reference amplitude flow, and the phase loss is determined based on the initial phase estimation flow and the reference phase flow.

[0062] Specifically, first, the initial amplitude estimation flow and the initial phase estimation flow are compared with the reference amplitude flow and the reference phase flow corresponding to the clear speech Speech respectively to obtain the corresponding amplitude loss and phase loss; then, the joint loss is calculated based on the amplitude loss and the phase loss, and the joint loss function is minimized using the Adam optimizer for training, and the joint loss is fed back forward, wherein the learning rate is set to 0.0001; finally, the above training process is executed in a loop until the initial double-flow network model converges, and the relevant network model parameters are saved to obtain the target double-flow network model.

[0063] Specifically, the amplitude loss is determined based on the initial amplitude estimation flow and the reference amplitude flow. Wherein, the amplitude loss function (complex compression mean square error) is as follows:

[0064] Wherein, is the first predicted complex signal of the amplitude flow estimation; is the first standard complex signal corresponding to the noisy amplitude flow, and C is a compression function. Specifically, the phase loss is determined based on the initial phase estimation flow and the reference phase flow, wherein the phase difference loss function is as follows:

[0065]

[0066] Wherein, is the phase flow estimation, is the standard phase flow corresponding to the noisy phase flow, is used to eliminate periodic error.

[0067] Specifically, the joint loss is calculated based on the amplitude loss and the phase loss, wherein the joint loss function is as follows:

[0068] Wherein, are the weighting coefficients of the amplitude loss function and the phase loss function, respectively.

[0069] The method provided by the embodiment of the application not only reduces the dependence on pre-built models, but also enhances the feature extraction capability and generalization capability of the model, and has high robustness and practicality.

[0070] Specifically, in the model training process, the training data set is 30000 mixed speech data, and the background noise is 527 random event audios. The traffic, music environment white noise test results are shown in Table 1.​​

[0071] Table 1 Performance indicators of speech enhancement based on deep cross- feature extraction and feature cross-fusion on the test data set

[0072] From the performance indicators in Table 1, it can be found that the speech enhancement method based on deep cross-feature extraction and feature cross-fusion has excellent performance in the signal-to-noise ratio (SNR) improvement, peak signal-to-noise ratio (PSNR), speech quality (PESQ), and mean square error (MSE) of three mixed noises. This is because the target double-flow network model has the following characteristics: 1) In a complex background environment, the feature extraction diversity problem is solved by a jump network; 2) The feature extraction process is fully performed during the encoding and decoding process of the U2-Net network structure; 3) Under the premise of effectively ensuring the network depth combining the residual structure, the network gradient and convergence speed are optimized; 4) The stacking of space-time convolution effectively ensures the space-time feature extraction; 5) The fusion processing of amplitude and phase features makes the features fully utilized; 6) The error between the true label and the prediction result is calculated by the loss function, and the continuous jump and residual network parameters are continuously updated.

[0073] Further, in order to verify that this method is a relatively advanced speech enhancement method at present, the performance of this method is compared with other commonly used methods, and the results are shown in Table 2.

[0074] Table 2 Performance comparison of different speech enhancement methods

[0075] From the performance comparison in Table 2, it can be found that the present method exceeds the existing speech enhancement methods in the four indicators, and has a good application prospect.

[0076] In summary, the present application designs an efficient intent recognition method based on a pre-trained language model from the perspective of generalization ability and feature fusion extraction. The experimental results show that the speech enhancement method based on deep cross-feature extraction and feature cross-fusion proposed in the present application can utilize the features processed by the encoding and decoding network, so that the model can stably capture deep audio information, thereby effectively solving the audio mixed noise problem, the prediction result is good and the error is small, and the method can be applied to various noise scenes and has high practical value.

[0077] The beneficial effects of the present application are as follows: (1) Jump link: in the process of feature extraction, the U2 The Net concatenates the high-resolution spatial information of each layer of the encoder with the semantic features of the corresponding layer of the decoder, retains the information (such as boundaries and contours) of the input data, significantly improves the feature extraction details, and the feature concatenation of the network enhances the multi-scale feature learning ability; (2) Residual network: the residual network is used to iteratively process the soft prompt, so that the speech T-F information can be understood and processed in a deeper level. Compared with the traditional network structure, the addition of the residual network can effectively increase the model depth, thereby effectively reducing the occurrence of gradient disappearance and gradient explosion, and significantly improving the processing effect of complex speech information.

[0078] (3) Iteration of amplitude feature and phase feature: in order to reduce the relevance loss caused by feature separation and extraction as much as possible, the amplitude feature and the phase feature are calculated together before feature mapping in the present application, so as to ensure multi-information fusion and cross.

[0079] (4) Spatio-temporal convolution: the spatio-temporal convolution combines spatial (Spatial) and temporal (Temporal) information processing, further extracts the information extracted by the U2-Encoder, and enhances the modeling ability of the model for long sequence data by stacking multiple layers of Temporal Block based on the standard TCN.

[0080] It should be noted that the above-described figures are only schematic representations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to limit the purpose. It is easy to understand that the processes shown in the above-described figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0081] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, such division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units.

[0082] Figure 4 A schematic diagram of an electronic device suitable for implementing embodiments of the present application is shown.

[0083] It should be noted that, Figure 4 The electronic device 1000 shown is only an example, and should not limit the functions and use range of the embodiments of the present application.

[0084] As Figure 4As shown, the electronic device 1000 includes a central processing unit (CPU) 1001 which can perform various appropriate actions and processes in accordance with a program stored in a read-only memory (ROM) 1002 or a program loaded from the storage section 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004. Further, the electronic device 1000 includes an FPGA device, a SOC device.

[0085] Connected to the I / O interface 1005 are an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as necessary. A removable media 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1010 as necessary, so that a computer program read out therefrom is installed in the storage section 1008 as necessary.

[0086] In particular, according to embodiments of the present application, the processes described below with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a storage medium, the computer program containing program code for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication section 1009, and / or installed from the removable media 1011. When the computer program is executed by the central processing unit (CPU) 1001, various functions defined in the system of the present application are performed.

[0087] In particular, the electronic device described above can be an onboard intelligent electronic device.

[0088] It should be noted that the storage medium shown in the embodiments of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the above two. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any storage medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or component. The program code contained in the storage medium can be transmitted by any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination of the above.

[0089] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a part of code containing one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0090] The units described in the embodiments of the present application can be implemented by software, or by hardware, or by a combination of software and hardware. The units described may

[0091] It should be noted that, as another aspect, the present application also provides a storage medium, which can be included in an electronic device, or can exist independently without being assembled into the electronic device. The storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to implement the method described in the embodiments. For example, the electronic device can implement each step of the method as shown in Figure 1

[0092] In one embodiment, the present application provides a computer program product, including a computer program which, when executed by a processor, implements the steps in the above method embodiments.

[0093] In addition, the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of the processes. In addition, it is also easy to understand that the processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0094] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present application cover any and all variations of the application that come within the scope of the claims and their equivalents. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0095] It should be understood that the present application is not limited to the precise construction that has been described above and illustrated in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the present application. The scope of the present application is limited only by the appended claims.​

Claims

1. A speech denoising method based on deep cross-feature extraction and cross-feature fusion, characterized in that, The method comprises: performing short-time Fourier transform on the to-be-processed voice signal to obtain an input amplitude stream and an input phase stream; modeling the input amplitude stream and the input phase stream based on a double-stream network model to generate a target amplitude estimation stream and a target phase estimation stream corresponding to the to-be-processed voice signal; wherein the target double-stream network model comprises an amplitude feature extraction branch and a phase feature extraction branch, the amplitude feature extraction branch is used to extract multi-scale amplitude features of the input amplitude stream, and the phase feature extraction branch is used to perform hierarchical coding on the input phase stream to extract phase structure features of the input phase stream; generating time-frequency estimation information based on the target amplitude estimation stream and the target phase estimation stream; and performing inverse short-time Fourier transform on the time-frequency estimation information in combination with a sampling rate to obtain a target voice signal; wherein the target voice signal is the to-be-processed voice signal after noise reduction.

2. The method of claim 1, wherein, The modeling of the input amplitude stream and the input phase stream based on the double-stream network model to generate the target amplitude estimation stream and the target phase estimation stream corresponding to the to-be-processed voice signal comprises: performing time series modeling on the input amplitude stream by using the amplitude feature extraction branch to obtain multi-scale amplitude features; and performing hierarchical coding on the input phase stream by using the phase feature extraction branch to obtain phase structure features; performing cross-stream feature fusion on the multi-scale amplitude features and the phase structure features to obtain cross-stream fusion features; performing feature mapping on the cross-stream fusion features based on the amplitude mapping branch and the phase mapping branch to generate an amplitude estimation stream and a phase estimation stream.

3. The method of claim 2, wherein, The time series modeling on the input amplitude stream by using the amplitude feature extraction branch to obtain multi-scale amplitude features comprises: inputting the input amplitude stream in a time sequence into a first multi-level coding layer, pre-processing the input amplitude stream by using a first coding pre-processing module of the first multi-level coding layer to obtain pre-processed amplitude features; at the same time, performing multi-scale feature extraction on the pre-processed amplitude features by using each coding layer in a first U-shaped network structure, and fusing the extracted amplitude features with the pre-processed amplitude features to obtain first amplitude coding features; and saving the amplitude coding features corresponding to each coding layer; inputting the first amplitude coding features into a time series convolution structure, and modeling the context information of the first amplitude coding features in the time dimension by using a space-time convolution structure to obtain amplitude enhancement features; performing step-by-step upsampling reconstruction and jump connection fusion processing on the amplitude enhancement features and the amplitude coding features corresponding to each coding layer by using a multi-level decoding layer to obtain multi-scale amplitude features; wherein the multi-scale amplitude features comprise local amplitude features and global time structure features.

4. The method of claim 3, wherein, The step-by-step upsampling reconstruction and jump connection fusion processing on the amplitude enhancement features and the amplitude coding features corresponding to each coding layer by using the multi-level decoding layer to obtain the multi-scale amplitude features comprises: performing upsampling processing on the amplitude enhancement features by using a first decoding layer of a first multi-level decoding layer to obtain first amplitude decoding features corresponding to the first decoding layer; performing same-scale jump connection on the first amplitude decoding features and the second amplitude coding features corresponding to the first coding layer saved to obtain fused second amplitude decoding features; performing step-by-step upsampling reconstruction and jump connection fusion processing on the phase enhancement features and the phase coding features corresponding to each coding layer by using a multi-level decoding layer to obtain multi-scale phase features; wherein the multi-scale phase features comprise local phase features and global time structure features. The step-by-step upsampling reconstruction and jump connection fusion processing on the phase enhancement features and the phase coding features corresponding to each coding layer by using the multi-level decoding layer to obtain the multi-scale phase features comprises: performing upsampling processing on the phase enhancement features by using a first decoding layer of a first multi-level decoding layer to obtain first phase decoding features corresponding to the first decoding layer; performing same-scale jump connection on the first phase decoding features and the second phase coding features corresponding to the first coding layer saved to obtain fused second phase decoding features; and performing step-by-step upsampling reconstruction and jump connection fusion processing on the phase enhancement features and the phase coding features corresponding to each coding layer by using a multi-level decoding layer to obtain multi-scale phase features; wherein the multi-scale phase features comprise local phase features and global time structure features. The second amplitude decoding feature is taken as an input of a next-level decoding layer, and up-sampling and same-scale skip connection processing are sequentially performed to recover time-frequency resolution and integrate amplitude decoding features of different scales, thereby obtaining multi-scale amplitude features.

5. The method of claim 2, wherein, The phase feature extraction branch is used to hierarchically encode the phase stream to obtain a phase structure feature, including: The phase stream is input into a second encoding preprocessing module of a first encoding layer of a second multi-level encoding layer in chronological order, the second encoding preprocessing module is used to perform gated filtering, amplitude normalization and nonlinear transformation on the input feature to obtain a preprocessed phase feature; The second U-shaped network structure is used to perform multi-scale feature extraction on the preprocessed phase feature to obtain a phase semantic representation; The phase semantic representation is fused with the preprocessed phase feature to obtain the phase structure feature.

6. The method of claim 3, wherein, The multi-scale amplitude feature and the phase structure feature are cross-stream feature fused to obtain a cross-stream fusion feature, including: The multi-scale amplitude feature output by the amplitude feature extraction branch is aligned with the phase structure feature output by the phase feature extraction branch; The aligned multi-scale amplitude feature and the phase structure feature are superimposed to obtain the cross-stream fusion feature.

7. The method of claim 6, wherein, The amplitude mapping branch includes a first convolutional layer, a bidirectional long short-term memory network layer and a multi-layer fully connected layer, the phase mapping branch includes a second convolutional layer and a normalization layer, and the cross-stream fusion feature is mapped by the amplitude mapping branch and the phase mapping branch respectively to generate an amplitude estimation stream and a phase estimation stream, including: The first convolutional layer is used to perform channel compression and linear mapping on the cross-stream fusion feature to obtain an amplitude initial feature; The bidirectional long short-term memory network is used to model the context association across time frames for the amplitude initial feature to obtain a time series enhanced amplitude feature; The multi-layer fully connected layer is used to perform nonlinear mapping and dimension transformation on the time series enhanced amplitude feature to generate the amplitude estimation stream; The second convolutional layer is used to perform channel compression and linear mapping on the cross-stream fusion feature to obtain a phase initial feature; The normalization layer is used to normalize the phase initial feature to generate the phase estimation stream.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: mixing the noise data set and the speech data set to generate training data for training the initial dual-stream network model, and performing short-time Fourier transform on the mixed speech signal and the reference speech signal in the training data set respectively to generate corresponding initial amplitude stream, initial phase stream, reference amplitude stream and reference phase stream; the initial dual-stream network model is used to model the initial amplitude stream and the initial phase stream in space and time to generate an initial amplitude estimation stream and an initial phase estimation stream; a joint loss is calculated according to the initial amplitude estimation stream, the initial phase estimation stream and the reference amplitude stream and the reference phase stream, the model parameters of the initial dual-stream network model are updated by using the joint loss and an optimizer to obtain a target dual-stream network model for speech denoising; wherein the joint loss includes an amplitude loss and a phase loss, the amplitude loss is determined based on the initial amplitude estimation stream and the reference amplitude stream, and the phase loss is determined based on the initial phase estimation stream and the reference phase stream.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored executable program, wherein the executable program, when executed, controls a device in which the storage medium is located to perform the method of any one of claims 1 to 8.

10. A computer program product, characterised in that, A computer program which, when executed by a processor, implements the method of any one of claims 1 to 8.

Images