A method and device for monophonic speech enhancement based on dual-branch network

Abstract

The present invention relates to the technical field of speech enhancement, and in particular to a monophonic speech enhancement method and device based on a dual-branch network. The present invention discloses a monophonic speech enhancement method based on a dual-branch network, firstly converting the noisy speech into the original speech spectrum, and introducing a decomposition strategy to decouple it into the original amplitude spectrum and the original complex spectrum; then based on the speech enhancement network with a dual-branch structure, extracting features from the original amplitude spectrum and the original complex spectrum in parallel, and obtaining an enhanced complex spectrum through information interaction processing; then inverse processing is performed based on the enhanced complex spectrum and the original speech spectrum, so as to obtain enhanced speech. Through simulation comparison on existing public data sets, it can be seen that the method of the present invention can further compress the complexity level of Gb / s while maintaining comparable performance, and achieve an average reduction in computational complexity of 8.3 times. The present invention solves the problem of high computational complexity of existing SE solutions.

Description

Technical Field

[0001] The present invention relates to the technical field of speech enhancement, and more specifically to: 1. a monophonic speech enhancement method based on a dual-branch network; 2. a monophonic speech enhancement device based on a dual-branch network. Background Art

[0002] Speech enhancement (SE) is a technique for recovering pure speech from noisy environments. The degradation of speech quality caused by background noise is not only perceptually annoying, but also significantly impairs the performance of automatic speech recognition (ASR). In addition, SE is indispensable in smart devices, in-vehicle systems, and home automation. And with the rapid popularity of online meetings, the demand for real-time SE solutions has surged, emphasizing the necessity of techniques that are both effective and computationally efficient.

[0003] Existing SE solutions can be roughly divided into two categories, namely the time domain and the time-frequency (TF) domain. Although existing SE solutions have good appearances, their high computational complexity may hinder the practical application of SE. Specifically: 1. As a front-end task, the impact of SE on downstream tasks (such as ASR) should be considered; 2. Effective deployment in edge or resource-constrained environments, such as online meetings and real-time communications, requires low-complexity SE solutions; 3. The computational complexity of sequence modeling can be large, and some transformer-based models reach hundreds of gigaflops (G / s) due to their quadratic attention mechanism, which will hinder practical applications. Summary of the invention

[0004] Based on this, it is necessary to provide a monophonic speech enhancement method and device based on a dual-branch network to address the problem of high computational complexity of existing SE solutions.

[0005] The present invention is implemented by the following technical solutions:

[0006] In a first aspect, the present invention discloses a monophonic speech enhancement method based on a dual-branch network, comprising:

[0007] Step 1: Obtain noisy speech and perform Fourier transform on the noisy speech to obtain the original speech spectrum Z , then Z Decouple to get the original complex spectrum X 0. Original Amplitude Spectrum Y 0.

[0008] Step 2: X 0. Y 0 is input into the trained speech enhancement network for processing to obtain the enhanced complex spectrum Z' .

[0009] Among them, the speech enhancement network includes: 2 frequency band segmentation units, 2 feature extraction units, 1 sequence modeling unit, 1 complex spectrum feature modeling unit, 1 amplitude spectrum feature modeling unit, and 1 frequency band synthesis unit.

[0010] The first frequency band division unit is used to X 0 performs frequency band segmentation processing to obtain the intermediate complex spectrum characteristics A ;

[0011] The second frequency band division unit is used to Y 0 performs frequency band segmentation processing to obtain the intermediate amplitude spectrum characteristics E ;

[0012] The first feature extraction part is used based on A , E Extracting intermediate features B ;

[0013] The second feature extraction part is used based on A , E Extracting intermediate features F ;

[0014] The sequence modeling department is built on Mamba and is used to B , F Perform sequence modeling to obtain intermediate features C , intermediate features H ;

[0015] The complex spectrum feature modeling unit is used to C Perform feature modeling to obtain the final complex spectrum features D ;

[0016] The amplitude spectrum feature modeling part is used to H Perform feature modeling to obtain the final amplitude spectrum characteristics L ;

[0017] The frequency band synthesis section is used to D , L Perform synthesis to obtain Z' .

[0018] Step 3: Z and Z' Masking is performed first, and then inverse Fourier transform is performed to obtain enhanced speech.

[0019] This monophonic speech enhancement method based on a dual-branch network implements the method or process according to an embodiment of the present disclosure.

[0020] In a second aspect, the present invention discloses a monophonic speech enhancement device based on a dual-branch network, which uses the monophonic speech enhancement method based on a dual-branch network disclosed in the first aspect.

[0021] The monophonic speech enhancement device based on a dual-branch network comprises: a pre-processing module, a speech enhancement module and a post-processing module.

[0022] The preprocessing module is used to obtain noisy speech and perform Fourier transform on the noisy speech to obtain the original speech spectrum. Z , then Z Decouple to get the original complex spectrum X 0. Original Amplitude Spectrum Y 0.

[0023] The speech enhancement module is used to X 0. Y 0 is input into the trained speech enhancement network for processing to obtain the enhanced complex spectrum Z' .

[0024] The post-processing module is used to Z and Z' Masking is performed first, and then inverse Fourier transform is performed to obtain enhanced speech.

[0025] The monophonic speech enhancement device based on the dual-branch network implements the method or process according to the embodiment of the present disclosure.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. The present invention first converts the noisy speech into the original speech spectrum, and introduces a decomposition strategy to decouple it into the original amplitude spectrum and the original complex spectrum; then, based on the speech enhancement network with a dual-branch structure, features are extracted from the original amplitude spectrum and the original complex spectrum in parallel, and an enhanced complex spectrum is obtained through information interaction processing; then, inverse processing is performed based on the enhanced complex spectrum and the original speech spectrum to obtain enhanced speech. Through simulation comparison with the existing SE solution on the existing public data set, it can be seen that the method of the present invention can further compress the complexity level of Gb / s while maintaining comparable performance, achieving an average reduction of 8.3 times in computational complexity.

[0028] 2. The speech enhancement network designed in the present invention adopts two frequency band segmentation parts to segment the amplitude spectrum and the complex spectrum respectively to compress the frequency band dimension, thereby significantly reducing the amount of data processing; an information interaction layer is introduced to complement the intermediate information of the dual-branch process; and a sequence modeling part based on Mamba is introduced to perform sequence modeling, using its linear complexity to reduce the complexity of modeling and maintain the modeling effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 A data flow diagram of a monophonic speech enhancement method based on a dual-branch network provided in Embodiment 1 of the present invention;

[0031] Figure 2 for Figure 1 The data flow diagram of the first frequency band division part and the second frequency band division part;

[0032] Figure 3 for Figure 1 The data flow diagram of the first feature extraction unit and the second frequency band division unit;

[0033] Figure 4 for Figure 1 Data flow diagram of the sequence modeling department;

[0034] Figure 5 for Figure 3 , Figure 4 Data flow diagram of the information interaction layer;

[0035] Figure 6 for Figure 4 Data flow diagram of the FT-Mamba department;

[0036] Figure 7 for Figure 6 Data flow diagram of the F-Mamba subsection;

[0037] Figure 8 for Figure 6 Data flow diagram of the T-Mamba sub-unit;

[0038] Fig. 9 for Figure 1 The data flow diagram of the complex spectrum feature modeling unit;

[0039] Fig.10 for Fig. 9 Data flow diagram of F1E-Real layer and F2E-Imag layer;

[0040] Fig.11 for Figure 1 Data flow diagram of the mid-amplitude spectrum feature modeling unit;

[0041] Fig.12 for Fig.11Data flow diagram of the F2E-Mask layer;

[0042] Fig.13 for Figure 1 Data flow diagram of the IF band synthesis unit. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] It should be noted that when a component is referred to as being "mounted on" another component, it may be directly on the other component or there may be a central component. When a component is considered to be "set on" another component, it may be directly set on the other component or there may be a central component at the same time. When a component is considered to be "fixed to" another component, it may be directly fixed on the other component or there may be a central component at the same time.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "or / and" used herein includes any and all combinations of one or more of the related listed items.

[0046] Example 1

[0047] See also Figure 1 , Figure 1 The data flow diagram of the monophonic speech enhancement method based on the dual-branch network is shown, which actually also shows the process of the monophonic speech enhancement method based on the dual-branch network.

[0048] like Figure 1 As shown, the monophonic speech enhancement method based on the dual-branch network includes the following steps:

[0049] Step 1: Obtain noisy speech and perform Fourier transform on the noisy speech to obtain the original speech spectrum Z , then Z Decouple to get the original complex spectrum X 0. Original Amplitude Spectrum Y 0.

[0050] Noisy speech can be represented as Voice(t,f), which includes clean speech signal(t,f) and noise noise(t,f); where t represents the time frame index and f represents the frequency index.

[0051] Typically, noisy speech is sampled at 16kHz and framed using a 20ms Hamming window with a 50% overlap between frames.

[0052] Step 2: X 0. Y 0 is input into the trained speech enhancement network (BSDB-Net for short) for processing to obtain the enhanced complex spectrum Z' .

[0053] For the speech enhancement network, it includes: 2 frequency band segmentation units, 2 feature extraction units, 1 sequence modeling unit, 1 complex spectrum feature modeling unit, 1 amplitude spectrum feature modeling unit, and 1 frequency band synthesis unit.

[0054] like Figure 1 As shown:

[0055] The first frequency band division unit is used to X 0 performs frequency band segmentation processing to obtain the intermediate complex spectrum characteristics A ;

[0056] The second frequency band division unit is used to Y 0 performs frequency band segmentation processing to obtain the intermediate amplitude spectrum characteristics E ;

[0057] The first feature extraction part is used based on A , E Extracting intermediate features B ;

[0058] The second feature extraction part is used based on A , E Extracting intermediate features F ;

[0059] The sequence modeling department is built on Mamba and is used to B , F Perform sequence modeling to obtain intermediate features C , intermediate features H ;

[0060] The complex spectrum feature modeling unit is used to C Perform feature modeling to obtain the final complex spectrum features D ;

[0061] The amplitude spectrum feature modeling part is used to H Perform feature modeling to obtain the final amplitude spectrum characteristicsL ;

[0062] The frequency band synthesis section is used to D , L Perform synthesis to obtain Z' .

[0063] in:

[0064] The first frequency band segmentation unit, the first feature extraction unit, the sequence modeling unit, and the complex spectrum feature modeling unit constitute the CEN branch. The CEN branch focuses on X 0 for processing;

[0065] The second frequency band segmentation unit, the second feature extraction unit, the sequence modeling unit, and the amplitude spectrum feature modeling unit constitute the MEN branch. The MEN branch focuses on Y 0 for processing.

[0066] The following is a description of each part:

[0067] (I) The two frequency band divisions are designed to reduce X 0. Y 0 dimension, reducing data complexity.

[0068] The frequency band division unit can be used Figure 2 The design includes: 1 spectrum segmentation layer, 1 normalization layer, 1 Layer layer, and 1 dimension merging layer.

[0069] 101, in the first frequency band division:

[0070] The spectrum segmentation layer is used to X 0 is split according to the preset specifications to obtain M Original complex band X 1;

[0071] The normalization layer is used to X 1 is normalized to obtain M Normalized complex spectral features X 2;

[0072] Layer is used to X 2 Unify the frequency dimension to obtain M Unified complex spectral features X 3;

[0073] The dimension merging layer is used to X 3 Merge in the frequency dimension to obtain A ;

[0074] in, X 1=[ x 11 ,x 12 ,…, x 1M ], x 1m express X 1, the mth original complex spectral band, m∈[1, M ];

[0075] X 2=[ x 21 , x 22 ,…, x 2M ], x 2m express X The mth normalized complex spectrum feature in 2;

[0076] X 3=[ x 31 , x 32 ,…, x 3M ], x 3m express X The mth unified complex spectral feature in 3.

[0077] That is, the processing of the first frequency band division unit can be formulated as:

[0078] ;

[0079] ;

[0080] ;

[0081] ;

[0082] In the formula, Split (.) indicates the processing process of the spectrum segmentation layer; FC (.) indicates the processing of the Layer layer. LN (.) represents the processing of the normalization layer, and Concat(.) represents the processing of the dimension merging layer.

[0083] 102, in the second frequency band division part:

[0084] The spectrum segmentation layer is used to Y 0 is split according to the preset specifications to obtain M Original Amplitude Bands Y 1;

[0085] The normalization layer is used to Y 1 is normalized to obtain M Normalized amplitude spectrum features Y 2;

[0086] Layer is used to Y 2 Unify the frequency dimension to obtain M Unified Amplitude Spectrum Features Y 3;

[0087] The dimension merging layer is used to Y 3 Merge in the frequency dimension to obtain E ;

[0088] in, Y 1=[ y 11 , y 12 ,…, y 1M ], y 1m express Y The mth original amplitude spectrum band in 1;

[0089] Y 2=[ y 21 , y 22 ,…, y 2M ], y 2m express Y The mth normalized amplitude spectrum feature in 2;

[0090] Y 3=[ y 31 , y 32 ,…, y 3M ], y 3m express Y The mth unified amplitude spectrum feature in 3.

[0091] That is, the processing of the second frequency band division unit can be formulated as:

[0092] ;

[0093] ;

[0094] ;

[0095] .

[0096] In the formula, Split (.) indicates the processing of the spectrum segmentation layer; FC (.) indicates the processing of the Layer layer. LN (.) represents the processing of the normalization layer, and Concat(.) represents the processing of the dimension merging layer.

[0097] In this embodiment 1, M is 20, and the preset specifications are [2, 3, 3, 3, 3, 3, 8, 8, 8, 8, 8, 8, 8, 12, 16, 16, 16, 17]; after a series of processing by the first frequency band division unit, the frequency band dimension of the complex spectrum can be reduced from 161 to 20; after a series of processing by the second frequency band division unit, the frequency band dimension of the amplitude spectrum can be reduced from 161 to 20; this can not only better capture the characteristics of different frequency bands, but also greatly reduce the data complexity.

[0098] (ii) The two feature extraction parts are respectively intended to further extract the features of the complex spectrum and the amplitude spectrum for better subsequent modeling.

[0099] The feature extraction part can be used Figure 3 The design includes: 1 two-dimensional convolution layer, 1 normalization layer, 1 PRelu activation function, and 1 information interaction layer.

[0100] 201, in the first feature extraction unit:

[0101] The two-dimensional convolutional layer is used to A Perform feature extraction to obtain complex spectrum information a 1;

[0102] The normalization layer is used to a 1Normalize to get complex spectrum information a 2;

[0103] The PRelu activation function is used to a 2Introducing nonlinear features to obtain complex spectrum information a 3;

[0104] The information exchange layer is used to a 3. E Interactive processing to obtain B , which can prevent information loss under double branches.

[0105] That is to say, the processing of the first feature extraction part can be formulated as:

[0106] ;

[0107] ;

[0108] In the formula, Conv 2 d (.) represents the processing process of the two-dimensional convolutional layer; LN (.) represents the processing of the sub-normalization layer; PRelu(.) represents the processing of the PRelu activation function; Inter (.) represents the processing process of the information interaction layer.

[0109] 202, in the second feature extraction unit:

[0110] The two-dimensional convolutional layer is used to E Perform feature extraction to obtain amplitude spectrum information e 1;

[0111] The normalization layer is used to e 1Normalize to get the amplitude spectrum information e 2;

[0112] The PRelu activation function is used to e 2Introducing nonlinear features to obtain amplitude spectrum information e 3;

[0113] The information exchange layer is used to e 3. A Interactive processing to obtain F , which can prevent information loss under double branches.

[0114] That is to say, the processing of the second feature extraction part can be formulated as:

[0115] ;

[0116] ;

[0117] In the formula, Conv 2 d (.) represents the processing process of the two-dimensional convolutional layer; LN (.) represents the processing of the sub-normalization layer; PRelu(.) represents the processing of the PRelu activation function; Inter (.) represents the processing process of the information interaction layer.

[0118] (iii) The sequence modeling part aims to model in the time domain and time-frequency domain with linear complexity.

[0119] Sequence modeling can be done using Figure 4 The design comprises: N Mamba model parts with the same structure.

[0120] Taking the nth Mamba model unit as an example, it includes: 1 information interaction layer, 2 overlay layers, and 2 FT-Mamba units.

[0121] In the nth Mamba Model Department:

[0122] The information interaction layer is used to analyze the intermediate spectrum features W n-1 , intermediate spectrum characteristics w n-1 Interactive processing to obtain intermediate spectrum features U n ;

[0123] The first overlay is used to U n and W n-1 Superposition to obtain the intermediate spectrum characteristics U_B n ;

[0124] The second overlay is used to U n and w n-1 Superposition to obtain the intermediate spectrum characteristics U_F n ;

[0125] The first FT-Mamba section is used to U_B n Sequence modeling based on Mamba to obtain intermediate spectrum features W n ;

[0126] The second FT-Mamba section is used to U_F n Sequence modeling based on Mamba to obtain intermediate spectrum features w n ; n∈[1,N].

[0127] in, B As the intermediate spectrum feature W 0, F As the intermediate spectrum feature w 0;

[0128] Intermediate spectrum characteristics W N As C , intermediate spectrum characteristics w N As H .

[0129] That is, the processing of the nth Mamba model unit can be formulated as:

[0130] ;

[0131] ;

[0132] ;

[0133] In the formula, Inter (.) represents the processing process of the information interaction layer; FT _ Mamba (.) indicates the processing of the FT-Mamba department.

[0134] It should be noted that N A value of 6 is recommended to achieve a balance between network performance and complexity.

[0135] 301, it should be noted that the information interaction layer appears in both the two feature extraction units and the sequence modeling unit. Here is a supplementary explanation of the structure of the information interaction layer:

[0136] The information exchange layer can be used Figure 5 The design includes: 1 joint layer, 1 sub-2D convolution layer, 1 sub-normalization layer, and 1 sub-PRelu activation function. The information interaction layer has dual inputs and a single output.

[0137] In any information exchange layer:

[0138] The union layer is used to connect the two inputs of the information exchange layer together to obtain intermediate data G 1;

[0139] The sub-2D convolutional layer is used to transform G 1 Perform feature extraction to obtain intermediate data G 2;

[0140] The sub-normalization layer is used to G 2Perform normalization to obtain intermediate data G 3;

[0141] The PRelu activation function is used to G 3. Introduce nonlinear features to obtain a single output of the information interaction layer.

[0142] That is, the above process can be formulated as:

[0143] ;

[0144] ;

[0145] In the formula, Input1 and Input2 represent the dual inputs of the information interaction layer, and Output represents the single input of the information interaction layer; Cat (.) indicates the processing of the joint layer; Conv 2 d (.) represents the processing process of the sub-two-dimensional convolutional layer; LN (.) represents the processing process of the sub-normalization layer; PRelu(.) represents the processing process of the sub-PRelu activation function.

[0146] Then, the dual input of the information interaction layer in the first feature extraction unit is a 3. E , a single output is B ;

[0147] The dual input of the information interaction layer in the second feature extraction unit is e 3. A , a single output is F ;

[0148] The dual input of the information interaction layer in the nth Mamba model is W n-1 , w n-1 , a single output is U n .

[0149] 302, FT_Mamba section includes 5 subsections.

[0150] The 5 subsections can be set up in two ways:

[0151] The first type: the first sub-part is the normalization layer, the second sub-part is the F_Mamba part, the third sub-part is the superposition layer, the fourth sub-part is the T_Mamba part, and the fifth sub-part is the superposition layer;

[0152] The second type: the first sub-part is the normalization layer, the second sub-part is the T_Mamba part, the third sub-part is the superposition layer, the fourth sub-part is the F_Mamba part, and the fifth sub-part is the superposition layer.

[0153] In any FT_Mamba group:

[0154] The first sub-part is used to normalize the input of the FT_Mamba part to obtain intermediate data J 1;

[0155] The second subsection is used for J 1 Perform sequence modeling on the frequency dimension to obtain intermediate data J 2;

[0156] The third subsection is used to connect the input of the FT_Mamba section to J 2. Overlay to get intermediate data J 3;

[0157] The fourth subsection is used to J 3. Perform sequence modeling on the time dimension to obtain intermediate data J 4;

[0158] The fifth subsection is used to J 3 and J 4 are superimposed to obtain the output of the FT_Mamba section.

[0159] See also Figure 6 , which shows the structure of FT_Mamba for the first setting. The processing can be formulated as follows:

[0160] ;

[0161] ;

[0162] In the formula, In 0 means the input of FT_Mamba. Out 0 means the output of FT_Mamba department; LN (.) represents the processing of the normalization layer; F_Mamba (.) indicates the processing of the F_Mamba department; T_Mamba (.) indicates the processing of T_Mamba section.

[0163] The following are the descriptions of the F_Mamba section and the T_Mamba section:

[0164] 3021, F_Mamba department can adopt Figure 7 The design includes: 2 Transpose layers, 1 Unfold layer, 1 F_Mamba layer, and 1 one-dimensional transposed convolution layer.

[0165] In any F_Mamba club:

[0166] The first Transpose layer is used to convert the input of the F_Mamba part into a tensor to obtain intermediate data. R 1;

[0167] Unfold layer is used to R 1 Upsample to get intermediate data R 2;

[0168] The F_Mamba layer is used to communicate with Mamba via bidirectional R2 Perform sequence modeling to obtain intermediate data R 3;

[0169] The one-dimensional transposed convolutional layer is used to R 3 Perform one-dimensional transposed convolution to obtain intermediate data R 4;

[0170] The second Transpose layer is used to R 4 Perform tensor conversion to obtain the output of this F_Mamba part.

[0171] That is, the processing of the F_Mamba department can be formulated as:

[0172] ;

[0173] ;

[0174] In the formula, In 1 indicates the input of F_Mamba. Out 1 indicates the output of the F_Mamba part; Tran(.) indicates the processing of the Transpose layer; Unfold(.) indicates the processing of the Unfold layer; F_Ma (.) represents the processing of the F_Mamba layer; Deconv(.) represents the processing of the one-dimensional transposed convolution layer.

[0175] Among them, see Figure 7 , the F_Mamba layer includes: 1 flip layer, 2 Mamba layers, 2 Norm layers, 2 overlay layers, 1 Constant layer, and 1 Layer layer.

[0176] In any F_Mamba layer:

[0177] The first Mamba layer is used to perform sequence modeling on the input of the F_Mamba layer to obtain the intermediate data P1;

[0178] The first Norm layer is used to normalize P1 to obtain the intermediate data P2;

[0179] The first overlay layer is used to overlay the input of the F_Mamba layer with P2 to obtain the intermediate data P3;

[0180] The flip layer is used to reverse the input of the F_Mamba layer to obtain the intermediate data P4;

[0181] The second Mamba layer is used to model P4 to obtain the intermediate data P5;

[0182] The second Norm layer is used to normalize P5 to obtain the intermediate data P6;

[0183] The second overlay layer is used to overlay P4 and P6 to obtain intermediate data P7;

[0184] The Constant layer is used to adjust the tensors of P3 and P7 to obtain the intermediate data P8;

[0185] The Layer layer is used to perform linear mapping on P8 to obtain the output of the F_Mamba layer.

[0186] That is, the processing of the F_Mamba layer can be formulated as:

[0187] ;

[0188] ;

[0189] ;

[0190] In the formula, in 1 represents the input of the F_Mamba layer, out 1 represents the output of the F_Mamba layer; Ma(.) represents the processing process of the Mamba layer; LN(.) represents the processing process of the Norm layer; flip(.) represents the processing process of the flip layer; constant(.) represents the processing process of the Constant layer; FC (.) represents the processing process of the flip layer.

[0191] 3022, T_Mamba department can adopt Figure 8 The design includes: 2 Transpose layers, 1 one-dimensional padding layer, 1 Unfold layer, 1 T_Mamba layer, and 1 one-dimensional transposed convolution layer.

[0192] In any T_Mamba club:

[0193] The first Transpose layer is used to convert the input of the T_Mamba part into intermediate data. T 1;

[0194] One-dimensional filling layer is used to T 1 Fill data to get intermediate data T 2;

[0195] Unfold layer is used to T2 Perform abstract feature extraction to obtain intermediate data T 3;

[0196] The T_Mamba layer is used to pass the one-way MambaT 3 Perform sequence modeling to obtain intermediate data T 4;

[0197] The one-dimensional transposed convolutional layer is used to T 4 Perform one-dimensional transposed convolution to obtain intermediate data T 5;

[0198] The second Transpose layer is used to T 5 Perform tensor transformation on the output of the T_Mamba unit.

[0199] That is, the processing of the T_Mamba department can be formulated as:

[0200] ;

[0201] ;

[0202] In the formula, In 2 represents the input of T_Mamba. Out 2 represents the output of the T_Mamba part; Tran(.) represents the processing of the Transpose layer; Pad(.) represents the processing of the one-dimensional padding layer; Unfold(.) represents the processing of the Unfold layer; T_Ma (.) represents the processing process of the T_Mamba layer; Deconv(.) represents the processing process of the one-dimensional transposed convolution layer.

[0203] Among them, see Figure 8 ,T_Mamba layer includes: 1 Mamba layer, 1 Norm layer, 1 Layer layer, and 1 overlay layer.

[0204] In any T_Mamba layer:

[0205] The Mamba layer is used to model the input of the T_Mamba layer to obtain the intermediate data Q1;

[0206] The Norm layer is used to normalize Q1 to obtain the intermediate data Q2;

[0207] The Layer layer is used to perform linear mapping on Q2 to obtain the intermediate data Q3;

[0208] The superposition layer is used to superimpose the input of the T_Mamba layer with Q3 to obtain the output of the T_Mamba layer.

[0209] That is, the processing of the T_Mamba layer can be formulated as:

[0210] ;

[0211] In the formula, in 2 represents the input of the T_Mamba layer, out 2 represents the output of the T_Mamba layer; Ma(.) represents the processing process of the Mamba layer; LN(.) represents the processing process of the Norm layer; FC(.) represents the processing process of the Layer layer.

[0212] (iv) The complex spectrum modeling unit aims to restore the complex spectrum features to their original dimensions for subsequent synthesis.

[0213] The complex spectrum feature modeling part can be used Fig. 9 The design includes: 1 F1E-Real layer, 1 F2E-Imag layer, and 1 overlay layer.

[0214] In the complex spectrum feature modeling section:

[0215] The F1E-Real layer is used for C Perform feature modeling to obtain the real intermediate feature c1;

[0216] The F2E-Imag layer is used to C Perform feature modeling to obtain the imaginary intermediate feature c2;

[0217] The overlay layer is used to overlay c1 and c2 to obtain D .

[0218] That is to say, the processing process of the complex spectrum feature modeling part can be formulated as:

[0219] ;

[0220] Wherein, F1E_Real(.) represents the processing process of the F1E-Real layer; F2E_Imag(.) represents the processing process of the F2E-Imag layer.

[0221] 401, F1E-Real layer can be used Fig.10 The design includes: 3 two-dimensional convolutional layers, 1 Norm layer, 2 PRelu activation functions, and 1 Sigmoid activation function.

[0222] In the F1E-Real layer:

[0223] The first two-dimensional convolutional layer is used to C Upsampling is performed to obtain intermediate data u1;

[0224] The Norm layer is used to normalize u1 to obtain the intermediate data u2;

[0225] The first PRelu activation function is used to introduce nonlinear features into u2 to obtain intermediate data u3;

[0226] The second two-dimensional convolution layer is used to perform two-dimensional convolution processing on u3 to obtain the intermediate data u4;

[0227] Sigmoid activation function is used to introduce linear features into u4 to obtain intermediate data u5;

[0228] The third two-dimensional convolutional layer is used to extract features from U5 to obtain intermediate data u6;

[0229] The second PRelu activation function is used to introduce nonlinear features to u6 to obtain c1.

[0230] That is to say, the processing of the F1E-Real layer can be formulated as:

[0231] ;

[0232] ;

[0233] In the formula, PRelu(.) represents the processing process of the PRelu activation function; Conv 2 d (.) represents the processing process of the two-dimensional convolutional layer; LN (.) indicates the processing of the Norm layer; Sigmoid (.) represents the processing process of the Sigmoid activation function.

[0234] 402, F2E-Imag layer can be used Fig.10 The design includes: 4 two-dimensional convolutional layers, 1 Norm layer, 2 PRelu activation functions, 1 Sigmoid activation function, 1 Tanh activation function, and 1 overlay layer.

[0235] In the F2E-Imag layer:

[0236] The first two-dimensional convolutional layer is used to upsample C to obtain the intermediate data U1;

[0237] The Norm layer is used to normalize U1 to obtain the intermediate data U2;

[0238] The first PRelu activation function is used to introduce nonlinear features into U2 to obtain intermediate data U3;

[0239] The second two-dimensional convolution layer is used to perform two-dimensional convolution processing on U3 to obtain intermediate data U4;

[0240] Sigmoid activation function is used to introduce linear features into U4 to obtain intermediate data U5;

[0241] The third two-dimensional convolution layer is used to perform two-dimensional convolution processing on U3 to obtain intermediate data U6;

[0242] Tanh activation function is used to introduce linear features into U6 to obtain intermediate data U7;

[0243] The overlay layer is used to overlay U5 and U7 to obtain the intermediate data U8;

[0244] The fourth two-dimensional convolutional layer is used to extract features from U8 to obtain intermediate data U9;

[0245] The second PRelu activation function is used to introduce nonlinear features into U9 to obtain c2.

[0246] That is, the processing of the F2E-Imag layer can be formulated as:

[0247] ;

[0248] ;

[0249] ;

[0250] ;

[0251] In the formula, PRelu(.) represents the processing process of the PRelu activation function; Conv 2 d (.) represents the processing process of the two-dimensional convolutional layer; LN (.) indicates the processing of the Norm layer; Sigmoid (.) represents the processing of Sigmoid activation function; Tanh (.) indicates the processing of Tanh activation function.

[0252] (V) The amplitude spectrum modeling part aims to restore the complex spectrum features to the original dimension for subsequent synthesis.

[0253] The amplitude spectrum feature modeling part can be used Fig.11 The design includes: 1 F2E-Mask layer and 1 matrix multiplication layer.

[0254] In the amplitude spectrum feature modeling section:

[0255] The F2E-Mask layer is used to H Modeling to obtain the amplitude spectrum feature mask H' ;

[0256] The matrix multiplication layer is used to transform H' and E Perform matrix multiplication to getL .

[0257] That is to say, the processing process of the amplitude spectrum feature modeling part can be formulated as:

[0258] ;

[0259] Wherein, F2E-Mask(.) represents the processing process of the F2E-Mask layer.

[0260] The F2E-Mask layer can be used Fig.12 The design includes: 4 two-dimensional convolutional layers, 1 Norm layer, 2 PRelu activation functions, 1 Sigmoid activation function, 1 Tanh activation function, and 1 overlay layer.

[0261] In the F2E-Mask layer:

[0262] The first two-dimensional convolutional layer is used to upsample H to obtain the intermediate data h1;

[0263] The Norm layer is used to normalize h1 to obtain the intermediate data h2;

[0264] The first PRelu activation function is used to introduce nonlinear features into h2 to obtain intermediate data h3;

[0265] The second two-dimensional convolution layer is used to perform two-dimensional convolution processing on h3 to obtain intermediate data h4;

[0266] Sigmoid activation function is used to introduce linear features into h4 to obtain intermediate data h5;

[0267] The third two-dimensional convolution layer is used to perform two-dimensional convolution processing on h3 to obtain intermediate data h6;

[0268] Tanh activation function is used to introduce linear features into h6 to obtain intermediate data h7;

[0269] The overlay layer is used to overlay h5 and h7 to obtain intermediate data h8;

[0270] The fourth two-dimensional convolutional layer is used to extract features from h8 to obtain intermediate data h9;

[0271] The second PRelu activation function is used to introduce nonlinear features into h9 to obtain H' .

[0272] In other words, the processing of the F2E-Mask layer can be formulated as:

[0273] ;

[0274] ;

[0275] ;

[0276] ;

[0277] In the formula, PRelu(.) represents the processing process of the PRelu activation function; Conv 2 d (.) represents the processing process of the two-dimensional convolutional layer; LN (.) indicates the processing of the Norm layer; Sigmoid (.) represents the processing of Sigmoid activation function; Tanh (.) represents the processing of Tanh activation function.

[0278] (vi) The frequency band synthesis unit is intended to re-combine the divided frequency bands into a complex spectrum.

[0279] The frequency band synthesis unit can use Fig.13 The design includes: 1 overlay layer, 1 block decomposition layer, 1 normalization layer, 1 Layer layer, 1 Tanh activation function, 1 GLU layer, and 1 band merging layer.

[0280] In the frequency band synthesis section:

[0281] Overlays are used to D , L Superposition is performed to obtain a complex spectrum feature block V0;

[0282] The block decomposition layer is used to decompose V0 into blocks to obtain M The original complex spectral tensor V1;

[0283] The normalization layer is used to normalize V1 to obtain the normalized complex spectrum tensor V2;

[0284] The Layer layer is used to perform a linear transformation on V2 to obtain a double complex spectral tensor V3.

[0285] Tanh activation function is used to introduce linear features into V3 to obtain the activated complex spectrum tensor V4;

[0286] The GLU layer is used to restore the dimension of V4 to obtain the final segmentation band mask V5;

[0287] The frequency band merging layer is used to merge V5 in the frequency dimension to obtain Z' ;

[0288] in, V 1=[ v 11 , v12 ,…, v 1M ], v 1m express V The mth original complex spectral tensor in 1;

[0289] V 2=[ v 21 , v 22 ,…, v 2M ], v 2m express V The mth normalized complex spectral tensor in 2;

[0290] V 3=[ v 31 , v 32 ,…, v 3M ], v 3m express V The mth double complex spectral tensor in 3;

[0291] V 4=[ v 41 , v 42 ,…, v 4M ], v 4m express V The mth activation complex spectral tensor in 4;

[0292] V 5=[ v 51 , v 52 ,…, v 5M ], v 5m express V The mth final segmentation band mask in 5.

[0293] That is, the frequency band synthesis part can be formulated as:

[0294] ;

[0295] ;

[0296] ;

[0297] ;

[0298] ;

[0299] In the formula, Exp (.) indicates the processing of block disassembly layer; LN (.) represents the processing of the normalization layer; FC(.) represents the processing of the Layer layer; Tanh(.) represents the processing of the Tanh activation function; GLU (.) indicates the processing of the GLU layer; Merge (.) indicates the processing process of the band merging layer.

[0300] In this embodiment 1, M is 20, and the size of V5 after dimensionality reduction is [2, 3, 3, 3, 3, 3, 8, 8, 8, 8, 8, 8, 8, 12, 16, 16, 16, 17].

[0301] It should be noted that this method requires the use of a trained speech enhancement network with optimal network parameters. The speech enhancement network is generally trained based on an existing known speech data set to ensure the network training effect.

[0302] The "RI+Mag" loss function is used when training the speech enhancement network. Loss To supervise the optimization of phase and amplitude components simultaneously. Among them, the "RI+Mag" loss function Loss It is expressed as:

[0303] ;

[0304] In the formula, L RI represents the minimum mean square error loss function of the complex spectrum; L Mag represents the minimum mean square error loss function of the amplitude spectrum; β Represents the weight coefficient, which is usually 0.5.

[0305] Simulation Verification

[0306] The existing datasets WSJ0-SI84, DNS-Challenge, VoiceBank, and Demand are used.

[0307] 1. Based on WSJ0-SI84+DNS-Challenge, an ablation experiment is conducted on BSDB-Net to evaluate the effectiveness of the dual-branch structure design.

[0308] Remove the CEN branch of BSDB-Net and keep the MEN branch, and train the MEN branch alone, which corresponds to BSDB-MEN; remove the MEN branch of BSDB-Net and keep the CEN branch, and train the CEN branch alone, which corresponds to BSDB-CEN.

[0309] Comparing the performance of BSDB-MEN, BSDB-CEN, and BSDB-Net, the results are shown in Table 1:

[0310] Table 1. Comparison results of ablation experiments

[0311]

[0312] Among them, PESQ is a perceptual evaluation index for narrowband and broadband speech quality, and the larger the value, the better; ESTOI is an extended version of the short-time objective intelligibility STOI for measuring speech quality, and the larger the value, the better; SI-SDR is used to evaluate the degree of speech distortion, and the larger the value, the better.

[0313] As shown in Table 1, the dual-path structure outperforms the single-path structure in all indicators. This means that the coordinated efforts of the CEN branch and the MEN branch can improve the quality of the target speech—the CEN branch filters out the main noise for a rough estimate, while the MEN branch continuously supplements the speech information, thereby improving the overall performance of the system.

[0314] 2. A replacement experiment was conducted on BSDB-Net based on WSJ0-SI84+DNS-Challenge to evaluate the effectiveness of the sequence modeling design.

[0315] Replace the sequence modeling part of BSDB-Net with LSTM, which corresponds to BSDB-LSTM; replace the sequence modeling part of BSDB-Net with Transformer, which corresponds to BSDB-Transformer.

[0316] Comparing the performance of BSDB-LSTM, BSDB-Transformer, and BSDB-Net, the results are shown in Table 2:

[0317] Table 2 Comparison results of replacement experiments

[0318]

[0319] Among them, PESQ is a perceptual evaluation index for narrowband and broadband speech quality, and the larger the value, the better; MACs represents network complexity; Parameters represents the number of network parameters.

[0320] As shown in Table 2, BSDB-Net has greatly reduced the complexity of the network and the number of network parameters while maintaining the best performance. It should be noted that although the performance improvement of BSDB-Net is not obvious compared with BSDB-Transformer, the network complexity has been significantly reduced and the number of network parameters has been significantly reduced.

[0321] 3. Based on WSJ0-SI84+DNS-Challenge, the complexity of BSDB-Net is compared with 9 other models (including: ConvTasNet, DPRNN, DDAEC, LSTM, CRN, GCRN, DCCRN, FullSubNet, CTSNet, GaGNet). The results are shown in Table 3.

[0322] Table 3 Complexity comparison results

[0323]

[0324] Among them, PESQ is a perceptual evaluation index for narrowband and broadband speech quality, and the larger the value, the better; MACs represents network complexity.

[0325] As shown in Table 3, BSDB-Net has an average computational complexity about 8 times lower than other models and has good performance.

[0326] 4. Based on VoiceBank+Demand, the performance of BSDB-Net is compared with 14 other models (including SEGAN, MMSEGAN, MetricGAN, SRTNET, Wavenet, PHASEN, MHSA-SPK, DCCRN, TSTNN, S4DSE, FDFnet, CTS-net, GaGnet, and CompNet). The results are shown in Table 4.

[0327] Table 4 Performance comparison results

[0328]

[0329] Among them, PESQ is a perceptual evaluation index for narrowband and broadband speech quality, and the larger the value, the better; STOI is a short-time objective intelligibility for measuring speech quality, and the larger the value, the better; CSIG, CBAK, and COVL are all indicators for evaluating speech quality, and the larger the value, the better.

[0330] As shown in Table 4, BSDB-Net achieves an average improvement of 0.32, 1.5%, 0.31, 0.17, and 0.35 over other models in PESQ, STOI, CSIG, CBAK, and COVL, respectively, which also illustrates the general applicability of this method.

[0331] Example 2

[0332] This embodiment 2 provides a monophonic speech enhancement device based on a dual-branch network, which uses the monophonic speech enhancement method based on a dual-branch network disclosed in embodiment 1.

[0333] The monophonic speech enhancement device based on a dual-branch network comprises: a pre-processing module, a speech enhancement module and a post-processing module.

[0334] The preprocessing module is used to obtain noisy speech and perform Fourier transform on the noisy speech to obtain the original speech spectrum. Z , then Z Decouple to get the original complex spectrum X 0. Original Amplitude Spectrum Y 0.

[0335] The speech enhancement module is used to X 0. Y 0 is input into the trained speech enhancement network for processing to obtain the enhanced complex spectrum Z' .

[0336] The post-processing module is used to Z and Z' Masking is performed first, and then inverse Fourier transform is performed to obtain enhanced speech.

[0337] Since the present device uses the monophonic speech enhancement method based on the dual-branch network in Example 1, it also has the same effect and will not be repeated here.

[0338] Example 3

[0339] This embodiment 3 discloses a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the monophonic speech enhancement method based on a dual-branch network disclosed in embodiment 1 are implemented.

[0340] This embodiment 3 also discloses a readable storage medium, in which computer program instructions are stored. When the computer program instructions are read and executed by a processor, the steps of the monophonic speech enhancement method based on a dual-branch network disclosed in embodiment 1 are executed.

[0341] This embodiment 3 also discloses a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the monophonic speech enhancement method based on a dual-branch network disclosed in embodiment 1 are implemented.

[0342] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A monophonic speech enhancement method based on a dual-branch network, characterized in that: include: Step 1: Obtain noisy speech and perform Fourier transform on the noisy speech to obtain the original speech spectrum Z , then Z Decouple to get the original complex spectrum X 0. Original Amplitude Spectrum Y 0; Step 2: X 0. Y 0 is input into the trained speech enhancement network for processing to obtain the enhanced complex spectrum Z' ; The speech enhancement network includes: 2 frequency band segmentation units, 2 feature extraction units, 1 sequence modeling unit, 1 complex spectrum feature modeling unit, 1 amplitude spectrum feature modeling unit, and 1 frequency band synthesis unit; The first frequency band division unit is used to X 0 performs frequency band segmentation processing to obtain the intermediate complex spectrum characteristics A ; The second frequency band division unit is used for Y 0 performs frequency band segmentation processing to obtain the intermediate amplitude spectrum characteristics E ; The first feature extraction unit is used based on A , E Extracting intermediate features B ; The second feature extraction unit is used based on A , E Extracting intermediate features F ; The sequence modeling department is built on Mamba and is used to B , F Perform sequence modeling to obtain intermediate features C , intermediate features H ; The complex spectrum feature modeling unit is used to C Perform feature modeling to obtain the final complex spectrum features D ; The amplitude spectrum feature modeling part is used to H Perform feature modeling to obtain the final amplitude spectrum characteristics L ; The frequency band synthesis unit is used to D , L Perform synthesis to obtain Z' ; Step 3: Z and Z' Masking is performed first, and then inverse Fourier transform is performed to obtain enhanced speech.

2. The monophonic speech enhancement method based on a dual-branch network according to claim 1, characterized in that: The frequency band segmentation unit includes: 1 spectrum segmentation layer, 1 normalization layer, 1 Layer layer, and 1 dimension merging layer; In the first frequency band division part: the spectrum division layer is used to X 0 is split according to the preset specifications to obtain M Original complex bands X 1. The normalization layer is used to X 1 is normalized to obtain M Normalized complex spectral features X 2. Layer is used for X 2 Unify in frequency dimension to obtain M Unified complex spectral features X 3. The dimension merging layer is used to X 3 Merge in the frequency dimension to obtain A ; in, X 1=[ x 11 , x 12 ,…, x 1M ], x 1m express X 1, the mth original complex spectral band, m∈[1, M ]; X 2=[ x 21 , x 22 ,…, x 2M ], x 2m express X The mth normalized complex spectrum feature in 2; X 3=[ x 31 , x 32 ,…, x 3M ], x 3m express X The mth unified complex spectrum feature in 3; In the second frequency band division part: the spectrum division layer is used to Y 0 is split according to the preset specifications to obtain M Original Amplitude Bands Y 1. The normalization layer is used to Y 1 is normalized to obtain M Normalized amplitude spectrum features Y 2. Layer is used for Y 2 Unify in frequency dimension to obtain M Unified Amplitude Spectrum Features Y 3. The dimension merging layer is used to Y 3 Merge in the frequency dimension to obtain E ; in, Y 1=[ y 11 , y 12 ,…, y 1M ], y 1m express Y The mth original amplitude spectrum band in 1; Y 2=[ y 21 , y 22 ,…, y 2M ], y 2m express Y The mth normalized amplitude spectrum feature in 2; Y 3=[ y 31 , y 32 ,…, y 3M ], y 3m express Y The mth unified amplitude spectrum feature in 3.

3. The monophonic speech enhancement method based on a dual-branch network according to claim 1, characterized in that: The feature extraction part includes: 1 two-dimensional convolution layer, 1 normalization layer, 1 PRelu activation function, and 1 information interaction layer; In the first feature extraction part: the two-dimensional convolutional layer is used to A Perform feature extraction to obtain complex spectrum information a 1. The normalization layer is used to a 1Normalize to get complex spectrum information a 2. PRelu activation function is used to a 2Introducing nonlinear features to obtain complex spectrum information a 3. The information interaction layer is used to a 3. E Interactive processing to obtain B ; In the second feature extraction part: the two-dimensional convolution layer is used to E Perform feature extraction to obtain amplitude spectrum information e 1. The normalization layer is used to e 1Normalize to get the amplitude spectrum information e 2. PRelu activation function is used to e 2Introducing nonlinear features to obtain amplitude spectrum information e 3. The information interaction layer is used to e 3. A Interactive processing to obtain F .

4. The monophonic speech enhancement method based on a dual-branch network according to claim 3, characterized in that: The sequence modeling part includes: N Mamba model parts with the same structure; The nth Mamba model part includes: 1 information interaction layer, 2 overlay layers, and 2 FT-Mamba parts; In the nth Mamba model: the information interaction layer is used to analyze the intermediate spectrum features W n-1 , intermediate spectrum characteristics w n-1 Interactive processing to obtain intermediate spectrum features U n , the first overlay is used to U n and W n-1 Superposition to obtain the intermediate spectrum characteristics U_B n , the second overlay is used to U n and w n-1 Superposition to obtain the intermediate spectrum characteristics U_F n , the first FT-Mamba section is used to U_B n Sequence modeling based on Mamba to obtain intermediate spectrum features W n , the second FT-Mamba section is used to U_F n Sequence modeling based on Mamba to obtain intermediate spectrum features w n ; n∈[1,N]; in, B As the intermediate spectrum feature W 0, F As the intermediate spectrum feature w 0; Intermediate spectrum characteristics W N As C , intermediate spectrum characteristics w N As H .

5. The monophonic speech enhancement method based on a dual-branch network according to claim 3 or 4, characterized in that: The information interaction layer includes: 1 joint layer, 1 sub-2D convolution layer, 1 sub-normalization layer, and 1 sub-PRelu activation function; The information exchange layer has dual inputs and a single output. In any information exchange layer, the union layer is used to connect the dual inputs of the information exchange layer to obtain intermediate data. G 1. The sub-two-dimensional convolutional layer is used to transform G 1 Perform feature extraction to obtain intermediate data G 2. The sub-normalization layer is used to G 2Perform normalization to obtain intermediate data G 3. The sub-PRelu activation function is used to G 3. Introduce nonlinear features to obtain a single output of the information interaction layer.

6. The monophonic speech enhancement method based on a dual-branch network according to claim 4, characterized in that: The FT_Mamba department includes 5 sub-sections; Among them, the first sub-part is the normalization layer, the second sub-part is the F_Mamba part, the third sub-part is the superposition layer, the fourth sub-part is the T_Mamba part, and the fifth sub-part is the superposition layer; Or, the first sub-part is the normalization layer, the second sub-part is the T_Mamba part, the third sub-part is the superposition layer, the fourth sub-part is the F_Mamba part, and the fifth sub-part is the superposition layer; In any FT_Mamba section: The first subsection is used to normalize the input of the FT_Mamba section to obtain intermediate data J 1, the second sub-section is used for J 1 Perform sequence modeling on the frequency dimension to obtain intermediate data J 2, the third sub-section is used to connect the input of the FT_Mamba section with J 2. Overlay to get intermediate data J 3, the fourth sub-section is used for J 3. Perform sequence modeling on the time dimension to obtain intermediate data J 4, the fifth subsection is used to J 3 and J 4 are superimposed to obtain the output of the FT_Mamba section.

7. The monophonic speech enhancement method based on a dual-branch network according to claim 6, characterized in that: The F_Mamba part includes: 2 Transpose layers, 1 Unfold layer, 1 F_Mamba layer, and 1 one-dimensional transposed convolution layer; In any F_Mamba unit: The first Transpose layer is used to perform tensor transformation on the input of the F_Mamba unit to obtain intermediate data R 1. The Unfold layer is used to R 1 Upsample to get intermediate data R 2. The F_Mamba layer is used to pass bidirectional Mamba pairs R 2 Perform sequence modeling to obtain intermediate data R 3. One-dimensional transposed convolutional layer is used to R 3 Perform one-dimensional transposed convolution to obtain intermediate data R 4. The second Transpose layer is used to R 4. Perform tensor conversion to obtain the output of the F_Mamba unit; Among them, the F_Mamba layer includes: 1 flip layer, 2 Mamba layers, 2 Norm layers, 2 overlay layers, 1 Constant layer, and 1 Layer layer; In any F_Mamba layer: the first Mamba layer is used to perform sequence modeling on the input of the F_Mamba layer to obtain intermediate data P1, the first Norm layer is used to normalize P1 to obtain intermediate data P2, the first superposition layer is used to superimpose the input of the F_Mamba layer with P2 to obtain intermediate data P3, the flip layer is used to reverse the input of the F_Mamba layer to obtain intermediate data P4, the second Mamba layer is used to model P4 to obtain intermediate data P5, the second Norm layer is used to normalize P5 to obtain intermediate data P6, the second superposition layer is used to superimpose P4 and P6 to obtain intermediate data P7, the Constant layer is used to perform tensor adjustment on P3 and P7 to obtain intermediate data P8, and the Layer layer is used to perform linear mapping on P8 to obtain the output of the F_Mamba layer; The T_Mamba part includes: 2 Transpose layers, 1 one-dimensional padding layer, 1 Unfold layer, 1 T_Mamba layer, and 1 one-dimensional transposed convolution layer; In any T_Mamba unit: The first Transpose layer is used to perform tensor transformation on the input of the T_Mamba unit to obtain intermediate data T 1. One-dimensional filling layer is used for T 1 Fill data to get intermediate data T 2. The Unfold layer is used to T2 Perform abstract feature extraction to obtain intermediate data T 3. The T_Mamba layer is used to pass the one-way Mamba pair T 3 Perform sequence modeling to obtain intermediate data T 4. One-dimensional transposed convolutional layer is used to T 4 Perform one-dimensional transposed convolution to obtain intermediate data T 5. The second Transpose layer is used to T 5. Perform tensor conversion on the output of the T_Mamba unit; Among them, the T_Mamba layer includes: 1 Mamba layer, 1 Norm layer, 1 Layer layer, and 1 overlay layer; In any T_Mamba layer: the Mamba layer is used to model the input of the T_Mamba layer to obtain the intermediate data Q1, the Norm layer is used to normalize Q1 to obtain the intermediate data Q2, the Layer layer is used to linearly map Q2 to obtain the intermediate data Q3, and the superposition layer is used to superimpose the input of the T_Mamba layer with Q3 to obtain the output of the T_Mamba layer.

8. The monophonic speech enhancement method based on a dual-branch network according to claim 1, characterized in that: The complex spectrum feature modeling part includes: 1 F1E-Real layer, 1 F2E-Imag layer, and 1 superposition layer; In the complex spectrum feature modeling section: the F1E-Real layer is used to C Perform feature modeling to obtain the real intermediate feature c1, and the F2E-Imag layer is used to C Feature modeling is performed to obtain the imaginary intermediate feature c2, and the superposition layer is used to superimpose c1 and c2 to obtain D ; Among them, the F1E-Real layer includes: 3 two-dimensional convolutional layers, 1 Norm layer, 2 PRelu activation functions, and 1 Sigmoid activation function; In the F1E-Real layer: The first two-dimensional convolutional layer is used to C Upsampling is performed to obtain intermediate data u1, the Norm layer is used to normalize u1 to obtain intermediate data u2, the first PRelu activation function is used to introduce nonlinear features to u2 to obtain intermediate data u3, the second two-dimensional convolution layer is used to perform two-dimensional convolution on u3 to obtain intermediate data u4, the Sigmoid activation function is used to introduce linear features to u4 to obtain intermediate data u5, the third two-dimensional convolution layer is used to extract features from U5 to obtain intermediate data u6, and the second PRelu activation function is used to introduce nonlinear features to u6 to obtain c1; Among them, the F2E-Imag layer includes: 4 two-dimensional convolutional layers, 1 Norm layer, 2 PRelu activation functions, 1 Sigmoid activation function, 1 Tanh activation function, and 1 overlay layer; In the F2E-Imag layer: the first two-dimensional convolution layer is used to upsample C to obtain intermediate data U1, the Norm layer is used to normalize U1 to obtain intermediate data U2, the first PRelu activation function is used to introduce nonlinear features to U2 to obtain intermediate data U3, the second two-dimensional convolution layer is used to perform two-dimensional convolution on U3 to obtain intermediate data U4, the Sigmoid activation function is used to introduce linear features to U4 to obtain intermediate data U5, the third two-dimensional convolution layer is used to perform two-dimensional convolution on U3 to obtain intermediate data U6, the Tanh activation function is used to introduce linear features to U6 to obtain intermediate data U7, the superposition layer is used to superimpose U5 and U7 to obtain intermediate data U8, the fourth two-dimensional convolution layer is used to extract features from U8 to obtain intermediate data U9, and the second PRelu activation function is used to introduce nonlinear features to U9 to obtain c2; The amplitude spectrum feature modeling part includes: 1 F2E-Mask layer and 1 matrix multiplication layer; In the amplitude spectrum feature modeling part: the F2E-Mask layer is used to H Modeling to obtain the amplitude spectrum feature mask H' , the matrix multiplication layer is used to convert H' and E Perform matrix multiplication to get L ; Among them, the F2E-Mask layer includes: 4 two-dimensional convolutional layers, 1 Norm layer, 2 PRelu activation functions, 1 Sigmoid activation function, 1 Tanh activation function, and 1 overlay layer; In the F2E-Mask layer: the first two-dimensional convolution layer is used to upsample H to obtain intermediate data h1, the Norm layer is used to normalize h1 to obtain intermediate data h2, the first PRelu activation function is used to introduce nonlinear features to h2 to obtain intermediate data h3, the second two-dimensional convolution layer is used to perform two-dimensional convolution on h3 to obtain intermediate data h4, the Sigmoid activation function is used to introduce linear features to h4 to obtain intermediate data h5, the third two-dimensional convolution layer is used to perform two-dimensional convolution on h3 to obtain intermediate data h6, the Tanh activation function is used to introduce linear features to h6 to obtain intermediate data h7, the superposition layer is used to superimpose h5 and h7 to obtain intermediate data h8, the fourth two-dimensional convolution layer is used to extract features from h8 to obtain intermediate data h9, and the second PRelu activation function is used to introduce nonlinear features to h9 to obtain H' .

9. The monophonic speech enhancement method based on a dual-branch network according to claim 1, characterized in that: The frequency band synthesis unit includes: 1 overlay layer, 1 block disassembly layer, 1 normalization layer, 1 Layer layer, 1 Tanh activation function, 1 GLU layer, and 1 frequency band merging layer; In the frequency band synthesis section, the overlay layer is used to D , L The complex spectrum feature block V0 is superimposed, and the block decomposition layer is used to decompose V0 into blocks to obtain M The original complex spectrum tensor V1, the normalization layer is used to normalize V1 to obtain the normalized complex spectrum tensor V2, the Layer layer is used to linearly transform V2 to obtain the double complex spectrum tensor V3, the Tanh activation function is used to introduce linear features into V3 to obtain the activated complex spectrum tensor V4, the GLU layer is used to restore the dimension of V4 to obtain the final segmentation band mask V5, and the band merging layer is used to merge V5 in the frequency dimension to obtain Z' ; in, V 1=[ v 11 , v 12 ,…, v 1M ], v 1m express V The mth original complex spectral tensor in 1; V 2=[ v 21 , v 22 ,…, v 2M ], v 2m express V The mth normalized complex spectral tensor in 2; V 3=[ v 31 , v 32 ,…, v 3M ], v 3m express V The mth double complex spectral tensor in 3; V 4=[ v 41 , v 42 ,…, v 4M ], v 4m express V The mth activation complex spectral tensor in 4; V 5=[ v 51 , v 52 ,…, v 5M ], v 5m express V The mth final segmentation band mask in 5.

10. A monophonic speech enhancement device based on a dual-branch network, characterized in that: It uses the monophonic speech enhancement method based on a dual-branch network as described in any one of claims 1-9; The monophonic speech enhancement device based on the dual-branch network comprises: The preprocessing module is used to obtain noisy speech and perform Fourier transform on the noisy speech to obtain the original speech spectrum. Z , then Z Decouple to get the original complex spectrum X 0. Original Amplitude Spectrum Y 0; The speech enhancement module is used to X 0. Y 0 is input into the trained speech enhancement network for processing to obtain the enhanced complex spectrum Z' ; as well as Post-processing module, which is used to Z and Z' Masking is performed first, and then inverse Fourier transform is performed to obtain enhanced speech.

Images