A neuromorphic speech enhancement method, device and electronic equipment based on a double-branch pulse neural network

Abstract

The application provides a neuromorphic speech enhancement method, device and electronic equipment based on a double-branch pulse neural network. The method comprises inputting a time-domain noisy speech signal into a pre-constructed neuromorphic speech enhancement model to obtain enhanced speech; wherein the processing steps of the model on the time-domain noisy speech signal comprise: extracting frequency-domain multi-dimensional features of the time-domain noisy speech signal; performing convolutional attention coding on the frequency-domain multi-dimensional features based on an encoder to obtain a high-dimensional latent feature map; performing feature modeling on the high-dimensional latent feature map based on a double-path gated pulse unit block stacked with each other to obtain a deep feature map; obtaining a complex spectrum estimation result through a complex spectrum decoding branch and an amplitude spectrum estimation result through an amplitude spectrum decoding branch for the deep feature map; fusing the complex spectrum estimation result and the amplitude spectrum estimation result, and reconstructing a time-domain speech signal based on the fusion result to obtain enhanced speech.

Description

Technical Field

[0001] This invention relates to the field of speech enhancement technology, specifically to a neuromorphic speech enhancement method, apparatus, and electronic device based on a dual-branch spiking neural network. Background Technology

[0002] The core objective of single-channel speech enhancement is to recover clean speech from noisy speech observation signals. It is a key front-end processing technology for edge hardware such as hearing aids, speech recognition terminals, and real-time voice communication devices. Deep learning has become the mainstream technology paradigm in this field, but existing high-performance speech enhancement models typically contain millions or even tens of millions of trainable parameters, resulting in significant computational and storage overhead. This fails to meet the core application requirements of edge devices for low power consumption, low latency, and small memory usage. Therefore, parameter-efficient and lightweight speech enhancement schemes have become one of the core research directions.

[0003] Research on lightweight speech enhancement has mainly progressed along two major technical routes: one is structural decomposition optimization, such as the DPRNN model which decomposes long speech sequences into frequency and time dimensions for decoupled modeling, reducing the computational complexity of long sequence modeling; the other is spectral representation optimization, such as complex spectrum modeling, spectral-temporal joint feature learning, and multi-domain feature fusion, which improve model efficiency and performance through more accurate speech feature representation. However, the efficiency improvements of the above methods all stem from improvements in macroscopic network topology. The recurrent modules (such as LSTM and GRU), which are the main body of model parameters, have not undergone fundamental changes. Their continuous value hidden state updates and dense matrix operations make them the core efficiency bottleneck of lightweight speech enhancement.

[0004] Spiking Neural Networks (SNNs), as a third-generation neural network model, leverages the inherent characteristics of event-driven computation and binary spike activation to significantly reduce computational load and power consumption, making them highly adaptable to resource-constrained edge devices. By replacing gradient training methods, SNNs have achieved performance comparable to traditional Artificial Neural Networks (ANNs) in various speech tasks such as speech recognition and speech synthesis. Among these, the Gated Spiking Unit (GSU) provides a lightweight modification to the traditional Long Short-Term Memory (LSTM) neural network, retaining only the forget gate structure and halving the parameters of the recurrent layers. This lays the core foundation for building ultra-lightweight SNN speech enhancement models and represents a key technological direction for solving the efficiency problem of traditional recurrent modules.

[0005] Building upon this, researchers have proposed lightweight speech enhancement models based on SNNs. Representative studies include Spiking-FullSubNet and DPSNN, both of which attempt to leverage the low power consumption and lightweight nature of SNNs to achieve speech enhancement. However, they exhibit significant limitations in structural design and performance shortcomings, as detailed below: The Spiking-FullSubNet model combines a spiking neural network with FullSubNet, focusing on the modeling and processing of complex spectral features. This achieves effective compression of model parameters, adapting to the resource requirements of edge devices. However, this model uses a single-branch architecture, modeling only the complex spectrum in a single dimension. It fails to explore the complementary information between the amplitude spectrum and the complex spectrum in noise suppression, phase recovery, and energy envelope estimation. As a result, the model still has a significant difference in sound quality compared to traditional high-performance ANN models, and its perceptual enhancement effect is poor.

[0006] The DPSNN model constructs a dual-path spiking neural network structure on the time-domain waveform and uniformly applies SNN units for feature processing, achieving low-latency streaming speech enhancement. However, this model does not study the compatibility of pulse dynamics characteristics in different signal domains (frequency domain / time domain), nor does it utilize complementary information from multiple spectral dimensions, and there is still considerable room for improvement in model parameter efficiency and enhancement performance.

[0007] In addition, traditional dual-path, multi-domain fusion speech enhancement methods (such as DPRNN and MP-SENet) can improve the enhancement effect through multi-dimensional feature learning, but they are all based on traditional ANN and do not combine the event-driven and binary activation characteristics of SNN. They cannot meet the low power consumption and low latency requirements of edge devices and are difficult to deploy on extremely resource-constrained hardware such as hearing aids. Summary of the Invention

[0008] In view of this, this application proposes a neuromorphic speech enhancement method, device and electronic device based on a dual-branch spiking neural network.

[0009] Specifically, this application is implemented through the following technical solution: According to a first aspect of the embodiments of this specification, a neuromorphic speech enhancement method based on a dual-branch spiking neural network is provided, wherein a noisy temporal speech signal is input into a pre-constructed neuromorphic speech enhancement model to obtain enhanced speech through the model; wherein the processing steps of the model on the noisy temporal speech signal are as follows: Step S1: Extract the frequency domain multi-dimensional features of the noisy time-domain speech signal; Step S2: Perform convolutional attention encoding on the frequency domain multi-dimensional features based on the encoder to obtain a high-dimensional latent feature map; Step S3: Based on stacked dual-path gated pulse unit blocks, feature modeling is performed on the high-dimensional latent feature map to obtain a deep feature map; each dual-path gated pulse unit block includes a frequency path and a time path. The frequency path uses bidirectional gated pulse units to perform bidirectional modeling along the frequency dimension; the time path uses unidirectional gated pulse units to perform unidirectional causal modeling along the time dimension. Step S4: For the depth feature map, obtain the complex spectrum estimation result through the complex spectrum decoding branch and obtain the amplitude spectrum estimation result through the amplitude spectrum decoding branch; Step S5: The complex spectrum estimation result is fused with the amplitude spectrum estimation result, and the time-domain speech signal is reconstructed based on the fusion result to obtain the enhanced speech.

[0010] According to a second aspect of the embodiments of this specification, a neuromorphic speech enhancement device based on a dual-branch spiking neural network is provided. The device is configured to process an input time-domain noisy speech signal based on a pre-built neuromorphic speech enhancement model to obtain enhanced speech; wherein the device includes: The feature extraction unit is used to extract the frequency domain multi-dimensional features of the noisy time-domain speech signal; The feature encoding unit is used to perform convolutional attention encoding on the multi-dimensional features in the frequency domain based on the encoder to obtain a high-dimensional latent feature map. The time-frequency feature processing unit is used to perform feature modeling on the high-dimensional latent feature map based on stacked dual-path gated pulse unit blocks to obtain a deep feature map. Each dual-path gated pulse unit block includes a frequency path and a time path. The frequency path uses bidirectional gated pulse units to perform bidirectional modeling along the frequency dimension. The time path uses unidirectional gated pulse units to perform unidirectional causal modeling along the time dimension. The feature decoding unit is used to obtain a complex spectrum estimation result through a complex spectrum decoding branch and an amplitude spectrum estimation result through an amplitude spectrum decoding branch for the depth feature map. The speech reconstruction unit is used to fuse the complex spectrum estimation result with the amplitude spectrum estimation result, and reconstruct the time-domain speech signal based on the fusion result to obtain the enhanced speech.

[0011] According to a third aspect of the embodiments of this specification, an electronic device is provided, comprising: processor; A computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method as described in the first aspect.

[0012] The embodiments of this application have at least the following technical effects: First, the embodiments of this application use a gated pulse unit as the core loop unit. Compared with traditional ANN speech enhancement models and SNN baseline models, it can significantly reduce the total number of model parameters. At the same time, by utilizing the event-driven calculation and binary pulse activation of the gated pulse unit, power consumption and computational overhead are greatly reduced, making it naturally suitable for edge devices with extremely limited resources, such as hearing aids and portable voice terminals. Second, by constructing a dual-branch decoding structure of amplitude spectrum and complex spectrum, the embodiments of this application can fully exploit the advantages of stable energy estimation of amplitude spectrum and accurate phase recovery of complex spectrum, and achieve complementary optimization of the two through fusion, thereby improving the speech enhancement effect of the model. Third, the embodiments of this application construct a feature separator with dual-path gated pulse units. The frequency path uses bidirectional gated pulse units to capture global spectral correlation, and the time path uses unidirectional gated pulse units to realize causal time modeling, forming a complete time-frequency dimension deep modeling framework. This solves the problem of insufficient capture of speech context information by traditional models and improves the robustness of the model to complex noise. Fourth, the embodiments of this application construct an end-to-end neuromorphic speech enhancement model from noisy speech input to enhanced speech output. Since the time path uses unidirectional gated pulse units to achieve causal modeling, no future frame information is required, supporting low-latency streaming speech enhancement and meeting the practical application needs of scenarios such as real-time communication and hearing aids. Attached Figure Description

[0013] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Some specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings in an exemplary and non-limiting manner. The same reference numerals in the drawings indicate the same or similar parts or components. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings: Figure 1 This is a schematic flowchart illustrating an exemplary embodiment of the present application of a neuromorphic speech enhancement method based on a dual-branch spiking neural network; Figure 2 This is a schematic diagram illustrating a speech signal processing flow according to an exemplary embodiment of this application; Figure 3 This is a schematic diagram of the internal processing flow of a dual-path gated pulse unit block, as illustrated in an exemplary embodiment of this application. Figure 4 This is a schematic diagram of the internal calculation process of a gated pulse unit according to an exemplary embodiment of this application; Figure 5 This is a structural block diagram of an electronic device illustrated in an exemplary embodiment of this application; Figure 6 This is a structural block diagram of a neuromorphic speech enhancement device based on a dual-branch spiking neural network, as illustrated in an exemplary embodiment of this application. Detailed Implementation

[0014] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0015] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0016] As mentioned earlier, the core bottleneck of traditional lightweight speech enhancement methods using ANNs lies in the design of the recurrent module. The continuous value hidden state update and dense matrix operations of LSTM / GRU result in a large number of model parameters and high power consumption. Even if the overhead is reduced through structural decomposition and feature optimization, it is still difficult to adapt to edge devices with extremely limited resources, such as hearing aids and portable voice terminals.

[0017] Existing SNN speech enhancement methods suffer from the following main shortcomings: First, these methods all employ a single-branch architecture, modeling only a single spectral representation (complex spectrum or time-domain waveform). This fails to fully utilize the complementary information between the amplitude spectrum and the complex spectrum. The amplitude spectrum excels at providing stable speech energy envelope estimation, while the complex spectrum excels in accurate phase recovery and noise suppression. Single-dimensional modeling severely limits enhancement performance. Second, existing methods do not design differentiated SNN modeling approaches for different signal dimensions. For example, the DPSNN model uniformly applies SNN units in both time-domain paths, failing to consider the adaptability of impulse dynamics characteristics across different dimensions. Third, the gate structure design of existing SNN recurrent units lacks targeted verification. The impact of gate structure complexity on model performance and parameter efficiency under binary output characteristics is not clearly defined, resulting in gate structure redundancy and low parameter utilization.

[0018] Furthermore, there are still technological gaps in cross-domain fusion. Currently, no research has organically combined a dual-path, dual-branch multi-domain fusion architecture with spiking neural networks, thus failing to simultaneously leverage the complementary advantages of multi-dimensional features and the low-power, lightweight characteristics of SNNs. This has become a core obstacle to the implementation of voice enhancement technology for edge devices.

[0019] Based on this, this application proposes a neuromorphic speech enhancement scheme based on a dual-branch spiking neural network, aiming to achieve a balance between low parameters, low power consumption, low latency, high perceptual quality, and high noise robustness, thus meeting the practical application needs of edge devices. The specific objectives are as follows: (1) Solving the problem of high parameters and high power consumption in traditional ANN speech enhancement models. By introducing gated pulse units into the speech enhancement architecture, the single-gate structure and binary pulse activation characteristics are utilized to significantly reduce model parameters and computational overhead, achieving extreme lightweighting. Breaking through the limitations of the single-branch architecture of existing SNN speech enhancement models, an amplitude spectrum-complex spectrum dual-branch decoding structure is constructed to achieve joint modeling and fusion of dual spectrum dimensions, fully exploring the complementary information of the two in energy estimation, phase recovery, and noise suppression, and improving enhancement performance.

[0020] (2) Differentiated SNN modeling methods are designed for different signal dimensions. A dual-path GSU feature processing module is constructed. The frequency path uses bidirectional BiGSU to capture global spectrum correlation, and the time path uses unidirectional GSU to realize causal time modeling, forming a complete time-frequency dimension deep modeling framework.

[0021] (3) Verify the optimal gate structure design of the SNN recurrent unit under the binary output bottleneck, clarify the optimality of the single gate structure, provide empirical evidence for the subsequent structural design of the spiking neural network, and build an end-to-end neuromorphic speech enhancement system to achieve complete and efficient processing from noisy speech input to enhanced speech output, and adapt to the deployment requirements of neuromorphic hardware and edge devices.

[0022] The embodiments described in this specification will now be described in detail.

[0023] This application provides a neuromorphic speech enhancement method based on a dual-branch spiking neural network, which inputs a noisy time-domain speech signal into a pre-constructed neuromorphic speech enhancement model to obtain enhanced speech through the model.

[0024] Figure 1 This is a flowchart illustrating an exemplary embodiment of a neuromorphic speech enhancement method based on a dual-branch spiking neural network, as shown in this application. Figure 1 As shown, the processing steps of the model for the noisy time-domain speech signal are as follows: Step S1: Extract the frequency domain multi-dimensional features of the noisy time-domain speech signal; Step S2: Perform convolutional attention encoding on the frequency domain multi-dimensional features based on the encoder to obtain a high-dimensional latent feature map; Step S3: Based on stacked dual-path gated pulse unit blocks, feature modeling is performed on the high-dimensional latent feature map to obtain a deep feature map; each dual-path gated pulse unit block includes a frequency path and a time path. The frequency path uses bidirectional gated pulse units to perform bidirectional modeling along the frequency dimension; the time path uses unidirectional gated pulse units to perform unidirectional causal modeling along the time dimension. Step S4: For the depth feature map, obtain the complex spectrum estimation result through the complex spectrum decoding branch and obtain the amplitude spectrum estimation result through the amplitude spectrum decoding branch; Step S5: The complex spectrum estimation result is fused with the amplitude spectrum estimation result, and the time-domain speech signal is reconstructed based on the fusion result to obtain the enhanced speech.

[0025] Next, combine Figures 2 to 4 Please explain each of the above steps in detail.

[0026] like Figure 2 As shown, the processing flow of the time-domain noisy speech signal in this embodiment includes STFT spectral feature extraction, feature encoding by convolutional attention encoder, time-frequency modeling based on dual-path gated pulse unit blocks, mask estimation based on dual branches, fusion of dual-branch decoding results, and ISTFT time-domain reconstruction.

[0027] In some embodiments, step S1 includes: A short-time Fourier transform is performed on the noisy speech signal in the time domain to generate a complex frequency spectrum; features in three dimensions—real part, imaginary part, and amplitude spectrum—are extracted from the complex spectrum; the real part, imaginary part, and amplitude spectrum are concatenated along the channel dimension to generate the multi-dimensional frequency domain features.

[0028] This embodiment converts noisy time-domain speech signals into multi-dimensional frequency-domain features, providing a foundation for subsequent encoding and modeling. Specifically, it includes the following sub-steps: S11: Acquire the noisy time-domain speech signal x(t).

[0029] For example, the sampling rate is set to 16kHz, which is a common sampling rate for edge devices and is suitable for actual application scenarios.

[0030] S12: Perform a short-time Fourier transform (STFT) on x(t) to generate a complex frequency spectrum X(f,t).

[0031] For example, the core parameters of STFT are set as follows: a Hanning window with 400 sampling points (corresponding to a window length of 25ms), a frame shift of 100 sampling points (corresponding to 6.25ms, with a frame shift rate of 25%), and 512-point FFT operation, ultimately generating a 257-dimensional frequency domain complex spectrum. ∈[0,256], where t is a time frame.

[0032] S13: Extract the real part Re(X), imaginary part Im(X), and amplitude spectrum |X| from the complex spectrum X(f,t), using the following formula: (1) in, To minimize the value and avoid numerical overflow; S14: Concatenate the real part, imaginary part, and amplitude spectrum features along the channel dimension to generate a three-channel spectrum input. ∈ T represents the number of time frames, which serves as the input to the encoder module.

[0033] In some embodiments, convolutional attention encoding is performed in step S2 to generate a high-dimensional latent feature map.

[0034] This embodiment uses a convolutional attention encoder to extract features, compress dimensions, and expand channels from the three-channel spectral input, generating a high-dimensional latent feature map. This provides a suitable feature representation for subsequent dual-path GSU modeling. The specific sub-steps are as follows: S21: Construct three concatenated convolutional attention blocks, each with the following structure: Conv2d→GroupNorm→PReLU→CBAM attention.

[0035] Conv2d is used to implement feature extraction and dimensionality transformation; GroupNorm is used to implement feature normalization to avoid the mini-batch problem of batch normalization; PreLU is used to implement non-linear activation; and CBAM attention is used to implement adaptive adjustment of feature weights in channel and spatial dimensions.

[0036] For example, the parameter settings for the first convolutional attention block are: Conv2d kernel size (5,2), stride (2,1), and 16 output channels. This performs frequency dimension compression and channel expansion on the three-channel input Xin, and outputs a feature map. ∈ ; The parameters for the second convolutional attention block are: Conv2d kernel size (5,2), stride (2,1), and output channels 32. Continue with frequency dimension compression and channel expansion to output feature maps. ∈ ; The parameters for the third convolutional attention block are: Conv2d kernel size (1,1), stride (1,1), and 64 output channels. Only channel expansion is performed, without changing the frequency and temporal dimensions, and the output is a latent feature map. ∈ , as input to the dual-path GSU block.

[0037] In some embodiments, step S3 performs dual-path GSU feature modeling to achieve deep feature learning in the time-frequency dimension.

[0038] This embodiment is the core feature processing module of the model. It uses two stacked dual-path GSU blocks to alternately model the latent feature map Z in the frequency and time dimensions, capturing the global spectral correlation and inter-frame temporal continuity of speech.

[0039] like Figure 3 As shown, each dual-path GSU block contains three sub-modules: frequency path processing, time path processing, and residual fusion. The core process is: latent feature map → frequency path BiGSU modeling → residual addition → time path GSU modeling → residual addition → output processed feature map; two dual-path GSU blocks are stacked to achieve deep modeling of features.

[0040] The bidirectional BiGSU modeling process for the frequency path is as follows: The high-dimensional latent feature map or the feature map output by the previous dual-path gated pulse unit block is rearranged in dimensions to adapt it to the input format of the bidirectional gated pulse unit layer. The rearranged feature map is input into the bidirectional gated pulse unit layer, and bidirectional feature modeling is performed along the forward and reverse order of the frequency dimension, respectively, and bidirectional fused features are output. Linear projection is performed on the bidirectional fused features to compress their number of channels to match the number of channels in the input feature map, and inverse dimensional rearrangement is performed to restore the original data format; The recovered feature map is normalized and then added to the residual of the input feature map of the frequency path to output the feature map after frequency path processing.

[0041] Specifically, the frequency path employs a bidirectional gated pulse unit (BiGSU) to bidirectionally model the feature map along the frequency dimension (65 dimensions), capturing global spectral correlations across frequencies. The specific sub-steps are as follows: S311: Transfer the latent feature map ∈ Perform dimensional rearrangement, becoming ∈ It adapts to the input dimensions of BiGSU (time frame × frequency point × channel). S312: Will Input a bidirectional BiGSU layer with a hidden dimension of 128, perform feature modeling along both the forward and reverse frequency sequences, and output bidirectional features. ; S313: Yes Perform linear projection to compress the number of channels from 256 to 64, generating ∈ And perform dimensional rearrangement to restore it to ; S314: Yes GroupNorm normalization is performed, and the residual is added to the original input Z to obtain the feature map after frequency path processing. To avoid gradient vanishing.

[0042] The time-path unidirectional GSU modeling process is as follows: The feature map output by the frequency path is rearranged in dimensions to fit the input format of the unidirectional gated pulse unit layer. The rearranged feature map is input into the unidirectional gated pulse unit layer, and unidirectional causal feature modeling is performed only along the positive order of the time dimension, and the time feature is output. Linear projection is performed on the time features to compress their number of channels to match the number of channels in the frequency path output feature map, and inverse dimensional rearrangement is performed to restore the original data format; The recovered feature map is normalized and added to the input feature map of the time path by residual, and the feature map after processing by the current dual-path gated pulse unit block is output.

[0043] Specifically, the time path employs a unidirectional gated pulse unit (GSU) to perform causal modeling of the feature map along the time dimension (T-dimensional), ensuring inter-frame continuity of speech and adapting to the low-latency requirements of streaming speech enhancement. The specific sub-steps are as follows: S321: Outputs the frequency path Perform dimensional rearrangement, becoming ∈ Adapts to GSU's input dimensions (frequency point × time frame × channel). S322: Will Input a one-way GSU layer with a hidden dimension of 128, perform causal feature modeling along the positive time sequence, and output the time path feature Ft∈R65×T×128; S323: Yes Perform linear projection to compress the number of channels from 128 to 64, generating And perform dimensional rearrangement to restore it to ; S324: Yes Perform GroupNorm normalization, and By summing the residuals, we obtain the feature map after processing the single and dual path GSU blocks. .

[0044] Will Input the second stacked dual-path GSU block, and repeat steps S311-S314 and S321-S325 above to finally output the depth feature map after dual-path GSU modeling. .

[0045] In this embodiment, the gated pulse unit adopts a single-gate structure based on a leak-integration ignition neuron, and its operation process includes: A joint linear projection is performed on the current input and the pulse output at the previous time step to obtain an intermediate vector; the first half of the features are extracted from the intermediate vector and activated by the sigmoid function to obtain a forget gate; the second half of the features are extracted from the intermediate vector and the membrane potential at the current time step is updated according to the forget gate; the membrane potential at the current time step step function is activated to generate a binary pulse output.

[0046] Specifically, such as Figure 4 As shown, the gated pulse unit in this embodiment retains only the single-gate structure of the forget gate, and includes two core operations: gated membrane potential update and binary pulse emission. The specific operation process is as follows: First, perform the joint linear projection: (2) in, Enter the current time. The pulse output from the previous moment. , This is the weight matrix. For bias.

[0047] Next, the forget gate calculation is performed: (3) Where σ is the sigmoid activation function, for The first half of the features.

[0048] The forgetting gate is used to control the time decay of membrane potential. Specifically, →1 Retain the previous state. →0 Focus on the current input.

[0049] Then, perform a membrane potential update: (4) Where ⊙ represents element-wise multiplication. for The latter half of the features, 1- As an implicit input gate, it requires no additional parameters and halves the parameters of the recurrent layer compared to LSTM.

[0050] Next, execute the binary pulse emission: =Θ( (5) Where Θ is the Heaviside step function. ≥0 =1, otherwise =0, Θ is used to implement binary pulse output.

[0051] In practical applications, since Θ is not differentiable, this embodiment can use a trigonometric substitution gradient approximation to achieve standard backpropagation in time (BPTT), as shown in the following formula: (6) In some embodiments, step S4 generates complex spectrum and amplitude spectrum estimation results through bi-branch spectral mask estimation.

[0052] In this embodiment, two independent deconvolutional decoders, a complex spectrum decoder and an amplitude spectrum decoder, are constructed. By combining the features of each layer of the U-Net skip connection fusion encoder, the complex mask and amplitude mask are estimated respectively, generating a dual-branch spectrum estimation result.

[0053] The complex spectrum decoder and amplitude spectrum decoder both adopt a deconvolution stacked structure, forming a symmetrical architecture with the encoder. The core structure is: TransConv2d→GroupNorm→PReLU. At the same time, U-Net skip connections are introduced to combine the output features of each layer of the encoder ( , The features of the corresponding layer of the decoder (Z) are concatenated to merge shallow and deep features, thereby improving the accuracy of mask estimation.

[0054] Specifically, step S4 obtains the complex spectrum estimation result through the complex spectrum decoding branch, including: The depth feature map is input into the complex spectrum decoder, which contains multiple cascaded deconvolutional layers. Each deconvolutional layer is connected to the feature map of the corresponding scale in the encoder through skip connections to perform channel concatenation. The last deconvolutional layer of the complex spectrum decoder outputs a complex spectrum mask. Apply the tanh activation function to the complex spectrum mask to generate DeepFilter coefficients; The noisy complex spectrum is filtered using the DeepFilter coefficients to obtain the complex spectrum estimation result.

[0055] And, step S4, obtaining the amplitude spectrum estimation result through the amplitude spectrum decoding branch, includes: The depth feature map is input into the amplitude spectrum decoder, which contains multiple cascaded deconvolution layers. Each deconvolution layer is connected to the feature map of the corresponding scale in the encoder through skip connections to perform channel concatenation. The last deconvolution layer of the amplitude spectrum decoder outputs a single-channel amplitude spectrum mask. Apply the sigmoid activation function to the amplitude spectrum mask to generate a normalized amplitude mask; The normalized amplitude mask is multiplied element-wise with the noisy amplitude spectrum to obtain the amplitude spectrum estimation result.

[0056] For example, step S4 includes the following sub-steps: S41, Complex spectrum branch decoding, generating complex spectrum estimation results. .

[0057] The precise correction of the phase information by focusing on the complex spectrum branch, the output of which is activated by tanh to generate DeepFilter coefficients, and the filtering of the noisy STFT spectrum, are detailed in the following sub-steps: S411: Transfer depth feature map Input the first layer of the complex spectrum decoder, TransConv2d, with the following parameters: kernel size (1,1), stride (1,1), and output channels 32. Output feature map. ∈ , with encoder After concatenation, input PReLU to activate; S412: Second layer TransConv2d parameter settings: kernel size (5,2), stride (2,1), output channels 16, output feature map. ∈ , with encoder After concatenation, input PReLU to activate; S413: Third layer TransConv2d parameter settings: kernel size (5,2), stride (2,1), output channels 2, output complex spectrum mask ∈ (Corresponding to real and imaginary part masks); S414: Yes Apply the tanh activation function to generate DeepFilter coefficients. The noisy complex spectrum X(f,t) is filtered to obtain the complex spectrum estimation result. .

[0058] S42, Amplitude Spectrum Branch Decoding, generates amplitude spectrum estimation results. .

[0059] The amplitude spectrum branch focuses on a stable estimate of the speech energy envelope. The output is activated by sigmoid to generate an amplitude mask, which is then multiplied element-wise with the noisy amplitude spectrum. The specific sub-steps are as follows: S421: Transfer depth feature map The first layer of the input amplitude spectrum decoder, TransConv2d, has the same parameters as S421, and outputs a feature map. , with encoder After concatenation, input PReLU to activate; S422: The parameters of the second-layer TransConv2d are the same as those of S422, and the output feature map is... , with encoder After concatenation, input PReLU to activate; S423: Third layer TransConv2d parameter settings: kernel size (5,2), stride (2,1), output channel 1, output amplitude spectrum mask. ; S424: Yes Apply the sigmoid activation function to generate a normalized magnitude mask. =σ( )( (∈[0,1]), and multiply element-wise with the noisy amplitude spectrum |X| to obtain the amplitude spectrum estimation result. = ⊙∣X∣.

[0060] In some embodiments, step S5 is used to perform bi-branch feature fusion and temporal reconstruction to generate the final enhanced speech.

[0061] This embodiment fully utilizes the complementary advantages of the two branches by weighted averaging and fusing the spectral estimation results. Then, the fused frequency domain features are converted into a time domain speech signal through inverse short-time Fourier transform (iSTFT) to obtain the final enhanced speech. The specific sub-steps are as follows: S51: Weighted fusion of dual-branch features, the formula is as follows: (7) Where α is the fusion weight, which was experimentally verified to be set to 0.5 (to balance the contributions of the complex spectrum and the amplitude spectrum), and Y is the enhanced complex spectrum after fusion.

[0062] S52: Perform inverse short-time Fourier transform (iSTFT) on the fused enhanced complex spectrum Y, and use parameters symmetrical to STFT (Hanning window, frame shift, number of FFT points) to generate time-domain speech features y′(t); S53: Normalize the amplitude of y′(t) to limit the signal value to a preset range, such as [-1,1], to avoid audio clipping, and obtain the final enhanced speech time-domain signal y(t), thus completing the entire speech enhancement process.

[0063] In summary, the technical solution of this application combines a neuromorphic speech enhancement method with dual-path gated pulse units and amplitude-complex spectrum dual-branch decoding, which breaks through the single-dimensional modeling limitations of existing SNN speech enhancement and achieves a balance between lightweight and high performance.

[0064] Specifically, the technical solution of this application proposes a fusion design of gated pulse unit and dual-path architecture: innovatively introducing a single gated pulse unit into the dual-path feature processing architecture, designing differentiated modeling methods of bidirectional BiGSU in the frequency path and unidirectional GSU in the time path to capture global spectral correlation and causal time continuity respectively, while utilizing the single-gate structure and binary pulse characteristics of GSU to achieve extreme lightweighting.

[0065] Furthermore, the technical solution of this application also proposes a dual-branch decoding and weighted fusion structure for amplitude spectrum and complex spectrum: for the first time, a dual-branch decoding structure is constructed in a spiking neural network, and independent deconvolution mask estimation is performed on amplitude spectrum and complex spectrum respectively. Dual-branch feature fusion is achieved by weighted averaging, making full use of the complementary information of the two in energy estimation, phase recovery and noise suppression.

[0066] Furthermore, the technical solution of this application optimizes the convolutional attention encoder and decoder: the CBAM channel-spatial attention module is combined with Conv2d, GroupNorm, and PReLU to construct a convolutional attention encoder, while U-Net skip connections are introduced to fuse the features of each layer of the encoder with the corresponding layer features of the decoder, thereby improving the accuracy of feature extraction and mask estimation.

[0067] Based on this, the technical solution of this application constructs a complete process of STFT feature extraction, convolutional attention coding, dual-path GSU modeling, dual-branch mask estimation, and weighted fusion iSTFT reconstruction, realizing efficient processing from noisy speech to enhanced speech, supporting low-latency streaming enhancement, and adapting to neuromorphic hardware and edge device deployment, thus forming the core protection framework of the overall solution.

[0068] Finally, the embodiments of this application also experimentally verify the speech enhancement scheme proposed in this application.

[0069] This embodiment uses the VoiceBank+DEMAND dataset, a common standard dataset in the field of speech enhancement, for experimental verification. The training set of this dataset contains 11,572 speech sentences from 28 speakers, mixed with 10 types of noise under four signal-to-noise ratios (SNRs): 0dB, 5dB, 10dB, and 15dB. The test set contains 824 speech sentences from 2 speakers who did not participate in the training, mixed with 5 types of unseen noise under SNRs of 2.5dB, 7.5dB, 12.5dB, and 17.5dB. The sampling rate of all speech signals is uniformly set to 16kHz, which conforms to the sampling standard for actual applications of edge devices.

[0070] The experiment employed common objective evaluation metrics in the field of speech enhancement, including Perceptual Evaluation of Speech Quality (PESQ), Composite Speech Signal Distortion (CSIG), Predictor of Background-noise Intrusiveness (CBAK), Coverage of Voiced Segments (COVL), Segmental Signal to Noise Ratio (SSNR), Short-time Objective Intelligibility (STOI), Scale-Invariant Signal to Noise Ratio (SI-SNR), and Deep Noise Suppression Mean Opinion Score (DNSMOS), to comprehensively measure the speech enhancement quality, noise suppression effect, speech intelligibility, and overall performance of the speech enhancement scheme proposed in this application. Among them, PESQ is used to evaluate the perceptual quality of speech, CSIG is used to evaluate the degree of speech signal distortion, CBAK is used to evaluate the background noise suppression effect, COVL is used to evaluate the overall speech quality, SSNR is the segmented signal-to-noise ratio, STOI is the speech intelligibility index, SI-SNR is the scale-invariant signal-to-noise ratio, and DNSMOS includes three items: signal quality (SIG), background noise (BAK), and overall quality (OVRL) to simulate subjective noise reduction quality evaluation.

[0071] Based on the above objective evaluation indicators, this embodiment compares the speech enhancement scheme of this application with traditional speech enhancement schemes on the VoiceBank+DEMAND dataset. The comparison results are shown in Table 1.

[0072] Table 1: Comparison results of this application and traditional speech enhancement schemes on the VoiceBank+DEMAND dataset.

[0073] In Table 1, apart from the number of parameters, higher values ​​indicate better performance, and lower parameter numbers indicate better performance. "-" indicates no results were obtained. Noisy is the original noisy speech, which is not enhanced in any way; the index score obtained by directly retaining the noisy speech is used. DCCRN (Deep Convolutional Neural Networks) is a deep convolutional neural network model. FullSubNet+ is an amplitude-spectrum channel attention full sub-network model for speech enhancement. GaGNet is a complex domain multi-stage iterative optimization dual-path collaborative learning deep neural network model for single-channel speech enhancement. TSTNN is an end-to-end neural network model designed for speech temporal denoising. Spiking-FSN is a speech enhancement model based on spiking neural networks. The speech enhancement process of these models used in this embodiment can be found in relevant technical literature by those skilled in the art, and will not be described in detail here.

[0074] As can be seen from Table 1, the speech enhancement scheme of this application achieved the highest scores in all evaluation metrics of the test set, and also had the lowest number of parameters.

[0075] Furthermore, to verify the necessity and effectiveness of the design of each core module of the speech enhancement model of this application, this embodiment conducts system ablation experiments from four key dimensions: dual-branch spectrum processing structure, dual-path GSU feature modeling, GSU ​​unit gate structure design, and model hidden dimension setting. The results of the ablation experiments are shown in Table 2.

[0076] Table 2: Ablation Experiment Results of this Application

[0077] As can be seen from the experimental results in Table 2, the PESQ value decreased significantly after removing the amplitude spectrum branch or the complex spectrum branch. This proves that the joint modeling of the amplitude spectrum and the complex spectrum can make full use of the complementary information of the two branches in energy envelope estimation, phase recovery and noise suppression. The single branch modeling cannot achieve the optimal enhancement effect.

[0078] The removal of the time or frequency path significantly reduced the model performance, indicating that dual-path differential modeling is indispensable. The frequency path uses bidirectional BiGSU to effectively capture global spectral correlation, while the time path uses unidirectional GSU to achieve causal time series modeling. The combination of the two forms a complete time-frequency feature modeling framework.

[0079] Replacing GSU with multi-gate SLSTM-2G and SLSTM-3G almost doubled the model parameters, but the performance decreased instead of improving. This verifies that under the constraint of the binary output bottleneck of SNN, the single forget gate GSU is the optimal recurrent unit design. Adding additional gate structures will only introduce redundant parameters and cannot improve the model performance.

[0080] Experimental results on adjusting the model's hidden dimension show that a hidden dimension H=128 achieves the optimal balance between model performance and the number of parameters. Too small a hidden dimension will lead to performance degradation, while too large a hidden dimension will result in parameter redundancy, which meets the requirements for lightweight deployment of edge devices.

[0081] Based on the above embodiments of this application, the speech enhancement solution of this application has at least the following advantages: (1) It has high parameter efficiency and is suitable for edge devices: This application uses a single-gate GSU as the core recurrent unit, and the total number of model parameters is only 394K, which is 4.5%–10.6% of the traditional representative ANN speech enhancement methods (DCCRN, FullSubNet+, GaGNet). Compared with the existing SNN baseline models (DPSNN, Spiking-FSN), the parameters are reduced by 31%–59%. At the same time, it utilizes event-driven computation and binary impulse activation to significantly reduce power consumption and computational overhead, making it naturally suitable for edge devices with extremely limited resources, such as hearing aids and portable voice terminals. (2) Multi-domain complementary fusion to improve speech enhancement performance: For the first time, a dual-branch decoding structure of amplitude spectrum and complex spectrum is constructed in SNN, which fully explores the advantages of stable energy estimation of amplitude spectrum and accurate phase recovery of complex spectrum. The complementary optimization of the two is achieved by weighted average fusion. All perceptual indicators on VoiceBank+DEMAND dataset are significantly better than traditional SNN speech enhancement models and surpass most traditional lightweight ANN methods. (3) Differentiated time-frequency modeling to capture complete speech features: Construct a dual-path GSU feature separator. The frequency path uses BiGSU to capture global spectral correlation, and the time path uses GSU to realize causal time modeling, forming a complete time-frequency dimension deep modeling framework. This solves the problem of insufficient capture of speech context information by traditional models and improves the robustness of the model to complex noise. (4) Adopting an end-to-end architecture design to achieve efficient streaming processing: This application constructs an end-to-end neuromorphic speech enhancement system from noisy speech input to enhanced speech output. The time path adopts unidirectional GSU to achieve causal modeling, which does not require future frame information, supports low-latency streaming speech enhancement, and meets the actual application needs of scenarios such as real-time communication and hearing aids.

[0082] (5) Feature fusion and attention mechanism are combined to improve feature utilization: The CBAM attention module is introduced into the encoder to realize adaptive adjustment of feature weights in the channel and spatial dimensions. At the same time, U-Net skip connections are introduced to fuse shallow and deep features of the encoder, which improves the accuracy of feature extraction and mask estimation, and further optimizes the perception quality of enhanced speech.

[0083] Figure 5 This is a schematic diagram of an electronic device illustrated in this specification according to an exemplary embodiment. Please refer to... Figure 5 At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, memory 508, a hardware acceleration device 510, and non-volatile memory 512, and may also include other hardware required for its functions. One or more embodiments of this application can be implemented in software, for example, the processor 502 reads the corresponding computer program from the non-volatile memory 512 into memory 508 and then runs it. Of course, in addition to software implementation, one or more embodiments of this application do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the above processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0084] Figure 6 This is a structural block diagram illustrating an exemplary embodiment of a neuromorphic speech enhancement device based on a dual-branch spiking neural network, which can be applied to, for example... Figure 5 The illustrated electronic device is used to implement the technical solution of this application. The device is configured to process an input time-domain noisy speech signal based on a pre-built neuromorphic speech enhancement model to obtain enhanced speech.

[0085] The speech enhancement device includes: a feature extraction unit 610, a feature encoding unit 620, a time-frequency feature processing unit 630, a feature decoding unit 640, and a speech reconstruction unit 650, wherein: The feature extraction unit 610 is used to extract the frequency domain multi-dimensional features of the noisy time-domain speech signal; The feature encoding unit 620 is used to perform convolutional attention encoding on the frequency domain multi-dimensional features based on the encoder to obtain a high-dimensional latent feature map. The time-frequency feature processing unit 630 is used to perform feature modeling on the high-dimensional latent feature map based on stacked dual-path gated pulse unit blocks to obtain a deep feature map; each dual-path gated pulse unit block includes a frequency path and a time path, the frequency path adopts a bidirectional gated pulse unit to perform bidirectional modeling along the frequency dimension; the time path adopts a unidirectional gated pulse unit to perform unidirectional causal modeling along the time dimension. The feature decoding unit 640 is used to obtain a complex spectrum estimation result through a complex spectrum decoding branch and an amplitude spectrum estimation result through an amplitude spectrum decoding branch for the depth feature map. The speech reconstruction unit 650 is used to fuse the complex spectrum estimation result with the amplitude spectrum estimation result, and reconstruct the time-domain speech signal based on the fusion result to obtain the enhanced speech.

[0086] In some embodiments, the time-frequency feature processing unit 630 includes a frequency processing module; The frequency processing module is used to rearrange the dimensions of the high-dimensional latent feature map or the feature map output from the previous dual-path gated pulse unit block to adapt it to the input format of the bidirectional gated pulse unit layer; input the rearranged feature map into the bidirectional gated pulse unit layer, perform bidirectional feature modeling along the forward and reverse order of the frequency dimension, and output bidirectional fused features; perform linear projection on the bidirectional fused features to compress its channel number to be consistent with the channel number of the input feature map, and perform inverse dimension rearrangement to restore the original data format; normalize the restored feature map, and add the residuals with the input feature map of the frequency path to output the frequency path processed feature map.

[0087] In some embodiments, the time-frequency feature processing unit 630 includes a time processing module; The time processing module is used to rearrange the dimensions of the feature map output by the frequency path to adapt it to the input format of the unidirectional gated pulse unit layer; input the rearranged feature map into the unidirectional gated pulse unit layer, perform unidirectional causal feature modeling only along the positive order of the time dimension, and output time features; perform linear projection on the time features to compress its number of channels to be consistent with the number of channels of the feature map output by the frequency path, and perform inverse dimension rearrangement to restore the original data format; normalize the restored feature map, and add the residuals with the input feature map of the time path to output the feature map processed by the current dual-path gated pulse unit block.

[0088] In some embodiments, the feature decoding unit 640 includes a complex spectrum decoding module; The complex spectrum decoding module is used to input the deep feature map into the complex spectrum decoder. The complex spectrum decoder contains multiple cascaded deconvolutional layers. Each deconvolutional layer and the feature map of the corresponding scale in the encoder are concatenated through skip connections. The last deconvolutional layer of the complex spectrum decoder outputs a complex spectrum mask. The tanh activation function is applied to the complex spectrum mask to generate DeepFilter coefficients. The DeepFilter coefficients are used to filter the noisy complex spectrum to obtain the complex spectrum estimation result.

[0089] In some embodiments, the feature decoding unit 640 includes an amplitude spectrum decoding module; The amplitude spectrum decoding module is used to input the depth feature map into the amplitude spectrum decoder. The amplitude spectrum decoder contains multiple cascaded deconvolutional layers. Each deconvolutional layer is concatenated with the feature map of the corresponding scale in the encoder through skip connections. The last deconvolutional layer of the amplitude spectrum decoder outputs a single-channel amplitude spectrum mask. The sigmoid activation function is applied to the amplitude spectrum mask to generate a normalized amplitude mask. The normalized amplitude mask is multiplied element-wise with the noisy amplitude spectrum to obtain the amplitude spectrum estimation result.

[0090] In some embodiments, the speech reconstruction unit 650 is configured to perform an inverse short-time Fourier transform on the fused enhanced complex spectrum to generate time-domain speech features; and to perform amplitude normalization processing on the time-domain speech features to limit the signal values ​​within a preset range to obtain the enhanced speech.

[0091] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0092] Accordingly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the above embodiments.

[0093] Accordingly, embodiments of this application also provide a computer program product configured to perform the methods described in any of the above embodiments.

[0094] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0095] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0096] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0097] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0098] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0099] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0100] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0101] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0102] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A neuromorphic speech enhancement method based on a dual-branch spiking neural network, characterized in that, A noisy time-domain speech signal is input into a pre-constructed neuromorphic speech enhancement model to obtain enhanced speech; wherein the processing steps of the model on the noisy time-domain speech signal are as follows: Step S1: Extract the frequency domain multi-dimensional features of the noisy time-domain speech signal; Step S2: Perform convolutional attention encoding on the frequency domain multi-dimensional features based on the encoder to obtain a high-dimensional latent feature map; Step S3: Based on stacked dual-path gated pulse unit blocks, feature modeling is performed on the high-dimensional latent feature map to obtain a deep feature map; each dual-path gated pulse unit block includes a frequency path and a time path. The frequency path uses bidirectional gated pulse units to perform bidirectional modeling along the frequency dimension; the time path uses unidirectional gated pulse units to perform unidirectional causal modeling along the time dimension. Step S4: For the depth feature map, obtain the complex spectrum estimation result through the complex spectrum decoding branch and obtain the amplitude spectrum estimation result through the amplitude spectrum decoding branch; Step S5: The complex spectrum estimation result is fused with the amplitude spectrum estimation result, and the time-domain speech signal is reconstructed based on the fusion result to obtain the enhanced speech.

2. The method according to claim 1, characterized in that, The frequency path processing steps in step S3 include: The high-dimensional latent feature map or the feature map output by the previous dual-path gated pulse unit block is rearranged in dimensions to adapt it to the input format of the bidirectional gated pulse unit layer. The rearranged feature map is input into the bidirectional gated pulse unit layer, and bidirectional feature modeling is performed along the forward and reverse order of the frequency dimension, respectively, and bidirectional fused features are output. Linear projection is performed on the bidirectional fused features to compress their number of channels to match the number of channels in the input feature map, and inverse dimensional rearrangement is performed to restore the original data format; The recovered feature map is normalized and then added to the residual of the input feature map of the frequency path to output the feature map after frequency path processing.

3. The method according to claim 1, characterized in that, The time path processing steps in step S3 include: The feature map output by the frequency path is rearranged in dimensions to fit the input format of the unidirectional gated pulse unit layer. The rearranged feature map is input into the unidirectional gated pulse unit layer, and unidirectional causal feature modeling is performed only along the positive order of the time dimension, and the time feature is output. Linear projection is performed on the time features to compress their number of channels to match the number of channels in the frequency path output feature map, and inverse dimensional rearrangement is performed to restore the original data format; The recovered feature map is normalized and added to the input feature map of the time path by residual, and the feature map after processing by the current dual-path gated pulse unit block is output.

4. The method according to claim 1, characterized in that, In step S3, the gated pulse unit adopts a single-gate structure based on a leak-integration ignition neuron, and its operation process includes: By performing a joint linear projection of the current input and the pulse output from the previous moment, an intermediate vector is obtained. The first half of the features is extracted from the intermediate vector and activated by the sigmoid function to obtain the forget gate; Extract the latter half of the features from the intermediate vector and update the membrane potential at the current time according to the forget gate; The membrane potential at the current moment is activated by a step function to generate a binary pulse output.

5. The method according to claim 1, characterized in that, The step S4, which obtains the complex spectrum estimation result through the complex spectrum decoding branch, includes: The depth feature map is input into the complex spectrum decoder, which contains multiple cascaded deconvolutional layers. Each deconvolutional layer is connected to the feature map of the corresponding scale in the encoder through skip connections to perform channel concatenation. The last deconvolutional layer of the complex spectrum decoder outputs a complex spectrum mask. Apply the tanh activation function to the complex spectrum mask to generate DeepFilter coefficients; The noisy complex spectrum is filtered using the DeepFilter coefficients to obtain the complex spectrum estimation result.

6. The method according to claim 1, characterized in that, The step S4, which obtains the amplitude spectrum estimation result through the amplitude spectrum decoding branch, includes: The depth feature map is input into the amplitude spectrum decoder, which contains multiple cascaded deconvolution layers. Each deconvolution layer is connected to the feature map of the corresponding scale in the encoder through skip connections to perform channel concatenation. The last deconvolution layer of the amplitude spectrum decoder outputs a single-channel amplitude spectrum mask. Apply the sigmoid activation function to the amplitude spectrum mask to generate a normalized amplitude mask; The normalized amplitude mask is multiplied element-wise with the noisy amplitude spectrum to obtain the amplitude spectrum estimation result.

7. The method according to claim 1, characterized in that, The fusion result includes the enhanced complex spectrum after fusion. Step S5, reconstructing the time-domain speech signal based on the fusion result, includes: Perform an inverse short-time Fourier transform on the fused enhanced complex spectrum to generate time-domain speech features; The time-domain speech features are subjected to amplitude normalization processing to limit the signal values ​​within a preset range, thereby obtaining the enhanced speech.

8. The method according to claim 1, characterized in that, Step S1 includes: Perform a short-time Fourier transform on the noisy time-domain speech signal to generate a complex frequency-domain spectrum; Extract features from the complex spectrum in three dimensions: real part, imaginary part, and amplitude spectrum. The real part, imaginary part, and amplitude spectrum are concatenated along the channel dimension to generate the multi-dimensional frequency domain feature.

9. A neuromorphic speech enhancement device based on a dual-branch spiking neural network, characterized in that, The device is configured to process an input time-domain noisy speech signal based on a pre-built neuromorphic speech enhancement model to obtain enhanced speech; wherein the device includes: The feature extraction unit is used to extract the frequency domain multi-dimensional features of the noisy time-domain speech signal; The feature encoding unit is used to perform convolutional attention encoding on the multi-dimensional features in the frequency domain based on the encoder to obtain a high-dimensional latent feature map. The time-frequency feature processing unit is used to perform feature modeling on the high-dimensional latent feature map based on stacked dual-path gated pulse unit blocks to obtain a deep feature map. Each dual-path gated pulse unit block includes a frequency path and a time path. The frequency path uses bidirectional gated pulse units to perform bidirectional modeling along the frequency dimension. The time path uses unidirectional gated pulse units to perform unidirectional causal modeling along the time dimension. The feature decoding unit is used to obtain a complex spectrum estimation result through a complex spectrum decoding branch and an amplitude spectrum estimation result through an amplitude spectrum decoding branch for the depth feature map. The speech reconstruction unit is used to fuse the complex spectrum estimation result with the amplitude spectrum estimation result, and reconstruct the time-domain speech signal based on the fusion result to obtain the enhanced speech.

10. An electronic device, characterized in that, include: processor; A computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method as described in any one of claims 1 to 8.

Images