Speech enhancement method and device based on real value and complex value network, terminal and medium

Abstract

This invention discloses a speech enhancement method, apparatus, terminal, and medium based on real-valued and complex-valued networks, relating to the field of signal processing. The method involves converting a noisy speech signal into an initial complex spectrum, and extracting an initial amplitude spectrum and an initial phase spectrum based on the initial complex spectrum. Real-valued and complex-valued speech enhancement networks are used to encode and extract features from the initial complex spectrum, initial amplitude spectrum, and initial phase spectrum, respectively, to determine the corresponding target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features. These target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features are then decoded and enhanced to determine the corresponding enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum. Finally, the enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum are fused and reconstructed to determine the enhanced speech signal. Therefore, this method effectively solves the performance limitations of existing time-frequency domain-based speech enhancement methods.

Description

Technical Field

[0001] This invention relates to the field of signal processing, and more particularly to speech enhancement methods, apparatus, terminals, and media based on real-valued and complex-valued networks. Background Technology

[0002] Speech enhancement (SE) technology is a core front-end technology for applications such as voice communication, speech recognition, and hearing aids. Its core purpose is to recover clear and clean speech from noisy speech signals, thereby improving speech quality and intelligibility.

[0003] Single-channel speech enhancement methods based on deep learning are mainly divided into two categories: time-domain and time-frequency domain. Time-domain methods directly manipulate the original waveform, which can completely preserve amplitude and phase coupling information, but they have high requirements for long-term modeling, a large number of parameters, and high computational cost, making them unsuitable for deployment on resource-constrained devices. Time-frequency domain methods are divided into real-valued and complex-valued neural network methods. Real-valued methods suffer from insufficient phase processing and limited robustness; while complex-valued methods can directly process complex features, they are prone to implicit compensation effects between amplitude and phase, leading to insufficient amplitude estimation accuracy and thus affecting the overall performance of speech enhancement.

[0004] Therefore, existing technologies still need improvement and development. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a speech enhancement method, apparatus, terminal and medium based on real-valued and complex-valued networks, in order to address the above-mentioned deficiencies of the prior art and solve the problem of limited performance of existing time-frequency domain-based speech enhancement methods.

[0006] The technical solution adopted by this invention to solve the problem is as follows: In a first aspect, embodiments of the present invention provide a speech enhancement method based on real-valued and complex-valued networks, wherein the method includes: The noisy speech signal is converted into an initial complex spectrum, and the initial amplitude spectrum and initial phase spectrum are extracted based on the initial complex spectrum; The initial complex spectrum, initial amplitude spectrum, and initial phase spectrum are encoded by a speech enhancement network based on real and complex values, respectively, and the corresponding initial complex spectrum features, initial amplitude spectrum features, and initial phase spectrum features are determined. The initial complex spectrum features, initial amplitude spectrum features, and initial phase spectrum features are input into the bottleneck module of the speech enhancement network based on real and complex values ​​for feature extraction and fusion interaction, and the target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features are determined accordingly. The bottleneck blocks in the bottleneck module are connected sequentially, and the output features of the previous bottleneck block are the input features of the next bottleneck block. The input features of the first bottleneck block and the output features of the last bottleneck block are connected in a skip connection. The target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features are decoded and enhanced by speech enhancement networks based on real and complex values, respectively, to determine the corresponding enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum. The enhanced speech signal is determined by fusing and reconstructing the enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum.

[0007] In one implementation, the input features of any bottleneck block include amplitude spectrum input features, phase spectrum input features, and complex spectrum input features; the output features include amplitude spectrum output features, phase spectrum output features, and complex spectrum output features, and the bottleneck block includes: The time-frequency transformation module determines the complex time-frequency characteristics based on the complex spectrum input characteristics, the amplitude time-frequency characteristics based on the amplitude spectrum input characteristics, and the phase time-frequency characteristics based on the phase spectrum input characteristics. The complex time-frequency features, amplitude time-frequency features, and phase time-frequency features are fused and interacted through the real-complex value fusion interaction module, and the corresponding complex interaction features, amplitude interaction features, and phase interaction features are output. The multidimensional collaborative attention module determines the corresponding complex spectrum attention, amplitude spectrum attention, and phase spectrum attention based on the complex interaction features, amplitude interaction features, and phase interaction features, respectively. The complex spectrum output features are determined based on the complex spectrum input features and complex spectrum attention; the amplitude spectrum output features are determined based on the amplitude spectrum input features and amplitude spectrum attention; and the phase spectrum output features are determined based on the phase spectrum input features and phase spectrum attention.

[0008] In one implementation, the time-frequency transformation module includes a time submodule and a frequency submodule. The time submodule is used to extract global dependencies and local dynamic features in the time dimension, and the frequency submodule is used to extract global and local dependencies in the frequency dimension. The time submodule and frequency submodule are connected sequentially, with the input and output terminals of the time submodule being connected in a skip connection, and the input and output terminals of the frequency submodule being connected in a skip connection. The temporal submodule is based on a sequentially connected feedforward neural network, a multi-head self-attention mechanism, a gated recurrent unit, and layer normalization. The frequency submodule is based on a sequentially connected feedforward neural network, a multi-head self-attention mechanism, convolution operations, and layer normalization.

[0009] In one implementation method, the complex time-frequency features, amplitude time-frequency features, and phase time-frequency features are fused and interacted through a real-complex value fusion interaction module, outputting corresponding complex interaction features, amplitude interaction features, and phase interaction features, including: Two-dimensional convolution, instance normalization, and nonlinear transformation are performed on the complex time-frequency features, amplitude time-frequency features, and phase time-frequency features respectively to determine the corresponding nonlinear complex features, nonlinear amplitude features, and nonlinear phase features. Amplitude adjustment characteristics are determined based on amplitude time-frequency characteristics and nonlinear amplitude characteristics; phase adjustment characteristics are determined based on phase time-frequency characteristics and nonlinear phase characteristics. Modulo and feature extraction are performed on nonlinear complex number features to determine real-valued amplitude features; amplitude interaction features are determined based on amplitude adjustment features and real-valued amplitude features. Phase and feature extraction are performed on nonlinear complex features to determine real-valued phase features; phase interaction features are determined based on phase adjustment features and real-valued phase features. By coupling nonlinear amplitude features and nonlinear phase features, deep complex features are determined; by fusing and extracting nonlinear complex features and deep complex features, complex interaction features are determined.

[0010] In one implementation method, a multi-dimensional collaborative attention module determines the corresponding complex spectral attention, amplitude spectral attention, and phase spectral attention based on complex interaction features, amplitude interaction features, and phase interaction features, respectively, including: Complex interaction features, amplitude interaction features, and phase interaction features are used as input features separately; The channel attention module determines the channel attention weights based on the input features, and then determines the channel attention based on the input features and the channel attention weights. The frequency attention is determined by applying frequency-dimensional attention weights to the input features using the frequency attention module. The temporal attention module performs temporal dimension attention weighting on the input features to determine the temporal attention; The output features corresponding to the input features are determined based on frequency attention, time attention, and channel attention. The output features are one of complex spectrum attention, amplitude spectrum attention, and phase spectrum attention.

[0011] In one implementation method, the target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features are decoded and enhanced respectively using a speech enhancement network based on real-valued and complex values, to determine the corresponding enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum, including: The target complex spectrum output features are decoded to determine the complex ratio mask and the first enhanced complex spectrum; the second enhanced complex spectrum is determined based on the complex ratio mask and the initial complex spectrum. The enhanced complex spectrum is determined based on the first and second enhanced complex spectra; Decode the output features of the target amplitude spectrum to determine the amplitude ratio mask; determine the enhanced amplitude spectrum based on the amplitude ratio mask and the initial amplitude spectrum. The target phase spectrum is output feature decoded to determine the enhanced phase spectrum.

[0012] In one implementation, the method further includes: A multi-objective joint loss function is constructed, and the parameters of the real-valued and complex-valued speech enhancement networks are optimized based on this multi-objective joint loss function. The multi-objective joint loss function is expressed as follows: , in, Indicates the joint loss of multiple objectives. Indicates time-domain loss, Indicates magnitude loss. Indicates actual loss. Indicates loss of the imaginary part. Indicates phase loss, This indicates the perceived loss in speech quality assessment. These represent the weights of each loss.

[0013] Secondly, embodiments of the present invention also provide a speech enhancement device based on real-valued and complex-valued networks, wherein the speech enhancement device based on real-valued and complex-valued networks includes: The signal conversion module is used to convert noisy speech signals into an initial complex spectrum, and to extract the initial amplitude spectrum and the initial phase spectrum based on the initial complex spectrum; The feature encoding module is used to encode the initial complex spectrum, initial amplitude spectrum, and initial phase spectrum respectively through a speech enhancement network based on real and complex values, and to determine the corresponding initial complex spectrum features, initial amplitude spectrum features, and initial phase spectrum features. The feature extraction module is used to input the initial complex spectrum features, initial amplitude spectrum features, and initial phase spectrum features into the bottleneck module of the speech enhancement network based on real and complex values ​​for feature extraction and fusion interaction, and correspondingly determine the target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features. The bottleneck blocks in the bottleneck module are connected sequentially, and the output feature of the previous bottleneck block is the input feature of the next bottleneck block. The input feature of the first bottleneck block and the output feature of the last bottleneck block are connected in a skip connection. The feature decoding module is used to decode and enhance the target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features respectively through a speech enhancement network based on real values ​​and complex values, to determine the enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum. The signal reconstruction module is used to fuse and reconstruct the enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum to determine the enhanced speech signal.

[0014] Thirdly, embodiments of the present invention also provide a terminal, the terminal including a memory and one or more processors; the memory stores one or more programs; the programs include instructions for executing the speech enhancement method based on real-valued and complex-valued networks as described above; the processor is used to execute the programs.

[0015] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a plurality of instructions, wherein the instructions are adapted to be loaded and executed by a processor to implement any of the above-described speech enhancement methods based on real-valued and complex-valued networks.

[0016] The beneficial effects of this invention are as follows: In this embodiment, the noisy speech signal is converted into an initial complex spectrum, and initial amplitude and phase spectra are extracted based on the initial complex spectrum. Real-valued and complex-valued speech enhancement networks are used to encode and extract features from the initial complex spectrum, initial amplitude spectrum, and initial phase spectrum, respectively, to determine the corresponding target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features. These target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features are then decoded and enhanced to determine the corresponding enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum. Finally, the enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum are fused and reconstructed to determine the enhanced speech signal. Therefore, this effectively solves the problem of limited performance in existing time-frequency domain-based speech enhancement methods. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the speech enhancement method based on real-valued and complex-valued networks provided in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the structure of a speech enhancement network based on real and complex values ​​provided in an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the structure of a dense encoder provided in an embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram of the time-frequency conversion module provided in an embodiment of the present invention.

[0022] Figure 5This is a schematic diagram of the structure of the real-complex value fusion interaction module provided in an embodiment of the present invention.

[0023] Figure 6 This is a schematic diagram of the structure of the multidimensional collaborative attention module provided in an embodiment of the present invention.

[0024] Figure 7 This is a schematic diagram of the mask decoder provided in an embodiment of the present invention.

[0025] Figure 8 This is a schematic diagram of the phase decoder provided in an embodiment of the present invention.

[0026] Figure 9 This is a schematic diagram of the complex mask decoder provided in an embodiment of the present invention.

[0027] Figure 10 This is a schematic diagram of the complex spectrum decoder provided in an embodiment of the present invention.

[0028] Figure 11 This is a schematic diagram of the internal modules of the speech enhancement device based on real-valued and complex-valued networks provided in an embodiment of the present invention.

[0029] Figure 12 This is a schematic diagram of the terminal provided in the embodiment of the present invention. Detailed Implementation

[0030] This invention discloses a speech enhancement method, apparatus, terminal, and medium based on real-valued and complex-valued networks. To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention.

[0031] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0032] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0033] Speech enhancement (SE) technology is a core front-end technology for applications such as voice communication, speech recognition, and hearing aids. Its core purpose is to recover clear and clean speech from noisy speech signals, thereby improving speech quality and intelligibility.

[0034] Single-channel speech enhancement methods based on deep learning are mainly divided into two categories: time-domain and time-frequency domain. Time-domain methods directly manipulate the original waveform, which can completely preserve amplitude and phase coupling information, but they have high requirements for long-term modeling, a large number of parameters, and high computational cost, making them unsuitable for deployment on resource-constrained devices. Time-frequency domain methods are divided into real-valued and complex-valued neural network methods. Real-valued methods suffer from insufficient phase processing and limited robustness; while complex-valued methods can directly process complex features, they are prone to implicit compensation effects between amplitude and phase, leading to insufficient amplitude estimation accuracy and thus affecting the overall performance of speech enhancement.

[0035] To address the aforementioned shortcomings of existing technologies, this invention provides a speech enhancement method based on real-valued and complex-valued networks. The method involves converting a noisy speech signal into an initial complex spectrum, and extracting initial amplitude and phase spectra based on this spectrum. Real-valued and complex-valued speech enhancement networks are then used to encode and extract features from the initial complex, amplitude, and phase spectra, respectively, to determine the corresponding target complex, amplitude, and phase spectrum output features. These output features are then decoded and enhanced to determine the corresponding enhanced complex, amplitude, and phase spectra. Finally, the enhanced complex, amplitude, and phase spectra are fused and reconstructed to determine the enhanced speech signal. Therefore, this method effectively solves the performance limitations of existing time-frequency domain-based speech enhancement methods.

[0036] Exemplary method: like Figure 1 As shown, the method includes: Step S100: Convert the noisy speech signal into an initial complex spectrum, and extract the initial amplitude spectrum and initial phase spectrum based on the initial complex spectrum; Acquiring noisy speech signals The noisy speech signal is preprocessed by framing and windowing to reduce spectral leakage between frames and improve the accuracy of subsequent spectral conversion. For each preprocessed frame of noisy speech signal, a Short-Time Fourier Transform (STFT) is performed to convert the noisy speech signal in the time domain into a complex spectrum in the frequency domain. This refers to the initial complex spectrum. Each frequency point in the initial complex spectrum corresponds to a complex number, and each complex number is composed of its real part. and the virtual part It consists of components used to fully characterize the signal features at the corresponding frequency point.

[0037] Based on the obtained initial complex spectrum, analytical calculations are performed on the complex numbers corresponding to each frequency point to extract the initial amplitude spectrum for each frequency point. and the initial phase spectrum (or the wrapped phase spectrum) The initial amplitude spectrum is obtained by calculating the modulus of each complex number. Specifically, the real and imaginary parts of the complex number are squared and summed, and the square root of the sum is taken to characterize the signal intensity at each frequency point. The initial phase spectrum is obtained by calculating the argument of each complex number. Specifically, the arctangent function is used to solve for the corresponding angle based on the real and imaginary parts of the complex number. This angle is used to characterize the phase information of the signal at each frequency point. Finally, the initial amplitude spectrum and initial phase spectrum corresponding to the initial complex number spectrum are obtained.

[0038] Step S200: Encode the initial complex spectrum, initial amplitude spectrum, and initial phase spectrum respectively using a speech enhancement network based on real and complex values, and determine the corresponding initial complex spectrum features, initial amplitude spectrum features, and initial phase spectrum features; The initial complex spectrum, initial amplitude spectrum, and initial phase spectrum are encoded into a potential time-frequency representation using a dense encoder. Specifically, the initial amplitude spectrum is encoded using a real-number dense encoder to obtain the initial amplitude spectrum features corresponding to the initial amplitude spectrum; the initial complex spectrum is encoded using a complex-number dense encoder to obtain the initial complex spectrum features corresponding to the initial complex spectrum; and the phase spectrum is encoded using a real-number dense encoder to determine the initial phase spectrum features corresponding to the initial phase spectrum.

[0039] The structures of real-number dense encoders and complex-number dense encoders are as follows: Figure 3 As shown, each encoder consists of multiple convolutional layers, where each convolutional operation is followed by instance normalization and a parameterized ReLU (PReLU) activation function. High-dimensional features are extracted from the initial complex spectrum, initial amplitude spectrum, or initial phase spectrum of the input through an initial convolutional layer, and these high-dimensional features are fed into N dilated convolutional layers, where the dilation rate is a power of 2. Specifically, the number of dilated convolutional layers can be four. Residual connections are included between the convolutional layers. The last convolutional layer of the encoder is used for downsampling to reduce the computational cost of the network.

[0040] like Figure 1As shown, the method further includes: Step S300: Input the initial complex spectrum features, initial amplitude spectrum features, and initial phase spectrum features into the bottleneck module of the speech enhancement network based on real and complex values ​​for feature extraction and fusion interaction, and determine the target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features accordingly. The bottleneck blocks in the bottleneck module are connected sequentially, and the output features of the previous bottleneck block are the input features of the next bottleneck block. The input features of the first bottleneck block and the output features of the last bottleneck block are connected in a skip connection. The encoded initial complex spectrum features, initial amplitude spectrum features, and initial phase spectrum features are input into the bottleneck module. Feature extraction and interaction are performed through the bottleneck blocks of the bottleneck module to obtain the target complex spectrum output features corresponding to the initial complex spectrum features, the target amplitude spectrum output features corresponding to the initial amplitude spectrum features, and the target phase spectrum output features corresponding to the initial phase spectrum features. The bottleneck module consists of several bottleneck blocks, which are sequentially connected, with the input features of the first bottleneck block and the output features of the last bottleneck block skipping connections. Optionally, when the number of bottleneck blocks is three, a speech enhancement network based on real-valued and complex values ​​can achieve better speech enhancement performance.

[0041] In one implementation, the input features of any bottleneck block include amplitude spectrum input features, phase spectrum input features, and complex spectrum input features; the output features include amplitude spectrum output features, phase spectrum output features, and complex spectrum output features, and the bottleneck block includes: Step S301: The time-frequency transformation module determines the complex time-frequency characteristics based on the complex spectrum input characteristics, the amplitude time-frequency characteristics based on the amplitude spectrum input characteristics, and the phase time-frequency characteristics based on the phase spectrum input characteristics. Step S302: The complex time-frequency features, amplitude time-frequency features, and phase time-frequency features are fused and interacted through the real-complex value fusion interaction module, and the corresponding complex interaction features, amplitude interaction features, and phase interaction features are output. Step S303: The multidimensional collaborative attention module determines the corresponding complex spectrum attention, amplitude spectrum attention, and phase spectrum attention based on the complex interaction features, amplitude interaction features, and phase interaction features, respectively. Step S304: Determine the complex spectrum output features based on the complex spectrum input features and complex spectrum attention; determine the amplitude spectrum output features based on the amplitude spectrum input features and amplitude spectrum attention; and determine the phase spectrum output features based on the phase spectrum input features and phase spectrum attention.

[0042] Each bottleneck block includes a Time-Frequency Transformer (TF-former) module, a Real-Complex Valued Fusion Interactive Module (RCFI), and a Multi-Dimensional Collaborative Attention Module (MCA). Within the bottleneck block, the TF-former module captures time-frequency dependencies; the RCFI is specifically designed for real-complex fusion networks to enable cross-stream interaction, integrating amplitude, phase, and complex features to allow the three streams to share complementary information, thereby improving overall performance; and the MCA captures the information distribution in the time-frequency domain based on the three types of network information streams.

[0043] If the input features of each bottleneck block are amplitude spectrum input features, phase spectrum input features, and complex spectrum input features, and the output features are amplitude spectrum output features, phase spectrum output features, and complex spectrum output features, then when the bottleneck block is the first bottleneck block, the amplitude spectrum input features, phase spectrum input features, and complex spectrum input features correspond to the initial amplitude spectrum features, initial phase spectrum features, and initial complex spectrum features, respectively. If the bottleneck block is not the first bottleneck block, the amplitude spectrum input features, phase spectrum input features, and complex spectrum input features correspond to the amplitude spectrum output features, phase spectrum output features, and complex spectrum output features of the previous bottleneck block, respectively.

[0044] Specifically, such as Figure 2 As shown, the bottleneck block is divided into three branches: a first complex branch, a first amplitude branch, and a first phase branch. For the first phase branch, the phase spectrum input characteristics are input into the time-frequency transformation module of this branch to determine the phase time-frequency characteristics; for the first amplitude branch, the amplitude spectrum input characteristics are input into the time-frequency transformation module of this branch to determine the amplitude time-frequency characteristics; for the first complex branch, the complex spectrum input characteristics are input into the time-frequency transformation module of this branch to determine the complex time-frequency characteristics.

[0045] The output features of the time-frequency transformation modules on each branch in each bottleneck block are input into the real-complex value fusion interaction module. That is, the complex time-frequency features, amplitude time-frequency features, and phase time-frequency features are input into the real-complex value fusion interaction module for fusion and interaction, and the complex interaction features corresponding to the complex time-frequency features, the amplitude interaction features corresponding to the amplitude time-frequency features, and the phase interaction features corresponding to the phase time-frequency features are output.

[0046] Phase interaction features are input into the multidimensional collaborative attention module on the first phase branch to obtain phase spectrum attention; amplitude interaction features are input into the multidimensional collaborative attention module on the first amplitude branch to obtain amplitude spectrum attention; complex interaction features are input into the multidimensional collaborative attention module on the first complex branch to obtain complex spectrum attention.

[0047] The output of the multidimensional collaborative attention module is residually connected with the input features of the bottleneck block to obtain the output features of the bottleneck block. Specifically, the complex spectrum output features are determined based on the complex spectrum input features and complex spectrum attention, the amplitude spectrum output features are determined based on the amplitude spectrum input features and amplitude spectrum attention, and the phase spectrum output features are determined based on the phase spectrum input features and phase spectrum attention.

[0048] If the current bottleneck block is not the last bottleneck block, the phase spectrum output feature can be used as the input feature of the time-frequency transformation of the first phase branch of the next bottleneck block; similarly, the amplitude spectrum output feature and the complex spectrum output feature can be used as the input features of the time-frequency transformation of the corresponding branches.

[0049] If the current bottleneck block is the last bottleneck block, then the output feature of the current bottleneck block is connected to the input feature of the first bottleneck block via a skip connection to obtain the target complex spectrum output feature, the target amplitude spectrum output feature, and the target phase spectrum output feature. Specifically, the initial complex spectrum feature is connected to the complex spectrum output feature (the last bottleneck block) via a skip connection to obtain the target complex spectrum output feature; the initial amplitude spectrum feature is connected to the amplitude spectrum output feature via a skip connection to obtain the target amplitude spectrum output feature; and the initial phase spectrum feature is connected to the phase spectrum output feature via a skip connection to obtain the target phase spectrum output feature.

[0050] The bottleneck block uses a multi-branch network with complex value branches, amplitude branches, and phase branches, where each branch learns complementary information in an independent feature space.

[0051] In one implementation, the time-frequency transformation module includes a time submodule and a frequency submodule, wherein the time submodule is used to extract global dependencies and local dynamic features in the time dimension; and the frequency submodule is used to extract global and local dependencies in the frequency dimension. The time submodule and frequency submodule are connected sequentially, with the input and output terminals of the time submodule being connected in a skip connection, and the input and output terminals of the frequency submodule being connected in a skip connection. The temporal submodule is based on a sequentially connected feedforward neural network, a multi-head self-attention mechanism, a gated recurrent unit, and layer normalization. The frequency submodule is based on a sequentially connected feedforward neural network, a multi-head self-attention mechanism, convolution operations, and layer normalization.

[0052] like Figure 4As shown, the time-frequency transformation module (TF-former module) leverages the powerful capabilities of the Transformer in modeling speech characteristics to extract richer feature information from the time-frequency domain. It includes sequentially connected time sub-modules (Time-GRUformer and Frequency-Conformer), with skip connections between the input and output of the time sub-module and the frequency sub-module. The Transformer, with its self-attention mechanism, effectively captures local and global dependencies and has achieved good results in various speech tasks.

[0053] The input features (first input features) of the time-frequency transformation module are processed by the time submodule. First, the input features are initially transformed and their dimensions are adjusted using a feedforward neural network (FFN). Then, the global dependencies are extracted in the time dimension using a multi-head self-attention mechanism (MSA module). Finally, the gated recurrent unit (GRU) is introduced to finely model the local dynamic features in the continuous time series, thereby capturing the temporal dynamics of the speech signal.

[0054] Secondly, through the frequency submodule, since the spectrum of speech signals typically exhibits local correlation and stationarity, convolution operations can efficiently extract local spectral features. The frequency submodule processes time-frequency features along the frequency dimension. First, a multi-head self-attention mechanism is used, followed by a local feature extraction method suitable for spectral data to capture global and local dependencies in the frequency dimension, thereby achieving a more comprehensive spectral analysis of the speech signal. Furthermore, in the first complex branch, all Transformer modules used are complex Transformers.

[0055] In one implementation, the complex time-frequency features, amplitude time-frequency features, and phase time-frequency features are fused and interacted through a real-complex value fusion interaction module, outputting corresponding complex interaction features, amplitude interaction features, and phase interaction features, including: Step S3021: Perform two-dimensional convolution, instance normalization, and nonlinear transformation on the complex time-frequency features, amplitude time-frequency features, and phase time-frequency features respectively to determine the corresponding nonlinear complex features, nonlinear amplitude features, and nonlinear phase features; Step S3022: Determine the amplitude adjustment characteristics based on the amplitude time-frequency characteristics and nonlinear amplitude characteristics; determine the phase adjustment characteristics based on the phase time-frequency characteristics and nonlinear phase characteristics. Step S3023: Take the modulus value and extract features of the nonlinear complex number characteristics to determine the real-valued amplitude features; determine the amplitude interaction features based on the amplitude adjustment features and the real-valued amplitude features; Step S3024: Extract phase and features from the nonlinear complex features to determine the real-valued phase features; determine the phase interaction features based on the phase adjustment features and the real-valued phase features; Step S3025: Couple nonlinear amplitude features and nonlinear phase features to determine deep complex features; fuse and extract features from nonlinear complex features and deep complex features to determine complex interaction features.

[0056] The real-complex fusion interaction module comprises three branches: a second complex branch for processing complex time-frequency features, a second amplitude branch for processing amplitude time-frequency features, and a second phase branch for processing phase time-frequency features. Input features undergo bidirectional feature transfer and gated fusion between the different branches. The amplitude-phase information output from the second complex branch is modulated into the real-valued branch features after modulo-phase conversion. The speech structure features extracted by the second real-valued branch (including the second phase branch and the second amplitude branch) are mapped to the complex domain and fused with the complex-valued branch features, thereby achieving bidirectional information interaction and collaborative optimization of real and complex features.

[0057] Specifically, such as Figure 5 As shown, in the second amplitude branch, the amplitude time-frequency characteristics are... Two-dimensional convolution, instance normalization, and Tanh activation are performed to obtain nonlinear amplitude features. Then, multiply the nonlinear amplitude feature with the amplitude time-frequency feature to obtain the amplitude adjustment feature. ,Right now , .

[0058] In the second phase branch, the phase time-frequency characteristics Two-dimensional convolution, instance normalization, and Tanh activation are performed to obtain nonlinear phase features. The nonlinear phase feature is then multiplied by the phase time-frequency feature to obtain the phase adjustment feature. ,Right now , .

[0059] In the second complex branch, the complex time-frequency features Complex convolution, complex normalization, and complex PReLU activation are performed to extract nonlinear complex features. ,Right now Next, the second complex branch interacts with the second amplitude branch and the second phase branch.

[0060] Since deep complex-valued networks cannot directly exchange information with deep real-valued networks, on the one hand, nonlinear complex features are incorporated. The modulus value is converted into a real-valued feature related to the amplitude, and then adjusted by a gating mechanism and amplitude adjustment feature. Interaction yields amplitude interaction features ,Right now On the other hand, the arctan function is used to obtain the nonlinear complex characteristics. The phase, and the phase adjustment features Phase interaction features are obtained through interaction. ,Right now .

[0061] To facilitate information transfer between the second amplitude branch, the second phase branch, and the second complex branch, the nonlinear amplitude characteristics of the second amplitude branch are... Nonlinear phase characteristics of the second phase branch The features are coupled to obtain deep complex features, which are then combined with the nonlinear complex features of the second complex branch. Add them together. After complex 2D convolution, complex instance normalization, and complex PReLU processing, the output of the second complex branch is obtained, representing the complex interaction features. The expression is: .

[0062] In one implementation, a multi-dimensional collaborative attention module determines the corresponding complex spectral attention, amplitude spectral attention, and phase spectral attention based on complex interaction features, amplitude interaction features, and phase interaction features, respectively, including: Step S3031: Take the complex interaction feature, amplitude interaction feature and phase interaction feature as input features respectively; Step S3032: Determine the channel attention weights based on the input features through the channel attention module, and determine the channel attention based on the input features and the channel attention weights; Step S3033: Apply frequency-dimension attention weights to the input features using the frequency attention module to determine the frequency attention; Step S3034: Apply time-dimensional attention weights to the input features using the time attention module to determine the time attention; Step S3035: Determine the output feature corresponding to the input feature based on frequency attention, time attention, and channel attention. The output feature is one of complex spectrum attention, amplitude spectrum attention, and phase spectrum attention.

[0063] like Figure 2 As shown, each branch of the bottleneck block includes a multidimensional collaborative attention module. The output feature of the multidimensional collaborative attention module corresponds to its input feature. If the input feature is a complex interaction feature, the output feature is complex attention; if the input feature is an amplitude interaction feature, the output feature is amplitude attention; if the input feature is a phase interaction feature, the output feature is phase attention.

[0064] Specifically, such as Figure 6 As shown, the structure of the multidimensional collaborative attention module mainly consists of three parallel attention modules. , and The module is composed of components used to extract attention features from the channel, frequency, and time dimensions. Channel Attention Module By analyzing the input features (second input features) Global information aggregation is performed to generate channel attention weights. Specifically, firstly, Input features in the channel dimension Global max pooling and global average pooling are used to obtain vectors. and Max pooling captures the most salient features in the feature map, while average pooling provides smoothing information about the overall features. Combining these two methods allows for the simultaneous acquisition of salient features and global information, thereby enhancing the model's expressive power. The vectors are then processed through a small, fully connected network with shared weights. and This process generates two channel attention weights. Next, these two channel attention weights are added together and then passed through a sigmoid activation function to obtain the final channel attention weights. Finally, the channel attention weights are multiplied by the input features to obtain the final channel attention. .

[0065] The frequency attention module performs the same pooling operation on the input features along the frequency dimension to obtain two matrices. and The two matrices are concatenated along the channel dimension, merging two different frequency dimension information into a single two-dimensional convolution operation. This extracts more representative frequency dimension features while reducing the number of channels, ensuring the final output matches the desired result. and The dimensions are matched. To further capture the energy distribution in the frequency dimension, two one-dimensional dilated convolutions are used to capture frequency dimension information under different receptive fields, enabling the model to simultaneously focus on local and global energy distributions. .in and These represent dilated convolutions with dilation rates of 1 and 2, respectively.

[0066] The temporal attention module, similar to the frequency attention module, operates on the input features along the temporal dimension to capture the temporal dynamics of speech. The frequency attention module performs the same pooling operation on the input features along the temporal dimension, resulting in two matrices. and The two matrices are then concatenated along the channel dimension, merging two different temporal dimensions into a single two-dimensional convolution operation. Two one-dimensional dilated convolutions are then used to capture temporal information under different receptive fields. .

[0067] Finally, the outputs of the frequency attention module, the temporal attention module, and the channel attention module are multiplied to obtain the final multidimensional collaborative attention. The multidimensional collaborative attention module can be represented as: , in, This represents matrix multiplication, while This indicates element-wise multiplication.

[0068] like Figure 1 As shown, the method further includes: Step S400: Decode and enhance the target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features respectively through a speech enhancement network based on real and complex values, and determine the corresponding enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum. In simple terms, the target complex spectrum output features are decoded and enhanced to determine the corresponding enhanced complex spectrum; the target amplitude spectrum output features are decoded to determine the enhanced amplitude spectrum; and the target phase spectrum output features are decoded to determine the enhanced phase spectrum.

[0069] In one implementation, a speech enhancement network based on real-valued and complex-valued methods decodes and enhances the target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features, respectively, to determine the corresponding enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum, including: Step S401: Decode the output features of the target complex spectrum to determine the complex ratio mask and the first enhanced complex spectrum; determine the second enhanced complex spectrum based on the complex ratio mask and the complex spectrum; Step S402: Determine the enhanced complex spectrum based on the first enhanced complex spectrum and the second enhanced complex spectrum; Step S403: Decode the output features of the target amplitude spectrum to determine the amplitude ratio mask; determine the enhanced amplitude spectrum based on the amplitude ratio mask and the initial amplitude spectrum features; Step S404: Decode the output features of the target phase spectrum to determine the enhanced phase spectrum.

[0070] like Figure 2 As shown, four decoders are used to predict each target. The amplitude branch uses a mask decoder to decode the target amplitude spectrum output features to obtain the amplitude ratio mask. The amplitude ratio mask is compared with the initial amplitude spectrum. Dot product yields the enhanced amplitude spectrum The phase branch uses a phase decoder to decode the output features of the target phase spectrum to obtain the enhanced phase spectrum. In the complex branch, two parallel decoders, a complex mask decoder and a complex decoder, are used. The complex mask decoder predicts the complex ratio mask based on the output features of the target complex spectrum. And predict the first enhanced complex spectrum based on the output features of the target complex spectrum using a complex decoder. The complex ratio mask is applied to the initial complex spectrum via complex multiplication to obtain the second enhanced complex spectrum. The enhanced complex spectrum is obtained by weighted summation of the first and second enhanced complex spectra. .

[0071] Specifically, the mask decoder is as follows: Figure 7 As shown, the target amplitude spectrum output features enter a dilated convolutional layer, and then are upsampled through a transposed convolutional layer to restore the depth-time frequency features to the same size as the input. Then, convolutional layers and operations are applied, and finally, the amplitude ratio mask is obtained through the LSigmoid activation function. The structure of a complex mask decoder is as follows: Figure 9 As shown, its structure is similar to that of a mask decoder. The main difference is that the complex mask decoder uses a deep complex-valued neural network and ends with Tanh as the activation function.

[0072] Phase decoder such as Figure 8 As shown, the target phase spectrum output features are fed into a dilated convolutional layer, followed by upsampling using a transposed convolutional layer to restore the time-frequency features to their original size. Then, the real and imaginary parts of the time-frequency features are predicted using two convolutional modules respectively, and finally, the enhanced phase spectrum is obtained through the Arctan2 function. Complex spectrum decoder, such as Figure 10 As shown, the structure of the complex spectrum decoder is similar to that of the phase decoder, except that a complex convolution operation is directly applied after upsampling to obtain the first enhanced complex spectrum. .

[0073] Step S500: The enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum are fused and reconstructed to determine the enhanced speech signal.

[0074] like Figure 2 As shown, the enhanced amplitude spectrum and enhanced phase spectrum are coupled to obtain the fused complex spectrum; the fused complex spectrum and enhanced complex spectrum are combined to obtain the speech enhancement spectrum; the speech enhancement spectrum is subjected to inverse short-time Fourier transform to obtain the enhanced speech signal.

[0075] In one implementation, the method further includes: To support the multi-branch joint learning of the complex spectrum, amplitude spectrum, and phase spectrum, a multi-objective joint loss function composed of multiple sub-loss components is constructed. Multi-objective optimization is achieved through weighted summation, i.e.: , in, Indicates the joint loss of multiple objectives. Indicates magnitude loss. , and These represent the amplitude spectra of the clean speech signal and the enhanced speech signal, respectively. express Norm; Indicates actual loss. , and Let represent the real parts of the complex spectra of the clean speech signal and the enhanced speech signal, respectively; Indicates loss of the imaginary part. , and Let represent the imaginary parts of the complex spectra of the clean speech signal and the enhanced speech signal, respectively. Indicates phase loss, This indicates the perceived loss in speech quality assessment. Indicates time-domain loss, , and These represent clean speech and enhanced speech, respectively. These represent the weights of each loss. In the experiment... Set to respectively This is used to balance the weights of the various loss components.

[0076] Because phase has an enveloping property, directly calculating the absolute difference between the clean phase spectrum and the enhanced phase spectrum does not accurately reflect the distance between them. Phase loss is defined. The specific components include instantaneous phase loss. Group delay phase loss and instantaneous angular frequency loss Three parts, among which, express Norm, and These represent the differentiation operations along the frequency and time directions, respectively. It is a type of unwrapping function. This can avoid error propagation caused by phase wrapping issues. This is a rounding operation to the nearest integer.

[0077] Using PESQ (Perceptual Speech Quality Assessment) as the target assessment metric, and training an assessment network, we obtain the perceptual loss for speech quality assessment. , is represented as: .

[0078] Based on the above embodiments, the present invention also provides a speech enhancement device based on real-valued and complex-valued networks, such as... Figure 11 As shown, the device includes: Signal conversion module 01 is used to convert noisy speech signals into initial complex spectra, and extract initial amplitude spectrum and initial phase spectrum based on the initial complex spectrum; The feature encoding module 02 is used to encode the initial complex spectrum, initial amplitude spectrum and initial phase spectrum respectively through a speech enhancement network based on real value and complex value, and to determine the corresponding initial complex spectrum features, initial amplitude spectrum features and initial phase spectrum features. The feature extraction module 03 is used to input the initial complex spectrum features, initial amplitude spectrum features, and initial phase spectrum features into the bottleneck module of the speech enhancement network based on real and complex values ​​for feature extraction and fusion interaction, and correspondingly determine the target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features. The bottleneck blocks in the bottleneck module are connected sequentially, and the output features of the previous bottleneck block are the input features of the next bottleneck block. The input features of the first bottleneck block and the output features of the last bottleneck block are connected in a skip connection. The feature decoding module 04 is used to decode and enhance the target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features respectively through a speech enhancement network based on real and complex values, and to determine the corresponding enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum. Signal reconstruction module 05 is used to fuse and reconstruct the enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum to determine the enhanced speech signal.

[0079] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 12 As shown, the terminal includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a speech enhancement method based on real-valued and complex-valued networks. The display screen can be an LCD screen or an e-ink screen.

[0080] Those skilled in the art will understand that Figure 12 The schematic diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0081] In one implementation, the terminal's memory stores one or more programs, and these programs are configured to be executed by one or more processors, and the programs contain instructions for performing a speech enhancement method based on real-valued and complex-valued networks.

[0082] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0083] In summary, this invention discloses a speech enhancement method, apparatus, terminal, and medium based on real-valued and complex-valued networks. The method converts a noisy speech signal into an initial complex spectrum, and extracts an initial amplitude spectrum and an initial phase spectrum based on the initial complex spectrum. It then encodes and extracts features from the initial complex spectrum, initial amplitude spectrum, and initial phase spectrum using real-valued and complex-valued speech enhancement networks, respectively, to determine the corresponding target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features. Finally, it decodes and enhances these target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features to determine the corresponding enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum. Finally, it fuses and reconstructs the enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum to determine the enhanced speech signal. Therefore, this method effectively solves the performance limitations of existing time-frequency domain-based speech enhancement methods.

[0084] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A speech enhancement method based on real and complex valued networks, characterized in that, The method includes: The noisy speech signal is converted into an initial complex spectrum, and the initial amplitude spectrum and initial phase spectrum are extracted based on the initial complex spectrum; The initial complex spectrum, initial amplitude spectrum, and initial phase spectrum are encoded by a speech enhancement network based on real and complex values, respectively, and the corresponding initial complex spectrum features, initial amplitude spectrum features, and initial phase spectrum features are determined. The initial complex spectrum features, initial amplitude spectrum features, and initial phase spectrum features are input into the bottleneck module of the speech enhancement network based on real and complex values ​​for feature extraction and fusion interaction, and the target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features are determined accordingly. The bottleneck blocks in the bottleneck module are connected sequentially, and the output features of the previous bottleneck block are the input features of the next bottleneck block. The input features of the first bottleneck block and the output features of the last bottleneck block are connected in a skip connection. The target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features are decoded and enhanced by speech enhancement networks based on real and complex values, respectively, to determine the enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum. The enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum are fused and reconstructed to determine the enhanced speech signal.

2. The speech enhancement method based on real-valued and complex-valued networks according to claim 1, characterized in that, The input features of any of the bottleneck blocks include amplitude spectrum input features, phase spectrum input features, and complex spectrum input features; The output features include amplitude spectrum output features, phase spectrum output features, and complex spectrum output features. The bottleneck block includes: The time-frequency transformation module determines the complex time-frequency characteristics based on the complex spectrum input characteristics, the amplitude time-frequency characteristics based on the amplitude spectrum input characteristics, and the phase time-frequency characteristics based on the phase spectrum input characteristics. The complex time-frequency features, amplitude time-frequency features, and phase time-frequency features are fused and interacted through the real-complex value fusion interaction module, and the corresponding complex interaction features, amplitude interaction features, and phase interaction features are output. The multidimensional collaborative attention module determines the corresponding complex spectrum attention, amplitude spectrum attention, and phase spectrum attention based on the complex interaction features, amplitude interaction features, and phase interaction features, respectively. The complex spectrum output features are determined based on the complex spectrum input features and complex spectrum attention; the amplitude spectrum output features are determined based on the amplitude spectrum input features and amplitude spectrum attention; and the phase spectrum output features are determined based on the phase spectrum input features and phase spectrum attention.

3. The speech enhancement method based on real-valued and complex-valued networks according to claim 2, characterized in that, The time-frequency transformation module includes a time submodule and a frequency submodule. The time submodule is used to extract global dependencies and local dynamic features in the time dimension; the frequency submodule is used to extract global and local dependencies in the frequency dimension. The time submodule and frequency submodule are connected sequentially, with the input and output terminals of the time submodule being connected in a skip connection, and the input and output terminals of the frequency submodule being connected in a skip connection. The temporal submodule is based on a sequentially connected feedforward neural network, a multi-head self-attention mechanism, a gated recurrent unit, and layer normalization. The frequency submodule is based on a sequentially connected feedforward neural network, a multi-head self-attention mechanism, convolution operations, and layer normalization.

4. The speech enhancement method based on real-valued and complex-valued networks according to claim 2, characterized in that, The complex time-frequency features, amplitude time-frequency features, and phase time-frequency features are fused and interacted through the real-complex value fusion interaction module, outputting the corresponding complex interaction features, amplitude interaction features, and phase interaction features, including: Two-dimensional convolution, instance normalization, and nonlinear transformation are performed on the complex time-frequency features, amplitude time-frequency features, and phase time-frequency features respectively to determine the corresponding nonlinear complex features, nonlinear amplitude features, and nonlinear phase features. Amplitude adjustment characteristics are determined based on amplitude time-frequency characteristics and nonlinear amplitude characteristics; phase adjustment characteristics are determined based on phase time-frequency characteristics and nonlinear phase characteristics. Modulo and feature extraction are performed on nonlinear complex number features to determine real-valued amplitude features; amplitude interaction features are determined based on amplitude adjustment features and real-valued amplitude features. Phase and feature extraction are performed on nonlinear complex features to determine real-valued phase features; phase interaction features are determined based on phase adjustment features and real-valued phase features. By coupling nonlinear amplitude features and nonlinear phase features, deep complex features are determined; by fusing and extracting nonlinear complex features and deep complex features, complex interaction features are determined.

5. The speech enhancement method based on real-valued and complex-valued networks according to claim 2, characterized in that, The multidimensional collaborative attention module determines the corresponding complex spectral attention, amplitude spectral attention, and phase spectral attention based on complex interaction features, amplitude interaction features, and phase interaction features, respectively, including: Complex interaction features, amplitude interaction features, and phase interaction features are used as input features separately; The channel attention module determines the channel attention weights based on the input features, and then determines the channel attention based on the input features and the channel attention weights. The frequency attention is determined by applying frequency-dimensional attention weights to the input features using the frequency attention module. The temporal attention module performs temporal dimension attention weighting on the input features to determine the temporal attention; The output features corresponding to the input features are determined based on frequency attention, time attention, and channel attention. The output features are one of complex spectrum attention, amplitude spectrum attention, and phase spectrum attention.

6. The speech enhancement method based on real-valued and complex-valued networks according to claim 1, characterized in that, The target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features are decoded and enhanced using real-valued and complex-valued speech enhancement networks, respectively, to determine the corresponding enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum, including: The target complex spectrum output features are decoded to determine the complex ratio mask and the first enhanced complex spectrum; the second enhanced complex spectrum is determined based on the complex ratio mask and the initial complex spectrum. The enhanced complex spectrum is determined based on the first and second enhanced complex spectra; Decode the output features of the target amplitude spectrum to determine the amplitude ratio mask; determine the enhanced amplitude spectrum based on the amplitude ratio mask and the initial amplitude spectrum. The target phase spectrum is output feature decoded to determine the enhanced phase spectrum.

7. The speech enhancement method based on real-valued and complex-valued networks according to claim 1, characterized in that, The method further includes: A multi-objective joint loss function is constructed, and the parameters of the real-valued and complex-valued speech enhancement networks are optimized based on this multi-objective joint loss function. The multi-objective joint loss function is expressed as follows: ， in, Indicates the joint loss of multiple objectives. Indicates time-domain loss. Indicates magnitude loss. Indicates actual loss. Indicates loss of the imaginary part. Indicates phase loss, This indicates the perceived loss in speech quality assessment. These represent the weights of each loss.

8. A speech enhancement device based on real-valued and complex-valued networks, characterized in that, The device includes: The signal conversion module is used to convert noisy speech signals into an initial complex spectrum, and to extract the initial amplitude spectrum and the initial phase spectrum based on the initial complex spectrum; The feature encoding module is used to encode the initial complex spectrum, initial amplitude spectrum, and initial phase spectrum respectively through a speech enhancement network based on real and complex values, and to determine the corresponding initial complex spectrum features, initial amplitude spectrum features, and initial phase spectrum features. The feature extraction module is used to input the initial complex spectrum features, initial amplitude spectrum features, and initial phase spectrum features into each bottleneck block of the speech enhancement network based on real and complex values ​​for feature extraction, and correspondingly determine the target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features. Each bottleneck block is connected sequentially, and residual connections are used within the bottleneck block. The input of the first bottleneck block and the output of the last bottleneck block are connected by a skip connection. The feature decoding module is used to decode and enhance the target complex spectrum output features, target amplitude spectrum output features, and target phase spectrum output features respectively through a speech enhancement network based on real values ​​and complex values, and to determine the corresponding enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum. The signal reconstruction module is used to fuse and reconstruct the enhanced complex spectrum, enhanced amplitude spectrum, and enhanced phase spectrum to determine the enhanced speech signal.

9. A terminal, characterized in that, The terminal includes a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the speech enhancement method based on real-valued and complex-valued networks as described in any one of claims 1-7; the processors are used to execute the programs.

10. A computer-readable storage medium storing a plurality of instructions thereon, characterized in that, The instructions are loaded and executed by the processor to implement the steps of the speech enhancement method based on real-valued and complex-valued networks as described in any one of claims 1-7.

Images