A cross-domain speech enhancement efficient fine-tuning method and system based on a time-frequency adaptive decoupling module

Abstract

The application discloses a cross-domain speech enhancement efficient fine-tuning method and system based on a time-frequency adaptive decoupling module, and belongs to the technical field of speech enhancement. In view of the problems of domain deviation, high cost of full-parameter fine-tuning and dependence on clean speech labels of existing speech enhancement technologies when facing unknown acoustic environments, a TFA module is introduced into a backbone network, the TFA module comprises layer normalization units, cascaded deep convolution blocks and SC-CTFA units connected in sequence, and is used for explicitly modeling time-frequency difference features of different acoustic environments. The parameter efficient fine-tuning strategy is as follows: after pre-training in a source domain, the backbone network is frozen, and only a small number of parameters of the TFA module are updated, so that low-cost cross-domain adaptation is realized; in a label-free scene, speech activity detection and pseudo-label generation are combined to construct a self-supervised pseudo-training pair, so that unsupervised domain adaptation is realized. Under the premise of not changing the structure of the backbone network, fast, efficient and stable adaptation to unknown acoustic environments is realized.

Description

Technical Field

[0001] This invention belongs to the field of speech enhancement technology, specifically relating to a cross-domain speech enhancement efficient fine-tuning method and system based on a time-frequency adaptive decoupling module. Background Technology

[0002] Speech enhancement technology aims to recover clear and intelligible speech from noisy speech signals, and has wide applications in communication, audio processing, hearing aids, and intelligent voice interaction. For example, patent application CN110428849B discloses a speech enhancement method based on generative adversarial networks. This prior art combines discriminator feature matching with traditional feature mapping methods, effectively reducing the difference between the feature distribution of the enhanced speech and the feature distribution of the clean speech.

[0003] In recent years, deep learning-based speech enhancement methods have made new progress. These methods typically construct a nonlinear mapping model between noisy speech and clean speech, and use a large amount of paired noisy speech and corresponding clean speech data for supervised learning during the training phase. When the acoustic environment is similar to or highly consistent with the training set (e.g., similar distributions of noise type, reverberation conditions, speaker features, etc.), such models can effectively improve the perceptual quality and intelligibility of speech.

[0004] Currently, data-driven deep speech enhancement models heavily rely on the acoustic scenarios covered by the training data. When the model is deployed to an acoustic environment (i.e., the target domain) that was not seen during the training phase or is significantly different, the model performance will degrade significantly due to the inconsistency between the training and test data distributions, resulting in a domain shift problem. To address the domain shift problem, existing technologies mainly employ the following three approaches: Method 1: Fine-tune the model's parameters using target domain data; Method 2: Retrain an independent model for a single new environment; Method 3: Introduce a teacher-student structure for unsupervised or weakly supervised training.

[0005] However, all three of the existing methods mentioned above have drawbacks in practical applications: When using method 1, the entire model parameters need to be retrained using data collected from the target domain. Although this method can improve the model's performance in the target domain, it requires a complete supervised training process, which is computationally and time-intensive and can easily lead to the model forgetting the knowledge it has learned in the source domain, thus causing catastrophic forgetting.

[0006] When using method 2, a dedicated model needs to be trained for each new acoustic environment. This method not only requires repeated investment of a large amount of training resources, but also causes a sharp increase in the complexity of model storage and maintenance when the target environment is diverse or dynamically changing, lacking flexibility and scalability.

[0007] When using method 3, it is necessary to adapt using unpaired or weakly labeled data in the target domain through multiple rounds of iteration and knowledge distillation. This method reduces the dependence on clean speech in the target domain to some extent, but the training process is still cumbersome, the convergence speed is slow, and it is often difficult to achieve a good balance between computational efficiency and performance enhancement. Summary of the Invention

[0008] To address the problems in related technologies, this invention proposes a cross-domain speech enhancement efficient fine-tuning method and system based on a time-frequency adaptive decoupling module, in order to overcome the aforementioned technical problems existing in the prior art. By introducing a time-frequency adaptive decoupling module, an efficient parameter fine-tuning mechanism, and a self-supervised data construction strategy, this invention achieves rapid, efficient, and stable adaptation to unknown acoustic environments without changing the backbone network structure.

[0009] The technical solution of this invention is implemented as follows: a highly efficient fine-tuning method for cross-domain speech enhancement based on a time-frequency adaptive decoupling module, the method comprising the following steps: Step S1: Acquire the initial noisy time-domain speech signal and perform preprocessing to obtain the preprocessed noisy time-domain speech signal; Step S2: Perform time-frequency transformation on the preprocessed noisy time-domain speech signal to obtain the complex time-spectral features, which are referred to as the complex spectrum features of the noisy time-domain speech signal. Step S3: Input the complex spectral features of the noisy time-domain speech signal into the complex convolutional coding module, and perform high-dimensional mapping and encoding processing on the complex spectral features to extract high-level semantic features; Step S4: Introduce a time-frequency adaptive decoupling module, abbreviated as TFA (Time-Frequency Adapter) module, into the complex convolutional coding module and decoding network; the TFA module includes a layer normalization unit, a cascaded deep convolutional block, and a Split-Complex Channel-wise Time-Frequency Attention (SC-CTFA) unit connected in sequence; the TFA module is used to model and correct the domain-related time-frequency difference features of the input high-level semantic features to obtain the input complex spectral features with enhanced feature discriminability; Step S5: In the complex-separated channel-level time-frequency attention unit, the input complex spectral features are decoupled into real and imaginary parts and processed separately. Specifically, this includes: extracting the frequency domain global vector and time domain global vector of the features respectively, and generating frequency domain attention weights and time domain attention weights through the first attention sub-network and the second attention sub-network respectively. The first attention sub-network uses a narrow convolution kernel, and the second attention sub-network uses a wide convolution kernel. After the generated weights are weighted and fused with the corresponding features, the enhanced real and imaginary features are output and reassembled into complex features. Step S6: Based on the complex features processed by the TFA module, generate a complex ideal ratio mask, which is used to estimate the complex spectral features of the clean speech signal from the complex spectral features of the noisy time-domain speech signal in step S2. Step S7: Efficient parameter fine-tuning; pre-train the speech enhancement model containing the TFA module based on a large-scale dataset from the source domain; when facing the target domain, freeze the backbone network parameters of the speech enhancement model except for the TFA module, and update the parameters in the TFA module using only the data from the target domain to complete the adaptation of the speech enhancement model to the target domain.

[0010] Furthermore, in step S1, the preprocessing includes speech framing and windowing; wherein the frame length ranges from 20 to 32 milliseconds, the frame shift ranges from 10 to 16 milliseconds, and the windowing function is either a Hamming window or a Hanning window.

[0011] In step S2, the Short-Time Fourier Transform (STFT) is preferentially used to perform time-frequency transformation on the noisy time-domain speech signal. After the STFT operation, the one-dimensional time-domain speech signal can be mapped to a two-dimensional time-frequency domain representation, thereby simultaneously characterizing the changes in the time and frequency dimensions of the speech signal.

[0012] Furthermore, in step S3, after obtaining the complex-form time-spectrum features, this embodiment further performs high-dimensional mapping and encoding processing on the complex features to enhance the expressive power of the features; Furthermore, the complex convolutional coding module includes a complex two-dimensional convolutional layer, a complex batch normalization layer, and a parameterized modified linear unit.

[0013] Further, in step S4, the cascaded deep convolutional block includes a first 3×3 deep convolutional layer, a nonlinear activation function, a 5×5 deep convolutional layer, and a second 3×3 deep convolutional layer connected in sequence. Furthermore, in step S4, the input feature tensor is set to... Where B is the batch size, C is the number of channels, F is the number of frequency bands, and T is the number of frames; the input feature tensor is introduced into the layer normalization unit and processed by the normalization function in the CFT three-dimensional space, and the calculation formula is as follows: ; in, The feature tensor after normalization; The layer normalization unit along The mean and variance are calculated in three dimensions, that is, the features are normalized for individual samples in each batch B. This can effectively reduce the feature distribution shift caused by different speakers and different types of noise in each batch, increase the model's ability to identify features during training, and improve the model's ability to generalize to input speech. The cascaded depthwise convolutional blocks are then used for processing, and the calculation formula is as follows: ; Where ReLU(.) refers to the nonlinear activation function; This refers to a 3×3 depthwise convolutional layer; The result refers to the result after processing with a non-linear activation function; This refers to the result after processing through the 5×5 depth convolutional layer; This refers to the result after processing by the second 3×3 depth convolutional layer.

[0014] Furthermore, in step S5, the size of the narrow convolution kernel is 3, and the size of the wide convolution kernel is 7.

[0015] Furthermore, in step S7, the parameters of the TFA module are updated 1 to 3 times.

[0016] Furthermore, it also includes step S8: efficient fine-tuning of self-supervised parameters, applied to scenarios where clean speech signals in the target domain cannot be obtained, specifically including: S8-1: Perform speech activity detection (VAD) on the noisy speech signal in the target domain and extract segments that are determined to be non-speech as estimated noise; S8-2: The pre-trained speech enhancement model is used to enhance the noisy speech signal in the target domain to obtain pseudo-clean speech; S8-3: Mix the estimated noise with the pseudo-clean speech at a random signal-to-noise ratio to generate pseudo-noisy speech, thereby constructing a pseudo-training pair consisting of the pseudo-noisy speech and the pseudo-clean speech; S8-4: Based on the pseudo-training pair, perform efficient parameter fine-tuning on the TFA module, update the parameters of the TFA module, and obtain the fine-tuned new enhanced model.

[0017] Further, in step S8, the calculation process for generating the pseudo-training pair is as follows: Single-channel noisy speech signal First, use a pre-trained speech enhancement model Obtain pseudo-clean voice The calculation formula is: ; in, Parameters of the TFA module including fine-tuning and frozen backbone network parameters ; In step S8-1, the noisy speech samples are simultaneously processed by a lightweight speech endpoint detection model to detect speech activity, specifically: The system outputs a binary mask (0 for noise segments and 1 for speech segments) to obtain pseudo-noisy speech without human voice activity. The pseudo-noisy speech is then concatenated to obtain the background noise corresponding to a sample. The calculation formula is as follows: ; ; Where K refers to the number of noise segments; It is a concatenation function; Then, the pseudo-clean voice and pseudo-noisy speech Generated after random mixing When the length of the noise segment is insufficient, it is cyclically added to the pseudo-clean speech. Finally, a new set of training pairs was obtained. ; Then, proceed to step S8-4.

[0018] A cross-domain speech enhancement system based on a time-frequency adaptive decoupling module is provided to implement the above-mentioned efficient fine-tuning method for cross-domain speech enhancement based on a time-frequency adaptive decoupling module. The cross-domain speech enhancement system includes a speech signal acquisition module, a speech preprocessing unit, a time-frequency transformation unit, a complex feature encoding module, a time-frequency adaptive decoupling module, a complex masking estimation unit, a parameter efficient fine-tuning control unit, a self-supervised data construction unit, and a model update and execution unit. The speech signal acquisition module is used to acquire the initial noisy time-domain speech signal; The speech preprocessing unit is used to preprocess the initial noisy time-domain speech signal; The time-frequency conversion unit is used to convert the preprocessed noisy time-domain speech signal into a complex time-frequency spectrum; The complex feature encoding module is used to perform high-dimensional feature encoding on the complex time spectrum; The time-frequency adaptive decoupling module is integrated into the network structure and includes a layer normalization unit, a cascaded deep convolution block, and a complex-separated channel-level time-frequency attention unit connected in sequence. It is used to perform layer normalization, deep convolution feature extraction, and time-frequency dual-path attention weighting. Complex masking estimation unit, used to generate complex ideal ratio mask; The parameter high-efficiency fine-tuning control unit is used to control and freeze the backbone parameters of the complex feature encoding module and the complex masking estimation unit in cross-domain adaptation scenarios, and only update the parameters of the time-frequency adaptive decoupling module. Self-supervised data construction unit is used to generate pseudo-training pairs in unlabeled scenarios through speech endpoint detection and speech enhancement; The model update and execution unit is used to load the fine-tuned parameters of the time-frequency adaptive decoupling module and perform enhancement processing on the input target domain noisy speech.

[0019] The beneficial effects of this invention are: (1) By introducing a time-frequency adaptive decoupling module, this invention explicitly models the time-frequency characteristics of different acoustic environments, effectively alleviates the performance degradation caused by domain offset, and maintains a stable enhancement effect in unknown and complex scenarios.

[0020] (2) By adopting a parameter-efficient fine-tuning strategy, only the parameters of the time-frequency adaptive decoupling module, which accounts for a small proportion, are updated, which greatly reduces the consumption of computing resources, training time and storage overhead, and facilitates deployment and rapid adaptation on resource-constrained devices.

[0021] (3) The time-frequency adaptive decoupling module of the present invention focuses on the learning domain difference features, so that the model can be effectively adapted with only a small number of target domain samples, reducing the dependence on large-scale pairing data.

[0022] (4) This invention proposes a self-supervised parameter efficient fine-tuning method, which utilizes lightweight VAD and pseudo-label technology to achieve effective model updates without the need for clean speech in the target domain, thus broadening the application scenarios. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the framework for steps S3 to S6 of the present invention; Figure 2 This is a schematic diagram of the TFA module of the present invention; Figure 3 This is a schematic diagram of the framework of the complex separated channel-level time-frequency attention unit (SC-CTFA) of the present invention; Figure 4 This is a schematic diagram of the time-frequency separation attention (CTFA) framework of the present invention; Figure 5This is a schematic diagram of the framework of the efficient parameter fine-tuning method based on the TFA module of the present invention; Figure 6 This is a schematic diagram of the supervised training framework based on a large-scale pre-trained dataset of the present invention; Figure 7 This is a schematic diagram of the framework of the TFA-PEFT process of the present invention; Figure 8 This is a schematic diagram of the framework generated by the pseudo-training pair of the present invention; Figure 9 This is a flowchart illustrating the steps of the cross-domain speech enhancement high-efficiency fine-tuning method of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0026] Example 1 like Figure 9 As shown, this embodiment provides a highly efficient fine-tuning method for cross-domain speech enhancement based on a time-frequency adaptive decoupling module. The method includes the following steps: Step S1: Acquisition and preprocessing of noisy time-domain speech signals; Step S2: Time-frequency transformation and complex spectrum feature extraction; Step S3: High-dimensional mapping and encoding of complex spectral features; Step S4: Introduction of the time-frequency adaptive decoupling module; Step S5: Introduction of complex separation channel-level time-frequency attention unit; Step S6: Generate a complex ideal ratio mask; Step S7: Efficient parameter fine-tuning method; Step S8: Self-supervised parameter fine-tuning method.

[0027] Specifically, a single-channel noisy time-domain speech signal is acquired through a speech signal acquisition module. This speech signal can originate from a mobile terminal, wearable device, embedded voice interaction terminal, or server-side speech acquisition interface. The acquired noisy time-domain speech signal is usually a continuous time-domain digital signal, and its sampling rate can be set to common speech sampling rates such as 8kHz, 16kHz, or 48kHz depending on the application scenario. Since speech signals are often affected by environmental noise, equipment noise and speaker differences in real-world applications, directly modeling the original speech signal can easily introduce unstable factors. Therefore, preprocessing of the speech signal is required after acquisition. The preprocessing includes at least one or more of the following processes: 1) Amplitude normalization processing is used to eliminate amplitude differences caused by different recording devices or different speakers, so that the dynamic range of the input speech signal is kept within a preset range. 2) DC component elimination processing is used to remove the DC bias component in the speech signal to avoid interference with subsequent spectrum analysis; 3) Pre-emphasis processing: First-order high-pass filtering is used to enhance the high-frequency components of the speech signal, thereby improving speech clarity; 4) Speech framing processing: The continuous time-domain speech signal is divided into multiple time frames, each containing a fixed length of sampling points. 5) Windowing: Apply a window function to each frame of the speech signal to reduce spectral leakage.

[0028] The frame length and frame shift can be set according to actual needs. For example, the frame length ranges from 20 to 32 ms, and the frame shift ranges from 10 to 16 ms. In this embodiment, a Hamming window or a Hanning window is used as the windowing function to achieve a balance between temporal continuity and frequency domain resolution. Through the above steps of speech signal acquisition and preprocessing, the initial noisy speech signal can be converted into a stable input form suitable for time-frequency analysis, providing a reliable foundation for the time-frequency transformation in step S2 and the deep feature modeling in step S4.

[0029] In step S2, the Short-Time Fourier Transform (STFT) is preferentially used to perform time-frequency transformation on the noisy time-domain speech signal. After the STFT operation, the one-dimensional time-domain speech signal can be mapped to a two-dimensional time-frequency domain representation, thereby simultaneously characterizing the variation characteristics of the speech signal in the time and frequency dimensions. The calculation process of step S2 is as follows: Let the preprocessed noisy speech signal be represented as x(t), where t represents the time index. Then its short-time Fourier transform formula can be expressed as: ; in, Represents the window function. Indicates the frame index. Indicates frequency index, The imaginary unit; After applying the above transformation formula, the complex spectrum characteristics containing both real and imaginary parts can be obtained. It should be noted that, unlike using only the amplitude spectrum, the complex spectrum retains both the amplitude and phase information of the speech signal, and can more completely describe the physical characteristics of the speech signal. In this embodiment, the complex spectrum feature is represented as a four-dimensional tensor, whose dimensions are ordered as follows: batch dimension, channel dimension, frequency dimension, and time dimension; wherein, the channel dimension is used to distinguish the real part and the imaginary part of the complex spectrum, or to carry the multi-channel complex feature after mapping. By representing the speech signal as complex time-frequency features in step S2, a richer information expression capability can be provided for the subsequent speech enhancement model, enabling the model to not only suppress noise amplitude but also implicitly model the phase structure, thereby improving the overall effect of speech enhancement.

[0030] like Figure 1 As shown, in step S3, after obtaining the complex-form time-spectrum features, this embodiment further performs high-dimensional mapping and encoding processing on the complex features to enhance the expressive power of the features; Specifically, step S3 uses a complex convolutional coding module to process the input complex spectrum; through this complex convolutional coding module, the original low-dimensional complex spectrum features can be mapped to a higher-dimensional feature space, thereby extracting more abstract and discriminative acoustic features. More specifically, in step S3, the complex convolutional coding module includes a complex two-dimensional convolutional layer, a complex batch normalization layer, and a parameterized modified linear unit.

[0031] In step S3, the input complex spectral features are set as follows: ;in, , Let represent the real part and the imaginary part, respectively. Then, the complex convolution operation is expressed by the following formula: ; in, , These represent the real and imaginary parts of the complex convolution kernel, respectively, and the symbol * indicates the convolution operation; After the convolution operation, the complex batch normalization layer performs complex batch normalization on the complex features to alleviate the problem of inconsistent distribution between different feature channels during training; then, the parameterized rectified linear unit (PReLU) is introduced as the activation function to introduce non-linear expressive power into the network. More specifically, the complex convolutional coding modules are repeatedly stacked in multiple layers to form an encoder structure, which is used to extract high-level semantic features of the speech signal layer by layer. Through the above-mentioned high-dimensional mapping and encoding of complex features, the network can construct feature representations that are more sensitive to the differences between speech and noise, laying the foundation for the subsequent introduction of a time-frequency adaptive decoupling module.

[0032] like Figure 2 As shown, in step S4, the cascaded deep convolutional block includes a first 3×3 deep convolutional layer, a nonlinear activation function, a 5×5 deep convolutional layer, and a second 3×3 deep convolutional layer connected in sequence. More specifically, in step S4, the input feature tensor is set to... Where B is the batch size, C is the number of channels, F is the number of frequency bands, and T is the number of frames; the input feature tensor is introduced into the layer normalization unit and processed by the normalization function in the CFT three-dimensional space, and the calculation formula is: ; in, The feature tensor after normalization; The layer normalization unit along The mean and variance are calculated in three dimensions, that is, the features are normalized for individual samples in each batch B. This can effectively reduce the feature distribution shift caused by different speakers and different types of noise in each batch, increase the model's ability to identify features during training, and improve the model's ability to generalize to input speech. The cascaded depthwise convolutional blocks are then used for processing, and the calculation formula is as follows: ; Where ReLU(.) refers to the nonlinear activation function; This refers to a 3×3 depthwise convolutional layer; The result refers to the result after processing with a non-linear activation function; This refers to the result after processing through the 5×5 depth convolutional layer; This refers to the result after processing by the second 3×3 depth convolutional layer; The cascaded deep convolutional block first uses a 3×3 convolutional layer with a compact kernel, which can initially extract local time-frequency features. Then, a nonlinear transformation is introduced through a nonlinear activation function (ReLU). The intermediate stage uses a 5×5 convolutional layer, which expands the receptive field by increasing the kernel size, effectively capturing more frequency bands and longer temporal features. The final 3×3 convolutional layer further adjusts the features, further enhancing the capture of local time-frequency features. All convolutional layers use deep convolution (groups=C), which, while maintaining time-frequency resolution, allows each channel to independently learn the acoustic features of specific frequency bands. This design can effectively capture the differential features under different acoustic environments, enhance the recognition, and effectively prevent the overfitting problem of subsequent parameter-efficient fine-tuning (PEFT).

[0033] Specifically, in step S5, the cascaded deep convolutional blocks have already performed deep extraction of complex features, effectively establishing local correlations of complex features in the time-frequency domain; to improve the sensitivity of the speech enhancement model to subtle spectral changes, thereby capturing the core differences in different acoustic environments, SC-CTFA is introduced, such as... Figure 3 As shown; It should be noted that SC-CTFA is an attention mechanism designed for complex features. Its core idea is to first decouple the real and imaginary parts of the complex spectrum, avoiding coupling interference between them. This allows the Channel-wise Time-Frequency Attention (CTFA) module to focus more on time-frequency domain modeling of the real or imaginary spectrum. The real and imaginary spectral features are then concatenated into a complex spectrum after processing by CTFA. Compared to transforming it into a form similar to complex convolution, this design has a much lower computational cost. CTFA achieves precise feature enhancement by combining frequency and temporal attention; its core structure is as follows: Figure 4 As shown; given that the preceding cascaded deep convolutional blocks have already extracted the spatial features in the time and frequency domains, CTFA first uses average pooling for feature compression to extract the frequency domain features. and temporal features This approach can capture the frequency domain energy distribution and the temporal features of speech separately, ensuring they do not interfere with each other. It allows for attention modeling of the features themselves, paying more attention to subtle changes in the features and effectively capturing the core differences in features across different acoustic scenarios. Next, the generation of dual-path attention in the time and frequency domains mainly consists of three stages: channel dimensionality reduction, local feature extraction, and channel dimensionality restoration, which can be represented by the following formula: ; Where A represents attention weights, W represents convolution, and * indicates performing a convolution operation; t represents the frequency domain; t represents the time domain; Depend on Figure 4 It can be seen that the generation process of time-domain and frequency-domain dual attention is quite similar. The main difference between the two lies in the size of the convolution kernel used for feature extraction after channel dimensionality reduction. In the frequency attention generation process, a narrow convolutional kernel (kernel_size=3) was selected to capture the correlation of harmonic structures between adjacent frequency bands. In the temporal attention generation process, since the differences between adjacent speech frames are small, a wider convolutional kernel (kernel_size=7) was selected to capture longer temporal dependencies through a wide temporal window. This can effectively distinguish speech steady-state features from transient noise components, and can also capture the difference features between different speakers, providing more discriminative temporal features for subsequent PEFT. Finally, the sigmoid activation function is used to generate attention weights, and the two generated attention weights are weighted into the input features to obtain the output features of the CTFA module.

[0034] In step S6, this embodiment uses a Complex Ideal Ratio Mask (cIRM) to mask the complex spectrum, which theoretically can achieve perfect reconstruction of the spectrum. The overall principle can be expressed as follows: ; in, , Let represent the complex spectra of the estimated clean speech signal and the noisy time-domain speech signal, respectively. Indicates the cIRM mask; The calculation process for Y, M, and X is shown below: ; From the two formulas above, we can obtain: ; Therefore, the mask cIRM can be represented by the following formula: ; It is worth noting that cIRM is learned by the network, rather than calculated from predicting clean and noisy speech; cIRM can reconstruct the real and imaginary parts of the clean speech signal, thereby implicitly enhancing the phase information and improving the perceptual quality of speech; while Ideal Ratio Mask (IRM) only masks the amplitude information and uses unprocessed phase information for speech reconstruction, which leads to a decrease in perceptual quality.

[0035] In step S7, the PEFT proposed in this embodiment refers to achieving domain adaptation of the model by fine-tuning only a small number of parameters while freezing the backbone network model parameters. It is generally suitable for fine-tuning large models. This embodiment introduces this design into the field of speech enhancement, such as... Figure 5 As shown; Figure 6 To employ a supervised training framework based on a large-scale pre-trained dataset; specifically, a pre-training set is first constructed to train paired samples. ,in This represents a noisy time-domain speech signal. This represents the corresponding clean time-domain speech signal, i.e., the reference signal; the noisy signal. After performing a short-time Fourier transform, the time-frequency domain information is fed into the network model. This network model denoises the noisy speech to obtain the predicted clean speech. The calculation formula is: ; in, For the network model (including the TFA module), For network parameters; The Inverse Short-Time Fourier Transform (ISTFT) function refers to the function of the Inverse Short-Time Fourier Transform. The SISNR loss function was then used to measure the loss. and The difference; Finally, the gradient of the entire network parameters is updated using the backpropagation algorithm, calculated as follows: ; ; in, The learning rate; SI-SNR stands for Scale-Invariant Signal-to-Noise Ratio, which is a function used to calculate the scale-invariant signal-to-noise ratio. The pre-training loss function; The above calculation process performed supervised pre-training on the speech enhancement model. Since the training set contains a large amount of complex noise and multiple speakers, a more general speech enhancement model can be constructed to prepare for subsequent PEFT. Figure 7 It's the TFA-PEFT process; first, the parameters of the pre-trained model are... The samples are then ported into the network and then paired using the target domain training set. Fine-tune the TFA; specifically, and These represent noisy and clean speech in the target domain training set, respectively. Enhanced speech is predicted after pre-training the model. Then, the sisnr loss function was used to measure the loss. and The difference; parameters of the frozen network backbone model (1.87M, 94%), only update the parameters of the TFA module. (0.12M, 6%); The TFA-PEFT update process is expressed by the following formula: ; The target domain loss function; In this embodiment, the parameters of the TFA module are... After 1-3 rounds of updates, the entire model can achieve better speech enhancement results in the target domain.

[0036] Specifically, the self-supervised parameter fine-tuning in step S8 is applied to scenarios where clean speech signals in the target domain cannot be obtained; step S8 specifically includes: S8-1: Perform speech activity detection (VAD) on the noisy speech signal in the target domain and extract segments that are determined to be non-speech as estimated noise; S8-2: The pre-trained speech enhancement model is used to enhance the noisy speech signal in the target domain to obtain pseudo-clean speech; S8-3: Mix the estimated noise with the pseudo-clean speech at a random signal-to-noise ratio to generate pseudo-noisy speech, thereby constructing a pseudo-training pair consisting of the pseudo-noisy speech and the pseudo-clean speech; S8-4: Based on the pseudo-training pair, perform efficient parameter fine-tuning on the TFA module, update the parameters of the TFA module, and obtain the fine-tuned new enhanced model.

[0037] It should be noted that in actual speech enhancement model application scenarios, obtaining clean speech signals is quite difficult, and the traditional supervised learning model that relies on clean-noisy speech pairing training faces challenges. To this end, step S8 proposes stringent constraints to construct the unlabeled domain adaptation scenario, namely, only single-channel noisy speech samples can be obtained within a short period of time, and stable background noise cannot be obtained through long-term recording. In this scenario, although the self-supervised learning method based on teacher-student model collaborative training can achieve domain adaptation, it has certain limitations. First, a high-performance speech separation model is needed to accurately estimate speech and corresponding noise. Second, the model requires full parameter fine-tuning and a large number of iterations; the classic RemixIT method requires 60 iterations. To perform domain adaptation on devices with lower resources, step S8 develops a self-supervised parameter-efficient fine-tuning method (SS-PEFT) based on the TFA module. Noise extraction is performed using Silo VAD, pseudo-labels are generated through a pre-trained model, and the two are mixed to generate pseudo-training pairs. Then, the TFA-PEFT method is used for speech enhancement domain transfer.

[0038] like Figure 8 As shown, in step S8, the calculation process for generating the pseudo-training pair is as follows: Single-channel noisy speech signal First, use a pre-trained speech enhancement model Obtain pseudo-clean voice The calculation formula is: ; in, Parameters of the TFA module including fine-tuning and frozen backbone network parameters ; In step S8-1, the noisy speech samples are simultaneously processed by a lightweight speech endpoint detection model to detect speech activity, specifically: The system outputs a binary mask (0 for noise segments and 1 for speech segments) to obtain pseudo-noisy speech without human voice activity. The pseudo-noisy speech is then concatenated to obtain the background noise corresponding to a sample. The calculation formula is as follows: ; ; Where K refers to the number of noise segments; It is a concatenation function; Then, the pseudo-clean voice and pseudo-noisy speech Generated after random mixing When the length of the noise segment is insufficient, it is cyclically added to the pseudo-clean speech. Finally, a new set of training pairs was obtained. ; Then, proceed to step S8-4. After completing the above steps, this embodiment forms a complete cross-domain speech enhancement operation process, applicable to different languages, different noise environments, and different computing resource conditions.

[0039] To address the domain offset issue, this embodiment utilizes a large-scale dataset to jointly train a baseline model and an embedded TFA module, constructing a general speech enhancement model. When facing the target domain, TFA-PEFT is performed, freezing 94% of the backbone network parameters and updating only 6% of the TFA parameters. This minor fine-tuning allows the entire network to achieve better speech enhancement results in the target domain. For scenarios where clean speech labels cannot be collected, a self-supervised, efficient parameter fine-tuning method is employed. Specifically, lightweight speech endpoint detection (VAD) is used to extract noise segments, which are then mixed with enhanced speech generated by the pre-trained model to construct pseudo-training pairs. TFA-PEFT is then performed to improve the model's noise suppression capability in unknown noise environments.

[0040] Example 2 This embodiment also provides a cross-domain speech enhancement system based on a time-frequency adaptive decoupling module, used to implement the efficient fine-tuning method for cross-domain speech enhancement based on a time-frequency adaptive decoupling module in Embodiment 1; features not explained in this embodiment can be explained using the methods in Embodiment 1, and will not be repeated here. The difference between this embodiment and Embodiment 1 is: The cross-domain speech enhancement system includes a speech signal acquisition module, a speech preprocessing unit, a time-frequency transformation unit, a complex feature encoding module, a time-frequency adaptive decoupling module, a complex masking estimation unit, a parameter efficient fine-tuning control unit, a self-supervised data construction unit, and a model update and execution unit. The speech signal acquisition module is used to acquire the initial noisy time-domain speech signal; The speech preprocessing unit is used to preprocess the initial noisy time-domain speech signal; The time-frequency conversion unit is used to convert the preprocessed noisy time-domain speech signal into a complex time-frequency spectrum; The complex feature encoding module is used to perform high-dimensional feature encoding on the complex time spectrum; The time-frequency adaptive decoupling module is integrated into the network structure and includes a layer normalization unit, a cascaded deep convolution block, and a complex-separated channel-level time-frequency attention unit connected in sequence. It is used to perform layer normalization, deep convolution feature extraction, and time-frequency dual-path attention weighting. Complex masking estimation unit, used to generate complex ideal ratio mask; The parameter high-efficiency fine-tuning control unit is used to control and freeze the backbone parameters of the complex feature encoding module and the complex masking estimation unit in cross-domain adaptation scenarios, and only update the parameters of the time-frequency adaptive decoupling module. Self-supervised data construction unit is used to generate pseudo-training pairs in unlabeled scenarios through speech endpoint detection and speech enhancement; The model update and execution unit is used to load the fine-tuned parameters of the time-frequency adaptive decoupling module and perform enhancement processing on the input target domain noisy speech.

[0041] Based on the above-mentioned efficient fine-tuning method for cross-domain speech enhancement using a time-frequency adaptive decoupling module, the overall operation flow of the cross-domain speech enhancement system is as follows: 1) Perform time-frequency transformation and complex modeling on the input speech based on steps S1~S3; 2) Extract general speech structure features through the backbone network; 3) Model the domain correlation differences using the time-frequency adaptive decoupling module described in steps S4-S6; 4) When a domain change is detected, the parameter fine-tuning mechanism in step S7 or step S8 is automatically triggered; 5) Write back the parameters of the fine-tuned speech enhancement model to the cross-domain speech enhancement system to achieve continuous adaptation.

[0042] Through the above design, this embodiment significantly reduces cross-domain migration costs while ensuring enhanced performance, and is applicable to various deployment forms such as smart terminals, edge devices and cloud services; Moreover, in response to the common problems of insufficient domain adaptability, high parameter update cost, and strong dependence on target domain labeled data in existing cross-domain speech enhancement technologies, the cross-domain speech enhancement system described in this embodiment introduces a time-frequency adaptive decoupling module, a parameter efficient fine-tuning control unit, and a self-supervised data construction unit, which achieves rapid, efficient, and stable adaptation to unknown acoustic environments without changing the backbone network structure.

[0043] Based on the disclosure and teachings of the foregoing specification, those skilled in the art can make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the present invention should also fall within the protection scope of the claims of the present invention. Furthermore, although some specific terms are used in this specification, these terms are only for convenience of explanation and do not constitute any limitation on the present invention.

Claims

1. A highly efficient fine-tuning method for cross-domain speech enhancement based on a time-frequency adaptive decoupling module, characterized in that, The method includes the following steps: Step S1: Acquire the initial noisy time-domain speech signal and perform preprocessing to obtain the preprocessed noisy time-domain speech signal; Step S2: Perform time-frequency transformation on the preprocessed noisy time-domain speech signal to obtain the complex time-spectral features, which are referred to as the complex spectrum features of the noisy time-domain speech signal. Step S3: Input the complex spectral features of the noisy time-domain speech signal into the complex convolutional encoder, and perform high-dimensional mapping and encoding processing on the complex spectral features to extract high-level semantic features; Step S4: Introduce a time-frequency adaptive decoupling module, abbreviated as TFA module, into the complex convolutional encoder and decoder network; the TFA module includes a layer normalization unit, a cascaded deep convolutional block and a complex separation channel-level time-frequency attention unit connected in sequence; the TFA module is used to perform domain-related time-frequency difference feature modeling and correction on the input high-level semantic features to obtain the input complex spectrum features with enhanced feature recognition. Step S5: In the complex-separated channel-level time-frequency attention unit, the input complex spectral features are decoupled into real and imaginary parts and processed separately. Specifically, this includes: extracting the frequency domain global vector and time domain global vector of the features respectively, and generating frequency domain attention weights and time domain attention weights through the first attention sub-network and the second attention sub-network respectively. The first attention sub-network uses a narrow convolution kernel, and the second attention sub-network uses a wide convolution kernel. After the generated weights are weighted and fused with the corresponding features, the enhanced real and imaginary features are output and reassembled into complex features. Step S6: Based on the complex features processed by the TFA module, generate a complex ideal ratio mask, which is used to estimate the complex spectral features of the clean speech signal from the complex spectral features of the noisy time-domain speech signal in step S2. Step S7: Pre-train a speech enhancement model containing the TFA module based on a large-scale dataset from the source domain; when facing the target domain, freeze the backbone network parameters of the speech enhancement model except for the TFA module, and update the parameters in the TFA module using only the data from the target domain to complete the adaptation of the speech enhancement model to the target domain.

2. The efficient fine-tuning method for cross-domain speech enhancement based on a time-frequency adaptive decoupling module according to claim 1, characterized in that, In step S4, the cascaded deep convolutional block includes a first 3×3 deep convolutional layer, a nonlinear activation function, a 5×5 deep convolutional layer, and a second 3×3 deep convolutional layer connected in sequence.

3. The efficient fine-tuning method for cross-domain speech enhancement based on a time-frequency adaptive decoupling module according to claim 2, characterized in that, In step S4, the input feature tensor is set to... Where B is the batch size, C is the number of channels, F is the number of frequency bands, and T is the number of frames; the input feature tensor is introduced into the layer normalization unit and processed by the normalization function in the CFT three-dimensional space, and the calculation formula is: ； in, The feature tensor after normalization; The cascaded depthwise convolutional blocks are then used for processing, and the calculation formula is as follows: ； in, Refers to a non-linear activation function; This refers to a 3×3 depthwise convolutional layer; The result refers to the result after processing with a non-linear activation function; This refers to the result after processing through the 5×5 depth convolutional layer; This refers to the result after processing by the second 3×3 depth convolutional layer.

4. The efficient fine-tuning method for cross-domain speech enhancement based on a time-frequency adaptive decoupling module according to claim 1, characterized in that, In step S5, the size of the narrow convolution kernel is 3, and the size of the wide convolution kernel is 7.

5. The efficient fine-tuning method for cross-domain speech enhancement based on a time-frequency adaptive decoupling module according to claim 1, characterized in that, It also includes step S8: efficient fine-tuning of self-supervised parameters, applied to scenarios where clean speech signals in the target domain cannot be obtained, specifically including: S8-1: Perform speech activity detection on the noisy speech signal in the target domain and extract segments that are determined to be non-speech as estimated noise; S8-2: The pre-trained speech enhancement model is used to enhance the noisy speech signal in the target domain to obtain pseudo-clean speech; S8-3: Mix the estimated noise with the pseudo-clean speech at a random signal-to-noise ratio to generate pseudo-noisy speech, thereby constructing a pseudo-training pair consisting of the pseudo-noisy speech and the pseudo-clean speech; S8-4: Based on the pseudo-training pair, perform efficient parameter fine-tuning on the TFA module, update the parameters of the TFA module, and obtain the fine-tuned new enhanced model.

6. The efficient fine-tuning method for cross-domain speech enhancement based on a time-frequency adaptive decoupling module according to claim 5, characterized in that, In step S8, the calculation process for generating the pseudo-training pair is as follows: Single-channel noisy speech signal First, use a pre-trained speech enhancement model Obtain pseudo-clean voice The calculation formula is: ； in, Parameters of the TFA module including fine-tuning and frozen backbone network parameters ; In step S8-1, the noisy speech samples are simultaneously processed by a lightweight speech endpoint detection model to detect speech activity, specifically: The system outputs a binary mask to obtain pseudo-noisy speech without human voice activity. Then, the pseudo-noisy speech is concatenated to obtain the background noise corresponding to a sample. The calculation formula is as follows: ； ； Where K refers to the number of noise segments; It is a concatenation function; Then, the pseudo-clean voice and pseudo-noisy speech Generated after random mixing When the length of the noise segment is insufficient, it is cyclically added to the pseudo-clean speech. Finally, a new set of training pairs was obtained. ; Then, proceed to step S8-4.

7. The efficient fine-tuning method for cross-domain speech enhancement based on a time-frequency adaptive decoupling module according to claim 1, characterized in that, In step S1, the preprocessing includes speech framing and windowing; wherein the frame length ranges from 20 to 32 milliseconds, the frame shift ranges from 10 to 16 milliseconds, and the windowing function is either a Hamming window or a Hanning window.

8. The efficient fine-tuning method for cross-domain speech enhancement based on a time-frequency adaptive decoupling module according to claim 1, characterized in that, In step S3, the complex convolutional encoder includes a complex two-dimensional convolutional layer, a complex batch normalization layer, and a parameterized modified linear unit.

9. The efficient fine-tuning method for cross-domain speech enhancement based on a time-frequency adaptive decoupling module according to claim 1, characterized in that, In step S7, the parameters of the TFA module are updated 1 to 3 times.

10. A cross-domain speech enhancement system based on a time-frequency adaptive decoupling module, characterized in that, This method is used to implement a cross-domain speech enhancement high-efficiency fine-tuning method based on a time-frequency adaptive decoupling module as described in any one of claims 1 to 9. The cross-domain speech enhancement system includes a speech signal acquisition module, a speech preprocessing unit, a time-frequency transformation unit, a complex feature encoding unit, a time-frequency adaptive decoupling module, a complex masking estimation unit, a parameter efficient fine-tuning control unit, a self-supervised data construction unit, and a model update and execution unit. The speech signal acquisition module is used to acquire the initial noisy time-domain speech signal; The speech preprocessing unit is used to preprocess the initial noisy time-domain speech signal; The time-frequency conversion unit is used to convert the preprocessed noisy time-domain speech signal into a complex time-frequency spectrum; A complex feature encoding unit is used to perform high-dimensional feature encoding on the complex temporal spectrum; the complex feature encoding unit includes a complex convolutional encoder; The time-frequency adaptive decoupling module is integrated into the network structure and includes a layer normalization unit, a cascaded deep convolution block, and a complex-separated channel-level time-frequency attention unit connected in sequence. It is used to perform layer normalization, deep convolution feature extraction, and time-frequency dual-path attention weighting. Complex masking estimation unit, used to generate complex ideal ratio mask; The parameter-efficient fine-tuning control unit is used to control and freeze the backbone parameters of the complex feature encoding unit and the complex masking estimation unit in cross-domain adaptation scenarios, and only update the parameters of the time-frequency adaptive decoupling module. Self-supervised data construction unit is used to generate pseudo-training pairs in unlabeled scenarios through speech endpoint detection and speech enhancement; The model update and execution unit is used to load the fine-tuned parameters of the time-frequency adaptive decoupling module and perform enhancement processing on the input target domain noisy speech.

Images