A lightweight single-channel noise reduction method for multi-axis Transformers

Abstract

This invention relates to noise reduction methods, and more particularly to a lightweight multi-axis Transformer single-channel noise reduction method. It can fully extract the latent time-frequency features of speech signals. A multi-head dynamic local self-attention module is employed to efficiently extract local features. The proposed method has fewer network parameters and lower computational cost, while remaining competitive with state-of-the-art methods in terms of speech signal quality and intelligibility. It can fully extract the latent time-frequency features of speech signals, effectively reducing computer resource consumption. Hint blocks are used to help the model better learn frequency feature information.

Description

Technical Field

[0001] This invention relates to noise reduction methods, and more particularly to a single-channel noise reduction method for a lightweight multi-axis Transformer. Background Technology

[0002] Speech enhancement effectively suppresses environmental noise and improves the intelligibility of speech signals, making it a key task in speech signal front-end processing. During interactive processes such as voice calls and video conferencing, speech signals are frequently interfered with by noise, severely impacting user experience. To address this, many noise reduction solutions have been proposed. Current single-channel speech enhancement methods suffer from insufficient analysis of the time-frequency features of audio, leading to significant performance degradation in complex acoustic scenarios. Furthermore, models based on the classic Transformer structure suffer from high computational costs, especially as audio duration increases, resulting in an exponential increase in computational demands and significantly increasing the model's reliance on computer resources.

[0003] Furthermore, in existing Transformer designs, all similarities between queries and keys are used for feature aggregation. However, since not all queries are related to keys, using all similarities does not effectively promote efficient speech signal enhancement. Summary of the Invention

[0004] To address the shortcomings and deficiencies in existing technologies, the technical problem this invention aims to solve is: to provide an effective, lightweight, dynamic multi-scale self-attention network for single-channel speech enhancement. Specifically, this paper proposes a simple yet effective multi-scale time-frequency feature analysis method that can fully extract the latent time-frequency features of speech signals.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is to provide a speech lightweight multi-axis transformer (Transformer) that includes two training objectives: an ideal scaling mask and complex spectrum compensation. The amplitude spectrum feature |Y(t,f)| and the complex spectrum feature Y(t,f) are input into a dense encoder block, where t and f represent the frame index and frequency point, respectively. The dense encoder block is used to capture low-level features, which are then processed by a cue block, a lightweight multi-axis transformer, and a residual channel attention block (RCAB) to extract high-level features.

[0006] As a further improvement of the present invention, the embedded fusion block further extracts and compresses these high-level features, and the system generates a masking value M(t,f) and a compensation value C(t,f) through a dense decoding block similar to a dense coding block. The enhanced complex spectrum can be expressed as:

[0007]

[0008] As a further improvement of the present invention, a prompt block is used to facilitate improved frequency feature extraction. The extraction calculation is as follows: the input feature X is transformed into X using global average pooling. GAP Simultaneously, a cue tensor P is introduced, which is a randomly initialized learnable tensor, obtained by adjusting X. GAP By applying a linear layer, we obtain X. Linear After the Softmax function completes, X is... Linear Multiplying it by the hint tensor P, we get P. Mul P Mul Summing in the first dimension yields P Sum By applying sigmoid activation, we obtain the frequency cue tensor, denoted as P. Sigmoid Then, element-wise multiplication is performed using X, followed by two-dimensional convolution and residual concatenation to produce the final output of the frequency cue module.

[0009] As a further improvement of the present invention, a low-complexity time-frequency multi-separation convolutional head self-attention method is used to analyze the time and frequency of the speech signal, including both time-frequency multi-separation convolutional head self-attention and frequency-frequency multi-separation convolutional head self-attention.

[0010] The temporal separable convolutional head self-attention calculation method is as follows: the input features are expanded in the channel dimension through point-to-point convolution, and three basic components are generated using depthwise separable convolution and splitting operations to obtain the attention map;

[0011] The attention map is extended by introducing a set of learnable parameters. The attention map is multiplied with the values, and then the output features are obtained through pointwise convolution.

[0012] As a further improvement of the present invention, the loss function uses MSE throughout the training process, and its specific calculation process is shown in Equation 2:

[0013] MSE=α·||S RI -S RI || 2 +β·||S Mag -S Mag || 2 (2)

[0014] As a further improvement of the present invention, the performance of the model is evaluated using metrics such as PESQ, STOI, SSNR, CSIG, CBAK, and COVL.

[0015] The beneficial effects of this invention are: This invention utilizes the most useful global features to facilitate the reconstruction of speech signals. The method proposed in this paper has fewer network parameters and lower computational cost, while being competitive with state-of-the-art methods in terms of speech signal quality and intelligibility. The main advantages of this algorithm are as follows: (1) This method can fully extract the potential time-frequency features of speech signals, making the model more effective in complex acoustic environments; (2) This method proposes a lightweight Transformer, which can effectively reduce the consumption of computer resources; (3) This method proposes a cue block to enable the model to better learn frequency feature information. Attached Figure Description

[0016] Figure 1 This is an overall structural diagram of the LMA-Transformer of the present invention;

[0017] Figure 2 This is a structural diagram of the main modules of the prompt block of the present invention;

[0018] Figure 3 This is a diagram of the self-attention structure of the time / frequency multi-separated convolution head of the present invention. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0020] like Figure 1 As shown, noise and speech are represented by y(t) and c(t) respectively. This invention proposes the Speechenhancement lightweight multi-axial Transformer (SE-LMA-Transformer), which achieves excellent noise reduction performance. The overall framework of the SE-LMA-Transformer is as follows: Figure 1 As shown.

[0021] The SE-LMA-Transformer consists of two training objectives: an ideal ratio mask (IRM) and complex spectrum compensation. The amplitude spectrum feature |Y(t,f)| and the complex spectrum feature Y(t,f) are fed into a dense encoder block, where t and f represent the frame index and frequency point, respectively. The dense encoder block captures low-level features. These features are then processed by a cue block, a lightweight multi-axis transformer, and a residual channel attention block (RCAB) to extract high-level features. Here, "multi-axis" refers to the channel-time axis and the channel-frequency axis. An embedding fusion block further extracts and compresses these high-level features. Finally, the system generates the mask value M(t,f) and the compensation value C(t,f) through a dense decoder block, similar to the dense encoder block. The enhanced complex spectrum can be represented as:

[0022]

[0023] The proposed prompt block aims to facilitate improved frequency feature extraction, such as... Figure 2 As shown, the input feature X is transformed into Xi through global average pooling. GAP Simultaneously, a cue tensor P is introduced, which is a randomly initialized learnable tensor. Through X... GAP By applying a linear layer, we obtain X. Linear After the Softmax function completes, X is... Linear Multiplying it by the hint tensor P, we get P. Mul P Mul Summing in the first dimension yields P Sum By applying sigmoid activation, we obtain the frequency cue tensor, denoted as P. Sigmoid Then, element-wise multiplication is performed using X, followed by 2D convolution and residual concatenation to produce the final output of the frequency cue module.

[0024] One of the main challenges faced by the classic Transformer architecture in processing time-frequency features is the exponential increase in computational complexity with the number of frames. When calculating the attention matrix using Multi-head Self-Attention (MHSA), the algorithm's complexity reaches [insert value here]. This leads to a significant increase in computational demands and memory usage, especially when processing longer speech segments. This invention proposes a low-complexity time-frequency multi-DConv head self-attention method, more suitable for time-frequency analysis of speech signals. Temporal multi-DConv head self-attention (T-MDHSA) and frequency multi-DConv head self-attention (F-MDHSA) not only solve the problem of insufficient utilization of local features by transformers but also reduce computational costs. Their structures are as follows: Figure 3 As shown.

[0025] Taking T-MDHSA as an example, the input features are initially expanded along the channel dimension using pointwise convolutions. Next, we use depthwise separable convolutions and splitting operations to generate three basic components: query, key, and value. It's important to note that our focus is on cross-channel application, not on the time or frequency dimension. We can then obtain an attention map, at which point the algorithm's complexity is O(n log n). Because C is smaller than T and also smaller than F, this T-MDHSA is more lightweight. To expand the attention map, we introduce a set of learnable parameters. These parameters are key to scaling the attention weights, a process that prepares the subsequent Softmax function. The attention map is multiplied by the values, and then the output features are obtained through pointwise convolution.

[0026] Throughout the training process, to better utilize amplitude and phase information, our loss function uses MSE. The specific calculation process is shown below:

[0027] MSE=α·||S RI -S RI || 2 +β·||S Mag -S Mag || 2 (2)

[0028] The Voice Bank+DEMAND dataset was used for both training and testing, comprising 11,572 noise-free speech clips for training and 824 noisy speech clips for testing. The average signal-to-noise ratio (SNR) of the test set was approximately 2.5 dB higher than that of the training set. All clips were downsampled to 16 kHz. Model performance was evaluated using metrics such as PESQ, STOI, SSNR, CSIG, CBAK, and COVL. These metrics serve as benchmarks for evaluating and quantifying the effectiveness of our model.

[0029] Table 1 shows the performance of our model compared to state-of-the-art models. SE-LMA-Transformer consistently outperforms existing state-of-the-art models in most metrics. For example, SE-LMA-Transformer improves DEMUCS by 0.33 PESQ, 0.7% STOI, 0.34 CSIG, 0.47 CBAK, and 0.49 COVL. We also compared the parameters of our method with those of some state-of-the-art methods, showing that the parameters of SE-LMA-Transformer are not significantly smaller than those of state-of-the-art comparable methods.

[0030] Table 1. Comparison of speech enhancement performance between SE-LMA-Transformer and advanced models.

[0031]

[0032]

[0033] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A single-channel noise reduction method for a lightweight multi-axis Transformer, characterized in that, Includes the following steps: The speech lightweight multi-axis transformer includes two training objectives: ideal scaling mask and complex spectrum compensation. The amplitude spectrum feature |Y(t,f)| and complex spectrum feature Y(t,f) are fed into a dense encoder block, where t and f represent the frame index and frequency point, respectively. The dense encoder block is used to capture low-level features, which are then processed by the cue block, the lightweight multi-axis transformer, and the remaining channel attention block to extract high-level features. The embedded fusion block further extracts and compresses these high-level features, and the system generates masking value M(t,f) and compensation value C(t,f) through dense decoding block similar to dense coding block; Hint blocks are used to facilitate improved frequency feature extraction, which is calculated as follows: Input feature X is transformed into Xi through global average pooling. GAP Simultaneously, a cue tensor P is introduced, which is a randomly initialized learnable tensor, obtained by adjusting X. GAP Applying a linear layer, we obtain the linear layer output X. Linear After processing by the Softmax function, X is... Linear Multiplying it by the hint tensor P, we get P. Mul The product result P Mul Summing in the first dimension yields the sum P. Sum By applying sigmoid activation, a frequency cue tensor is obtained, denoted as P. Sigmoid Then, element-wise multiplication is performed using the input feature X, followed by two-dimensional convolution and residual concatenation to generate the final output of the frequency cue module.

2. The single-channel noise reduction method for lightweight multi-axis Transformer according to claim 1, characterized in that: Low-complexity time-frequency multi-separation convolutional head self-attention is used to analyze the time and frequency of speech signals. This involves both time-frequency and frequency-frequency multi-separation convolutional head self-attention. The temporal separable convolutional head self-attention calculation method is as follows: the input features are expanded in the channel dimension through point-to-point convolution, and three basic components are generated using depthwise separable convolution and splitting operations to obtain the attention map; The attention map is extended by introducing a set of learnable parameters. The attention map is multiplied with the values, and then the output features are obtained through pointwise convolution.

Images