Two-way lightweight time-frequency domain adaptive neural network model and use method thereof

Abstract

The invention belongs to the technical field of speech enhancement. The invention provides a double-path lightweight time-frequency domain adaptive neural network model and a use method thereof. According to the embodiment of the invention, the depth separable convolutional layer is adopted, so that the calculation amount of the model can be reduced, and the noise reduction delay of the model is reduced; the proposed lightweight Transform layer adopts an MLP network, and multi-head attention is replaced by single-head attention, so that the quantity of model parameters is greatly reduced; time-frequency domain interactive fusion is carried out in the middle layer, so that the model can learn more time-frequency domain joint hidden features in the training process, and the noise reduction effect is greatly improved; a meta-learning dynamic activation function is introduced, the dynamic activation function adapts to non-stationary noise, a meta-training framework supports few-sample fine tuning, and the deployment cost is reduced; in the time-frequency feature adaptive fusion process, a regular term is added, so that the model does not excessively deviate from a frequency domain or a time domain in the training process, and the training effect of the model is ensured.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of speech enhancement technology, and more particularly to a dual-path lightweight time-frequency domain adaptive neural network model and a method for using the same. Background Art

[0002] In recent years, deep learning has made significant progress in the field of speech noise reduction. Existing speech noise reduction technologies mainly include traditional methods based on frequency domain processing, single-domain processing solutions based on deep learning, noise reduction solutions based on real-time speech processing, and speech enhancement models based on Transformers. Among them, there is a speech enhancement method based on neural networks, which performs a time-frequency transform on the original speech signal to obtain the original amplitude spectrum of the original speech signal; uses a time-dimensional convolution kernel to extract features from the original amplitude spectrum to obtain a time-domain smoothed feature map; uses a frequency-dimensional convolution kernel to extract features from the original amplitude spectrum to obtain a frequency-domain smoothed feature map; performs a combined feature extraction on the original amplitude spectrum, time-domain smoothed feature map, and frequency-domain smoothed feature map to obtain an enhanced amplitude spectrum of the original speech signal; and performs an inverse time-frequency transform on the enhanced amplitude spectrum to obtain an enhanced speech signal. By extracting time-frequency smoothing features from the original speech signal, dual-axis noise reduction can be achieved on both the time and frequency axes, and combined with a deep neural network, self-learning of noise reduction parameters can be achieved. However, the dual-axis convolution and deep neural network of this method lead to high computing delays, making it difficult to meet real-time requirements; fixed convolution kernel weights are difficult to adapt to complex noise changes; and the independent processing of time domain and frequency domain features may cause the model to be biased towards one domain.

[0003] The method also includes a speech enhancement method, which uses an encoder module to extract preset features from the original noisy speech signal; uses a dual-path module to model the preset features on the frequency axis to obtain frequency-axis features, and uses a dual-path module to model the preset features on the time axis to obtain time-axis features; uses an excitation decoder and a channel decoder to process the time-axis features to obtain a first denoised speech signal; uses a complex spectrum decoder to process the frequency-axis features to obtain a second denoised speech signal; and uses a merging module to merge the first denoised speech signal, the second denoised speech signal, and the original noisy speech signal to obtain a final estimated speech signal. However, this method uses multiple decoders (excitation, channel, and complex spectrum decoders) and merging modules, resulting in many model parameters and high computational complexity; poor effect on non-stationary noise, and limited generalization ability of the neural network; and only fuses time-frequency features through branch merging, without explicitly modeling time-frequency domain interactions.

[0004] Therefore, it is necessary to improve one or more problems existing in the above-mentioned related technical solutions.

[0005] It should be noted that this section is intended to provide background or context for the technical solutions of the present disclosure stated in the claims. The description herein is not admitted to be prior art by virtue of being included in this section. Summary of the Invention

[0006] The purpose of the embodiments of the present disclosure is to provide a dual-path lightweight time-frequency domain adaptive neural network model and a method for using the same, thereby overcoming one or more problems caused by the limitations and defects of related technologies, at least to a certain extent.

[0007] According to a first aspect of an embodiment of the present disclosure, a dual-path lightweight time-frequency domain adaptive neural network model is provided, comprising:

[0008] Input layer, parallel frequency domain branch network and time domain branch network, time-frequency feature adaptive fusion layer and output layer. The input layer is connected to the frequency domain branch network and the time domain branch network respectively. The frequency domain branch network and the time domain branch network are both connected to the time-frequency feature adaptive fusion layer. The time-frequency feature adaptive fusion layer is connected to the output layer.

[0009] The frequency domain branch network includes a serial FFT layer, a depthwise separable convolution layer, a lightweight Transformer layer, a frequency domain ECANet interaction layer, a frequency domain mask generation layer, and an IFFT layer. The FFT layer is connected to the input layer, and the IFFT layer is connected to the time-frequency feature adaptive fusion layer. The depthwise separable convolution layer includes a serial depthwise convolution layer, a pointwise convolution layer, and a batch normalization layer. The lightweight Transformer layer includes a serial 2-layer encoder, a 2-layer decoder, and a Softmax layer. The frequency domain mask generation layer includes a frequency domain meta-network and a frequency domain fully connected layer.

[0010] The time domain branch network includes a MobileNet-CNN subnetwork, two TCN modules, a time domain ECANet interaction layer and a time domain mask generation layer in series. The MobileNet-CNN subnetwork is connected to the input layer, the time domain fully connected layer is connected to the time-frequency feature adaptive fusion layer, and the frequency domain ECANet interaction layer is connected to the time domain ECANet interaction layer; the time domain mask generation layer includes a time domain meta-network and a time domain fully connected layer.

[0011] Furthermore, the depthwise separable convolution layer has 64 channels and a 5×3 convolution kernel, which performs convolution on each input channel separately. The pointwise convolution layer has a 1×1 convolution kernel, which is used to convolve the output of the depthwise convolution.

[0012] The decoder in the lightweight Transformer layer consists of a 2-layer MLP network and a normalization layer. The input dimension of the MLP network is 64 and the output dimension is 32.

[0013] The frequency domain meta-network in the frequency domain fully connected layer includes one convolutional layer and one RNN layer. The number of channels in the convolutional layer is 32, and the number of nodes in the RNN layer is 128.

[0014] Furthermore, the number of channels of the MobileNet-CNN subnetwork is 64, and the convolution kernel is 5×3;

[0015] The number of channels of the TCN module is 32, the convolution kernel is 5×5, and the forgetting rate is 0.2.

[0016] According to a second aspect of an embodiment of the present disclosure, a method for using a dual-path lightweight time-frequency domain adaptive neural network model is provided, the method comprising:

[0017] Obtain real-time speech signals and input them into the input layer for preprocessing to obtain the original time domain signals;

[0018] Use the FFT layer to perform FFT transformation on the original time domain signal to obtain the frequency domain signal;

[0019] Use the depth-wise separable convolutional layer to extract features from the frequency domain signal to obtain frequency domain features;

[0020] Use the lightweight Transformer layer to process the frequency domain features to obtain the frequency domain feature matrix array;

[0021] Use the MobileNet-CNN sub-network to extract the features of the original time domain signal to obtain the time domain features;

[0022] Use two TCN modules to map the time domain features to obtain a time domain feature matrix array;

[0023] The frequency domain ECANet interaction layer and the time domain ECANet interaction layer are used to perform time-frequency domain interaction on the frequency domain feature matrix array and the time domain feature matrix array to obtain frequency domain interaction results and time domain interaction results;

[0024] The frequency domain element network in the frequency domain mask generation layer is used to obtain the parameters of the frequency domain dynamic activation function according to the frequency domain interaction results, and the frequency domain fully connected layer is used to obtain the frequency domain mask according to the parameters of the frequency domain dynamic activation function and the frequency domain interaction results;

[0025] Multiply the frequency domain mask and the frequency domain feature to obtain the masked frequency domain feature;

[0026] Performing an inverse Fourier transform on the masked frequency domain features using an IFFT layer to obtain the first time domain signal;

[0027] The time domain meta-network in the time domain mask generation layer is used to obtain the parameters of the time domain dynamic activation function according to the time domain interaction results, and the time domain fully connected layer is used to obtain the time domain mask according to the parameters of the time domain dynamic activation function and the time domain interaction results;

[0028] Multiplying the time domain mask and the time domain feature to obtain the second time domain signal;

[0029] Performing signal fusion on the first time domain signal and the second time domain signal using a time-frequency feature adaptive fusion layer to obtain an audio signal;

[0030] Use the output layer to output the audio signal.

[0031] Furthermore, the steps of performing FFT transformation on the original time domain signal using the FFT layer to obtain the frequency domain signal specifically include:

[0032]

[0033] Where X(f) is the frequency domain signal of the current frame; f is the frequency point; x(n) is the original time domain signal, n = 0, 1, 2, ..., N, N is the number of sampling points per frame of audio; e is a constant;

[0034] The steps of extracting features from frequency domain signals using a depthwise separable convolutional layer to obtain frequency domain features specifically include:

[0035] F f =f DSConv (X(f))

[0036] Where, F f is the frequency domain feature, f DSConv (·) is the nonlinear transformation function of the multi-layer separable convolution stack;

[0037] The steps of parallelizing the frequency domain features using the lightweight Transformer layer to obtain the frequency domain feature matrix array include:

[0038] Input preparation: frequency domain feature F f Converted into sequence form As input to the lightweight Transformer layer:

[0039]

[0040] Where Reshape(·) is the operation of converting to sequence form;

[0041] Self-attention calculation: The attention mechanism is used to model the context of the frequency domain sequence; the query, key and value vector calculation formulas are:

[0042]

[0043] Where Q, K, and V are the query, key, and value vectors in the lightweight Transformer layer, respectively, and W Q 、W K 、W V are the weight matrices for query, key, and value vectors respectively;

[0044] Self-attention weight calculation: Use Linformer's efficient attention mechanism to reduce complexity:

[0045]

[0046] Where, α f is the attention weight of the frequency domain path, K T represents the transpose of K, d k is the dimension of the key vector;

[0047] Apply MLP network to obtain frequency domain audio features

[0048]

[0049] Residual connection and normalization: To improve stability, residual connection and normalization are introduced to obtain the frequency domain feature matrix array:

[0050]

[0051] Where, is the frequency domain feature matrix array, Norm(·) is the normalization operation;

[0052] The steps of using the frequency domain ECANet interaction layer and the time domain ECANet interaction layer to perform time-frequency domain interaction on the frequency domain feature matrix array and the time domain feature matrix array to obtain the frequency domain interaction result and the time domain interaction result specifically include:

[0053] Use the frequency domain ECANet interaction layer to model the interaction between the time domain and the frequency domain:

[0054] Time-frequency domain features are spliced to obtain the spliced time-frequency features F inter :

[0055]

[0056] Where, is the frequency domain feature matrix array, is the time domain feature matrix array;

[0057] Calculate the time-frequency domain interaction attention weights:

[0058]

[0059] Where c ft is the global average of the time domain interaction features, H and W are the feature maps F inter height and width; is the time-frequency domain interaction attention weight, σ(·) is the sigmoid function, and Conv1D(·) is the one-dimensional convolution;

[0060] Time-frequency domain interactive output:

[0061]

[0062] Where, It is the frequency domain interaction result output by the frequency domain path after the time-frequency domain interaction;

[0063]

[0064] Where F′ t_out It is the time domain interaction result output by the time domain path after the time-frequency domain interaction;

[0065] The steps of obtaining the parameters of the frequency domain dynamic activation function according to the frequency domain interaction results using the frequency domain element network in the frequency domain mask generation layer, and obtaining the frequency domain mask according to the parameters of the frequency domain dynamic activation function and the frequency domain interaction results using the frequency domain fully connected layer include:

[0066] Frequency domain element network input: the frequency domain interaction result of the frequency domain path output As the input of the frequency domain element network;

[0067] Dynamic parameter generation: The frequency domain element network outputs the parameters θ of the frequency domain dynamic activation function; wherein the parameters θ of the frequency domain dynamic activation function include the mask amplitude adjustment coefficient α and the nonlinear response slope β;

[0068] Dynamic activation calculation: Apply the parameters α and β of the frequency domain dynamic activation function to the Sigmoid function to generate a frequency domain mask:

[0069]

[0070] Where M(f) is the frequency domain mask of the frequency domain path, f is the frequency point, α is the frequency domain mask amplitude adjustment coefficient, β is the frequency domain nonlinear response slope, FC(·) is the frequency domain fully connected network layer, and MetaNet is the frequency domain meta-network;

[0071] The specific steps of multiplying the frequency domain mask and the frequency domain feature to obtain the masked frequency domain feature include:

[0072] Apply the frequency domain mask M(f) to the original frequency domain feature F f , to mask the noise in the spectrum and achieve the effect of frequency domain noise reduction;

[0073]

[0074] Where, It is the frequency domain feature after mask processing;

[0075] The specific steps of performing an inverse Fourier transform on the masked frequency domain features using the IFFT layer to obtain the first time domain signal include:

[0076] Frequency domain features after mask processing Perform inverse Fourier transform to restore the time domain signal:

[0077]

[0078] Where, It is the first time domain signal obtained after frequency domain denoising.

[0079] Furthermore, the MobileNet-CNN subnetwork is used to extract the features of the original time domain signal to obtain the time domain features. The specific steps include:

[0080] The input is a frame of original time domain signal after preprocessing, denoted as x(t), where t represents time, t=1, 2, 3, ..., T.

[0081] Extract temporal features F based on MobileNet-CNN sub-network t :

[0082] F t =A(R(C(x(t))))

[0083] Where C(·) is the separable convolution calculation; R(·) is the residual connection calculation; A(·) is the HardSwish activation function of the MobileNet-CNN subnetwork;

[0084] The specific steps of using two TCN modules to map the time domain features to obtain the time domain feature matrix array include:

[0085] Based on two TCN modules, the time domain features are further mapped, including causal convolution, dilated convolution, residual connection and activation function;

[0086] 1) For the time domain feature F t Perform causal convolution to capture hidden features in the time domain:

[0087]

[0088] Where, F t (1)is the result of causal convolution, t is the t-th time step; Z is the size of the convolution kernel, generally 5×5 convolution kernel, that is, Z=5; p is the p-th convolution step, W p is the causal convolution weight parameter;

[0089] 2) For the time domain feature F t Perform dilated convolution:

[0090]

[0091] Where, F t (2) is the result of the dilated convolution, d is the dilation factor, which represents the interval of the convolution kernel on the input sequence, and W d is the dilated convolution weight parameter;

[0092] 3) Residual connection: Add the causal convolution results, dilated convolution results and time domain features to obtain the output after residual connection:

[0093]

[0094] 4) Use the ReLU function to activate the output of the residual connection:

[0095]

[0096] Where ReLU(·) is the ReLU activation function;

[0097] The specific steps of obtaining the parameters of the time domain dynamic activation function according to the time domain interaction results by using the time domain meta-network in the time domain mask generation layer, and obtaining the time domain mask according to the parameters of the time domain dynamic activation function and the time domain interaction results by using the time domain fully connected layer include:

[0098] Time domain meta-network input: the time domain interaction result F′ output by the time domain path t_out As input to the time domain meta-network;

[0099] Dynamic parameter generation: The time domain meta-network outputs the parameters of the time domain dynamic activation function; the parameters of the time domain dynamic activation function include the time domain mask amplitude adjustment coefficient α t and the time domain nonlinear response slope β t ;

[0100] Dynamic activation calculation: The parameter α of the time domain dynamic activation function t and β t Applied to the Sigmoid function to generate a frequency domain mask:

[0101] m(t)=α t Sigmoid(β t FC t(F′ t_out ))

[0102] Among them, m(t) is the mask of the time domain path, α t is the time domain mask amplitude adjustment coefficient, β t The time domain is the nonlinear response slope, FC t (·) is the time domain fully connected network layer;

[0103] The step of multiplying the time domain mask and the time domain feature to obtain the second time domain signal specifically includes:

[0104] Apply the temporal mask m(t) to the temporal features F t , to mask the noise in the time domain and achieve the effect of time domain noise reduction:

[0105]

[0106] Where, It is the second time domain signal after time domain noise reduction.

[0107] Furthermore, the step of fusing the first time domain signal and the second time domain signal using the time-frequency feature adaptive fusion layer to obtain the audio signal includes:

[0108] The time-frequency domain fusion signal is calculated by the following formula:

[0109]

[0110] Where, is the fused audio signal, W f is the frequency domain weight, W t is the time domain weight, λ is the fusion coefficient, and its value is 0.04.

[0111] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0112] In the embodiments of the present disclosure, through the above-mentioned dual-path lightweight time-frequency domain adaptive neural network model and its use method, on the one hand, by combining time domain features and frequency domain feature processing, more efficient noise suppression can be achieved in complex noise environments, significantly improving the signal-to-noise ratio of speech signals.

[0113] Secondly, the use of a lightweight neural network structure reduces the consumption of computing resources, enabling this application to achieve real-time processing on resource-constrained devices, and is suitable for mobile devices and embedded systems. Among them, the use of a deep separable convolutional layer can reduce the amount of computation of the model, thereby reducing the noise reduction delay of the model; the lightweight Transformer layer proposed in this application uses an MLP network, thereby replacing multi-head attention with single-head attention, to significantly reduce the number of model parameters; this application performs interactive fusion of time and frequency domains in the middle layer, so that the model can learn more joint hidden features of time and frequency domains during training, which greatly improves the noise reduction effect; this application introduces a meta-learning dynamic activation function, which not only adapts to non-stationary noise, but also supports small sample fine-tuning in the meta-training framework, reducing deployment costs; this application adds a regularization term in the process of adaptive fusion of time and frequency features, which can prevent the model from being too biased towards the frequency domain or time domain during training, thereby ensuring the training effect of the model.

[0114] Thirdly, while removing noise, the naturalness and clarity of the speech signal are preserved to the greatest extent possible, reducing speech distortion and improving the user's listening experience. The use of a dynamic activation function makes the model highly robust to complex noise and can dynamically adapt to non-stationary noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0115] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0116] Figure 1 A schematic diagram illustrating a framework of a dual-path lightweight time-frequency domain adaptive neural network model in an exemplary embodiment of the present disclosure is shown;

[0117] Figure 2 A diagram showing the steps of a method for using a dual-path lightweight time-frequency domain adaptive neural network model in an exemplary embodiment of the present disclosure;

[0118] Figure 3 A schematic diagram showing a lightweight Transformer layer in an exemplary embodiment of the present disclosure;

[0119] Figure 4 A comparison chart showing the results of Example 1 in the present disclosure is shown;

[0120] Figure 5 A comparison chart showing the results of Example 2 in the present disclosure is shown;

[0121] Figure 6 A comparison chart showing the results of Example 3 in the present disclosure is shown;

[0122] Figure 7 A comparison chart showing the results of Example 4 in the present disclosure is shown;

[0123] Figure 8 A comparison chart showing the results of Example 5 in the present disclosure is shown;

[0124] Figure 9 A comparison chart showing the results of Example 6 in this disclosure is shown.

[0125] Figure 10 A comparison chart of the results of Example 7 in the present disclosure is shown. DETAILED DESCRIPTION

[0126] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0127] In addition, the accompanying drawings are merely schematic illustrations of embodiments of the present disclosure and are not necessarily drawn to scale. Like reference numerals in the figures represent like or similar parts, and thus repeated descriptions thereof will be omitted. Some of the blocks shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically separate entities.

[0128] This example implementation first provides a dual-path lightweight time-frequency domain adaptive neural network model. Figure 1 As shown in , the dual-path lightweight time-frequency domain adaptive neural network model may include:

[0129] Input layer, parallel frequency domain branch network and time domain branch network, time-frequency feature adaptive fusion layer and output layer. The input layer is connected to the frequency domain branch network and the time domain branch network respectively. The frequency domain branch network and the time domain branch network are both connected to the time-frequency feature adaptive fusion layer. The time-frequency feature adaptive fusion layer is connected to the output layer.

[0130] The frequency domain branch network includes a serial FFT layer, a depthwise separable convolution layer, a lightweight Transformer layer, a frequency domain ECANet interaction layer, a frequency domain mask generation layer, and an IFFT layer. The FFT layer is connected to the input layer, and the IFFT layer is connected to the time-frequency feature adaptive fusion layer. The depthwise separable convolution layer includes a serial depthwise convolution layer, a pointwise convolution layer, and a batch normalization layer. The lightweight Transformer layer includes a serial 2-layer encoder, a 2-layer decoder, and a Softmax layer. The frequency domain mask generation layer includes a frequency domain meta-network and a frequency domain fully connected layer.

[0131] The time domain branch network includes a MobileNet-CNN subnetwork, two TCN modules, a time domain ECANet interaction layer and a time domain mask generation layer in series. The MobileNet-CNN subnetwork is connected to the input layer, the time domain fully connected layer is connected to the time-frequency feature adaptive fusion layer, and the frequency domain ECANet interaction layer is connected to the time domain ECANet interaction layer; the time domain mask generation layer includes a time domain meta-network and a time domain fully connected layer.

[0132] Through the above-mentioned dual-path lightweight time-frequency domain adaptive neural network model, firstly, by combining time domain feature and frequency domain feature processing, it is possible to achieve more efficient noise suppression in complex noise environments and significantly improve the signal-to-noise ratio of speech signals.

[0133] Secondly, the use of a lightweight neural network structure reduces the consumption of computing resources, enabling this application to achieve real-time processing on resource-constrained devices, and is suitable for mobile devices and embedded systems. Among them, the use of a deep separable convolutional layer can reduce the amount of computation of the model, thereby reducing the noise reduction delay of the model; the lightweight Transformer layer proposed in this application uses an MLP network, thereby replacing multi-head attention with single-head attention, to significantly reduce the number of model parameters; this application performs interactive fusion of time and frequency domains in the middle layer, so that the model can learn more joint hidden features of time and frequency domains during training, which greatly improves the noise reduction effect; this application introduces a meta-learning dynamic activation function, which not only adapts to non-stationary noise, but also supports small sample fine-tuning in the meta-training framework, reducing deployment costs; this application adds a regularization term in the process of adaptive fusion of time and frequency features, which can prevent the model from being too biased towards the frequency domain or time domain during training, thereby ensuring the training effect of the model.

[0134] Thirdly, while removing noise, the naturalness and clarity of the speech signal are preserved to the greatest extent possible, reducing speech distortion and improving the user's listening experience. The use of a dynamic activation function makes the model highly robust to complex noise and can dynamically adapt to non-stationary noise.

[0135] Below, we will refer to Figure 1The various parts of the above-mentioned dual-path lightweight time-frequency domain adaptive neural network model in this example implementation are described in more detail.

[0136] In one embodiment, the decoder in the lightweight Transformer layer includes a 2-layer MLP network and a normalization layer. The input dimension of the MLP network is 64 and the output dimension is 32. The frequency domain meta-network in the frequency domain fully connected layer includes 1 convolution layer and 1 RNN layer. The number of channels of the convolution layer is 32, and the number of nodes of the RNN layer is 128.

[0137] In one embodiment, the number of channels of the MobileNet-CNN subnetwork is 64, the convolution kernel is 5×3; the number of channels of the TCN module is 32, the convolution kernel is 5×5, and the forgetting rate is 0.2.

[0138] Furthermore, this example embodiment also provides a method for using a dual-path lightweight time-frequency domain adaptive neural network model. Figure 2 As shown in , the method for using the dual-path lightweight time-frequency domain adaptive neural network model may include:

[0139] Step S101: obtaining a real-time speech signal and inputting it into the input layer for preprocessing to obtain an original time domain signal;

[0140] Step S102: using the FFT layer to perform FFT transformation on the original time domain signal to obtain a frequency domain signal;

[0141] Step S103: extracting features from the frequency domain signal using a depthwise separable convolutional layer to obtain frequency domain features;

[0142] Step S104: Processing the frequency domain features using a lightweight Transformer layer to obtain a frequency domain feature matrix array;

[0143] Step S105: using the MobileNet-CNN sub-network to extract features of the original time domain signal to obtain time domain features;

[0144] Step S106: Mapping the time domain features using two TCN modules to obtain a time domain feature matrix array;

[0145] Step S107: using the frequency domain ECANet interaction layer and the time domain ECANet interaction layer to perform time-frequency domain interaction on the frequency domain feature matrix array and the time domain feature matrix array to obtain a frequency domain interaction result and a time domain interaction result;

[0146] Step S108: using the frequency domain element network in the frequency domain mask generation layer to obtain the parameters of the frequency domain dynamic activation function according to the frequency domain interaction result, and using the frequency domain fully connected layer to obtain the frequency domain mask according to the parameters of the frequency domain dynamic activation function and the frequency domain interaction result;

[0147] Step S109: multiplying the frequency domain mask and the frequency domain feature to obtain the masked frequency domain feature;

[0148] Step S110: using an IFFT layer to perform an inverse Fourier transform on the masked frequency domain features to obtain the first time domain signal;

[0149] Step S111: using the time domain meta-network in the time domain mask generation layer to obtain the parameters of the time domain dynamic activation function according to the time domain interaction result, and using the time domain fully connected layer to obtain the time domain mask according to the parameters of the time domain dynamic activation function and the time domain interaction result;

[0150] Step S112: multiplying the time domain mask and the time domain feature to obtain the second time domain signal;

[0151] Step S113: using a time-frequency feature adaptive fusion layer to perform signal fusion on the first time domain signal and the second time domain signal to obtain an audio signal;

[0152] Step S114: Output the audio signal using the output layer.

[0153] In step S101 , a real-time speech signal is acquired and input into an input layer for preprocessing to obtain an original time domain signal.

[0154] Specifically, 1. Voice signal collection:

[0155] Use a microphone or other audio acquisition device to obtain real-time voice signals, and convert the collected real-time voice signals into digital signals through AD analog-to-digital conversion;

[0156] 2. Feature Standardization:

[0157] The z-score normalization method was used to standardize the digital signal to eliminate the influence of different signal amplitudes.

[0158] 3. Feature framing:

[0159] The standardized digital signal is framed to obtain the original time domain signal. Each frame of data is 16ms (16KHz sampling rate). Overlapping framing technology is used to ensure smooth transition between frames. The overlap rate is 50%, that is, 8ms frame overlap.

[0160] In step S102, an FFT layer is used to perform an FFT transform on the original time domain signal to obtain a frequency domain signal.

[0161] Specifically, perform FFT transformation on the original time domain signal:

[0162]

[0163] Where X(f) is the frequency domain signal of the current frame; f is the frequency point; x(n) is the original time domain signal, n = 0, 1, 2, ..., N, N is the number of sampling points per frame of audio; e is a constant.

[0164] In step S103, a depthwise separable convolutional layer is used to extract features from the frequency domain signal to obtain frequency domain features.

[0165] Specifically, a depthwise separable convolution layer is used to extract features from the frequency domain amplitude spectrum X(f):

[0166] F f =f DSConv (X(f))

[0167] Where, F f is the frequency domain feature, f DSConv (·) is the nonlinear transformation function of the multi-layer separable convolution stack. Its implementation steps include depthwise convolution and pointwise convolution as well as batch normalization and activation:

[0168] 1) Depthwise convolution: The number of channels is 64, the convolution kernel is 5×3, and convolution is performed on each input channel separately.

[0169] 2) Point-wise convolution: Convolve the output of depthwise convolution with a 1×1 convolution kernel.

[0170] 3) Batch normalization and activation: Normalize the convolution result and use the ReLU function for nonlinear activation to obtain the final frequency domain feature F f .

[0171] The use of the deep separable convolutional neural network (ds-cnn) layer is to reduce the computational complexity of the model, thereby reducing the denoising delay of the model.

[0172] In step S104, the frequency domain features are parallelized using a lightweight Transformer layer to obtain a frequency domain feature matrix array.

[0173] Specifically, the lightweight Transform layer consists of a two-layer encoder, a two-layer decoder, and a softmax layer. The multi-head attention module in each decoder layer is replaced with a two-layer MLP network to reduce the number of model parameters. The input dimension is 64, and the output dimension is 32.

[0174] 1) Input preparation: frequency domain feature F f Converted into sequence form As input to the lightweight Transformer layer:

[0175]

[0176] Where Reshape(·) is the operation of converting to sequence form.

[0177] 2) Self-attention calculation: The attention mechanism is used to model the context of the frequency domain sequence. The query, key, and value vectors are calculated as follows:

[0178]

[0179] Where Q, K, and V are the query, key, and value vectors in the lightweight Transformer layer, respectively, and W Q 、W K 、W V are the weight matrices for query, key, and value vectors respectively.

[0180] 3) Self-attention weight calculation: Use Linformer’s efficient attention mechanism to reduce complexity:

[0181]

[0182] Where, α f is the attention weight of the frequency domain path, K T represents the transpose of K, d k is the dimension of the key vector.

[0183] 4) Apply MLP network to obtain frequency domain audio features

[0184]

[0185] Where MLP(·) represents an MLP multilayer perceptron network with 64 input nodes, 32 output nodes, and 2 output layers.

[0186] 5) Residual connection and normalization: To improve stability, residual connection and normalization are introduced to obtain the frequency domain feature matrix array:

[0187]

[0188] Where, is the frequency domain feature matrix array, and Norm(·) is the normalization operation.

[0189] Since the existing Transformer layer has a large amount of computation, it will affect the operating efficiency of the model and make it difficult to run in real time on the edge. Therefore, this application proposes to replace the existing Transformer layer's multi-head attention mechanism with an MLP multi-layer perceptron (this application uses a 2-layer MLP structure), thereby replacing the multi-head attention with a single-head attention to significantly reduce the number of model parameters. The implementation method is as follows Figure 3 shown.

[0190] By using an improved lightweight attention module to significantly reduce the number of model parameters, the model can be deployed on low-resource hardware.

[0191] In step S107, the frequency domain ECANet interaction layer and the time domain ECANet interaction layer are used to perform time-frequency domain interaction on the frequency domain feature matrix array and the time domain feature matrix array to obtain a frequency domain interaction result and a time domain interaction result.

[0192] Specifically, after the lightweight Transformer output, the frequency domain ECANet interaction layer is used to model the interaction relationship between the time domain and the frequency domain:

[0193] 1) Time-frequency domain feature splicing to obtain the spliced time-frequency feature F inter :

[0194]

[0195] Where, is the frequency domain feature matrix array, is the time domain feature matrix array.

[0196] 2) Calculate the time-frequency domain interaction attention weights:

[0197]

[0198] Where c ft is the global average of the time domain interaction features; H and W are the feature maps F inter height and width; is the time-frequency domain interaction attention weight; σ(·) is the sigmoid function; Conv1D(·) is a one-dimensional convolution layer with 32 channels, a convolution kernel of 5×3, and a forgetting rate of 0.3.

[0199] 3) Time-frequency domain interactive output:

[0200]

[0201] Where, It is the frequency domain interaction result output by the frequency domain path after the time-frequency domain interaction.

[0202]

[0203] Where F′ t_out It is the time domain interaction result output by the time domain path after the time-frequency domain interaction.

[0204] Ordinary time-frequency domain dual-path network structures often only perform fusion in the output layer of the time-frequency domain, while this method performs time-frequency domain interactive fusion in the middle layer of the network, allowing the model to learn more joint hidden features in the time-frequency domain during training.

[0205] In step S108, the frequency domain element network in the frequency domain mask generation layer is used to obtain the parameters of the frequency domain dynamic activation function according to the frequency domain interaction result, and the frequency domain fully connected layer is used to obtain the frequency domain mask according to the parameters of the frequency domain dynamic activation function and the frequency domain interaction result.

[0206] Specifically, a dynamic activation function is used to obtain the mask, and the dynamic activation function implements parameter adaptation through the meta-network:

[0207] 1) Frequency domain element network input: the frequency domain interaction result output by the frequency domain path As the input of the frequency domain element network;

[0208] 2) Dynamic parameter generation: The frequency domain element network outputs the parameters θ of the frequency domain dynamic activation function; wherein the parameters θ of the frequency domain dynamic activation function include the mask amplitude adjustment coefficient α and the nonlinear response slope β;

[0209] 3) Dynamic activation calculation: Apply the parameters α and β of the frequency domain dynamic activation function to the Sigmoid function to generate a frequency domain mask:

[0210]

[0211] Where M(f) is the frequency domain mask of the frequency domain path, f is the frequency point, α is the frequency domain mask amplitude adjustment coefficient, β is the frequency domain nonlinear response slope, and FC(·) is the frequency domain fully connected network layer. MetaNet is a frequency domain meta-network consisting of one convolutional layer (32 channels) and one RNN layer (128 nodes).

[0212] It is understandable that environmental adaptation: compared with fixed functions, meta-learning dynamic activation can automatically adjust parameters according to the noise type (such as sudden knocking sounds vs. continuous engine sounds) to achieve targeted suppression.

[0213] Edge efficiency: MetaNet requires only 0.05M parameters and adds less than 0.3ms of latency on the RV1126 chip.

[0214] Few-sample adaptation: Through the meta-training framework, the model can be fine-tuned with only 10 target scene voices, adapting to vertical scenarios such as factories and vehicles.

[0215] In step S109 , the frequency domain mask is multiplied by the frequency domain feature to obtain the masked frequency domain feature.

[0216] Specifically, the frequency domain mask M(f) is applied to the original frequency domain feature F f , to mask the noise in the spectrum and achieve the effect of frequency domain noise reduction;

[0217]

[0218] Where, It is the frequency domain feature after mask processing;

[0219] In step S110, an inverse Fourier transform is performed on the masked frequency domain features using an IFFT layer to obtain the first time domain signal.

[0220] Specifically, the frequency domain features after mask processing Perform inverse Fourier transform to restore the time domain signal:

[0221]

[0222] Where, It is the first time domain signal obtained after frequency domain denoising.

[0223] In step S105, the features of the original time domain signal are extracted using the MobileNet-CNN sub-network to obtain time domain features.

[0224] Specifically, time domain feature extraction (based on the MobileNet-CNN subnetwork, the number of channels of the MobileNet-CNN subnetwork is 64, and the convolution kernel is 5×3):

[0225] The input is a frame of original time domain signal after preprocessing, denoted as x(t), where t represents time, t=1, 2, 3, ..., T.

[0226] Extract temporal features F based on MobileNet-CNN sub-network t :

[0227] F t =A(R(C(x(t))))

[0228] Where C(·) is the separable convolution calculation; R(·) is the residual connection calculation; A(·) is the HardSwish activation function of the MobileNet-CNN subnetwork;

[0229] In step S106, two TCN modules are used to map the time domain features to obtain a time domain feature matrix array.

[0230] Specifically, the number of channels in each layer of the TCN network is 32, the convolution kernel is 5×5, and the forgetting rate is 0.2.

[0231] Based on two TCN modules, the time domain features are further mapped, including causal convolution, dilated convolution, residual connection and activation function;

[0232] 1) For the time domain feature F t Perform causal convolution to capture hidden features in the time domain:

[0233]

[0234] Where, F t (1) is the result of causal convolution, t is the t-th time step; Z is the size of the convolution kernel, generally 5×5 convolution kernel, that is, Z=5; p is the p-th convolution step, W p is the causal convolution weight parameter;

[0235] 2) For the time domain feature F t Perform dilated convolution:

[0236]

[0237] Where, F t (2) is the result of the dilated convolution, d is the dilation factor, which represents the interval of the convolution kernel on the input sequence, and W d is the dilated convolution weight parameter;

[0238] 3) Residual connection: Add the causal convolution results, dilated convolution results and time domain features to obtain the output after residual connection:

[0239]

[0240] 4) Use the ReLU function to activate the output of the residual connection:

[0241]

[0242] Where ReLU(·) is the ReLU activation function;

[0243] In step S111, the time domain meta-network in the time domain mask generation layer is used to obtain the parameters of the time domain dynamic activation function according to the time domain interaction result, and the time domain fully connected layer is used to obtain the time domain mask according to the parameters of the time domain dynamic activation function and the time domain interaction result.

[0244] Specifically, a dynamic activation function is used to obtain the mask, and the dynamic activation function is used to achieve parameter adaptation through the meta-network:

[0245] 1) Time domain meta-network input: the time domain interaction result F′ output by the time domain path t_out As the input of the time domain meta-network;

[0246] 2) Dynamic parameter generation: The time domain meta-network outputs the parameters of the time domain dynamic activation function; wherein, the parameters of the time domain dynamic activation function include the time domain mask amplitude adjustment coefficient α t and the time domain nonlinear response slope β t ;

[0247] 3) Dynamic activation calculation: The parameter α of the time domain dynamic activation function t and β t Applied to the Sigmoid function to generate a frequency domain mask:

[0248] m(t)=α t Sigmoid(β t FC t (F′ t_out ))

[0249] Among them, m(t) is the mask of the time domain path; α t is the time domain mask amplitude adjustment coefficient, β t The time domain is a nonlinear response slope, which is calculated in the same way as the mask calculation of the frequency domain path. It is calculated using the frequency domain meta-network, which is composed of a 1D convolution layer and a 1-layer RNN layer. t (·) is the time domain fully connected network layer.

[0250] In step S112, the time domain mask is multiplied by the time domain feature to obtain the second time domain signal.

[0251] Specifically, the time domain mask m(t) is applied to the time domain feature F t , to mask the noise in the time domain and achieve the effect of time domain noise reduction.

[0252]

[0253] Where, It is the second time domain signal after time domain noise reduction.

[0254] In step S113, the first time domain signal and the second time domain signal are fused using a time-frequency feature adaptive fusion layer to obtain an audio signal.

[0255] Specifically, the purpose of the adaptive fusion layer is to combine the signals processed in the frequency domain and time domain and dynamically adjust their weights to achieve optimal signal fusion. The following is a detailed implementation process:

[0256] Input signal:

[0257] The signal after frequency domain processing:

[0258] The signal after time domain processing:

[0259] Signal Fusion:

[0260] The time-frequency domain fusion signal is calculated by the following formula:

[0261]

[0262] Where, is the fused audio signal, W f is the frequency domain weight, W t is the time domain weight, λ is the fusion coefficient, and its value is 0.04.

[0263] In addition, the formula The addition of is equivalent to adding a regular term to the model, which can prevent the model from being too biased towards the frequency domain or time domain during training, thereby ensuring the training effect of the model.

[0264] In step S114 , the audio signal is output using the output layer.

[0265] Specifically, the audio signal after fusion and noise reduction is converted into digital-to-analog form and then output to headphones or speakers.

[0266] In a specific embodiment, a dual-path enhanced neural network in the time-frequency domain is trained.

[0267] Specifically, we used open source datasets, the librisspeech corpus for speech signals and the Audioset corpus for noise signals. We created speech-noise mixed signals with a signal-to-noise ratio range of -5 to 25dB and divided them into 80% training set and 20% test set. The loss function used was the mean square error function. Training strategy: The Adam optimizer used a learning rate of 10 -3 , if the validation set loss does not improve for three consecutive epochs, the learning rate is halved.

[0268] Parameter settings:

[0269] Voice input: 16ms, 16kHz; FFT frequency setting: 512;

[0270] Frequency domain path parameters: DS-CNN depthwise convolution: 64 channels, 5×3 convolution kernel, DS-CNN pointwise convolution: 64 channels, 1×1 convolution kernel; lightweight Transform layer parameters: input dimension 64, output dimension 32, number of encoder layers: 2 layers, number of decoder layers 2 layers; number of single-head attention MLP layers: 2 layers; 1D convolution parameters of the time-frequency domain interaction layer: 32 channels, 5×3 convolution kernel, forgetting rate 0.3; meta-network of the mask calculation layer: 1 convolution layer (32 channels) + 1 RNN layer (128 nodes).

[0271] Temporal path parameters: MobileNet-CNN network: 64 channels, 5×3 convolution kernel; Dual TCN module: 2-layer TCN network, 32 channels, 5×5 convolution kernel, forgetting rate 0.2.

[0272] Time-frequency domain fusion layer: The fusion coefficient is 0.04.

[0273] To support deployment on edge devices, the frequency and time domain paths are parallelized for computation, significantly improving computational efficiency. The model is then quantized to INT8 fixed-point and deployed on the RV1126 chip (1GHz). The computational latency is reduced to just 6.5ms, meeting the real-time communication requirements of low-resource edge devices and suppressing noise by over 30dB.

[0274] Table 1 Comparison of various noise reduction methods

[0275]

[0276] In Example 1, Figure 4 As shown, this is an example of audio after noise reduction of -20db noise audio using the time-frequency domain joint modeling speech noise reduction method for edge devices of this application.

[0277] In Example 2, Figure 5 As shown, this is an example of audio after noise reduction is performed on an ambient recording using the time-frequency domain joint modeling speech noise reduction method for edge devices of the present application.

[0278] It can be seen that the human voice is preserved intact without any damage to the speech, while the noise is eliminated very cleanly.

[0279] In Example 3, Figure 6 The figure below shows an example comparison of the noise-reduced audio obtained by adding noise to a pure human voice using the time-frequency domain joint modeling speech noise reduction method for edge devices of this application. As can be seen, in the energy graph of the "noisy frequency" image, the "horizontal line" at the bottom of the image represents the main noise, while in the energy graph of the "noised audio" image, it can be seen that the noise has been eliminated. Conclusion: Noise removal is significant, and the human voice is preserved intact, with excellent results.

[0280] In Example 4, Figure 7 The following is an example of an audio signal after noise reduction is applied to non-stationary ambient noise in a public place using the time-frequency domain joint modeling speech noise reduction method for edge devices of this application. As can be seen from the time domain waveform in the box, the noise is filtered out very cleanly.

[0281] In Example 5, Figure 8The following is an example of audio after using the time-frequency domain joint modeling speech noise reduction method for edge devices in this application to reduce non-stationary music noise. As can be seen from the time domain waveform in the upper frame, this application can still effectively filter out non-stationary noise such as music.

[0282] In Example 6, Figure 9 The following is an example of audio after noise reduction was performed on non-stationary burst clapping noise using the time-frequency domain joint modeling speech noise reduction method for edge devices in this application. As can be seen from the waveform, the burst noise of "clapping" is also filtered out after processing in this application.

[0283] In Example 7, Figure 10 The following is an example of audio after noise reduction using this application's time-frequency domain joint modeling speech noise reduction method for edge devices. As can be seen from the waveform, this application can effectively filter out the noise and restore the human voice for steady-state sounds such as "air conditioning noise."

[0284] Furthermore, we conducted an experimental verification on the performance of the lightweight Transformer module. The comparison results are shown in Table 2:

[0285] Table 2 Transformer module performance comparison results

[0286]

[0287] It can be seen that the improved lightweight attention module proposed in this application significantly reduces the number of model parameters by 67%, while the signal-to-noise ratio value is only reduced by 4.6% and the speech intelligibility PESQ is only reduced by 5.8%.

[0288] Furthermore, the frequency domain ECANet interaction layer was experimentally verified and compared with data. The results are shown in Table 3:

[0289] Table 3 Comparison results with and without frequency domain ECANet interaction layer

[0290] Method comparison SISNR(dB) PESQ The time and frequency domains are independent 17.96 2.81 Adding time-frequency domain interaction layer 19.87 2.94

[0291] It can be seen that adding interaction layers to the network structures of the respective branches in the time and frequency domains improves the noise reduction effect of the model.

[0292] Furthermore, the performance of fixed mask activation and meta-learning dynamic activation is compared, and the results are shown in Table 4:

[0293] Table 4 Comparison results between fixed mask activation and meta-learning dynamic activation

[0294] Noise reduction methods Parameter quantity (M) SISNR(dB) PESQ Fixed mask activation 0.81 18.24 2.67 Meta-learning dynamic activation 0.82 19.87 2.94

[0295] Edge deployment verification

[0296] Hardware platform: RV1126 chip (1GHz main frequency, INT8 quantization);

[0297] Real-time performance: The total latency increased from 6.2ms to 6.5ms, still meeting the real-time requirements of the edge.

[0298] Energy efficiency ratio: Power consumption increased by 0.01W, which is within an acceptable range.

[0299] It can be seen that by introducing the meta-learning dynamic activation function, the following advantages can be achieved:

[0300] 1. Complex noise robustness: Dynamic parameters adapt to non-stationary noise, improving SISNR by 1.63dB;

[0301] 2. Rapid scenario adaptation: The meta-training framework supports fine-tuning with a small number of samples, reducing deployment costs.

[0302] 3. Edge-friendly: Lightweight design ensures real-time performance and is compatible with low-computing-power chips such as RV1126.

[0303] Through the above-mentioned dual-path lightweight time-frequency domain adaptive neural network model and its usage method, on the one hand, by combining time domain feature and frequency domain feature processing, more efficient noise suppression can be achieved in complex noise environments, significantly improving the signal-to-noise ratio of speech signals.

[0304] Secondly, the use of a lightweight neural network structure reduces the consumption of computing resources, enabling this application to achieve real-time processing on resource-constrained devices, and is suitable for mobile devices and embedded systems. Among them, the use of a deep separable convolutional layer can reduce the amount of computation of the model, thereby reducing the noise reduction delay of the model; the lightweight Transformer layer proposed in this application uses an MLP network, thereby replacing multi-head attention with single-head attention, to significantly reduce the number of model parameters; this application performs interactive fusion of time and frequency domains in the middle layer, so that the model can learn more joint hidden features of time and frequency domains during training, which greatly improves the noise reduction effect; this application introduces a meta-learning dynamic activation function, which not only adapts to non-stationary noise, but also supports small sample fine-tuning in the meta-training framework, reducing deployment costs; this application adds a regularization term in the process of adaptive fusion of time and frequency features, which can prevent the model from being too biased towards the frequency domain or time domain during training, thereby ensuring the training effect of the model.

[0305] Thirdly, while removing noise, the naturalness and clarity of the speech signal are preserved to the greatest extent possible, reducing speech distortion and improving the user's listening experience. The use of a dynamic activation function makes the model highly robust to complex noise and can dynamically adapt to non-stationary noise.

[0306] In addition, this application reduces processing delays, meets the needs of real-time voice communication and interaction on embedded low-resource chips, and improves the practicality of the system.

[0307] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0308] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A dual-path lightweight time-frequency domain adaptive neural network model, characterized by: include: Input layer, parallel frequency domain branch network and time domain branch network, time-frequency feature adaptive fusion layer and output layer. The input layer is connected to the frequency domain branch network and the time domain branch network respectively. The frequency domain branch network and the time domain branch network are both connected to the time-frequency feature adaptive fusion layer. The time-frequency feature adaptive fusion layer is connected to the output layer. The frequency domain branch network includes a serial FFT layer, a depthwise separable convolution layer, a lightweight Transformer layer, a frequency domain ECANet interaction layer, a frequency domain mask generation layer, and an IFFT layer. The FFT layer is connected to the input layer, and the IFFT layer is connected to the time-frequency feature adaptive fusion layer. The depthwise separable convolution layer includes a serial depthwise convolution layer, a pointwise convolution layer, and a batch normalization layer. The lightweight Transformer layer includes a serial 2-layer encoder, a 2-layer decoder, and a Softmax layer. The frequency domain mask generation layer includes a frequency domain meta-network and a frequency domain fully connected layer. The time domain branch network includes a MobileNet-CNN subnetwork, two TCN modules, a time domain ECANet interaction layer and a time domain mask generation layer in series. The MobileNet-CNN subnetwork is connected to the input layer, the time domain fully connected layer is connected to the time-frequency feature adaptive fusion layer, and the frequency domain ECANet interaction layer is connected to the time domain ECANet interaction layer; the time domain mask generation layer includes a time domain meta-network and a time domain fully connected layer.

2. The dual-path lightweight time-frequency domain adaptive neural network model according to claim 1, characterized in that: The depthwise separable convolution layer has 64 channels and a 5×3 convolution kernel, which performs convolution on each input channel separately. The pointwise convolution layer has a 1×1 convolution kernel, which is used to convolve the output of the depthwise convolution. The decoder in the lightweight Transformer layer consists of a 2-layer MLP network and a normalization layer. The input dimension of the MLP network is 64 and the output dimension is 32. The frequency domain meta-network in the frequency domain fully connected layer includes one convolutional layer and one RNN layer. The number of channels in the convolutional layer is 32, and the number of nodes in the RNN layer is 128.

3. The dual-path lightweight time-frequency domain adaptive neural network model according to claim 1, characterized in that: The number of channels of the MobileNet-CNN subnetwork is 64, and the convolution kernel is 5×3; The number of channels of the TCN module is 32, the convolution kernel is 5×5, and the forgetting rate is 0.

2.

4. A method for using a dual-path lightweight time-frequency domain adaptive neural network model, characterized in that: The method includes: Obtain real-time speech signals and input them into the input layer for preprocessing to obtain the original time domain signals; Use the FFT layer to perform FFT transformation on the original time domain signal to obtain the frequency domain signal; Use the depth-wise separable convolutional layer to extract the frequency domain signal to obtain the frequency domain features; Use the lightweight Transformer layer to process the frequency domain features to obtain the frequency domain feature matrix array; Use the MobileNet-CNN sub-network to extract the features of the original time domain signal to obtain the time domain features; Use two TCN modules to map the time domain features to obtain a time domain feature matrix array; The frequency domain ECANet interaction layer and the time domain ECANet interaction layer are used to perform time-frequency domain interaction on the frequency domain feature matrix array and the time domain feature matrix array to obtain frequency domain interaction results and time domain interaction results; The frequency domain element network in the frequency domain mask generation layer is used to obtain the parameters of the frequency domain dynamic activation function according to the frequency domain interaction results, and the frequency domain fully connected layer is used to obtain the frequency domain mask according to the parameters of the frequency domain dynamic activation function and the frequency domain interaction results; Multiply the frequency domain mask and the frequency domain feature to obtain the masked frequency domain feature; Performing an inverse Fourier transform on the masked frequency domain features using an IFFT layer to obtain the first time domain signal; The time domain meta-network in the time domain mask generation layer is used to obtain the parameters of the time domain dynamic activation function according to the time domain interaction results, and the time domain fully connected layer is used to obtain the time domain mask according to the parameters of the time domain dynamic activation function and the time domain interaction results; Multiplying the time domain mask and the time domain feature to obtain the second time domain signal; Performing signal fusion on the first time domain signal and the second time domain signal using a time-frequency feature adaptive fusion layer to obtain an audio signal; Use the output layer to output the audio signal.

5. The method for using the dual-path lightweight time-frequency domain adaptive neural network model according to claim 4 is characterized in that: The steps of using the FFT layer to perform FFT transformation on the original time domain signal to obtain the frequency domain signal specifically include: Where X(f) is the frequency domain signal of the current frame; f is the frequency point; x(n) is the original time domain signal, n = 0, 1, 2, ..., N, N is the number of sampling points per frame of audio; e is a constant; The steps of extracting features from frequency domain signals using a depthwise separable convolutional layer to obtain frequency domain features specifically include: F f =f DSConv (X(f)) Where, F f is the frequency domain feature, f DSConv (·) is the nonlinear transformation function of the multi-layer separable convolution stack; The steps of parallelizing the frequency domain features using the lightweight Transformer layer to obtain the frequency domain feature matrix array include: Input preparation: frequency domain feature F f Converted into sequence form As input to the lightweight Transformer layer: Where Reshape(·) is the operation of converting to sequence form; Self-attention calculation: The attention mechanism is used to model the context of the frequency domain sequence; the query, key and value vector calculation formulas are: Where Q, K, and V are the query, key, and value vectors in the lightweight Transformer layer, respectively, and W Q 、W K 、W V are the weight matrices for query, key, and value vectors respectively; Self-attention weight calculation: Use Linformer's efficient attention mechanism to reduce complexity: Where, α f is the attention weight of the frequency domain path, K T represents the transpose of K, d k is the dimension of the key vector; Apply MLP network to obtain frequency domain audio features Residual connection and normalization: To improve stability, residual connection and normalization are introduced to obtain the frequency domain feature matrix array: Where, is the frequency domain feature matrix array, Norm(·) is the normalization operation; The steps of using the frequency domain ECANet interaction layer and the time domain ECANet interaction layer to perform time-frequency domain interaction on the frequency domain feature matrix array and the time domain feature matrix array to obtain the frequency domain interaction result and the time domain interaction result specifically include: Use the frequency domain ECANet interaction layer to model the interaction between the time domain and the frequency domain: Time-frequency domain features are spliced to obtain the spliced time-frequency features F inter : Where, is the frequency domain feature matrix array, is the time domain feature matrix array; Calculate the time-frequency domain interaction attention weights: Where c ft is the global average of the time domain interaction features, H and W are the feature maps F inter height and width; is the time-frequency domain interaction attention weight, σ(·) is the sigmoid function, and Conv1D(·) is the one-dimensional convolution; Time-frequency domain interactive output: Where, It is the frequency domain interaction result output by the frequency domain path after the time-frequency domain interaction; Where F′ t_out It is the time domain interaction result output by the time domain path after the time-frequency domain interaction; The steps of obtaining the parameters of the frequency domain dynamic activation function according to the frequency domain interaction results using the frequency domain element network in the frequency domain mask generation layer, and obtaining the frequency domain mask according to the parameters of the frequency domain dynamic activation function and the frequency domain interaction results using the frequency domain fully connected layer include: Frequency domain element network input: the frequency domain interaction result of the frequency domain path output As the input of the frequency domain element network; Dynamic parameter generation: The frequency domain element network outputs the parameters θ of the frequency domain dynamic activation function; wherein the parameters θ of the frequency domain dynamic activation function include the mask amplitude adjustment coefficient α and the nonlinear response slope β; Dynamic activation calculation: Apply the parameters α and β of the frequency domain dynamic activation function to the Sigmoid function to generate a frequency domain mask: Where M(f) is the frequency domain mask of the frequency domain path, f is the frequency point, α is the frequency domain mask amplitude adjustment coefficient, β is the frequency domain nonlinear response slope, FC(·) is the frequency domain fully connected network layer, and MetaNet is the frequency domain meta-network; The specific steps of multiplying the frequency domain mask and the frequency domain feature to obtain the masked frequency domain feature include: Apply the frequency domain mask M(f) to the original frequency domain feature F f , to mask the noise in the spectrum and achieve the effect of frequency domain noise reduction; Where, It is the frequency domain feature after mask processing; The specific steps of performing an inverse Fourier transform on the masked frequency domain features using the IFFT layer to obtain the first time domain signal include: Frequency domain features after mask processing Perform inverse Fourier transform to restore the time domain signal: Where, It is the first time domain signal obtained after frequency domain denoising.

6. The method for using the dual-path lightweight time-frequency domain adaptive neural network model according to claim 5, characterized in that: The specific steps of using the MobileNet-CNN subnetwork to extract the features of the original time domain signal to obtain the time domain features include: The input is a frame of original time domain signal after preprocessing, denoted as x(t), where t represents time, t=1, 2, 3, ..., T. Extract temporal features F based on MobileNet-CNN sub-network t : F t =A(R(C(x(t)))) Where C(·) is the separable convolution calculation; R(·) is the residual connection calculation; A(·) is the HardSwish activation function of the MobileNet-CNN subnetwork; The specific steps of using two TCN modules to map the time domain features to obtain the time domain feature matrix array include: Based on two TCN modules, the time domain features are further mapped, including causal convolution, dilated convolution, residual connection and activation function; 1) For the time domain feature F t Perform causal convolution to capture hidden features in the time domain: Where, is the result of causal convolution, t is the t-th time step; Z is the size of the convolution kernel, generally 5×5 convolution kernel, that is, Z=5; p is the p-th convolution step, W p is the causal convolution weight parameter; 2) For the time domain feature F t Perform dilated convolution: Where, is the result of the dilated convolution, d is the dilation factor, which represents the interval of the convolution kernel on the input sequence, and W d is the dilated convolution weight parameter; 3) Residual connection: Add the causal convolution results, dilated convolution results and time domain features to obtain the output after residual connection: 4) Use the ReLU function to activate the output of the residual connection: Where ReLU(·) is the ReLU activation function; The specific steps of obtaining the parameters of the time domain dynamic activation function according to the time domain interaction results by using the time domain meta-network in the time domain mask generation layer, and obtaining the time domain mask according to the parameters of the time domain dynamic activation function and the time domain interaction results by using the time domain fully connected layer include: Time domain meta-network input: the time domain interaction result F′ output by the time domain path t_out As input to the time domain meta-network; Dynamic parameter generation: The time domain meta-network outputs the parameters of the time domain dynamic activation function; the parameters of the time domain dynamic activation function include the time domain mask amplitude adjustment coefficient α t and the time domain nonlinear response slope β t ; Dynamic activation calculation: The parameter α of the time domain dynamic activation function t and β t Applied to the Sigmoid function to generate a frequency domain mask: m(t)=α t ·Sigmoid(β t ·FC t (F′ t_out )) Among them, m(t) is the mask of the time domain path, α t is the time domain mask amplitude adjustment coefficient, β t The time domain is the nonlinear response slope, FC t (·) is the time domain fully connected network layer; The step of multiplying the time domain mask and the time domain feature to obtain the second time domain signal specifically includes: Apply the time domain mask m(t) to the time domain feature F t , to mask the noise in the time domain and achieve the effect of time domain noise reduction: Where, It is the second time domain signal after time domain noise reduction.

7. The method for using the dual-path lightweight time-frequency domain adaptive neural network model according to claim 6, characterized in that: The step of fusing the first time domain signal and the second time domain signal using the time-frequency feature adaptive fusion layer to obtain an audio signal includes: The time-frequency domain fusion signal is calculated by the following formula: Where, is the fused audio signal, W f is the frequency domain weight, W t is the time domain weight, λ is the fusion coefficient, and its value is 0.04.

Images