A dual-path lightweight time-frequency domain adaptive neural network model and a use method thereof

Abstract

The application belongs to the technical field of voice enhancement. The application provides a dual-path lightweight time-frequency domain adaptive neural network model and a use method thereof. The deep separable convolution layer is adopted in the embodiment of the disclosure to reduce the calculation amount of the model and reduce the noise reduction delay of the model. The lightweight Transformer layer proposed adopts an MLP network, replaces the multi-head attention with single-head attention, and greatly reduces the model parameter amount. Time-frequency domain interaction fusion is performed in the middle layer, so that the model can learn more time-frequency domain joint hidden features in the training process, and the noise reduction effect is greatly improved. By introducing a meta-learning dynamic activation function, the dynamic activation function adapts to non-stationary noise, the meta-training framework supports few-shot fine-tuning, and the deployment cost is reduced. In the process of time-frequency feature adaptive fusion, a regularization term is added, so that the model will not excessively deviate to one side of the frequency domain or the time domain during the training process, and the training effect of the model is ensured.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of speech enhancement, and in particular to a dual-path lightweight time-frequency domain adaptive neural network model and a method of using the same. BACKGROUND

[0002] In recent years, deep learning has made significant progress in the field of speech denoising. Existing speech denoising technologies mainly include traditional methods based on frequency domain processing, single-domain processing schemes based on deep learning, real-time speech processing-based denoising schemes, and speech enhancement models based on Transformers. Among them, a speech enhancement method based on a neural network is used to perform time-frequency transformation on an original speech signal to obtain an original amplitude spectrum of the original speech signal; a time-domain smoothing feature map is obtained by using a time-dimension convolution kernel to extract features from the original amplitude spectrum; a frequency-domain smoothing feature map is obtained by using a frequency-dimension convolution kernel to extract features from the original amplitude spectrum; an enhanced amplitude spectrum of the original speech signal is obtained by combining feature extraction on the original amplitude spectrum, the time-domain smoothing feature map, and the frequency-domain smoothing feature map; and an enhanced speech signal is obtained by performing time-frequency inverse transformation on the enhanced amplitude spectrum. By extracting time-frequency smoothing features from the original speech signal, two-axis denoising can be achieved on the time axis and the frequency axis, and self-learning of denoising parameters can be achieved by combining a deep neural network. However, the two-axis convolution and the deep neural network of this method result in high computational delay, which is difficult to meet real-time requirements; the fixed convolution kernel weights are difficult to adapt to complex noise changes; and the time-domain and frequency-domain features are processed independently, which may cause the model to be biased towards a certain domain.

[0003] Another speech enhancement method also includes extracting preset features in an original noisy speech signal using an encoder module; modeling and processing the preset features on a frequency axis using a dual-path module to obtain frequency-axis features, and modeling and processing the preset features on a time axis using the dual-path module to obtain time-axis features; processing the time-axis features using an excitation decoder and a vocal tract decoder to obtain a first denoised speech signal; processing the frequency-axis features using a complex spectrum decoder to obtain a second denoised speech signal; and merging the first denoised speech signal, the second denoised speech signal, and the original noisy speech signal using a merging module to obtain a final estimated speech signal. However, this method uses multiple decoders (excitation, vocal tract, and complex spectrum decoders) and a merging module, resulting in a large number of model parameters and high computational complexity; the effect on non-stationary noise is poor, and the generalization ability of the neural network is limited; and only time-frequency features are fused by branching, without explicitly modeling the interaction between the time and frequency domains.

[0004] Therefore, it is necessary to improve one or more problems in the above-mentioned related technical solutions.

[0005] It should be noted that this section is intended to provide background or context to the technical solutions of the disclosure stated in the claims. The description herein does not admit to be prior art because it is included in this section. SUMMARY

[0006] The purpose of the embodiments of the present disclosure is to provide a dual-path lightweight time-frequency domain adaptive neural network model and a method for using the same, thereby at least partially overcoming one or more problems caused by the limitations and defects of the related art.

[0007] According to a first aspect of the embodiments of the present disclosure, a dual-path lightweight time-frequency domain adaptive neural network model is provided, comprising:

[0008] An input layer, parallel frequency domain branch networks and time domain branch networks, a time-frequency feature adaptive fusion layer, and an output layer, the input layer is connected with the frequency domain branch networks and the time domain branch networks respectively, the frequency domain branch networks and the time domain branch networks are connected with the time-frequency feature adaptive fusion layer, and the time-frequency feature adaptive fusion layer is connected with the output layer; wherein,

[0009] The frequency domain branch network comprises an FFT layer, a depth separable convolution layer, a lightweight Transformer layer, a frequency domain ECANet interaction layer, a frequency domain mask generation layer, and an IFFT layer connected in series, the FFT layer is connected with the input layer, and the IFFT layer is connected with the time-frequency feature adaptive fusion layer; the depth separable convolution layer comprises a depth convolution layer, a point-wise convolution layer, and a batch normalization layer connected in series; the lightweight Transformer layer comprises two layers of encoders, two layers of decoders, and a layer of Softmax connected in series; and the frequency domain mask generation layer comprises a frequency domain element network and a frequency domain full connection layer;

[0010] The time domain branch network comprises a MobileNet-CNN subnetwork, two TCN modules, a time domain ECANet interaction layer, and a time domain mask generation layer connected in series, the MobileNet-CNN subnetwork is connected with the input layer, the time domain full connection layer is connected with the time-frequency feature adaptive fusion layer, and the frequency domain ECANet interaction layer is connected with the time domain ECANet interaction layer; and the time domain mask generation layer comprises a time domain element network and a time domain full connection layer.

[0011] Further, the depth convolution layer of the depth separable convolution layer has a channel number of 64, a convolution kernel of 5x3, and performs convolution on each input channel separately; the point-wise convolution layer has a convolution kernel of 1x1 and is used for convolution on the output of the depth convolution;

[0012] The decoder in the lightweight Transformer layer comprises two layers of MLP networks and a normalization layer, the input dimension of the MLP network is 64, and the output dimension is 32;

[0013] The frequency domain element network in the frequency domain full connection layer comprises one convolution layer and one RNN layer, the number of channels of the convolution layer is 32, and the number of nodes of the RNN layer is 128.

[0014] Further, the number of channels of the MobileNet-CNN subnetwork is 64, and the convolution kernel is 5x3.

[0015] The number of channels of the TCN module is 32, the convolution kernel is 5x5, and the forgetting rate is 0.2.

[0016] According to a second aspect of the embodiments of the present disclosure, a use method of a dual-path lightweight time-frequency domain adaptive neural network model is provided, the method comprising:

[0017] An original time domain signal is obtained by inputting a real-time voice signal to an input layer for preprocessing.

[0018] An FFT layer is used to perform FFT transformation on the original time domain signal to obtain a frequency domain signal.

[0019] A deep separable convolution layer is used to extract features of the frequency domain signal to obtain frequency domain features.

[0020] A lightweight Transformer layer is used to process the frequency domain features to obtain a frequency domain feature matrix array.

[0021] A MobileNet-CNN subnetwork is used to extract features of the original time domain signal to obtain time domain features.

[0022] Two TCN modules are used to map the time domain features to obtain a time domain feature matrix array.

[0023] A frequency domain ECANet interaction layer and a time domain ECANet interaction layer are used to perform time-frequency domain interaction on the frequency domain feature matrix array and the time domain feature matrix array to obtain a frequency domain interaction result and a time domain interaction result.

[0024] A frequency domain element network in a frequency domain mask generation layer is used to obtain parameters of a frequency domain dynamic activation function according to the frequency domain interaction result, and a frequency domain full connection layer is used to obtain a frequency domain mask according to the parameters of the frequency domain dynamic activation function and the frequency domain interaction result.

[0025] The frequency domain mask is multiplied by the frequency domain features to obtain frequency domain features processed by the mask.

[0026] An IFFT layer is used to perform inverse Fourier transformation on the frequency domain features processed by the mask to obtain the first time domain signal.

[0027] The time domain meta-network in the time domain mask generation layer obtains parameters of a time domain dynamic activation function according to the time domain interaction result, and the time domain fully connected layer obtains a time domain mask according to the parameters of the time domain dynamic activation function and the time domain interaction result;

[0028] The time domain mask is multiplied by the time domain feature to obtain the second time domain signal;

[0029] The first time domain signal and the second time domain signal are fused by using the time-frequency feature adaptive fusion layer to obtain the audio signal;

[0030] The audio signal is output by using the output layer.

[0031] Further, the step of performing FFT transformation on the original time domain signal by using the FFT layer to obtain the frequency domain signal specifically comprises:

[0032]

[0033] In the formula, X(f) is the frequency domain signal of the current frame; f is a frequency point; x(n) is the original time domain signal, n=0, 1, 2, …, N, N is the number of sampling points of each frame of audio; and e is a constant;

[0034] The step of extracting features from the frequency domain signal by using the depth separable convolution layer to obtain the frequency domain feature specifically comprises:

[0035] F f =f DSConv (X(f))

[0036] In the formula, F f is the frequency domain feature, and f DSConv (·) is a nonlinear transformation function of a multi-layer separable convolution stack;

[0037] The step of performing parallel processing on the frequency domain feature by using the lightweight Transformer layer to obtain a frequency domain feature matrix array specifically comprises:

[0038] Input preparation: convert the frequency domain feature F f into a sequence form to obtain as the input of the lightweight Transformer layer:

[0039]

[0040] In the formula, Reshape(·) is an operation of converting into a sequence form;

[0041] Self-attention calculation: the attention mechanism is used to model the context of the frequency domain sequence; wherein, the query, key and value vector calculation formula is:

[0042]

[0043] In the formula, Q, K, V are query, key, value vectors in the lightweight Transformer layer, respectively, and W Q , W K , W V are weight matrices of the query, key, value vectors, respectively.

[0044] Self-attention weight calculation: use Linformer efficient attention mechanism to reduce complexity:

[0045]

[0046] In the formula, α f is the attention weight of the frequency domain path, K T represents the transpose of K, and d k is the dimension of the key vector.

[0047] Apply the MLP network to obtain the frequency domain audio feature

[0048]

[0049] Residual connection and normalization: to improve stability, residual connection and normalization are introduced to obtain the frequency domain feature matrix array:

[0050]

[0051] In the formula, F is the frequency domain feature matrix array, and Norm(·) is the normalization operation.

[0052] The steps of using the frequency domain ECANet interaction layer and the time domain ECANet interaction layer to interact between the frequency domain feature matrix array and the time domain feature matrix array to obtain the frequency domain interaction result and the time domain interaction result include:

[0053] Model the interaction relationship between the time domain and the frequency domain using the frequency domain ECANet interaction layer:

[0054] Time-frequency domain feature splicing to obtain the spliced time-frequency feature F inter :

[0055]

[0056] In the formula, F is the frequency domain feature matrix array, is the time domain feature matrix array.

[0057] Calculate the time-frequency domain interaction attention weight:

[0058]

[0059] In the formula, c ft H represents the global average value of the temporal interaction features, and H and W represent the feature maps F, respectively. inter Height and width; σ(·) represents the time-frequency domain interactive attention weights, σ(·) represents the sigmoid function, and Conv1D(·) represents the one-dimensional convolution.

[0060] Interactive output in the time and frequency domain:

[0061]

[0062] In the formula, This represents the frequency domain interaction result output by the frequency domain path after the time-frequency domain interaction.

[0063]

[0064] In the formula, F′ t_out This represents the time-domain interaction result output by the time-domain path after the time-frequency domain interaction.

[0065] The steps of obtaining the frequency domain mask using the frequency domain meta-network in the frequency domain mask generation layer based on the frequency domain interaction results, and then using the frequency domain fully connected layer to obtain the frequency domain mask based on the parameters of the frequency domain dynamic activation function and the frequency domain interaction results, specifically include:

[0066] Frequency domain meta-network input: the frequency domain interaction result of the frequency domain path output. As input to the frequency domain meta-network;

[0067] Dynamic parameter generation: The frequency domain meta-network outputs the parameters θ of the frequency domain dynamic activation function; where the parameters θ of the frequency domain dynamic activation function include the mask amplitude adjustment coefficient α and the nonlinear response slope β;

[0068] Dynamic activation calculation: The parameters α and β of the frequency domain dynamic activation function are applied to the Sigmoid function to generate a frequency domain mask.

[0069]

[0070] In the formula, M(f) is the frequency domain mask of the frequency domain path, f is the frequency point, α is the frequency domain mask amplitude adjustment coefficient, β is the frequency domain nonlinear response slope, FC(·) is the frequency domain fully connected network layer, and MetaNet is the frequency domain meta-network.

[0071] The specific steps for multiplying the frequency domain mask by the frequency domain features to obtain the frequency domain features after masking include:

[0072] Apply the frequency domain mask M(f) to the original frequency domain feature F f This is done to mask noise in the frequency spectrum and achieve frequency domain noise reduction.

[0073]

[0074] In the formula, is the frequency domain feature after mask processing;

[0075] The specific steps for performing inverse Fourier transform on the frequency domain feature after mask processing by using the IFFT layer to obtain the first time domain signal include:

[0076] The frequency domain feature after mask processing is inverse Fourier transformed to restore it to a time domain signal:

[0077]

[0078] In the formula, is the first time domain signal obtained after frequency domain noise reduction.

[0079] Further, the specific steps for extracting the feature of the original time domain signal by using the MobileNet-CNN subnetwork to obtain the time domain feature include:

[0080] The input is a preprocessed original time domain signal, denoted as x(t), where t represents time, t = 1, 2, 3, …, T.

[0081] The time domain feature F t is extracted based on the MobileNet-CNN subnetwork:

[0082] F t = A(R(C(x(t))))

[0083] In the formula, C(·) is a separable convolution calculation; R(·) is a residual connection calculation; A(·) is a HardSwish activation function of the MobileNet-CNN subnetwork;

[0084] The specific steps for mapping the time domain feature by using two TCN modules to obtain a time domain feature matrix array include:

[0085] Based on the two TCN modules, the time domain feature is further mapped, including causal convolution, dilated convolution, residual connection, and activation function in sequence;

[0086] 1) The time domain feature F t is subjected to causal convolution to capture time domain hidden features:

[0087]

[0088] In the formula, F t (1)The result represents the causal convolution, where t is the t-th time step; Z is the kernel size, typically a 5×5 kernel, i.e., Z = 5; p is the p-th convolution step, and W... p These are the causal convolution weight parameters;

[0089] 2) For time-domain features F t Perform dilated convolution:

[0090]

[0091] In the formula, F t (2) For the dilated convolution result, d is the dilation factor, representing the spacing of the convolution kernel on the input sequence, and W is the dilation factor. d These are the parameters for dilating the convolution weights;

[0092] 3) Residual connection: The results of causal convolution, dilated convolution, and temporal features are added together to obtain the output after residual connection.

[0093]

[0094] 4) Use the ReLU function to activate the output of the residual connection result:

[0095]

[0096] In the formula, ReLU(·) is the ReLU activation function;

[0097] The specific steps for obtaining the time-domain mask using the time-domain meta-network in the time-domain mask generation layer based on the time-domain interaction results, and then using the time-domain fully connected layer to obtain the time-domain mask based on the parameters of the time-domain dynamic activation function and the time-domain interaction results, include:

[0098] Temporal meta-network input: the temporal interaction result F′ of the temporal path output. t_out As input to the temporal meta-network;

[0099] Dynamic parameter generation: The temporal meta-network outputs the parameters of the temporal dynamic activation function; among which, the parameters of the temporal dynamic activation function include the temporal mask amplitude adjustment coefficient α. t With the slope β of the time-domain nonlinear response t ;

[0100] Dynamic activation computation: The parameter α of the time-domain dynamic activation function... t With β t Applied to the Sigmoid function, it generates a frequency domain mask:

[0101] m(t)=α t ·Sigmoid(β t FC t(F′ t_out ))

[0102] wherein m(t) is a time-domain mask, α t is a time-domain mask amplitude adjustment coefficient, β t is a time-domain nonlinear response slope, FC t is a time-domain full connection network layer;

[0103] The step of multiplying the time-domain mask with the time-domain feature to obtain the second time-domain signal specifically includes:

[0104] The time-domain mask m(t) is applied to the time-domain feature F t to mask the noise in the time domain, achieving the effect of time-domain noise reduction:

[0105]

[0106] In the formula, is the second time-domain signal after time-domain noise reduction.

[0107] Further, in the step of using the time-frequency feature adaptive fusion layer to perform signal fusion on the first time-domain signal and the second time-domain signal to obtain the audio signal, the step includes:

[0108] The time-frequency domain fusion signal is calculated by the following formula:

[0109]

[0110] In the formula, is the fused audio signal, W f is a frequency domain weight, W t is a time domain weight, and λ is a fusion coefficient, taking a value of 0.04.

[0111] The technical scheme provided by the embodiments of the present disclosure can include the following beneficial effects:

[0112] In the embodiments of the present disclosure, through the above-mentioned dual-path lightweight time-frequency domain adaptive neural network model and the use method thereof, in the first aspect, by combining time-domain feature processing and frequency-domain feature processing, more efficient noise suppression can be achieved in a complex noise environment, and the signal-to-noise ratio of the speech signal is significantly improved.

[0113] In a second aspect, a lightweight neural network structure is adopted to reduce the consumption of computing resources, so that the application can realize real-time processing on resource-limited devices, and is suitable for mobile devices and embedded systems. Among them, the deep separable convolution layer can reduce the computational amount of the model, thereby reducing the noise reduction delay of the model; the lightweight Transformer layer proposed in the application adopts an MLP network, thereby replacing the multi-head attention with single-head attention to greatly reduce the model parameter amount; the application performs time-frequency domain interaction fusion in the middle layer, so that the model can learn more time-frequency domain joint hidden features in the training process, and the noise reduction effect is greatly improved; the application introduces a meta-learning dynamic activation function, which not only adapts to non-stationary noise, but also supports few-shot fine-tuning in the meta-training framework, thereby reducing the deployment cost; in the process of time-frequency feature adaptive fusion, the application adds a regularization term, so that the model will not excessively deviate to one side of the frequency domain or the time domain in the training process, thereby ensuring the training effect of the model.

[0114] In a third aspect, while removing noise, the naturalness and clarity of the speech signal are maximally preserved, speech distortion is reduced, and the user's auditory experience is improved. The use of a dynamic activation function enables the model to have high robustness to complex noise and dynamically adapt to non-stationary noise. BRIEF DESCRIPTION OF DRAWINGS

[0115] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure. It is apparent that the accompanying drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained from these drawings without creative labor for those skilled in the art.

[0116] Figure 1 FIG. 1 shows a schematic diagram of a two-path lightweight time-frequency domain adaptive neural network model in an exemplary embodiment of the present disclosure;

[0117] Figure 2 FIG. 2 shows a step diagram of a use method of a two-path lightweight time-frequency domain adaptive neural network model in an exemplary embodiment of the present disclosure;

[0118] Figure 3 FIG. 3 shows a schematic diagram of a lightweight Transformer layer in an exemplary embodiment of the present disclosure;

[0119] Figure 4 FIG. 4 shows a result comparison diagram of embodiment 1 in the present disclosure;

[0120] Figure 5 FIG. 5 shows a result comparison diagram of embodiment 2 in the present disclosure;

[0121] Figure 6 A result comparison chart showing the results of Example 3 in the present disclosure is shown;

[0122] Figure 7 A result comparison chart showing the results of Example 4 in the present disclosure is shown;

[0123] Figure 8 A result comparison chart showing the results of Example 5 in the present disclosure is shown;

[0124] Figure 9 A result comparison chart showing the results of Example 6 in the present disclosure is shown

[0125] Figure 10 A result comparison chart showing the results of Example 7 in the present disclosure is shown. DETAILED DESCRIPTION

[0126] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the concept of an example implementation to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more implementations.

[0127] In addition, the drawings are only schematic and are non-limiting. Identical components have been given the same reference numerals in the various drawings and the same reference designators have been used throughout the drawings and the detailed description for consistency and ease of understanding. Some of the blocks in the drawings are functionally related blocks, and do not necessarily have a direct correspondence to physical or logical components.

[0128] In the present example implementation, a dual-path lightweight time-frequency domain adaptive neural network model is first provided. Referring to FIG. 1, the dual-path lightweight time-frequency domain adaptive neural network model can include: Figure 1

[0129] an input layer, parallel frequency domain branch network and time domain branch network, time-frequency feature adaptive fusion layer and output layer, the input layer is connected with the frequency domain branch network and the time domain branch network respectively, the frequency domain branch network and the time domain branch network are connected with the time-frequency feature adaptive fusion layer, and the time-frequency feature adaptive fusion layer is connected with the output layer; wherein,

[0130] ​The frequency domain branch network comprises, in sequence and in series, an FFT layer, a deep separable convolution layer, a lightweight Transformer layer, a frequency domain ECANet interaction layer, a frequency domain mask generation layer and an IFFT layer, the FFT layer is connected with the input layer, and the IFFT layer is connected with the time-frequency feature adaptive fusion layer; the deep separable convolution layer comprises, in sequence and in series, a deep convolution layer, a point-by-point convolution layer and a batch normalization layer; the lightweight Transformer layer comprises, in sequence and in series, two layers of encoders, two layers of decoders and one layer of Softmax; and the frequency domain mask generation layer comprises a frequency domain element network and a frequency domain full connection layer.

[0131] The time domain branch network comprises, in sequence and in series, a MobileNet-CNN subnetwork, two TCN modules, a time domain ECANet interaction layer and a time domain mask generation layer, the MobileNet-CNN subnetwork is connected with the input layer, the time domain full connection layer is connected with the time-frequency feature adaptive fusion layer, and the frequency domain ECANet interaction layer is connected with the time domain ECANet interaction layer; and the time domain mask generation layer comprises a time domain element network and a time domain full connection layer.

[0132] Through the above-mentioned dual-path lightweight time-frequency domain adaptive neural network model, in the first aspect, by combining time domain features and frequency domain features for processing, more efficient noise suppression can be realized in a complex noise environment, and the signal-to-noise ratio of a speech signal is significantly improved.

[0133] In the second aspect, a lightweight neural network structure is adopted, the consumption of computing resources is reduced, real-time processing can be realized on a resource-limited device, and the application is suitable for mobile devices and embedded systems. The deep separable convolution layer is adopted to reduce the calculation amount of the model, thereby reducing the noise reduction delay of the model; the lightweight Transformer layer proposed in the application adopts an MLP network, thereby replacing multi-head attention with single-head attention to greatly reduce the model parameter amount; the time-frequency domain interaction fusion is performed in the middle layer, so that the model can learn more time-frequency domain joint hidden features in the training process, and the noise reduction effect is greatly improved; the meta-learning dynamic activation function is introduced, the dynamic activation function not only adapts to non-stationary noise, but also supports few-shot fine-tuning in the meta-training framework, thereby reducing the deployment cost; and in the process of time-frequency feature adaptive fusion, a regularization term is added, so that the model will not excessively deviate to one side of the frequency domain or the time domain in the training process, thereby ensuring the training effect of the model.

[0134] In the third aspect, while removing noise, the naturalness and clarity of the speech signal are maximally preserved, speech distortion is reduced, and the user's auditory experience is improved. The dynamic activation function is used, so that the model has high robustness to complex noise and can dynamically adapt to non-stationary noise.

[0135] In the following, reference will be made to Figure 1The above-mentioned two-path lightweight time-frequency domain adaptive neural network model in the present example embodiment will be described in more detail.

[0136] In one embodiment, the decoder in the lightweight Transformer layer includes 2 layers of MLP network and a normalization layer, the input dimension of the MLP network is 64, and the output dimension is 32; the frequency domain element network in the frequency domain full connection layer includes 1 layer of convolution layer and 1 layer of RNN layer, the channel number of the convolution layer is 32, and the node number of the RNN layer is 128.

[0137] In one embodiment, the channel number of the MobileNet-CNN subnetwork is 64, and the convolution kernel is 5x3; the channel number of the TCN module is 32, the convolution kernel is 5x5, and the forgetting rate is 0.2.

[0138] Further, the present example embodiment also provides a use method of the two-path lightweight time-frequency domain adaptive neural network model. Referring to Figure 2 the use method of the two-path lightweight time-frequency domain adaptive neural network model can include:

[0139] Step S101: obtaining a real-time speech signal and inputting it to an input layer for preprocessing to obtain an original time domain signal;

[0140] Step S102: performing FFT transformation on the original time domain signal by using an FFT layer to obtain a frequency domain signal;

[0141] Step S103: performing feature extraction on the frequency domain signal by using a depth separable convolution layer to obtain a frequency domain feature;

[0142] Step S104: processing the frequency domain feature by using a lightweight Transformer layer to obtain a frequency domain feature matrix array;

[0143] Step S105: extracting features of the original time domain signal by using a MobileNet-CNN subnetwork to obtain a time domain feature;

[0144] Step S106: mapping the time domain feature by using two TCN modules to obtain a time domain feature matrix array;

[0145] Step S107: performing time-frequency domain interaction on the frequency domain feature matrix array and the time domain feature matrix array by using a frequency domain ECANet interaction layer and a time domain ECANet interaction layer to obtain a frequency domain interaction result and a time domain interaction result;

[0146] Step S108: obtaining parameters of a frequency domain dynamic activation function according to the frequency domain interaction result by using a frequency domain element network in a frequency domain mask generation layer, and obtaining a frequency domain mask according to the parameters of the frequency domain dynamic activation function and the frequency domain interaction result by using a frequency domain full connection layer.

[0147] Step S109: multiplying the frequency domain mask with the frequency domain feature to obtain a mask-processed frequency domain feature;

[0148] Step S110: performing inverse Fourier transform on the mask-processed frequency domain feature by using an IFFT layer to obtain the first time domain signal;

[0149] Step S111: obtaining a time domain dynamic activation function parameter according to the time domain interaction result by using a time domain meta network in the time domain mask generation layer, and obtaining a time domain mask according to the time domain dynamic activation function parameter and the time domain interaction result by using a time domain full connection layer;

[0150] Step S112: multiplying the time domain mask with the time domain feature to obtain the second time domain signal;

[0151] Step S113: performing signal fusion on the first time domain signal and the second time domain signal by using a time-frequency feature adaptive fusion layer to obtain an audio signal;

[0152] Step S114: outputting the audio signal by using an output layer.

[0153] In step S101, a real-time speech signal is obtained and input to an input layer for preprocessing to obtain an original time domain signal.

[0154] Specifically, 1. Speech signal collection:

[0155] A real-time speech signal is obtained by using a microphone or other audio collection device, and the collected real-time speech signal is converted into a digital signal through AD analog-digital conversion;

[0156] 2. Feature standardization:

[0157] The z-score normalization method is used to standardize the digital signal to eliminate the influence of different signal amplitudes.

[0158] 3. Feature framing:

[0159] The standardized digital signal is subjected to framing processing to obtain an original time domain signal, each frame of data being 16 ms (16 KHz sampling rate), and an overlap framing technique is used to ensure smooth transition between frames, with an overlap rate of 50%, i.e., 8 ms of frame overlap.

[0160] In step S102, the original time domain signal is subjected to FFT transformation by using an FFT layer to obtain a frequency domain signal.

[0161] Specifically, the original time domain signal is subjected to FFT transformation:

[0162]

[0163] wherein X(f) is a frequency domain signal of a current frame; f is a frequency point; x(n) is an original time domain signal, n = 0, 1, 2, …, N, N is the number of sampling points of each frame of audio; and e is a constant.

[0164] In step S103, a deep separable convolution layer is used to extract features from the frequency domain signal to obtain a frequency domain feature.

[0165] Specifically, a deep separable convolution layer (Depthwise Separable Convolution) is used to extract features from the frequency domain amplitude spectrum X(f):

[0166] F f = f DSConv (X(f))

[0167] wherein F f is a frequency domain feature, f DSConv (·) is a nonlinear transformation function of a multi-layer separable convolution stack. The implementation steps include deep convolution, point-by-point convolution, batch normalization, and activation:

[0168] 1) Deep convolution: the number of channels is 64, the convolution kernel is 5x3, and convolution is performed on each input channel separately.

[0169] 2) Point-by-point convolution: a 1x1 convolution kernel is used to convolve the output of the deep convolution.

[0170] 3) Batch normalization and activation: the convolution result is normalized, and a ReLU function is used for nonlinear activation to obtain the final frequency domain feature F f .

[0171] The deep separable convolution neural network (ds-cnn) layer is used to reduce the computational load of the model and thus reduce the noise reduction delay of the model.

[0172] In step S104, a lightweight Transformer layer is used to perform parallel processing on the frequency domain feature to obtain a frequency domain feature matrix array.

[0173] Specifically, the lightweight Transform layer is composed of 2 layers of encoder and 2 layers of decoder and one layer of Softmax. The multi-head attention module in each decoder layer is replaced by a 2-layer MLP network to reduce the model parameter amount. The input dimension is 64 and the output dimension is 32.

[0174] 1) Input preparation: convert the frequency domain feature F f into a sequence form to obtain as the input of the lightweight Transformer layer:

[0175]

[0176] In the formula, Reshape(·) is an operation of converting to a sequence form.

[0177] 2) Self-attention calculation: using attention mechanism to model the context of the frequency domain sequence; wherein the query, key and value vector calculation formula:

[0178]

[0179] In the formula, Q, K, V are respectively the query, key and value vectors in the lightweight Transformer layer, W Q , W K , W V are respectively the weight matrices of the query, key and value vectors.

[0180] 3) Self-attention weight calculation: using Linformer efficient attention mechanism to reduce complexity:

[0181]

[0182] In the formula, α f is the attention weight of the frequency domain path, K T represents the transpose of K, d k is the dimension of the key vector.

[0183] 4) Application of MLP network to obtain frequency domain audio features

[0184]

[0185] In the formula, MLP(·) represents an MLP multi-layer perceptron network, the input node is 64, the output node is 32, and the output layer is 2 layers.

[0186] 5) Residual connection and normalization: to improve stability, residual connection and normalization are introduced to obtain a frequency domain feature matrix array:

[0187]

[0188] In the formula, is the frequency domain feature matrix array, and Norm(·) is a normalization operation.

[0189] Since the existing Transformer layer has a large amount of calculation, it will affect the running efficiency of the model, and it is difficult to run in real time on the edge side, therefore, the present application proposes to replace the multi-head attention mechanism of the existing Transformer layer with an MLP multi-layer perceptron (2-layer MLP structure is used in the present application) to replace the multi-head attention with single-head attention, so as to greatly reduce the model parameter amount. The implementation manner is as follows:Figure 3 as shown.

[0190] By using the improved lightweight attention module, the model parameter quantity is greatly reduced, so that the model can be deployed on low-resource hardware.

[0191] In step S107, the frequency domain feature matrix array and the time domain feature matrix array are time-frequency domain interaction by using the frequency domain ECANet interaction layer and the time domain ECANet interaction layer, to obtain the frequency domain interaction result and the time domain interaction result.

[0192] Specifically, after the lightweight Transformer output, the frequency domain ECANet interaction layer is used to model the interaction relationship between the time domain and the frequency domain:

[0193] 1) Time-frequency domain feature splicing, to obtain the spliced time-frequency feature F inter :

[0194]

[0195] In the formula, is the frequency domain feature matrix array, is the time domain feature matrix array.

[0196] 2) Calculate the time-frequency domain interaction attention weight:

[0197]

[0198] In the formula, c ft is the global average value of the time domain interaction feature; H and W are the height and width of the feature map F inter respectively; is the time-frequency domain interaction attention weight; σ(·) is the sigmoid function; Conv1D(·) is a one-dimensional convolution with 32 channels, a convolution kernel of 5x3, and a forgetting rate of 0.3.

[0199] 3) Time-frequency domain interaction output:

[0200]

[0201] In the formula, is the frequency domain interaction result output by the frequency domain path after the time-frequency domain interaction.

[0202]

[0203] In the formula, F′ t_out is the time domain interaction result output by the time domain path after the time-frequency domain interaction.

[0204] The common time-frequency domain dual-path network structure often only performs fusion at the output layer in the time-frequency domain, and the method performs time-frequency domain interactive fusion at the middle layer of the network, so that the model can learn more time-frequency domain joint hidden features in the training process.

[0205] In step S108, the frequency domain meta network in the frequency domain mask generation layer obtains the parameters of the frequency domain dynamic activation function according to the frequency domain interaction result, and obtains the frequency domain mask by using the frequency domain full connection layer according to the parameters of the frequency domain dynamic activation function and the frequency domain interaction result.

[0206] Specifically, the mask is obtained by using the dynamic activation function, and the dynamic activation function realizes parameter self-adaptation through the meta network:

[0207] 1) Frequency domain meta network input: the frequency domain interaction result output by the frequency domain path as the input of the frequency domain meta network;

[0208] 2) Dynamic parameter generation: the frequency domain meta network outputs the parameters θ of the frequency domain dynamic activation function; wherein the parameters θ of the frequency domain dynamic activation function include a mask amplitude adjustment coefficient α and a nonlinear response slope β;

[0209] 3) Dynamic activation calculation: apply the parameters α and β of the frequency domain dynamic activation function to the Sigmoid function to generate the frequency domain mask:

[0210]

[0211] In the formula, M(f) is the frequency domain mask of the frequency domain path, f is the frequency point, α is the frequency domain mask amplitude adjustment coefficient, β is the frequency domain nonlinear response slope, FC(·) is the frequency domain full connection network layer. MetaNet is the frequency domain meta network, including 1 layer of convolution layer (channel number 32) + 1 layer of RNN layer (node number 128).

[0212] It can be understood that the environment is self-adaptive: compared with the fixed function, the meta-learning dynamic activation can automatically adjust the parameters according to the noise type (such as sudden knocking sound vs. continuous engine sound), and realize targeted suppression.

[0213] Edge efficiency: MetaNet only needs 0.05M parameters, and the delay on RV1126 chip is less than 0.3ms.

[0214] Few sample adaptation: through the meta-training framework, the model only needs 10 target scene speeches to fine-tune, and adapts to factory, vehicle-mounted and other vertical scenes.

[0215] In step S109, the frequency domain mask is multiplied by the frequency domain feature to obtain the frequency domain feature processed by the mask.

[0216] Specifically, the frequency domain mask M(f) is applied to the original frequency domain feature F f to mask the noise in the spectrum, achieving the effect of frequency domain noise reduction.

[0217]

[0218] In the formula, is the frequency domain feature after mask processing;

[0219] In step S110, the mask-processed frequency domain feature is inverse Fourier transformed by using the IFFT layer to obtain the first time domain signal.

[0220] Specifically, the mask-processed frequency domain feature is inverse Fourier transformed to restore it to a time domain signal:

[0221]

[0222] In the formula, is the first time domain signal obtained after frequency domain noise reduction.

[0223] In step S105, the MobileNet-CNN subnetwork is used to extract the features of the original time domain signal to obtain the time domain features.

[0224] Specifically, the time domain feature extraction (based on the MobileNet-CNN subnetwork, the number of channels of the MobileNet-CNN subnetwork is 64, and the convolution kernel is 5x3):

[0225] The input is a pre-processed original time domain signal, denoted as x(t), where t represents time, t = 1, 2, 3, …, T.

[0226] The MobileNet-CNN subnetwork is used to extract the time domain features F t :

[0227] F t = A(R(C(x(t))))

[0228] In the formula, C(·) is a separable convolution calculation; R(·) is a residual connection calculation; A(·) is the HardSwish activation function of the MobileNet-CNN subnetwork;

[0229] In step S106, two TCN modules are used to map the time domain features to obtain a time domain feature matrix array.

[0230] Specifically, the number of channels of each layer of the TCN network is 32, the convolution kernel is 5x5, and the forgetting rate is 0.2.

[0231] Based on the two TCN modules, the time domain features are further mapped, including causal convolution, dilated convolution, residual connection and activation function in turn;

[0232] 1) The time domain feature F t is subjected to causal convolution to capture the time domain hidden feature:

[0233]

[0234] In the formula, F t (1) is the causal convolution result, t is the tth time step; Z is the convolution kernel size, generally 5*5 convolution kernel, that is, Z=5; p is the pth convolution step, W p is the causal convolution weight parameter;

[0235] 2) The time domain feature F t is subjected to dilated convolution:

[0236]

[0237] In the formula, F t (2) is the dilated convolution result, d is the dilation factor, which represents the interval of the convolution kernel on the input sequence, W d is the dilated convolution weight parameter;

[0238] 3) Residual connection: the causal convolution result, the dilated convolution result and the time domain feature are added to obtain the output after residual connection:

[0239]

[0240] 4) The output result after residual connection is activated using the ReLU function:

[0241]

[0242] In the formula, ReLU(·) is the ReLU activation function;

[0243] In step S111, the time domain meta network in the time domain mask generation layer obtains the parameters of the time domain dynamic activation function according to the time domain interaction result, and obtains the time domain mask using the time domain full connection layer according to the parameters of the time domain dynamic activation function and the time domain interaction result.

[0244] Specifically, the mask is obtained using the dynamic activation function, and the dynamic activation function realizes parameter self-adaptation through the meta network:

[0245] 1) Time domain meta network input: the time domain interaction result F′ t_out output by the time domain path is taken as the input of the time domain meta network;

[0246] 2) Dynamic parameter generation: the time domain meta network outputs the parameters of the time domain dynamic activation function; wherein the parameters of the time domain dynamic activation function include a time domain mask amplitude adjustment coefficient α t and a time domain nonlinear response slope β t ;

[0247] 3) Dynamic activation calculation: the parameters α t and β t of the time domain dynamic activation function are applied to the Sigmoid function to generate a frequency domain mask:

[0248] m(t) = α t · Sigmoid(β t · FC t (F′ t_out ))

[0249] wherein m(t) is a mask of a time domain path; α t is a time domain mask amplitude adjustment coefficient, β t is a time domain nonlinear response slope, and the mask calculation manner of the frequency domain path is the same, which is calculated by a frequency domain meta network, the frequency domain meta network is composed of a 1D convolution layer and a 1-layer RNN layer; FC t (·) is a time domain full connection network layer.

[0250] In step S112, the time domain mask is multiplied by the time domain feature to obtain the second time domain signal.

[0251] Specifically, the time domain mask m(t) is applied to the time domain feature F t to mask the noise in the time domain, achieving the effect of time domain noise reduction.

[0252]

[0253] In the formula, is the second time domain signal after time domain noise reduction.

[0254] In step S113, the first time domain signal and the second time domain signal are fused by using a time-frequency feature adaptive fusion layer to obtain an audio signal.

[0255] Specifically, the purpose of the adaptive fusion layer is to combine the signals processed in the frequency domain and the time domain, dynamically adjust their weights, and achieve optimal signal fusion. The following is a detailed implementation process:

[0256] Input signal:

[0257] Signal after frequency domain processing:

[0258] Signal after time domain processing:

[0259] Signal fusion:

[0260] The time-frequency domain fusion signal is calculated by the following formula:

[0261]

[0262] In the formula, is the fused audio signal, W f is the frequency domain weight, W t is the time domain weight, and λ is the fusion coefficient, which is 0.04.

[0263] In addition, the addition of in the formula is equivalent to adding a regularization term to the model, which can prevent the model from excessively deviating to one side of the frequency domain or the time domain during training, thereby ensuring the training effect of the model.

[0264] In step S114, the output layer outputs an audio signal.

[0265] Specifically, the fused noise-reduced audio signal is converted to digital-analog after DA and output to earphones or loudspeakers.

[0266] In one specific embodiment, the time-frequency domain dual-path enhancement neural network is trained.

[0267] Specifically, an open source dataset is used, the speech signal uses the librisspeech corpus, the noise signal uses the Audioset corpus, the speech-noise mixed signal is created according to the -5~25db signal-to-noise ratio range, and the training set is divided according to the 80% training set and 20% test set ratio. The loss function uses the mean square error function. The learning rate used by the Adam optimizer is 10 -3 If the loss of the validation set does not improve for three consecutive stages, the learning rate is halved.

[0268] Parameter settings:

[0269] Speech input: 16ms, 16kHz; FFT frequency point setting: 512;

[0270] Frequency domain path parameters: DS-CNN deep convolution: channel number 64, convolution kernel 5x3, DS-CNN point-by-point convolution: channel number 64, using 1x1 convolution kernel; Lightweight Transform layer parameters: input dimension 64, output dimension 32, encoder layer number: 2 layers, decoder layer number 2 layers; Single-head attention MLP layer number: 2 layers; 1D convolution parameters of the time-frequency domain interaction layer: channel number 32, convolution kernel 5x3, forgetting rate 0.3; Mask calculation layer meta-network: 1 convolution layer (channel number 32) + 1 RNN layer (node number 128).

[0271] Temporal path parameters: MobileNet-CNN network: channel number 64, convolution kernel 5x3; double TCN module: 2 layers of TCN network, channel number 32, convolution kernel: 5x5, forgetting rate 0.2.

[0272] Time-frequency domain fusion layer: fusion coefficient value 0.04.

[0273] For edge device deployment, the frequency domain path and the time domain path are set to parallel computing, which can greatly improve the computing efficiency. After the model is quantized to INT8, it is deployed in RV1126 chip (1GHz frequency), and the operation delay is only 6.5ms, which meets the real-time communication requirements of low-resource edge, and the noise suppression can reach more than 30db.

[0274] Table 1 Comparison table of various noise reduction methods

[0275]

[0276] In embodiment 1, as shown in Figure 4 , it is an audio example graph after noise reduction of -20db noise audio by using the edge device-oriented time-frequency domain joint modeling speech noise reduction method of the present application.

[0277] In embodiment 2, as shown in Figure 5 , it is an audio example graph after noise reduction of environmental recording by using the edge device-oriented time-frequency domain joint modeling speech noise reduction method of the present application.

[0278] As can be seen, the human voice is well preserved and there is no damage to the speech, and the noise is completely eliminated.

[0279] In embodiment 3, as shown in Figure 6 , it is a comparison example graph of noise-free audio after noise reduction of noise-added noise-free audio by using the edge device-oriented time-frequency domain joint modeling speech noise reduction method of the present application. As can be seen, in the energy graph of "noise-added audio", the "horizontal line" at the bottom of the picture represents the main noise, and in the energy graph of "noise-reduced audio", it can be seen that the noise is eliminated. Conclusion: noise removal is obvious, human voice is well preserved, and the effect is excellent.

[0280] In embodiment 4, as shown in Figure 7 , it is an audio example graph after noise reduction of public place non-stable environmental noise by using the edge device-oriented time-frequency domain joint modeling speech noise reduction method of the present application. As can be seen from the time domain waveform in the box, the noise is completely filtered out.

[0281] In embodiment 5, as shown in Figure 8As shown in the figure, it is an audio example after the non-steady-state music noise is denoised by the edge device-oriented time-frequency domain joint modeling speech denoising method of the present application. As can be seen from the time domain waveform in the upper box, for music, the present application can still well filter out the non-steady-state noise.

[0282] In embodiment 6, as shown in the figure, it is an audio example after the non-steady-state burst handclap noise is denoised by the edge device-oriented time-frequency domain joint modeling speech denoising method of the present application. As can be seen from the waveform graph, for the burst noise of “handclap”, the burst noise is also filtered out after the processing of the present application. Figure 9

[0283] In embodiment 7, as shown in the figure, it is an audio example after the air conditioner noise is denoised by the edge device-oriented time-frequency domain joint modeling speech denoising method of the present application. As can be seen from the waveform graph, for the steady-state sound of “air conditioner noise”, the present application can well filter out the noise and restore the human voice. Figure 10

[0284] Further, the performance of the lightweight Transformer module is experimentally verified, and the comparison results are shown in Table 2.

[0285] Table 2 Comparison results of Transformer module performance

[0286]

[0287] As can be seen, the improved lightweight attention module proposed in the present application reduces the model parameter amount by 67%, while the signal-to-noise ratio is only reduced by 4.6%, and the speech intelligibility PESQ is only reduced by 5.8%.

[0288] Further, the frequency domain ECANet interaction layer is experimentally verified and compared with data, and the results are shown in Table 3.

[0289] Table 3 Comparison results of frequency domain ECANet interaction layer

[0290] Method comparison SISNR (dB) PESQ Separate in time and frequency domain 17.96 2.81 Add interactive layer in time and frequency domain 19.87 2.94

[0291] As can be seen, the addition of the interaction layer in the network structure of each branch in the time-frequency domain improves the denoising effect of the model.

[0292] Further, the performance of the fixed mask activation and the meta-learning dynamic activation is compared, and the results are shown in Table 4.

[0293] Table 4 Comparison results of fixed mask activation and meta-learning dynamic activation

[0294] Noise reduction method Parameter amount (M) SISNR (dB) PESQ Fixed mask activation 0.81 18.24 2.67 Meta-learning dynamic activation 0.82 19.87 2.94

[0295] ​​Edge deployment verification

[0296] Hardware platform: RV1126 chip (1GHz frequency, INT8 quantization);

[0297] Real-time performance: the total delay increased from 6.2ms to 6.5ms, still meeting the edge real-time requirements;

[0298] Energy efficiency ratio: power consumption increased by 0.01W, within an acceptable range.

[0299] As can be seen, by introducing meta-learning dynamic activation function, the following advantages can be achieved:

[0300] 1. Robustness to complex noise: dynamic parameters adapt to non-stationary noise, and SISNR is improved by 1.63db;

[0301] 2. Rapid adaptation to scenarios: the meta-training framework supports few-shot fine-tuning, reducing deployment costs;

[0302] 3. Edge friendliness: lightweight design ensures real-time performance, and adapts to low-power chips such as RV1126.

[0303] Through the above dual-path lightweight time-frequency domain adaptive neural network model and its use method, in the first aspect, by combining time domain feature and frequency domain feature processing, more efficient noise suppression can be realized in complex noise environment, and the signal-to-noise ratio of the speech signal is significantly improved.

[0304] In the second aspect, a lightweight neural network structure is adopted, which reduces the consumption of computing resources, so that the present application can realize real-time processing on resource-limited devices, and is suitable for mobile devices and embedded systems. Among them, the use of depth separable convolution layer can reduce the computational amount of the model, thereby reducing the noise reduction delay of the model; the lightweight Transformer layer proposed in the present application adopts an MLP network, thereby replacing the multi-head attention with a single-head attention to greatly reduce the model parameter amount; the present application performs time-frequency domain interactive fusion in the middle layer, so that the model can learn more time-frequency domain joint hidden features in the training process, and the noise reduction effect is greatly improved; the present application introduces a meta-learning dynamic activation function, which not only adapts to non-stationary noise, but also supports few-shot fine-tuning in the meta-training framework, reducing deployment costs; in the process of time-frequency feature adaptive fusion, the present application adds a regularization term, which can prevent the model from being biased too much to one side of the frequency domain or the time domain during the training process, thereby ensuring the training effect of the model.

[0305] In the third aspect, while removing noise, the naturalness and clarity of the speech signal are maximally preserved, the speech distortion is reduced, and the user's auditory experience is improved. The use of dynamic activation function makes the model have high robustness to complex noise and can dynamically adapt to non-stationary noise.

[0306] In addition, the processing delay is reduced by the application, the real-time voice communication and interaction on the embedded low-resource chip is met, and the practicability of the system is improved.

[0307] In the description of the present specification, the description referring to the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in the specification.

[0308] Other embodiments of the present disclosure will be apparent to those skilled in the art upon consideration of the specification and practice of the applications disclosed. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure following the general principles of the present disclosure and including common knowledge or conventional techniques in the art not disclosed by the present disclosure. The specification and examples are only considered as exemplary, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A method for using a dual-path lightweight time-frequency domain adaptive neural network model, characterized in that, The method includes: A dual-path lightweight time-frequency domain adaptive neural network model is constructed. The dual-path lightweight time-frequency domain adaptive neural network model includes an input layer, parallel frequency domain branch networks and time domain branch networks, a time-frequency feature adaptive fusion layer, and an output layer. The input layer is connected to both the frequency domain branch networks and the time domain branch networks. Both the frequency domain branch networks and the time domain branch networks are connected to the time-frequency feature adaptive fusion layer. The time-frequency feature adaptive fusion layer is connected to the output layer. The frequency domain branch network consists of sequentially arranged FFT layers, depthwise separable convolutional layers, lightweight Transformer layers, frequency domain ECANet interaction layers, frequency domain mask generation layers, and IFFT layers. The FFT layers are connected to the input layer, and the IFFT layers are connected to the time-frequency feature adaptive fusion layer. The depthwise separable convolutional layers consist of sequentially arranged depthwise convolutional layers, pointwise convolutional layers, and batch normalization layers. The lightweight Transformer layers consist of sequentially arranged 2-layer encoders, 2-layer decoders, and a Softmax layer. The frequency domain mask generation layer consists of a frequency domain meta-network and a frequency domain fully connected layer. The temporal branch network consists of a MobileNet-CNN sub-network, two TCN modules, a temporal ECANet interaction layer, and a temporal mask generation layer, which are connected sequentially. The MobileNet-CNN sub-network is connected to the input layer, the temporal fully connected layer is connected to the time-frequency feature adaptive fusion layer, and the frequency-domain ECANet interaction layer is connected to the temporal ECANet interaction layer. The temporal mask generation layer consists of a temporal meta-network and a temporal fully connected layer. The real-time speech signal is acquired and input into the input layer for preprocessing to obtain the original time-domain signal; The original time-domain signal is transformed using an FFT layer to obtain the frequency-domain signal; Depth-separable convolutional layers are used to extract features from frequency domain signals to obtain frequency domain features; A lightweight Transformer layer is used to process the frequency domain features to obtain an array of frequency domain feature matrices; The features of the original time-domain signal are extracted using the MobileNet-CNN subnetwork to obtain the time-domain features; Two TCN modules are used to map the time-domain features to obtain a time-domain feature matrix array; The frequency domain ECANet interaction layer and the time domain ECANet interaction layer are used to perform time-frequency domain interaction on the frequency domain feature matrix array and the time domain feature matrix array to obtain the frequency domain interaction result and the time domain interaction result; The parameters of the frequency domain dynamic activation function are obtained by using the frequency domain meta-network in the frequency domain mask generation layer based on the frequency domain interaction results, and the frequency domain mask is obtained by using the frequency domain fully connected layer based on the parameters of the frequency domain dynamic activation function and the frequency domain interaction results. Multiply the frequency domain mask by the frequency domain features to obtain the frequency domain features after masking. The frequency domain features after masking are subjected to inverse Fourier transform using an IFFT layer to obtain the first time domain signal. The parameters of the temporal dynamic activation function are obtained by using the temporal meta-network in the temporal mask generation layer based on the temporal interaction results, and the temporal mask is obtained by using the temporal fully connected layer based on the parameters of the temporal dynamic activation function and the temporal interaction results. Multiply the time-domain mask by the time-domain feature to obtain the second time-domain signal; An adaptive fusion layer based on time-frequency features is used to fuse the first time-domain signal and the second time-domain signal to obtain an audio signal; The audio signal is output using the output layer.

2. The method of using the dual-path lightweight time-frequency domain adaptive neural network model according to claim 1, characterized in that, The depthwise separable convolutional layer has 64 channels and a 5×3 kernel, performing convolution on each input channel individually; the pointwise convolutional layer has a 1×1 kernel, used to convolve the output of the depthwise convolution. The decoder in the lightweight Transformer layer consists of two MLP network layers and a normalization layer. The input dimension of the MLP network is 64 and the output dimension is 32. The frequency domain meta-network in the frequency domain fully connected layer consists of one convolutional layer and one RNN layer. The convolutional layer has 32 channels, and the RNN layer has 128 nodes.

3. The method of using the dual-path lightweight time-frequency domain adaptive neural network model according to claim 2, characterized in that, The MobileNet-CNN subnetwork has 64 channels and 5×3 convolutional kernels; The TCN module has 32 channels, a 5×5 convolution kernel, and a forgetting rate of 0.

2.

4. The method of using the dual-path lightweight time-frequency domain adaptive neural network model according to claim 3, characterized in that, The specific steps of using an FFT layer to perform an FFT transform on the original time-domain signal to obtain a frequency-domain signal include: In the formula, This is the frequency domain signal of the current frame; Frequency point; The original time-domain signal is n=0, 1, 2, ..., N, where N is the number of sampling points per frame of audio. It is a constant; The specific steps for extracting features from a frequency domain signal using depthwise separable convolutional layers to obtain frequency domain features include: In the formula, For frequency domain characteristics, It is a nonlinear transformation function for multiple layers of separable convolution stacks; The steps for parallelizing the frequency domain features using a lightweight Transformer layer to obtain an array of frequency domain feature matrices specifically include: Input preparation: frequency domain features Convert to sequence form As input to the lightweight Transformer layer: In the formula, This is an operation to convert the data into sequence form; Self-attention computation: This employs an attention mechanism to model the context of frequency domain sequences; the formulas for calculating query, key, and value vectors are as follows: In the formula, Q, K, and V are the query, key, and value vectors in the lightweight Transformer layer, respectively. , , These are the weight matrices for the query, key, and value vectors, respectively. Self-attention weight calculation: Using the efficient attention mechanism of Linformer to reduce complexity: In the formula, For the attention weights of the frequency domain path, This represents the transpose of K. The dimension of the key vector; Frequency domain audio features obtained by applying MLP networks : Residual connection and normalization: To improve stability, residual connection and normalization are introduced to obtain the frequency domain feature matrix array: In the formula, It is an array of frequency domain feature matrices. This is a normalization operation; The specific steps for performing time-frequency domain interaction on the frequency domain feature matrix array and the time domain feature matrix array using the frequency domain ECANet interaction layer and the time domain ECANet interaction layer to obtain the frequency domain interaction result and the time domain interaction result include: The interaction relationship between the time domain and the frequency domain is modeled using a frequency domain ECANet interaction layer: Time-frequency domain features are concatenated to obtain the concatenated time-frequency features. : In the formula, It is an array of frequency domain feature matrices. It is an array of time-domain feature matrices; Calculate the time-frequency domain interactive attention weighting: In the formula, The global average value of the temporal interaction features. and Feature maps Height and width; For time-frequency domain interaction attention weights, For the sigmoid function, It is a one-dimensional convolution; Interactive output in the time and frequency domain: In the formula, This represents the frequency domain interaction result output by the frequency domain path after the time-frequency domain interaction. In the formula, This represents the time-domain interaction result output by the time-domain path after the time-frequency domain interaction. The steps of obtaining the frequency domain mask using the frequency domain meta-network in the frequency domain mask generation layer based on the frequency domain interaction results, and then using the frequency domain fully connected layer to obtain the frequency domain mask based on the parameters of the frequency domain dynamic activation function and the frequency domain interaction results, specifically include: Frequency domain meta-network input: the frequency domain interaction result of the frequency domain path output. As input to the frequency domain meta-network; Dynamic parameter generation: Parameters of the frequency domain dynamic activation function output by the frequency domain meta-network. Among them, the parameters of the frequency domain dynamic activation function Including mask amplitude adjustment coefficient With nonlinear response slope ; Dynamic activation calculation: The parameters of the frequency domain dynamic activation function... and Applied to the Sigmoid function, it generates a frequency domain mask: In the formula, For the frequency domain path, For frequency points, This is the frequency domain mask amplitude adjustment coefficient. The slope of the frequency domain nonlinear response. It is a frequency-domain fully connected network layer. It is a frequency domain meta-network; The specific steps for multiplying the frequency domain mask by the frequency domain features to obtain the frequency domain features after masking include: frequency domain mask Applied to the original frequency domain features This is done to mask noise in the frequency spectrum and achieve frequency domain noise reduction. In the formula, These are the frequency domain features after masking. The specific steps for performing an inverse Fourier transform on the frequency domain features after masking using an IFFT layer to obtain the first time-domain signal include: Frequency domain features after masking Perform an inverse Fourier transform to recover the time-domain signal: In the formula, It is the first time-domain signal obtained after frequency-domain noise reduction.

5. The method of using the dual-path lightweight time-frequency domain adaptive neural network model according to claim 4, characterized in that, The specific steps for extracting features from the original time-domain signal using the MobileNet-CNN subnetwork to obtain time-domain features include: The input is a preprocessed frame of the original time-domain signal, denoted as . Where t represents time, t = 1, 2, 3, ..., T; Temporal features are extracted based on MobileNet-CNN subnetwork. : In the formula, For separable convolution computation; Calculate the residual connection; A The HardSwish activation function for the MobileNet-CNN subnetwork; The specific steps for mapping temporal features using two TCN modules to obtain a temporal feature matrix array include: Based on two TCN modules, the temporal features are further mapped, including causal convolution, dilated convolution, residual connection and activation function in sequence. 1) Regarding time-domain characteristics Perform causal convolution to capture temporal hidden features: In the formula, For the result of causal convolution, For the first One time step; Z Z represents the kernel size, typically a 5×5 kernel, i.e., Z=5; p represents the p-th convolution step. These are the causal convolution weight parameters; 2) Time-domain characteristics Perform dilated convolution: In the formula, To dilate the convolution result, d The dilation factor represents the spacing of the convolution kernel on the input sequence. These are the parameters for dilating the convolution weights; 3) Residual connection: The results of causal convolution and dilated convolution are added to the temporal features to obtain the output after residual connection. 4) Use the ReLU function to activate the output of the residual connection result: In the formula, It is the ReLU activation function; The specific steps for obtaining the time-domain mask using the time-domain meta-network in the time-domain mask generation layer based on the time-domain interaction results, and then using the time-domain fully connected layer to obtain the time-domain mask based on the parameters of the time-domain dynamic activation function and the time-domain interaction results, include: Temporal meta-network input: Temporal interaction results of the temporal path output As input to the temporal meta-network; Dynamic parameter generation: The temporal meta-network outputs the parameters of the temporal dynamic activation function; among which, the parameters of the temporal dynamic activation function include the temporal mask amplitude adjustment coefficient. With the slope of the time-domain nonlinear response ; Dynamic activation computation: The parameters of the time-domain dynamic activation function... and Applied to the Sigmoid function, it generates a frequency domain mask: in, For the time-domain path mask, This is the time-domain mask amplitude adjustment coefficient. The slope of the nonlinear response in the time domain. It is a time-domain fully connected network layer; The steps of multiplying the time-domain mask by the time-domain features to obtain the second time-domain signal specifically include: Time domain mask Applied to time domain features This is done to mask noise in the time domain, achieving the effect of time-domain noise reduction. In the formula, It is the second time-domain signal after time-domain noise reduction.

6. The method of using the dual-path lightweight time-frequency domain adaptive neural network model according to claim 5, characterized in that, The step of fusing the first time-domain signal and the second time-domain signal using a time-frequency feature adaptive fusion layer to obtain an audio signal includes: The time-frequency domain fused signal is calculated using the following formula: In the formula, The fused audio signal. For frequency domain weights, For time domain weights, This is the fusion coefficient, with a value of 0.04.

Images