KAN convolution improvement-based DCU-Net intelligent sound box speech recognition and noise reduction method

Abstract

According to the KAN convolution improvement-based DCU-Net intelligent sound box voice recognition noise reduction method disclosed by the invention, a KAN convolution improvement-based DCU-Net noise reduction network model is constructed and trained, the complex frequency spectrum of audio data is encoded through a model encoder layer, feature information is extracted and enhanced, and then weighting processing is performed by using a local sparse attention mechanism, so that the noise reduction performance of the voice recognition noise reduction system is improved. Decoding the feature information and reconstructing the feature information into audio data after model noise reduction; and inputting the de-noised audio data of the model and the corresponding pure voice into an LwSDR loss function to calculate a loss value and update model parameters, freezing model parameter updating after training is completed, switching a verification set, and adjusting and optimizing model hyper-parameters. According to the method, the complexity of the model is effectively reduced, the robustness of the model in processing a complex audio scene is enhanced, meanwhile, efficient audio noise suppression is realized, and the voice recognition accuracy in the intelligent loudspeaker box is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of audio noise reduction processing methods, and in particular relates to a DCU-Net smart speaker speech recognition noise reduction method based on KAN convolution improvement. Background Art

[0002] Audio noise reduction is a key area in signal processing that aims to recover clear speech or sound from noisy audio signals. This technology is essential for a variety of applications, including voice communication, audio and video editing, hearing aids, surveillance systems, etc. Noise in audio signals can be classified into stationary noise and non-stationary noise, each of which has different characteristics and processing challenges.

[0003] Traditional audio noise reduction technology usually adopts the following five schemes: designing high and low bandpass filters for specific noise characteristics; using frequency domain transformation to estimate the noise power spectrum, such as spectral subtraction; using the statistical characteristics of the signal and noise, such as autoregression (AR) and moving average (MA) models for noise reduction; using subspace methods to decompose the signal into noise subspace and signal subspace, and then suppressing the noise component; and performing singular value decomposition (SVD) on the short-time Fourier transform (STFT) matrix of the signal to remove low singular value components to suppress noise.

[0004] In recent years, the development of deep learning technology has brought new solutions to audio denoising. Compared with traditional methods, deep learning-based methods can not only effectively process steady-state noise, but also perform well in processing non-steady-state noise. Current audio denoising technologies based on deep learning mainly include: Convolutional Neural Networks (CNN) extract features in the time-frequency domain for denoising; Long Short-Term Memory Networks (LSTM) process audio data with time series characteristics; Autoencoders (AE) perform reconstruction denoising by learning effective representations of data; Generative Adversarial Networks (GAN) generate high-quality denoised audio through adversarial training; Attention mechanisms help the model focus on the key parts of the signal and improve denoising performance; and U-Net structure networks, which were originally used for medical image segmentation and have also been introduced into the field of audio denoising, especially in tasks that require precise positioning and detail restoration.

[0005] In the field of audio noise reduction, the storage capacity and computing power limitations of end devices are the focus of research. Most existing methods are aimed at non-real-time noise reduction, and the proposed parameters are complex and difficult to integrate into the large network of end devices. These methods may not be applicable to all audio processing fields in practical applications. In addition, some networks are only designed for steady-state noise and lack adaptability to non-steady-state noise. In order to solve these problems, it is particularly important to explore lightweight networks with fewer parameters and higher computational efficiency, as well as models that can adapt to different audio environments and noise conditions. Through technologies such as neural architecture search (NAS), model compression, multi-task learning, and end-to-end learning, the parameter efficiency, real-time performance, and generalization ability of the model can be improved. At the same time, the application of speech noise reduction technology can further improve the adaptability of the model to different noise types. These optimization directions help promote the development of audio noise reduction technology, making it more suitable for end devices and real-time application scenarios, while improving the processing effect of steady-state and non-steady-state noise. Summary of the invention

[0006] The purpose of the present invention is to provide a DCU-Net smart speaker speech recognition denoising method based on KAN convolution improvement, which solves the problem of low adaptability caused by the complex number of model parameters used in existing denoising methods.

[0007] The technical solution adopted by the present invention is: a DCU-Net smart speaker speech recognition denoising method based on KAN convolution improvement, comprising the following steps:

[0008] Step 1: Obtain pure speech and background noise data, synthesize audio data sets and perform preprocessing;

[0009] Step 2: Divide the preprocessed audio data set into training set, validation set and test set;

[0010] Step 3: Build a DCU-Net denoising network model based on KAN convolution improvement and input the training set for training;

[0011] Step 4: Use L during model training wSDR The loss function calculates the loss value and updates the model parameters through the gradient descent method. After the training is completed, the model parameters are frozen, the validation set is switched, and the model hyperparameters are tuned.

[0012] Step 5: Embed the tuned model into the speech recognition system of the smart speaker.

[0013] The present invention is also characterized in that:

[0014] The preprocessing method in step 1 is as follows: first, remove the silent segments in the audio; then, calculate the frequency of each noise category in the audio data set, assign a penalty weight to each category, and use the inverse proportional method to give higher weights to categories with lower frequency of occurrence; finally, truncate or zero-fill the audio with inconsistent lengths to standardize the length of all audio.

[0015] In step 2, when dividing the audio data set, the noise type is evenly distributed in each subset, and a random seed is set.

[0016] The model constructed in step 3 includes three encoder layers Encoder1, Encoder2, Encoder3 with different receptive fields, a local sparse attention module LSA and three decoder layers Decoder1, Decoder2, Decoder3;

[0017] The three encoder layers encode the complex spectrum of the audio data to extract and enhance the feature information; each encoder layer consists of a KAN convolution block, complex batch normalization, and a complex Leaky ReLU activation function, and there is an ECA module behind each encoder layer to perform weighted processing on the feature matrix according to the channel. The convolution kernel of the KAN convolution block is composed of a nonlinear B-spline function with learnable parameters. The convolution kernel of the KAN convolution is recorded as The result of convolution is In the formula It is a nonlinear B-spline function, and the formula is as follows:

[0018]

[0019] In the formula, C i is the control point, is the learnable parameter, N i,k (x) is the i-th basis function, k is the order of the spline;

[0020] The local sparse attention module performs local windowing and weighting processing on the feature matrix after three encoder layers and the corresponding ECA module;

[0021] The feature matrix is ​​then decoded and reconstructed through three decoder layers and jump connections. Each decoder layer consists of a transposed convolution block, complex batch normalization, and a complex Leaky ReLU activation function. Finally, the reconstructed feature matrix is ​​subjected to an inverse Fourier transform to obtain the denoised audio data.

[0022] The complex spectrum of the audio data is obtained through short-time Fourier transform. The window type of short-time Fourier transform is Hamming window, the window length is 512, the step size is 256, the FFT point number is 512, and the sampling rate is consistent with the original audio sampling rate.

[0023] Encoder1 consists of a KAN convolution block with a convolution kernel of 1*1, a stride of 1, a padding of 0, an input channel of 2, and an output channel of 8, a complex batch normalization layer with num_features of 8, and a complex Leaky ReLU activation function with a leaky_slope of 0.05. The corresponding ECA module has 8 channels;

[0024] Encoder2 consists of a KAN convolution block with a convolution kernel of 3*3, a stride of 1*2, a padding of 1, 8 input channels, and 16 output channels, a complex batch normalization layer with num_features of 16, and a complex Leaky ReLU activation function with a leaky_slope of 0.05. The number of channels of the corresponding ECA module is 16;

[0025] Encoder3 consists of a KAN convolution block with a convolution kernel of 3*3, a stride of 2*1, a padding of 1, 16 input channels, and 32 output channels, a complex batch normalization layer with num_features of 32, and a complex LeakyReLU activation function with leaky_slope of 0.05. The corresponding ECA module has 32 channels;

[0026] LSA's embed_dim is 256, num_heads is 4, and window_size is 16;

[0027] The input channel of the transposed convolution block in Decoder1 is 32, the output channel is 16, the convolution kernel is 3*3, the stride is 2*1, and the padding is 1; the input channel of the transposed convolution block in Decoder2 is 16, the output channel is 8, the convolution kernel is 3*3, the stride is 1*2, and the padding is 1; the input channel of the transposed convolution block in Decoder3 is 8, the output channel is 2, the convolution kernel is 1*1, the stride is 1*1, and the padding is 1.

[0028] The specific method of extracting and enhancing feature information is:

[0029] Step 3.1, input the complex spectrum x of the audio data into Encoder1 to obtain the preliminary extracted local features F1(x) of 8*512*256;

[0030] Step 3.2: Input the local feature F1(x) into the ECA module with 8 channels to obtain the feature matrix E1(x) that has been weighted by the channel attention mechanism and whose shape remains unchanged.

[0031] Step 3.3, input the feature matrix E1(x) into Encoder2, and extract the feature matrix F2(x) of complex features 16*512*128 across time periods;

[0032] Step 3.4, input the feature matrix F2(x) into the ECA module with 16 channels to obtain the feature matrix E2(x) that has been weighted by the channel attention mechanism and whose shape remains unchanged;

[0033] Step 3.5, input the feature matrix E2(x) into Encoder3 to obtain the 32*256*128 global feature F3(x) across frequency segments and time segments;

[0034] Step 3.6: Input the global feature F3(x) into the ECA module with 32 channels to obtain the feature matrix E3(x) that has been weighted by the channel attention mechanism and whose shape remains unchanged.

[0035] The specific method of local windowing weighted processing is:

[0036] The feature matrix E3(x) is divided into multiple local windows with a window size of 16 according to the time frame. In each local window, Q i , Represents the characteristics of the time step within the window; using Q within the local window i and The matrix calculates the attention weights, and then uses the calculated attention weights to The matrix is ​​weighted to obtain the weighted feature matrix Attention in each time step i , the weighted feature matrix Attention in each time step i The output lsa(x) of the local sparse attention module is obtained by splicing, and the shape is kept as 32*256*128. The calculation formula of the output lsa(x) is as follows:

[0037] lsa(x)=[Attention1,Attention2,…,Attention L ]

[0038]

[0039] Where L is the number of windows, Q, K, and V represent query, key, and value matrices respectively; Q i is the query vector at the ith position, is the key matrix in the i-th position window, is the value matrix in the i-th position window, d is the vector dimension, is the scaling factor.

[0040] The specific method for decoding and reconstructing the feature matrix is:

[0041] After Decoder1 adds the feature matrix lsa(x) and the global feature F3(x) element by element, C1(x) is obtained, and then Decoder1 upsamples the feature matrix to a 16*512*128 data shape to obtain D1(x); the obtained feature matrix D1(x) is added to the feature matrix F2(x) element by element to obtain C2(x), and then Decoder2 obtains a feature matrix with a data shape of 8*512*256, denoted as D2(x); the local feature F1(x) and the feature matrix D2(x) are added and input into Decoder3 to obtain the denoised audio data spectrum D3(x) with a data shape of 2*512*256, and finally D3(x) is subjected to inverse Fourier transform to obtain the model denoised audio data y pred .

[0042] The audio data y after the model denoising in step 4 pred , original noisy audio data x, and clean speech data y true Enter L wSDR Loss function, calculate the loss value, L wSDR The loss function formula is as follows:

[0043]

[0044] wSDR = a·SDR(y true ,y pred )+(1-a)SDR(z true ,z pred )

[0045]

[0046] In the formula, z true =xy true is the real noise signal, z pred =xy pred is the predicted noise signal L wSDR is the loss function, N is the number of samples in the batch, and ∈ is a decimal to prevent the denominator from being zero.

[0047] The beneficial effects of the present invention are as follows: the DCU-Net smart speaker speech recognition denoising method based on KAN convolution improvement of the present invention adopts a cross-scale feature extraction method through the improved DCU-Net network structure, which can fully capture the information of different scales in the audio signal and fuse these features, so that the features finally input into the loss function are more comprehensive and representative, which effectively reduces the complexity of the model, enhances the robustness of the model in processing complex audio scenes, and realizes efficient audio noise suppression, thereby improving the speech recognition accuracy in the smart speaker. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of the DCU-Net smart speaker speech recognition denoising method based on KAN convolution improvement of the present invention;

[0049] Figure 2 It is a schematic diagram of a denoising network model constructed by the DCU-Net smart speaker speech recognition denoising method based on KAN convolution improvement of the present invention;

[0050] Figure 3 It is a schematic diagram of KAN convolution in the DCU-Net smart speaker speech recognition denoising method improved based on KAN convolution of the present invention;

[0051] Figure 4 It is a schematic diagram of the sparse attention mechanism in the DCU-Net smart speaker speech recognition denoising method based on KAN convolution improvement of the present invention;

[0052] Figure 5 This is a comparison chart of the denoising effects of synthetic data and real data in the DCU-Net smart speaker speech recognition denoising method based on KAN convolution improvement of the present invention. DETAILED DESCRIPTION

[0053] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] Example 1

[0055] The present invention provides a DCU-Net smart speaker speech recognition denoising method based on KAN convolution improvement, such as Figure 1 As shown, the specific implementation steps are as follows:

[0056] Step 1: Get the original audio from the recording data of speakers of different genders and ages, and select 10 scene sounds in UrbanSound8K, including "air_conditioner", "car_horn", "children_playing", "dog_bark", "drilling", "engine_idling", "gun_shot", "jackhammer", "siren" and "street_music", as well as additional noise sources such as white noise and pink noise, to generate noisy audio data under three signal-to-noise ratio conditions of 10dB, 15dB and 20dB.

[0057] Step 2: Divide the generated noisy audio data into training set, validation set and test set with a data division ratio of 3:1:1; uniformly resample all audio data to 16kHz, standardize the audio length, and trim or pad the audio to 66302 sampling points (about 4.14 seconds) to ensure the consistency of all input data; trimming or padding the audio data length ensures that all input model data has the same length and format, providing consistent input for subsequent model training to avoid affecting the model training effect due to inconsistent data length.

[0058] Step 3: Build a denoising network model based on the improved DCU-Net and perform initial training to enable it to learn basic denoising capabilities. The model includes three encoder layers with different receptive fields. Each encoder layer consists of a KAN convolution block, complex batch normalization, and a complex Leaky ReLu activation function. Each encoder layer is followed by its own ECA (channel attention mechanism) module to enhance the channel weights of the features after each encoder. Among them, the KAN convolution block uses a nonlinear function to replace the traditional convolution kernel. The parameters in the nonlinear function are learnable. In this way, each convolution operation can extract more refined features from the input data, thereby improving the model's fit to the audio signal. The sparse attention mechanism is used to weight the key parts of the audio signal and enhance the effective features of the signal, thereby further improving the denoising effect. Subsequently, the three decoder layers combine the feature matrix of the jump connection to upsample the feature matrix weighted by the sparse attention mechanism. Finally, the feature matrix that has passed through the three decoder layers is subjected to ISTFT (inverse short-time Fourier transform) to obtain the audio data after the model denoises.

[0059] Step 4: During the initial training of the model, use the clean speech data before synthetic noise and the audio data after network noise reduction in the training set to input L wSDR The loss function calculates the loss value, updates the model parameters through the gradient descent algorithm, gradually reduces the value of the loss function, and freezes the model parameter update after training is completed. The data set is switched to the validation set, and hyperparameters such as batch size, learning rate, Dropout Rate, and Regularization Parameters are tuned. Finally, the best model is saved for subsequent audio denoising tasks. Among them, the custom L wSDR The loss function measures the difference between the model output signal and the target noise-free signal through the weighted signal-to-noise ratio (WSDR). The loss function can adapt to different types of noise and audio scenes, thereby improving the noise reduction performance and robustness of the model.

[0060] Step 5: In the testing phase, the best model saved in step 4 is used to test the test set. The robustness and generalization performance of the model are evaluated through the CSIG (signal distortion), PESQ (speech quality score), STOI (speech intelligibility) and SNR (signal-to-noise ratio) indicators to verify the noise reduction effect of the model in practical applications. The evaluation indicators in the testing phase include but are not limited to CSIG, PESQ, STOI and SNR. Through the analysis of these indicators, the noise reduction effect, signal fidelity, background noise suppression ability, overall audio quality, speech intelligibility and signal-to-noise ratio of the model are comprehensively evaluated, thereby verifying the performance and applicability of the model in practical applications.

[0061] Example 2

[0062] The present invention provides a DCU-Net smart speaker speech recognition denoising method based on KAN convolution improvement, which is specifically implemented according to the following steps:

[0063] Step 1. Download the recording data of 28 speakers from [https: / / datashare.ed.ac.uk / handle / 10283 / 2791], covering multiple languages ​​and different sound quality. Use the UrbanSound8k dataset [https: / / urbansounddataset.weebly.com / urbansound8k.html] as the background noise source, and additionally generate white noise, pink noise, etc., and superimpose them on the speaker recordings with different signal-to-noise ratios (SNR) to form a diverse noisy audio dataset.

[0064] In step 1, first, we downloaded recordings of 28 speakers from the LibriSpeech dataset. These speakers cover different genders, ages, and accents to ensure data diversity. The recordings were all sampled at 16kHz, and the silent segments in the audio were removed to ensure high audio quality. In addition, UrbanSound8k was selected as the background noise source, which includes a variety of common urban noise types such as traffic noise, street noise, and construction noise. To further expand the types of noise, white noise, pink noise, etc. were generated, and these noises were added to the original noise-free audio with different signal-to-noise ratios (10dB, 15dB, and 20dB) by linear superposition. The selection of these SNR values ​​is based on the typical noise levels in real environments to ensure that the model can perform well in a variety of noise environments. The final dataset consists of thousands of audio samples with added noise, covering a variety of noise types and signal-to-noise ratios. The dataset was preprocessed in detail, including resampling and volume normalization, and the length of each audio sample was ensured to be 66,302 samples (about 4.14 seconds), laying the foundation for subsequent model training.

[0065] Step 2: Calculate the penalty weight of the category based on the frequency of the noise category in the data set when synthesizing the noise data set in step 1, and use the inverse proportional method to give higher weights to categories with lower frequencies.

[0066] In step 2, in order to deal with the problem of imbalanced samples of different noise categories in the data set, a balanced weight strategy is adopted. Assume that there are N different noise categories in the data set, and the number of samples of category i is n i , category weight C i Calculated by the ratio of the number of category samples to the total number of samples. The specific weight formula is: Where N total is the total number of all samples in the dataset, N is the total number of categories, and ni is the number of samples of category i. In this way, categories with fewer samples will be given higher weights to reduce the impact of category imbalance on model training.

[0067] In order to ensure that the weights are stable and reasonable during the calculation process, normalization is performed next. The normalization formula is: This step ensures that the sum of the weights of all categories is 1, avoiding the adverse effects of too large or too small weights on the loss calculation of the training process.

[0068] Next, the sampling rate of all audio samples is unified to 16kHz, and the audio length is unified to 66302 sampling points. For audio with insufficient length, zero padding is performed, and for audio with excessive length, truncation is performed to ensure the consistency of input data. The formula is as follows:

[0069]

[0070] Where τ(t) is the data after the unified length, x(t) is the original data, Lx is the data length, and the insufficient length is padded with 0. In the process of adding noise, the noise intensity is controlled according to the target signal-to-noise ratio (SNR), and the calculation formula is as follows:

[0071]

[0072]

[0073] Among them, SNR dB is the target signal-to-noise ratio, Psignal is the clean audio data signal power, and Pnoise is the noise power. N(i) is the noise signal adjusted according to the target signal-to-noise ratio. n(i) is the noise signal adjusted according to the target signal-to-noise ratio.

[0074] Step 3: The overall data set is divided into training set, validation set, and test set in a ratio of 3:1:1 to ensure that the noise types are evenly distributed in each subset. A random seed is set during the division process to ensure the repeatability of the experiment.

[0075] In step 3, after the dataset is prepared, it is divided into training set, validation set and test set in a ratio of 3:1:1, that is, 60% for training, 20% for validation, and 20% for testing. This ratio ensures that each subset reasonably represents the diversity of the entire dataset. In order to avoid data leakage, stratified sampling is used to ensure that the same speaker or the same noise scene does not appear in different subsets at the same time. At the same time, a fixed random seed is set to ensure the reproducibility of the partitioning process. The validation set is used for parameter adjustment and monitoring overfitting, while the test set is used to evaluate the final generalization ability of the model. After the division, each subset is checked for balance and quality to ensure that its distribution is reasonable.

[0076] Step 4: Figure 2 As shown, first, each audio file is subjected to a short-time Fourier transform (STFT) using a Hann window to obtain a complex spectrum x with a shape of 2*512*256, where 512 represents the frequency resolution, 256 represents the number of time frames, and 2 represents the real and imaginary parts of the complex spectrum. Next, the audio data x is processed through three encoder layers to extract feature information at different levels.

[0077] Step 4.1: After the first encoder layer is processed, the preliminary feature matrix F1(x) is obtained. Then, F1(x) is weighted by ECA (channel attention mechanism) to obtain the enhanced feature matrix E1(x), whose shape remains unchanged. The short-time Fourier transform formula is as follows:

[0078]

[0079] Among them, the short-time Fourier transform (STFT) decomposes the time domain signal into a joint representation of time and frequency. The X(m,k) in the formula represents the kth frequency component of the mth time frame, which is a complex value in the spectrum diagram. The input signal x[n] is cut out through the window function w[n-mR]. The window function is a Hamming window, and the window length is N, which corresponds to the number of points of the Fourier transform and determines the frequency resolution. The jump step length R determines the moving step of the time frame and affects the overlap between adjacent frames. The Fourier transform kernel e -j2πkn / N Used to convert the intercepted signal into the frequency domain, generating a two-dimensional representation of time and frequency.

[0080] like Figure 3As shown in the figure, the data passes through the encoder layer composed of KAN convolution, complex batch normalization, and complex Leaky ReLu activation function. The convolution kernel size in the first encoder layer is 1*1. Unlike traditional convolution, the convolution kernel of KAN convolution is not composed of traditional learnable parameters but of B-spline spline nonlinear functions. The parameters in the B-spline spline function are learnable. This enables each convolution operation to learn a more delicate expression of the data and has a better fit to the data. Specifically, for the traditional convolution operation, the 2*2 convolution kernel is recorded as The feature matrix is ​​simply written as The result of a convolution is result = k1*f1+k2*f2+k3*f3+k4*f4+bias. The convolution kernel of KAN convolution is The result of convolution is in It is a nonlinear B-spline function, and the formula is as follows:

[0081]

[0082] Among them, C i is the control point, is the learnable parameter, N i,k (x) is the i-th basis function, and k is the order of the spline. The traditional convolution kernel performs weighted summation on the input features, which is a linear operation. However, each convolution kernel in the KAN convolution is a nonlinear B-spline function, which performs nonlinear transformation on the input features, can capture more complex feature relationships, and provide stronger support for subsequent audio noise reduction. The comparison between the traditional convolution kernel and the KAN convolution is shown in Table 1 below:

[0083] Table 1 Comparison of traditional convolution kernels and KAN convolution

[0084]

[0085] After the first encoder layer Encoder1 (KAN convolution block: convolution kernel 1*1; step size 1*1; padding 0, input channel 2, output channel 8; complex batch normalization layer num_features is 8; complex Leaky ReLU activation function leaky_slope is 0.05), the feature matrix is ​​in the shape of (8*512*256). The main function of the first layer is to perform preliminary feature extraction on the features after short-time Fourier transform, such as local features such as the basic frequency pattern of the audio signal, and introduce nonlinear operations to enhance the local perception ability of the model, so as to facilitate the subsequent deep-level features. After the first encoding layer, the above-mentioned feature F1(x) is obtained. Subsequently, the ECA module is applied to each feature matrix that passes through the encoder layer. Specifically, global average pooling is performed on each channel and compressed to 8 dimensions (16 for the second ECA module and 32 after the third ECA module). Then, these average values ​​are processed by one-dimensional convolution to generate the attention weights of each channel. Finally, the weights are normalized using the Sigmoid activation function and applied to the original feature map to obtain the enhanced feature E1(x), which is calculated as follows:

[0086] X′c=Sc·Xc

[0087] Sc=σ(Ac)

[0088] Ac=Conv1D(Zc)

[0089]

[0090] Where Xc(i,j) represents the value of the i-th row and j-th column of the feature map on channel c, H and W are the height and width of the feature map, Ac is the attention weight of channel c, σ(·) represents the Sigmoid function, and X′c represents the feature map after weight enhancement.

[0091] Step 4.2, the convolution kernel of the KAN convolution in the second encoder Encoder2 is 3*3, the step size is 1*2, the padding is 1, the input channel is 8, and the output channel is 16; the num_features of the complex batch normalization layer is 16; the leaky_slope of the complex Leaky ReLU activation function is 0.05. After the second encoder, the feature matrix F2(x) is obtained, and then the feature matrix E2(x) is obtained after weighting by the ECA module with 16 channels.

[0092] Step 4.3, the convolution kernel of the third encoder Encoder3 is 3*3, the step size is 2*1, the padding is 1, the input channels are 16, and the output channels are 32; the num_features of the complex batch normalization layer is 32; the leaky_slope of the complex Leaky ReLU activation function is 0.05. After the third encoder, the feature matrix F3(x) is obtained, and then the feature matrix E3(x) is obtained after weighting by the ECA module with 32 channels. After that, the feature matrix E3(x) obtained after the three encoders and their respective ECAs has a shape of 32*256*128, representing high-level features across frequency bands and time periods.

[0093] Step 5: Figure 4 As shown in the figure, after passing through the encoder layers with three different receptive fields and the ECA mechanism, the local sparse attention mechanism is further applied to weight the final E3(x) obtained in step 4.3. The embed_dim of LSA is 256, num_heads is 4, and window_size is 16, and the weighted feature matrix lsa(x) is obtained. The calculation method is as follows:

[0094] lsa(x)=[Attention1,Attention2,…,Attention L ]

[0095]

[0096] Where is the weighted output matrix calculated by lsa(x) as a whole, Attention i is the attention calculation result in the i-th window, L is the number of windows, Q, K, V represent the query, key and value matrices respectively; Q i is the query vector at the ith position, is the key matrix in the i-th position window, is the value matrix in the i-th position window, d is the vector dimension, is the scaling factor. The data is then passed through three decoder layers to upsample the data to the original shape of 2*512*256. The input channel of the transposed convolution in Decoder1 is 32, the output channel is 16, the convolution kernel is 3*3, the step size is 2*1, and the padding is 1. The input channel of the transposed convolution block in Decoder2 is 16, the output channel is 8, the convolution kernel is 3*3, the step size is 1*2, and the padding is 1. The input channel of the transposed convolution block in Decoder3 is 8, the output channel is 2, the convolution kernel is 1*1, the step size is 1*1, and the padding is 1.

[0097] In step 5, the feature matrix lsa(x) obtained after weighting by the local sparse attention mechanism is added to F3(x) of the same shape obtained in step 4.3 to obtain C1(x), and then passes through a decoder layer composed of transposed convolution, complex batch normalization, and complex LeakyRelu activation function to obtain the feature matrix D1(x). Similarly, D1(x) is added to F2(x) in step 4.2, and the feature matrix D2(x) is obtained after passing through the second decoder. Finally, D2(x) is added to F1(x) in step 4.1, and the feature matrix D3(x) is obtained after passing through the last layer of decoder. The feature matrix D3(x) is then subjected to ISTFT to obtain the audio data y after model denoising. pred .

[0098] Step 6: Calculate the loss using the original noise-free audio and the audio data processed by the model, using a custom weighted signal-to-noise-distortion ratio loss function (L wSDR ) as the loss function. The Adam optimizer and StepLR were used to adjust the learning rate and train for 50 Epochs to improve the convergence effect and stability of the model.

[0099] In step 6, the custom L wSDR In the loss function, the calculation method of the weighted signal-to-noise ratio wSDR and the weighting coefficient a is as follows:

[0100]

[0101] wSDR = a·SDR(y true ,y pred )+(1-a)SDR(y true ,y pred )

[0102]

[0103] Where x is the original noisy frequency signal, y true is the target noise-free frequency signal, y pred is the noise-free frequency signal predicted by the model, z true =xy true is the real noise signal, z pred =xy pred is the predicted noise signal L wSDR is the loss function, N is the number of samples in the batch, and ∈ is a decimal to prevent the denominator from being zero. The Adam optimizer was used in the training process, and the StepLR learning rate scheduler was used to optimize the convergence speed and final performance of the model by gradually reducing the learning rate. In order to prevent overfitting, the training process includes data enhancement and regularization techniques, and gradient clipping is used to maintain the stability of the training.

[0104] Step 7: After the initial training is completed, freeze the model parameters, switch the validation set and tune the model hyperparameters. The adjusted deep learning model is embedded in the speech recognition system of the smart speaker, and the accuracy of speech recognition is effectively improved through precise audio noise reduction processing. First, after the model is trained on a large amount of noisy and clear speech data in the above steps, it can accurately separate the speech signal and reduce the interference of environmental noise. Then, the model is integrated into the real-time audio processing pipeline of the smart speaker, so that the speaker can apply the noise reduction algorithm in real time to the received speech signal. With the support of hardware accelerators, efficient processing is ensured under low latency, thereby greatly improving the speech recognition ability of the speaker in complex noise environments, providing users with a clearer and more accurate voice interaction experience. This embedded noise reduction solution not only improves the accuracy of speech recognition, but also improves the overall user experience, and is suitable for various home and office scenarios. For smart speakers with low computing power, post-training quantization technology can be used to convert the weights and activation values ​​of the model into 8-bit integers to reduce memory usage. Although this will slightly reduce the noise reduction effect, it can ensure that the model runs stably on the device.

[0105] Example 3

[0106] The present invention provides a DCU-Net smart speaker speech recognition denoising method based on KAN convolution improvement, which is specifically implemented according to the following steps:

[0107] Step 1. The original data set contains 20,076 recordings of speakers of different genders and ages, and the lengths of these recordings vary. 10 scene sounds from UrbanSound8K are selected, namely "air_conditioner", "car_horn", "children_playing", "dog_bark", "drilling", "engine_idling", "gun_shot", "jackhammer", "siren" and "street_music" as noise data. In addition, pink noise and white noise are added, totaling 12 noise sources. Through a custom noise mixing method, a noise audio is randomly selected and synthesized with the original audio under three signal-to-noise ratio conditions of 10dB, 15dB, and 20dB. Finally, 60,228 noisy synthetic audio data are generated, and the number of synthetic audios with different noises is recorded at the same time, which is convenient for adding penalty weights to different noises later.

[0108] Step 2: Based on the above step 1, the data is divided into a training set, a validation set, and a test set with a ratio of 3:1:1. The training set has 36,136 noisy audio files, the validation set has 12,045 noisy audio files, and the test set has 12,045 noisy audio files. When the model loads the data, the audio data is first resampled to 16kHz, and the length of each audio segment is cut to 66,302 sampling points (about 4.14 seconds), and the insufficient length is filled with 0. At the same time, set batchSize to 16, that is, 64 data samples are processed each time, and they are input into the model as a small batch for training.

[0109] Step 3: When training the denoising network model, the basic network structure uses DCU-Net. The model includes three encoder layers with different receptive fields. After being processed by the first layer of encoder, the data shape becomes 8*512*256, and the feature is enhanced by the ECA module, and the data remains unchanged. The second layer of encoder processes the data into 16*512*128, and the features are also enhanced by the ECA module. After passing through the third layer of encoder, the data shape becomes 32*256*128, and is enhanced again by the ECA module. Subsequently, the 32*256*128 data enters the sparse attention mechanism layer to further extract key features. Next, the feature matrix weighted by the sparse attention mechanism is restored to its original shape by three decoders composed of transposed convolution, complex batch normalization, and complex Leaky ReLu activation functions, with reference to the saved data of the jump connection. While the decoder is decoding, the jump connection mechanism retains the original data features. Finally, the audio data after model denoising is obtained by ISTFT (inverse short-time Fourier transform), and is sent to the loss function to calculate the loss, and the model parameters are updated by back-propagating gradients.

[0110] Step 4: The training set processed in step 2 will be used as the input of the denoising network model built in step 3, using the custom L wSDR The loss function is used as the loss function of the model for training. After training, freeze the model parameter updates, switch the data set to the validation set, and tune the hyperparameters such as batch size, learning rate, Dropout Rate, Regularization Parameters, etc. In this process, the model parameters will not be updated, but the following audio denoising indicators will be used to evaluate the impact of the hyperparameters, and the model with the best effect will be saved as the audio denoising model. Figure 5 As shown in the figure, group a is pure speech without noise, the left side of group b is artificially synthesized noisy speech, the right side of group b is the spectrum diagram with real-world noise, and group c corresponds to the spectrum diagram of the audio data after the model denoises the synthetic data and the real data. Figure 5The overall denoising effect of the model is shown.

[0111] Step 5. In the testing phase, the best model saved in step 4 is tested using the test set of 12045 noisy audio segments divided in step 2. During the test, the same data indicators as step 4 are used, including CSIG, PESQ, STOI, and SNR, to comprehensively evaluate the performance of the model.

[0112] CSIG: It is used to evaluate the degree of distortion of the signal itself, focusing on the fidelity of the audio signal during processing. The CSIG score ranges from 1 to 5, with higher scores indicating higher signal fidelity and less distortion.

[0113] PESQ: A speech quality score based on the human auditory perception model, ranging from -0.5 to 4.5. A higher score means that the processed speech quality is closer to the original signal and the distortion is smaller.

[0114] STOI: An objective indicator for evaluating the intelligibility of speech signals. Its value range is from 0 to 1. The closer to 1, the higher the speech intelligibility.

[0115] SNR: A basic indicator of signal quality, which represents the ratio of signal power to noise power in decibels (dB). A higher SNR indicates a clearer signal and less noise.

[0116] These indicators are used to verify the robustness and generalization performance of the model, ensuring that the model can still maintain high-quality denoising effects on unseen test data. The test results will be used to further analyze the practical application capabilities of the model and may provide a basis for subsequent optimization. The results of performance tuning are shown in Table 2 below. The following data are all average values ​​​​statisticed on the validation set:

[0117] Table 2 Performance tuning results

[0118]

[0119]

[0120] In summary, the present invention significantly improves the denoising ability of the model by constructing encoder layers with different receptive fields, introducing the ECA module to enhance feature extraction, and combining the sparse attention mechanism. wSDRThe loss function is used for training, and the model is fine-tuned by adjusting hyperparameters such as batch size, learning rate, Dropout Rate, RegularizationParameters, etc. Finally, the robustness and generalization performance of the model are verified by evaluation on an independent test set, ensuring that it can effectively reduce noise and retain the fidelity and clarity of audio signals in practical applications. The method of the present invention has strong practicality and broad application prospects, and can be used for noise reduction tasks in a variety of audio processing scenarios.

Claims

1. The DCU-Net smart speaker speech recognition denoising method based on KAN convolution is characterized by: The following steps are involved: Step 1: Obtain pure speech and background noise data, synthesize audio data sets and perform preprocessing; Step 2: Divide the preprocessed audio data set into a training set, a validation set, and a test set; Step 3: Build a DCU-Net denoising network model based on KAN convolution improvement and input the training set for training; Step 4: Use L during model training wSDR The loss function calculates the loss value and updates the model parameters through the gradient descent method. After the training is completed, the model parameters are frozen, the validation set is switched, and the model hyperparameters are tuned. Step 5: Embed the tuned model into the speech recognition system of the smart speaker.

2. The DCU-Net intelligent speaker speech recognition denoising method based on KAN convolution improvement as claimed in claim 1 is characterized in that: The preprocessing method in step 1 is specifically as follows: first, remove the silent segments in the audio; then, calculate the frequency of occurrence of each noise category in the audio data set, assign a penalty weight to each category, and use the inverse proportional method to give a higher weight to the category with a lower frequency of occurrence; finally, truncate or zero-fill the audio with inconsistent lengths to standardize the length of all audio.

3. The DCU-Net intelligent speaker speech recognition denoising method based on KAN convolution improvement as claimed in claim 1 is characterized in that: In step 2, when dividing the audio data set, the noise type is evenly distributed in each subset, and a random seed is set.

4. The DCU-Net intelligent speaker speech recognition denoising method based on KAN convolution improvement as claimed in claim 1, characterized in that: The model constructed in step 3 includes three encoder layers Encoder1, Encoder2, Encoder3 with different receptive fields, a local sparse attention module LSA and three decoder layers Decoder1, Decoder2, Decoder3; The three encoder layers encode the complex spectrum of the audio data to extract and enhance the feature information; each encoder layer consists of a KAN convolution block, complex batch normalization, and a complex Leaky ReLU activation function, and there is an ECA module behind each encoder layer to perform weighted processing on the feature matrix according to the channel. The convolution kernel of the KAN convolution block is composed of a nonlinear B-spline function with learnable parameters. The convolution kernel of the KAN convolution is recorded as The result of convolution is In the formula is a nonlinear B-spline function, and the formula is as follows: In the formula, C i is the control point, is the learnable parameter, N i,k (x) is the i-th basis function, k is the order of the spline; The local sparse attention module performs local windowing and weighting processing on the feature matrix after three encoder layers and the corresponding ECA module; The feature matrix is ​​then decoded and reconstructed through three decoder layers and jump connections. Each decoder layer consists of a transposed convolution block, complex batch normalization, and a complex Leaky ReLU activation function. Finally, the reconstructed feature matrix is ​​subjected to an inverse Fourier transform to obtain the denoised audio data.

5. The DCU-Net intelligent speaker speech recognition denoising method based on KAN convolution improvement as claimed in claim 4 is characterized in that: The complex spectrum of the audio data is obtained by short-time Fourier transform. The window type of short-time Fourier transform adopts Hamming window, the window length is 512, the step size is 256, the FFT point number is 512, and the sampling rate is consistent with the original audio sampling rate.

6. The DCU-Net intelligent speaker speech recognition denoising method based on KAN convolution improvement as claimed in claim 4 is characterized in that: The Encoder1 consists of a KAN convolution block with a convolution kernel of 1*1, a stride of 1, a padding of 0, an input channel of 2, and an output channel of 8, a complex batch normalization layer with num_features of 8, and a complex Leaky ReLU activation function with a leaky_slope of 0.

05. The number of channels of the corresponding ECA module is 8; Encoder2 consists of a KAN convolution block with a convolution kernel of 3*3, a stride of 1*2, a padding of 1, 8 input channels, and 16 output channels, a complex batch normalization layer with num_features of 16, and a complex Leaky ReLU activation function with a leaky_slope of 0.

05. The number of channels of the corresponding ECA module is 16; Encoder3 consists of a KAN convolution block with a convolution kernel of 3*3, a stride of 2*1, a padding of 1, 16 input channels, and 32 output channels, a complex batch normalization layer with num_features of 32, and a complex Leaky ReLU activation function with leaky_slope of 0.

05. The corresponding ECA module has 32 channels. LSA's embed_dim is 256, num_heads is 4, and window_size is 16; The input channel of the transposed convolution block in Decoder1 is 32, the output channel is 16, the convolution kernel is 3*3, the stride is 2*1, and the padding is 1; the input channel of the transposed convolution block in Decoder2 is 16, the output channel is 8, the convolution kernel is 3*3, the stride is 1*2, and the padding is 1; the input channel of the transposed convolution block in Decoder3 is 8, the output channel is 2, the convolution kernel is 1*1, the stride is 1*1, and the padding is 1.

7. The DCU-Net intelligent speaker speech recognition denoising method based on KAN convolution improvement as claimed in claim 4 is characterized in that: The specific method of extracting and enhancing feature information is: Step 3.1, input the complex spectrum x of the audio data into Encoder1 to obtain the preliminary extracted local features F1(x) of 8*512*256; Step 3.2: Input the local feature F1(x) into the ECA module with 8 channels to obtain the feature matrix E1(x) that has been weighted by the channel attention mechanism and whose shape remains unchanged. Step 3.3, input the feature matrix E1(x) into Encoder2, and extract the feature matrix F2(x) of complex features 16*512*128 across time periods; Step 3.4, input the feature matrix F2(x) into the ECA module with 16 channels to obtain the feature matrix E2(x) that has been weighted by the channel attention mechanism and whose shape remains unchanged; Step 3.5, input the feature matrix E2(x) into Encoder3 to obtain the 32*256*128 global feature F3(x) across frequency segments and time segments; Step 3.6: Input the global feature F3(x) into the ECA module with 32 channels to obtain the feature matrix E3(x) that has been weighted by the channel attention mechanism and whose shape remains unchanged.

8. The DCU-Net intelligent speaker speech recognition denoising method based on KAN convolution improvement as claimed in claim 7 is characterized in that: The specific method of the local windowing weighted processing is: The feature matrix E3(x) is divided into multiple local windows with a window size of 16 according to the time frame. In each local window, Q i , Represents the characteristics of the time step within the window; using Q within the local window i and The matrix calculates the attention weights, and then uses the calculated attention weights to The matrix is ​​weighted to obtain the weighted feature matrix Attention in each time step i , the weighted feature matrix Attention in each time step i The output lsa(x) of the local sparse attention module is obtained by splicing, and the shape is kept as 32*256*128. The calculation formula of the output lsa(x) is as follows: lsa(x)=[Attention1,Attention2,…,Attention L ] Where L is the number of windows, Q, K, and V represent query, key, and value matrices respectively; Q i is the query vector at the ith position, is the key matrix in the i-th position window, is the value matrix in the i-th position window, d is the vector dimension, is the scaling factor.

9. The DCU-Net intelligent speaker speech recognition denoising method based on KAN convolution improvement as claimed in claim 8, characterized in that: The specific method for decoding and reconstructing the feature matrix is: After Decoder1, the feature matrix lsa(x) and the global feature F3(x) are added element by element to obtain C1(x), and then Decoder1 upsamples the feature matrix to a 16*512*128 data shape to obtain D1(x); the obtained feature matrix D1(x) is added element by element to the feature matrix F2(x) to obtain C2(x), and then Decoder2 obtains a feature matrix with a data shape of 8*512*256, recorded as D2(x); the local feature F1(x) and the feature matrix D2(x) are added and input into Decoder3 to obtain the denoised audio data spectrum D3(x) with a data shape of 2*512*256, and finally D3(x) is subjected to inverse Fourier transform to obtain the model denoised audio data y pred .

10. The DCU-Net intelligent speaker speech recognition denoising method based on KAN convolution improvement as claimed in claim 9, characterized in that: In step 4, the audio data y after model denoising is pred , original noisy audio data x, and clean speech data y true Enter L wSDR Loss function, calculate the loss value, L wSDR The loss function formula is as follows: <h2 style=";text-align:left;direction:ltr">wSDR = aSDR(y)<h2 style=";text-align:left;direction:ltr"> true <h2 style=";text-align:left;direction:ltr"> ,y<h2 style=";text-align:left;direction:ltr"> pred <h2 style=";text-align:left;direction:ltr"> )+(1-a)SDR(z<h2 style=";text-align:left;direction:ltr"> true <h2 style=";text-align:left;direction:ltr"> ,z<h2 style=";text-align:left;direction:ltr"> pred <h2 style=";text-align:left;direction:ltr"> ) In the formula, z true =xy true is the real noise signal, z pred =xy pred is the predicted noise signal L wSDR is the loss function, N is the number of samples in the batch, and ∈ is a decimal to prevent the denominator from being zero.

Images