An audio denoising method and apparatus

Abstract

The application discloses an audio denoising method and device, and belongs to the technical field of audio, which can effectively separate pure speech, effectively remove noise and retain speech characteristics in a complex noise environment, improves the audio denoising effect, and improves the audio quality after denoising.

Description

Technical Field

[0001] This application relates to the field of audio technology, and in particular to an audio noise reduction method and apparatus. Background Technology

[0002] In fields such as intelligent voice interaction and speech recognition, high-quality voice data is fundamental to ensuring system performance. However, in actual voice communication, noise interference becomes a key factor affecting voice quality and communication effectiveness.

[0003] Existing audio denoising methods face numerous challenges when processing speech signals in complex noisy environments. Many traditional methods lack data diversity and cannot accurately simulate real-world, complex noise environments. This results in models failing to fully learn the characteristics and relationships between speech and noise at different noise intensities during training, leading to poor denoising performance. Furthermore, existing methods primarily focus on processing the amplitude spectrum, neglecting phase information, which is crucial for speech reconstruction quality. This further limits the clarity and naturalness of the denoised speech. To further improve the quality and stability of voice communication, there is an urgent need for an audio denoising method that can effectively remove noise and preserve speech features in complex noisy environments. Summary of the Invention

[0004] This application provides an audio noise reduction method and apparatus to solve the problem of difficulty in effectively separating clean speech.

[0005] In a first aspect, this application provides an audio denoising method, which is applied to a pre-trained audio denoising model, the audio denoising model including a complex feature extraction network, a complex convolutional encoder, and a complex convolutional decoder, the method comprising:

[0006] The original audio to be processed is input into the complex feature extraction network, which extracts audio signal features at multiple different scales from the original audio to be processed, and obtains the complex time spectrum of the original audio to be processed based on the multiple audio signal features at different scales.

[0007] The complex time-frequency spectrum of the original audio to be processed is input into the complex convolutional encoder, and the complex convolutional encoder obtains the encoding features of the original audio to be processed based on the multi-level time-frequency features of the complex time-frequency spectrum.

[0008] The encoded features of the original audio to be processed are input into the complex convolutional decoder. The complex convolutional decoder obtains the noise-reduced frequency corresponding to the original audio by upsampling the encoded features layer by layer and concatenating the encoded features of the corresponding layers.

[0009] Secondly, this application provides an audio denoising apparatus, which is applied to a pre-trained audio denoising model, the audio denoising model including a complex feature extraction network, a complex convolutional encoder, and a complex convolutional decoder, the apparatus comprising:

[0010] The first unit is used to input the original audio to be processed into the complex feature extraction network, the complex feature extraction network extracts audio signal features at multiple different scales from the original audio to be processed, and obtains the complex time spectrum of the original audio to be processed based on the audio signal features at multiple different scales;

[0011] The second unit is used to input the complex time spectrum of the original audio to be processed into the complex convolutional encoder, and the complex convolutional encoder obtains the encoding features of the original audio to be processed based on the multi-level time-frequency features of the complex time spectrum.

[0012] The third unit is used to input the encoded features of the original audio to be processed into the complex convolutional decoder. The complex convolutional decoder obtains the noise-reduced frequency corresponding to the original audio by upsampling the encoded features layer by layer and concatenating the encoded features of the corresponding layers.

[0013] Thirdly, this application provides a readable medium including executable instructions, which, when executed by a processor of an electronic device, cause the electronic device to perform any of the methods described in the first aspect.

[0014] Fourthly, this application provides an electronic device including a processor and a memory storing execution instructions, wherein when the processor executes the execution instructions stored in the memory, the processor performs the method as described in any of the first aspects.

[0015] As can be seen from the above technical solution, the audio denoising method provided in this application is specifically designed for pre-trained audio denoising models, which encompass a complex feature extraction network, a complex convolutional encoder, and a complex convolutional decoder. In practical applications, the original audio to be processed first enters the complex feature extraction network, which can extract features of the audio signal from different scales and generate a complex time-frequency spectrum accordingly. This process allows the rich information of the audio data in the time-frequency domain to be fully captured. Subsequently, the complex time-frequency spectrum is input to the complex convolutional encoder. The encoder, leveraging its ability to analyze the multi-level time-frequency features of the complex time-frequency spectrum, outputs the encoded features of the audio to be processed. This step enables the model to learn complex features at different frequencies and time scales. Finally, the complex convolutional decoder receives the encoded features, upsamples layer by layer, and combines the encoded features of the corresponding layers to generate the denoised audio. This method achieves direct processing of time-frequency features in the complex domain, simultaneously optimizing amplitude and phase information, and significantly improving the quality of audio reconstruction. Specifically, the complex time-frequency spectrum that preserves phase information enables the model to reconstruct speech signals more accurately, effectively reducing speech distortion and making the reconstructed speech clearer and more natural, thereby enhancing the model's ability to understand and process speech signals. Furthermore, by extracting multi-level time-frequency features, the model learns more comprehensively the complex features of the speech signal. The layer-by-layer upsampling and splicing of corresponding layer-coded features not only solves the shortcomings of existing decoders but also fully utilizes feature information at different levels and scales, helping to recover high-frequency details of the speech, thus improving the clarity and realism of the reconstructed speech. Overall, this method can effectively separate clean speech, effectively remove noise and preserve speech features in complex noise environments, improve the effect of audio denoising, and enhance the quality of the denoised audio. The further effects of the aforementioned non-conventional preferred method will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0016] To more clearly illustrate the embodiments of this application or the existing technical solutions, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating an audio denoising method provided in this application;

[0018] Figure 2 This application provides a schematic diagram of the structure of an audio denoising model.

[0019] Figure 3 This application provides a schematic diagram of the structure of a complex convolution decoder.

[0020] Figure 4 This application provides a schematic diagram of the structure of an audio noise reduction device;

[0021] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] The various non-limiting embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0024] The inventors have discovered that existing audio denoising methods face numerous challenges when processing speech signals in complex noisy environments. Many traditional methods lack data diversity, failing to accurately simulate real-world, complex noise environments. This results in models being unable to fully learn the characteristics and relationships between speech and noise at different noise intensities during training, leading to poor denoising performance. Furthermore, existing methods primarily focus on processing the amplitude spectrum, neglecting phase information, which is crucial for speech reconstruction quality. This further limits the clarity and naturalness of the denoised speech. To further improve the quality and stability of voice communication, there is an urgent need for an audio denoising method that can effectively remove noise and preserve speech features in complex noisy environments. Therefore, this application proposes an audio denoising method to meet the above requirements.

[0025] See Figure 1 This paper illustrates an audio denoising method according to an embodiment of this application. The method is applied to a pre-trained audio denoising model, which includes a complex feature extraction network, a complex convolutional encoder, and a complex convolutional decoder. The complex feature extraction network, the complex convolutional encoder, and the complex convolutional decoder are sequentially connected. The method includes the following steps:

[0026] S101: Input the original audio to be processed into the complex feature extraction network. The complex feature extraction network extracts audio signal features at multiple different scales from the original audio to be processed, and obtains the complex time spectrum of the original audio to be processed based on the audio signal features at multiple different scales.

[0027] In this embodiment, as Figure 2As shown, the complex feature extraction network may include a first complex spectrum conversion module M1, a second complex spectrum conversion module M2, and a third complex spectrum conversion module M3. It can be understood that the complex feature extraction network uses a three-layer complex spectrum conversion module structure to achieve time-frequency analysis and complex feature extraction. The frame length and frame shift of each of the first, second, and third complex spectrum conversion modules are different. For example, the frame length of the first complex spectrum conversion module... Frame shift The frame length of the second complex spectrum conversion module Frame shift The frame length of the third complex spectrum conversion module Frame shift After acquiring the original audio to be processed, the complex feature extraction network can use a first complex spectrum conversion module, a second complex spectrum conversion module, and a third complex spectrum conversion module to extract audio signal features at multiple different scales from the original audio to be processed. Based on these multiple audio signal features at different scales, the complex time-frequency spectrum of the original audio to be processed is obtained. In other words, this embodiment converts the speech signal of the original audio to be processed into a complex time-frequency domain representation, thereby facilitating the processing of speech features in the time-frequency domain. The input to the complex feature extraction network is the original audio to be processed (i.e., the noisy speech waveform signal). The output is the spectrum when it is a complex number. It should be noted that most existing methods only process the amplitude spectrum, ignoring phase information, which has a significant impact on speech reconstruction quality. To jointly optimize amplitude and phase to improve reconstruction quality, this step designs a complex feature extraction network including a first complex spectrum conversion module, a second complex spectrum conversion module, and a third complex spectrum conversion module, proposing a method to directly preserve the complex spectrum, as follows:

[0028] The STFT transformation is performed on each frame of the original audio to be processed, using the following formula:

[0029]

[0030] Where n is the sample index, which takes values ​​from 0 to N-1 and is used to traverse each sample point in a frame of speech signal. y(t) is the noisy speech (i.e., the original audio to be processed) time-domain signal, a continuous or discrete time series representing the amplitude variation of the entire noisy speech in the time domain. y(mL+n) is the signal representation of the noisy speech after frame division. Where m is the frame index, L is the frame shift of the complex spectrum conversion module, and n is the sample index. mL determines the starting position of the current frame in the entire speech signal (i.e., the original audio to be processed), n indexes the sample points within the frame, and y(mL+n) represents the signal value of the nth sample point in the m-th frame. It is the specific intra-frame sample value obtained after framing y(t) according to the frame shift and frame length, that is, y(mL+n) is the local representation of y(t) after frame division. Y is the complex time spectrum. Frame index, used to identify different speech frames; Here, k represents a frequency point, indicating a different frequency component, where k = 0, 1, 2, ..., K-1; This refers to the frame shift of the complex spectrum conversion module, i.e., the interval between two adjacent frames; The Hamming window is used to reduce spectral leakage. This is the number of points for the Fast Fourier Transform (FFT), which can be manually set to 256. After the STFT transformation, a complex spectrum (i.e., the complex time spectrum of the original audio to be processed) is obtained, which contains the real part. and the virtual part Through the above steps, a complex spectrum conversion module was designed. In order to obtain the features of speech signals at different scales from coarse to fine, and to more comprehensively describe the characteristics of speech, so as to realize the progressively refined analysis and processing of signals, a three-scale cascaded complex spectrum conversion module was designed.

[0031] The speech phase information of the original audio to be processed contains the relative relationship between the time and frequency changes of the signal, which is of great value to the time-domain waveform and perceptual quality of the speech. This application uses a complex feature extraction network to convert the speech signal of the original audio to be processed into a complex time-frequency domain representation (i.e., a complex time-spectrum) and retain the complex spectrum. By retaining the real and imaginary parts of the complex time-spectrum and without performing amplitude-phase separation, subsequent processing can jointly optimize amplitude and phase, thereby improving the reconstruction quality.

[0032] It should be noted that the calculation methods of the first, second, and third complex spectrum conversion modules are similar, but the input signals of each module are different: Figure 2 The first complex spectrum conversion module in The input is a noisy speech frame signal. (i.e., the audio signal characteristics corresponding to the frame audio of the original audio to be processed), second complex spectrum conversion module The input is Processed first complex time spectrum The third complex spectrum conversion module The input is Processed second complex time spectrum Before input signals can be processed, they must be segmented into frames according to the frame length and frame shift of their respective modules.

[0033] Specifically, the first complex spectrum conversion module can perform frame-by-frame processing on the original audio to be processed according to the frame length and frame shift corresponding to the first complex spectrum conversion module, to obtain audio signal features corresponding to several frames of audio. The first complex spectrum conversion module can multiply the audio signal features corresponding to each frame of audio by a Hamming window function and then perform a short-time Fourier transform to obtain the first complex time spectrum corresponding to the frame of audio. For example, using the frame length of the first complex spectrum conversion module... Frame shift The frequency domain is divided into two sub-bands with a coarser granularity; the input noisy speech signal is framed. (i.e., the original audio to be processed), according to frame length and frame shift Perform frame segmentation; multiply each frame signal by a Hamming window function. To reduce spectral leakage, the speech signal after frame-windowing can be calculated as follows:

[0034]

[0035] in Here, n is the frame index, representing the nth sample point. y frame The audio signal is framed and windowed. Then, a Short-Time Fourier Transform (STFT) is performed on each windowed frame (i.e., the framed and windowed audio signal), as shown in the formula:

[0036]

[0037] Obtaining the spectrum of complex numbers Its dimensions are ,in, For frame number, This represents the number of frequency points. , This indicates a floor operation. S represents the total number of sample points in the signal. S is calculated as sampling rate × duration.

[0038] set up The number of points in the FFT. For module The frame length of (i.e., the first complex spectrum conversion module) is then The formula that satisfies this condition is: .

[0039] The second complex spectrum conversion module can perform frame-based processing on the first complex time spectrum corresponding to the original audio to be processed, based on the frame length and frame shift corresponding to the second complex spectrum conversion module, to obtain audio signal features corresponding to several frames of audio. The second complex spectrum conversion module can then multiply the audio signal features corresponding to each frame of audio by a Hamming window function and perform a short-time Fourier transform to obtain the second complex time spectrum corresponding to that frame of audio. It should be noted that the processing procedure of the second complex spectrum conversion module is the same as that of the first complex spectrum conversion module, and will not be repeated here. Similarly, the second complex spectrum conversion module... Design: Frame length using the second complex spectrum conversion module Frame shift The medium-granularity frequency domain is divided into 4 sub-bands. The second complex spectrum conversion module receives data from the first complex spectrum conversion module. Output The new complex time spectrum is obtained. Its dimensions are ,in For frame number, This represents the number of frequency points.

[0040] The third complex spectrum conversion module performs frame-by-frame processing on the second complex time spectrum corresponding to the original audio to be processed, based on the frame length and frame shift corresponding to the module, to obtain audio signal features corresponding to several frames of audio. The third complex spectrum conversion module can multiply the audio signal features corresponding to each frame of audio by a Hamming window function and then perform a short-time Fourier transform to obtain the third complex time spectrum corresponding to that frame of audio. It should be noted that the processing procedure of the third complex spectrum conversion module is the same as that of the first complex spectrum conversion module, and will not be repeated here. Use frame length Frame shift The fine frequency domain is divided into 8 sub-bands. The third complex spectrum conversion module receives data from the second complex spectrum conversion module. Output Following the above method, the complex time spectrum is obtained. Its dimensions are ,in For frame number, This represents the number of frequency points.

[0041] The entire third complex time spectrum corresponding to the original audio to be processed is taken as the complex time spectrum of the original audio to be processed. That is, the above three complex spectrum conversion modules can be cascaded to finally obtain the complex time spectrum after progressive refinement. .

[0042] It should be noted that the complex spectrum obtained through STFT transformation, combined with a Hamming window to reduce spectral leakage, prevents biases in frequency component analysis that could affect the accurate judgment of speech features. This time-frequency analysis and complex feature extraction method can more fully capture the rich information of the speech signal in the time and frequency domains, providing data for the model to learn speech features in depth. Compared with methods that rely solely on the amplitude spectrum, the complex spectrum, which preserves phase information, allows the model to reconstruct the speech signal more accurately, reduces speech distortion, improves the clarity and naturalness of the reconstructed speech, and enhances the model's ability to understand and process speech signals.

[0043] S102: Input the complex time spectrum of the original audio to be processed into the complex convolutional encoder. The complex convolutional encoder obtains the encoding features of the original audio to be processed based on the multi-level time-frequency features of the complex time spectrum.

[0044] In this embodiment, the complex convolution encoder includes a first complex convolution module, a second complex convolution module, a third complex convolution module, and a fourth complex convolution module; wherein, as... Figure 2 As shown, the first complex convolution module, the second complex convolution module, the third complex convolution module, and the fourth complex convolution module all include a complex convolution layer ComplexConv2D, a complex batch normalization layer ComplexBN, and a max pooling layer Maxpod.

[0045] In this embodiment, the complex convolutional encoder obtains the coding features of the original audio to be processed based on the multi-level time-frequency characteristics of the complex time-spectrum. Specifically, the complex time-spectrum is input into the first complex convolution module to obtain a first coding feature E1; the first coding feature E1 is input into the second complex convolution module to obtain a second coding feature E2; the second coding feature E2 is input into the third complex convolution module to obtain a third coding feature E3; and the third coding feature E3 is input into the fourth complex convolution module to obtain a fourth coding feature E4. The time-frequency feature levels corresponding to the first, second, third, and fourth coding features are all different. The complex convolutional layers in the first, second, third, and fourth complex convolutional modules are all used to simultaneously process the real and imaginary parts of the input data to obtain the features of the complex domain of the input data. The complex batch normalization layers in these modules are used to accelerate model convergence and improve model stability. The max pooling layers in these modules are used to progressively compress the time-frequency resolution and extract high-level abstract features. It should be noted that the input data is one of the complex time-frequency spectrum, the first encoded feature, the second encoded feature, and the third encoded feature.

[0046] Understandably, complex convolutional encoders can extract multi-level time-frequency features for subsequent, more in-depth speech analysis. For example... Figure 2 As shown, the input to the complex convolutional encoder is the complex time spectrum of the original audio to be processed, which is finally obtained by S101. Let it be denoted as E0, and the output is the encoded feature. To preserve the properties of the complex spectrum, this step proposes using a complex convolutional layer. Specifically, a complex convolutional layer (ComplexConv2D) is used to process the input, as shown in the following formula:

[0047]

[0048] Where l=1,2,3,4, the kernel weights of the complex convolutional layer ComplexConv2D. W is the kernel weight of the complex convolutional layer (ComplexConv2D), which is a complex number; For its part, The imaginary part is represented by a complex convolutional kernel. Complex convolutional kernels can process both the real and imaginary parts of the input complex spectrum simultaneously, thus better preserving and utilizing the features of the speech signal in the complex domain. Complex convolutional layers can process both real and imaginary information simultaneously. The number of input convolutional kernels is doubled layer by layer (16, 32, 64, 128), prompting the model to learn richer features. Each layer is followed by Complex Batch Normalization (ComplexBN) to accelerate model convergence and improve model stability, followed by 2×2 max pooling to progressively compress the time-frequency resolution and extract high-level abstract features.

[0049] It should be noted that the complex convolutional encoder is designed using complex convolutional layers to process the complex spectrum, fully considering the characteristics of speech signals in the complex domain. Traditional real convolutions struggle to effectively model the correlation between real and imaginary parts in the time-frequency domain, while the weights of complex convolutional kernels, composed of both real and imaginary parts, can process both information simultaneously, resulting in more accurate extraction of multi-level time-frequency features of speech. As the network layers deepen, the number of input channels doubles layer by layer, enabling the model to learn rich and complex features covering different frequencies and time scales. Complex batch normalization (ComplexBN) is employed to normalize the input data, reducing internal covariate bias.

[0050] like Figure 2 As shown, the second complex convolution module and the third complex convolution module can each be connected to a Transformer encoder. Accordingly, the method may further include the following steps:

[0051] The second encoded feature is split into real and imaginary parts and concatenated into a real tensor; the real tensor is compressed in dimension by linear projection and input into the Transformer encoder connected to the second complex convolution module to obtain the enhanced feature corresponding to the second encoded feature.

[0052] The third encoded feature is split into real and imaginary parts and concatenated into a real tensor; the real tensor is compressed in dimension by linear projection and input into a Transformer encoder connected to the third complex convolution module to obtain the enhanced feature corresponding to the third encoded feature.

[0053] This embodiment enhances long-range dependency modeling across frames, improving the ability to suppress non-stationary noise. The input to the Transformer encoder is the encoded features of the original audio to be processed. The output is the enhanced features corresponding to the encoded features of the original audio to be processed. Traditional U-Net skip connections only transmit features and lack global interaction, failing to effectively capture cross-frame contextual dependencies. To effectively capture cross-frame contextual dependencies, this step proposes inserting lightweight Transformer modules into the U-Net skip connections. Specifically, this involves inserting lightweight Transformer modules into the outputs of the encoder's 2nd and 3rd layers (i.e., the second and third complex convolutional modules). , Perform the following processing:

[0054] The Transformer encoders linked to the second and third complex convolution modules all process the data according to the following steps: First, the encoded features are split into real and imaginary parts and concatenated into a real tensor. This transforms complex features into real tensors suitable for processing by the Transformer encoder. Then, the dimensions are compressed using linear projection and input into the Transformer encoder, with the following formula:

[0055]

[0056] The Transformer is designed with 4-head attention, a feedforward dimension of 512, and learnable parameters for positional encoding. The multi-head attention mechanism allows the model to learn features from different subspaces, the feedforward dimension setting controls the model's expressive power, and the learnable positional encoding allows the model to better capture positional information within the sequence.

[0057] It's important to note that traditional U-Net skip connections only transmit features and lack global interaction capabilities, making it difficult to capture cross-frame contextual dependencies. Designing Transformer skip connections addresses this by recognizing the time-series nature of speech signals and the close connections between frames, especially crucial when dealing with non-stationary noise. Inserting a lightweight Transformer module into the U-Net skip connections effectively solves the shortcomings of traditional U-Net skip connections. By splitting complex features into real and imaginary parts and concatenating them into a real tensor, then compressing the dimensions through linear projection before inputting it into the Transformer encoder, the Transformer is able to enhance the learning of long-distance cross-frame dependencies, more accurately distinguish between speech and noise, effectively suppress non-stationary noise, improve the quality of denoised speech, and enhance the model's adaptability in complex noisy environments.

[0058] S103: Input the encoded features of the original audio to be processed into the complex convolutional decoder. The complex convolutional decoder obtains the noise-reduced frequency corresponding to the original audio by upsampling the encoded features layer by layer and splicing the corresponding layer encoded features.

[0059] In this embodiment, the complex convolutional decoder includes a first upsampling layer UpSample (i.e., us), a second upsampling layer UpSample (i.e., us), a third upsampling layer UpSample (i.e., us), a first complex convolutional layer ComplexConv (i.e., CConv), a second complex convolutional layer ComplexConv (i.e., CConv), a third complex convolutional layer ComplexConv (i.e., CConv), and a fourth complex convolutional layer ComplexConv (i.e., CConv).

[0060] The complex convolutional decoder obtains the noise-reduced frequency corresponding to the original audio by upsampling the encoded features layer by layer and concatenating the corresponding layer encoded features. The specific implementation of this method is described below. For example... Figure 3 As shown, the fourth encoded feature E4 is input as input D4 to the first upsampling layer to obtain the first upsampling feature; the first upsampling feature and the enhanced feature corresponding to the third encoded feature are then combined. The first concatenated feature is obtained by concatenating the features along the channel dimension; the first concatenated feature is then input into the first complex convolutional layer to obtain the first complex domain feature D3; the first complex domain feature D3 is then input into the second upsampling layer to obtain the second upsampling feature; the second upsampling feature and the corresponding enhancement feature of the second encoded feature are then combined. The second concatenated feature is obtained by concatenating the features along the channel dimension; the second concatenated feature is then input into the second complex convolutional layer to obtain the second complex domain feature D2; the second complex domain feature D2 is then input into the third upsampling layer to obtain the third upsampling feature; the third upsampling feature and the first encoded feature are concatenated along the channel dimension to obtain the third concatenated feature; the third concatenated feature is then input into the third complex convolutional layer to obtain the third complex domain feature D1; the third complex domain feature D1 is then input into the fourth complex convolutional layer to obtain the denoised complex spectrum; and the denoised frequency corresponding to the original audio to be processed is obtained based on the denoised complex spectrum.

[0061] Understandably, this step is designed to reconstruct the denoised complex spectrum. The input to this step is the encoded features. , , , The output is a denoised complex spectrum. Existing decoders may lose high-frequency detail information during reconstruction and cannot guarantee the phase consistency of the output spectrum. To reconstruct a high-quality denoised complex spectrum, this step proposes a method of upsampling layer by layer and concatenating the corresponding layer's encoded features.

[0062] Specifically, the features are upsampled layer by layer and concatenated with the corresponding layer's encoded features, as shown in the formula:

[0063]

[0064] ComplexConv is a complex convolution operation that performs convolution operations on the concatenated complex features to extract features. Concat is a concatenation operation that performs convolution operations on the upsampled complex features. Features and Features are concatenated along the channel dimension. UpSample is an upsampling operation, which... Features are upsampled to increase the feature map size. The value of l is 2 or 3.

[0065] In this process, the number of channels in D4-D1 is halved layer by layer from 128 to 16, which gradually restores the feature dimensionality. The final layer uses a 1×1 complex convolution ComplexConv (denoted as CConv in the diagram) on D1 to output the final layer. and Complex convolution is used to ensure phase consistency of the output spectrum. The schematic diagram is shown below, where US represents UpSample. Finally, the noise-reduced frequency corresponding to the original audio to be processed is obtained. .

[0066] It should be noted that the method of designing a complex convolutional decoder, which upsamples layer by layer and concatenates the encoded features of corresponding layers, effectively solves the problems existing in existing decoders. Existing decoders are prone to losing high-frequency detail information during reconstruction, resulting in a decrease in the clarity and naturalness of the reconstructed speech. Our method, by progressively upsampling from low-resolution feature maps to the original resolution, and simultaneously concatenating the encoded features of corresponding layers, fully utilizes feature information at different levels and scales in the encoder, which helps to recover high-frequency details of the speech, making the reconstructed speech clearer and more realistic. This step effectively improves the performance of the entire denoising algorithm and enhances speech quality.

[0067] It should be noted that the loss function of the audio denoising model is... Among them, L total L represents the total loss value. STFT L is the frequency domain loss value. wave This represents the time-domain loss value. This step is designed to jointly optimize the time-domain waveform and frequency-domain features, allowing the model to better fit real speech in both the time and frequency domains. The input to this step is the predicted speech. (i.e., noise-reduced audio) and real speech (i.e., the original audio to be processed), the output is the loss value. Existing loss functions may only focus on one aspect in the time or frequency domain, failing to comprehensively optimize the model. To jointly optimize time and frequency domain features, this step proposes a method combining multi-resolution STFT loss and time-domain L1 loss.

[0068] Specifically, the multi-resolution STFT loss formula is as follows:

[0069]

[0070] in For different FFT sizes, the corresponding frequency points in S101 k=0,...,K-1. Different The values ​​correspond to different frequencies. It's real voice. The frequency domain amplitude value obtained after performing a short-time Fourier transform (STFT) at a specific FFT size K. It is the representation of the real speech signal at the corresponding FFT size after converting it from the time domain to the frequency domain. It is predictive speech The frequency domain amplitude value obtained after performing a Short-Time Fourier Transform (STFT) at a specific FFT size K. This represents the frequency domain representation of the predicted speech signal after STFT transformation at the corresponding FFT size. By using different FFT sizes, frequency domain features can be constrained at multiple resolutions. This formula indicates that STFT transformations are first performed on the real and predicted speech at these three FFT sizes to obtain frequency domain representations at different resolutions, and then the loss value is calculated. ||·||1 represents the L1 norm, which is the sum of the absolute values ​​of all elements of the vector.

[0071] The time-domain L1 loss formula is:

[0072]

[0073] L1 loss can preserve temporal details and avoid over-smoothing.

[0074] The total loss formula is:

[0075]

[0076] By weighted summing of the two, the characteristics in both the time and frequency domains can be optimized simultaneously.

[0077] It's important to note that traditional loss functions often focus only on one aspect of the time or frequency domain, failing to comprehensively optimize the model. This method combines multi-resolution STFT loss and temporal L1 loss to overcome the limitations of traditional loss functions. Multi-resolution STFT loss constrains frequency domain features at multiple resolutions using different FFT sizes. Different FFT sizes capture the feature differences of speech signals at different frequency resolutions, enabling the model to more accurately fit the frequency domain characteristics of real speech and avoid the loss or deviation of frequency domain information. Temporal L1 loss preserves temporal details, avoids over-smoothing, and prevents speech from losing its original clarity and naturalness. By weighted joint optimization of time and frequency domain features, the algorithm can more comprehensively fit real speech, improving the model's generalization ability, enhancing denoising performance and speech quality, and ensuring that the model can stably output high-quality denoised speech in various scenarios.

[0078] Next, we will introduce the model training process of the audio denoising model. To generate diverse training data, this step proposes a method of mixing speech and noise according to the signal-to-noise ratio (SNR) to construct a noisy speech-clean speech pairing dataset, providing a data foundation for subsequent model training of the audio denoising model. The input to this step is a clean speech library and a noisy speech library (such as NOISEX-92) recorded in a noise-free environment, and the output is a noisy speech waveform. and corresponding clean speech waveform .

[0079] Specifically, firstly, clean speech is randomly selected from the speech database. Randomly select environmental noise from the noise database Examples include white noise and bubble noise. Next, noisy speech is generated by mixing these noises according to their signal-to-noise ratio (SNR), using the following formula:

[0080]

[0081] in This is the noise scaling factor, which controls the intensity of the noise. It is adjusted... It can achieve different signal-to-noise ratios, thereby better simulating various real-world noise environments. Noisy speech waveform Clean speech waveform : Environmental noise randomly selected from the noise library.

[0082] Then, the speech and noise are standardized, and... The amplitude is normalized to [-1,1], thereby ensuring that the data are on the same scale, which facilitates model learning.

[0083] Finally, a sliding window is used for frame segmentation, with a frame length of 25ms and a frame shift of 10ms, to obtain the noisy speech frame signal. (i.e., noisy speech waveform) This facilitates subsequent time-frequency analysis of the speech.

[0084] As can be seen, the data preprocessing method designed in this application, by mixing speech and noise according to the signal-to-noise ratio to simulate various noise environments, can effectively improve the diversity of training data. This allows the model to be exposed to a rich variety of noisy speech samples during training, fully learning the characteristics and relationships between speech and noise under different noise intensities, thereby enhancing the model's adaptability to various complex noise environments. Framing provides a foundation for subsequent time-frequency analysis. On the one hand, an appropriate frame length setting can completely preserve the local features of speech, such as the pronunciation features of phonemes; on the other hand, a small frame shift can ensure a certain overlap between adjacent frames, reducing information loss and enabling the model to capture the dynamic changes of speech in the time dimension more meticulously, laying the foundation for accurately extracting time-frequency features and achieving precise denoising.

[0085] As can be seen from the above technical solution, the audio denoising method provided in this application is specifically designed for pre-trained audio denoising models, which encompass a complex feature extraction network, a complex convolutional encoder, and a complex convolutional decoder. In practical applications, the original audio to be processed first enters the complex feature extraction network, which can extract features of the audio signal from different scales and generate a complex time-frequency spectrum accordingly. This process allows the rich information of the audio data in the time-frequency domain to be fully captured. Subsequently, the complex time-frequency spectrum is input to the complex convolutional encoder. The encoder, leveraging its ability to analyze the multi-level time-frequency features of the complex time-frequency spectrum, outputs the encoded features of the audio to be processed. This step enables the model to learn complex features at different frequencies and time scales. Finally, the complex convolutional decoder receives the encoded features, upsamples layer by layer, and combines the encoded features of the corresponding layers to generate the denoised audio. This method achieves direct processing of time-frequency features in the complex domain, simultaneously optimizing amplitude and phase information, and significantly improving the quality of audio reconstruction. Specifically, the complex time-frequency spectrum that preserves phase information enables the model to reconstruct speech signals more accurately, effectively reducing speech distortion and making the reconstructed speech clearer and more natural, thereby enhancing the model's ability to understand and process speech signals. Furthermore, by extracting multi-level time-frequency features, the model learns the complex features of speech signals more comprehensively. The layer-by-layer upsampling and splicing of corresponding layer-encoded features not only solves the shortcomings of existing decoders but also fully utilizes feature information at different levels and scales, helping to recover high-frequency details of speech. This, in turn, improves the clarity and realism of the reconstructed speech. Overall, this method can effectively separate clean speech, effectively remove noise and preserve speech features in complex noise environments, improve the effect of audio denoising, and enhance the quality of the denoised audio.

[0086] like Figure 4 The image shows a specific embodiment of an audio denoising apparatus provided in this application. The apparatus described in this embodiment is a physical device used to perform the method described in the above embodiments. Its technical solution is essentially the same as that of the above embodiments, and the corresponding descriptions in the above embodiments are also applicable to this embodiment. The apparatus is applied to a pre-trained audio denoising model, which includes a complex feature extraction network, a complex convolutional encoder, and a complex convolutional decoder. The apparatus in this embodiment includes:

[0087] The first unit 401 is used to input the original audio to be processed into the complex feature extraction network, the complex feature extraction network extracts audio signal features at multiple different scales from the original audio to be processed, and obtains the complex time spectrum of the original audio to be processed based on the audio signal features at multiple different scales.

[0088] The second unit 402 is used to input the complex time spectrum of the original audio to be processed into the complex convolutional encoder, and the complex convolutional encoder obtains the encoding features of the original audio to be processed based on the multi-level time-frequency features of the complex time spectrum.

[0089] The third unit 403 is used to input the encoded features of the original audio to be processed into the complex convolutional decoder. The complex convolutional decoder obtains the noise-reduced frequency corresponding to the original audio to be processed by upsampling the encoded features layer by layer and splicing the corresponding layer encoded features.

[0090] Optionally, the complex feature extraction network includes a first complex spectrum conversion module, a second complex spectrum conversion module, and a third complex spectrum conversion module; the frame length and frame shift of the first complex spectrum conversion module, the second complex spectrum conversion module, and the third complex spectrum conversion module are all different.

[0091] The first unit 401 is used for:

[0092] The first complex spectrum conversion module performs frame-by-frame processing on the original audio to be processed according to the frame length and frame shift corresponding to the first complex spectrum conversion module, and obtains audio signal features corresponding to several frames of audio.

[0093] The first complex spectrum conversion module multiplies the audio signal features corresponding to each frame of audio by a Hamming window function and then performs a short-time Fourier transform to obtain the first complex time spectrum corresponding to the frame of audio.

[0094] The second complex spectrum conversion module performs frame-segmentation processing on the first complex time spectrum corresponding to the original audio to be processed according to the frame length and frame shift corresponding to the second complex spectrum conversion module, and obtains audio signal features corresponding to several frames of audio.

[0095] The second complex spectrum conversion module multiplies the audio signal features corresponding to each frame of audio by a Hamming window function and then performs a short-time Fourier transform to obtain the second complex time spectrum corresponding to the frame of audio.

[0096] The third complex spectrum conversion module performs frame-segmentation processing on the second complex time spectrum corresponding to the original audio to be processed according to the frame length and frame shift corresponding to the third complex spectrum conversion module, and obtains audio signal features corresponding to several frames of audio.

[0097] The third complex spectrum conversion module multiplies the audio signal features corresponding to each frame of audio by a Hamming window function and then performs a short-time Fourier transform to obtain the third complex time spectrum corresponding to the frame of audio.

[0098] The entire third complex time spectrum corresponding to the original audio to be processed is taken as the complex time spectrum of the original audio to be processed.

[0099] Optionally, the complex convolution encoder includes a first complex convolution module, a second complex convolution module, a third complex convolution module, and a fourth complex convolution module; wherein the first complex convolution module, the second complex convolution module, the third complex convolution module, and the fourth complex convolution module each include a complex convolution layer, a complex batch normalization layer, and a max pooling layer.

[0100] Optionally, the second unit 402 is used for:

[0101] The complex time-frequency spectrum is input into the first complex convolution module to obtain the first encoded feature;

[0102] The first encoded feature is input into the second complex convolution module to obtain the second encoded feature;

[0103] The second encoded feature is input into the third complex convolution module to obtain the third encoded feature;

[0104] The third encoding feature is input into the fourth complex convolution module to obtain the fourth encoding feature;

[0105] Among them, the time-frequency feature levels corresponding to the first coding feature, the second coding feature, the third coding feature and the fourth coding feature are all different;

[0106] The complex convolutional layers in the first, second, third, and fourth complex convolutional modules are all used to simultaneously process the information of the real and imaginary parts of the input data to obtain the features of the complex domain of the input data;

[0107] The complex batch normalization layers in the first complex convolution module, the second complex convolution module, the third complex convolution module, and the fourth complex convolution module are all used to accelerate model convergence and improve model stability.

[0108] The max pooling layers in the first, second, third, and fourth complex convolution modules are all used to progressively compress the time-frequency resolution and extract high-level abstract features.

[0109] The input data is one of the complex time spectrum, the first coding feature, the second coding feature, and the third coding feature.

[0110] Optionally, the second complex convolution module and the third complex convolution module are each connected to a Transformer encoder.

[0111] Optionally, the device further includes a fourth unit for:

[0112] The second encoded feature is split into real and imaginary parts and concatenated into a real tensor; the real tensor is compressed in dimension by linear projection and input into the Transformer encoder connected to the second complex convolution module to obtain the enhanced feature corresponding to the second encoded feature;

[0113] The third encoded feature is split into real and imaginary parts and concatenated into a real tensor; the real tensor is compressed in dimension by linear projection and input into a Transformer encoder connected to the third complex convolution module to obtain the enhanced feature corresponding to the third encoded feature.

[0114] Optionally, the complex convolutional decoder includes a first upsampling layer, a second upsampling layer, a third upsampling layer, a first complex convolutional layer, a second complex convolutional layer, a third complex convolutional layer, and a fourth complex convolutional layer; the third unit 403 is used for:

[0115] The fourth encoded feature is input into the first upsampling layer to obtain the first upsampling feature;

[0116] The enhanced features corresponding to the first upsampled feature and the third encoded feature are concatenated along the channel dimension to obtain the first concatenated feature;

[0117] The first concatenated feature is input into the first complex convolutional layer to obtain the first complex domain feature;

[0118] The first complex domain feature is input into the second upsampling layer to obtain the second upsampling feature;

[0119] The second upsampled feature and the enhanced feature corresponding to the second encoded feature are concatenated along the channel dimension to obtain the second concatenated feature;

[0120] The second concatenated feature is input into the second complex convolutional layer to obtain the second complex domain feature;

[0121] The second complex domain feature is input into the third upsampling layer to obtain the third upsampling feature;

[0122] The third upsampling feature and the first encoded feature are concatenated along the channel dimension to obtain the third concatenated feature;

[0123] The third concatenated feature is input into the third complex convolutional layer to obtain the third complex domain feature;

[0124] The third complex domain feature is input into the fourth complex convolutional layer to obtain the denoised complex spectrum;

[0125] Based on the denoised complex spectrum, the denoised frequency corresponding to the original audio to be processed is obtained.

[0126] Optionally, the loss function of the audio denoising model is: Among them, L total L represents the total loss value. STFT L is the frequency domain loss value. wave This represents the time-domain loss value.

[0127] In this way, the device achieves direct processing of time-frequency features in the complex domain, simultaneously optimizing amplitude and phase information, and significantly improving the quality of audio reconstruction. Specifically, the complex time-spectrum, which preserves phase information, enables the model to reconstruct speech signals more accurately, effectively reducing speech distortion and making the reconstructed speech clearer and more natural, thereby enhancing the model's ability to understand and process speech signals. Furthermore, by extracting multi-level time-frequency features, the model learns more comprehensively the complex features of speech signals. The layer-by-layer upsampling and splicing of corresponding layer-coded features not only solves the shortcomings of existing decoders but also fully utilizes feature information at different levels and scales, helping to recover high-frequency details of speech, thus improving the clarity and realism of the reconstructed speech. Overall, this method can effectively separate clean speech, effectively remove noise and preserve speech features in complex noise environments, improve the effect of audio denoising, and enhance the quality of the denoised audio.

[0128] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include RAM, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk storage device. Of course, the electronic device may also include other hardware required for other services.

[0129] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0130] Memory is used to store instructions for execution. Specifically, instructions for execution are computer programs that can be executed. Memory can include main memory and non-volatile memory, and it provides the processor with execution instructions and data.

[0131] In one possible implementation, the processor reads the corresponding execution instructions from non-volatile memory into memory and then executes them. Alternatively, it may obtain the corresponding execution instructions from other devices to form an audio denoising device at the logical level. The processor executes the execution instructions stored in memory to implement the audio denoising method provided in any embodiment of this application through the executed instructions.

[0132] The above is as stated in this application. Figure 1The audio noise reduction device provided in the illustrated embodiment can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0133] The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0134] This application also proposes a readable medium that stores execution instructions. When the stored execution instructions are executed by the processor of an electronic device, the electronic device can execute the audio denoising method provided in any embodiment of this application, and specifically perform the above-mentioned evaluation method.

[0135] The electronic devices described in the foregoing embodiments may be computers.

[0136] Those skilled in the art will understand that the embodiments of this application can be provided as methods or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or a combination of software and hardware.

[0137] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0138] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0139] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An audio noise reduction method, characterized in that, The method is applied to a pre-trained audio denoising model, which includes a complex feature extraction network, a complex convolutional encoder, and a complex convolutional decoder. The method includes: The original audio to be processed is input into the complex feature extraction network, which extracts audio signal features at multiple different scales from the original audio to be processed, and obtains the complex time spectrum of the original audio to be processed based on the audio signal features at multiple different scales. The complex time-frequency spectrum of the original audio to be processed is input into the complex convolutional encoder, and the complex convolutional encoder obtains the encoding features of the original audio to be processed based on the multi-level time-frequency features of the complex time-frequency spectrum. The encoded features of the original audio to be processed are input into the complex convolutional decoder. The complex convolutional decoder obtains the noise-reduced frequency corresponding to the original audio by upsampling the encoded features layer by layer and concatenating the encoded features of the corresponding layers. The complex convolution encoder includes a first complex convolution module, a second complex convolution module, a third complex convolution module, and a fourth complex convolution module; wherein, the first complex convolution module, the second complex convolution module, the third complex convolution module, and the fourth complex convolution module each include a complex convolution layer, a complex batch normalization layer, and a max pooling layer; The complex convolutional encoder obtains the encoded features of the original audio to be processed based on the multi-level time-frequency features of the complex time spectrum, including: The complex time-frequency spectrum is input into the first complex convolution module to obtain the first encoded feature; The first encoded feature is input into the second complex convolution module to obtain the second encoded feature; The second encoded feature is input into the third complex convolution module to obtain the third encoded feature; The third encoding feature is input into the fourth complex convolution module to obtain the fourth encoding feature; Among them, the time-frequency feature levels corresponding to the first coding feature, the second coding feature, the third coding feature and the fourth coding feature are all different; The complex convolutional layers in the first, second, third, and fourth complex convolutional modules are all used to simultaneously process the information of the real and imaginary parts of the input data to obtain the features of the complex domain of the input data; The complex batch normalization layers in the first complex convolution module, the second complex convolution module, the third complex convolution module, and the fourth complex convolution module are all used to accelerate model convergence and improve model stability. The max pooling layers in the first complex convolution module, the second complex convolution module, the third complex convolution module, and the fourth complex convolution module are all used to progressively compress the time-frequency resolution and extract high-level abstract features; The input data is one of the complex time spectrum, the first coding feature, the second coding feature, and the third coding feature; The second complex convolution module and the third complex convolution module are each connected to a Transformer encoder; The method further includes: The second encoded feature is split into real and imaginary parts and concatenated into a real tensor; the real tensor is compressed in dimension by linear projection and input into the Transformer encoder connected to the second complex convolution module to obtain the enhanced feature corresponding to the second encoded feature; The third encoded feature is split into real and imaginary parts and concatenated into a real tensor; the real tensor is compressed in dimension by linear projection and input into a Transformer encoder connected to the third complex convolution module to obtain the enhanced feature corresponding to the third encoded feature.

2. The method according to claim 1, characterized in that, The complex feature extraction network includes a first complex spectrum conversion module, a second complex spectrum conversion module, and a third complex spectrum conversion module; the first complex spectrum conversion module, the second complex spectrum conversion module, and the third complex spectrum conversion module each correspond to different frame lengths and frame shifts; The complex feature extraction network extracts audio signal features at multiple different scales from the original audio to be processed, and obtains the complex time spectrum of the original audio based on the multiple different scale audio signal features, including: The first complex spectrum conversion module performs frame-by-frame processing on the original audio to be processed according to the frame length and frame shift corresponding to the first complex spectrum conversion module, and obtains audio signal features corresponding to several frames of audio. The first complex spectrum conversion module multiplies the audio signal features corresponding to each frame of audio by a Hamming window function and then performs a short-time Fourier transform to obtain the first complex time spectrum corresponding to the frame of audio. The second complex spectrum conversion module performs frame-segmentation processing on the first complex time spectrum corresponding to the original audio to be processed according to the frame length and frame shift corresponding to the second complex spectrum conversion module, and obtains audio signal features corresponding to several frames of audio; The second complex spectrum conversion module multiplies the audio signal features corresponding to each frame of audio by a Hamming window function and then performs a short-time Fourier transform to obtain the second complex time spectrum corresponding to the frame of audio. The third complex spectrum conversion module performs frame-segmentation processing on the second complex time spectrum corresponding to the original audio to be processed according to the frame length and frame shift corresponding to the third complex spectrum conversion module, and obtains audio signal features corresponding to several frames of audio. The third complex spectrum conversion module multiplies the audio signal features corresponding to each frame of audio by a Hamming window function and then performs a short-time Fourier transform to obtain the third complex time spectrum corresponding to the frame of audio. The entire third complex time spectrum corresponding to the original audio to be processed is taken as the complex time spectrum of the original audio to be processed.

3. The method according to claim 1, characterized in that, The complex convolutional decoder includes a first upsampling layer, a second upsampling layer, a third upsampling layer, a first complex convolutional layer, a second complex convolutional layer, a third complex convolutional layer, and a fourth complex convolutional layer; The complex convolutional decoder obtains the noise-reduced frequency corresponding to the original audio to be processed by upsampling the encoded features layer by layer and concatenating the corresponding layer encoded features, including: The fourth encoded feature is input into the first upsampling layer to obtain the first upsampling feature; The enhanced features corresponding to the first upsampled feature and the third encoded feature are concatenated along the channel dimension to obtain the first concatenated feature; The first concatenated feature is input into the first complex convolutional layer to obtain the first complex domain feature; The first complex domain feature is input into the second upsampling layer to obtain the second upsampling feature; The second upsampled feature and the enhanced feature corresponding to the second encoded feature are concatenated along the channel dimension to obtain the second concatenated feature; The second concatenated feature is input into the second complex convolutional layer to obtain the second complex domain feature; The second complex domain feature is input into the third upsampling layer to obtain the third upsampling feature; The third upsampling feature and the first encoded feature are concatenated along the channel dimension to obtain the third concatenated feature; The third concatenated feature is input into the third complex convolutional layer to obtain the third complex domain feature; The third complex domain feature is input into the fourth complex convolutional layer to obtain the denoised complex spectrum; Based on the denoised complex spectrum, the denoised frequency corresponding to the original audio to be processed is obtained.

4. The method according to claim 1, characterized in that, The loss function of the audio denoising model is: ; in, L total This is the total loss value. L STFT This is the frequency domain loss value. L wave This represents the time-domain loss value.

5. An audio noise reduction device, characterized in that, The device is applied to a pre-trained audio denoising model, which includes a complex feature extraction network, a complex convolutional encoder, and a complex convolutional decoder. The device includes: The first unit is used to input the original audio to be processed into the complex feature extraction network, the complex feature extraction network extracts audio signal features at multiple different scales from the original audio to be processed, and obtains the complex time spectrum of the original audio to be processed based on the audio signal features at multiple different scales; The second unit is used to input the complex time spectrum of the original audio to be processed into the complex convolutional encoder, and the complex convolutional encoder obtains the encoding features of the original audio to be processed based on the multi-level time-frequency features of the complex time spectrum. The third unit is used to input the encoded features of the original audio to be processed into the complex convolutional decoder. The complex convolutional decoder obtains the noise-reduced frequency corresponding to the original audio by upsampling the encoded features layer by layer and concatenating the encoded features of the corresponding layers. The complex convolution encoder includes a first complex convolution module, a second complex convolution module, a third complex convolution module, and a fourth complex convolution module; wherein, the first complex convolution module, the second complex convolution module, the third complex convolution module, and the fourth complex convolution module each include a complex convolution layer, a complex batch normalization layer, and a max pooling layer; The complex convolutional encoder obtains the encoded features of the original audio to be processed based on the multi-level time-frequency features of the complex time spectrum, including: The complex time-frequency spectrum is input into the first complex convolution module to obtain the first encoded feature; The first encoded feature is input into the second complex convolution module to obtain the second encoded feature; The second encoded feature is input into the third complex convolution module to obtain the third encoded feature; The third encoding feature is input into the fourth complex convolution module to obtain the fourth encoding feature; Among them, the time-frequency feature levels corresponding to the first coding feature, the second coding feature, the third coding feature and the fourth coding feature are all different; The complex convolutional layers in the first, second, third, and fourth complex convolutional modules are all used to simultaneously process the information of the real and imaginary parts of the input data to obtain the features of the complex domain of the input data; The complex batch normalization layers in the first complex convolution module, the second complex convolution module, the third complex convolution module, and the fourth complex convolution module are all used to accelerate model convergence and improve model stability. The max pooling layers in the first complex convolution module, the second complex convolution module, the third complex convolution module, and the fourth complex convolution module are all used to progressively compress the time-frequency resolution and extract high-level abstract features; The input data is one of the complex time spectrum, the first coding feature, the second coding feature, and the third coding feature; The second complex convolution module and the third complex convolution module are each connected to a Transformer encoder; The device is also used for: The second encoded feature is split into real and imaginary parts and concatenated into a real tensor; the real tensor is compressed in dimension by linear projection and input into the Transformer encoder connected to the second complex convolution module to obtain the enhanced feature corresponding to the second encoded feature; The third encoded feature is split into real and imaginary parts and concatenated into a real tensor; the real tensor is compressed in dimension by linear projection and input into a Transformer encoder connected to the third complex convolution module to obtain the enhanced feature corresponding to the third encoded feature.

6. A readable medium, characterized in that, The readable medium includes execution instructions that, when executed by the processor of the electronic device, cause the electronic device to perform the method as described in any one of claims 1-4.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory storing execution instructions. When the processor executes the execution instructions stored in the memory, the processor performs the method as described in any one of claims 1-4.

Images