Audio noise reduction method, computer device and storage medium

Abstract

The application relates to an audio noise reduction method, a computer device, a storage medium and a computer program product. The method comprises the following steps: performing frequency division processing on to-be-reduced-noise audio data to obtain to-be-reduced-noise sub-audio data of the to-be-reduced-noise audio data; acquiring a spectrum feature of the to-be-reduced-noise sub-audio data; performing noise reduction processing on the spectrum feature through a pre-trained audio noise reduction model to obtain a noise-reduced spectrum feature of the to-be-reduced-noise sub-audio data; performing signal reconstruction processing on the noise-reduced spectrum feature to obtain noise-reduced sub-audio data corresponding to the to-be-reduced-noise sub-audio data; and combining the noise-reduced sub-audio data to obtain noise-reduced audio data corresponding to the to-be-reduced-noise audio data. The method can improve the audio noise reduction effect.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to an audio noise reduction method, computer device, storage medium, and computer program product. Background Technology

[0002] With the development of internet technology, various singing apps have emerged. However, due to the influence of non-professional equipment and recording environment, the singing recorded through these apps is more prone to noise, such as microphone friction noise and ambient background noise. Therefore, noise reduction of audio data is very important.

[0003] In traditional techniques, when denoising audio data, digital signal processing methods are mainly used, such as statistical signal processing, to extract clean audio data from the audio data. However, this denoising method only has a certain denoising effect on steady-state noise, and it is difficult to achieve ideal results for other more complex and variable background noise (such as non-steady-state noise), resulting in poor audio denoising effect. Summary of the Invention

[0004] Therefore, it is necessary to provide an audio noise reduction method, computer device, computer-readable storage medium, and computer program product that can improve the audio noise reduction effect in response to the above-mentioned technical problems.

[0005] Firstly, this application provides an audio noise reduction method. The method includes:

[0006] The noise reduction frequency data is divided into frequencies to obtain the noise reduction sub-audio data of the noise reduction frequency data;

[0007] Obtain the spectral characteristics of the sub-audio data to be denoised;

[0008] The spectral features are denoised using a pre-trained audio denoising model to obtain the denoised spectral features of the sub-audio data to be denoised.

[0009] The denoised spectral features are subjected to signal reconstruction processing to obtain the denoised sub-audio data corresponding to the sub-audio data to be denoised;

[0010] The noise-reduced sub-audio data are combined to obtain the noise-reduced audio data corresponding to the noise-reduced audio data.

[0011] In one embodiment, the step of performing noise reduction processing on the spectral features using a pre-trained audio noise reduction model to obtain the noise-reduced spectral features of the sub-audio data to be denoised includes:

[0012] The spectral features are extracted using a pre-trained audio denoising model to obtain the processed spectral features of the sub-audio data to be denoised.

[0013] The processed spectral features are subjected to frequency domain-dependent noise reduction processing to obtain the first noise-reduced spectral features of the sub-audio data to be denoised.

[0014] The first denoised spectral features are subjected to time-related denoising processing to obtain the second denoised spectral features of the sub-audio data to be denoised.

[0015] The second denoised spectral features are subjected to feature extraction processing to obtain the denoised spectral features of the sub-audio data to be denoised.

[0016] In one embodiment, the pre-trained audio noise reduction model includes a first convolutional layer, a first dilated convolutional network, a second dilated convolutional network, and a second convolutional layer connected in sequence. The first dilated convolutional network and the second dilated convolutional network each include multiple sub-dilated convolutional networks connected in sequence. Each sub-dilated convolutional network includes a dilated convolutional layer, a normalization layer, and an activation layer connected in sequence.

[0017] In one embodiment, obtaining the spectral features of the sub-audio data to be denoised includes:

[0018] The sub-audio data to be noise-reduced is processed by frame segmentation to obtain the noise-reducing frequency frames of the sub-audio data to be noise-reduced.

[0019] The frequency frame to be noise-reduced is subjected to feature extraction processing to obtain the spectral features of the frequency frame to be noise-reduced;

[0020] The spectral features of the frequency frames to be noise-reduced are combined to obtain the spectral features of the sub-audio data to be noise-reduced.

[0021] In one embodiment, the step of performing signal reconstruction processing on the denoised spectral features to obtain the denoised sub-audio data corresponding to the sub-audio data to be denoised includes:

[0022] The denoised spectral features of the frequency frame to be denoised are identified from the denoised spectral features.

[0023] The denoised spectral features of the frequency frame to be denoised are processed to obtain the denoised audio frame of the frequency frame to be denoised.

[0024] The denoised audio frames are combined to obtain the denoised sub-audio data corresponding to the sub-audio data to be denoised.

[0025] In one embodiment, the pre-trained audio noise reduction model is obtained in the following manner:

[0026] Obtain clean audio samples and the corresponding noisy frequency samples of the clean audio samples;

[0027] The spectral features of the noisy sub-audio samples of the noisy frequency samples are input into the audio denoising model to be trained to obtain the denoised spectral features of the noisy sub-audio samples.

[0028] The denoised spectral features of the noisy sub-audio sample are subjected to signal reconstruction processing to obtain the denoised sub-audio sample corresponding to the noisy sub-audio sample;

[0029] The denoised sub-audio samples are combined to obtain the denoised audio samples corresponding to the noisy frequency samples;

[0030] Based on the difference between the spectral characteristics of the denoised audio sample and the spectral characteristics of the clean audio sample, the audio denoising model to be trained is trained to obtain the pre-trained audio denoising model.

[0031] In one embodiment, training the audio denoising model to be trained based on the difference between the spectral features of the denoised audio sample and the spectral features of the clean audio sample to obtain the pre-trained audio denoising model includes:

[0032] The loss value is obtained based on the difference between the spectral characteristics of the denoised audio sample and the spectral characteristics of the clean audio sample;

[0033] The audio noise reduction model to be trained is trained based on the loss value until the training termination condition is met.

[0034] The trained audio denoising model that meets the training termination condition is identified as the pre-trained audio denoising model.

[0035] In one embodiment, the noisy frequency sample corresponding to the clean audio sample is obtained in the following manner:

[0036] Acquire noisy audio samples and preset signal-to-noise ratio;

[0037] According to the preset signal-to-noise ratio, the clean audio sample and the noisy audio sample are fused to obtain the noisy frequency sample corresponding to the clean audio sample.

[0038] Secondly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0039] The noise reduction frequency data is divided into frequencies to obtain the noise reduction sub-audio data of the noise reduction frequency data;

[0040] Obtain the spectral characteristics of the sub-audio data to be denoised;

[0041] The spectral features are denoised using a pre-trained audio denoising model to obtain the denoised spectral features of the sub-audio data to be denoised.

[0042] The denoised spectral features are subjected to signal reconstruction processing to obtain the denoised sub-audio data corresponding to the sub-audio data to be denoised;

[0043] The noise-reduced sub-audio data are combined to obtain the noise-reduced audio data corresponding to the noise-reduced audio data.

[0044] Thirdly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0045] The noise reduction frequency data is divided into frequencies to obtain the noise reduction sub-audio data of the noise reduction frequency data;

[0046] Obtain the spectral characteristics of the sub-audio data to be denoised;

[0047] The spectral features are denoised using a pre-trained audio denoising model to obtain the denoised spectral features of the sub-audio data to be denoised.

[0048] The denoised spectral features are subjected to signal reconstruction processing to obtain the denoised sub-audio data corresponding to the sub-audio data to be denoised;

[0049] The noise-reduced sub-audio data are combined to obtain the noise-reduced audio data corresponding to the noise-reduced audio data.

[0050] Fourthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0051] The noise reduction frequency data is divided into frequencies to obtain the noise reduction sub-audio data of the noise reduction frequency data;

[0052] Obtain the spectral characteristics of the sub-audio data to be denoised;

[0053] The spectral features are denoised using a pre-trained audio denoising model to obtain the denoised spectral features of the sub-audio data to be denoised.

[0054] The denoised spectral features are subjected to signal reconstruction processing to obtain the denoised sub-audio data corresponding to the sub-audio data to be denoised;

[0055] The noise-reduced sub-audio data are combined to obtain the noise-reduced audio data corresponding to the noise-reduced audio data.

[0056] The aforementioned audio noise reduction method, computer equipment, storage medium, and computer program product obtain sub-audio data to be denoised by frequency division of the audio data to be denoised; then, the spectral characteristics of the sub-audio data to be denoised are acquired, and the spectral characteristics are denoised using a pre-trained audio noise reduction model to obtain the denoised spectral characteristics of the sub-audio data to be denoised; next, the denoised spectral characteristics are reconstructed to obtain the denoised sub-audio data corresponding to the sub-audio data to be denoised; finally, the denoised sub-audio data are combined to obtain the denoised audio data corresponding to the audio data to be denoised. In this way, by performing frequency division processing on the audio data to be denoised, we can better capture human voices in the low-frequency band and noise in various frequency bands. This makes it easier to use a pre-trained audio denoising model to perform targeted denoising processing on the spectral features of each sub-audio data to be denoised. This facilitates more comprehensive denoising processing, avoids missing any noise to be processed, and results in higher audio quality of the denoised audio data reconstructed from the denoised spectral features of the sub-audio data to be denoised, thereby improving the denoising effect of the audio data to be denoised. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating an audio noise reduction method in one embodiment;

[0058] Figure 2 This is a flowchart illustrating the steps of denoising spectral features in one embodiment;

[0059] Figure 3 This is a schematic diagram of the structure of an audio noise reduction model in one embodiment;

[0060] Figure 4 This is a schematic diagram of a frequency frame to be noise-reduced in one embodiment;

[0061] Figure 5 This is a flowchart illustrating the training steps of an audio noise reduction model in one embodiment.

[0062] Figure 6 This is a flowchart illustrating an audio noise reduction method in another embodiment;

[0063] Figure 7 This is a flowchart illustrating the training method of an audio noise reduction model in one embodiment;

[0064] Figure 8 This is a flowchart illustrating the training method of the audio noise reduction model in another embodiment;

[0065] Figure 9 This is a flowchart illustrating the audio noise reduction method in yet another embodiment;

[0066] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] In one embodiment, such as Figure 1 As shown, an audio noise reduction method is provided. This embodiment illustrates the method's application to a terminal, but it is understood that the method can also be applied to a server, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. The server can be a standalone server or a server cluster consisting of multiple servers. In this embodiment, the method includes the following steps:

[0069] Step S101: Perform frequency division processing on the noise frequency data to be reduced to obtain the noise sub-audio data to be reduced.

[0070] The audio data to be noise-reduced refers to the audio data that needs noise reduction processing, and is generally represented by waveform signals, such as time-domain waveform signals. In practical scenarios, the audio data to be noise-reduced can refer to singing data acquired by audio acquisition devices (such as microphones), singing data recorded by singing applications installed on terminals, singing data stored in databases, or singing data obtained from servers.

[0071] It should be noted that the noise reduction frequency data can also refer to other forms of audio data. This application mainly uses the noise reduction frequency data as singing data as an example for illustration.

[0072] The singing data contains noise (such as microphone friction noise, ambient background noise, etc.) and accompaniment (such as instrumental accompaniment, original song accompaniment, etc.). Here, the accompaniment can be regarded as noise; therefore, this application mainly aims to eliminate noise and accompaniment in the singing data.

[0073] Generally, in vocal recording scenarios, the user's vocals need to undergo dynamic compression, vocal enhancement, and reverb processing before being mixed with the accompaniment for loudness equalization and mixing to create a complete singing performance. However, when recording with a microphone, in addition to the user's vocals, the accompaniment and ambient noise are also captured. When applying dynamic compression, vocal enhancement, and reverb, the accompaniment and noise significantly affect the recording, making it difficult to achieve the desired effect. This is especially true in accompaniment recordings, where the captured accompaniment (the first accompaniment) is played through a speaker and, due to different speakers and microphones, differs from the original, well-produced accompaniment (the second accompaniment). Furthermore, the vocals must be mixed with this original, well-produced accompaniment, resulting in a final singing performance with two layers of accompaniment.

[0074] Frequency division processing refers to decomposing the noise-reducing audio data into multiple noise-reducing sub-audio data, each corresponding to a different frequency band; that is, each noise-reducing sub-audio data is a sub-band, and different sub-bands correspond to different frequency bands. In practical scenarios, frequency division processing can be implemented using PQMF (Pseudo Quadrature Mirror Filter), or other frequency division techniques, such as wavelet decomposition.

[0075] It should be noted that, on the one hand, by performing frequency division processing on the audio data to be denoised, the data volume of each sub-audio data to be denoised is smaller than that of the audio data to be denoised, thereby reducing the amount of data processed by the audio denoising model each time and reducing the computational complexity of the audio denoising model, making the audio denoising model more lightweight. On the other hand, considering that more energy in human voices is concentrated in the low frequency range, and that different instruments in accompaniment are distributed in different frequency ranges, by decomposing the waveform signal of the audio data to be denoised into multiple sub-bands (i.e., sub-audio data to be denoised), targeted noise reduction processing can be performed on each sub-band, which is conducive to better eliminating accompaniment and noise in the audio, effectively improving the clarity and intelligibility of the audio, thereby improving the audio noise reduction effect.

[0076] Specifically, the terminal acquires the noise reduction frequency data through an audio acquisition device deployed on the terminal or through a separate audio acquisition device, or obtains the noise reduction frequency data from a database or server; then, the terminal uses a signal frequency division instruction to perform frequency division processing on the noise reduction frequency data to obtain multiple noise reduction sub-audio data of the noise reduction frequency data.

[0077] For example, a user records a song using a singing application deployed on a terminal and triggers a noise reduction request. Then, in response to the noise reduction request, the terminal obtains the user's recorded singing data and uses a pseudo-orthogonal mirror filter to decompose the waveform signal of the singing data into multiple sub-bands, such as four sub-bands. Alternatively, the terminal performs wavelet decomposition on the singing data to obtain multiple sub-bands.

[0078] Step S102: Obtain the spectral characteristics of the sub-audio data to be denoised.

[0079] Among them, spectral features are used to characterize the spectral information of the sub-audio data to be denoised, specifically including amplitude features and phase features. In practical scenarios, by performing FFT (fast Fourier transform) processing on the sub-audio data to be denoised, that is, transforming the time-domain waveform signal of the sub-audio data to be denoised from the time domain to the frequency domain, the spectral features of the sub-audio data to be denoised can be obtained.

[0080] Specifically, the terminal can obtain the spectral features of the audio data to be denoised by performing feature extraction processing. For example, the terminal can perform a Fourier transform on the audio data to be denoised to obtain its spectral features; alternatively, the terminal can input the audio data to be denoised into a pre-trained spectral feature extraction model for feature extraction processing to obtain its spectral features. The pre-trained spectral feature extraction model is a neural network model used to extract the spectral features of audio data.

[0081] Step S103: The spectral features are denoised using a pre-trained audio denoising model to obtain the denoised spectral features of the sub-audio data to be denoised.

[0082] Among them, a pre-trained audio denoising model is a neural network model used to denoise audio data, specifically referring to CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), DNN (Deep Neural Network), etc. Of course, a pre-trained audio denoising model can also refer to other lightweight neural network models, such as... Figure 3 The model shown.

[0083] In this context, denoising the spectral features refers to eliminating noise and accompaniment carried within the spectral features. The denoised spectral features are those obtained after denoising the original spectral features. It can be understood that for the same sub-audio data to be denoised, the denoised spectral features will contain less noise than the original spectral features.

[0084] Specifically, the terminal retrieves clean audio samples and their corresponding noisy frequency samples from the database. Based on these clean audio samples and their corresponding noisy frequency samples, it iteratively trains the audio denoising model to be trained, obtaining a trained audio denoising model, which serves as the pre-trained audio denoising model. Next, the terminal inputs the spectral features of the sub-audio data to be denoised into the pre-trained audio denoising model. The pre-trained audio denoising model then performs multiple denoising processes on the spectral features of the sub-audio data to be denoised, obtaining the denoised spectral features of the sub-audio data.

[0085] Step S104: Perform signal reconstruction processing on the denoised spectral features to obtain the denoised sub-audio data corresponding to the sub-audio data to be denoised.

[0086] Signal reconstruction processing refers to restoring the spectral characteristics of the denoised sub-audio data to the same format (i.e., waveform signal format) as the sub-audio data to be denoised, thereby obtaining the denoised sub-audio data corresponding to the sub-audio data to be denoised. It can be understood that the noise in the denoised sub-audio data is less than that in the sub-audio data to be denoised, and its clarity, signal fidelity, etc. are higher than those of the sub-audio data to be denoised.

[0087] The denoised sub-audio data, like the sub-audio data to be denoised, is also represented by waveform signals, such as time-domain waveform signals. In practical scenarios, by performing IFFT (Inverse Fast Fourier Transform) processing on the denoised spectral features, that is, transforming the denoised spectral features to the time domain, the denoised sub-audio data corresponding to the sub-audio data to be denoised can be obtained.

[0088] Specifically, the terminal performs signal reconstruction processing on the denoised spectral features through signal reconstruction instructions, that is, restores the denoised spectral features to the same format as the sub-audio data to be denoised, thereby obtaining denoised sub-audio data with less noise and higher clarity corresponding to the sub-audio data to be denoised.

[0089] For example, the terminal can perform an inverse Fourier transform on the denoised spectral features to obtain the denoised sub-audio data corresponding to the sub-audio data to be denoised. Alternatively, the terminal can input the denoised spectral features into a pre-trained signal reconstruction model, and the signal reconstruction model can perform signal reconstruction processing on the denoised spectral features to obtain the denoised sub-audio data corresponding to the sub-audio data to be denoised. The pre-trained signal reconstruction model is a neural network model used to reconstruct signals from the spectral features of audio data.

[0090] Step S105: Combine the noise-reduced sub-audio data to obtain the noise-reduced audio data corresponding to the noise-reduced audio data.

[0091] In this context, the noise-reduced audio data refers to audio data with less noise and a purer signal than the audio data to be noise-reduced, such as audio data with higher clarity, intelligibility, and signal fidelity.

[0092] Specifically, the terminal obtains the arrangement order of the denoised sub-audio data and combines them according to this order to obtain combined audio data with less noise and a purer signal, which serves as the denoised audio data corresponding to the noise-reducing frequency data. For example, the terminal combines several denoised sub-bands of the denoised sub-audio data through sub-band synthesis to finally obtain the denoised audio data corresponding to the noise-reducing frequency data.

[0093] Furthermore, the terminal can input the denoised sub-audio data into a pre-trained audio quality prediction model to predict the audio quality value of the denoised sub-audio data. If the audio quality values ​​of all the denoised sub-audio data are greater than the preset quality value, all the denoised sub-audio data are combined to obtain the denoised audio data corresponding to the audio data to be denoised. If the audio quality value of at least one denoised sub-audio data is less than or equal to the preset quality value, the denoised sub-audio data is used as the sub-audio data to be denoised, and steps S102 to S104 are repeated until the audio quality value of the finally obtained denoised sub-audio data is greater than the preset quality value.

[0094] In the aforementioned audio denoising method, frequency division is performed on the frequency data to be denoised to obtain sub-audio data to be denoised. Then, the spectral features of the sub-audio data to be denoised are acquired, and a pre-trained audio denoising model is used to denoise these features, resulting in denoised spectral features. Next, signal reconstruction processing is performed on the denoised spectral features to obtain denoised sub-audio data corresponding to the sub-audio data to be denoised. Finally, the denoised sub-audio data are combined to obtain the denoised audio data corresponding to the frequency data to be denoised. In this way, by performing frequency division on the frequency data to be denoised, lower-frequency human voices and noise in various frequency bands can be better captured. This facilitates targeted denoising of the spectral features of each sub-audio data to be denoised using the pre-trained audio denoising model, resulting in more comprehensive denoising and avoiding omissions of noise. Consequently, the audio quality of the denoised audio data reconstructed from the denoised spectral features of the sub-audio data to be denoised is higher, thus improving the denoising effect on the frequency data to be denoised.

[0095] In one embodiment, such as Figure 2As shown, step S103 above, through a pre-trained audio denoising model, performs denoising processing on the spectral features to obtain the denoised spectral features of the sub-audio data to be denoised, specifically including the following steps:

[0096] Step S201: Using a pre-trained audio denoising model, feature extraction is performed on the spectral features to obtain the processed spectral features of the sub-audio data to be denoised.

[0097] Here, the processed spectral features refer to the spectral features obtained after feature extraction. Feature extraction specifically refers to convolution processing; it can be understood that after feature extraction, the processed spectral features contain only the key spectral features.

[0098] Step S202: Perform frequency domain-dependent noise reduction processing on the processed spectral features to obtain the first noise-reduced spectral features of the sub-audio data to be denoised.

[0099] Among them, frequency domain-related noise reduction processing refers to noise reduction processing of the processed spectral features on the frequency domain axis. It mainly involves fully learning the information within the frame to achieve a better fidelity effect for harmonic components or overtones in the singing data, and can effectively suppress transient noise in the singing data (such as door opening sounds, keyboard sounds, saliva sounds, etc.).

[0100] Understandably, considering the presence of various harmonic components or overtones in vocal data, and the frequency-domain related noise reduction processing, which mainly involves fully learning the intra-frame information on the frequency domain axis, the larger field of view established on the frequency domain axis can better capture the correlation in the frequency domain, thus achieving a better fidelity effect for the harmonic components or overtones in the vocal data, while also having a strong ability to suppress transient noise in the vocal data.

[0101] The first denoised spectral feature refers to the spectral feature obtained after performing frequency-domain correlated denoising on the processed spectral feature. It can be understood that after frequency-domain correlated denoising, the first denoised spectral feature has high signal fidelity and contains less, or even no, transient noise.

[0102] Step S203: Perform time-related denoising processing on the first denoised spectral features to obtain the second denoised spectral features of the sub-audio data to be denoised.

[0103] Among them, time-related noise reduction processing refers to noise reduction processing on the spectral features after the first noise reduction on the time axis. It mainly involves fully learning the information between frames, which helps to better preserve the integrity of the singing and avoid swallowing sounds.

[0104] It is understandable that, considering the strong contextual relationships in singing data, such as lengthening vowels to make the singing more rhythmic, time-related noise reduction mainly involves learning the information between frames on the timeline. The larger field of view established on the timeline can better capture temporal correlations. By lengthening a word and retaining it for a longer period of time, it helps to better preserve the integrity of the singing and avoid swallowing sounds.

[0105] The second denoised spectral feature refers to the spectral feature obtained after applying time-related denoising processing to the first denoised spectral feature. It can be understood that after time-related denoising processing, the second denoised spectral feature exhibits higher vocal integrity, and its signal fidelity is further improved.

[0106] Step S204: Perform feature extraction processing on the second denoised spectral features to obtain the denoised spectral features of the sub-audio data to be denoised.

[0107] Here, the denoised spectral features refer to the spectral features obtained after feature extraction processing of the second denoised spectral features. The feature extraction processing here also refers to convolution processing; it can be understood that after this feature extraction processing, the denoised spectral features are further optimized, thereby further improving their signal fidelity, clarity, etc.

[0108] Specifically, the terminal inputs the spectral features of the sub-audio data to be denoised into a pre-trained audio denoising model. The audio denoising model first performs feature extraction on the spectral features to obtain key spectral features, which serve as the processed spectral features of the sub-audio data to be denoised. Then, frequency-domain related denoising is performed on the processed spectral features to suppress transient noise and other noises while maintaining good fidelity for harmonic components and overtones in the audio data, thus obtaining the first denoised spectral features of the sub-audio data to be denoised. Next, time-related denoising is performed on the first denoised spectral features to suppress noise while better preserving the integrity of the singing and avoiding swallowing sounds, thus obtaining the second denoised spectral features of the sub-audio data to be denoised. Finally, feature extraction is performed on the second denoised spectral features to obtain the target spectral features, which serve as the denoised spectral features of the sub-audio data to be denoised.

[0109] In this embodiment, after feature extraction of the spectral features, the resulting processed spectral features undergo two noise reduction processes. This results in cleaner final noise-reduced spectral features, which further enhances the audio noise reduction effect. Simultaneously, frequency-domain correlated noise reduction better captures frequency domain correlations, effectively reducing distortion of harmonic components or overtones in the audio data. Furthermore, time-correlated noise reduction better captures temporal correlations, helping to better preserve the integrity of the vocals and avoid swallowing sounds.

[0110] In one embodiment, such as Figure 3 As shown, the pre-trained audio noise reduction model includes a first convolutional layer, a first dilated convolutional network, a second dilated convolutional network, and a second convolutional layer connected in sequence. The first and second dilated convolutional networks each include multiple sub-dilated convolutional networks connected in sequence. Each sub-dilated convolutional network includes a dilated convolutional layer, a normalization layer, and an activation layer connected in sequence.

[0111] Among them, reference Figure 3 The first and second convolutional layers are both ordinary convolutional layers, mainly used for spectral feature extraction. For example, the first convolutional layer is mainly used to extract spectral features from the sub-audio data to be denoised, while the second convolutional layer is mainly used to extract the second denoised spectral features from the sub-audio data.

[0112] Among them, reference Figure 3 The first dilated convolutional network comprises multiple (e.g., six) sequentially connected sub-dilated convolutional networks. The dilation rate of the dilated convolutional layers in each sub-dilated convolutional network increases exponentially by a factor of 2, such as 1, 2, 4, 8, 16, and 32. This gives the entire first dilated convolutional network a larger field of view in the frequency domain, facilitating better capture of correlations in the frequency domain. The first dilated convolutional network primarily performs frequency-domain correlated noise reduction; for example, it is mainly used to perform frequency-domain correlated noise reduction on the processed spectral features.

[0113] Among them, reference Figure 3 In each sub-dilated convolutional network, the dilated convolutional layer (i.e., DilationConvolution) mainly performs dilated convolution processing, the normalization layer (such as the BatchNorm layer) mainly performs normalization processing, and the activation layer (such as the PReLu layer) mainly performs activation processing.

[0114] Among them, reference Figure 3The second dilated convolutional network also includes multiple (e.g., 6) sequentially connected sub-dilated convolutional networks. The dilation rate of the dilated convolutional layers in each sub-dilated convolutional network increases exponentially by 2, such as 1, 2, 4, 8, 16, and 32. This gives the entire second dilated convolutional network a wider field of view on the time axis, making it easier to capture temporal correlations. The second dilated convolutional network mainly performs time-related denoising; for example, it is mainly used to perform time-related denoising on the spectral features after the first denoising.

[0115] It should be noted that the first and second dilated convolutional networks have the same network structure and parameters, ensuring that no additional processing such as feature dimension transformation is required during processing. This eliminates the need to add other network structures, making the entire audio denoising model more lightweight. Furthermore, the entire audio denoising model uses convolutional neural network results, which can be parallelized or implemented using separable convolutions. Therefore, the audio denoising model is a lightweight network model capable of real-time denoising.

[0116] In real-world scenarios, refer to Figure 3 When the number of audio data to be denoised is 4, the first convolutional layer is a two-dimensional convolutional layer with 4 input channels and 8 output channels; the first spatial convolutional network and the second dilated convolutional network both have 8 input channels and 8 output channels; the second convolutional layer is a two-dimensional convolutional layer with 8 input channels and 4 output channels. It is understandable that the network structure of the audio denoising model can be adaptively adjusted according to the actual scenario.

[0117] Specifically, refer to Figure 3The terminal inputs the spectral features of the sub-audio data to be denoised into a pre-trained audio denoising model. First, the first convolutional layer of the audio denoising model extracts key spectral features, which are then used as the processed spectral features of the sub-audio data to be denoised. Next, the processed spectral features are input into the first dilated convolutional network of the audio denoising model. This network performs frequency-domain related denoising on the processed spectral features, aiming to suppress transient noise and other noise in the processed spectral features while maintaining good fidelity for harmonic components and overtones in the audio data. This process yields the sub-audio data to be denoised. The first denoised spectral feature is then input into the second dilated convolutional network in the audio denoising model. The second dilated convolutional network performs time-related denoising on the first denoised spectral feature to suppress noise in the first denoised spectral feature while better preserving the integrity of the singing and avoiding swallowing sounds, thus obtaining the second denoised spectral feature of the sub-audio data to be denoised. Finally, the second denoised spectral feature is input into the second convolutional layer in the audio denoising model. The second convolutional layer performs feature extraction on the second denoised spectral feature to obtain the target spectral feature, which is used as the denoised spectral feature of the sub-audio data to be denoised.

[0118] In this embodiment, the spectral features are subjected to multiple denoising processes through the first convolutional layer, the first dilated convolutional network, the second dilated convolutional network, and the second convolutional layer in the audio denoising model. This effectively eliminates noise in the spectral features, facilitating the subsequent generation of cleaner denoised audio data based on the denoised spectral features, thereby further improving the audio denoising effect. Simultaneously, using a lightweight audio denoising model to denoise the spectral features effectively shortens the audio denoising time, thus improving the audio denoising efficiency.

[0119] In one embodiment, step S102, obtaining the spectral features of the sub-audio data to be denoised, specifically includes the following: performing frame segmentation processing on the sub-audio data to be denoised to obtain the noise reduction frequency frames of the sub-audio data to be denoised; performing feature extraction processing on the noise reduction frequency frames to obtain the spectral features of the noise reduction frequency frames; and combining the spectral features of the noise reduction frequency frames to obtain the spectral features of the sub-audio data to be denoised.

[0120] Frame segmentation refers to separating the audio data to be denoised into multiple frames of noise reduction frequency data, each with the same duration, for example, 20ms. A noise reduction frequency frame is the audio frame obtained after framing the audio data to be denoised. Feature extraction here refers to Fourier transform.

[0121] The spectral characteristics of the noise-to-be-reduced frequency frame are used to characterize its spectral information, specifically including amplitude and phase characteristics. The spectral characteristics of each noise-to-be-reduced sub-audio data are composed of the spectral characteristics of multiple noise-to-be-reduced frequency frames.

[0122] Specifically, the terminal performs frame segmentation processing on the sub-audio data to be denoised according to the frame length L (usually a power of 2, such as 1024) and the frame shift P (such as 0.5L), so as to divide the sub-audio data to be denoised into multiple frames, thereby obtaining multiple frames of noise reduction frequency frames of the sub-audio data to be denoised; then, the terminal performs Fourier transform on each frame of noise reduction frequency frame, or inputs each frame of noise reduction frequency frame into a pre-trained spectral feature extraction model for feature extraction processing, to obtain the spectral features of each frame of noise reduction frequency frame; finally, the spectral features of each frame of noise reduction frequency frame are combined to obtain combined spectral features, which are used as the spectral features of the sub-audio data to be denoised.

[0123] It should be noted that, assuming the frame length is L, after performing a Fourier transform on the audio data (such as the audio data to be denoised, the frequency frame to be denoised, etc.), there are a total of L frequency points; due to the symmetric conjugate of the frequency points, it is generally sufficient to take L / 2+1 frequency points.

[0124] For example, refer to Figure 4 The terminal performs frame-by-frame processing on the input audio data to be noise-reduced, with a frame length of 20ms and a frame shift of 10ms. Then, a windowing operation is performed to obtain n frames of noise-reduced frequency data: frame 1 (0 to 20ms), frame 2 (10ms to 30ms), frame 3 (20ms to 40ms), frame 4 (30ms to 50ms), and so on, up to frame n. Next, the terminal performs a Fourier transform on each of the n frames to obtain their spectral characteristics. Finally, the spectral characteristics of the n frames are combined to obtain the spectral characteristics of the audio data to be noise-reduced.

[0125] It should be noted that after performing a Fourier transform on the frequency frame (or sub-audio data) to be noise-reduced, the terminal can calculate the amplitude and phase characteristics of the frequency frame (or sub-audio data) based on the real and imaginary parts of the Fourier transform result (complex form), thereby obtaining the spectral characteristics of the frequency frame (or sub-audio data). The Fourier transform result is as follows:

[0126] X = X r +iX i ;

[0127] Among them, X r X represents the real part. i Representing the imaginary part, the corresponding amplitude characteristic is: The corresponding phase characteristic is α = arctan(X)i / X r ); arctan represents the arctangent function.

[0128] In this embodiment, by performing frame segmentation processing on the sub-audio data to be denoised, the sub-audio data to be denoised can be divided into multiple frames of noise reduction frequency frames. Feature extraction processing is performed on each frame of noise reduction frequency frame, which can better obtain the spectral features of the sub-audio data to be denoised. This facilitates subsequent effective noise reduction processing of the sub-audio data to be denoised based on the spectral features of the sub-audio data to be denoised.

[0129] In one embodiment, step S104, which involves performing signal reconstruction processing on the denoised spectral features to obtain the denoised sub-audio data corresponding to the sub-audio data to be denoised, specifically includes the following: identifying the denoised spectral features of the frequency frame to be denoised from the denoised spectral features; performing signal reconstruction processing on the denoised spectral features of the frequency frame to be denoised to obtain the denoised audio frame of the frequency frame to be denoised; and combining the denoised audio frames to obtain the denoised sub-audio data corresponding to the sub-audio data to be denoised.

[0130] The denoised spectral features of the sub-audio data to be denoised are composed of the denoised spectral features of multiple noise-reduced frequency frames of the sub-audio data. It is understandable that for the same noise-reduced frequency frame, the noise in its denoised spectral features is less than the noise in its overall spectral features.

[0131] Specifically, signal reconstruction processing refers to inverse Fourier transform, used to reconstruct the denoised audio frame of the frequency frame to be denoised. It can be understood that the denoised audio frame of the frequency frame to be denoised is an audio frame containing less noise and with higher signal fidelity than the frequency frame to be denoised.

[0132] Among them, the noise-reduced sub-audio data corresponding to the noise-reduced sub-audio data is composed of the noise-reduced audio frames of multiple noise-reduced frequency frames of the noise-reduced sub-audio data.

[0133] Specifically, since the denoised spectral characteristics of the sub-audio data to be denoised are composed of the denoised spectral characteristics of multiple frames of the sub-audio data to be denoised, the terminal can identify the denoised spectral characteristics of each frame of the sub-audio data to be denoised from the denoised spectral characteristics of the sub-audio data to be denoised. Then, an inverse Fourier transform is performed on the denoised spectral characteristics of each frame of the sub-audio data to transform the denoised spectral characteristics of each frame of the sub-audio data to be denoised to the time domain, thereby obtaining the denoised audio frame of each frame of the sub-audio data to be denoised. Finally, the terminal combines the denoised audio frames of each frame of the sub-audio data to obtain a combined audio, which serves as the denoised sub-audio data corresponding to the sub-audio data to be denoised.

[0134] In this embodiment, by identifying the denoised spectral features of each noise-reduced frequency frame from the denoised spectral features of the sub-audio data to be denoised, and then performing signal reconstruction processing on the denoised spectral features of each noise-reduced frequency frame, the denoised audio frame of each noise-reduced frequency frame can be reconstructed more effectively. This results in higher audio quality of the denoised sub-audio data obtained based on the combination of denoised audio frames, further improving the audio denoising effect.

[0135] In one embodiment, such as Figure 5 As shown, the audio denoising method provided in this application also includes a training step for the audio denoising model, specifically including the following steps:

[0136] Step S501: Obtain clean audio samples and the corresponding noisy frequency samples of the clean audio samples.

[0137] Clean audio samples refer to audio samples without noise; noisy audio samples refer to audio samples containing noise, specifically the mixed audio samples obtained by fusing clean and noisy audio samples. In other words, noisy audio samples contain both clean and noisy audio samples.

[0138] Step S502: Input the spectral features of the noisy audio samples with noise frequency samples into the audio denoising model to be trained to obtain the denoised spectral features of the noisy audio samples.

[0139] Among them, the noisy sub-audio samples of the noisy frequency samples are obtained by frequency division of the noisy frequency samples; the spectral features of the noisy sub-audio samples are obtained by feature extraction of the noisy sub-audio samples.

[0140] The audio denoising model to be trained can refer to various neural network models such as CNN, RNN, and DNN, or it can refer to... Figure 3 The model shown.

[0141] Step S503: Perform signal reconstruction processing on the denoised spectral features of the noisy sub-audio sample to obtain the denoised sub-audio sample corresponding to the noisy sub-audio sample.

[0142] Step S504: Combine the denoised sub-audio samples to obtain the denoised audio samples corresponding to the noisy frequency samples.

[0143] Step S505: Based on the difference between the spectral characteristics of the denoised audio sample and the spectral characteristics of the clean audio sample, train the audio denoising model to be trained to obtain the pre-trained audio denoising model.

[0144] Specifically, the terminal retrieves clean audio samples and their corresponding noisy frequency samples from the database. Next, it performs frequency division on the noisy frequency samples to obtain noisy sub-audio samples. Then, it performs a Fourier transform on the noisy sub-audio samples to obtain their spectral features. This spectral feature is then input into the audio denoising model to be trained. The model performs a series of denoising processes on the spectral features of the noisy sub-audio samples to obtain their denoised spectral features. Finally, an inverse Fourier transform is performed on the denoised spectral features of the noisy sub-audio samples. The process involves transforming the noisy sub-audio samples to obtain the denoised sub-audio samples corresponding to the noisy sub-audio samples. Following the order of the denoised sub-audio samples, these samples are combined to obtain the denoised audio samples corresponding to the noisy sub-audio samples. Finally, Fourier transforms are performed on the denoised audio samples and the clean audio samples to obtain the spectral features of the denoised audio samples and the clean audio samples, respectively. Based on the difference between the spectral features of the denoised audio samples and the clean audio samples, the audio denoising model to be trained is iteratively trained to obtain the trained audio denoising model, which serves as the pre-trained audio denoising model.

[0145] For example, during the training of the audio denoising model, the terminal uses clean audio samples as the training target and calculates the loss value based on the difference between the spectral characteristics of the denoised audio samples and the spectral characteristics of the clean audio samples. If the loss value is greater than or equal to the preset loss value, the audio denoising model to be trained is iteratively trained based on the loss value, and steps S502 to S505 are repeated until the loss value obtained from the output of the trained audio denoising model is less than the preset loss value. Then, the training stops, and the trained audio denoising model is used as the pre-trained audio denoising model.

[0146] In this embodiment, the spectral features of noisy sub-audio samples from noisy frequency samples are input into the audio denoising model to be trained to obtain the denoised spectral features of the noisy sub-audio samples. Then, based on the denoised spectral features of the noisy sub-audio samples, denoised sub-audio samples are obtained and combined to obtain denoised audio samples. Finally, based on the difference between the spectral features of the denoised audio samples and the spectral features of the clean audio samples, the audio denoising model to be trained is iteratively trained so that the spectral features of the denoised audio samples obtained based on the audio denoising model continuously approach the spectral features of the clean audio samples. This helps to improve the denoising capability of the audio denoising model and further enhances the audio denoising effect.

[0147] In one embodiment, step S505 above, which trains the audio denoising model to be trained based on the difference between the spectral characteristics of the denoised audio sample and the spectral characteristics of the clean audio sample, to obtain a pre-trained audio denoising model, specifically includes the following: obtaining a loss value based on the difference between the spectral characteristics of the denoised audio sample and the spectral characteristics of the clean audio sample; training the audio denoising model to be trained based on the loss value until the training termination condition is met; and confirming the trained audio denoising model that has met the training termination condition as the pre-trained audio denoising model.

[0148] The loss value can refer to MSE (Mean Square Error) or SI-SDR (Scale-invariant Signal-to-Distortion Ratio).

[0149] The training termination condition can be either that the loss value is less than the preset loss value, or that the total number of training iterations reaches the preset number of training iterations.

[0150] Specifically, the terminal calculates the MSE or SI-SDR based on the difference between the spectral characteristics of the denoised audio sample and the spectral characteristics of the clean audio sample, and uses the MSE or SI-SDR as the loss value of the audio denoising model to be trained. Then, using the clean audio sample as the training target, the audio denoising model to be trained is iteratively trained according to the loss value to continuously adjust the model parameters in the audio denoising model until the trained audio denoising model meets the training termination condition. For example, if the loss value calculated based on the output result of the trained audio denoising model is less than a preset threshold, then the trained audio denoising model is used as the pre-trained audio denoising model.

[0151] In this embodiment, the audio denoising model to be trained is iteratively trained based on the difference between the spectral features of the denoised audio sample and the spectral features of the clean audio sample. This can improve the accuracy of the denoised spectral features output by the trained audio denoising model, thereby improving the denoising effect of the audio denoising model.

[0152] In one embodiment, step S501 above, obtaining the noisy frequency sample corresponding to the clean audio sample, specifically includes the following: obtaining the noisy audio sample and a preset signal-to-noise ratio; performing a fusion process on the clean audio sample and the noisy audio sample according to the preset signal-to-noise ratio to obtain the noisy frequency sample corresponding to the clean audio sample.

[0153] The noise audio samples include both noise audio and accompaniment audio. Noise audio refers to various types of noise, such as noise from squares, roads, conference rooms, restaurants, cafes, and keyboard typing. Accompaniment audio refers to various types of accompaniment audio, such as accompaniment from instruments like piano, guitar, and drums, and original songs.

[0154] The preset signal-to-noise ratio is within the preset signal-to-noise ratio range of [-5dB, 20dB], and can be selected according to the actual scenario.

[0155] Among them, the noisy audio sample can be formed by superimposing clean audio samples and noisy audio, or by superimposing clean audio samples and accompaniment audio, or by superimposing clean audio samples, noisy audio and accompaniment audio.

[0156] Specifically, the terminal retrieves noisy audio and accompaniment audio from the database and treats both as noisy audio samples. Then, based on the current noise reduction scenario, it determines a preset signal-to-noise ratio from a preset signal-to-noise ratio range. Then, according to the preset signal-to-noise ratio, it mixes and superimposes the clean audio samples and noisy audio samples to obtain multiple types of noisy audio samples. These are the first type of noisy audio sample formed by mixing and superimposing clean audio samples and noisy audio, the second type of noisy audio sample formed by mixing and superimposing clean audio samples and accompaniment audio, and the third type of noisy audio sample formed by mixing and superimposing clean audio samples, noisy audio, and accompaniment audio.

[0157] In this embodiment, clean audio samples and noisy audio samples are fused according to a preset signal-to-noise ratio to obtain noisy frequency samples corresponding to the clean audio samples. This can simulate more complex and varied background noise (such as complex noise and non-steady-state noise in real singing scenarios), so that the audio denoising model trained based on noisy frequency samples has a strong denoising capability for audio data carrying different types of noise, thereby further improving the audio denoising effect of the audio denoising model.

[0158] In one embodiment, such as Figure 6 As shown, another audio noise reduction method is provided. Taking the application of this method to a terminal as an example, the method includes the following steps:

[0159] Step S601: Perform frequency division processing on the noise frequency data to be reduced to obtain the noise sub-audio data to be reduced.

[0160] Step S602: Perform frame segmentation processing on the sub-audio data to be denoised to obtain the noise reduction frequency frame of the sub-audio data to be denoised; perform feature extraction processing on the noise reduction frequency frame to obtain the spectral features of the noise reduction frequency frame.

[0161] Step S603: Combine the spectral features of the frequency frame to be denoised to obtain the spectral features of the sub-audio data to be denoised.

[0162] Step S604: Using a pre-trained audio denoising model, feature extraction is performed on the spectral features to obtain the processed spectral features of the sub-audio data to be denoised.

[0163] Step S605: Perform frequency-domain-related denoising processing on the processed spectral features to obtain the first denoised spectral features of the sub-audio data to be denoised; perform time-related denoising processing on the first denoised spectral features to obtain the second denoised spectral features of the sub-audio data to be denoised.

[0164] Step S606: Perform feature extraction processing on the second denoised spectral features to obtain the denoised spectral features of the sub-audio data to be denoised.

[0165] Step S607: Identify the denoised spectral features of the frequency frame to be denoised from the denoised spectral features; perform signal reconstruction processing on the denoised spectral features of the frequency frame to be denoised to obtain the denoised audio frame of the frequency frame to be denoised.

[0166] Step S608: Combine the denoised audio frames to obtain the denoised sub-audio data corresponding to the sub-audio data to be denoised.

[0167] Step S609: Combine the noise-reduced sub-audio data to obtain the noise-reduced audio data corresponding to the noise-reduced audio data.

[0168] In this embodiment, by performing frequency division processing on the noise-reducing frequency data, the human voice in the low-frequency band and noise in each frequency band can be better captured. This facilitates the subsequent targeted noise reduction processing of the spectral features of each sub-audio data to be denoised using a pre-trained audio denoising model. This allows for more comprehensive noise reduction processing, avoids missing any noise to be processed, and results in higher audio quality of the denoised audio data reconstructed based on the denoised spectral features of the sub-audio data to be denoised, thereby improving the noise reduction effect of the noise-reducing frequency data.

[0169] In one embodiment, such as Figure 7 As shown, a training method for an audio noise reduction model is provided. Taking the application of this method to a terminal as an example, the method includes the following steps:

[0170] Step S701: Obtain clean audio samples, noisy audio samples, and preset signal-to-noise ratio.

[0171] Step S702: According to the preset signal-to-noise ratio, the clean audio sample and the noisy audio sample are fused to obtain the noisy frequency sample corresponding to the clean audio sample.

[0172] Step S703: Perform frequency division processing on the noisy frequency sample to obtain the noisy sub-audio sample of the noisy frequency sample.

[0173] Step S704: Obtain the spectral characteristics of the audio sample with noise.

[0174] Step S705: Input the spectral features of the noisy audio sample into the audio denoising model to be trained to obtain the denoised spectral features of the noisy audio sample.

[0175] Step S706: Perform signal reconstruction processing on the denoised spectral features of the noisy sub-audio sample to obtain the denoised sub-audio sample corresponding to the noisy sub-audio sample.

[0176] Step S707: Combine the denoised sub-audio samples to obtain the denoised audio samples corresponding to the noisy frequency samples.

[0177] Step S708: Obtain the loss value based on the difference between the spectral characteristics of the denoised audio sample and the spectral characteristics of the clean audio sample.

[0178] Step S709: Train the audio denoising model to be trained according to the loss value until the training termination condition is met.

[0179] Step S710: The trained audio denoising model that has reached the end of training condition is confirmed as the pre-trained audio denoising model.

[0180] In this embodiment, the audio denoising model to be trained is iteratively trained based on clean audio samples and the noisy frequency samples corresponding to the clean audio samples. This makes the spectral characteristics of the denoised audio samples obtained based on the audio denoising model increasingly approach the spectral characteristics of the clean audio samples, which is beneficial to improving the denoising capability of the audio denoising model and further improving the audio denoising effect.

[0181] In one embodiment, to more clearly illustrate the audio noise reduction method provided in this application, the following specific embodiment will be used to describe the audio noise reduction method in detail. In one embodiment, refer to Figure 8 This application also provides another method for training an audio noise reduction model, which can be applied to a terminal, specifically including the following:

[0182] refer to Figure 8In the scenario of vocal data denoising, the terminal retrieves accompaniment audio samples, noisy audio samples, and clean audio samples from the database. Based on a preset signal-to-noise ratio, it performs mixed superposition processing on the clean audio samples and accompaniment audio samples, or on the clean audio samples and noisy audio samples, or on the clean audio samples, accompaniment audio samples, and noisy audio samples, resulting in multiple noisy frequency samples. Next, the noisy frequency samples are frequency-divided to obtain multiple sub-bands (i.e., noisy sub-audio samples). A Fourier transform is performed on each sub-band to obtain its spectral characteristics. These spectral characteristics are then input into the audio denoising model to be trained to obtain the denoised spectral characteristics of each sub-band. Finally, an inverse Fourier transform is performed on the denoised spectral characteristics of each sub-band. Multiple reconstructed sub-bands are obtained. These sub-bands are then combined to obtain denoised audio samples. Fourier transforms are performed on the clean audio samples and the denoised audio samples to obtain their spectral characteristics. Finally, the clean audio samples are used as the training target, and the loss value is calculated based on the difference between the spectral characteristics of the denoised audio samples and the clean audio samples. If the loss value is greater than or equal to a preset loss value, the audio denoising model to be trained is iteratively trained according to the loss value until the loss value obtained from the output of the trained audio denoising model is less than the preset loss value. At this point, training stops, and the trained audio denoising model is used as the pre-trained audio denoising model.

[0183] In addition, refer to Figure 9 This application also provides another audio noise reduction method that can be applied to a terminal, specifically including the following:

[0184] refer to Figure 9 The terminal acquires the user's recorded singing data as the noise reduction audio data; it performs frequency division processing on the noise reduction audio data to obtain multiple sub-bands (i.e., sub-audio data to be denoised); it performs Fourier transform on each sub-band to obtain the spectral features of each sub-band; it inputs the spectral features of each sub-band into a pre-trained audio denoising model for denoising processing to obtain the denoised spectral features of each sub-band; it performs inverse Fourier transform on the denoised spectral features of each sub-band to obtain multiple reconstructed sub-bands; and it combines the multiple reconstructed sub-bands through sub-band synthesis to obtain the denoised audio data, thus obtaining the denoised singing data.

[0185] In this embodiment, on the one hand, the audio denoising model to be trained is iteratively trained using clean audio samples and their corresponding noisy frequency samples. This allows the spectral characteristics of the denoised audio samples obtained based on the audio denoising model to continuously approach the spectral characteristics of the clean audio samples, which helps improve the denoising capability of the audio denoising model and further enhances the audio denoising effect. On the other hand, by using the pre-trained audio denoising model to perform targeted denoising processing on each sub-band, more comprehensive denoising processing is possible, avoiding omissions of noise to be processed. This results in cleaner reconstructed denoised audio data, with effectively improved clarity, intelligibility, and signal fidelity, further enhancing the denoising effect on the noisy frequency data.

[0186] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0187] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an audio noise reduction method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0188] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0189] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0190] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0191] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0192] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0193] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0194] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0195] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An audio noise reduction method, characterized in that, The method includes: The noise reduction frequency data is divided into frequencies to obtain the noise reduction sub-audio data of the noise reduction frequency data; Obtain the spectral characteristics of the sub-audio data to be denoised; The spectral features are denoised using a pre-trained audio denoising model to obtain the denoised spectral features of the sub-audio data to be denoised. The denoised spectral features are subjected to signal reconstruction processing to obtain the denoised sub-audio data corresponding to the sub-audio data to be denoised; The denoised sub-audio data are combined to obtain the denoised audio data corresponding to the denoised audio data to be denoised. The pre-trained audio noise reduction model is obtained in the following way: Obtain clean audio samples and the corresponding noisy frequency samples of the clean audio samples; The spectral features of the noisy sub-audio samples of the noisy frequency samples are input into the audio denoising model to be trained to obtain the denoised spectral features of the noisy sub-audio samples. The denoised spectral features of the noisy sub-audio sample are subjected to signal reconstruction processing to obtain the denoised sub-audio sample corresponding to the noisy sub-audio sample; The denoised sub-audio samples are combined to obtain the denoised audio samples corresponding to the noisy frequency samples; Based on the difference between the spectral characteristics of the denoised audio sample and the spectral characteristics of the clean audio sample, the audio denoising model to be trained is trained to obtain the pre-trained audio denoising model.

2. The method according to claim 1, characterized in that, The step involves using a pre-trained audio denoising model to denoise the spectral features, resulting in denoised spectral features of the sub-audio data to be denoised, including: The spectral features are extracted using a pre-trained audio denoising model to obtain the processed spectral features of the sub-audio data to be denoised. The processed spectral features are subjected to frequency domain-dependent noise reduction processing to obtain the first noise-reduced spectral features of the sub-audio data to be denoised. The first denoised spectral features are subjected to time-related denoising processing to obtain the second denoised spectral features of the sub-audio data to be denoised. The second denoised spectral features are subjected to feature extraction processing to obtain the denoised spectral features of the sub-audio data to be denoised.

3. The method according to claim 2, characterized in that, The pre-trained audio noise reduction model includes a first convolutional layer, a first dilated convolutional network, a second dilated convolutional network, and a second convolutional layer connected in sequence. The first dilated convolutional network and the second dilated convolutional network each include multiple sub-dilated convolutional networks connected in sequence. Each sub-dilated convolutional network includes a dilated convolutional layer, a normalization layer, and an activation layer connected in sequence.

4. The method according to claim 1, characterized in that, The acquisition of the spectral features of the sub-audio data to be denoised includes: The sub-audio data to be noise-reduced is processed by frame segmentation to obtain the noise-reducing frequency frames of the sub-audio data to be noise-reduced. The frequency frame to be noise-reduced is subjected to feature extraction processing to obtain the spectral features of the frequency frame to be noise-reduced; The spectral features of the frequency frames to be noise-reduced are combined to obtain the spectral features of the sub-audio data to be noise-reduced.

5. The method according to claim 4, characterized in that, The step of performing signal reconstruction processing on the denoised spectral features to obtain the denoised sub-audio data corresponding to the sub-audio data to be denoised includes: The denoised spectral features of the frequency frame to be denoised are identified from the denoised spectral features. The denoised spectral features of the frequency frame to be denoised are processed to obtain the denoised audio frame of the frequency frame to be denoised. The denoised audio frames are combined to obtain the denoised sub-audio data corresponding to the sub-audio data to be denoised.

6. The method according to claim 1, characterized in that, The step of training the audio denoising model to be trained based on the difference between the spectral features of the denoised audio sample and the spectral features of the clean audio sample, to obtain the pre-trained audio denoising model, includes: The loss value is obtained based on the difference between the spectral characteristics of the denoised audio sample and the spectral characteristics of the clean audio sample; The audio noise reduction model to be trained is trained based on the loss value until the training termination condition is met. The trained audio denoising model that meets the training termination condition is identified as the pre-trained audio denoising model.

7. The method according to claim 1, characterized in that, The noisy frequency samples corresponding to the clean audio samples are obtained in the following way: Acquire noisy audio samples and preset signal-to-noise ratio; According to the preset signal-to-noise ratio, the clean audio sample and the noisy audio sample are fused to obtain the noisy frequency sample corresponding to the clean audio sample.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images