Audio noise reduction method, computer device, storage medium and computer program product

Abstract

This application relates to an audio noise reduction method, computer device, storage medium, and computer program product. The method includes: acquiring the frequency domain features of audio data to be denoised; performing spectral compression processing on the frequency domain features according to a pre-trained spectral compressor to obtain processed frequency domain features of the audio data to be denoised; performing noise reduction processing on the processed frequency domain features to obtain denoised frequency domain features of the audio data to be denoised; and performing signal reconstruction processing on the denoised frequency domain features according to the pre-trained spectral compressor to obtain target denoised audio data corresponding to the audio data to be denoised. This method can improve the noise reduction effect on audio data.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an audio noise reduction method, computer device, storage medium, and computer program product. Background Technology

[0002] Currently, when users record their singing through karaoke software, they are limited by non-professional recording equipment and environments, which makes it easy for microphone friction noise and ambient background noise to mix into the user's recorded singing, which has a significant impact on the quality of the user's recorded singing.

[0003] Traditional techniques often rely on digital signal processing to process noise in audio. For example, statistical signal processing can be used to extract a clean vocal signal from a recorded song. However, this noise reduction method is only effective for stable noise scenes and is unlikely to be ideal for the more complex and varied background noise in real life. Summary of the Invention

[0004] Therefore, it is necessary to provide an audio noise reduction method, computer device, computer-readable storage medium, and computer program product that can improve the noise reduction effect on audio data in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides an audio noise reduction method. The method includes:

[0006] Obtain the frequency domain characteristics of the noise reduction frequency data;

[0007] The frequency domain features are subjected to spectral compression processing based on a pre-trained spectral compressor to obtain the processed frequency domain features of the noise-reduced frequency data.

[0008] The processed frequency domain features are subjected to noise reduction processing to obtain the noise-reduced frequency domain features of the frequency data to be noise-reduced;

[0009] Based on the pre-trained spectrum compressor, the noise-reduced frequency domain features are processed for signal reconstruction to obtain the target noise-reduced frequency data corresponding to the noise-reduced frequency data.

[0010] In one embodiment, the frequency domain features are subjected to spectral compression processing based on a pre-trained spectral compressor to obtain the processed frequency domain features of the noise-reduced frequency data, including:

[0011] Obtain the amplitude and phase features of the frequency domain features;

[0012] The amplitude and phase features are compressed using the pre-trained spectrum compressor to obtain the processed frequency domain features of the noise reduction frequency data.

[0013] In one embodiment, the noise-reduced frequency domain features are reconstructed using the pre-trained spectrum compressor to obtain the target noise-reduced frequency data corresponding to the noise-reduced frequency data, including:

[0014] Determine the target amplitude and target phase characteristics of the frequency domain features after noise reduction;

[0015] Based on the pre-trained spectrum compressor, the target amplitude features and the target phase features are subjected to audio enhancement processing to obtain enhanced frequency domain features;

[0016] The enhanced frequency domain features are processed for signal reconstruction to obtain the target noise-reduced frequency data corresponding to the noise-reduced frequency data.

[0017] In one embodiment, the processed frequency domain features are subjected to noise reduction processing to obtain the noise-reduced frequency domain features of the frequency data to be noise-reduced, including:

[0018] The processed frequency domain features are encoded according to the pre-trained encoder to obtain the encoded features of the noise reduction frequency data;

[0019] Based on a pre-trained denoiser, the encoded features are denoised to obtain the denoised encoded features of the noise-reduced frequency data.

[0020] The denoised encoded features are decoded using a pre-trained decoder to obtain the denoised frequency domain features of the noise-reduced frequency data.

[0021] In one embodiment, obtaining the frequency domain features of the noise reduction frequency data includes:

[0022] The noise frequency data to be reduced is processed into frames to obtain the noise frequency frames to be reduced.

[0023] Perform a Fourier transform on the frequency frame to be noise-reduced to obtain the frequency domain characteristics of the frequency data to be noise-reduced.

[0024] In one embodiment, the method further includes:

[0025] Obtain noise-free audio samples and corresponding mixed audio samples;

[0026] The frequency domain features of the mixed audio samples are input into the audio denoising model to be trained to obtain the denoised frequency domain features of the mixed audio samples.

[0027] The loss value of the audio denoising model to be trained is obtained based on the difference between the frequency domain features of the denoised mixed audio samples and the frequency domain features of the noise-free audio samples.

[0028] Based on the loss value, the audio denoising model to be trained is trained to obtain a pre-trained audio denoising model; the pre-trained audio denoising model includes the pre-trained spectrum compressor, the pre-trained encoder, the pre-trained denoiser, and the pre-trained decoder.

[0029] In one embodiment, obtaining the mixed audio sample corresponding to the noise-free audio sample includes:

[0030] Acquire noise samples and preset signal-to-noise ratio;

[0031] Based on the preset signal-to-noise ratio, the noise-free audio sample and the noise sample are fused to obtain the mixed audio sample corresponding to the noise-free audio sample.

[0032] Secondly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0033] Obtain the frequency domain characteristics of the noise reduction frequency data;

[0034] The frequency domain features are subjected to spectral compression processing based on a pre-trained spectral compressor to obtain the processed frequency domain features of the noise-reduced frequency data.

[0035] The processed frequency domain features are subjected to noise reduction processing to obtain the noise-reduced frequency domain features of the frequency data to be noise-reduced;

[0036] Based on the pre-trained spectrum compressor, the noise-reduced frequency domain features are processed for signal reconstruction to obtain the target noise-reduced frequency data corresponding to the noise-reduced frequency data.

[0037] Thirdly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0038] Obtain the frequency domain characteristics of the noise reduction frequency data;

[0039] The frequency domain features are subjected to spectral compression processing based on a pre-trained spectral compressor to obtain the processed frequency domain features of the noise-reduced frequency data.

[0040] The processed frequency domain features are subjected to noise reduction processing to obtain the noise-reduced frequency domain features of the frequency data to be noise-reduced;

[0041] Based on the pre-trained spectrum compressor, the noise-reduced frequency domain features are processed for signal reconstruction to obtain the target noise-reduced frequency data corresponding to the noise-reduced frequency data.

[0042] Fourthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0043] Obtain the frequency domain characteristics of the noise reduction frequency data;

[0044] The frequency domain features are subjected to spectral compression processing based on a pre-trained spectral compressor to obtain the processed frequency domain features of the noise-reduced frequency data.

[0045] The processed frequency domain features are subjected to noise reduction processing to obtain the noise-reduced frequency domain features of the frequency data to be noise-reduced;

[0046] Based on the pre-trained spectrum compressor, the noise-reduced frequency domain features are processed for signal reconstruction to obtain the target noise-reduced frequency data corresponding to the noise-reduced frequency data.

[0047] The aforementioned audio noise reduction method, computer equipment, storage medium, and computer program product acquire the frequency domain characteristics of the audio data to be noise-reduced; perform spectral compression processing on the frequency domain characteristics according to a pre-trained spectrum compressor to obtain the processed frequency domain characteristics of the audio data to be noise-reduced; perform noise reduction processing on the processed frequency domain characteristics to obtain the denoised frequency domain characteristics of the audio data to be noise-reduced; and perform signal reconstruction processing on the denoised frequency domain characteristics according to the pre-trained spectrum compressor to obtain the target noise-reduced audio data corresponding to the audio data to be noise-reduced. Using this method, by appropriately compressing and transforming the frequency domain characteristics of the noise-reduced audio data using a spectrum compressor, more detailed frequency information in the audio data to be noise-reduced can be captured. This is beneficial for reconstructing weak spectral information caused by sudden noise or reverberation in the noise-reduced audio data, effectively improving the signal quality of the reconstructed target noise-reduced audio data, thereby effectively improving the noise reduction effect on the audio data to be noise-reduced. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating an audio noise reduction method in one embodiment;

[0049] Figure 2 This is a flowchart illustrating the steps for obtaining the frequency domain characteristics of the noise-reducing frequency data in one embodiment.

[0050] Figure 3 This is a flowchart illustrating the steps for obtaining the denoised frequency domain features of the frequency data to be denoised in one embodiment.

[0051] Figure 4 This is a schematic diagram illustrating the principle of obtaining the denoised encoded features of the noise-reducing frequency data in one embodiment;

[0052] Figure 5 This is a schematic diagram of a frequency frame to be noise-reduced in one embodiment;

[0053] Figure 6 This is a flowchart illustrating an audio noise reduction method in another embodiment;

[0054] Figure 7 This is a flowchart illustrating the audio noise reduction method in yet another embodiment;

[0055] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] In one embodiment, such as Figure 1 As shown, an audio noise reduction method is provided. This embodiment illustrates the method's application to a terminal, but it is understood that the method can also be applied to a server, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. The server can be a standalone server or a server cluster consisting of multiple servers. In this embodiment, the method includes the following steps:

[0058] Step S101: Obtain the frequency domain characteristics of the noise reduction frequency data.

[0059] The audio data to be noise-reduced refers to audio data that requires noise reduction processing. For example, the audio data to be noise-reduced can be singing data recorded by a user through singing software installed on a terminal, or it can be in the form of a waveform signal.

[0060] Frequency domain features refer to the characteristic information that describes the frequency properties of audio data. For example, frequency domain features can be characteristic information such as the spectrum and frequency bands.

[0061] Specifically, the terminal can be equipped with an audio acquisition device to collect the noise reduction frequency data. Alternatively, the noise reduction frequency data can be collected using a separate audio acquisition device and sent to the terminal. The terminal can also retrieve the noise reduction frequency data from a server or database, or obtain it through other means. Furthermore, the terminal extracts frequency domain features from the noise reduction frequency data.

[0062] Step S102: Based on the pre-trained spectrum compressor, the frequency domain features are subjected to spectrum compression processing to obtain the processed frequency domain features of the noise reduction frequency data.

[0063] Among them, the spectrum compressor is used to dynamically adjust the energy of different frequencies in the frequency domain characteristics.

[0064] Specifically, the terminal trains the spectrum compressor to be trained using mixed audio samples to obtain a pre-trained spectrum compressor. The number of parameters in the pre-trained spectrum compressor is the sum of the number of preset frequency points divided by two and then one; for example, if the number of preset frequency points is 1024, then the number of parameters in the pre-trained spectrum compressor is 513 (1024 / 2+1). Then, the terminal uses the pre-trained spectrum compressor to perform spectrum compression processing on the frequency information in the frequency domain features, thus obtaining the processed frequency domain features of the noise-reduced frequency data.

[0065] It should be noted that after the audio data (such as mixed audio samples, noise reduction frequency data) undergoes Fourier transform, there are a total of L frequency points. Due to the symmetry and conjugation of frequency points, the number of preset frequency points is generally taken as L / 2+1, which can be marked as P.

[0066] Step S103: Perform noise reduction processing on the processed frequency domain features to obtain the noise-reduced frequency domain features of the frequency data to be noise-reduced.

[0067] Here, the denoised frequency domain features refer to the audio signal obtained after denoising. It can be understood that the noise in the denoised frequency domain features is less than the noise in the frequency domain features of the original audio data.

[0068] Specifically, the terminal performs noise reduction processing on the processed frequency domain features. This can be done by first encoding the processed frequency domain features to obtain the encoded features of the frequency data to be noise-reduced, then performing noise reduction processing on the encoded features to obtain the noise-reduced encoded features of the frequency data to be noise-reduced, and finally performing decoding processing on the noise-reduced encoded features to obtain the noise-reduced frequency domain features of the frequency data to be noise-reduced.

[0069] Step S104: Based on the pre-trained spectrum compressor, perform signal reconstruction processing on the frequency domain features after noise reduction to obtain the target noise-reduced frequency data corresponding to the noise-reduced frequency data.

[0070] The target noise-reduced audio data refers to the audio data that has less noise and a cleaner signal than the audio data to be noise-reduced after being processed by the above audio noise reduction method.

[0071] Specifically, after obtaining the denoised frequency domain features of the noise-reduced frequency data, the terminal can also use the pre-trained spectrum compressor in step S102 above to perform signal reconstruction processing on the denoised frequency domain features, that is, restore the denoised frequency domain features to the form of the noise-reduced frequency data. Then the terminal obtains the target noise-reduced frequency data corresponding to the noise-reduced frequency data, thereby obtaining target noise-reduced frequency data with less noise and higher signal fidelity.

[0072] In the aforementioned audio noise reduction method, the frequency domain features of the audio data to be noise-reduced are obtained; based on a pre-trained spectrum compressor, the frequency domain features are subjected to spectrum compression processing to obtain the processed frequency domain features of the audio data to be noise-reduced; the processed frequency domain features are then subjected to noise reduction processing to obtain the denoised frequency domain features of the audio data to be noise-reduced; finally, based on the pre-trained spectrum compressor, the denoised frequency domain features are subjected to signal reconstruction processing to obtain the target noise-reduced audio data corresponding to the audio data to be noise-reduced. Using this method, by appropriately compressing and transforming the frequency domain features of the noise-reduced audio data using a spectrum compressor, more detailed frequency information in the audio data to be noise-reduced can be captured. This is beneficial for reconstructing weak spectral information caused by sudden noise or reverberation in the noise-reduced audio data, effectively improving the signal quality of the reconstructed target noise-reduced audio data, thereby effectively improving the noise reduction effect on the audio data to be noise-reduced.

[0073] In one embodiment, such as Figure 2 As shown, step S102 above, based on a pre-trained spectrum compressor, performs spectrum compression processing on the frequency domain features to obtain the processed frequency domain features of the noise-reduced frequency data, specifically including the following:

[0074] Step S201: Obtain the amplitude and phase features of the frequency domain features.

[0075] Specifically, frequency domain features can be in complex form, that is, frequency domain features can be expressed as X = X r +iX i ;where X r Let X represent the real part of the frequency domain feature X. r ∈R P×C R represents a real number, P represents the number of preset frequency points, C represents the feature dimension of the frequency domain feature, and C is equal to the number of audio frames of the frequency to be noise-reduced; X i X represents the imaginary part of the frequency domain feature X. i ∈R P×CFurthermore, the terminal can calculate the amplitude and phase characteristics of the frequency domain features based on the real and imaginary parts of the frequency domain features.

[0076] In practical applications, the terminal can calculate the amplitude characteristics of the frequency domain features using formula (1):

[0077]

[0078] The terminal can also calculate the phase characteristics of the frequency domain features using formula (2):

[0079] α = arctan(X) i / X r (2)

[0080] Step S202: Based on the pre-trained spectrum compressor, perform spectrum compression processing on the amplitude and phase features to obtain the processed frequency domain features of the noise reduction frequency data.

[0081] Amplitude characteristics refer to the features that characterize the amplitude properties of a signal in the frequency domain. Phase characteristics refer to the features that characterize the phase properties of a signal in the frequency domain.

[0082] Specifically, the terminal performs spectrum compression processing on the amplitude and phase features based on a pre-trained spectrum compressor. This can be achieved by processing the amplitude features using the pre-trained spectrum compressor to obtain processed amplitude features, and simultaneously processing the phase features using cosine and sine functions to obtain cosine-processed and sine-processed phase features, respectively. The terminal then obtains the processed real part based on the processed amplitude and cosine-processed phase features, and the processed imaginary part based on the processed amplitude and sine-processed phase features. Finally, the processed frequency domain features are constructed from the processed real and processed imaginary parts, thus providing the terminal with the processed frequency domain features of the noise-reduced frequency data.

[0083] In practical applications, a pre-trained spectrum compressor can be represented by a spectrum compression matrix. The terminal can then input the spectrum compression matrix, amplitude features, and phase features corresponding to the pre-trained spectrum compressor into formula (3) to obtain the processed frequency domain features of the noise reduction frequency data.

[0084] Y = Y r +iYi i =|X| H cos(α)+i|X| H sin(α) (3)

[0085] Where Y represents the processed frequency domain features; Y r Y represents the real part of the frequency domain feature after processing; iH represents the imaginary part of the processed frequency domain features; H represents the pre-trained spectrum compressor (or the spectrum compression matrix corresponding to the pre-trained spectrum compressor).

[0086] Understandably, when users record their singing using karaoke software, loud noise may suddenly appear in the recording environment, suppressing the user's singing (or vocals), or causing the singing (or vocals) to become blurred due to excessive reverberation. In such cases, the singing is easily identified as noise and is thus removed, resulting in a poor presentation of the singing. Therefore, this embodiment uses a pre-trained spectrum compressor to extract detailed information on weak frequency bands in the frequency domain features, thereby accurately identifying the vocal signal for subsequent reconstruction.

[0087] In this embodiment, by dynamically adjusting the energy of different frequencies in the frequency domain features through a spectrum compressor, the low, medium, and high frequencies in the frequency domain features of the noise-reduced frequency data are appropriately compressed and transformed. This allows for the capture of more detailed frequency information in the noise-reduced frequency data, which is beneficial for reconstructing weak spectral information caused by sudden noise or reverberation in the noise-reduced frequency data. This solves the defect in traditional technology where human voice signals are easily distorted due to noise reduction processing. It not only improves the fidelity of human voice signals in the noise-reduced frequency data but also enhances the clarity of human voice signals in the noise-reduced frequency data.

[0088] In one embodiment, step S104 above, which involves performing signal reconstruction processing on the denoised frequency domain features based on a pre-trained spectrum compressor to obtain the target denoised frequency data corresponding to the denoised frequency data, specifically includes the following: determining the target amplitude features and target phase features of the denoised frequency domain features; performing audio enhancement processing on the target amplitude features and target phase features based on the pre-trained spectrum compressor to obtain enhanced frequency domain features; and performing signal reconstruction processing on the enhanced frequency domain features to obtain the target denoised frequency data corresponding to the denoised frequency data.

[0089] Specifically, the frequency domain characteristics after noise reduction can be in complex form, that is, the frequency domain characteristics after noise reduction can be expressed as V = V r +iV i , where Y r V represents the real part of the frequency domain feature V. i This represents the imaginary part of the frequency domain feature V; therefore, after obtaining the denoised frequency domain features, the terminal can calculate the target amplitude feature and target phase feature of the denoised frequency domain features based on the real part and imaginary part of the denoised frequency domain features.

[0090] In practical applications, the terminal can calculate the target amplitude feature |V| of the frequency domain after noise reduction using formula (4):

[0091]

[0092] The terminal can also calculate the target phase feature β of the frequency domain features after noise reduction using formula (5):

[0093] β=arctan(V i / V r (5)

[0094] Furthermore, the terminal performs audio enhancement processing on the target amplitude and phase features based on the pre-trained spectrum compressor. This can be achieved by first determining the spectrum compression matrix corresponding to the pre-trained spectrum compressor, then using the reciprocal of the spectrum compression matrix to process the target amplitude features, obtaining the processed target amplitude features. Simultaneously, the target phase features are processed using cosine and sine functions respectively, obtaining the cosine-processed and sine-processed phase features of the target phase features. The terminal obtains the enhanced real part based on the processed amplitude and cosine-processed phase features, and the enhanced imaginary part based on the processed amplitude and sine-processed phase features. Finally, the enhanced frequency domain features are constructed from the enhanced real and enhanced imaginary parts, i.e., the terminal obtains the enhanced frequency domain features of the noise-reduced frequency data.

[0095] In practical applications, the terminal can input the reciprocal of the spectrum compression matrix, the target amplitude feature and the target phase feature into formula (6) to obtain the enhanced frequency domain features of the noise reduction frequency data.

[0096]

[0097] Where Z represents the enhanced frequency domain feature; Z r Z represents the real part of the enhanced frequency domain characteristics; i 1 / H represents the imaginary part of the enhanced frequency domain characteristics; 1 / H represents the reciprocal of the spectral compression matrix.

[0098] Furthermore, the terminal performs signal reconstruction processing on the enhanced frequency domain features, which can be done by performing a short-time inverse Fourier transform on the enhanced frequency domain features, so as to restore the frequency domain features obtained after Fourier transform processing to audio data. Then the terminal obtains the target noise-reduced frequency data corresponding to the noise-reduced frequency data.

[0099] In this embodiment, the target amplitude features and target phase features of the denoised frequency domain features are determined; then, based on a pre-trained spectrum compressor, audio enhancement processing is performed on the target amplitude features and target phase features to obtain enhanced frequency domain features. This can further enhance the spectral information in the denoised frequency domain features, so that the target denoised audio data reconstructed through the enhanced spectrum features has higher clarity and greatly improves the denoising effect of the target denoised audio data.

[0100] In one embodiment, such as Figure 3As shown, step S103 above involves denoising the processed frequency domain features to obtain the denoised frequency domain features of the frequency data to be denoised, specifically including the following:

[0101] Step S301: Based on the pre-trained encoder, the processed frequency domain features are encoded to obtain the encoded features of the noise reduction frequency data.

[0102] Specifically, the terminal trains the encoder to be trained using mixed audio samples to obtain a pre-trained encoder; wherein, the pre-trained encoder can be in the form of a two-dimensional matrix, that is, the pre-trained encoder can be represented as an encoding matrix E∈R. P×N R represents a real number, P represents the number of preset frequency points, and N represents the feature dimension of the decoder. N can be set to 128 or 256, or other dimensions. It is understandable that the encoding matrix can also have a real part and an imaginary part, represented by a two-dimensional real matrix E. r ∈R P×N And the imaginary two-dimensional matrix E i ∈R P×N Then, the terminal encodes the processed frequency domain features based on the pre-trained encoder. This can be done by multiplying the transpose of the encoding matrix with the processed frequency domain features, i.e., F = E. T Y, F∈R N×C C represents the feature dimension of the coding feature (which is also equal to the feature dimension of the frequency domain feature), and the terminal obtains the coding feature F of the noise reduction frequency data.

[0103] Step S302: Based on the pre-trained denoiser, the encoded features are denoised to obtain the denoised encoded features of the noise frequency data to be denoised.

[0104] Specifically, the terminal trains the noise denoiser to be trained by mixing audio samples to obtain a pre-trained encoder; then, the terminal inputs the real and imaginary parts of the encoded features into the pre-trained encoder to obtain masking features, such as... Figure 4 As shown, the terminal can simultaneously input the real and imaginary parts of the encoded features into the real part processing network and the imaginary part processing network in the pre-trained encoder. The real part processing network outputs a real part masking sequence, and the imaginary part processing network outputs an imaginary part masking sequence. The terminal constructs a masking feature based on the real and imaginary part masking sequences. The masking feature is a sequence consisting of the probability that each time-frequency point in the encoded features belongs to normal audio (or noise). This probability is predicted by the pre-trained encoder; the closer the probability of a time-frequency point is to 1, the greater the probability that the time-frequency point belongs to normal audio. The terminal performs a dot product between the real and imaginary parts of the encoded features and the masking feature to obtain the denoised real part and the denoised imaginary part of the encoded features. Based on the denoised real and imaginary parts, the terminal obtains the denoised encoded features of the noise-generating frequency data.

[0105] In practical applications, the masking features can be denoted as M = M r +iM i M r M represents the real part masking sequence. i This represents the imaginary part masking sequence. The terminal then performs a dot product between the masking feature M and the encoded feature F, which can be represented by the following formulas (7) and (8):

[0106] S r =F r *M r -F i *M i (7)

[0107] S i =F r *M i +F i *M r (8)

[0108] Among them, S r S represents the real part after noise reduction; i This represents the imaginary part after noise reduction.

[0109] The terminal obtains the denoised coding features of the noise-reduced frequency data based on the denoised real and imaginary parts, i.e., S = S0 r +iS i .

[0110] Step S303: Based on the pre-trained decoder, the denoised encoded features are decoded to obtain the denoised frequency domain features of the frequency data to be denoised.

[0111] Specifically, the terminal trains the decoder to be trained by mixing audio samples to obtain a pre-trained decoder; wherein, the pre-trained decoder can be in the form of a two-dimensional matrix, that is, the pre-trained encoder can be represented as a decoding matrix D∈R. N×P Similar to the pre-trained encoder in step S301 above, the terminal decodes the denoised encoded features based on the pre-trained decoder. This can be achieved by multiplying the transpose of the decoding matrix with the denoised encoded features, i.e., V = D. T S, V∈R P×C Then the terminal obtains the denoised frequency domain features V of the noise-reduced frequency data to be reduced.

[0112] In this embodiment, the processed frequency domain features are first encoded using a pre-trained encoder to obtain richer coded features. Then, a pre-trained denoiser is used to denoise the coded features, utilizing a mask sequence to eliminate noise and generate cleaner denoised coded features. Finally, a pre-trained decoder decodes the denoised coded features, restoring them to waveform signal form, thus obtaining the denoised frequency domain features of the frequency data to be denoised. Furthermore, the spectral compression of the frequency domain features by the spectral compressor in the preceding steps enables the denoiser to more accurately identify noise at time-frequency points in the coded features, improving the denoising effect of the processed frequency domain features.

[0113] In one embodiment, step S101, obtaining the frequency domain features of the noise reduction frequency data, specifically includes the following: performing frame segmentation processing on the noise reduction frequency data to obtain the noise reduction frequency frame of the noise reduction frequency data; performing Fourier transform on the noise reduction frequency frame to obtain the frequency domain features of the noise reduction frequency data.

[0114] Specifically, the terminal can label the noise frequency data to be reduced as x∈R 1×T Where T represents the number of sampling points for the noise frequency data to be reduced. The noise frequency to be reduced can be in the form of a waveform signal; the terminal performs frame processing on the noise frequency x to be reduced, which can be done by dividing the noise frequency x into C frames according to the frame length L (usually a power of 2, such as 1024) and the frame shift M (which can be 0.5L), thus the terminal obtains C frames of the noise frequency to be reduced. A schematic diagram of the noise frequency frame to be reduced is shown below. Figure 5 As shown, Figure 5 The frame shift of the noise reduction frequency frame in the C-frame is 10ms, and the frame length is 20ms. Then, the terminal performs a Fourier transform on the noise reduction frequency frame in the C-frame to obtain the frequency domain characteristics of the noise reduction frequency frame in the C-frame. The frequency domain characteristics of the noise reduction frequency frame in the C-frame together constitute the frequency domain characteristics of the noise reduction frequency data.

[0115] In this embodiment, by performing frame segmentation processing on the noise reduction frequency data, the noise reduction frequency data can be divided into noise reduction frequency frames; then, Fourier transform is performed on the noise reduction frequency frames to obtain the frequency domain features of the noise reduction frequency data, so that the subsequent audio noise reduction steps can be performed based on the frequency domain features.

[0116] In one embodiment, the audio denoising method further includes the step of training a pre-trained spectrum compressor, a pre-trained encoder, a pre-trained denoiser, and a pre-trained decoder, specifically including the following: acquiring noise-free audio samples and mixed audio samples corresponding to the noise-free audio samples; inputting the frequency domain features of the mixed audio samples into the audio denoising model to be trained to obtain the denoised frequency domain features of the mixed audio samples; obtaining the loss value of the audio denoising model to be trained based on the difference between the denoised frequency domain features of the mixed audio samples and the frequency domain features of the noise-free audio samples; training the audio denoising model to be trained based on the loss value to obtain the pre-trained audio denoising model; the pre-trained audio denoising model includes a pre-trained spectrum compressor, a pre-trained encoder, a pre-trained denoiser, and a pre-trained decoder.

[0117] An audio denoising model is a model used to reduce noise in audio data or audio samples (such as audio data to be denoised or mixed audio samples). An audio denoising model includes a spectrum compressor, encoder, denoiser, and decoder.

[0118] Among them, noise-free audio samples refer to pure speech signals that do not contain noise.

[0119] Specifically, the terminal acquires noise-free audio samples and their corresponding mixed audio samples. It then performs a Fourier transform on the mixed audio samples to obtain their frequency domain features. The terminal inputs these features into the audio denoising model to be trained, sequentially performing spectral compression, encoding, denoising, and decoding. The trained audio denoising model outputs the denoised frequency domain features of the mixed audio samples. Using noise-free audio samples as the training target, the terminal obtains the loss value of the audio denoising model to be trained based on the difference between the denoised frequency domain features of the mixed audio samples and the frequency domain features of the noise-free audio samples. This loss value can be SI-SDR (Scale-invariant Signal-to-Distortion Ratio) or MSE (Mean Square Error). Based on the loss value, the terminal iteratively trains the spectral compressor, encoder, denoiser, and decoder in the audio denoising model to be trained until the trained audio denoising model meets a preset training termination condition. The terminal then obtains the pre-trained audio denoising model.

[0120] In this embodiment, the frequency domain features of the mixed audio samples are input into the audio denoising model to be trained to obtain the denoised frequency domain features of the mixed audio samples; then, based on the difference between the denoised frequency domain features of the mixed audio samples and the frequency domain features of the noise-free audio samples, the loss value of the audio denoising model to be trained is obtained; based on the loss value, the audio denoising model to be trained is trained so that the denoised frequency domain features output by the audio denoising model continuously approach the frequency domain features of the noise-free audio samples, thereby improving the denoising capability of the pre-trained audio denoising model.

[0121] In one embodiment, obtaining a mixed audio sample corresponding to a noise-free audio sample specifically includes the following: obtaining a noise sample and a preset signal-to-noise ratio; and performing a fusion process on the noise-free audio sample and the noise sample according to the preset signal-to-noise ratio to obtain a mixed audio sample corresponding to the noise-free audio sample.

[0122] The preset signal-to-noise ratio can be obtained by filtering from a preset signal-to-noise ratio range. For example, the preset signal-to-noise ratio range can be set to [-5, 20].

[0123] Specifically, the terminal determines the preset signal-to-noise ratio from the preset signal-to-noise ratio range and obtains noise samples and noise-free audio samples; then, the terminal performs a fusion process on the noise-free audio samples and noise samples according to the preset signal-to-noise ratio, which can be a mixture of noise-free audio samples and noise samples according to the preset signal-to-noise ratio, so that the terminal obtains the mixed audio sample corresponding to the noise-free audio sample.

[0124] In this embodiment, after determining the preset signal-to-noise ratio, the noise-free audio samples and the noise samples are fused according to the preset signal-to-noise ratio to obtain the mixed audio samples corresponding to the noise-free audio samples. This can simulate the complex noise and non-stationary noise in real singing scenarios, so that the audio denoising model trained by the mixed audio samples also has a strong denoising capability for audio data carrying complex noise and non-stationary noise, thereby improving the denoising effect of the pre-trained audio denoising model on the audio data to be denoised.

[0125] In one embodiment, such as Figure 6 As shown, another audio noise reduction method is provided. Taking the application of this method to a terminal as an example, the method includes the following steps:

[0126] Step S601: Perform frame segmentation on the noise reduction frequency data to be processed to obtain the noise reduction frequency frame of the noise reduction frequency data.

[0127] Step S602: Perform Fourier transform on the frequency frame to be noise-reduced to obtain the frequency domain characteristics of the frequency data to be noise-reduced.

[0128] Step S603: Obtain the amplitude and phase features of the frequency domain features; perform spectral compression processing on the amplitude and phase features according to the pre-trained spectrum compressor to obtain the processed frequency domain features of the noise-reduced frequency data.

[0129] Step S604: Based on the pre-trained encoder, the processed frequency domain features are encoded to obtain the encoded features of the noise-reduced frequency data.

[0130] Step S605: Based on the pre-trained denoiser, the encoded features are denoised to obtain the denoised encoded features of the noise-reduced frequency data.

[0131] Step S606: Based on the pre-trained decoder, the denoised encoded features are decoded to obtain the denoised frequency domain features of the frequency data to be denoised.

[0132] Step S607: Determine the target amplitude features and target phase features of the frequency domain features after noise reduction; perform audio enhancement processing on the target amplitude features and target phase features according to the pre-trained spectrum compressor to obtain the enhanced frequency domain features.

[0133] Step S608: Perform signal reconstruction processing on the enhanced frequency domain features to obtain the target noise-reduced frequency data corresponding to the noise-reduced frequency data.

[0134] The above-mentioned audio noise reduction method can achieve the following beneficial effects: by performing appropriate compression transformation on the frequency domain characteristics of the noise-reduced audio data through a spectrum compressor, more detailed frequency information in the noise-reduced audio data can be captured, which is beneficial for reconstructing the weak spectrum information caused by sudden noise or reverberation in the noise-reduced audio data. This effectively improves the signal quality of the reconstructed target noise-reduced audio data, thereby effectively improving the noise reduction effect of the noise-reduced audio data.

[0135] To more clearly illustrate the audio noise reduction method provided in this disclosure, a specific embodiment is given below for detailed description. Another audio noise reduction method is also provided, which can be applied to a terminal, and specifically includes the following:

[0136] like Figure 7 As shown, the terminal acquires the noise reduction frequency data, then performs frame segmentation on the noise reduction frequency data to obtain the noise reduction frequency frame of the noise reduction frequency data; and performs Fourier transform on the noise reduction frequency frame to obtain the frequency domain characteristics of the noise reduction frequency data.

[0137] The terminal inputs the frequency domain features into a pre-trained audio denoising model and obtains the real and imaginary parts of the frequency domain features in complex form. Then, based on the real and imaginary parts of the frequency domain features, it calculates the amplitude and phase features. The amplitude and phase features are then compressed using a pre-trained spectrum compressor within the pre-trained audio denoising model, resulting in the processed frequency domain features of the audio data to be denoised. The processed real and imaginary parts of the processed frequency domain features are then encoded using a pre-trained encoder within the pre-trained audio denoising model, resulting in the encoded features of the audio data to be denoised. The encoded real and imaginary parts of the encoded features are then denoised using a pre-trained denoiser within the pre-trained audio denoising model, resulting in the denoised encoded features of the audio data to be denoised. Finally, the denoised encoded features are decoded using a pre-trained decoder within the pre-trained audio denoising model, resulting in the denoised frequency domain features of the audio data to be denoised. The pre-trained audio denoising model then outputs the denoised frequency domain features.

[0138] The terminal calculates the target amplitude and target phase features of the denoised frequency domain features, and then performs audio enhancement processing on the target amplitude and target phase features according to the pre-trained spectrum compressor to obtain the enhanced frequency domain features; the enhanced frequency domain features are then subjected to short-time inverse Fourier transform to obtain the target denoised frequency data corresponding to the noise-reduced frequency data.

[0139] In this embodiment, by appropriately compressing and transforming the frequency domain features of the noise-reduced audio data using a spectrum compressor, more detailed frequency information in the noise-reduced audio data can be captured. This is beneficial for reconstructing weak spectral information caused by sudden noise or reverberation in the noise-reduced audio data, solving the defect in traditional technology where human voice signals are easily distorted due to noise reduction processing. This improves the fidelity of human voice signals in the noise-reduced audio data and can further enhance the spectral information in the frequency domain features after noise reduction. As a result, the target noise-reduced audio data reconstructed through the enhanced spectral features has higher clarity, greatly improving the noise reduction effect of the target noise-reduced audio data.

[0140] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0141] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an audio noise reduction method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0142] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0143] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0144] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0145] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0146] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0147] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0148] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An audio noise reduction method, characterized in that, The method includes: Obtain the frequency domain characteristics of the noise reduction frequency data; Based on the spectrum compression matrix corresponding to the pre-trained spectrum compressor, the amplitude features of the frequency domain features are exponentially processed to obtain the processed amplitude features. The phase features of the frequency domain features are processed by cosine function and sine function respectively to obtain the phase features after cosine processing and the phase features after sine processing. Based on the processed amplitude characteristics and the processed phase characteristics of the cosine, the processed real part is obtained, and based on the processed amplitude characteristics and the processed phase characteristics of the sine, the processed imaginary part is obtained. Based on the processed real part and the processed imaginary part, the processed frequency domain features of the noise reduction frequency data are obtained; The processed frequency domain features are subjected to noise reduction processing to obtain the noise-reduced frequency domain features of the frequency data to be noise-reduced; Based on the reciprocal of the spectrum compression matrix corresponding to the pre-trained spectrum compressor, the target amplitude feature of the denoised frequency domain feature is exponentially processed to obtain the processed target amplitude feature. The target phase features of the noise-reduced frequency domain features are processed by cosine function and sine function respectively to obtain the phase features after cosine processing and the phase features after sine processing. Based on the processed target amplitude characteristics, the cosine-processed phase characteristics, and the sine-processed phase characteristics, the enhanced frequency domain characteristics are obtained. The enhanced frequency domain features are processed for signal reconstruction to obtain the target noise-reduced frequency data corresponding to the noise-reduced frequency data.

2. The method according to claim 1, characterized in that, Before performing exponential processing on the amplitude features of the frequency domain features according to the spectral compression matrix corresponding to the pre-trained spectral compressor to obtain the processed amplitude features, the process further includes: The amplitude and phase characteristics of the frequency domain features are obtained from the real and imaginary parts of the frequency domain features.

3. The method according to claim 1, characterized in that, Before obtaining the target noise-reduced frequency data corresponding to the noise-reduced frequency data to be reduced, the process also includes: Determine the target amplitude and target phase characteristics of the frequency domain features after noise reduction.

4. The method according to claim 1, characterized in that, The step of denoising the processed frequency domain features to obtain the denoised frequency domain features of the frequency data to be denoised includes: The processed frequency domain features are encoded according to the pre-trained encoder to obtain the encoded features of the noise reduction frequency data; Based on a pre-trained denoiser, the encoded features are denoised to obtain the denoised encoded features of the noise-reduced frequency data. The denoised encoded features are decoded using a pre-trained decoder to obtain the denoised frequency domain features of the noise-reduced frequency data.

5. The method according to claim 1, characterized in that, The frequency domain features of the noise reduction frequency data to be acquired include: The noise frequency data to be reduced is processed into frames to obtain the noise frequency frames to be reduced. Perform a Fourier transform on the frequency frame to be noise-reduced to obtain the frequency domain characteristics of the frequency data to be noise-reduced.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Obtain noise-free audio samples and corresponding mixed audio samples; The frequency domain features of the mixed audio samples are input into the audio denoising model to be trained to obtain the denoised frequency domain features of the mixed audio samples. The loss value of the audio denoising model to be trained is obtained based on the difference between the frequency domain features of the denoised mixed audio samples and the frequency domain features of the noise-free audio samples. Based on the loss value, the audio denoising model to be trained is trained to obtain a pre-trained audio denoising model; the pre-trained audio denoising model includes the pre-trained spectrum compressor, the pre-trained encoder, the pre-trained denoiser, and the pre-trained decoder.

7. The method according to claim 6, characterized in that, The step of obtaining the mixed audio sample corresponding to the noise-free audio sample includes: Acquire noise samples and preset signal-to-noise ratio; Based on the preset signal-to-noise ratio, the noise-free audio sample and the noise sample are fused to obtain the mixed audio sample corresponding to the noise-free audio sample.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images