Audio processing method and apparatus, device, and computer-readable storage medium

Abstract

An audio processing method and apparatus, a device, and a computer-readable storage medium, relating to the technical field of audio processing. The audio processing method comprises: acquiring an input sound pickup signal (S100); detecting the input sound pickup signal, and identifying a wind noise frame in the input sound pickup signal (S200); and inputting the wind noise frame into a preset noise reduction model to obtain a noise reduction sound pickup signal corresponding to the wind noise frame, wherein the preset noise reduction model is obtained by training a generative adversarial network using a user speech vector, a noisy sample and a pure sample (S300). The method solves the technical problem that the suppression effect of the current pickup device on environmental noise is poor.

Description

Audio processing method, device, equipment and computer-readable storage medium Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio processing method, apparatus, device, and computer-readable storage medium. Background Art

[0002] Wind noise is the most common type of ambient noise in outdoor sound pickup, severely impacting the quality of outdoor sound pickup systems such as mobile phones and hearing aids. Wind noise is caused by the interaction between airflow and obstacles (such as buildings, the human body, and the microphone cavity). Therefore, the characteristics of wind noise caused by different obstacles vary. In many cases, wind noise sound pressure levels can reach 80dB SPL, completely masking speech signals, significantly reducing speech intelligibility and causing auditory discomfort.

[0003] Traditional methods for dealing with wind noise include windscreens, commonly found on handheld microphones and professional shotgun microphones. These come in a variety of materials, including sponge, artificial fur, and iron mesh. Their principle is to reduce the air velocity near the microphone diaphragm, breaking up the airflow and reducing turbulence. Windscreens can also be used on small pickup devices to suppress wind noise, but they require a larger size to improve noise reduction. Even with windscreens, small pickup devices like mobile phones and TWS earphones have very limited wind noise suppression.

[0004] Therefore, how to effectively reduce the ambient noise of the sound pickup device is a technical problem that those skilled in the art urgently need to solve.

[0005] Summary of the Invention

[0006] The main purpose of this application is to provide an audio processing method, apparatus, device and computer-readable storage medium, aiming to solve the technical problem that current sound pickup devices have poor suppression effect on ambient noise.

[0007] To achieve the above objectives, the present application provides an audio processing method, which is applied to an audio processing device. The audio processing method includes:

[0008] Get the input pickup signal;

[0009] detecting the input sound pickup signal and identifying a wind noise frame in the input sound pickup signal;

[0010] The wind noise frame is input into a preset noise reduction model to obtain a noise-reduced sound pickup signal corresponding to the wind noise frame, wherein the preset noise reduction model is obtained by training a generative adversarial network using a user voice vector, noisy samples, and clean samples.

[0011] Optionally, the preset noise reduction model includes an encoding layer, a recurrent neural network, an attention network, a decoding layer, and a fully connected layer. The step of inputting the wind noise frame into the preset noise reduction model to obtain the noise-reduced sound pickup signal corresponding to the wind noise frame includes:

[0012] Input the wind noise frame into the encoding layer of the preset noise reduction model for downsampling to obtain each encoded sound pickup feature;

[0013] Inputting the encoded sound pickup feature into the recurrent neural network so that the recurrent neural network outputs a corresponding user sound pickup feature based on the user voice vector, the preset random noise vector and the encoded sound pickup feature;

[0014] Inputting the encoded voice pickup feature and the user voice pickup feature into the attention network to obtain an attention feature;

[0015] Inputting the user sound pickup features into the decoding layer for upsampling to obtain each decoded sound pickup feature;

[0016] The attention feature is spliced ​​to the corresponding decoded sound pickup feature to obtain a spliced ​​sound pickup feature, and the spliced ​​sound pickup feature is input into a fully connected layer to obtain a noise reduction sound pickup signal corresponding to the wind noise frame.

[0017] Optionally, the input sound pickup signal includes at least two paths of sound pickup signals, and the step of detecting the input sound pickup signal and identifying a wind noise frame in the input sound pickup signal includes:

[0018] Acquiring frequency spectrum characteristics of each of the picked-up sound signals, and determining frequency spectrum correlations of each of the picked-up sound signals based on the frequency spectrum characteristics;

[0019] performing low-pass filtering on the main-path sound pickup signal in each of the sound pickup signals to determine the low-frequency energy of the input sound pickup signal;

[0020] determining the wind speed of each signal frame in the input sound pickup signal according to the spectral correlation and the low-frequency energy;

[0021] According to the wind speed, a wind noise frame in each of the signal frames is identified.

[0022] Optionally, before the step of detecting the input sound pickup signal and identifying a wind noise frame in the input sound pickup signal, the method further includes:

[0023] Performing echo cancellation processing on the input sound pickup signal to obtain an echo cancellation signal;

[0024] The echo cancellation signal is used as a new input sound pickup signal, and the following steps are performed: the input sound pickup signal is detected to identify wind noise frames in the input sound pickup signal.

[0025] Optionally, before the step of detecting the input sound pickup signal and identifying the wind noise frame in the input sound pickup signal, the method further includes:

[0026] Performing beamforming processing on the input sound pickup signal to obtain a directional enhancement signal;

[0027] The directional enhancement signal is used as a new input sound pickup signal, and the following steps are performed: the input sound pickup signal is detected to identify wind noise frames in the input sound pickup signal.

[0028] Optionally, after the step of inputting the wind noise frame into a preset noise reduction model to obtain a noise reduction pickup signal corresponding to the wind noise frame, the audio processing method further includes:

[0029] determining wind-noise-free frames other than the wind-noise frames in the input sound pickup signal, and calculating a posterior signal-to-noise ratio of the wind-noise-free frames;

[0030] Based on the a posteriori signal-to-noise ratio, performing noise elimination processing on the wind-noise-free frame to obtain an enhanced sound pickup signal;

[0031] Dynamic range control is performed on the noise reduction sound pickup signal and the enhanced sound pickup signal to obtain an output sound pickup signal.

[0032] Optionally, before the step of inputting the wind noise frame into a preset noise reduction model to obtain the noise reduction pickup signal corresponding to the wind noise frame, the step includes:

[0033] Obtain user speech vectors, noisy samples, and clean samples;

[0034] Based on the noisy samples and the clean samples, a discriminant network in a generative adversarial network is trained to obtain a trained discriminator;

[0035] Inputting a preset random noise vector, the user voice vector, and the noisy sample into a generative network in the generative adversarial network to obtain a noise-reduced sample;

[0036] Inputting the denoised sample into the discriminator to obtain a loss function value of the generating network;

[0037] The generation network is updated based on the loss function value until the generation network converges to obtain a preset noise reduction model.

[0038] The present application also provides an audio processing device, which is applied to an audio processing device. The audio processing device includes:

[0039] An acquisition module, used to acquire an input pickup signal;

[0040] a detection module, configured to detect the input sound pickup signal and identify wind noise frames in the input sound pickup signal;

[0041] The noise reduction module is used to input the wind noise frame into a preset noise reduction model to obtain a noise-reduced sound pickup signal corresponding to the wind noise frame, wherein the preset noise reduction model is obtained by training a generative adversarial network using user voice vectors, noisy samples and clean samples.

[0042] Optionally, the preset denoising model includes an encoding layer, a recurrent neural network, an attention network, a decoding layer and a fully connected layer, and a denoising module, and is further used to:

[0043] Input the wind noise frame into the encoding layer of the preset noise reduction model for downsampling to obtain each encoded sound pickup feature;

[0044] Inputting the encoded sound pickup feature into the recurrent neural network so that the recurrent neural network outputs a corresponding user sound pickup feature based on the user voice vector, the preset random noise vector and the encoded sound pickup feature;

[0045] Inputting the encoded voice pickup feature and the user voice pickup feature into the attention network to obtain an attention feature;

[0046] Inputting the user sound pickup features into the decoding layer for upsampling to obtain each decoded sound pickup feature;

[0047] The attention feature is spliced ​​to the corresponding decoded sound pickup feature to obtain a spliced ​​sound pickup feature, and the spliced ​​sound pickup feature is input into a fully connected layer to obtain a noise reduction sound pickup signal corresponding to the wind noise frame.

[0048] Optionally, the input sound pickup signal includes at least two paths of sound pickup signals, and the detection module is further configured to:

[0049] Acquiring frequency spectrum characteristics of each of the picked-up sound signals, and determining frequency spectrum correlations of each of the picked-up sound signals based on the frequency spectrum characteristics;

[0050] performing low-pass filtering on the main-path sound pickup signal in each of the sound pickup signals to determine the low-frequency energy of the input sound pickup signal;

[0051] determining the wind speed of each signal frame in the input sound pickup signal according to the spectral correlation and the low-frequency energy;

[0052] According to the wind speed, a wind noise frame in each of the signal frames is identified.

[0053] Optionally, the audio processing device further includes a pre-processing module configured to:

[0054] Performing echo cancellation processing on the input sound pickup signal to obtain an echo cancellation signal;

[0055] The echo cancellation signal is used as a new input sound pickup signal, and the following steps are performed: the input sound pickup signal is detected to identify wind noise frames in the input sound pickup signal.

[0056] Optionally, the preprocessing module is further configured to:

[0057] Performing beamforming processing on the input sound pickup signal to obtain a directional enhancement signal;

[0058] The directional enhancement signal is used as a new input sound pickup signal, and the following steps are performed: the input sound pickup signal is detected to identify wind noise frames in the input sound pickup signal.

[0059] Optionally, the audio processing device further includes an output module configured to:

[0060] determining wind-noise-free frames other than the wind-noise frames in the input sound pickup signal, and calculating a posterior signal-to-noise ratio of the wind-noise-free frames;

[0061] Based on the a posteriori signal-to-noise ratio, performing noise elimination processing on the wind-noise-free frame to obtain an enhanced sound pickup signal;

[0062] Dynamic range control is performed on the noise reduction sound pickup signal and the enhanced sound pickup signal to obtain an output sound pickup signal.

[0063] Optionally, the audio processing device further includes a model training module, configured to:

[0064] Obtain user speech vectors, noisy samples, and clean samples;

[0065] Based on the noisy samples and the clean samples, a discriminant network in a generative adversarial network is trained to obtain a trained discriminator;

[0066] Inputting a preset random noise vector, the user voice vector, and the noisy sample into a generative network in the generative adversarial network to obtain a noise-reduced sample;

[0067] Inputting the denoised sample into the discriminator to obtain a loss function value of the generating network;

[0068] The generation network is updated based on the loss function value until the generation network converges to obtain a preset noise reduction model.

[0069] The present application also provides an audio processing device, which includes: a memory, a processor, and a program of an audio processing method stored in the memory and runnable on the processor. When the program of the audio processing method is executed by the processor, the steps of the audio processing method described above can be implemented.

[0070] The present application also provides a computer-readable storage medium, on which a program for implementing the audio processing method is stored. When the program of the audio processing method is executed by a processor, the steps of the audio processing method as described above are implemented.

[0071] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned audio processing method when executed by a processor.

[0072] The present application provides an audio processing method, apparatus, device, and computer-readable storage medium. By acquiring an input sound pickup signal and detecting the input sound pickup signal, wind noise frames within the input sound pickup signal are identified. This method enables the identification of signal frames containing wind noise (i.e., wind noise frames), thereby avoiding wind noise suppression for signal frames without wind noise. This improves the efficiency of wind noise suppression and avoids affecting signal frames without wind noise. The wind noise frames are then input into a preset noise reduction model to obtain a noise-reduced sound pickup signal corresponding to the wind noise frames. The preset noise reduction model is obtained by training a generative adversarial network using user speech vectors, noisy samples, and clean samples. Because the preset noise reduction model is obtained by training a generative adversarial network using user speech vectors, noisy samples, and clean samples, the preset noise reduction model can more accurately identify the user's speech within the input sound pickup signal based on the user speech vectors, thereby suppressing wind noise and other environmental noises such as human voices in the sound pickup environment. This application thus addresses the technical problem of current sound pickup devices having poor environmental noise suppression performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0074] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0075] FIG1 is a flow chart of a first embodiment of an audio processing method of the present application;

[0076] FIG2 is a flowchart of a second embodiment of the audio processing method of the present application;

[0077] FIG3 is a schematic diagram of the structure of the attention network involved in the embodiment of the present application;

[0078] FIG4 is a schematic diagram of the structure of a preset noise reduction model involved in an embodiment of the present application;

[0079] FIG5 is a schematic structural diagram of an audio processing device according to an embodiment of the present application;

[0080] FIG6 is a schematic diagram of the device structure of the hardware operating environment involved in an embodiment of the present application.

[0081] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0082] To make the above-mentioned purposes, features, and advantages of the present application more clearly understood, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making any creative work are within the scope of protection of this application.

[0083] Refer to FIG. 1 , which is a flow chart of a first embodiment of an audio processing method of the present application.

[0084] The present application provides an audio processing method. In a first embodiment of the audio processing method of the present application, the audio processing method includes:

[0085] Step S100, obtaining an input sound pickup signal;

[0086] In this embodiment, it should be noted that the audio processing method is applied to an audio processing device, which may be an AR (Augmented Reality) device, a VR (Virtual Reality) device, a TWS (True Wireless Stereo) headset, a smartphone, a PC (Personal Computer), a tablet computer, a portable computer or a server.

[0087] As an example, this embodiment can collect sound using at least two microphones to obtain at least two picked-up sound signals as the input picked-up sound signals. Alternatively, a remote audio signal sent by a remote device (such as a mobile phone, a surveillance camera, etc.) can be received and used as the input picked-up sound signal, wherein the remote audio signal includes at least two picked-up sound signals.

[0088] As another example, after obtaining the input sound pickup signal, the input sound pickup signal can be preprocessed and used as a new input sound pickup signal. The preprocessing includes pre-emphasis, framing, windowing, and other processing operations to improve the quality of subsequent audio processing.

[0089] Step S200, detecting the input sound pickup signal and identifying a wind noise frame in the input sound pickup signal;

[0090] In this embodiment, it should be noted that it is understandable that the input sound pickup signal may exist in the form of frames, that is, the input sound pickup signal includes signal frames. In addition, the input sound pickup signal includes at least two sound pickup signals.

[0091] As an example, the spectral characteristics of each of the picked-up sound signals can be obtained, and the spectral correlation of each of the picked-up sound signals can be determined based on the spectral characteristics. Furthermore, based on the spectral correlation, whether wind noise exists in each of the signal frames in the input picked-up sound signal is determined, thereby identifying a wind noise frame in the input picked-up sound signal. Exemplarily, when the spectral correlation of each of the picked-up sound signals in the signal frame exceeds a first correlation threshold, it is determined that wind noise does not exist in the signal frame; and when the spectral correlation of each of the picked-up sound signals in the signal frame is lower than a second correlation threshold, it is determined that wind noise does not exist in the signal frame and the signal frame is a wind noise frame. The second correlation threshold is lower than the first correlation threshold.

[0092] As another example, the input sound pickup signal can be detected by selecting one of the sound pickup signals as the main sound pickup signal, calculating the low-frequency energy of each signal frame in the main sound pickup signal, obtaining spectral characteristics of each of the sound pickup signals, and determining the spectral correlation of each of the sound pickup signals based on the spectral characteristics. The wind speed of each signal frame in the input sound pickup signal is then determined based on the spectral correlation and the low-frequency energy, and wind noise frames are identified in each of the signal frames based on the wind speed.

[0093] The input sound pickup signal in step S200 includes at least two paths of sound pickup signals, and the step of detecting the input sound pickup signal and identifying the wind noise frame in the input sound pickup signal includes:

[0094] Step S210, obtaining the spectrum characteristics of each of the picked-up sound signals, and determining the spectrum correlation of each of the picked-up sound signals according to the spectrum characteristics;

[0095] Step S220, performing low-pass filtering on the main-channel sound pickup signal in each of the sound pickup signals to determine the low-frequency energy of the input sound pickup signal;

[0096] Step S230, determining the wind speed of each signal frame in the input sound pickup signal according to the spectrum correlation and the low-frequency energy;

[0097] Step S240: identifying wind noise frames in each of the signal frames according to the wind speed.

[0098] In this embodiment, it should be noted that the frequency spectrum characteristics may include characteristics such as the frequency and bandwidth of each of the picked-up sound signals.

[0099] As an example, taking the case where the input sound pickup signal includes two sound pickup signals, the following preset correlation function formula can be used to calculate the spectral correlation of each of the sound pickup signals. The preset correlation function formula is as follows:

[0100] Spectral correlation

[0101] Among them, O 12 , 0 22 are the elements in the matrix corresponding to the following spectral characteristics:

[0102] in, Indicates the number of points where the first channel's pickup signal is positive from time 0 to k. Indicates the number of points where the second channel's pickup signal is positive at time 0 to k. Indicates the number of points where the first channel's pickup signal is negative from time 0 to k. Indicates the number of points where the second channel's sound pickup signal is negative at time 0 to k.

[0103] As an example, taking the case where the input sound pickup signal includes two sound pickup signals, the Fourier transform can be performed on each of the sound pickup signals, and then substituted into a preset coherence coefficient calculation formula to calculate the spectral correlation of each of the sound pickup signals. The preset coherence coefficient calculation formula is as follows:

[0104] Spectral correlation

[0105] Wherein, Y1(K) is the Fourier transform of the first-channel picked-up signal in the bandwidth 1 to 32, and Y2(K) is the Fourier transform of the second-channel picked-up signal in the bandwidth 1 to 32.

[0106] In this way, the frequency spectrum correlation of each of the picked-up sound signals can be calculated.

[0107] Furthermore, in this embodiment, the low-frequency energy of the input sound pickup signal can be determined by low-pass filtering the main sound pickup signal in each of the sound pickup signals. It can be understood that the main sound pickup signal is the sound pickup signal collected by the main microphone (such as the feedforward microphone) in each of the sound pickup signals. Of course, any sound pickup signal from each of the sound pickup signals can also be selected as the main sound pickup signal. Then, the main sound pickup signal can be passed through a low-pass filter, and the main sound pickup signal can be low-pass filtered based on a preset upper cutoff frequency (such as 1300Hz, 1500Hz, 1800Hz), wherein the low-pass filter can generally be implemented using an IIR (Infinite Impulse Response) filter, and of course other digital filters can also be used. Then, based on the main sound pickup signal after low-pass filtering, the low-frequency energy of each signal frame in the input sound pickup signal is calculated. Exemplarily, the calculation formula for the low-frequency energy is as follows:

[0108] Among them, P low is low frequency energy, x1 LP is the main sound pickup signal after low-pass filtering, and k is time.

[0109] In this embodiment, a preset correspondence between spectral correlation, low-frequency energy, and wind speed can be pre-set. The preset correspondence can be expressed in the form of a mapping table, function, or the like. Thus, this embodiment can query the preset correspondence based on the spectral correlation and the low-frequency energy to obtain the wind speed of each signal frame in the input sound pickup signal. Signal frames whose wind speed exceeds a preset wind speed threshold (e.g., 0.1 m / s, 0.2 m / s, 0.3 m / s, etc.) are then identified as wind noise frames, thereby identifying wind noise frames among the signal frames.

[0110] In this embodiment, the wind speed of each frame in the input sound pickup signal is determined by combining the spectral correlation of each sound pickup signal in each signal frame and the low-frequency energy of each signal frame, thereby identifying wind noise frames. This more accurately filters out wind noise frames from the input sound pickup signal, allowing for targeted noise reduction of these wind noise frames, effectively improving the subsequent noise reduction effect.

[0111] Before the step of detecting the input sound pickup signal and identifying the wind noise frame in the input sound pickup signal in step S100, the following steps are included:

[0112] Step S110, performing echo cancellation processing on the input sound pickup signal to obtain an echo-cancelled signal;

[0113] Step S111 : using the echo-cancelled signal as a new input sound pickup signal, and executing the following steps: detecting the input sound pickup signal, and identifying wind noise frames in the input sound pickup signal.

[0114] Because the input sound pickup signal may contain not only ambient noise but also echoes, this embodiment, before detecting the input sound pickup signal, performs echo cancellation on the input sound pickup signal to obtain an echo-cancelled signal. This echo-cancelled signal is then used as a new input sound pickup signal to perform the following steps: detecting the input sound pickup signal and identifying wind noise frames within the input sound pickup signal. This embodiment effectively eliminates echoes from the input sound pickup signal, thereby improving the accuracy of wind noise frame identification within the input sound pickup signal and enhancing subsequent noise reduction effects.

[0115] Before the step of detecting the input sound pickup signal and identifying the wind noise frame in the input sound pickup signal in step S100, the method further includes:

[0116] Step S120, performing beamforming processing on the input sound pickup signal to obtain a directional enhancement signal;

[0117] Step S121 : using the directional enhancement signal as a new input sound pickup signal, executing the following steps: detecting the input sound pickup signal, and identifying wind noise frames in the input sound pickup signal.

[0118] In this embodiment, it should be noted that the beamforming (or spatial filtering) process is to perform weighted summation on the input sound pickup signals to suppress noise other than directivity.

[0119] To enhance the signal from a specific direction and eliminate or suppress noise from other directions, this embodiment performs beamforming processing on the input sound pickup signal to obtain a directional-enhanced signal. This directional-enhanced signal can then be used as a new input sound pickup signal. The following steps are performed: detecting the input sound pickup signal and identifying wind noise frames within the input sound pickup signal. This embodiment effectively eliminates or suppresses noise other than directional noise in the input sound pickup signal, thereby improving the accuracy of wind noise frame identification within the input sound pickup signal and enhancing subsequent noise reduction effects.

[0120] Step S300: Input the wind noise frame into a preset noise reduction model to obtain a noise-reduced sound pickup signal corresponding to the wind noise frame, wherein the preset noise reduction model is obtained by training a generative adversarial network using a user speech vector, noisy samples, and clean samples.

[0121] In this embodiment, it should be noted that the preset noise reduction model is obtained by training a generative adversarial network using user speech vectors, noisy samples, and clean samples. The user speech vector is a feature vector obtained by extracting features from the user's speech information and can be used to identify the user's speech in the wind noise frame. The noisy samples are speech signal samples with ambient noise, and the clean samples are speech signal samples without ambient noise.

[0122] In this embodiment, the preset noise reduction model is obtained by training a generative adversarial network using user voice vectors, noisy samples, and clean samples. Therefore, the preset noise reduction model can more accurately identify the user voice in the input sound pickup signal based on the user voice vector, and thus suppress environmental noise such as wind noise and other human voices in the sound pickup environment. In this embodiment, by inputting the wind noise frame into the preset noise reduction model, the preset noise reduction model can, on the one hand, separate the human voice from the environmental noise in the wind noise frame based on the characteristics of the generative adversarial network, and on the other hand, can suppress the human voice in the wind noise frame that does not match the user voice vector based on the user voice vector. Therefore, based on the preset noise reduction model, the environmental noise in the wind noise frame and the human voice that does not match the user voice vector can be effectively suppressed, thereby effectively improving the noise reduction effect of obtaining the noise reduction pickup signal corresponding to the wind noise frame.

[0123] Wherein, after the step of inputting the wind noise frame into a preset noise reduction model to obtain a noise reduction pickup signal corresponding to the wind noise frame in step S300, the audio processing method further includes:

[0124] Step S400, determining wind-noise-free frames other than the wind-noise frames in the input sound pickup signal, and calculating a posteriori signal-to-noise ratios of the wind-noise-free frames;

[0125] Step S410: performing noise elimination processing on the wind-noise-free frame based on the posterior signal-to-noise ratio to obtain an enhanced sound pickup signal;

[0126] Step S420: Perform dynamic range control on the noise reduction sound pickup signal and the enhanced sound pickup signal to obtain an output sound pickup signal.

[0127] In this embodiment, it should be noted that the wind-noise-free frame is a signal frame in the input sound pickup signal excluding the wind-noise frame.

[0128] Since the input sound pickup signal includes a wind-free frame in which no wind noise exists in addition to the wind-free frame, it is necessary to perform noise reduction processing on the wind-free frame. The posterior signal-to-noise ratio of the wind-free frame can be calculated according to a preset signal-to-noise ratio calculation formula. The preset signal-to-noise ratio calculation formula is as follows:

[0129] Among them, γt (k) is the posterior signal-to-noise ratio of the wind-free frame, P n (k, t) is the noise power spectrum function of the wind-free frame, Y t (k) is the frame without wind noise.

[0130] Then, based on the a posteriori signal-to-noise ratio and a preset enhancement function, the wind-noise-free frame can be subjected to noise elimination processing to obtain an enhanced sound pickup signal. The preset enhancement function is as follows:

[0131] in, is the enhanced sound pickup signal, Y t (k) is the wind-noise-free frame, H t (k) is the gain function, the over-reduction factor λ is a constant, usually 0.9, γ t (k) is the posterior signal-to-noise ratio of the wind-noise-free frame.

[0132] Furthermore, dynamic range control (DRC) may be performed on both the noise reduction sound pickup signal and the enhanced sound pickup signal to obtain an output sound pickup signal.

[0133] In the first embodiment of the present application, an input sound pickup signal is obtained and detected to identify wind noise frames in the input sound pickup signal. This enables the identification of signal frames containing wind noise (i.e., wind noise frames) in the input sound pickup signal, thereby avoiding the need to perform wind noise suppression on signal frames without wind noise. This improves the efficiency of wind noise suppression and avoids the impact on signal frames without wind noise. The wind noise frames are then input into a preset noise reduction model to obtain a noise-reduced sound pickup signal corresponding to the wind noise frame, wherein the preset noise reduction model is obtained by training a generative adversarial network using a user voice vector, noisy samples, and clean samples. Since the preset noise reduction model is obtained by training a generative adversarial network using a user voice vector, noisy samples, and clean samples, the preset noise reduction model can more accurately identify the user voice in the input sound pickup signal based on the user voice vector, thereby suppressing environmental noise such as wind noise and other human voices in the sound pickup environment. This embodiment thus solves the technical problem that current sound pickup devices have a poor suppression effect on environmental noise.

[0134] Further, referring to FIG. 2 , FIG. 2 is a flow chart of a second embodiment of the audio processing method of the present application.

[0135] Based on the first embodiment of the present application, in another embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction and will not be repeated hereafter. The preset noise reduction model includes an encoding layer, a recurrent neural network, an attention network, a decoding layer, and a fully connected layer. The step of inputting the wind noise frame into the preset noise reduction model in step S300 to obtain the noise reduction pickup signal corresponding to the wind noise frame includes:

[0136] Step S310: input the wind noise frame into the coding layer of the preset noise reduction model for downsampling to obtain various coded sound pickup features;

[0137] Step S320: inputting the encoded sound pickup feature into the recurrent neural network, so that the recurrent neural network outputs a corresponding user sound pickup feature based on the user voice vector, the preset random noise vector, and the encoded sound pickup feature;

[0138] Step S330, inputting the encoded sound pickup feature and the user sound pickup feature into the attention network to obtain an attention feature;

[0139] Step S340: input the user sound pickup features into the decoding layer for upsampling to obtain various decoded sound pickup features;

[0140] Step S350: splicing the attention feature to the corresponding decoded sound pickup feature to obtain a spliced ​​sound pickup feature, and inputting the spliced ​​sound pickup feature into a fully connected layer to obtain a denoised sound pickup signal corresponding to the wind noise frame.

[0141] In this embodiment, it should be noted that the preset denoising model includes an encoding layer, a recurrent neural network, an attention network, a decoding layer and a fully connected layer. The preset denoising model is a generator in a generative adversarial network, and the generator is composed of an encoding layer, a recurrent neural network, an attention network, a decoding layer and a fully connected layer.

[0142] Among them, the encoding layer can use one-dimensional convolution (1D-conv), the downsampling can use a 2*2 pooling layer, and the activation function of the encoding layer can use the LeakyRelu function, which is as follows:

[0143] The structure of the encoding layer and decoding layer can be described in the form of [batchsize, channel, datasize].

[0144] Coding layer structure: [Batch,1,4096], [Batch,16,2048], [Batch,32,1024], [Batch,64,512], [Batch,128,256], [Batch,64,128].

[0145] Decoding layer structure: [Batch,128*2,256], [Batch,64*2,512], [Batch,32*2,1024], [Batch,16*2,2048], [Batch,1,4096].

[0146] The recurrent neural network can be a recurrent neural network with a long short-term memory function, such as LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit).

[0147] The fully connected layer is a fully connected network, that is, all nodes in the fully connected network are connected to all dimensions of the input data. The activation function of the fully connected layer can be selected from the Tanh function and the RELU function, where:

[0148] The Tanh function is:

[0149] The RELU function is RELU(x)=max(0,x).

[0150] See Figure 3, which is a schematic diagram of the structure of the attention network involved in the embodiment of the present application. Figure 3 can be expressed by the following formula:

[0151] Among them, e1 is the decoding pickup feature (i.e., the grid square in Figure 3), and d1 is the encoding pickup feature (i.e., the white square in Figure 3). Denotes convolution, ⊙ denotes dot product, ReLU and sigmoid (denoted by s in the figure) are activation functions, and w1, w2, w3, and w4 are weight coefficients obtained through training. The encoded audio features from the encoding layer are multiplied by these weight coefficients and then concatenated with the decoded audio features. This helps balance the importance of features across multiple encoding layers.

[0152] The preset random noise vector can be generated using a random function. For example, a 128-dimensional noise vector can be generated as the preset random noise vector. It is understood that the preset random noise vector is optional, meaning that the recurrent neural network can output corresponding user voice pickup features based on the user voice vector and the encoded voice pickup features.

[0153] The user voice vector is a feature vector extracted from the user voice information. For example, the frequency domain signal mean and variance of the user voice information can be used as input features, and ResNet (residual neural network) can be used as a feature extraction model. The model structure of the feature extraction model is as follows:

[0154] Thus, a 128-dimensional feature vector of the user voice information can be extracted as the user voice vector.

[0155] Refer to Figure 4, which is a schematic diagram of the structure of the preset noise reduction model involved in the embodiment of the present application. In this embodiment, the wind noise frame is input into the encoding layer of the preset noise reduction model for downsampling to obtain various encoded sound pickup features (i.e., the white squares in Figure 4). The encoded sound pickup features are then input into the recurrent neural network so that the recurrent neural network outputs corresponding user sound pickup features based on the user voice vector, the preset random noise vector and the encoded sound pickup features. The encoded sound pickup features and the user sound pickup features are input into the attention network (i.e., the "G" square in Figure 4) to obtain attention features (i.e., the gray squares in Figure 4). The user sound pickup features are input into the decoding layer for upsampling to obtain various decoded sound pickup features (i.e., the grid squares in Figure 4). The attention features can then be spliced ​​to the corresponding decoded sound pickup features to obtain spliced ​​sound pickup features, and the spliced ​​sound pickup features are input into the fully connected layer to obtain the denoised sound pickup signal corresponding to the wind noise frame. In this embodiment, an attention mechanism (i.e., an attention network) is added to the generative adversarial network. This can increase the attention to a certain audio region (such as real speech) in the wind noise frame, while reducing the attention to other audio regions (such as wind noise), to achieve the goal of enhancing speech segments and suppressing noise. At the same time, recurrent neural networks such as LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) are added to maintain long and short-term memory. This allows accurate recognition of the user's voice in the wind noise frame based on the user's voice vector.

[0156] Before the step of inputting the wind noise frame into a preset noise reduction model to obtain the noise reduction pickup signal corresponding to the wind noise frame in step S300, the following steps are included:

[0157] Step A10: obtaining user speech vectors, noisy samples, and clean samples;

[0158] Step A20: training a discriminant network in a generative adversarial network based on the noisy samples and the clean samples to obtain a trained discriminator;

[0159] Step A30: inputting a preset random noise vector, the user speech vector, and the noisy sample into a generative network in the generative adversarial network to obtain a noise-reduced sample;

[0160] Step A40: inputting the denoised sample into the discriminator to obtain a loss function value of the generator network;

[0161] Step A50: Update the generation network based on the loss function value until the generation network converges to obtain a preset noise reduction model.

[0162] In this embodiment, it should be noted that the generative adversarial network includes a generative network and a discriminative network. The generative network is used to generate a denoised sample after denoising the noisy sample based on a preset random noise vector, the user voice vector and the noisy sample. The discriminative network is used to determine whether the input audio sample meets the noise reduction requirements.

[0163] In this embodiment, it is necessary to first train the discriminant network in the generative adversarial network to obtain a trained discriminator. When a pure sample (i.e., an audio sample without noise) is input into the trained discriminator, the trained discriminator outputs a probability value close to 1; when a noisy sample (i.e., an audio sample with noise) is input into the trained discriminator, the trained discriminator outputs a probability value close to 0. In this embodiment, the discriminant network in the generative adversarial network is trained based on the noisy samples and the pure samples by obtaining user voice vectors, noisy samples, and pure samples until the discriminant network converges (i.e., the number of iterations of the discriminant network reaches a preset number threshold or the loss function value is less than a preset function threshold) to obtain a trained discriminator.

[0164] Then, the preset random noise vector, the user speech vector, and the noisy sample can be input into the generative network of the generative adversarial network to obtain a denoised sample corresponding to the noisy sample. The denoised sample is then input into the discriminator to obtain the loss function value of the generative network. The generative network is then updated based on the loss function value until the generative network converges. The converged generative network can then be used as the preset denoising model.

[0165] In this embodiment, a user voice vector, a noisy sample, and a clean sample are obtained; based on the noisy sample and the clean sample, the discriminant network in the generative adversarial network is trained to obtain a trained discriminator; a preset random noise vector, the user voice vector, and the noisy sample are input into the generative network in the generative adversarial network to obtain a noise reduction sample; the noise reduction sample is input into the discriminator to obtain a loss function value of the generative network; based on the loss function value, the generative network is updated until the generative network converges to obtain a preset noise reduction model. In this embodiment, the generative adversarial network is trained using the user voice vector, the noisy sample, and the clean sample, so that the preset noise reduction model can, on the one hand, separate the human voice from the ambient noise in the wind noise frame based on the characteristics of the generative adversarial network, and on the other hand, can suppress the human voice in the wind noise frame that does not match the user voice vector based on the user voice vector. This effectively improves the noise reduction effect of the preset noise reduction model.

[0166] In a second embodiment of the present application, the wind noise frame is input into the encoding layer of a preset noise reduction model for downsampling to obtain encoded sound pickup features; the encoded sound pickup features are input into a recurrent neural network, so that the recurrent neural network outputs corresponding user sound pickup features based on the user voice vector, a preset random noise vector, and the encoded sound pickup features; the encoded sound pickup features and the user sound pickup features are input into an attention network to obtain attention features; the user sound pickup features are input into the decoding layer for upsampling to obtain decoded sound pickup features; the attention features are concatenated with the corresponding decoded sound pickup features to obtain concatenated sound pickup features, and the concatenated sound pickup features are input into a fully connected layer to obtain the denoised sound pickup signal corresponding to the wind noise frame. This embodiment adds an attention network to the generative adversarial network to achieve the goal of enhancing speech segments and suppressing noise. The addition of the recurrent neural network can also accurately identify the user voice in the wind noise frame based on the user voice vector and the preset random noise vector, thereby further improving the suppression of ambient noise.

[0167] 5 , FIG5 shows an embodiment of the present application further providing an audio processing device, the audio processing device being applied to an audio processing device, and the audio processing device comprising:

[0168] An acquisition module 10 is used to acquire an input sound pickup signal;

[0169] The detection module 20 is used to detect the input sound pickup signal and identify the wind noise frame in the input sound pickup signal;

[0170] The noise reduction module 30 is used to input the wind noise frame into a preset noise reduction model to obtain a noise-reduced sound pickup signal corresponding to the wind noise frame, wherein the preset noise reduction model is obtained by training a generative adversarial network using user speech vectors, noisy samples and clean samples.

[0171] Optionally, the preset denoising model includes an encoding layer, a recurrent neural network, an attention network, a decoding layer and a fully connected layer, and the denoising module 30 is further used to:

[0172] Input the wind noise frame into the encoding layer of the preset noise reduction model for downsampling to obtain each encoded sound pickup feature;

[0173] Inputting the encoded sound pickup feature into the recurrent neural network so that the recurrent neural network outputs a corresponding user sound pickup feature based on the user voice vector, the preset random noise vector and the encoded sound pickup feature;

[0174] Inputting the encoded voice pickup feature and the user voice pickup feature into the attention network to obtain an attention feature;

[0175] Inputting the user sound pickup features into the decoding layer for upsampling to obtain each decoded sound pickup feature;

[0176] The attention feature is spliced ​​to the corresponding decoded sound pickup feature to obtain a spliced ​​sound pickup feature, and the spliced ​​sound pickup feature is input into a fully connected layer to obtain a noise reduction sound pickup signal corresponding to the wind noise frame.

[0177] Optionally, the input sound pickup signal includes at least two paths of sound pickup signals, and the detection module 20 is further configured to:

[0178] Acquiring frequency spectrum characteristics of each of the picked-up sound signals, and determining frequency spectrum correlations of each of the picked-up sound signals based on the frequency spectrum characteristics;

[0179] performing low-pass filtering on the main-path sound pickup signal in each of the sound pickup signals to determine the low-frequency energy of the input sound pickup signal;

[0180] determining the wind speed of each signal frame in the input sound pickup signal according to the spectral correlation and the low-frequency energy;

[0181] According to the wind speed, a wind noise frame in each of the signal frames is identified.

[0182] Optionally, the audio processing device further includes a pre-processing module configured to:

[0183] Performing echo cancellation processing on the input sound pickup signal to obtain an echo cancellation signal;

[0184] The echo cancellation signal is used as a new input sound pickup signal, and the following steps are performed: the input sound pickup signal is detected to identify wind noise frames in the input sound pickup signal.

[0185] Optionally, the preprocessing module is further configured to:

[0186] Performing beamforming processing on the input sound pickup signal to obtain a directional enhancement signal;

[0187] The directional enhancement signal is used as a new input sound pickup signal, and the following steps are performed: the input sound pickup signal is detected to identify wind noise frames in the input sound pickup signal.

[0188] Optionally, the audio processing device further includes an output module configured to:

[0189] determining wind-noise-free frames other than the wind-noise frames in the input sound pickup signal, and calculating a posterior signal-to-noise ratio of the wind-noise-free frames;

[0190] Based on the a posteriori signal-to-noise ratio, performing noise elimination processing on the wind-noise-free frame to obtain an enhanced sound pickup signal;

[0191] Dynamic range control is performed on the noise reduction sound pickup signal and the enhanced sound pickup signal to obtain an output sound pickup signal.

[0192] Optionally, the audio processing device further includes a model training module, configured to:

[0193] Obtain user speech vectors, noisy samples, and clean samples;

[0194] Based on the noisy samples and the clean samples, a discriminant network in a generative adversarial network is trained to obtain a trained discriminator;

[0195] Inputting a preset random noise vector, the user voice vector, and the noisy sample into a generative network in the generative adversarial network to obtain a noise-reduced sample;

[0196] Inputting the denoised sample into the discriminator to obtain a loss function value of the generating network;

[0197] The generation network is updated based on the loss function value until the generation network converges to obtain a preset noise reduction model.

[0198] The audio processing device provided in this application utilizes the audio processing methods described in the aforementioned embodiments to address the technical issue of current sound pickup devices having poor suppression of ambient noise. Compared to the prior art, the beneficial effects of the audio processing device provided in the embodiments of this application are the same as those of the audio processing methods described in the aforementioned embodiments. Other technical features of the audio processing device are the same as those disclosed in the aforementioned embodiments and are not further elaborated here.

[0199] As shown in FIG6 , FIG6 is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application.

[0200] Specifically, the audio processing device can be an AR (Augmented Reality) device, a VR (Virtual Reality) device, a TWS (True Wireless Stereo) headset, a smartphone, a PC (Personal Computer), a tablet computer, a portable computer or a server.

[0201] As shown in Figure 6, the audio processing device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to implement connection and communication between these components. The user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard). Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a wireless fidelity (WI-FI) interface). The memory 1005 may be a high-speed random access memory (RAM) memory or a stable non-volatile memory (NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0202] Those skilled in the art will appreciate that the device structure shown in FIG6 does not limit the audio processing device, and may include more or fewer components than shown, or a combination of certain components, or a different arrangement of components.

[0203] As shown in FIG6 , the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an audio processing application.

[0204] In the device shown in Figure 6, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; and the processor 1001 can be used to call the audio processing program stored in the memory 1005 to implement the operations in the audio processing method provided in the above embodiment.

[0205] In addition, an embodiment of the present application further proposes a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, the operations in the audio processing method provided in the above embodiment are implemented. The specific steps are not described in detail here.

[0206] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity / operation / object from another entity / operation / object, and do not necessarily require or imply any actual relationship or order between these entities / operations / objects; the terms "include", "comprise", or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or system that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "includes a ..." does not exclude the presence of other identical elements in the process, method, article, or system that includes the element.

[0207] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For relevant details, please refer to the partial description of the method embodiment. The device embodiment described above is merely illustrative, and the units described as separate components may or may not be physically separated. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present application scheme. Those of ordinary skill in the art can understand and implement it without paying any creative work.

[0208] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0209] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the methods described in each embodiment of the present application.

[0210] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An audio processing method, characterized in that: The audio processing method comprises: Get the input pickup signal; Detecting the input sound pickup signal and identifying a wind noise frame in the input sound pickup signal; The wind noise frame is input into a preset noise reduction model to obtain a noise reduction pickup signal corresponding to the wind noise frame, wherein the preset noise reduction model is obtained by training a generative adversarial network using a user voice vector, noisy samples and clean samples.

2. The audio processing method according to claim 1, characterized in that: The preset noise reduction model includes an encoding layer, a recurrent neural network, an attention network, a decoding layer and a fully connected layer. The step of inputting the wind noise frame into the preset noise reduction model to obtain the noise reduction pickup signal corresponding to the wind noise frame includes: Input the wind noise frame into the encoding layer of the preset noise reduction model for downsampling to obtain each encoding sound pickup feature; Inputting the encoded sound pickup feature into the recurrent neural network so that the recurrent neural network outputs a corresponding user sound pickup feature based on the user voice vector, the preset random noise vector and the encoded sound pickup feature; Inputting the encoded sound pickup feature and the user sound pickup feature into the attention network to obtain an attention feature; Inputting the user sound pickup features into the decoding layer for upsampling to obtain each decoded sound pickup feature; The attention feature is spliced ​​to the corresponding decoded sound pickup feature to obtain a spliced ​​sound pickup feature, and the spliced ​​sound pickup feature is input into a fully connected layer to obtain a denoised sound pickup signal corresponding to the wind noise frame.

3. The audio processing method according to claim 1, characterized in that: The input sound pickup signal includes at least two paths of sound pickup signals, and the step of detecting the input sound pickup signal and identifying the wind noise frame in the input sound pickup signal includes: Acquiring frequency spectrum characteristics of each of the picked-up sound signals, and determining frequency spectrum correlation of each of the picked-up sound signals according to the frequency spectrum characteristics; Performing low-pass filtering on the main sound pickup signal in each of the sound pickup signals to determine the low-frequency energy of the input sound pickup signal; Determining the wind speed of each signal frame in the input sound pickup signal according to the frequency spectrum correlation and the low-frequency energy; According to the wind speed, a wind noise frame in each of the signal frames is identified.

4. The audio processing method according to claim 1, characterized in that: Before the step of detecting the input sound pickup signal and identifying the wind noise frame in the input sound pickup signal, the method includes: Performing echo cancellation processing on the input sound pickup signal to obtain an echo cancellation signal; The echo cancellation signal is used as a new input sound pickup signal, and the steps are performed: the input sound pickup signal is detected to identify the wind noise frame in the input sound pickup signal.

5. The audio processing method according to claim 4, characterized in that: Before the step of detecting the input sound pickup signal and identifying the wind noise frame in the input sound pickup signal, the method further includes: Performing beamforming processing on the input sound pickup signal to obtain a directional enhancement signal; The directional enhancement signal is used as a new input sound pickup signal, and the steps are performed: the input sound pickup signal is detected to identify the wind noise frame in the input sound pickup signal.

6. The audio processing method according to claim 5, characterized in that: After the step of inputting the wind noise frame into a preset noise reduction model to obtain a noise reduction pickup signal corresponding to the wind noise frame, the audio processing method further includes: Determine wind-noise-free frames other than the wind-noise frames in the input sound pickup signal, and calculate a posterior signal-to-noise ratio of the wind-noise-free frames; Based on the a posteriori signal-to-noise ratio, performing noise elimination processing on the wind-noise-free frame to obtain an enhanced sound pickup signal; Dynamic range control is performed on the noise reduction sound pickup signal and the enhanced sound pickup signal to obtain an output sound pickup signal.

7. The audio processing method according to any one of claims 1 to 6, characterized in that: Before the step of inputting the wind noise frame into a preset noise reduction model to obtain a noise reduction pickup signal corresponding to the wind noise frame, the method includes: Obtain user speech vectors, noisy samples, and clean samples; Based on the noisy samples and the clean samples, a discriminant network in a generative adversarial network is trained to obtain a trained discriminator; Inputting a preset random noise vector, the user voice vector and the noisy sample into a generative network in the generative adversarial network to obtain a noise reduction sample; Inputting the denoised sample into the discriminator to obtain a loss function value of the generating network; The generation network is updated based on the loss function value until the generation network converges to obtain a preset noise reduction model.

8. An audio processing device, characterized in that: The audio processing device comprises: An acquisition module, used for acquiring an input sound pickup signal; A detection module, used to detect the input sound pickup signal and identify the wind noise frame in the input sound pickup signal; The noise reduction module is used to input the wind noise frame into a preset noise reduction model to obtain a noise-reduced sound pickup signal corresponding to the wind noise frame, wherein the preset noise reduction model is obtained by training a generative adversarial network using a user voice vector, noisy samples and clean samples.

9. An audio processing device, characterized in that: The audio processing device comprises: a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the steps of the audio processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an audio processing program, and when the audio processing program is executed by a processor, the steps of the audio processing method according to any one of claims 1 to 7 are implemented.

Images