Audio processing method, device and equipment and computer readable storage medium

Abstract

The invention discloses an audio processing method, device and equipment and a computer readable storage medium, and relates to the technical field of audio processing, and the audio processing method comprises the steps: obtaining an input pickup signal; detecting the input pickup signal, and identifying a wind noise frame in the input pickup signal; and inputting the wind noise frame into a preset noise reduction model to obtain a noise reduction pickup signal corresponding to the wind noise frame, the preset noise reduction model being obtained by training a generative adversarial network by a user voice vector, a noisy sample and a pure sample. The technical problem that the suppression effect of current pickup equipment on environment noise is poor is solved.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio processing method, device, equipment and computer-readable storage medium. Background Art

[0002] Wind noise is the most common type of environmental noise in outdoor sound pickup, which seriously affects the sound pickup quality of outdoor sound pickup systems such as mobile phones and hearing aids. Wind noise is caused by the interaction between airflow and obstacles (such as buildings, human bodies, microphone cavities, etc.), so the characteristics of wind noise caused by different obstacles will also be different. In many cases, the sound pressure level of wind noise can reach 80dB SPL, which can completely cover the voice signal, greatly reduce the recognition of the voice and cause auditory discomfort.

[0003] Traditional methods of dealing with wind noise include windscreens, which are common in handheld microphones and professional gun-type microphones. There are many materials such as sponge, artificial fur, iron mesh, etc. The principle is mainly to reduce the air flow rate near the microphone diaphragm, break up the airflow and reduce the generation of turbulence. Windscreens can also be used on small pickup devices to suppress wind noise, but the size of the windscreen needs to be increased to increase the noise reduction. For small pickup devices such as mobile phones and TWS headphones, even if windscreens are set up, the effect of suppressing wind noise is very limited.

[0004] Therefore, how to effectively reduce the ambient noise of the sound pickup device is a technical problem that those skilled in the art need to solve urgently. Summary of the invention

[0005] The main purpose of the present application is to provide an audio processing method, device, equipment and computer-readable storage medium, aiming to solve the technical problem that the current sound pickup equipment has a poor suppression effect on environmental noise.

[0006] To achieve the above object, the present application provides an audio processing method, which is applied to an audio processing device. The audio processing method includes:

[0007] Get the input pickup signal;

[0008] Detecting the input sound pickup signal and identifying a wind noise frame in the input sound pickup signal;

[0009] The wind noise frame is input into a preset noise reduction model to obtain a noise reduction pickup signal corresponding to the wind noise frame, wherein the preset noise reduction model is obtained by training a generative adversarial network using a user voice vector, noisy samples and clean samples.

[0010] Optionally, the preset noise reduction model includes an encoding layer, a recurrent neural network, an attention network, a decoding layer and a fully connected layer, and the step of inputting the wind noise frame into the preset noise reduction model to obtain the noise reduction pickup signal corresponding to the wind noise frame includes:

[0011] Input the wind noise frame into the encoding layer of the preset noise reduction model for downsampling to obtain each encoding sound pickup feature;

[0012] Inputting the encoded sound pickup feature into the recurrent neural network so that the recurrent neural network outputs a corresponding user sound pickup feature based on the user voice vector, the preset random noise vector and the encoded sound pickup feature;

[0013] Inputting the encoded sound pickup feature and the user sound pickup feature into the attention network to obtain an attention feature;

[0014] Inputting the user sound pickup features into the decoding layer for upsampling to obtain each decoded sound pickup feature;

[0015] The attention feature is spliced ​​to the corresponding decoded sound pickup feature to obtain a spliced ​​sound pickup feature, and the spliced ​​sound pickup feature is input into a fully connected layer to obtain a denoised sound pickup signal corresponding to the wind noise frame.

[0016] Optionally, the input sound pickup signal includes at least two paths of sound pickup signals, and the step of detecting the input sound pickup signal and identifying a wind noise frame in the input sound pickup signal includes:

[0017] Acquiring frequency spectrum characteristics of each of the picked-up sound signals, and determining frequency spectrum correlation of each of the picked-up sound signals according to the frequency spectrum characteristics;

[0018] Performing low-pass filtering on the main sound pickup signal in each of the sound pickup signals to determine the low-frequency energy of the input sound pickup signal;

[0019] Determining the wind speed of each signal frame in the input sound pickup signal according to the frequency spectrum correlation and the low-frequency energy;

[0020] According to the wind speed, a wind noise frame in each of the signal frames is identified.

[0021] Optionally, before the step of detecting the input sound pickup signal and identifying a wind noise frame in the input sound pickup signal, the method further comprises:

[0022] Performing echo cancellation processing on the input sound pickup signal to obtain an echo cancellation signal;

[0023] The echo cancellation signal is used as a new input sound pickup signal, and the steps are performed: the input sound pickup signal is detected to identify the wind noise frame in the input sound pickup signal.

[0024] Optionally, before the step of detecting the input sound pickup signal and identifying the wind noise frame in the input sound pickup signal, the method further includes:

[0025] Performing beamforming processing on the input sound pickup signal to obtain a directional enhancement signal;

[0026] The directional enhancement signal is used as a new input sound pickup signal, and the steps are performed: the input sound pickup signal is detected to identify the wind noise frame in the input sound pickup signal.

[0027] Optionally, after the step of inputting the wind noise frame into a preset noise reduction model to obtain a noise reduction pickup signal corresponding to the wind noise frame, the audio processing method further includes:

[0028] Determine wind-noise-free frames other than the wind-noise frames in the input sound pickup signal, and calculate a posterior signal-to-noise ratio of the wind-noise-free frames;

[0029] Based on the a posteriori signal-to-noise ratio, performing noise elimination processing on the wind-noise-free frame to obtain an enhanced sound pickup signal;

[0030] Dynamic range control is performed on the noise reduction sound pickup signal and the enhanced sound pickup signal to obtain an output sound pickup signal.

[0031] Optionally, before the step of inputting the wind noise frame into a preset noise reduction model to obtain the noise reduction pickup signal corresponding to the wind noise frame, the step includes:

[0032] Obtain user speech vectors, noisy samples, and clean samples;

[0033] Based on the noisy samples and the clean samples, a discriminant network in a generative adversarial network is trained to obtain a trained discriminator;

[0034] Inputting a preset random noise vector, the user voice vector and the noisy sample into a generative network in the generative adversarial network to obtain a noise reduction sample;

[0035] Inputting the denoised sample into the discriminator to obtain a loss function value of the generating network;

[0036] The generation network is updated based on the loss function value until the generation network converges to obtain a preset noise reduction model.

[0037] The present application also provides an audio processing device, which is applied to an audio processing device, and the audio processing device includes:

[0038] An acquisition module, used for acquiring an input sound pickup signal;

[0039] A detection module, used to detect the input sound pickup signal and identify the wind noise frame in the input sound pickup signal;

[0040] The noise reduction module is used to input the wind noise frame into a preset noise reduction model to obtain a noise-reduced sound pickup signal corresponding to the wind noise frame, wherein the preset noise reduction model is obtained by training a generative adversarial network using a user voice vector, noisy samples and clean samples.

[0041] Optionally, the preset denoising model includes an encoding layer, a recurrent neural network, an attention network, a decoding layer and a fully connected layer, and a denoising module, and is further used for:

[0042] Input the wind noise frame into the encoding layer of the preset noise reduction model for downsampling to obtain each encoding sound pickup feature;

[0043] Inputting the encoded sound pickup feature into the recurrent neural network so that the recurrent neural network outputs a corresponding user sound pickup feature based on the user voice vector, the preset random noise vector and the encoded sound pickup feature;

[0044] Inputting the encoded sound pickup feature and the user sound pickup feature into the attention network to obtain an attention feature;

[0045] Inputting the user sound pickup features into the decoding layer for upsampling to obtain each decoded sound pickup feature;

[0046] The attention feature is spliced ​​to the corresponding decoded sound pickup feature to obtain a spliced ​​sound pickup feature, and the spliced ​​sound pickup feature is input into a fully connected layer to obtain a denoised sound pickup signal corresponding to the wind noise frame.

[0047] Optionally, the input sound pickup signal includes at least two paths of sound pickup signals, and the detection module is further used to:

[0048] Acquiring frequency spectrum characteristics of each of the picked-up sound signals, and determining frequency spectrum correlation of each of the picked-up sound signals according to the frequency spectrum characteristics;

[0049] Performing low-pass filtering on the main sound pickup signal in each of the sound pickup signals to determine the low-frequency energy of the input sound pickup signal;

[0050] Determining the wind speed of each signal frame in the input sound pickup signal according to the frequency spectrum correlation and the low-frequency energy;

[0051] According to the wind speed, a wind noise frame in each of the signal frames is identified.

[0052] Optionally, the audio processing device further includes: a pre-processing module, configured to:

[0053] Performing echo cancellation processing on the input sound pickup signal to obtain an echo cancellation signal;

[0054] The echo cancellation signal is used as a new input sound pickup signal, and the steps are performed: the input sound pickup signal is detected to identify the wind noise frame in the input sound pickup signal.

[0055] Optionally, the preprocessing module is further used to:

[0056] Performing beamforming processing on the input sound pickup signal to obtain a directional enhancement signal;

[0057] The directional enhancement signal is used as a new input sound pickup signal, and the steps are performed: the input sound pickup signal is detected to identify the wind noise frame in the input sound pickup signal.

[0058] Optionally, the audio processing device further includes: an output module, configured to:

[0059] Determine wind-noise-free frames other than the wind-noise frames in the input sound pickup signal, and calculate a posterior signal-to-noise ratio of the wind-noise-free frames;

[0060] Based on the a posteriori signal-to-noise ratio, performing noise elimination processing on the wind-noise-free frame to obtain an enhanced sound pickup signal;

[0061] Dynamic range control is performed on the noise reduction sound pickup signal and the enhanced sound pickup signal to obtain an output sound pickup signal.

[0062] Optionally, the audio processing device further includes: a model training module, configured to:

[0063] Obtain user speech vectors, noisy samples, and clean samples;

[0064] Based on the noisy samples and the clean samples, a discriminant network in a generative adversarial network is trained to obtain a trained discriminator;

[0065] Inputting a preset random noise vector, the user voice vector and the noisy sample into a generative network in the generative adversarial network to obtain a noise reduction sample;

[0066] Inputting the denoised sample into the discriminator to obtain a loss function value of the generating network;

[0067] The generation network is updated based on the loss function value until the generation network converges to obtain a preset noise reduction model.

[0068] The present application also provides an audio processing device, which includes: a memory, a processor, and a program of an audio processing method stored in the memory and executable on the processor. When the program of the audio processing method is executed by the processor, the steps of the audio processing method as described above can be implemented.

[0069] The present application also provides a computer-readable storage medium, on which is stored a program for implementing the audio processing method. When the program of the audio processing method is executed by a processor, the steps of the audio processing method as described above are implemented.

[0070] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned audio processing method when the computer program is executed by a processor.

[0071] The present application provides an audio processing method, device, equipment and computer-readable storage medium, which obtains an input sound pickup signal and detects the input sound pickup signal to identify the wind noise frame in the input sound pickup signal. The signal frame (i.e., wind noise frame) with wind noise in the input sound pickup signal is identified, thereby avoiding the signal frame without wind noise from being processed by wind noise suppression, that is, improving the efficiency of wind noise suppression processing, and avoiding the influence on the signal frame without wind noise. Then, the wind noise frame is input into a preset noise reduction model to obtain the noise reduction sound pickup signal corresponding to the wind noise frame, wherein the preset noise reduction model is obtained by training a generative adversarial network with a user voice vector, a noisy sample and a clean sample. Since the preset noise reduction model is obtained by training a generative adversarial network with a user voice vector, a noisy sample and a clean sample, the preset noise reduction model can more accurately identify the user voice in the input sound pickup signal based on the user voice vector, thereby suppressing wind noise and other environmental noises such as human voice in the sound pickup environment. Therefore, the present application solves the technical problem that the current sound pickup device has a poor suppression effect on environmental noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0073] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0074] Figure 1 This is a flowchart of the first embodiment of the audio processing method of the present application;

[0075] Figure 2 A flowchart of a second embodiment of the audio processing method of the present application;

[0076] Figure 3 This is a schematic diagram of the structure of the attention network involved in the embodiment of the present application;

[0077] Figure 4 This is a schematic diagram of the structure of a preset noise reduction model involved in the embodiment of the present application;

[0078] Figure 5 This is a schematic diagram of the structure of an audio processing device in an embodiment of the present application;

[0079] Figure 6 A schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present application.

[0080] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0081] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of this application.

[0082] Reference Figure 1 , Figure 1 This is a flowchart diagram of the first embodiment of the audio processing method of the present application.

[0083] The present application provides an audio processing method. In a first embodiment of the audio processing method of the present application, the audio processing method includes:

[0084] Step S100, obtaining an input sound pickup signal;

[0085] In this embodiment, it should be noted that the audio processing method is applied to an audio processing device, and the audio processing device can be an AR (Augmented Reality) device, a VR (Virtual Reality) device, a TWS (True Wireless Stereo) headset, a smart phone, a PC (Personal Computer), a tablet computer, a portable computer or a server and other devices.

[0086] As an example, this embodiment can collect sound through at least two microphones to obtain at least two picked-up sound signals as the input picked-up sound signals. Alternatively, a remote audio signal sent by a remote device (such as a mobile phone, a surveillance camera, etc.) is received and the remote audio signal is used as the input picked-up sound signal, wherein the remote audio signal includes at least two picked-up sound signals.

[0087] As another example, after obtaining the input sound pickup signal, the input sound pickup signal may be preprocessed and the preprocessed input sound pickup signal may be used as a new input sound pickup signal. The preprocessing may include pre-emphasis, framing, windowing and other processing operations, so as to improve the quality of subsequent audio processing.

[0088] Step S200, detecting the input sound pickup signal and identifying a wind noise frame in the input sound pickup signal;

[0089] In this embodiment, it should be noted that it is understandable that the input sound pickup signal may exist in the form of frames, that is, the input sound pickup signal includes signal frames. In addition, the input sound pickup signal includes at least two sound pickup signals.

[0090] As an example, the spectrum characteristics of each of the picked-up sound signals can be obtained, and the spectrum correlation of each of the picked-up sound signals can be determined based on the spectrum characteristics. Then, whether there is wind noise in each of the signal frames in the input picked-up sound signal can be determined based on the spectrum correlation, so as to identify the wind noise frame in the input picked-up sound signal. Exemplarily, when the spectrum correlation of each of the picked-up sound signals in the signal frame exceeds a first correlation threshold, it is determined that there is no wind noise in the signal frame; when the spectrum correlation of each of the picked-up sound signals in the signal frame is lower than a second correlation threshold, it is determined that there is no wind noise in the signal frame, and the signal frame is a wind noise frame. The second correlation threshold is less than the first correlation threshold.

[0091] As another example, one of the picked-up signals may be selected as the main picked-up signal, the low-frequency energy of each signal frame in the main picked-up signal may be calculated, and the spectrum characteristics of each picked-up signal may be obtained, and the spectrum correlation of each picked-up signal may be determined based on the spectrum characteristics, so as to detect the input picked-up signal. Then, the wind speed of each signal frame in the input picked-up signal may be determined based on the spectrum correlation and the low-frequency energy, and the wind noise frame in each signal frame may be identified based on the wind speed.

[0092] The input sound pickup signal in step S200 includes at least two sound pickup signals, and the step of detecting the input sound pickup signal and identifying the wind noise frame in the input sound pickup signal includes:

[0093] Step S210, acquiring the frequency spectrum characteristics of each of the picked-up sound signals, and determining the frequency spectrum correlation of each of the picked-up sound signals according to the frequency spectrum characteristics;

[0094] Step S220, performing low-pass filtering on the main sound pickup signal in each of the sound pickup signals to determine the low-frequency energy of the input sound pickup signal;

[0095] Step S230, determining the wind speed of each signal frame in the input sound pickup signal according to the frequency spectrum correlation and the low-frequency energy;

[0096] Step S240: identifying wind noise frames in each of the signal frames according to the wind speed.

[0097] In this embodiment, it should be noted that the frequency spectrum characteristics may include characteristics such as the frequency and bandwidth of each of the picked-up sound signals.

[0098] As an example, taking the case where the input sound pickup signal includes two sound pickup signals, the following preset correlation function formula can be used to calculate the spectral correlation of each of the sound pickup signals. The preset correlation function formula is as follows:

[0099] Spectral Correlation

[0100] Among them, O 12 , 0 22 is the element in the matrix corresponding to the following spectral characteristics:

[0101]

[0102] in, Indicates the number of points where the first channel's sound pickup signal is positive at time 0 to k. Indicates the number of points where the second channel's sound pickup signal is positive at time 0 to k. Indicates the number of points where the first channel's pickup signal is negative at time 0 to k. Indicates the number of points where the second channel's sound pickup signal is negative at time 0-k.

[0103] As an example, taking the input sound pickup signal including two sound pickup signals as an example, each of the sound pickup signals can be Fourier transformed, and then substituted into a preset coherence coefficient calculation formula to calculate the spectral correlation of each of the sound pickup signals. The preset coherence coefficient calculation formula is as follows:

[0104] Spectral Correlation

[0105] Wherein, Y1(K) is the Fourier transform of the first-channel sound pickup signal in the bandwidth 1-32, and Y2(K) is the Fourier transform of the second-channel sound pickup signal in the bandwidth 1-32.

[0106] In this way, the frequency spectrum correlation of each of the picked-up sound signals can be calculated.

[0107] Furthermore, in this embodiment, the low-frequency energy of the input sound pickup signal can be determined by low-pass filtering the main sound pickup signal in each of the sound pickup signals. It can be understood that the main sound pickup signal is the sound pickup signal collected by the main microphone (such as a feedforward microphone) in each of the sound pickup signals. Of course, any sound pickup signal from each of the sound pickup signals can be selected as the main sound pickup signal. Then, the main sound pickup signal can be passed through a low-pass filter, and the main sound pickup signal can be low-pass filtered based on a preset upper cutoff frequency (such as 1300Hz, 1500Hz, 1800Hz), wherein the low-pass filter can generally be implemented using an IIR (Infinite Impulse Response) filter, and of course other digital filters can also be used. Then, based on the main sound pickup signal after low-pass filtering, the low-frequency energy of each signal frame in the input sound pickup signal is calculated. Exemplarily, the calculation formula for the low-frequency energy is as follows:

[0108]

[0109] Among them, P low is low frequency energy, x1 LP is the main sound pickup signal after low-pass filtering, and k is time.

[0110] In this embodiment, a preset correspondence between the spectrum correlation and the low-frequency energy and the wind speed can be pre-set, and the preset correspondence can be expressed in the form of a mapping table, a function, etc. Therefore, this embodiment can query the preset correspondence according to the spectrum correlation and the low-frequency energy to obtain the wind speed of each signal frame in the input sound pickup signal. Then, the signal frame whose wind speed exceeds the preset wind speed threshold (such as 0.1m / s, 0.2m / s, 0.3m / s, etc.) is used as a wind noise frame to identify the wind noise frame in each of the signal frames.

[0111] In this embodiment, the wind speed of each signal frame in the input sound pickup signal is determined by combining the spectral correlation of each sound pickup signal in each signal frame and the low-frequency energy of each signal frame, and then the wind noise frame is identified. In this way, the wind noise frame with wind noise can be more accurately screened out from the input sound pickup signal, so that the wind noise frame can be specifically denoised, effectively improving the subsequent noise reduction effect.

[0112] Before the step of detecting the input sound pickup signal and identifying the wind noise frame in the input sound pickup signal in step S100, the following steps are included:

[0113] Step S110, performing echo cancellation processing on the input sound pickup signal to obtain an echo cancellation signal;

[0114] Step S111: taking the echo cancellation signal as a new input sound pickup signal, executing the steps of: detecting the input sound pickup signal, and identifying a wind noise frame in the input sound pickup signal.

[0115] Since the input sound pickup signal may contain not only environmental noise but also echoes, before detecting the input sound pickup signal, the present embodiment performs echo cancellation processing on the input sound pickup signal to obtain an echo cancellation signal, and then uses the echo cancellation signal as a new input sound pickup signal to perform the following steps: detecting the input sound pickup signal and identifying the wind noise frame in the input sound pickup signal. Thus, the present embodiment effectively eliminates the echo in the input sound pickup signal, that is, improves the accuracy of identifying the wind noise frame in the input sound pickup signal, and is also conducive to improving the subsequent noise reduction effect.

[0116] Before the step of detecting the input sound pickup signal and identifying the wind noise frame in the input sound pickup signal in step S100, the method further includes:

[0117] Step S120, performing beamforming processing on the input sound pickup signal to obtain a directional enhancement signal;

[0118] Step S121, taking the directional enhancement signal as a new input sound pickup signal, and executing the steps of: detecting the input sound pickup signal, and identifying a wind noise frame in the input sound pickup signal.

[0119] In this embodiment, it should be noted that the beamforming (or spatial domain filtering) process is to perform weighted summation on the input sound pickup signal to suppress noise other than directivity.

[0120] In order to enhance the signal in a specific direction and eliminate or suppress the noise in other directions. This embodiment performs beamforming processing on the input sound pickup signal to obtain a directional enhancement signal, and then the directional enhancement signal can be used as a new input sound pickup signal, and the steps are performed: the input sound pickup signal is detected to identify the wind noise frame in the input sound pickup signal. Therefore, this embodiment effectively eliminates or suppresses the noise in the input sound pickup signal except the directivity, that is, improves the accuracy of identifying the wind noise frame in the input sound pickup signal, and is also conducive to improving the subsequent noise reduction effect.

[0121] Step S300: input the wind noise frame into a preset noise reduction model to obtain a noise reduction pickup signal corresponding to the wind noise frame, wherein the preset noise reduction model is obtained by training a generative adversarial network using a user voice vector, noisy samples, and clean samples.

[0122] In this embodiment, it should be noted that the preset noise reduction model is obtained by training a generative adversarial network using a user voice vector, noisy samples and clean samples. The user voice vector is a feature vector obtained after feature extraction of the user voice information, which can be used to identify the user's voice in the wind noise frame. The noisy sample is a voice signal sample with environmental noise, and the clean sample is a voice signal sample without environmental noise.

[0123] In this embodiment, the preset noise reduction model is obtained by training a generative adversarial network using user voice vectors, noisy samples, and clean samples. Therefore, the preset noise reduction model can more accurately identify the user voice in the input sound pickup signal based on the user voice vector, and thus suppress environmental noise such as wind noise and other human voices in the sound pickup environment. In this embodiment, by inputting the wind noise frame into the preset noise reduction model, the preset noise reduction model can separate the human voice from the environmental noise in the wind noise frame based on the characteristics of the generative adversarial network on the one hand, and can suppress the human voice in the wind noise frame that does not match the user voice vector on the other hand based on the user voice vector. Therefore, based on the preset noise reduction model, the environmental noise in the wind noise frame and the human voice that does not match the user voice vector can be effectively suppressed, thereby effectively improving the noise reduction effect of obtaining the noise reduction pickup signal corresponding to the wind noise frame.

[0124] Wherein, after the step of inputting the wind noise frame into a preset noise reduction model to obtain a noise reduction pickup signal corresponding to the wind noise frame in step S300, the audio processing method further includes:

[0125] Step S400, determining a wind-noise-free frame other than the wind-noise frame in the input sound pickup signal, and calculating a posterior signal-to-noise ratio of the wind-noise-free frame;

[0126] Step S410, performing noise elimination processing on the wind-noise-free frame based on the posterior signal-to-noise ratio to obtain an enhanced sound pickup signal;

[0127] Step S420: performing dynamic range control on the noise reduction sound pickup signal and the enhanced sound pickup signal to obtain an output sound pickup signal.

[0128] In this embodiment, it should be noted that the wind-noise-free frame is a signal frame in the input sound pickup signal except the wind noise frame.

[0129] Since the input sound pickup signal includes a wind-free frame without wind noise in addition to the wind noise frame, it is also necessary to perform noise reduction processing on the wind-free frame. The posterior signal-to-noise ratio of the wind-free frame can be calculated according to a preset signal-to-noise ratio calculation formula. The preset signal-to-noise ratio calculation formula is as follows:

[0130]

[0131] Among them, γ t (k) is the posterior signal-to-noise ratio of the wind-free frame, P n (k, t) is the noise power spectrum function of the wind-free frame, Y t (k) is the wind-noise-free frame.

[0132] Then, based on the a posteriori signal-to-noise ratio and a preset enhancement function, the wind-noise-free frame can be subjected to noise elimination processing to obtain an enhanced sound pickup signal. The preset enhancement function is as follows:

[0133]

[0134] in, is the enhanced sound pickup signal, Y t (k) is the wind-noise-free frame, H t (k) is the gain function, the over-reduction factor λ is a constant, usually 0.9, γ t (k) is the posterior signal-to-noise ratio of the wind-noise-free frame.

[0135] Furthermore, dynamic range control (DRC) can be performed on both the noise reduction sound pickup signal and the enhanced sound pickup signal to obtain an output sound pickup signal.

[0136] In the first embodiment of the present application, by acquiring an input sound pickup signal and detecting the input sound pickup signal, a wind noise frame in the input sound pickup signal is identified. The signal frame (i.e., wind noise frame) with wind noise in the input sound pickup signal is identified, thereby avoiding the signal frame without wind noise from being subjected to wind noise suppression, that is, improving the efficiency of wind noise suppression processing, and avoiding the influence on the signal frame without wind noise. Then, the wind noise frame is input into a preset noise reduction model to obtain a noise reduction sound pickup signal corresponding to the wind noise frame, wherein the preset noise reduction model is obtained by training a generative adversarial network with a user voice vector, a noisy sample, and a clean sample. Since the preset noise reduction model is obtained by training a generative adversarial network with a user voice vector, a noisy sample, and a clean sample, the preset noise reduction model can more accurately identify the user voice in the input sound pickup signal based on the user voice vector, thereby suppressing environmental noise such as wind noise and other human voices in the sound pickup environment. Therefore, the technical problem that the current sound pickup device has a poor suppression effect on environmental noise is solved in this embodiment.

[0137] Further, refer to Figure 2 , refer to Figure 2 , Figure 2 This is a flowchart diagram of the second embodiment of the audio processing method of the present application.

[0138] Based on the first embodiment of the present application, in another embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be described in detail later. The preset noise reduction model includes an encoding layer, a recurrent neural network, an attention network, a decoding layer, and a fully connected layer. The step of inputting the wind noise frame into the preset noise reduction model in step S300 to obtain the noise reduction pickup signal corresponding to the wind noise frame includes:

[0139] Step S310, inputting the wind noise frame into the coding layer in the preset noise reduction model for downsampling to obtain various coded sound pickup features;

[0140] Step S320, inputting the encoded sound pickup feature into the recurrent neural network, so that the recurrent neural network outputs a corresponding user sound pickup feature based on the user voice vector, the preset random noise vector and the encoded sound pickup feature;

[0141] Step S330, inputting the encoded sound pickup feature and the user sound pickup feature into the attention network to obtain an attention feature;

[0142] Step S340, inputting the user sound pickup features into the decoding layer for up-sampling to obtain various decoded sound pickup features;

[0143] Step S350, splicing the attention feature to the corresponding decoded sound pickup feature to obtain a spliced ​​sound pickup feature, and inputting the spliced ​​sound pickup feature into a fully connected layer to obtain a denoised sound pickup signal corresponding to the wind noise frame.

[0144] In this embodiment, it should be noted that the preset denoising model includes an encoding layer, a recurrent neural network, an attention network, a decoding layer and a fully connected layer. The preset denoising model is a generator in a generative adversarial network, and the generator is composed of an encoding layer, a recurrent neural network, an attention network, a decoding layer and a fully connected layer.

[0145] Among them, the encoding layer can use one-dimensional convolution (1D-conv), the downsampling can use a 2*2 pooling layer, and the activation function of the encoding layer can use the LeakyRelu function, which is as follows:

[0146] LeakyRelu:

[0147] The structure of the encoding layer and the decoding layer can be described in the form of [batchsize, channel, datasize].

[0148] Coding layer structure: [Batch,1,4096], [Batch,16,2048], [Batch,32,1024], [Batch,64,512], [Batch,128,256], [Batch,64,128].

[0149] Decoding layer structure: [Batch,128*2,256], [Batch,64*2,512], [Batch,32*2,1024], [Batch,16*2,2048], [Batch,1,4096].

[0150] The recurrent neural network can be a recurrent neural network with a long short-term memory function, such as LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit).

[0151] The fully connected layer is a fully connected network, that is, all nodes in the fully connected network are connected to all input data of all dimensions. The activation function of the fully connected layer can be selected from the Tanh function and the RELU function, where:

[0152] The Tanh function is:

[0153] The RELU function is RELU(x)=max(0,x).

[0154] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of the attention network involved in the embodiment of the present application. Figure 3 It can be expressed by the following formula:

[0155]

[0156] Among them, e1 is the decoding pickup feature (i.e. Figure 3 ), d1 is the encoded sound pickup feature (i.e. Figure 3 The white square in the above formula Denotes convolution, ⊙ denotes dot product, Relu and sigmoid (denoted by s in the figure) are both activation functions, w1, w2, w3, w4 are all weight coefficients obtained through training. The encoded sound pickup features from the encoding layer are multiplied by the weight coefficients and then connected to the decoded sound pickup features, which helps to weigh the importance of features in multiple encoding layers.

[0157] The preset random noise vector can be generated using a random function. For example, a 128-dimensional noise vector can be generated as the preset random noise vector. It can be understood that the preset random noise vector is an optional vector, that is, the recurrent neural network can output the corresponding user pickup feature based on the user voice vector and the encoded pickup feature.

[0158] The user voice vector is a feature vector extracted from the user voice information. Exemplarily, the frequency domain signal mean and variance of the user voice information can be used as input features, and ResNet (residual neural network) can be used as a feature extraction model. The model structure of the feature extraction model is as follows:

[0159]

[0160]

[0161] Thus, a 128-dimensional feature vector of the user voice information can be extracted as the user voice vector.

[0162] See also Figure 4 , Figure 4 Schematic diagram of the structure of the preset noise reduction model involved in the embodiment of the present application. In this embodiment, the wind noise frame is input into the encoding layer of the preset noise reduction model for downsampling to obtain the encoding sound pickup features (i.e. Figure 4 The encoded sound pickup feature is then input into the recurrent neural network, so that the recurrent neural network outputs the corresponding user sound pickup feature based on the user voice vector, the preset random noise vector and the encoded sound pickup feature. The encoded sound pickup feature and the user sound pickup feature are input into the attention network (i.e. Figure 4 The "G" block in the figure) obtains the attention feature (i.e. Figure 4 The user sound pickup features are input into the decoding layer for up-sampling to obtain each decoded sound pickup feature (i.e. Figure 4). The attention feature can then be spliced ​​to the corresponding decoded sound pickup feature to obtain a spliced ​​sound pickup feature, and the spliced ​​sound pickup feature is input into the fully connected layer to obtain the noise reduction sound pickup signal corresponding to the wind noise frame. In this embodiment, an attention mechanism (i.e., attention network) is added to the generative adversarial network, which can increase the attention of a certain audio area (such as real speech) of the wind noise frame, while reducing the attention of certain audio areas (such as wind noise) to achieve the goal of enhancing the speech segment and suppressing noise. At the same time, recurrent neural networks such as LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) are added to have the function of maintaining long and short-term memory. Therefore, based on the user voice vector, the user voice in the wind noise frame can be accurately identified.

[0163] Wherein, before the step of inputting the wind noise frame into a preset noise reduction model to obtain the noise reduction pickup signal corresponding to the wind noise frame in step S300, the following steps are included:

[0164] Step A10, obtaining a user speech vector, a noisy sample, and a clean sample;

[0165] Step A20, training a discriminant network in a generative adversarial network based on the noisy samples and the clean samples to obtain a trained discriminator;

[0166] Step A30, inputting a preset random noise vector, the user voice vector and the noisy sample into a generative network in the generative adversarial network to obtain a noise reduction sample;

[0167] Step A40, inputting the denoised sample into the discriminator to obtain the loss function value of the generating network;

[0168] Step A50, updating the generation network based on the loss function value until the generation network converges to obtain a preset noise reduction model.

[0169] In this embodiment, it should be noted that the generative adversarial network includes a generative network and a discriminative network. The generative network is used to generate a denoised sample after denoising the noisy sample based on a preset random noise vector, the user voice vector and the noisy sample, and the discriminative network is used to determine whether the input audio sample meets the noise reduction requirements.

[0170] In this embodiment, it is necessary to first train the discriminant network in the generative adversarial network to obtain a trained discriminator. When a pure sample (i.e., an audio sample without noise) is input into the trained discriminator, the trained discriminator outputs a probability value close to 1; when a noisy sample (i.e., an audio sample with noise) is input into the trained discriminator, the trained discriminator outputs a probability value close to 0. In this embodiment, the discriminant network in the generative adversarial network is trained based on the noisy samples and the pure samples by obtaining user voice vectors, noisy samples, and pure samples until the discriminant network converges (i.e., the number of iterations of the discriminant network reaches a preset number threshold or the loss function value is less than a preset function threshold) to obtain a trained discriminator.

[0171] Then, the preset random noise vector, the user voice vector and the noisy sample can be input into the generative network in the generative adversarial network to obtain the denoised sample corresponding to the noisy sample. Then, the denoised sample is input into the discriminator to obtain the loss function value of the generative network, and then the generative network is updated based on the loss function value until the generative network converges, and the converged generative network can be used as the preset denoising model.

[0172] In this embodiment, by obtaining a user voice vector, a noisy sample and a clean sample; based on the noisy sample and the clean sample, the discriminant network in the generative adversarial network is trained to obtain a trained discriminator; the preset random noise vector, the user voice vector and the noisy sample are input into the generative network in the generative adversarial network to obtain a denoised sample; the denoised sample is input into the discriminator to obtain a loss function value of the generative network; based on the loss function value, the generative network is updated until the generative network converges to obtain a preset denoising model. In this embodiment, the generative adversarial network is trained by the user voice vector, the noisy sample and the clean sample, so that the preset denoising model can, on the one hand, separate the human voice in the wind noise frame from the ambient noise based on the characteristics of the generative adversarial network, and on the other hand, can suppress the human voice in the wind noise frame that does not match the user voice vector based on the user voice vector. The denoising effect of the preset denoising model is effectively improved.

[0173] In the second embodiment of the present application, the wind noise frame is input into the encoding layer in the preset noise reduction model for downsampling to obtain each encoded sound pickup feature; the encoded sound pickup feature is input into the recurrent neural network so that the recurrent neural network outputs the corresponding user sound pickup feature based on the user voice vector, the preset random noise vector and the encoded sound pickup feature; the encoded sound pickup feature and the user sound pickup feature are input into the attention network to obtain the attention feature; the user sound pickup feature is input into the decoding layer for upsampling to obtain each decoded sound pickup feature; the attention feature is spliced ​​to the corresponding decoded sound pickup feature to obtain the spliced ​​sound pickup feature, and the spliced ​​sound pickup feature is input into the fully connected layer to obtain the noise reduction sound pickup signal corresponding to the wind noise frame. This embodiment adds an attention network to the generative adversarial network to achieve the goal of enhancing the speech segment and suppressing noise. At the same time, the addition of a recurrent neural network can accurately identify the user voice in the wind noise frame based on the user voice vector and the preset random noise vector. Thereby further improving the suppression effect of environmental noise.

[0174] Reference Figure 5 , Figure 5 An audio processing device is also provided for an embodiment of the present application. The audio processing device is applied to an audio processing device. The audio processing device includes:

[0175] An acquisition module 10 is used to acquire an input sound pickup signal;

[0176] A detection module 20, configured to detect the input sound pickup signal and identify a wind noise frame in the input sound pickup signal;

[0177] The noise reduction module 30 is used to input the wind noise frame into a preset noise reduction model to obtain a noise-reduced sound pickup signal corresponding to the wind noise frame, wherein the preset noise reduction model is obtained by training a generative adversarial network using a user voice vector, noisy samples and clean samples.

[0178] Optionally, the preset denoising model includes an encoding layer, a recurrent neural network, an attention network, a decoding layer and a fully connected layer, and the denoising module 30 is further used for:

[0179] Input the wind noise frame into the encoding layer of the preset noise reduction model for downsampling to obtain each encoding sound pickup feature;

[0180] Inputting the encoded sound pickup feature into the recurrent neural network so that the recurrent neural network outputs a corresponding user sound pickup feature based on the user voice vector, the preset random noise vector and the encoded sound pickup feature;

[0181] Inputting the encoded sound pickup feature and the user sound pickup feature into the attention network to obtain an attention feature;

[0182] Inputting the user sound pickup features into the decoding layer for upsampling to obtain each decoded sound pickup feature;

[0183] The attention feature is spliced ​​to the corresponding decoded sound pickup feature to obtain a spliced ​​sound pickup feature, and the spliced ​​sound pickup feature is input into a fully connected layer to obtain a denoised sound pickup signal corresponding to the wind noise frame.

[0184] Optionally, the input sound pickup signal includes at least two paths of sound pickup signals, and the detection module 20 is further used for:

[0185] Acquiring frequency spectrum characteristics of each of the picked-up sound signals, and determining frequency spectrum correlation of each of the picked-up sound signals according to the frequency spectrum characteristics;

[0186] Performing low-pass filtering on the main sound pickup signal in each of the sound pickup signals to determine the low-frequency energy of the input sound pickup signal;

[0187] Determining the wind speed of each signal frame in the input sound pickup signal according to the frequency spectrum correlation and the low-frequency energy;

[0188] According to the wind speed, a wind noise frame in each of the signal frames is identified.

[0189] Optionally, the audio processing device further includes: a pre-processing module, configured to:

[0190] Performing echo cancellation processing on the input sound pickup signal to obtain an echo cancellation signal;

[0191] The echo cancellation signal is used as a new input sound pickup signal, and the steps are performed: the input sound pickup signal is detected to identify the wind noise frame in the input sound pickup signal.

[0192] Optionally, the preprocessing module is further used to:

[0193] Performing beamforming processing on the input sound pickup signal to obtain a directional enhancement signal;

[0194] The directional enhancement signal is used as a new input sound pickup signal, and the steps are performed: the input sound pickup signal is detected to identify the wind noise frame in the input sound pickup signal.

[0195] Optionally, the audio processing device further includes: an output module, configured to:

[0196] Determine wind-noise-free frames other than the wind-noise frames in the input sound pickup signal, and calculate a posterior signal-to-noise ratio of the wind-noise-free frames;

[0197] Based on the a posteriori signal-to-noise ratio, performing noise elimination processing on the wind-noise-free frame to obtain an enhanced sound pickup signal;

[0198] Dynamic range control is performed on the noise reduction sound pickup signal and the enhanced sound pickup signal to obtain an output sound pickup signal.

[0199] Optionally, the audio processing device further includes: a model training module, configured to:

[0200] Obtain user speech vectors, noisy samples, and clean samples;

[0201] Based on the noisy samples and the clean samples, a discriminant network in a generative adversarial network is trained to obtain a trained discriminator;

[0202] Inputting a preset random noise vector, the user voice vector and the noisy sample into a generative network in the generative adversarial network to obtain a noise reduction sample;

[0203] Inputting the denoised sample into the discriminator to obtain a loss function value of the generating network;

[0204] The generation network is updated based on the loss function value until the generation network converges to obtain a preset noise reduction model.

[0205] The audio processing device provided by the present application adopts the audio processing method in the above-mentioned embodiments to solve the technical problem that the current sound pickup device has a poor suppression effect on environmental noise. Compared with the prior art, the beneficial effects of the audio processing device provided by the embodiment of the present application are the same as the beneficial effects of the audio processing method provided by the above-mentioned embodiment, and other technical features in the audio processing device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0206] like Figure 6 As shown, Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application.

[0207] Specifically, the audio processing device can be an AR (Augmented Reality) device, a VR (Virtual Reality) device, a TWS (True Wireless Stereo) headset, a smart phone, a PC (Personal Computer), a tablet computer, a portable computer or a server and other devices.

[0208] like Figure 6As shown, the audio processing device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0209] Those skilled in the art will understand that Figure 6 The device structure shown in the figure does not constitute a limitation on the audio processing device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0210] like Figure 6 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an audio processing application.

[0211] exist Figure 6 In the device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; and the processor 1001 can be used to call the audio processing program stored in the memory 1005 to implement the operations in the audio processing method provided in the above embodiment.

[0212] In addition, an embodiment of the present application further proposes a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, the operations in the audio processing method provided in the above embodiment are implemented, and the specific steps are not repeated here.

[0213] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity / operation / object from another entity / operation / object, and do not necessarily require or imply any such actual relationship or order between these entities / operations / objects; the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0214] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is only schematic, and the units described as separate components may or may not be physically separated. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present application scheme. Those of ordinary skill in the art can understand and implement it without paying creative work.

[0215] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0216] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, including a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the methods described in each embodiment of the present application.

[0217] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An audio processing method, characterized in that: The audio processing method comprises: Get the input pickup signal; Detecting the input sound pickup signal and identifying a wind noise frame in the input sound pickup signal; The wind noise frame is input into a preset noise reduction model to obtain a noise reduction pickup signal corresponding to the wind noise frame, wherein the preset noise reduction model is obtained by training a generative adversarial network using a user voice vector, noisy samples and clean samples.

2. The audio processing method according to claim 1, characterized in that: The preset noise reduction model includes an encoding layer, a recurrent neural network, an attention network, a decoding layer and a fully connected layer. The step of inputting the wind noise frame into the preset noise reduction model to obtain the noise reduction pickup signal corresponding to the wind noise frame includes: Input the wind noise frame into the encoding layer of the preset noise reduction model for downsampling to obtain each encoding sound pickup feature; Inputting the encoded sound pickup feature into the recurrent neural network so that the recurrent neural network outputs a corresponding user sound pickup feature based on the user voice vector, the preset random noise vector and the encoded sound pickup feature; Inputting the encoded sound pickup feature and the user sound pickup feature into the attention network to obtain an attention feature; Inputting the user sound pickup features into the decoding layer for upsampling to obtain each decoded sound pickup feature; The attention feature is spliced ​​to the corresponding decoded sound pickup feature to obtain a spliced ​​sound pickup feature, and the spliced ​​sound pickup feature is input into a fully connected layer to obtain a denoised sound pickup signal corresponding to the wind noise frame.

3. The audio processing method according to claim 1, characterized in that: The input sound pickup signal includes at least two paths of sound pickup signals, and the step of detecting the input sound pickup signal and identifying the wind noise frame in the input sound pickup signal includes: Acquiring frequency spectrum characteristics of each of the picked-up sound signals, and determining frequency spectrum correlation of each of the picked-up sound signals according to the frequency spectrum characteristics; Performing low-pass filtering on the main sound pickup signal in each of the sound pickup signals to determine the low-frequency energy of the input sound pickup signal; Determining the wind speed of each signal frame in the input sound pickup signal according to the frequency spectrum correlation and the low-frequency energy; According to the wind speed, a wind noise frame in each of the signal frames is identified.

4. The audio processing method according to claim 1, characterized in that: Before the step of detecting the input sound pickup signal and identifying the wind noise frame in the input sound pickup signal, the method includes: Performing echo cancellation processing on the input sound pickup signal to obtain an echo cancellation signal; The echo cancellation signal is used as a new input sound pickup signal, and the steps are performed: the input sound pickup signal is detected to identify the wind noise frame in the input sound pickup signal.

5. The audio processing method according to claim 4, characterized in that: Before the step of detecting the input sound pickup signal and identifying the wind noise frame in the input sound pickup signal, the method further includes: Performing beamforming processing on the input sound pickup signal to obtain a directional enhancement signal; The directional enhancement signal is used as a new input sound pickup signal, and the steps are performed: the input sound pickup signal is detected to identify the wind noise frame in the input sound pickup signal.

6. The audio processing method according to claim 5, characterized in that: After the step of inputting the wind noise frame into a preset noise reduction model to obtain a noise reduction pickup signal corresponding to the wind noise frame, the audio processing method further includes: Determine wind-noise-free frames other than the wind-noise frames in the input sound pickup signal, and calculate a posterior signal-to-noise ratio of the wind-noise-free frames; Based on the a posteriori signal-to-noise ratio, performing noise elimination processing on the wind-noise-free frame to obtain an enhanced sound pickup signal; Dynamic range control is performed on the noise reduction sound pickup signal and the enhanced sound pickup signal to obtain an output sound pickup signal.

7. The audio processing method according to any one of claims 1 to 6, characterized in that: Before the step of inputting the wind noise frame into a preset noise reduction model to obtain a noise reduction pickup signal corresponding to the wind noise frame, the method includes: Obtain user speech vectors, noisy samples, and clean samples; Based on the noisy samples and the clean samples, a discriminant network in a generative adversarial network is trained to obtain a trained discriminator; Inputting a preset random noise vector, the user voice vector and the noisy sample into a generative network in the generative adversarial network to obtain a noise reduction sample; Inputting the denoised sample into the discriminator to obtain a loss function value of the generating network; The generation network is updated based on the loss function value until the generation network converges to obtain a preset noise reduction model.

8. An audio processing device, characterized in that: The audio processing device comprises: An acquisition module, used for acquiring an input sound pickup signal; A detection module, used to detect the input sound pickup signal and identify the wind noise frame in the input sound pickup signal; The noise reduction module is used to input the wind noise frame into a preset noise reduction model to obtain a noise-reduced sound pickup signal corresponding to the wind noise frame, wherein the preset noise reduction model is obtained by training a generative adversarial network using a user voice vector, noisy samples and clean samples.

9. An audio processing device, characterized in that: The audio processing device comprises: a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the steps of the audio processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an audio processing program, and when the audio processing program is executed by a processor, the steps of the audio processing method according to any one of claims 1 to 7 are implemented.

Images