Model training method and device, audio processing method and device and electronic equipment
By model training on audio with wind noise and other noises, a target model that can remove wind noise and retain other noises is obtained, which solves the problem of unsatisfactory noise reduction processing in the prior art and achieves better audio Processing effect.
Patent Information
- Application Number
- CN202311001653.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-09
- Publication Date
- 2025-02-18
AI Technical Summary
When the prior art reduces the wind noise in audio, it is difficult to effectively remove wind noise without affecting other noises, resulting in unsatisfactory audio processing effect.
By obtaining audio with wind noise and other noises, training the preset model to obtain a target model that can remove wind noise and retain other noises. The audio input target model with wind noise is then processed to remove wind noise and retain other noises.
It realizes that in the process of removing wind noise, reduces the impact on other noise, avoids distortion caused by excessive audio cleaning, and improves the audio processing effect.
Smart Images
Figure CN119479671A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of audio processing technology, and in particular to a model training method, an audio processing method, a device and an electronic device. Background Art
[0002] When audio is collected using an audio collection device, the collected audio usually carries wind noise, which affects the audio playback effect.
[0003] In order to improve the audio playback effect, after the audio with wind noise is collected, the wind noise in the audio is usually reduced, but the processing effect is not ideal. Summary of the invention
[0004] The present application provides a model training method, an audio processing method, an apparatus, a storage medium and an electronic device.
[0005] In a first aspect, an embodiment of the present application provides a model training method, the method comprising:
[0006] Acquire a first audio; the first audio has a first noise and a second noise; the first noise includes wind noise, and the second noise is different from the first noise;
[0007] A preset model is trained based on the first audio to obtain a target model, wherein the target model is used to remove the first noise and retain the second noise.
[0008] Optionally, obtaining the first audio includes:
[0009] Acquire a second audio, wherein the second audio includes the second noise;
[0010] The first noise is added to the second audio to obtain the first audio.
[0011] Optionally, adding the first noise to the second audio to obtain the first audio includes:
[0012] collecting the first noise,
[0013] Attenuating a specified part of the first noise to obtain the wind noise; wherein the frequency of the specified part is greater than a first preset value;
[0014] The wind noise is added to the second audio to obtain the first audio.
[0015] Optionally, obtaining the second audio includes:
[0016] collecting a fourth audio frequency;
[0017] The second noise is added to the fourth audio to obtain the second audio.
[0018] Optionally, the training the preset model based on the first audio to obtain a target model includes:
[0019] Inputting the first audio into the preset model to obtain a first gain function output by the preset model;
[0020] Processing the first audio based on the first gain function to obtain a fifth audio having the second noise;
[0021] Based on the fifth audio and the second audio, the model parameters of the preset model are adjusted to obtain the target model.
[0022] Optionally, inputting the first audio into the preset model to obtain a first gain function output by the preset model includes:
[0023] Performing frequency domain conversion on the first audio to obtain first audio data and second audio data, wherein the first audio data is used to indicate an amplitude feature and / or a frequency feature of the first audio, and the second audio data is used to indicate a phase feature of the first audio;
[0024] The first audio data and the second audio data are input into the preset model to obtain the first gain function output by the preset model.
[0025] Optionally, the processing the first audio based on the first gain function to obtain a fifth audio having the second noise includes:
[0026] Adjusting the first audio data based on the first gain function to obtain third audio data;
[0027] Adjusting the second audio data based on the first gain function to obtain fourth audio data;
[0028] The fifth audio is generated based on the third audio data and the fourth audio data.
[0029] In a second aspect, an embodiment of the present application provides an audio processing method, the method comprising:
[0030] Inputting the audio to be processed having first noise and second noise into the target model described above to obtain a second gain coefficient output by the target model, wherein the first noise includes wind noise, and the second noise is different from the first noise;
[0031] The audio to be processed is processed based on the second gain coefficient to obtain target audio, where the target audio is audio from which the first noise has been removed and the second noise is retained.
[0032] In a third aspect, an embodiment of the present application provides a model training device, the device comprising:
[0033] An acquisition module, configured to acquire a first audio; the first audio has a first noise and a second noise; the first noise includes wind noise, and the second noise is different from the first noise;
[0034] A training module is used to train a preset model based on the first audio to obtain a target model, wherein the target model is used to remove the first noise and retain the second noise.
[0035] Optionally, the acquisition module includes:
[0036] An acquiring unit, configured to acquire a second audio, wherein the second audio includes the second noise;
[0037] An adding unit is used to add the first noise to the second audio to obtain the first audio.
[0038] Optionally, the adding unit is used to include:
[0039] collecting the first noise,
[0040] Attenuating a specified part of the first noise to obtain the wind noise; wherein the frequency of the specified part is greater than a first preset value;
[0041] The wind noise is added to the second audio to obtain the first audio.
[0042] Optionally, the acquiring unit is used to:
[0043] collecting a fourth audio frequency;
[0044] The second noise is added to the fourth audio to obtain the second audio.
[0045] Optionally, the training module includes:
[0046] An output unit, used for inputting the first audio into the preset model to obtain a first gain function output by the preset model;
[0047] a processing unit, configured to process the first audio based on the first gain function to obtain a fifth audio having the second noise;
[0048] An adjustment unit is used to adjust the model parameters of the preset model based on the fifth audio and the second audio to obtain the target model.
[0049] Optionally, the output unit is used to:
[0050] Performing frequency domain conversion on the first audio to obtain first audio data and second audio data, wherein the first audio data is used to indicate an amplitude feature and / or a frequency feature of the first audio, and the second audio data is used to indicate a phase feature of the first audio;
[0051] The first audio data and the second audio data are input into the preset model to obtain the first gain function output by the preset model.
[0052] Optionally, the processing unit is used to:
[0053] Adjusting the first audio data based on the first gain function to obtain third audio data;
[0054] Adjusting the second audio data based on the first gain function to obtain fourth audio data;
[0055] The fifth audio is generated based on the third audio data and the fourth audio data.
[0056] In a fourth aspect, an embodiment of the present application provides an audio processing device, the device comprising:
[0057] An input module, used for inputting the audio to be processed having first noise and second noise into the target model to obtain a second gain coefficient output by the target model, wherein the first noise includes wind noise, and the second noise is different from the first noise;
[0058] A processing module is used to process the audio to be processed based on the second gain coefficient to obtain a target audio, where the target audio is the audio from which the first noise has been removed and the second noise is retained.
[0059] In a fifth aspect, an embodiment of the present application provides a non-temporary computer-readable storage medium, which, when the instructions in the storage medium are executed by a processor of a computer, enables the computer to execute the model training method and / or audio processing method as described above.
[0060] In a sixth aspect, an embodiment of the present application provides an electronic device, including:
[0061] a memory for storing processor-executable instructions;
[0062] A processor connected to the memory;
[0063] Wherein, the processor is used to execute any of the model training methods and / or audio processing methods described above.
[0064] The technical solution provided by the embodiments of the present application may have the following beneficial effects:
[0065] In the embodiment of the present application, the preset model is trained by the first audio with the first noise and the second noise to obtain the target model with the ability to remove the first noise and retain the second noise. Based on this, the audio with wind noise is processed by the target model, and when the wind noise is removed, the non-wind noise can be retained, which reduces the impact of the wind noise removal process on the non-wind noise in the audio, thereby reducing the distortion caused by the audio denoising being too clean, and improving the audio processing effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0067] Figure 1 A flowchart of an audio processing method is provided;
[0068] Figure 2 is a flowchart of a model training method according to an exemplary embodiment;
[0069] Figure 3 is a schematic structural diagram of a preset model according to an exemplary embodiment;
[0070] Figure 4 is a schematic diagram of a process of processing first noise to obtain wind noise according to an exemplary embodiment;
[0071] Figure 5 is a schematic diagram of a process of processing first noise to obtain wind noise according to another exemplary embodiment;
[0072] Figure 6 is a flowchart of a model training method according to another exemplary embodiment;
[0073] Figure 7 is a flowchart of an audio processing method according to another exemplary embodiment;
[0074] Figure 8 is a flowchart of a model training method and an audio processing method according to an exemplary embodiment;
[0075] Fig. 9is a flowchart of a model training method according to another exemplary embodiment;
[0076] Fig.10 is a structural schematic diagram of a model training device according to an exemplary embodiment;
[0077] Fig.11 is a structural schematic diagram of an audio processing device according to an exemplary embodiment;
[0078] Fig.12 It is a schematic structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION
[0079] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices consistent with some aspects of the present application as detailed in the appended claims.
[0080] Before explaining the embodiments of the present application in detail, the application scenarios of the embodiments of the present application are first introduced, as follows:
[0081] When audio is collected through an audio collection component, if the audio collection component is in a windy environment, the audio collected by the audio collection component includes not only the sound of the target object and the noise emitted by other sound-emitting bodies in the environment where the target object is located, but also the wind noise generated by the turbulence formed by the wind around the audio collection component. Wind noise will affect the audio playback effect. Based on this, after collecting audio with wind noise, the audio is usually processed by noise reduction technology to remove the wind noise.
[0082] However, if the collected audio contains non-wind noise in addition to wind noise, when the wind noise in the audio is reduced using noise reduction technology in related technologies, part of the non-wind noise will also be removed, resulting in distortion of the non-wind noise in the audio after noise reduction processing.
[0083] For example, when collecting audio from a concert in a windy environment, the collected audio may contain not only the singer's voice, but also the sounds of musical instruments, the cheers of the audience, and wind noise. In order to improve the playback effect of the collected audio, you can use Figure 1 The method described above removes wind noise from the collected concert audio. However, in the process of removing the wind noise, some background sounds and cheers of the audience may be attenuated and distorted, affecting the audio playback effect.
[0084] In view of this, an embodiment of the present application provides a model training to train a preset model to obtain a target model that can remove wind noise in audio and retain non-wind noise. Wind noise in audio is removed based on the trained target model, which can improve the audio processing effect.
[0085] refer to Figure 2 , the embodiment of the present application provides a model training method, comprising:
[0086] S201: Acquire a first audio; the first audio has a first noise and a second noise; the first noise includes wind noise, and the second noise is different from the first noise;
[0087] S202: Training a preset model based on the first audio to obtain a target model, wherein the target model is used to remove the first noise and retain the second noise.
[0088] The model training method of the embodiment of the present application can be executed on an electronic device with certain computing capabilities, for example, on a laptop or desktop computer.
[0089] Exemplarily, the first noise mentioned above may be audio in which the proportion of wind noise is greater than a second preset value, wherein the second preset value may be any value greater than or equal to 80%.
[0090] The second noise mentioned above may be any non-wind noise. Exemplarily, the second noise may include sound emitted by any sound-emitting body except the target object.
[0091] For example, in a concert scene, the second noise may be non-wind noise emitted by any sound source other than the singer. For example, the second noise may include: the voice of a non-target sound source, cheers and claps from the audience, and background sounds from musical instruments.
[0092] The first audio may be audio collected in an environment with wind noise and no wind noise, or may be audio collected in a noise-free environment by adding the first noise and the second noise.
[0093] The above-mentioned preset model can be any deep neural network model, for example, a convolutional neural network model, a recursive neural network model, a deep belief neural network model, and a generative adversarial network.
[0094] For example, reference Figure 3 The structure of the preset model in the embodiment of the present application can be as follows: Figure 3 As shown, the preset model may include a feature extraction module ( Figure 3 The encoder in the Figure 3LSTM) and gain function output module ( Figure 3 decoder in ).
[0095] Among them, the feature extraction module is used to extract features of the input first audio; the feature recognition module is used to identify the features of the first noise from the features output by the feature extraction module; and the gain function output module is used to output the first gain function according to the features of the first noise identified by the feature recognition module.
[0096] The feature extraction module may be composed of multiple convolutional layers (Conv2D), and illustratively, may be composed of 3-10 Conv2D layers, for example, 5 Conv2D layers, wherein the convolution kernel size (Kernel Size) used by each Conv2D is (3, 1) and the stride (Stride) is (2, 1).
[0097] In the feature extraction module, the output channel of each layer is twice that of the previous layer. Therefore, when the feature extraction module has 5 convolution layers, the size of the convolution kernel is (3, 1), and the step size is (2, 1), the feature extraction module has 64 output channels.
[0098] The feature recognition module may use LSTM (Long Short-Term Memory), wherein the number of LSTM units is the same as the number of output channels of the feature extraction module. For example, when the feature extraction module has 64 output channels, the number of LSTM units is also 64.
[0099] The gain function output module may be composed of a plurality of deconvolution (ConvTranspose2D) layers, wherein the number of deconvolution layers included in the gain function output module may be determined according to the number of convolution layers included in the feature extraction module. For example, when the feature extraction module includes 5 Conv2Ds, 5 ConvTranspose2Ds may be included in the gain function output module, and the Kernel Size is (3, 1) and the Stride is (2, 1).
[0100] In an embodiment of the present application, the number of convolutional layers included in the encoder of the target model and the number of deconvolutional layers included in the decoder can be designed according to the computing power of the electronic device that executes the method. When the computing power of the electronic device is limited, a smaller number of convolutional layers and a smaller number of deconvolutional layers can be used, such as 3 convolutional layers and 3 deconvolutional layers. Therefore, the trained target model can be migrated to more types of electronic devices for processing the audio collected by the audio collection component in the electronic device, thereby increasing the scope of application of the target model.
[0101] In the embodiment of the present application, the preset model is trained by the first audio with the first noise and the second noise to obtain the target model with the ability to remove the first noise and retain the second noise. Based on this, the audio with wind noise is processed by the target model, and when the wind noise is removed, the non-wind noise can be retained, which reduces the impact of the wind noise removal process on the non-wind noise in the audio, thereby reducing the distortion caused by the audio denoising being too clean, and improving the audio processing effect.
[0102] In one embodiment, obtaining the first audio includes:
[0103] Acquire a second audio, wherein the second audio includes a second noise;
[0104] The first noise is added to the second audio to obtain the first audio.
[0105] Exemplarily, the second audio may be audio corresponding to the target collected in an environment without wind and with non-wind noise, for example, audio of the target object collected in an environment without wind and with background music.
[0106] The first noise mentioned above is audio collected in a quiet and windy environment.
[0107] Among them, considering that different wind speeds, different wind volumes and / or different wind directions will have different effects on the collected first noise, therefore, when collecting the first noise, different types of first noise can be obtained by changing the wind speed, wind volume and / or wind direction.
[0108] Exemplarily, a windy environment can be created by using a wind-generating tool, such as a fan or a blower, and the wind speed, wind volume and / or wind direction flowing through the audio collection component can be adjusted by adjusting the wind outlet direction and power of the wind-generating tool to obtain different types of first noise.
[0109] Exemplarily, considering that the first noise collected by audio collection components of different models may be different when the wind speed, wind volume and wind direction are the same, the first noise can also be obtained by collecting audio at different wind speeds, different wind volumes and / or different wind directions through audio collection components of different models.
[0110] After the second audio and the first noise are obtained, the first audio with different wind noises can be obtained by fusing different first noises with different second audio.
[0111] In an embodiment of the present application, the first noise is obtained by using different models of audio collection components to collect audio at different wind speeds, different wind volumes and / or different wind directions. This can make the obtained first audio contain multiple types of wind noise. By training the preset model with such first audio, the obtained target model can recognize more types of wind noise, thereby improving the target model's ability to recognize wind noise.
[0112] In one embodiment, obtaining the second audio includes:
[0113] collecting a fourth audio frequency;
[0114] The second noise is added to the fourth audio to obtain the second audio.
[0115] Exemplarily, in order to enhance the recognition ability of the preset model for non-wind noise, the audio of different target objects can be collected in a quiet environment to obtain a fourth audio without noise or with less noise. The sound emitted by other sound-emitting bodies except the target object is collected in a windless environment to obtain the second noise.
[0116] Afterwards, the fourth audio collected from different target objects and the second noise collected in different environments are fused to obtain the second audio.
[0117] The method of fusing the collected fourth audio of different target objects with the collected second noise in different environments to obtain the second audio can be expressed by the following formula (1).
[0118] x(n)=s(n)+d(n) (1)
[0119] Wherein, x(n) is the second audio, s(n) is the fourth audio without noise or with less noise, and d(n) is the second noise.
[0120] In one embodiment, adding the first noise to the second audio to obtain the first audio includes:
[0121] Collect the first noise,
[0122] Attenuating a specified part of the first noise to obtain the wind noise; wherein the frequency of the specified part is greater than a first preset value;
[0123] The wind noise is added to the second audio to obtain the first audio.
[0124] For example, reference Figure 4In an environment with natural wind or artificial wind, the first noise is collected. Considering that the collected first noise may contain non-wind noise generated by the sound of wind-making tools or other objects, and the frequency of the non-wind noise is usually greater than the frequency of the wind noise, the part of the first noise with a frequency greater than the first preset value can be attenuated to obtain wind noise.
[0125] For example, refer to Figure 5 After the first noise is collected, the first noise is converted from the time domain to the frequency domain through short-time Fourier transform (STFT) to obtain the amplitude information and phase information of the first noise, and based on the amplitude information and phase information of the first noise, a search is performed from the low-frequency channel to the high-frequency channel to determine the first channel whose frequency is less than a first preset value, and the amplitude of the audio signal in the first channel whose amplitude is less than a second preset value is attenuated, and then the processed first noise is converted from the frequency domain to the time domain to obtain wind noise.
[0126] Among them, the sizes of the first preset value and the second preset value can be set as needed, and the embodiment of the present application does not limit this.
[0127] For example, the first preset value and the second preset value may be set according to the type of wind noise included in the collected first noise.
[0128] After the wind noise is obtained, the wind noise is added to the second audio to obtain the first audio.
[0129] Based on the following formula (2), wind noise can be added to the second audio to obtain the first audio.
[0130] y(n)=s(n)+d(n)+w(n)=x(n)+w(n) (2)
[0131] Among them, y(n) is the first audio, x(n) is the second audio, and w(n) is the wind noise.
[0132] In an embodiment of the present application, non-wind noise in the first noise is removed by attenuating the part of the collected first noise whose frequency is greater than a first preset value. This can increase the proportion of wind noise in the first noise, thereby improving the preset model's ability to recognize wind noise in the first audio.
[0133] refer to Figure 6 In one embodiment, the step S202 includes:
[0134] S2021: Inputting the first audio into the preset model to obtain a first gain function output by the preset model;
[0135] S2022: Process the first audio based on the first gain function to obtain a fifth audio having the second noise;
[0136] S2023: Based on the fifth audio and the second audio, adjust the model parameters of the preset model to obtain the target model.
[0137] Exemplarily, after obtaining a first audio having various wind noises and non-wind noises, the first audio can be framed to obtain multiple audio frames, and then each audio frame is converted from the time domain to the frequency domain through Fourier transform, and the obtained frequency domain signal is input into a preset model, which is used for the preset model to identify the characteristics of the wind noise from the frequency domain signal obtained from the first audio conversion, and output a first gain function based on the identified characteristics of the wind noise.
[0138] After obtaining the first gain function, the wind noise in the first audio can be attenuated by the first gain function to obtain a fifth audio with the second noise. Then, based on the difference between the fifth audio and the second audio, the model parameters of the preset model are adjusted until the preset model converges or reaches a preset number of training times to obtain a target model.
[0139] In one embodiment, the S2021 includes:
[0140] Performing frequency domain conversion on the first audio to obtain first audio data and second audio data, wherein the first audio data is used to indicate an amplitude feature and / or a frequency feature of the first audio, and the second audio data is used to indicate a phase feature of the first audio;
[0141] The first audio data and the second audio data are input into the preset model to obtain the first gain function output by the preset model.
[0142] Exemplarily, after obtaining the first audio with different wind noise and non-wind noise, the first audio is framed to obtain multiple audio frames, and the multiple audio frames are converted from time domain to frequency domain through short-time Fourier transform to obtain first audio data and second audio data of the first audio.
[0143] After obtaining the first audio data and the second audio data of the first audio, the first audio data and the second audio data are input into the preset model. Since the first audio data can indicate the amplitude characteristics and / or frequency characteristics of the first audio, and the second audio characteristics can indicate the phase characteristics of the first audio. Therefore, after the first audio data and the second audio data are obtained by the preset model, the corresponding data of the wind noise can be identified from the first audio data and the second audio data based on the amplitude characteristics and / or frequency characteristics of the first audio and the phase characteristics of the first audio, and the first gain function is output based on the corresponding data of the wind noise.
[0144] Exemplarily, the first audio data may be real data obtained after converting the first audio from a time domain signal into a frequency domain signal through a short-time Fourier transform, and the second audio data may be imaginary data obtained after converting the first audio from a time domain signal into a frequency domain signal through a short-time Fourier transform.
[0145] In one embodiment, inputting the first audio data and the second audio data into the preset model to obtain the first gain function output by the preset model includes:
[0146] Extracting a first feature from the first audio data based on the preset model, and extracting a second feature from the second audio data;
[0147] Based on the first feature and the second feature, the first gain function is output.
[0148] Exemplarily, after the first audio data and the second audio data are input into a preset model, the feature extraction module of the preset model can extract the first feature from the first audio data and extract the second feature from the second audio data. After the feature extraction module obtains the first feature and the second feature, the first feature and the second feature are input into the feature recognition module, the feature recognition module recognizes the feature of the wind noise from the first feature and the second feature, and the gain function output module outputs the first gain function based on the proportion of the feature of the wind noise in the first feature and the second feature.
[0149] For example, when the first audio data is the real data obtained after the first audio is transformed by short-time Fourier transform, and the second audio data is the imaginary data obtained after the first audio is transformed by short-time Fourier transform, the first gain function output by the gain function output module includes a real gain function corresponding to the real data of the first audio and an imaginary gain function corresponding to the imaginary data of the first audio.
[0150] In one embodiment, the processing the first audio based on the first gain function to obtain a fifth audio having the second noise includes:
[0151] Adjusting the first audio data based on the first gain function to obtain third audio data;
[0152] Adjusting the second audio data based on the first gain function to obtain fourth audio data;
[0153] The fifth audio is generated based on the third audio data and the fourth audio data.
[0154] From the above introduction, it can be seen that when the first audio data is the real data obtained after the first audio is transformed by short-time Fourier transform, and the second audio data is the imaginary data obtained after the short-time Fourier transform, the first gain function includes a real gain function corresponding to the real data of the first audio, and an imaginary gain function corresponding to the imaginary data of the first audio.
[0155] After obtaining the real gain function and the imaginary gain function, the real data and the imaginary data of the first audio can be adjusted based on the first calculation method by the real gain function and the imaginary gain function to remove the audio data related to wind noise in the real data and the imaginary data to obtain the third audio data.
[0156] Based on the second calculation method, the real data and imaginary data of the first audio are adjusted by a real gain function and an imaginary gain function to remove audio data related to wind noise in the real data and the imaginary data to obtain fourth audio data.
[0157] The first calculation method mentioned above may be the calculation method shown in the following formula (3).
[0158]
[0159] in, For the third audio data, Gain R (T, F) is the real gain function, Y R (T, F) real data, Gain I (T, F) is the imaginary gain function, Y I (T, F) is the imaginary part data, T is the Tth audio frame of the first audio, and F is the frequency of the first audio.
[0160] The above-mentioned second calculation method can be expressed by the following formula (4).
[0161]
[0162] in, It is the fourth audio data.
[0163] After obtaining the third audio data and the fourth audio data, audio reorganization can be performed based on the third audio data and the fourth audio data to obtain fifth audio data, and the fifth audio data can be obtained by performing an inverse short-time Fourier transform on the fifth audio data.
[0164] Exemplarily, the third audio data and the fourth audio data may be reorganized by the following formula (5) to obtain the fifth audio data.
[0165]
[0166] in, is the fifth audio data, is the third-degree audio data, 1j is the imaginary unit, It is the fourth audio data.
[0167] In one embodiment, adjusting the model parameters of the preset model based on the fifth audio and the second audio to obtain the target model includes:
[0168] determining a loss value based on a difference between the fifth audio frequency and the second audio frequency;
[0169] Based on the loss value, the model parameters of the preset model are adjusted to obtain the target model.
[0170] Exemplarily, when the fifth audio and the second audio are obtained, the mean squared error (MSE), mean absolute error (MAE) / or scale-invariant signal-to-distortion ratio (SI-SDR) of the fifth audio and the second audio can be calculated, and based on the mean squared error, mean absolute error / or scale-invariant signal-to-distortion ratio of the fifth audio and the second audio, the model parameters of the preset model are adjusted through the back propagation algorithm (BP) until the preset model converges or reaches a preset number of training times to obtain the target model.
[0171] refer to Figure 7 , the embodiment of the present application also provides an audio processing method, including:
[0172] Inputting the audio to be processed having first noise and second noise into the target model described above to obtain a second gain coefficient output by the target model, wherein the first noise includes wind noise, and the second noise is different from the first noise;
[0173] The audio to be processed is processed based on the second gain coefficient to obtain target audio, where the target audio is audio from which the first noise has been removed and the second noise is retained.
[0174] The audio processing method of the embodiment of the present application is executed by an electronic device having an audio acquisition component or capable of being connected to an audio acquisition device.
[0175] Exemplarily, the electronic device used to execute the audio processing method of the embodiment of the present application may be a mobile phone, a tablet computer, a phone watch, a learning device, a wearable device, and a smart home appliance.
[0176] Among them, smart home appliances include smart speakers and smart TVs.
[0177] Exemplarily, after the audio acquisition component in the electronic device or the audio acquisition device connected to the electronic device acquires the audio to be processed having the first noise and the second noise, the audio to be processed is converted from the time domain to the frequency domain through short-time Fourier transform to obtain the real data and imaginary data of the audio to be processed, and the real data and imaginary data of the audio to be processed are input into the trained preset model, that is, the real data and imaginary data of the audio to be processed are input into the target model, so that the target model can output a second gain function based on the real data and imaginary data of the audio to be processed.
[0178] After obtaining the second gain function, the real data and imaginary data of the processed audio are adjusted by the second gain function, the adjusted real data and imaginary data are fused, and the frequency domain is converted to the time domain to obtain the target audio with the first noise removed and the second noise retained.
[0179] Among them, the implementation method of the target model outputting the second gain function based on the real data and imaginary data of the audio to be processed can refer to the implementation method of the above-mentioned preset model outputting the first gain function based on the first audio data and the second audio data, and the embodiments of the present application will not be repeated here.
[0180] The implementation process of adjusting the real data and imaginary data of the processed audio based on the second gain function can refer to the implementation process of adjusting the first audio data and the second audio data based on the first gain function, and the embodiments of the present application will not be repeated here.
[0181] refer to Figure 8 and Fig. 9 , the embodiment of the present application also provides a model training method, including:
[0182] refer to Figure 8In the training phase of the preset model, the first audio is subjected to a short-time Fourier transform to obtain the real data and imaginary data of the first audio, the real data and imaginary data of the first audio are input into the preset model, and the loss value of the fifth audio and the second audio obtained by processing the first audio according to the preset model is calculated. Based on the loss value, the model parameters are optimized by the back propagation algorithm to obtain a target model capable of removing the first noise and retaining the second noise.
[0183] In the audio processing stage, the audio to be processed with the first noise and the second noise is input into the target model, the enhanced audio features are estimated, and finally the time domain waveform is reconstructed to obtain the target audio.
[0184] In the above process, the noise reduction process will not optimize the model parameters, and the training phase determines the wind noise reduction performance of the model. The model training process includes:
[0185] S901: training data generation;
[0186] Training data, also known as the first audio, is one of the keys to achieving the wind noise reduction task using a deep neural network. How to obtain ideal training data is the difficulty of this wind noise reduction task.
[0187] The synthesis process of the first audio is shown in formula (2).
[0188] The synthesis process of the second audio is shown in formula (1).
[0189] In the embodiment of the present application, in order to synthesize y(n) and x(n), s(n), d(n) and w(n) need to be collected. For x(n), s(n) and d(n) can be collected separately or simultaneously.
[0190] As for w(n), it can be collected in the following way:
[0191] Wind noise is the noise that is recorded by the microphone when the wind forms turbulence on the surface of the microphone. For different microphones, the turbulence generated by the wind on their surface will be different, resulting in different characteristics of the wind noise recorded by the microphone. When collecting wind noise, the ideal state is to be in an environment with wind and no other noise, but in the actual collection process, there is often a certain amount of environmental noise, and the obtained is wind noise mixed with non-wind noise.
[0192] To account for the differences in microphones and the impact of ambient noise, refer to Figure 4 and Figure 5In terms of the selection of the collection environment, a natural wind environment or an artificial wind environment with low interference noise should be selected (using wind-making tools, such as fans); in terms of microphone selection, since different microphones have different wind noise characteristics, different types of microphones can be used to collect wind noise to reduce the difference between the training data and the actual wind noise.
[0193] After the first noise is acquired, Figure 5 The non-wind noise in the first noise is attenuated in a manner to increase the proportion of wind noise in the first noise.
[0194] like Figure 5 As shown, firstly, the first noise recorded by the microphone is subjected to a short-time discrete Fourier transform (STFT), and the amplitude spectrum and phase spectrum are calculated. The wind noise is characterized by a large overall energy in the spectrum, which gradually decreases from the low-frequency channel to the high-frequency channel. According to the masking characteristics of the human ear, the wind noise energy is large in the low-frequency signal, which makes the non-wind noise signal with smaller energy unable to be perceived, and as the frequency increases, the energy of the wind noise signal will gradually decrease. When the frequency of the first noise exceeds the first preset value, the energy of the non-wind noise included in the first noise will be greater than the energy of the wind noise. Therefore, the demarcation point between the wind noise and the non-wind noise in the first noise can be determined by the first preset value, and the search is performed from the low-frequency channel to the high-frequency channel. When a spectrum channel with a frequency greater than the first preset value is encountered, the amplitude of the signal with an amplitude less than the second preset value in the first pass with a frequency greater than the first preset value is attenuated to remove the non-wind noise in the first noise.
[0195] The amplitude spectrum of the processed first noise is subjected to inverse short time discrete Fourier transform (ISTFT) to obtain wind noise.
[0196] S902: preset model design;
[0197] Considering the limited computing power of electronic devices, the limitation of computational complexity needs to be considered when designing the preset model. The preset model in the embodiment of the present application is a lightweight deep neural network model. The framework of the preset model is as follows Figure 3As shown. In the encoder module, it consists of 5 layers of Conv2D, kernel_size is (3,1), strides is (2,1), and the output channel of each layer is twice that of the previous layer, so the final output channel is 64. The number of LSTM units is set to 64, and the output channel of the encoder is used as the input of the LSTM layer to keep the input and output dimensions consistent. In the decoder, it consists of 5 layers of ConvTranspose2D, kernel_size is (3,1), strides is (2,1), and the input of each layer is composed of the output of the previous layer and the two sets of features of the corresponding layer in the encoder spliced in the channel. The output channel of each layer is half of the output of the previous layer, and the final network input and output dimensions are consistent. Through the encoder-decoder structure, the effective information of the network input features can be deeply mined, so that the LSTM layer can better learn data features. In addition, LSTM has the ability to model temporal correlation, which has great advantages in speech enhancement tasks.
[0198] In this preset model, the number of Conv2D and ConvTranspose2D layers in the codec is relatively small, and the number of units in the LSTM layer is only 64. The overall computational complexity is relatively small, and audio processing can be implemented during audio recording.
[0199] S903: Preset model training:
[0200] The training process of the preset model mainly consists of three key parts: feature extraction, training objectives and loss function.
[0201] In the feature extraction process, STFT is used for frequency domain conversion, and the real data and imaginary data of the calculated frequency domain information are input into the preset model. After being processed by the preset model, the output is the real gain function and the imaginary gain function, and the output dimension is consistent with the input dimension. The real gain function and the imaginary gain function output by the preset model are compared with the input real data and imaginary data to obtain enhanced real data and imaginary data.
[0202] After obtaining the enhanced real data and imaginary data, audio reorganization and inverse short-time Fourier transform are performed to obtain the fifth audio with wind noise reduced.
[0203] In order to make the fifth audio after wind noise reduction as close as possible to the second audio, it is necessary to adjust the parameters of the preset model according to the difference between the second audio and the fifth chip. The calculation of the difference is the loss function. Common loss functions include mean square error, mean absolute error and scale-invariant signal distortion rate (SI-SDR). After calculating the loss value, the network parameters are updated through the back propagation algorithm. After a large amount of training data and multiple rounds of updates, the preset model has an effective gain function estimated from the characteristics of the speech with wind noise, so that the wind noise signal in the speech with wind noise is effectively suppressed.
[0204] refer to Fig.10 The present application also provides a model training device, which includes:
[0205] An acquisition module 1001 is used to acquire a first audio; the first audio has a first noise and a second noise; the first noise includes wind noise, and the second noise is different from the first noise;
[0206] The training module 1002 is used to train a preset model based on the first audio to obtain a target model, wherein the target model is used to remove the first noise and retain the second noise.
[0207] Optionally, the acquisition module 1001 includes:
[0208] An acquiring unit, configured to acquire a second audio, wherein the second audio includes a second noise;
[0209] An adding unit is used to add the first noise to the second audio to obtain the first audio.
[0210] Optionally, the adding unit is used to include:
[0211] collecting the first noise,
[0212] Attenuating a specified part of the first noise to obtain the wind noise; wherein the frequency of the specified part is greater than a first preset value;
[0213] The wind noise is added to the second audio to obtain the first audio.
[0214] Optionally, the acquiring unit is used to:
[0215] collecting a fourth audio frequency;
[0216] The second noise is added to the fourth audio to obtain the second audio.
[0217] Optionally, the training module 1002 includes:
[0218] An output unit, used for inputting the first audio into the preset model to obtain a first gain function output by the preset model;
[0219] a processing unit, configured to process the first audio based on the first gain function to obtain a fifth audio having the second noise;
[0220] An adjustment unit is used to adjust the model parameters of the preset model based on the fifth audio and the second audio to obtain the target model.
[0221] Optionally, the output unit is used to:
[0222] Performing frequency domain conversion on the first audio to obtain first audio data and second audio data, wherein the first audio data is used to indicate an amplitude feature and / or a frequency feature of the first audio, and the second audio data is used to indicate a phase feature of the first audio;
[0223] The first audio data and the second audio data are input into the preset model to obtain the first gain function output by the preset model.
[0224] Optionally, the processing unit is used to:
[0225] Adjusting the first audio data based on the first gain function to obtain third audio data;
[0226] Adjusting the second audio data based on the first gain function to obtain fourth audio data;
[0227] The fifth audio is generated based on the third audio data and the fourth audio data.
[0228] refer to Fig.11 , an embodiment of the present application provides an audio processing device, the device comprising:
[0229] An input module 1101 is used to input the audio to be processed having a first noise and a second noise into the target model described above, to obtain a second gain coefficient output by the target model, wherein the first noise includes wind noise, and the second noise is different from the first noise;
[0230] The processing module 1102 is configured to process the audio to be processed based on the second gain coefficient to obtain target audio, where the target audio is audio from which the first noise has been removed and the second noise is retained.
[0231] refer to Fig.12In an embodiment of the present application, an electronic device 1200 is provided, including:
[0232] Memory 1204 for storing processor executable instructions;
[0233] A processor 1220 connected to the memory 1204;
[0234] Among them, the processor 1220 is configured to execute the model training method and / or audio processing method provided by any of the aforementioned technical solutions.
[0235] A block diagram of an electronic device 1200 is shown according to an exemplary embodiment. For example, the electronic device 1200 may be a smart terminal such as a smart phone, a tablet computer, or a smart watch with an audio acquisition component.
[0236] refer to Fig.12 , the electronic device 1200 may include one or more of the following components: a processing component 1202 , a memory 1204 , a power component 1208 , a multimedia component 1208 , an audio component 1210 , an input / output (I / O) interface 1212 , a sensor component 1214 , and a communication component 1218 .
[0237] The processing component 1202 generally controls the overall operation of the electronic device 1200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1202 may include one or more processors 1220 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 1202 may include one or more modules to facilitate the interaction between the processing component 1202 and other components. For example, the processing component 1202 may include a multimedia module to facilitate the interaction between the multimedia component 1208 and the processing component 1202.
[0238] The memory 1204 is configured to store various types of data to support operations on the electronic device 1200. Examples of such data include instructions for any application or method operating on the electronic device 1200, contact data, phone book data, messages, pictures, videos, etc. The memory 1204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0239] The power supply component 1208 provides power to the various components of the electronic device 1200. The power supply component 1208 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1200.
[0240] The multimedia component 1208 includes a screen that provides an output interface between the electronic device 1200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1208 includes a front camera and / or a rear camera. When the electronic device 1200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0241] The audio component 1210 is configured to output and / or input audio signals. For example, the audio component 1210 includes a microphone (MIC), and when the electronic device 1200 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1204 or sent via the communication component 1218. In some embodiments, the audio component 1210 also includes a speaker for outputting audio signals.
[0242] I / O interface 1212 provides an interface between processing component 1202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.
[0243] The sensor assembly 1214 includes one or more sensors for providing various aspects of status assessment for the electronic device 1200. For example, the sensor assembly 1214 can detect the open / closed state of the electronic device 1200, the relative positioning of components, such as the display and keypad of the electronic device 1200, and the sensor assembly 1214 can also detect the position change of the electronic device 1200 or a component of the electronic device 1200, the presence or absence of user contact with the electronic device 1200, the orientation or acceleration / deceleration of the electronic device 1200, and the temperature change of the electronic device 1200. The sensor assembly 1214 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 1214 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1214 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0244] The communication component 1218 is configured to facilitate wired or wireless communication between the electronic device 1200 and other devices. The electronic device 1200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1218 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1218 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0245] In an exemplary embodiment, the electronic device 1200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.
[0246] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1204 including instructions, and the instructions can be executed by a processor 1220 of the electronic device 1200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0247] An embodiment of the present application provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by a processor of a computer, the computer can execute the model training method and / or audio processing method described in one or more of the aforementioned technical solutions.
[0248] When the processor executes the instruction, it can at least perform the following steps:
[0249] Acquire a first audio; the first audio has a first noise and a second noise; the first noise includes wind noise, and the second noise is different from the first noise;
[0250] A preset model is trained based on the first audio to obtain a target model, wherein the target model is used to remove the first noise and retain the second noise.
[0251] Optionally, obtaining the first audio includes:
[0252] Acquire a second audio, wherein the second audio includes a second noise;
[0253] The first noise is added to the second audio to obtain the first audio.
[0254] Optionally, adding the first noise to the second audio to obtain the first audio includes:
[0255] collecting the first noise,
[0256] Attenuating a specified part of the first noise to obtain the wind noise; wherein the frequency of the specified part is greater than a first preset value;
[0257] The wind noise is added to the second audio to obtain the first audio.
[0258] Optionally, obtaining the second audio includes:
[0259] collecting a fourth audio frequency;
[0260] The second noise is added to the fourth audio to obtain the second audio.
[0261] Optionally, the training the preset model based on the first audio to obtain a target model includes:
[0262] Inputting the first audio into the preset model to obtain a first gain function output by the preset model;
[0263] Processing the first audio based on the first gain function to obtain a fifth audio having the second noise;
[0264] Based on the fifth audio and the second audio, the model parameters of the preset model are adjusted to obtain the target model.
[0265] Optionally, inputting the first audio into the preset model to obtain a first gain function output by the preset model includes:
[0266] Performing frequency domain conversion on the first audio to obtain first audio data and second audio data, wherein the first audio data is used to indicate an amplitude feature and / or a frequency feature of the first audio, and the second audio data is used to indicate a phase feature of the first audio;
[0267] The first audio data and the second audio data are input into the preset model to obtain the first gain function output by the preset model.
[0268] Optionally, the processing the first audio based on the first gain function to obtain a fifth audio having the second noise includes:
[0269] Adjusting the first audio data based on the first gain function to obtain third audio data;
[0270] Adjusting the second audio data based on the first gain function to obtain fourth audio data;
[0271] The fifth audio is generated based on the third audio data and the fourth audio data.
[0272] When the processor executes the instruction, it can at least perform the following steps:
[0273] Inputting the audio to be processed having first noise and second noise into the target model described above to obtain a second gain coefficient output by the target model, wherein the first noise includes wind noise, and the second noise is different from the first noise;
[0274] The audio to be processed is processed based on the second gain coefficient to obtain target audio, where the target audio is audio from which the first noise has been removed and the second noise is retained.
[0275] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0276] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A model training method, characterized in that: The method comprises: Acquire a first audio; the first audio has a first noise and a second noise; the first noise includes wind noise, and the second noise is different from the first noise; A preset model is trained based on the first audio to obtain a target model, wherein the target model is used to remove the first noise and retain the second noise.
2. The method according to claim 1, characterized in that The obtaining of the first audio includes: Acquire a second audio, wherein the second audio includes the second noise; The first noise is added to the second audio to obtain the first audio.
3. The method according to claim 2, characterized in that The adding the first noise to the second audio to obtain the first audio includes: collecting the first noise, Attenuating a specified part of the first noise to obtain the wind noise; wherein the frequency of the specified part is greater than a first preset value; The wind noise is added to the second audio to obtain the first audio.
4. The method according to claim 2, characterized in that: The obtaining of the second audio includes: collecting a fourth audio frequency; The second noise is added to the fourth audio to obtain the second audio.
5. The method according to any one of claims 2 to 4, characterized in that: The step of training the preset model based on the first audio to obtain a target model includes: Inputting the first audio into the preset model to obtain a first gain function output by the preset model; Processing the first audio based on the first gain function to obtain a fifth audio having the second noise; Based on the fifth audio and the second audio, the model parameters of the preset model are adjusted to obtain the target model.
6. The method according to claim 5, characterized in that The step of inputting the first audio into the preset model to obtain a first gain function output by the preset model includes: Performing frequency domain conversion on the first audio to obtain first audio data and second audio data, wherein the first audio data is used to indicate an amplitude feature and / or a frequency feature of the first audio, and the second audio data is used to indicate a phase feature of the first audio; The first audio data and the second audio data are input into the preset model to obtain the first gain function output by the preset model.
7. The method according to claim 6, characterized in that The step of processing the first audio based on the first gain function to obtain a fifth audio having the second noise includes: Adjusting the first audio data based on the first gain function to obtain third audio data; Adjusting the second audio data based on the first gain function to obtain fourth audio data; The fifth audio is generated based on the third audio data and the fourth audio data.
8. An audio processing method, characterized in that: The method comprises: Inputting the audio to be processed having the first noise and the second noise into the target model according to any one of claims 1 to 7, to obtain a second gain coefficient output by the target model, wherein the first noise includes wind noise, and the second noise is different from the first noise; The audio to be processed is processed based on the second gain coefficient to obtain target audio, where the target audio is audio from which the first noise has been removed and the second noise is retained.
9. A model training device, characterized in that: The device comprises: An acquisition module, configured to acquire a first audio; the first audio has a first noise and a second noise; the first noise includes wind noise, and the second noise is different from the first noise; A training module is used to train a preset model based on the first audio to obtain a target model, wherein the target model is used to remove the first noise and retain the second noise.
10. An audio processing device, characterized in that: The device comprises: An input module, used for inputting the audio to be processed having a first noise and a second noise into the target model according to any one of claims 1 to 7, to obtain a second gain coefficient output by the target model, wherein the first noise includes wind noise, and the second noise is different from the first noise; A processing module is used to process the audio to be processed based on the second gain coefficient to obtain a target audio, where the target audio is the audio from which the first noise has been removed and the second noise is retained.
11. A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a computer, enables the computer to execute the model training method as described in any one of claims 1 to 7, and / or the audio processing method as described in claim 8.
12. An electronic device, characterized in that: include: a memory for storing processor-executable instructions; A processor connected to the memory; The processor is used to execute the model training method as described in any one of claims 1 to 7, and / or the audio processing method as described in claim 8.