Audio noise reduction method, device and storage medium
By using an audio denoising model based on Mel-frequency cepstral coefficients and frequency domain features, the problem of audio data loss in traditional methods is solved, achieving better audio data processing results and training efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-27
- Publication Date
- 2026-03-20
AI Technical Summary
Traditional audio noise reduction methods may eliminate human voices in audio data during differential amplification, resulting in data loss and affecting the processing effect.
The audio noise reduction model is based on Mel-frequency cepstral coefficients and frequency domain features. Filtering parameters are generated through two network branches for filtering, and then corrected by a feature splicing layer to suppress noise and amplify human voice details.
It improves the efficiency of audio data processing, reduces the computational cost of neural networks, enhances the preservation and generalization of feature information, and improves training efficiency.
Smart Images
Figure CN116013348B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the field of audio data processing, and particularly relates to an audio noise reduction method, device and storage medium. BACKGROUND
[0002] At present, multiple audio acquisition components are usually installed in an electronic device, and the electronic device needs to process audio data collected by the multiple audio acquisition components to achieve the effect of reducing noise of the audio data.
[0003] A conventional electronic device usually has at least two audio acquisition components, and when performing audio noise reduction, first audio data collected by a first audio acquisition component and second audio data collected by a second audio acquisition component are obtained; the second audio data is used to perform differential amplification processing on the first audio data to eliminate noise in the first audio data, and obtain the first audio data after noise elimination; after the first audio data after noise elimination is input into a pre-trained audio noise reduction model, audio data after noise reduction is obtained.
[0004] However, when eliminating noise in the first audio data during differential amplification processing, the original human voice in the first audio data may be eliminated at the same time, so the processed first audio data may have data loss, which may result in poor audio data processing effect. SUMMARY
[0005] The application provides a network management method, device and storage medium, which can eliminate the original human voice in the first audio data when eliminating noise in the first audio data during differential amplification processing, so the processed first audio data may have data loss, which may result in poor audio data processing effect. The application provides the following technical solutions:
[0006] In a first aspect, an audio noise reduction method is provided, and the method comprises:
[0007] obtaining first audio data collected by a first audio acquisition component and second audio data collected by a second audio acquisition component; the first audio acquisition component and the second audio acquisition component belong to the same electronic device and synchronously collect audio data;
[0008] determining a mel-frequency cepstral coefficient corresponding to the first audio data;
[0009] determining a first frequency domain feature corresponding to the first audio data and a second frequency domain feature corresponding to the second audio data;
[0010] input the mel-frequency cepstral coefficients into a first network branch of a pre-trained audio noise reduction model, input the first frequency domain feature and the second frequency domain feature into a second network branch of the audio noise reduction model, to generate an initial noise reduction spectrum based on the mel-frequency cepstral coefficients through the first network branch; determine the filtering parameters generated based on the first audio data and the second audio data through the second network branch; a filtering layer connected to the first network branch and the second network branch respectively is used to filter the initial noise reduction spectrum using the filtering parameters to obtain the first audio data after noise reduction; the filtering parameters are used to suppress the audio to be suppressed in the current environment and amplify the audio to be amplified;
[0011] The second network branch has a feature splicing layer connected to a preset hidden layer in the first network branch to obtain an intermediate result output by the preset hidden layer, and the feature splicing layer is used to correct feature data input into the feature splicing layer of the second network branch using the intermediate result.
[0012] Optionally, the first network branch includes m first separable convolution layers connected in sequence and m second separable convolution layers connected in sequence, and the i th first separable convolution layer is further connected to the m-i+1 th second separable convolution layer through a convolution connection jump layer; m is an integer greater than 1, and i is a positive integer from 1 to m in sequence;
[0013] The second network branch includes a separable convolution layer, at least one first grouped linear layer, the feature splicing layer, a first feature extraction layer, a second feature extraction layer connected to the first feature extraction layer, and a plurality of second grouped linear layers connected in sequence; the second feature extraction layer is connected to the first network branch through a grouped recurrent network and a grouped linear layer; the plurality of second grouped linear layers are used to calculate the filtering parameters; and the feature splicing layer is connected to the m th first separable convolution layer.
[0014] The first feature extraction layer and the second feature extraction layer each include at least one grouped linear layer and a grouped recurrent network connected in sequence.
[0015] Optionally, the pre-trained audio noise reduction model is obtained by:
[0016] The training data includes: sample first audio data, sample mel-frequency cepstral coefficients corresponding to the sample first audio data, and standard spectrum corresponding to the sample first audio data, sample first audio data and sample second audio data, sample first frequency domain features corresponding to the sample first audio data and sample second frequency domain features corresponding to the sample second audio data, and standard filtering parameters corresponding to the sample first audio data and the sample second audio data;
[0017] inputting the sample mel-frequency cepstral coefficient into a pre-created first network branch to obtain a first training result;
[0018] inputting the first training result and a standard spectrum corresponding to the sample first audio data into a pre-set first loss function to obtain a first loss function value;
[0019] performing iterative training on the pre-created first network branch using the first loss function value to obtain a trained first network branch;
[0020] keeping the network parameters of the trained first network branch unchanged, inputting the sample first frequency domain feature and the sample second frequency domain feature into a pre-created second network branch to obtain a second training result;
[0021] inputting the second training result and standard filter coefficients corresponding to the sample first audio data and the sample second audio data into a pre-set second loss function to obtain a second loss function value;
[0022] performing iterative training on the pre-created second network branch using the second loss function value to obtain a trained second network branch.
[0023] Optionally, the obtaining of the training data comprises:
[0024] obtaining simulated audio data of human voice emitted at different angles simulated by an audio simulation program;
[0025] performing Fourier transform on the simulated audio data to obtain mel-frequency cepstral coefficients corresponding to the simulated audio data and frequency domain features corresponding to the simulated audio data.
[0026] Optionally, the determining of the mel-frequency cepstral coefficients corresponding to the first audio data comprises:
[0027] performing Fourier transform on the first audio data to obtain first audio data in a frequency domain;
[0028] mapping the sample first audio data in the frequency domain by a pre-set value to obtain the mel-frequency cepstral coefficients.
[0029] Optionally, the determining of the first frequency domain feature corresponding to the first audio data and the second frequency domain feature corresponding to the second audio data comprises:
[0030] performing Fourier transform on the first audio data and the second audio data to obtain first audio data and sample second audio data in a frequency domain;
[0031] determining phase information and amplitude information of the first audio data and the second audio data;
[0032] The phase information and the amplitude information of the preset numerical frequency point are taken as the frequency domain feature corresponding to the first audio data and the frequency domain feature of the second audio data.
[0033] Optionally, the method further includes:
[0034] The data volume of the training data is in a positive correlation with the training duration.
[0035] Optionally, the first audio acquisition component is the audio acquisition component closest to the sound source, and the second audio acquisition component is the other audio acquisition component in the electronic device except the first audio acquisition component.
[0036] In a second aspect, an electronic device is provided, which includes a processor and a memory; the memory stores a program which is loaded and executed by the processor to implement the audio noise reduction method according to any one of the first aspect.
[0037] In a third aspect, a computer readable storage medium is provided, which stores a program which is executed by a processor to implement the audio noise reduction method according to any one of the first aspect.
[0038] The application has the beneficial effects that: the first audio data collected by the first audio collection component and the second audio data collected by the second audio collection component are obtained; the first audio collection component and the second audio collection component belong to the same electronic device, and audio data is collected synchronously; the mel-frequency cepstral coefficient corresponding to the first audio data is determined; the first frequency domain feature corresponding to the first audio data and the second frequency domain feature corresponding to the second audio data are determined; the mel-frequency cepstral coefficient is input into the first network branch in the pre-trained audio noise reduction model, and the first frequency domain feature and the second frequency domain feature are input into the second network branch in the audio noise reduction model, so as to generate an initial noise reduction spectrum based on the mel-frequency cepstral coefficient through the first network branch; the filter parameter is determined based on the first audio data and the second audio data through the second network branch; the filter layer connected with the first network branch and the second network branch is used for filtering the initial noise reduction spectrum by using the filter parameter to obtain the first audio data after noise reduction; the filter parameter is used for suppressing the audio to be suppressed in the current environment and amplifying the audio to be amplified; wherein the feature splicing layer is arranged in the second network branch, the feature splicing layer is connected with the preset hidden layer in the first network branch, so as to obtain the intermediate result output by the preset hidden layer, and the feature data input into the feature splicing layer of the second network branch is corrected by using the intermediate result; the problem that the traditional audio noise reduction method may eliminate the original human voice in the first audio data when eliminating the noise of the first audio data during differential amplification processing, so that the processed first audio data may exist data loss, and the audio data processing effect is poor, can be solved; since the first audio data and the second audio data are simultaneously input into the audio noise reduction model, and the noise reduction spectrum of the first audio data is adjusted by using the output filter coefficient, unlike the traditional audio noise reduction method which only inputs the first audio data into the audio noise reduction network, the filter parameter output by the audio noise reduction model can make the noise in the noise reduction spectrum more easily eliminated, and the human voice details in the noise reduction spectrum can also be amplified, so that the effect of audio data processing can be improved.
[0039] In addition, since the first audio data is converted from the time domain to the frequency domain and the features are extracted by the mel-frequency cepstrum, such operation enables us to still retain more information into the network under the condition of less calculation times, and the information entering the network is double information of amplitude and phase, so that the network can perceive more feature information.
[0040] In addition, since the phase information and amplitude information of the preset number of frequency points are used as the input of the network, it is not necessary to input the phase information and amplitude information of all frequency points, so that the size of the input data can be further compressed, and at the same time, the network can perceive more feature information.
[0041] In addition, because the traditional convolutional layer needs to consume high network computing power when extracting features from input information, resulting in low network output efficiency, based on the above technical problems, in the embodiment, the mel cepstrum coefficient is extracted by adopting a separable convolutional layer and a grouped linear layer, which can compress the network structure, thereby greatly reducing the computing power of the neural network, and at the same time, the output noise reduction spectrum can maintain excellent noise reduction performance.
[0042] In addition, after the mel cepstrum coefficient is compressed and extracted by the separable convolutional layer, the compressed mel cepstrum coefficient is restored by using the separable convolutional layer, and a plurality of convolutional skip layers are used to correct the error in the restoration process, so that a large amount of distorted information not introduced in the compression process is added to the restored features, so that the restored noise reduction spectrum restores more details of the human voice.
[0043] In addition, by obtaining the training data simulated by the audio simulation software, the simulation data is closer to the actual audio data application, and the flexibility of the simulation data is adopted, and the generalization of the data is further improved, thereby greatly reducing the human cost of algorithm research and development, and facilitating the learning of the network.
[0044] In addition, because the learning efficiency of the audio noise reduction model gradually decreases during the training process, if the training amount is kept unchanged, some feature information cannot be learned by the audio noise reduction model, based on the above technical problems, in the embodiment, the training data amount is gradually increased during the training process of the audio noise reduction model, which can reduce the learning rate of the audio noise reduction model, and the feature information in the stable process of the audio noise reduction network region can be more perceived by the audio noise reduction model, thereby improving the training efficiency of the audio noise reduction model. BRIEF DESCRIPTION OF DRAWINGS
[0045] Figure 1 is a flowchart of an audio noise reduction method provided by an embodiment of the present application;
[0046] Figure 2 is a schematic diagram of an audio noise reduction model structure provided by an embodiment of the present application;
[0047] Figure 3 is a waveform diagram of original audio data provided by an embodiment of the present application;
[0048] Figure 4 is a waveform diagram of audio data processed by a traditional audio noise reduction model provided by an embodiment of the present application;
[0049] Figure 5 is a waveform diagram of audio data processed by an audio noise reduction model provided by an embodiment of the present application;
[0050] Figure 6 is a block diagram of an audio noise reduction device provided by an embodiment of the present application;
[0051] Figure 7 is a block diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0052] The technical solutions of the present application will be described in detail below with reference to the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Hereinafter, the present application will be described in detail with reference to the drawings and in conjunction with the embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.
[0053] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.
[0054] In the application, the orientation words such as "up", "down", "top", "bottom" are generally used for the direction shown in the drawings, or for the vertical, vertical or gravity direction of the components themselves; similarly, for the convenience of understanding and description, "inner", "outer" refers to the inner and outer relative to the contour of each component, but the above-mentioned orientation words are not used to limit the present application.
[0055] The audio noise reduction method provided by the present application will be described in detail below.
[0056] Optionally, the execution subject of the audio noise reduction method provided by the present application is an electronic device, which can be a computer, a mobile phone, a tablet computer, a camera terminal, or also a server, and the present embodiment does not limit the implementation manner of the electronic device.
[0057] The present embodiment provides an audio noise reduction method, as shown in the figure, which comprises at least the following steps: Figure 1
[0058] Step 101, obtaining first audio data collected by a first audio collection component and second audio data collected by a second audio collection component.
[0059] Among them, the first audio collection component and the second audio collection component belong to the same electronic device, and collect audio data synchronously.
[0060] Optionally, the first audio collection component is the audio collection component closest to the sound source, and the second audio collection component is the other audio collection component in the electronic device except the first audio collection component.
[0061] wherein the sound source refers to an object that is occurring.
[0062] Optionally, the first audio acquisition component and the second audio acquisition component can be a microphone or a microphone array, etc. The embodiment does not limit the type of the first audio acquisition component and the second audio acquisition component.
[0063] For example, taking the electronic device as a mobile phone, and the first audio acquisition component and the second audio acquisition component as microphones, the mobile phone usually has one microphone on the front and one microphone on the back, respectively, for collecting audio data. Since the user is closer to the front microphone than the back microphone when using the mobile phone to make a call, the front microphone of the mobile phone is taken as the first audio acquisition component, and the back microphone of the mobile phone is taken as the second audio acquisition component. At this time, the first audio acquisition component is mainly used to collect human voice, and the second audio acquisition component is mainly used to collect human voice and other noise audio data.
[0064] Step 102, determining the mel-frequency cepstral coefficient corresponding to the first audio data.
[0065] Optionally, determining the mel-frequency cepstral coefficient corresponding to the first audio data comprises: performing Fourier transform on the first audio data to obtain the first audio data in the frequency domain; and mapping the sample first audio data in the frequency domain with a preset value to obtain the mel-frequency cepstral coefficient.
[0066] wherein performing Fourier transform on the first audio data to obtain the first audio data in the frequency domain comprises: performing frame processing on the first audio data to obtain the first audio data signal after frame processing; and performing short-time Fourier transform on the first audio data signal after frame processing to obtain the first audio data signal in the frequency domain.
[0067] Specifically, the first audio data is processed by frame, and each 10 ms is set as a frame, and there are N frames in total, N being a positive integer; the 0th frame is set as a compensation frame in front of the 1st frame, and the N+1th frame is set as a compensation frame behind the Nth frame, and 20 ms of audio data signal of the current frame and the previous frame is processed each time from the 1st frame to the Nth frame, and there is 10 ms of overlap between adjacent frames, and each frame from the 1st frame to the Nth frame is processed twice by short-time Fourier transform to enhance the reliability of the algorithm, and the conversion of the first audio data signal to the first audio data signal in the frequency domain is completed after the 1st frame to the Nth frame are processed.
[0068] Specifically, the sample first audio data in the frequency domain is mapped with a preset number of values to obtain mel-frequency cepstral coefficients, specifically: using mel scale to simulate the nonlinear perception of human ears to the frequency band, arranging a group of band-pass filters according to the size of the critical bandwidth from low frequency to high frequency, each band-pass filter filters the input digital audio signal in the frequency domain; the signal energy output by each band-pass filter is taken as the basic feature of the digital audio signal in the frequency domain, and the mel-frequency cepstral coefficients of the basic feature are calculated; further, 32 frequency bands are taken in this embodiment, 32 band-pass filters are used, and 32 mel-frequency cepstral coefficients are generated by 32 band-pass filters, which compresses the input information while reducing the influence on the sound quality.
[0069] In this embodiment, since the first audio data is converted from the time domain to the frequency domain and the features are extracted by the mel-frequency cepstrum, such operation enables us to retain more information into the network under the condition of less operation times, and the information entering the network is retained with the dual information of amplitude and phase, so that the network can perceive more feature information.
[0070] Step 103, determining the first frequency domain feature corresponding to the first audio data and the second frequency domain feature corresponding to the second audio data.
[0071] Optionally, determining the first frequency domain feature corresponding to the first audio data and the second frequency domain feature corresponding to the second audio data comprises: performing Fourier transform on the first audio data and the second audio data to obtain the first audio data and the sample second audio data in the frequency domain; determining the phase information and the amplitude information of the first audio data and the second audio data; taking the phase information and the amplitude information of the preset number of frequency points as the frequency domain feature of the first audio data and the frequency domain feature of the second audio data.
[0072] Optionally, the preset number of frequency points is pre-stored in the electronic device.
[0073] For example, the preset number of frequency points can be 96.
[0074] In this embodiment, since the phase information and the amplitude information of the preset number of frequency points are used as the input of the network, it is not necessary to input the phase information and the amplitude information of all frequency points, which can further compress the size of the input data while ensuring that the network can perceive more feature information.
[0075] In step 104, the mel-frequency cepstral coefficients are input into a first network branch of a pre-trained audio noise reduction model, the first frequency domain feature and the second frequency domain feature are input into a second network branch of the audio noise reduction model, an initial noise reduction spectrum is generated based on the mel-frequency cepstral coefficients by the first network branch, a filtering parameter is determined based on the first audio data and the second audio data by the second network branch, and a filtering layer connected to the first network branch and the second network branch respectively is configured to filter the initial noise reduction spectrum using the filtering parameter to obtain the first audio data after noise reduction.
[0076] The filtering parameter is used to suppress the audio to be suppressed in the current environment and amplify the audio to be amplified.
[0077] The second network branch has a feature splicing layer connected to a preset hidden layer in the first network branch to obtain an intermediate result output by the preset hidden layer and correct feature data input into the feature splicing layer of the second network branch using the intermediate result.
[0078] As shown in Figure 2 The first network branch includes m first separable convolution layers connected in sequence and m second separable convolution layers connected in sequence, and the i th first separable convolution layer is further connected to the m-i+1 th second separable convolution layer through a convolution connection jump layer, where m is an integer greater than 1, i is a positive integer from 1 to m.
[0079] The second network branch includes a separable convolution layer, at least one first grouped linear layer, a feature splicing layer, a first feature extraction layer, a second feature extraction layer connected to the first feature extraction layer, and a plurality of second grouped linear layers connected in sequence; the second feature extraction layer is connected to the first network branch through a grouped recurrent network and a grouped linear layer; the plurality of second grouped linear layers are used to calculate the filtering parameter; and the feature splicing layer is connected to the m th first separable convolution layer.
[0080] The first feature extraction layer and the second feature extraction layer each include at least one grouped linear layer and a grouped recurrent network connected in sequence.
[0081] Since the traditional convolution layer needs to consume high network computing power when extracting features from input information, the network output efficiency is low. Based on the above technical problems, in the embodiment, the mel-frequency cepstral coefficients are extracted by the separable convolution layer and the grouped linear layer, which can greatly reduce the computing power of the neural network, and at the same time ensure that the output noise reduction spectrum maintains excellent noise reduction performance.
[0082] In addition, after the mel-frequency cepstrum coefficient is compressed and extracted by the separable convolution layer, the separable convolution layer is also used for restoration when the compressed mel-frequency cepstrum coefficient is restored. In order to correct the error in the restoration process, a plurality of convolution skip layers are used. In this way, a large amount of distorted information not introduced in the compression process can be added to the restored feature, so that the restored noise reduction spectrum can restore more details of the human voice.
[0083] Optionally, the pre-trained audio noise reduction model is obtained, and at least the following steps S1 to S7 are included.
[0084] In step S1, training data is obtained, and the training data includes: sample first audio data, sample mel-frequency cepstrum coefficients corresponding to the sample first audio data, and standard spectrum corresponding to the sample first audio data, sample first audio data and sample second audio data, sample first frequency domain features corresponding to the sample first audio data and sample second frequency domain features corresponding to the sample second audio data, and standard filtering parameters corresponding to the sample first audio data and the sample second audio data.
[0085] In one example, the training data is obtained, including: obtaining simulated audio data of human voice emitted at different angles by an audio simulation program; and performing Fourier transform on the simulated audio data to obtain mel-frequency cepstrum coefficients corresponding to the simulated audio data and frequency domain features corresponding to the simulated audio data.
[0086] For example, taking a mobile phone as an example, the audio simulation program sets the position of the human voice to be within 10 cm of the first audio acquisition component, and the second audio acquisition component is at any position within 12-15 cm from the first audio acquisition component, thereby simulating different handheld modes in handheld calls. At the same time, by changing the parameters of the audio simulation program, the directivity of the mouth can be changed, thereby simulating simulated audio data of human voice emitted at different angles.
[0087] In this embodiment, the training data is obtained by the audio simulation software, so that the simulation data is closer to the actual audio data application, and the flexibility of the simulation data is also improved, thereby further improving the generalization of the data, greatly reducing the human cost of algorithm development, and being more conducive to network learning.
[0088] In another example, the training data is obtained, including: obtaining audio data collected by different audio acquisition components; and performing Fourier transform on the audio data to obtain mel-frequency cepstrum coefficients corresponding to the audio data and frequency domain features corresponding to the audio data.
[0089] In step S2, the sample mel-frequency cepstrum coefficients are input into a pre-created first network branch to obtain a first training result.
[0090] Step S3: Input the first training result and the standard spectrum corresponding to the first audio data of the sample into the preset first loss function to obtain the value of the first loss function.
[0091] Step S4: Iteratively train the pre-created first network branch using the first loss function value to obtain the trained first network branch.
[0092] Step S5: Keep the network parameters of the first network branch after training unchanged, input the first frequency domain features and the second frequency domain features of the sample into the pre-created second network branch to obtain the second training result.
[0093] Step S6: Input the second training result and the standard filtering coefficients corresponding to the first audio data and the second audio data of the sample into the preset second loss function to obtain the value of the second loss function.
[0094] Step S7: Iteratively train the pre-created second network branch using the second loss function value to obtain the trained second network branch.
[0095] In this embodiment, during the training of the audio noise reduction model, the amount of training data is positively correlated with the training duration.
[0096] As the learning efficiency of audio denoising models gradually decreases during training, keeping the training amount constant will result in some feature information not being learned by the audio denoising model. Based on the above technical problem, in this embodiment, the amount of training data is gradually increased during the training process of the audio denoising model. This can reduce the learning rate of the audio denoising model, and at the same time, the feature information can be perceived by the audio denoising model to a greater extent during the stabilization process of the audio denoising network region, thereby improving the training efficiency of the audio denoising model.
[0097] To sum up, the audio noise reduction method provided by the embodiment, by acquiring first audio data collected by a first audio collection component and second audio data collected by a second audio collection component; the first audio collection component and the second audio collection component belong to the same electronic device, and audio data is collected synchronously; the mel-frequency cepstral coefficient corresponding to the first audio data is determined; the first frequency domain feature corresponding to the first audio data and the second frequency domain feature corresponding to the second audio data are determined; the mel-frequency cepstral coefficient is input into a first network branch in a pre-trained audio noise reduction model, and the first frequency domain feature and the second frequency domain feature are input into a second network branch in the audio noise reduction model, so as to generate an initial noise reduction spectrum based on the mel-frequency cepstral coefficient through the first network branch; the filter parameter is determined based on the first audio data and the second audio data through the second network branch; the filter layer connected with the first network branch and the second network branch is used for filtering the initial noise reduction spectrum using the filter parameter to obtain the first audio data after noise reduction; the filter parameter is used to suppress the audio to be suppressed in the current environment and amplify the audio to be amplified; wherein the feature splicing layer is arranged in the second network branch, the feature splicing layer is connected with the preset hidden layer in the first network branch, so as to obtain the intermediate result output by the preset hidden layer, and the feature data input into the feature splicing layer of the second network branch is corrected using the intermediate result; the problem that the processed first audio data has data loss can be solved, which is caused by the fact that the noise of the first audio data is eliminated when the differential amplification processing is performed in the traditional audio noise reduction method, and the original human voice in the first audio data can be eliminated at the same time, so that the processed first audio data has data loss, which can cause poor audio data processing effect; since the first audio data and the second audio data are input into the audio noise reduction model at the same time, and the noise reduction spectrum of the first audio data is adjusted through the output filter coefficient, unlike the traditional audio noise reduction method which only inputs the first audio data into the audio noise reduction network, the filter parameter output by the audio noise reduction model can make the noise in the noise reduction spectrum more easily eliminated, and the human voice details in the noise reduction spectrum can also be amplified, so that the effect of audio data processing can be improved.
[0098] In addition, since the first audio data is converted from the time domain to the frequency domain and the features are extracted through the mel-frequency cepstrum, such operation enables us to retain more information into the network under the condition of less calculation times, and the information entering the network is double information of amplitude and phase, so that the network can perceive more feature information.
[0099] In addition, since the phase information and amplitude information of the preset number of frequency points are used as the input of the network, it is not necessary to input the phase information and amplitude information of all frequency points, so that the size of the input data can be further compressed, and at the same time, the network can perceive more feature information.
[0100] In addition, because the traditional convolutional layer needs to consume high network computing power when extracting features from the input information, resulting in low network output efficiency, based on the above technical problems, in the embodiment, the mel cepstrum coefficient is extracted by adopting the separable convolutional layer and the grouped linear layer, which can compress the network structure, thereby greatly reducing the computing power of the neural network, and at the same time, the output noise reduction spectrum can maintain excellent noise reduction performance.
[0101] In addition, after the mel cepstrum coefficient is compressed and extracted by the separable convolutional layer, the separable convolutional layer is also used for restoration when the compressed mel cepstrum coefficient is restored, and a plurality of convolutional skip layers are used to correct the error in the restoration process, so that a large amount of distorted information not introduced in the compression process is added to the restored features, so that the restored noise reduction spectrum restores more details of the human voice.
[0102] In addition, by obtaining the training data simulated by the audio simulation software, the simulation data is closer to the actual audio data application, and at the same time, due to the flexibility of the simulation data, the generalization of the data is further improved, thereby greatly reducing the human cost of algorithm development, and the network learning is more beneficial.
[0103] In addition, because the learning efficiency of the audio noise reduction model gradually decreases during the training process, if the training amount is kept unchanged, some feature information cannot be learned by the audio noise reduction model. Based on the above technical problems, in the embodiment, the training data amount is gradually increased during the training process of the audio noise reduction model, which can reduce the learning rate of the audio noise reduction model, and at the same time, the feature information in the stable process of the audio noise reduction network region can be more perceived by the audio noise reduction model, thereby improving the training efficiency of the audio noise reduction model.
[0104] In order to more clearly illustrate the technical effects of the above-mentioned audio noise reduction method compared with the traditional audio noise reduction method, the following will be described in detail in combination with Table 1, Figure 3 , Figure 4 and Figure 5 .
[0105] Table 1 shows the comparison of related parameters of the traditional audio noise reduction model and the audio noise reduction model in the embodiment. According to Table 1, the operation speed and the speech score after noise reduction of the audio noise reduction model in the embodiment are significantly improved compared with the traditional audio noise reduction model.
[0106]
[0107] Figure 3 The original audio data waveform diagram before the audio data is processed is shown in FIG. 1,Figure 4 shows a waveform diagram obtained by processing only the input first audio data in the conventional technology, Figure 5 shows a waveform diagram obtained by processing the input first audio data and the second audio data in the embodiment, according to Figure 4 and Figure 5 The comparison of the two clearly shows that the original audio data noise elimination has a smaller elimination effect.
[0108] Figure 6 is a block diagram of an audio noise reduction device provided by an embodiment of the present application, and the device includes at least the following modules: a data acquisition module 610, a first determination module 620, a second determination module 630, and a data noise reduction module 640.
[0109] The data acquisition module 610 is configured to acquire first audio data collected by a first audio collection component and second audio data collected by a second audio collection component; the first audio collection component and the second audio collection component belong to the same electronic device and synchronously collect audio data.
[0110] The first determination module 620 is configured to determine a mel-frequency cepstral coefficient corresponding to the first audio data.
[0111] The second determination module 630 is configured to determine a first frequency domain feature corresponding to the first audio data and a second frequency domain feature corresponding to the second audio data.
[0112] The data noise reduction module 640 is configured to input the mel-frequency cepstral coefficient into a first network branch of a pre-trained audio noise reduction model, and input the first frequency domain feature and the second frequency domain feature into a second network branch of the audio noise reduction model, so as to generate an initial noise reduction spectrum based on the mel-frequency cepstral coefficient through the first network branch; determine a filtering parameter generated based on the first audio data and the second audio data through the second network branch; and a filtering layer connected to the first network branch and the second network branch is configured to filter the initial noise reduction spectrum using the filtering parameter to obtain the first audio data after noise reduction.
[0113] For related details, refer to the above embodiments.
[0114] It should be noted that the audio noise reduction device provided in the above embodiments is only used for example to divide the above functional modules when performing audio noise reduction, and in actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the entity content generation device is divided into different functional modules to complete all or part of the functions described above. In addition, the audio noise reduction device and the audio noise reduction method provided in the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be repeated here.
[0115] The present embodiment provides an electronic device, such asFigure 7 As shown, the electronic device at least includes a processor 701 and a memory 702.
[0116] The processor 701 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 701 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 701 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also known as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 701 can be integrated with a GPU (Graphics Processing Unit) that is responsible for rendering and drawing the content to be displayed by the display screen. In some embodiments, the processor 701 can also include an AI (Artificial Intelligence) processor for processing machine learning-related computing operations.
[0117] The memory 702 can include one or more computer-readable storage media that can be non-transitory. The memory 702 can also include a high-speed random access memory, and a non-volatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 702 is used to store at least one instruction for being executed by the processor 701 to implement the audio noise reduction method provided by the method embodiments in the present application.
[0118] In some embodiments, the electronic device can also optionally include a peripheral device interface and at least one peripheral device. The processor 701, the memory 702, and the peripheral device interface can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface through a bus, a signal line, or a circuit board. Illustratively, the peripheral devices include, but are not limited to, a radio frequency circuit, a touch display screen, an audio circuit, and a power supply, etc.
[0119] Of course, the electronic device can also include fewer or more components, which are not limited in the present embodiment.
[0120] Optionally, the present application also provides a computer readable storage medium, the computer readable storage medium stores a program, the program is loaded and executed by a processor to realize the audio noise reduction method of the above method embodiment.
[0121] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present disclosure.
[0122] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.
Claims
1. An audio noise reduction method, characterized in that, The method includes: Acquire first audio data collected by a first audio acquisition component and second audio data collected by a second audio acquisition component; the first audio acquisition component and the second audio acquisition component belong to the same electronic device and acquire audio data synchronously; Determine the Mel-frequency cepstral coefficients corresponding to the first audio data; Determine the first frequency domain feature corresponding to the first audio data and the second frequency domain feature corresponding to the second audio data; The Mel-frequency cepstral coefficients are input into the first network branch of a pre-trained audio denoising model, and the first and second frequency domain features are input into the second network branch of the audio denoising model. An initial denoised spectrum is generated based on the Mel-frequency cepstral coefficients through the first network branch. Filtering parameters are generated based on the first and second audio data through the second network branch. Filtering layers connected to the first and second network branches respectively are used to filter the initial denoised spectrum using the filtering parameters to obtain denoised first audio data. The filtering parameters are used to suppress audio to be suppressed and amplify audio to be amplified in the current environment. The second network branch has a feature splicing layer, which is connected to a preset hidden layer in the first network branch to obtain intermediate results output by the preset hidden layer, and uses the intermediate results to correct the feature data input to the feature splicing layer from the second network branch. The first network branch includes m sequentially connected first separable convolutional layers and m sequentially connected second separable convolutional layers. The i-th first separable convolutional layer is also connected to the (m-i+1)-th second separable convolutional layer through a convolutional jump layer. The m is an integer greater than 1, and the i is a positive integer from 1 to m. The second network branch includes a separable convolutional layer, at least one first grouped linear layer, the feature concatenation layer, a first feature extraction layer, a second feature extraction layer connected to the first feature extraction layer, and multiple second grouped linear layers connected in sequence; the second feature extraction layer is connected to the first second separable convolutional layer of the first network branch through a grouped recurrent network and a grouped linear layer; the multiple second grouped linear layers are used to calculate the filtering parameters; the feature concatenation layer is connected to the m-th first separable convolutional layer; The first feature extraction layer and the second feature extraction layer each include at least one grouped linear layer and a grouped recurrent network connected in sequence.
2. The method according to claim 1, characterized in that, Obtain a pre-trained audio denoising model, including: Acquire training data, which includes: sample first audio data, sample Mel-Cepstral coefficients corresponding to sample first audio data, standard spectrum corresponding to sample first audio data, sample first audio data and sample second audio data, sample first frequency domain features corresponding to sample first audio data and sample second frequency domain features corresponding to sample second audio data, and standard filtering parameters corresponding to sample first audio data and sample second audio data. The sample Mel-frequency cepstral coefficients are input into a pre-created first network branch to obtain the first training result; The first training result and the standard spectrum corresponding to the first audio data of the sample are input into a preset first loss function to obtain the value of the first loss function; The first loss function value is used to iteratively train the pre-created first network branch to obtain the trained first network branch. Keeping the network parameters of the first network branch after training unchanged, the first frequency domain feature and the second frequency domain feature of the sample are input into the pre-created second network branch to obtain the second training result; The second training result and the standard filtering coefficients corresponding to the first audio data and the second audio data of the sample are input into a preset second loss function to obtain the value of the second loss function. The pre-created second network branch is iteratively trained using the second loss function value to obtain the trained second network branch.
3. The method according to claim 2, characterized in that, The acquisition of training data includes: Acquire simulated audio data from different angles, generated by an audio simulation program; The simulated audio data is subjected to Fourier transform to obtain the Mel-frequency cepstral coefficients and frequency domain characteristics corresponding to the simulated audio data.
4. The method according to claim 1, characterized in that, Determining the Mel-spectral coefficients corresponding to the first audio data includes: Perform a Fourier transform on the first audio data to obtain the first audio data in the frequency domain; The first audio data of the frequency domain sample is mapped with a preset value to obtain the Mel-spectral coefficients.
5. The method according to claim 1, characterized in that, Determining the first frequency domain feature corresponding to the first audio data and the second frequency domain feature corresponding to the second audio data includes: Perform Fourier transform on the first audio data and the second audio data to obtain the first audio data and sampled second audio data in the frequency domain; Determine the phase information and amplitude information of the first audio data and the second audio data; The phase information and amplitude information of the preset numerical frequency points are used as the frequency domain features of the first audio data and the frequency domain features of the second audio data.
6. The method according to claim 2, characterized in that, The method further includes: The amount of training data is positively correlated with the training duration.
7. The method according to claim 1, characterized in that, The first audio acquisition component is the audio acquisition component closest to the sound source, and the second audio acquisition component is any other audio acquisition component in the electronic device besides the first audio acquisition component.
8. An electronic device, characterized in that, The device includes a processor and a memory; the memory stores a program that is loaded and executed by the processor to implement the audio noise reduction method as described in any one of claims 1 to 7.
9. A computer-readable storage medium, characterized in that, The storage medium stores a program that, when executed by a processor, is used to implement the audio noise reduction method as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Speech enhancement method and system
CN112581973A