Voice noise reduction method, model training method, device, apparatus, and medium

Abstract

A method for reducing audio noise, a model training method, an apparatus, a device, a medium, and a product. The method for reducing audio noise employs a preset voice activity detection algorithm to detect the current audio frame waiting for processing, and obtains a corresponding algorithm activity detection result [101]; performs an integration process on the model activity detection result corresponding to the previous audio frame and the algorithm activity detection result corresponding to the current audio frame, obtains a target activity detection result corresponding to the current audio frame, and the model activity detection result is output from a preset audio noise reduction network model [102]; based on the target activity detection result, performs noise estimation and noise removal on the current audio frame to obtain an initial noise reduction audio frame [103]; inputs the initial noise reduction audio frame into a preset audio noise reduction network model to output a target noise reduction audio frame and a model activity detection result corresponding to the current audio frame [104]. By adopting the above solution, the audio noise reduction effect can be improved, and the stability and robustness of the audio noise reduction solution can be improved.

Description

Technical Field

[0001] This application claims the priority of Chinese Patent Application No. 202210864010.4, filed with the State Intellectual Property Office of China on July 21, 2022, and all of its contents are incorporated herein by reference.

[0002] This application relates to the field of audio processing technologies, for example, to voice noise reduction methods, model training methods, devices, apparatuses, media, and products.

Background Art

[0003] With the rapid development of multimedia technologies, various conference, social, and entertainment applications have emerged one after another, among which many scenarios are related to voice calls, audio and video live broadcasts, multi-person conferences, etc. Voice quality has become an important indicator for judging the performance of an application.

[0004] The voice collected by the microphone of a terminal device usually has a certain degree of noise, and the clarity and quality of the voice can be improved by suppressing the noise contained in the voice through a voice noise reduction algorithm.

[0005] Currently, voice noise reduction solutions can mainly be divided into two types: traditional noise reduction solutions and artificial intelligence (AI) noise reduction solutions. The traditional noise reduction solution is to reduce voice noise by means of signal processing, and it cannot remove non-stationary noise, that is, its noise reduction ability for sudden noise is weak. The AI noise reduction solution has excellent noise reduction ability for both stationary noise and non-stationary noise. However, since this solution is a data-driven solution, it is highly dependent on training samples. In the process of model training, if there are scenarios not considered (for example, situations with a low signal-to-noise ratio), when encountering this scenario during actual application, it may cause an impossible-to-estimate signal output, and thus may cause the collapse of the system.

Summary of the Invention

[0006] Embodiments of the present application provide a voice noise reduction method, a model training method, an apparatus, a device, a medium, and a product, which can effectively combine a traditional noise reduction solution and an AI noise reduction solution to improve the voice noise reduction effect.

[0007] According to an aspect of the present application, a voice noise reduction method is provided, and the method includes: Adopting a preset voice activity detection algorithm to perform detection on the current audio frame waiting for processing to obtain a corresponding algorithm activity detection result; Performing integrated processing on the model activity detection result corresponding to the previous audio frame and the algorithm activity detection result corresponding to the current audio frame to obtain a target activity detection result corresponding to the current audio frame, where the model activity detection result is output from a preset voice noise reduction network model; Performing noise estimation and noise removal on the current audio frame based on the target activity detection result to obtain an initial noise reduction audio frame; Inputting the initial noise reduction audio frame into the preset voice noise reduction network model to output a target noise reduction audio frame and a model activity detection result corresponding to the current audio frame; and the like.

[0008] According to another aspect of the present application, a model training method is provided, and the method includes: Adopting a preset voice activity detection algorithm to perform detection on the current sample audio frame to obtain a corresponding sample algorithm activity detection result, where an activity detection label and a pure audio frame are associated with the current sample audio frame; Perform integrated processing on the sample model activity detection result corresponding to the previous sample audio frame and the sample algorithm activity detection result corresponding to the current sample audio frame to obtain the target sample activity detection result corresponding to the current sample audio frame, and the sample model activity detection result is output from the voice noise reduction network model, and Based on the target activity sample detection result, perform noise estimation and noise removal on the current sample audio frame to obtain an initial noise reduction sample audio frame, and By inputting the initial noise reduction sample audio frame into the voice noise reduction network model, output the target sample noise reduction audio frame and the sample model activity detection result corresponding to the current sample audio frame, and Based on the target sample noise reduction audio frame and the pure audio frame, determine a first loss relationship, and based on the sample model activity detection result and the activity detection label, determine a second loss relationship, and based on the first loss relationship and the second loss relationship, train the voice noise reduction network model, and include.

[0009] According to another aspect of the present application, a voice noise reduction device is provided, and the device includes a voice activity detection module, a detection result integration module, a noise reduction processing module, and a model input module. The voice activity detection module is installed to perform detection on the current audio frame waiting for processing by adopting a preset voice activity detection algorithm to obtain a corresponding algorithm activity detection result. The detection result integration module performs integration processing on the model activity detection result corresponding to the previous audio frame and the algorithm activity detection result corresponding to the current audio frame, and is installed to obtain the target activity detection result corresponding to the current audio frame. The model activity detection result is output from a preset voice noise reduction network model. The noise reduction processing module performs noise estimation and noise removal on the current audio frame based on the target activity detection result, and is installed to obtain an initial noise reduction audio frame. The model input module is installed to output a target noise reduction audio frame and the model activity detection result corresponding to the current audio frame by inputting the initial noise reduction audio frame into the preset voice noise reduction network model.

[0010] According to another aspect of the present application, a model training device is provided. The device includes a voice detection module, an integration module, a noise removal module, a network model input module, and a network model training module. The voice detection module is installed to perform detection on the current sample audio frame waiting for processing by adopting a preset voice activity detection algorithm, and obtain the corresponding sample algorithm activity detection result. The current sample audio frame is associated with an activity detection label and a clean audio frame. The integration module performs integration processing on the sample model activity detection result corresponding to the previous sample audio frame and the sample algorithm activity detection result corresponding to the current sample audio frame, and is installed to obtain the target sample activity detection result corresponding to the current sample audio frame. The sample model activity detection result is output from a voice noise reduction network model. The noise removal module is installed to perform noise estimation and noise removal on the current sample audio frame based on the target activity sample detection result, and obtain an initial noise reduction sample audio frame. The network model input module is installed to input the initial noise reduction sample audio frame into the audio noise reduction network model, and output a target sample noise reduction audio frame and a sample model activity detection result corresponding to the current sample audio frame. The network model training module is installed to determine a first loss relationship based on the target sample noise reduction audio frame and the clean audio frame, determine a second loss relationship based on the sample model activity detection result and the activity detection label, and perform training on the audio noise reduction network model based on the first loss relationship and the second loss relationship.

[0011] According to another aspect of the present application, an electronic device is provided. The electronic device includes at least one processor, a memory communicatively connected to the at least one processor, and includes The memory stores a computer program executable by the at least one processor. When the computer program is executed by the at least one processor, the at least one processor can execute the audio noise reduction method and / or the model training method described in any embodiment of the present application.

[0012] According to another aspect of the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and the computer program is used to implement the audio noise reduction method and / or the model training method described in any embodiment of the present application when executed by a processor.

[0013] According to another aspect of the present application, a computer program product is provided, the computer program product includes a computer program, and when the computer program is executed by a processor, the voice noise reduction method and / or the model training method described in any embodiment of the present application are realized.

[0014] The voice noise reduction solution provided by the embodiments of this application adopts a preset voice activity detection algorithm to detect the current audio frame waiting for processing, obtains the corresponding algorithm activity detection result, and performs integrated processing on the model activity detection result corresponding to the previous audio frame and the algorithm activity detection result corresponding to the current audio frame, so as to obtain the target activity detection result corresponding to the current audio frame. The model activity detection result is output from a preset voice noise reduction network model. Based on the target activity detection result, noise estimation and noise removal are performed on the current audio frame to obtain an initial noise reduction audio frame. By inputting the initial noise reduction audio frame into a preset voice noise reduction network model, a target noise reduction audio frame and the model activity detection result corresponding to the current audio frame are output. By adopting the above solution, the preset voice noise reduction network model can output the model activity detection result. When processing the current audio frame using the traditional voice noise reduction algorithm, the model activity detection result of the previous audio frame can be combined with the algorithm activity detection result obtained by the traditional voice noise reduction algorithm, enabling the traditional noise reduction algorithm to obtain more activity detection information and enabling the voice activity detection result to be determined more reasonably and accurately. Based on this result, noise estimation and noise removal are performed to better protect the voice, remove more noise, and obtain a traditional noise reduction result with a higher signal-to-noise ratio. Furthermore, the traditional noise reduction result is used as the input of a preset voice noise reduction network model to obtain a more effective noise reduction audio frame, reducing the possibility that the preset voice noise reduction network model processes poor data. The traditional noise reduction algorithm and the AI noise reduction method promote each other, have excellent noise reduction capabilities for various noises, improve the voice noise reduction effect, and enhance the stability and robustness of the overall voice noise reduction solution.

[0015] The following is an explanation of the drawings that need to be used in the description of the embodiments. The drawings described below are only some embodiments of the present application. It is obvious to those skilled in the art that other drawings can also be obtained based on these drawings without creative labor.

Brief Description of the Drawings

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Modes for Carrying Out the Invention

[0017] To enable those skilled in the art to better understand the solution of the present application, the following will combine the drawings in the embodiments of the present application to describe the embodiments of the present application. The described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the present application.

[0018] In addition, in the specification, claims, and the above-mentioned drawings of the present application, terms such as "first" and "second" do not necessarily indicate a specific order or sequence, but are used to distinguish similar objects. It should be understood that the data used in this way can be exchanged when appropriate so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "comprising", "having", and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to clearly including these steps or units that are clearly shown, but may include steps or units that are not clearly shown, or other steps or units inherent to these processes, methods, products, or devices.

[0019] FIG. 1 is a schematic flowchart of a voice noise reduction method provided by an embodiment of the present application. This embodiment can be applied when performing noise reduction on voice, and can be applied to various scenarios such as voice calls, audio-video live broadcasts, and multi-person conferences. The method can be executed by a voice noise reduction device, and the voice noise reduction device is realized in the form of hardware and / or software, and the voice noise reduction device can be arranged in an electronic device such as a voice noise reduction device. The electronic device may be a mobile device such as a mobile phone, a smart watch, a tablet, and a personal digital assistant, or may be other devices such as a desktop computer. As shown in FIG. 1, the method includes the following steps.

[0020] Step 101: Detect the current audio frame waiting for processing by adopting a preset voice activity detection algorithm, and obtain the corresponding algorithm activity detection result.

[0021] Exemplarily, the current audio frame waiting for processing can be understood as an audio frame that currently needs to undergo voice noise reduction processing, and the current audio frame can be included in an audio file or an audio stream. As one option, the current audio frame may be the original audio frame in the audio file or audio stream, or may be an audio frame obtained by performing pre-processing on the original audio frame.

[0022] In the embodiments of the present application, the entire voice noise reduction solution can be understood as a voice noise reduction system, and the current audio frame can be understood as the input signal of the voice noise reduction system. In the voice noise reduction solution, a traditional voice noise reduction algorithm and an AI voice noise reduction model can be included.

[0023] Here, the type of the traditional voice noise reduction algorithm may be, for example, an Adaptive Noise Suppression (ANS) algorithm in Web Real-Time Communication (webRTC), a linear filtering method, spectral subtraction, a statistical model algorithm, or a subspace algorithm, etc. In the traditional voice noise reduction algorithm, there are mainly three main parts: voice activity detection (VAD) estimation, noise estimation, and noise removal. Voice activity detection is also called voice endpoint detection or voice boundary detection, and can identify long silent periods from the voice signal stream. The preset voice activity detection algorithm in the embodiments of the present application may be the voice activity detection algorithm in any traditional voice noise reduction algorithm.

[0024] Here, the preset voice noise reduction network model in the present application may be an AI voice noise reduction model. For example, it can include the RNNoise model, or the Dual-Signal Transformation LSTM Network for Real-Time Noise Suppression (DTLN) noise reduction model, etc. The preset voice noise reduction network model includes two branches. One branch is for outputting noise-reduced voice (which can be abbreviated as the noise reduction branch), and the other branch is for outputting the voice activity detection result (which can be abbreviated as the detection branch). For the AI voice noise reduction model that already includes the detection branch, the original model structure can be maintained. For the AI voice noise reduction model that does not include the detection branch, the detection branch can be added based on the backbone network. The network structure of the detection branch can include, for example, a convolutional layer and / or a fully connected layer, etc.

[0025] Here, RNNoise is a noise reduction solution that adopts the combination of audio feature extraction and a deep neural network.

[0026] Exemplarily, in order to easily distinguish the voice activity detection results from different sources, after performing detection on the current audio frame waiting for processing by adopting a preset voice activity detection algorithm, the obtained detection result is recorded as the algorithm activity detection result, and the activity detection result output by the preset voice noise reduction network model can be recorded as the model activity detection result.

[0027] Step 102: Perform integration processing on the model activity detection result corresponding to the previous audio frame and the algorithm activity detection result corresponding to the current audio frame to obtain the target activity detection result corresponding to the current audio frame. Here, the model activity detection result is output from a preset voice noise reduction network model.

[0028] Exemplarily, the previous audio frame can be understood as the latest one audio frame before the current audio frame, that is, the previous audio frame is located before the current audio frame, and the frame sequence numbers of the previous audio frame and the current audio frame are adjacent. When performing voice noise reduction processing on the previous audio frame, the preset voice noise reduction network model can output the noise-reduced audio frame and the model activity detection result corresponding to the previous audio frame, and cache the model activity detection result for the noise reduction processing of the current audio frame.

[0029] In the embodiments of the present application, when processing the current audio frame, by integrating the model activity detection result corresponding to the previous audio frame and the algorithm activity detection result corresponding to the current audio frame, the activity detection result (target activity detection result) used for noise estimation and noise removal in the traditional voice noise reduction algorithm can be determined. Compared with simply adopting the traditional voice noise reduction algorithm for voice activity detection, the traditional noise reduction algorithm can be made to obtain more VAD information, thereby obtaining a more accurate noise estimation, better protecting the voice, and more accurately removing the noise, and improving the output signal-to-noise ratio (SNR) of the traditional noise reduction algorithm.

[0030] Step 103: Based on the target activity detection result, perform noise estimation and noise removal on the current audio frame to obtain an initial noise-reduced audio frame.

[0031] Exemplarily, after obtaining the target activity detection result, use the noise estimation algorithm and noise removal algorithm in the traditional audio noise reduction algorithm to perform corresponding processing on the current audio frame, and the audio frame obtained after processing can be recorded as the initial noise-reduced audio frame.

[0032] Step 104: By inputting the initial noise-reduced audio frame into the preset audio noise reduction network model, output the target noise-reduced audio frame and the model activity detection result corresponding to the current audio frame.

[0033] Exemplarily, after obtaining the initial noise-reduced audio frame, the initial noise-reduced audio frame can be directly used as the input to the preset audio noise reduction network model, or the initial noise-reduced audio frame can be converted based on the characteristics of the preset audio noise reduction network model. For example, it can be converted into a signal with a preset dimension, and the preset dimension can be, for example, the frequency domain, the time domain, or other dimensional domains.

[0034] The voice noise reduction method provided in the embodiments of this application adopts a preset voice activity detection algorithm to detect the current audio frame waiting to be processed, obtains the corresponding algorithm activity detection result, and performs integrated processing on the model activity detection result corresponding to the previous audio frame and the algorithm activity detection result corresponding to the current audio frame to obtain the target activity detection result corresponding to the current audio frame. The model activity detection result is output from a preset voice noise reduction network model. Based on the target activity detection result, noise estimation and noise removal are performed on the current audio frame to obtain an initial noise reduction audio frame. By inputting the initial noise reduction audio frame into a preset voice noise reduction network model, a target noise reduction audio frame and the model activity detection result corresponding to the current audio frame are output. By adopting the above solution, the preset voice noise reduction network model can output the model activity detection result. When processing the current audio frame using a traditional voice noise reduction algorithm, the model activity detection result of the previous audio frame and the algorithm activity detection result by the traditional voice noise reduction algorithm can be combined, enabling the traditional noise reduction algorithm to obtain more activity detection information, and enabling the voice activity detection result to be determined more reasonably and accurately. By performing noise estimation and noise removal based on the result, the voice can be better protected, more noise can be removed, and a traditional noise reduction result with a higher signal-to-noise ratio can be obtained. Furthermore, by using the traditional noise reduction result as the input to a preset voice noise reduction network model, a more effective noise reduction audio frame can be obtained, reducing the possibility that the preset voice noise reduction network model processes poor data. The traditional noise reduction algorithm and the AI noise reduction method promote each other, have excellent noise reduction capabilities for various noises, and enhance the stability and robustness of the overall solution.

[0035] In an embodiment of the present application, the voice activity detection may be at the frame level or at the frequency point level, and the detection result can be represented by one or more probability values.

[0036] In some embodiments, the algorithm activity detection result includes a first probability value indicating the presence of voice in the corresponding audio frame, and the model activity detection result includes a second probability value indicating the presence of voice in the corresponding audio frame. Here, for the model activity detection result corresponding to the previous audio frame and the algorithm activity detection result corresponding to the current audio frame, an integration process is performed to obtain the target activity detection result corresponding to the current audio frame, which includes adopting a preset calculation method to calculate the first probability value in the model activity detection result corresponding to the previous audio frame and the second probability value in the algorithm activity detection result corresponding to the current audio frame to obtain a third probability value, and determining the target activity detection result corresponding to the current audio frame based on the third probability value. By setting it in this way, for the frame-level voice activity detection, the target activity detection result can be accurately determined.

[0037] Here, the first probability value is used to represent the probability that the corresponding audio frame contains audio after detection is performed on the corresponding audio frame by adopting a preset voice activity detection algorithm. The corresponding audio frame here may be any audio frame, or the current audio frame, or the previous audio frame. The first probability values corresponding to different audio frames may be different. The second probability value is used to represent the probability that the corresponding audio frame contains audio output by a preset voice noise reduction network model. The corresponding audio frame here may be any audio frame, and the second probability values corresponding to different audio frames may be different.

[0038] Exemplarily, the first probability value in the algorithm activity detection result corresponding to the current audio frame can be used to represent the probability that the current audio frame contains audio after detection is performed on the current audio frame (recorded as A for example) by adopting a preset voice activity detection algorithm, and can be recorded as Pa. The second probability value in the model activity detection result corresponding to the previous audio frame can be used to represent the probability that the previous audio frame contains audio predicted by a preset voice noise reduction network model when voice noise reduction processing is performed on the previous audio frame (recorded as B for example), and can be recorded as Pb. Pa and Pb are calculated by adopting a preset calculation method to obtain a third probability value, which can be recorded as Pc. Exemplarily, the third probability value can be used as the target activity detection result corresponding to the current audio frame.

[0039] Exemplarily, the preset calculation method is any one of obtaining a maximum value, obtaining a minimum value, calculating an average value, addition, calculating a weighted sum, and calculating a weighted average value. Taking the obtaining of the maximum value as an example, Pc = max(Pa, Pb).

[0040] In some embodiments, the algorithm activity detection result includes, for each frequency point among a preset number of frequency points in the corresponding audio frame, a fourth probability value indicating the presence of voice, and the model activity detection result includes, for each frequency point among the preset number of frequency points in the corresponding audio frame, a fifth probability value indicating the presence of voice. Here, for the model activity detection result corresponding to the previous audio frame and the algorithm activity detection result corresponding to the current audio frame, performing an integration process to obtain the target activity detection result corresponding to the current audio frame includes, for each frequency point among the preset number of frequency points, adopting a preset calculation method to calculate the fifth probability value of a single frequency point in the model activity detection result corresponding to the previous audio frame and the fourth probability value of the corresponding single frequency point in the algorithm activity detection result corresponding to the current audio frame to obtain a sixth probability value, and determining the target activity detection result corresponding to the current audio frame based on the preset number of sixth probability values. By setting it in this way, voice activity detection at the frequency point level can be adopted to more accurately determine the target activity detection result.

[0041] Exemplarily, the preset number (recorded as n) can be set according to actual needs. For example, it is determined based on the number of points adopted in the fast Fourier transform during the preprocessing stage. For example, n is 256. The fourth probability value corresponding to the current audio frame is used to represent the probability that each frequency point among the preset number of frequency points in the current audio frame (recorded as A for example) contains audio after detecting the current audio frame (using a preset voice activity detection algorithm), and can be recorded as PA[n]. PA[n] can be understood as a vector containing n elements (n bits). The value of each element takes a value from 0 to 1, and the value of one element is used to represent the probability that the corresponding frequency point contains audio. The fifth probability value corresponding to the previous audio frame is used to represent the probability that each frequency point among the preset number of frequency points in the previous audio frame (recorded as B for example) contains audio predicted by the preset voice noise reduction network model when performing voice noise reduction processing on the previous audio frame, and can be recorded as PB[n]. The preset calculation method is adopted to calculate PA[n] and PB[n], and a preset number of sixth probability values are obtained, and can be recorded as PC[n] for example. Exemplarily, the vector containing the sixth probability value can be used as the target activity detection result corresponding to the current audio frame.

[0042] Exemplarily, the preset calculation method is any one of obtaining the maximum value, obtaining the minimum value, calculating the average value, addition, calculating the weighted sum, and calculating the weighted average value. Taking the obtaining of the maximum value as an example, PC[n]=max(PA[n], PB[n]). For example, for the first frequency point in the current audio frame, the maximum value among the corresponding fourth probability value and the fifth probability value is the sixth probability value corresponding to the first frequency point in the current audio frame, and the same applies to subsequent frequency points.

[0043] In some embodiments, inputting the initial noise-reduced audio frame into the preset voice noise reduction network model includes performing feature extraction of preset feature dimensions on the initial noise-reduced audio frame to obtain a target input signal, and inputting the target input signal into the preset voice noise reduction network model, or inputting the target input signal and the initial noise-reduced audio frame into the preset voice noise reduction network model. By setting it in this way, purposeful feature extraction can be performed, and the prediction accuracy and precision of the preset voice noise reduction network model can be improved.

[0044] As one option, the preset feature dimension may include a dominant feature dimension, which may be a fundamental frequency feature, for example, a pitch frequency, or a per-channel energy normalization (PCEN) feature for each channel, or a Mel Frequency Cepstrum Coefficient (MFCC) feature, etc. The preset feature dimension can be determined based on the network structure or characteristics of the preset voice noise reduction network model.

[0045] Figure 2 is a schematic flowchart of another voice noise reduction method provided by the embodiments of the present application. The method is optimized based on each of the above selectable embodiments. Figure 3 is a schematic flowchart of the inference process of the voice noise reduction method provided by the embodiments of the present application. In combination with Figures 2 and 3, the embodiment solutions of the present application can be understood. Here, as shown in Figure 2, the method can include the following steps.

[0046] Step 201: Obtain an original audio frame, perform preprocessing on the original audio frame, and obtain a current audio frame waiting for processing.

[0047] Exemplarily, the original audio frame is included in an audio file or an audio stream. For example, it may be an audio stream in a voice call scene. In order to ensure call quality, it is necessary to perform noise reduction on the call audio. The preprocessing can include processes such as framing, window processing, and Fourier transform. The noisy voice frame after preprocessing is the current audio frame waiting to be processed, and is used as the input signal (recorded as S0) of a preset traditional noise reduction algorithm.

[0048] Step 202, adopt a preset voice activity detection algorithm among the preset traditional noise reduction algorithms, perform detection on the current audio frame waiting to be processed, and obtain the corresponding algorithm activity detection result.

[0049] Exemplarily, the preset traditional noise reduction algorithm may be the ANS algorithm. Using the preset voice activity detection algorithm corresponding to the VAD estimation function module in the ANS algorithm, perform detection on S0. Assuming detection at the frequency point level, the voice presence probability Pf

[0256] of 256 frequency points can be obtained, that is, the algorithm activity detection result corresponding to S0 can be obtained.

[0050] Step 203, determine whether there is a previous audio frame for the current audio frame. If there is, execute Step 204; if not, execute Step 206.

[0051] Exemplarily, for the first audio frame, since there is no previous audio frame, it is not necessary to obtain the model activity detection result of the previous audio frame. Execute Step 206, and noise estimation and noise removal can be performed based on the algorithm activity detection result corresponding to the current audio frame.

[0052] In step 204, obtain the model activity detection result corresponding to the previous audio frame, perform integration processing on the obtained model activity detection result and the algorithm activity detection result corresponding to the current audio frame, and obtain the target activity detection result corresponding to the current audio frame.

[0053] Exemplarily, the model activity detection result corresponding to the previous audio frame may be output from a preset voice noise reduction network model based on artificial intelligence, and may be the voice presence probability PF

[0256] of 256 frequency points in the previous audio frame. The integrated VAD estimation result (target activity detection result): P

[0256] =max(Pf

[0256] , PF

[0256] ) can be obtained by adopting the method of taking the maximum value.

[0054] In step 205, based on the target activity detection result, use the preset traditional noise reduction algorithm to perform noise estimation and noise removal on the current audio frame to obtain an initial noise reduction audio frame, and execute step 207.

[0055] Exemplarily, the preset traditional noise reduction algorithm realizes noise estimation and noise removal based on P

[0256] , and obtains an audio signal S1 after traditional noise reduction processing, that is, an initial noise reduction audio frame.

[0056] In step 206, based on the algorithm activity detection result corresponding to the current audio frame, use the preset traditional noise reduction algorithm to perform noise estimation and noise removal on the current audio frame to obtain an initial noise reduction audio frame.

[0057] Exemplarily, the preset traditional noise reduction algorithm realizes noise estimation and noise removal based on Pf

[0256] , and obtains an audio signal S1 after traditional noise reduction processing, that is, an initial noise reduction audio frame.

[0058] Step 207: Perform feature extraction of preset feature dimensions on the initial noise-reduced voice to obtain a target input signal.

[0059] Exemplarily, S1 may be a signal in the frequency domain, time domain, or other dimensional domains as the input signal of a preset voice noise reduction network model. Based on the differences in the model design of the preset voice noise reduction network model, there may be an extraction calculation of the dominant feature (e.g., fundamental frequency feature) in one step, and the extracted feature information is recorded as the target input signal S2.

[0060] Step 208: Input the target input signal and / or the initial noise-reduced audio frame into a preset voice noise reduction network model to output a target noise-reduced audio frame and a model activity detection result corresponding to the current audio frame.

[0061] As one option, S1 or S2 may be used as the input to the model, or both S1 and S2 may be used as the input to the model. Input them into the preset voice noise reduction network model for inference calculation to obtain an output signal. The output signal includes two parts. The first part is the output S3 of the final noise-reduced voice of the voice noise reduction method, and the second part is the VAD output PF

[0256] of the model, which is used when the traditional voice noise reduction algorithm processes the next audio frame.

[0062] Step 209: Determine whether there is an original audio frame waiting to be processed. If so, return to the execution of Step 201; if not, end the process.

[0063] Exemplarily, when a voice call ends, if all original audio frames have already been subjected to noise reduction processing at this point, the process can be terminated. If there are original audio frames that have not been noise-reduced, the process returns to the execution of step 201, and the noise reduction processing can continue.

[0064] The voice noise reduction method provided in the embodiments of the present application is such that a preset voice noise reduction network model based on artificial intelligence feeds back information to a traditional noise reduction algorithm, enabling the traditional noise reduction algorithm to obtain more VAD information. Both the VAD estimation of traditional noise reduction and AI noise reduction adopt the frequency point level, enabling more accurate noise estimation. The traditional noise reduction algorithm can better protect the voice, remove more noise, improve the output signal-to-noise ratio of traditional noise reduction. After the initial noise-reduced voice signal with a high signal-to-noise ratio undergoes feature extraction, it enriches the input of the preset voice noise reduction network model, reduces the possibility of the preset voice noise reduction network model processing poor data, improves the voice noise reduction effect of the model, and can improve the voice noise reduction performance.

[0065] FIG. 4 is a schematic flowchart of the model training method provided by an embodiment of the present application, and FIG. 5 is a schematic diagram of the training process of the model training method provided by an embodiment of the present application. In combination with FIGS. 4 and 5, the embodiments of the present application can be understood. This embodiment can be applied when training an artificial intelligence-based voice noise reduction network model, and the model can be applied to various scenarios such as voice calls, audio and video live broadcasts, and multi-person conferences. The method can be executed by a model training device, and the device can be implemented in the form of hardware and / or software, and the device can be arranged in an electronic device such as a model training device. The electronic device may be a mobile device such as a mobile phone, a smart watch, a tablet, and a personal digital assistant, or may be other devices such as a desktop computer. The voice noise reduction network model trained by adopting the embodiment of the present application can be applied to the voice noise reduction method provided by any embodiment in the present application.

[0066] As shown in FIG. 4, the method includes the following steps.

[0067] Step 401: Detect the current sample audio frame by adopting a preset voice activity detection algorithm, and obtain a corresponding sample algorithm activity detection result. Here, an activity detection label and a pure audio frame are associated with the current sample audio frame.

[0068] Exemplarily, a pure (clean) audio data set and a noise data set can be mixed into noisy audio data according to a preset mixing rule, and the preset mixing rule can be set based on, for example, a signal-to-noise ratio or a room impulse response (RIR). As one option, the mixed noisy audio data set and the pure audio data set are taken together as the training set of the model. The current sample audio frame may be an audio frame in the training set. The current sample audio frame can have an activity detection label, and the label can be added by a method of manual markup. Taking the frame level as an example, when audio is included, the label can be set to 1, and when no audio is included, the label can be set to 0. Taking the frequency point level as an example, the label may be a vector containing a preset number of elements, the value of each element is 1 or 0, when the corresponding frequency point contains audio, the value is set to 1, and when the corresponding frequency point does not contain audio, the value is set to 0.

[0069] Step 402, perform an integration process on the sample model activity detection result corresponding to the previous sample audio frame and the sample algorithm activity detection result corresponding to the current sample audio frame to obtain the target sample activity detection result corresponding to the current sample audio frame. Here, the sample model activity detection result is output from the audio noise reduction network model.

[0070] Exemplarily, the integration process of the activity detection result in this step can be similar to the integration process in the audio noise reduction method provided in the embodiments of the present application. For example, it can be an integration at the frequency point level or an integration at the frame level, etc., and a similar preset calculation method can be adopted to perform integration on the corresponding frequency values. Specific details can refer to the relevant content of the present application, and detailed description is omitted here.

[0071] Step 403: Based on the target activity sample detection result, perform noise estimation and noise removal on the current sample audio frame to obtain an initial noise reduction sample audio frame.

[0072] Step 404: By inputting the initial noise reduction sample audio frame into the audio noise reduction network model, output a target sample noise reduction audio frame and a sample model activity detection result corresponding to the current sample audio frame.

[0073] Step 405: Based on the target sample noise reduction audio frame and the pure audio frame, determine a first loss relationship, based on the sample model activity detection result and the activity detection label, determine a second loss relationship, and based on the first loss relationship and the second loss relationship, train the audio noise reduction network model.

[0074] Exemplarily, the loss relationship can be used to characterize the difference between two types of data, can be represented by a loss value, for example, can be calculated by adopting a loss function. The first loss relationship is used to characterize the difference between the target sample noise reduction audio frame and the pure audio frame, and the second loss relationship is used to characterize the difference between the sample model activity detection result and the activity detection label. Here, the function types of the first loss function for calculating the first loss relationship and the second loss function for calculating the second loss relationship can be set according to actual needs.

[0075] Exemplarily, based on the first loss relationship and the second loss relationship, a target loss relationship can be calculated, and the calculation method can be, for example, weighted sum, etc.

[0076] Exemplarily, based on the target loss relationship, training is performed on the voice noise reduction network model. During the training process, the goal is to minimize the target loss relationship, and until the preset training stop condition is met, training means such as backpropagation can be used to optimize the weight parameter values in the voice noise reduction network model. The training stop condition can be set according to actual needs and can be set, for example, based on the number of iterations, the degree of convergence of the loss value, or the model accuracy rate, etc.

[0077] The model training method provided in the embodiments of the present application, during the training process, takes the traditional noise reduction algorithm and the voice noise reduction network model as a whole, and can avoid the risk of data mismatch by connecting the traditional noise reduction algorithm in series to the voice noise reduction network model that is trained independently. The model obtained after training is used for voice noise reduction, has excellent noise reduction ability for various noises, and improves the noise reduction effect.

[0078] As one of the options, the sample algorithm activity detection result includes a first sample probability value indicating the presence of voice in the corresponding sample audio frame, and the sample model activity detection result includes a second sample probability value indicating the presence of voice in the corresponding sample audio frame. Here, for the sample model activity detection result corresponding to the previous sample audio frame and the sample algorithm activity detection result corresponding to the current sample audio frame, performing an integration process to obtain the target sample activity detection result corresponding to the current sample audio frame includes adopting a preset calculation method to calculate the second sample probability value in the sample model activity detection result corresponding to the previous sample audio frame and the first sample probability value in the sample algorithm activity detection result corresponding to the current sample audio frame, obtaining a third sample probability value, and determining the target sample activity detection result corresponding to the current sample audio frame based on the third sample probability value.

[0079] As one option, the sample algorithm activity detection result includes, for each frequency point among a preset number of frequency points in the corresponding audio frame, the fourth sample probability value indicating the presence of audio, and the model activity detection result includes, for each frequency point among the preset number of frequency points in the corresponding audio frame, the fifth sample probability value indicating the presence of audio. Here, performing integration processing on the sample model activity detection result corresponding to the previous sample audio frame and the sample algorithm activity detection result corresponding to the current sample audio frame to obtain the target sample activity detection result corresponding to the current sample audio frame means that for each frequency point among the preset number of frequency points, adopting a preset calculation method to calculate the fifth sample probability value of a single frequency point in the sample model activity detection result corresponding to the previous sample audio frame and the fourth sample probability value of the corresponding single frequency point in the sample algorithm activity detection result corresponding to the current sample audio frame to obtain a sixth sample probability value, and determining the target sample activity detection result corresponding to the current sample audio frame based on the preset number of sixth sample probability values.

[0080] As an option, inputting the initial noise reduction sample audio frame into the voice noise reduction network model includes performing feature extraction of a preset feature dimension on the initial noise reduction sample audio frame to obtain a target input signal, and inputting the target input signal into the voice noise reduction network model, or inputting the target input signal and the initial noise reduction sample audio frame into the voice noise reduction network model.

[0081] FIG. 6 is a structural block diagram of a voice noise reduction device provided in an embodiment of the present application. The device can be realized by software and / or hardware, and can generally be integrated into an electronic device such as a voice noise reduction device, and can perform voice noise reduction by executing a voice noise reduction method. As shown in FIG. 6, the device includes a voice activity detection module 601, a detection result integration module 602, a noise reduction processing module 603, and a model input module 604. The voice activity detection module 601 is installed to perform detection on the current audio frame waiting for processing by adopting a preset voice activity detection algorithm, and obtain the corresponding algorithm activity detection result. The detection result integration module 602 is installed to perform integration processing on the model activity detection result corresponding to the previous audio frame and the algorithm activity detection result corresponding to the current audio frame, and obtain the target activity detection result corresponding to the current audio frame. The model activity detection result is output from a preset voice noise reduction network model. The noise reduction processing module 603 is installed to perform noise estimation and noise removal on the current audio frame based on the target activity detection result, and obtain an initial noise reduction audio frame. The model input module 604 is installed to input the initial noise reduction audio frame into the preset voice noise reduction network model, and output the target noise reduction audio frame and the model activity detection result corresponding to the current audio frame.

[0082] The voice noise reduction device provided in the embodiment of the present application adopts a preset voice activity detection algorithm to detect the current audio frame waiting for processing, obtains the corresponding algorithm activity detection result, and performs integrated processing on the model activity detection result corresponding to the previous audio frame and the algorithm activity detection result corresponding to the current audio frame to obtain the target activity detection result corresponding to the current audio frame. The model activity detection result is output from a preset voice noise reduction network model. Based on the target activity detection result, noise estimation and noise removal are performed on the current audio frame to obtain an initial noise reduction audio frame. By inputting the initial noise reduction audio frame into a preset voice noise reduction network model, a target noise reduction audio frame and the model activity detection result corresponding to the current audio frame are output. By adopting the above solution, the preset voice noise reduction network model can output the model activity detection result. When processing the current audio frame by adopting the traditional voice noise reduction algorithm, the model activity detection result of the previous audio frame can be combined with the algorithm activity detection result by the traditional voice noise reduction algorithm, so that the traditional noise reduction algorithm can obtain more activity detection information, and the voice activity detection result can be determined more reasonably and accurately. By performing noise estimation and noise removal based on the result, the voice can be better protected, more noise can be removed, and a traditional noise reduction result with a higher signal-to-noise ratio can be obtained. Furthermore, by using the traditional noise reduction result as the input of a preset voice noise reduction network model, a more effective noise reduction audio frame can be obtained, the possibility that the preset voice noise reduction network model processes poor data can be reduced, the traditional noise reduction algorithm and the AI noise reduction method promote each other, have excellent noise reduction capabilities for various noises, and enhance the stability and robustness of the overall solution.

[0083] As one option, the algorithm activity detection result includes a first probability value indicating the presence of voice in the corresponding audio frame, and the model activity detection result includes a second probability value indicating the presence of voice in the corresponding audio frame. Here, the detection result integration module 602 is configured to perform integration processing on the model activity detection result and the algorithm activity detection result in the following manner to obtain a target activity detection result corresponding to the current audio frame. Adopt a preset calculation method to calculate the second probability value in the model activity detection result corresponding to the previous audio frame and the first probability value in the algorithm activity detection result corresponding to the current audio frame to obtain a third probability value, and determine the target activity detection result corresponding to the current audio frame based on the third probability value.

[0084] As one option, the algorithm activity detection result includes a fourth probability value indicating the presence of voice at each of a preset number of frequency points in the corresponding audio frame, and the model activity detection result includes a fifth probability value indicating the presence of voice at each of the preset number of frequency points in the corresponding audio frame. Here, the detection result integration module 602 is further configured to perform integration processing on the model activity detection result and the algorithm activity detection result in the following manner to obtain a target activity detection result corresponding to the current audio frame. For each of the preset number of frequency points, adopting a preset calculation method, calculate the fifth probability value of a single frequency point in the model activity detection result corresponding to the previous audio frame and the fourth probability value of the corresponding single frequency point in the algorithm activity detection result corresponding to the current audio frame, obtain a sixth probability value, and determine the target activity detection result corresponding to the current audio frame based on the preset number of sixth probability values.

[0085] As one option, the preset calculation method is any one of obtaining a maximum value, obtaining a minimum value, calculating an average value, addition, calculating a weighted sum, and calculating a weighted average value.

[0086] As one option, the model input module includes a feature extraction unit and a signal input unit. The feature extraction unit performs feature extraction of a preset feature dimension on the initial noise-reduced voice and is arranged to obtain a target input signal. The signal input unit inputs the target input signal into the preset voice noise reduction network model, or inputs the target input signal and the initial noise-reduced audio frame into the preset voice noise reduction network model, so as to output a target noise-reduced audio frame and a model activity detection result corresponding to the current audio frame.

[0087] FIG. 7 is a structural block diagram of a model training device provided by an embodiment of the present application. The device can be realized by software and / or hardware, and can generally be integrated into an electronic device such as a model training device, and can perform model training by executing a model training method. As shown in FIG. 7, the device includes a voice detection module 701, an integration module 702, a noise removal module 703, a network model input module 704, and a network model training module 705. The voice detection module 701 is installed to perform detection on the current sample audio frame waiting for processing by adopting a preset voice activity detection algorithm, and obtain the corresponding sample algorithm activity detection result. The current sample audio frame is associated with an activity detection label and a clean audio frame. The integration module 702 is installed to perform integration processing on the sample model activity detection result corresponding to the previous sample audio frame and the sample algorithm activity detection result corresponding to the current sample audio frame, and obtain the target sample activity detection result corresponding to the current sample audio frame. The sample model activity detection result is output from the voice noise reduction network model. The noise removal module 703 is installed to perform noise estimation and noise removal on the current sample audio frame based on the target activity sample detection result, and obtain the initial noise reduction sample audio frame. The network model input module 704 is installed to input the initial noise reduction sample audio frame into the voice noise reduction network model, and output the target sample noise reduction audio frame and the sample model activity detection result corresponding to the current sample audio frame. The network model training module 705 is installed to determine a first loss relationship based on the target sample noise reduction audio frame and the clean audio frame, determine a second loss relationship based on the sample model activity detection result and the activity detection label, and perform training on the voice noise reduction network model based on the first loss relationship and the second loss relationship.

[0088] The model training device provided in the embodiments of the present application can, during the training process, take the traditional noise reduction algorithm and the voice noise reduction network model as a whole, and avoid the risk of data mismatch by connecting the traditional noise reduction algorithm in series to the voice noise reduction network model trained independently. The model obtained after training is used for voice noise reduction, has excellent noise reduction ability for various noises, and improves the noise reduction effect.

[0089] Embodiments of the present application provide an electronic device, in which the voice noise reduction device and / or the model training device provided in the embodiments of the present application can be integrated. FIG. 8 is a structural block diagram of the electronic device provided in the embodiments of the present application. The electronic device 800 includes a processor 801 and a memory 802 communicatively connected to the processor 801. The memory 802 stores a computer program executable by the processor 801. When the computer program is executed by the processor 801, the processor 801 can execute the voice noise reduction method and / or the model training method described in any embodiment of the present application. Here, the number of processors may be one or more. In FIG. 8, one processor is taken as an example.

[0090] Embodiments of the present application further provide a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the voice noise reduction method and / or the model training method described in any embodiment of the present application.

[0091] Embodiments of the present application further provide a computer program product, which includes a computer program. When the computer program is executed by a processor, for example, it implements the voice noise reduction method and / or the model training method provided in the embodiments of the present application.

[0092] The voice noise reduction device, model training device, electronic device, storage medium, and product provided in the above embodiments can execute the voice noise reduction method or model training method provided in the corresponding embodiments of this application, and have the corresponding functional modules and beneficial effects for executing the method. For technical details not specifically described in the above embodiments, reference can be made to the voice noise reduction method or model training method provided in any embodiment of this application.

Claims

1. Detect the current audio frame waiting for processing by adopting a preset voice activity detection algorithm, and obtain the corresponding algorithm activity detection result; Perform integrated processing on the model activity detection result corresponding to the previous audio frame and the algorithm activity detection result corresponding to the current audio frame to obtain the target activity detection result corresponding to the current audio frame, and the model activity detection result is output from a preset voice noise reduction network model; Perform noise estimation and noise removal on the current audio frame based on the target activity detection result to obtain an initial noise reduction audio frame; Input the initial noise reduction audio frame into the preset voice noise reduction network model to output the target noise reduction audio frame and the model activity detection result corresponding to the current audio frame; including A method for reducing audio noise.

2. The algorithm activity detection result includes a first probability value indicating the presence of voice in the corresponding audio frame, and the model activity detection result includes a second probability value indicating the presence of voice in the corresponding audio frame. Performing integrated processing on the model activity detection result corresponding to the previous audio frame and the algorithm activity detection result corresponding to the current audio frame to obtain the target activity detection result corresponding to the current audio frame includes Adopting a preset calculation method to calculate the second probability value in the model activity detection result corresponding to the previous audio frame and the first probability value in the algorithm activity detection result corresponding to the current audio frame to obtain a third probability value, and determining the target activity detection result corresponding to the current audio frame based on the third probability value. The method according to claim 1.

3. The algorithm activity detection result includes a fourth probability value indicating the presence of voice at each of a preset number of frequency points in the corresponding audio frame, and the model activity detection result includes a fifth probability value indicating the presence of voice at each of the preset number of frequency points in the corresponding audio frame. Performing integration processing on the model activity detection result corresponding to the previous audio frame and the algorithm activity detection result corresponding to the current audio frame to obtain a target activity detection result corresponding to the current audio frame is For each of the preset number of frequency points, adopting a preset calculation method to calculate the fifth probability value of a single frequency point in the model activity detection result corresponding to the previous audio frame and the corresponding fourth probability value of the single frequency point in the algorithm activity detection result corresponding to the current audio frame, and obtaining a sixth probability value. Determining a target activity detection result corresponding to the current audio frame based on the preset number of sixth probability values. including The method according to claim 1.

4. The preset calculation method is any one of obtaining a maximum value, obtaining a minimum value, calculating an average value, addition, calculating a weighted sum, and calculating a weighted average value. The method according to claim 2 or 3.

5. Inputting the initial noise-reduced audio frame into the preset voice noise reduction network model is Performing feature extraction of a preset feature dimension on the initial noise-reduced audio frame to obtain a target input signal. Inputting the target input signal into the preset voice noise reduction network model, or inputting the target input signal and the initial noise-reduced audio frame into the preset voice noise reduction network model. including The method according to claim 1.

6. Adopt a pre-set voice activity detection algorithm to perform detection on the current sample audio frame, obtain the corresponding sample algorithm activity detection result, and in the current sample audio frame, an activity detection label and a pure audio frame are associated, and Integrate the sample model activity detection result corresponding to the previous sample audio frame and the sample algorithm activity detection result corresponding to the current sample audio frame to obtain the target sample activity detection result corresponding to the current sample audio frame, and the sample model activity detection result is output from the voice noise reduction network model, and Based on the target activity sample detection result, perform noise estimation and noise removal on the current sample audio frame to obtain an initial noise reduction sample audio frame, and By inputting the initial noise reduction sample audio frame into the voice noise reduction network model, output the target sample noise reduction audio frame and the sample model activity detection result corresponding to the current sample audio frame, and Based on the target sample noise reduction audio frame and the pure audio frame, determine a first loss relationship, and based on the sample model activity detection result and the activity detection label, determine a second loss relationship, and based on the first loss relationship and the second loss relationship, train the voice noise reduction network model, and including A model training method.

7. Including a voice activity detection module, a detection result integration module, a noise reduction processing module, and a model input module, The voice activity detection module is installed to perform detection on the current audio frame waiting for processing by adopting a pre-set voice activity detection algorithm and obtain the corresponding algorithm activity detection result. The detection result integration module is configured to perform integration processing on the model activity detection result corresponding to the previous audio frame and the algorithm activity detection result corresponding to the current audio frame, so as to obtain the target activity detection result corresponding to the current audio frame. The model activity detection result is output from a preset voice noise reduction network model. The noise reduction processing module is configured to perform noise estimation and noise removal on the current audio frame based on the target activity detection result, so as to obtain an initial noise reduction audio frame. The model input module is configured to input the initial noise reduction audio frame into the preset voice noise reduction network model, so as to output a target noise reduction audio frame and the model activity detection result corresponding to the current audio frame. Voice noise reduction device.

8. It includes a voice detection module, an integration module, a noise removal module, a network model input module, and a network model training module. The voice detection module is configured to perform detection on the current sample audio frame waiting for processing by adopting a preset voice activity detection algorithm, so as to obtain the corresponding sample algorithm activity detection result. The current sample audio frame is associated with an activity detection label and a clean audio frame. The integration module is configured to perform integration processing on the sample model activity detection result corresponding to the previous sample audio frame and the sample algorithm activity detection result corresponding to the current sample audio frame, so as to obtain the target sample activity detection result corresponding to the current sample audio frame. The sample model activity detection result is output from a voice noise reduction network model. The noise removal module is configured to perform noise estimation and noise removal on the current sample audio frame based on the target activity sample detection result, so as to obtain an initial noise reduction sample audio frame. The network model input module is installed to input the initial noise reduction sample audio frame into the audio noise reduction network model, so as to output a target sample noise reduction audio frame and a sample model activity detection result corresponding to the current sample audio frame. The network model training module is installed to determine a first loss relationship based on the target sample noise reduction audio frame and the clean audio frame, determine a second loss relationship based on the sample model activity detection result and the activity detection label, and perform training on the audio noise reduction network model based on the first loss relationship and the second loss relationship. Model training device.

9. At least one processor; A memory communicatively connected to the at least one processor; Comprising: The memory stores a computer program executable by the at least one processor. When the computer program is executed by the at least one processor, the at least one processor can execute the audio noise reduction method according to any one of claims 1 to 5 and / or the model training method according to claim 6. Electronic device.

10. A computer program is stored. When the computer program is executed by a processor, it is used to implement the audio noise reduction method according to any one of claims 1 to 5 and / or the model training method according to claim 6. Computer-readable storage medium.

11. Including a computer program. When the computer program is executed by a processor, it realizes the audio noise reduction method according to any one of claims 1 to 5 and / or the model training method according to claim 6. Computer program product.

Images