Audio noise reduction method and device, storage medium and electronic device

Abstract

The embodiment of the application provides an audio noise reduction method and device, a storage medium and an electronic device, the method comprises the following steps: obtaining first noisy audio data at a current moment and second noisy audio data at a historical moment; performing feature extraction according to the first noisy audio data and the second noisy audio data to obtain target feature information; inputting the target feature information into a target neural network structure to obtain a target audio signal corresponding to the first noisy audio data. Through the application, the problem that the related art trains a noise reduction model by using a pure audio signal, resulting in relatively low accuracy of noise reduction of the noise reduction model, is solved.

Description

Technical Field

[0001] The present invention relates to the field of audio noise reduction technology, and more specifically, to an audio noise reduction method and apparatus, a storage medium and an electronic device. Background Technology

[0002] In real life, when people use mobile phones to make hands-free calls or video conferencing terminals to conduct video conferences, due to the randomness of the environment, the device will collect various environmental noises along with the target person's voice. These noises can seriously affect the acquisition of the target person's voice. Therefore, audio noise reduction is needed to suppress various environmental noises. Generally speaking, noise is divided into steady-state noise and non-steady-state noise. For steady-state noise, there are many classic noise reduction algorithms that can process it, such as Wiener filtering. However, for non-steady-state noise, classic noise reduction algorithms cannot process it. With the rise of neural networks, non-steady-state noise can be effectively removed through this method due to the powerful nonlinear fitting ability of neural networks. However, existing neural networks use ideal voice data as clean data for training, resulting in relatively poor audio noise reduction performance.

[0003] There is currently no effective solution to the problem that training a noise reduction model with a clean audio signal results in low noise reduction accuracy in related technologies. Summary of the Invention

[0004] This invention provides an audio noise reduction method, apparatus, storage medium, and electronic device to at least solve the problem in related technologies where training a noise reduction model with a clean audio signal results in low noise reduction accuracy.

[0005] According to an embodiment of the present invention, an audio noise reduction method is provided, comprising: acquiring first noisy frequency data at the current moment and second noisy frequency data at a historical moment; performing feature extraction based on the first noisy frequency data and the second noisy frequency data to obtain target feature information; inputting the target feature information into a target neural network structure to obtain a target audio signal corresponding to the first noisy frequency data, wherein the target neural network structure is trained by a noisy frequency training sample set, the noisy frequency training sample set is composed of multiple noisy frequency training samples and a preset audio signal corresponding to each noisy frequency training sample, and each noisy frequency training sample is obtained through the preset audio signal and the target noise signal.

[0006] In an exemplary embodiment, the target neural network structure comprises an encoding module, a timing propagation module, a cross-layer connection module, and a decoding module. The cross-layer connection module is used to connect the convolutional layer in the encoding module and the transposed convolutional layer in the decoding module. The convolutional kernel in the cross-layer connection module is determined by the number of layers of the data output by the encoding module.

[0007] In an exemplary embodiment, feature extraction based on the first noisy frequency data and the second noisy frequency data to obtain target feature information includes: performing a short-time Fourier transform on each frame of the first noisy frequency data to obtain first time-frequency domain information corresponding to the first noisy frequency data; performing a short-time Fourier transform on each frame of the second noisy frequency data to obtain second time-frequency domain information corresponding to the second noisy frequency data; and concatenating the first time-frequency domain information and the second time-frequency domain information to obtain the target feature information.

[0008] In an exemplary embodiment, inputting the target feature information into a target neural network structure to obtain the target audio signal corresponding to the first noisy frequency data includes: encoding the target feature information through the encoding module to obtain a target feature matrix; passing the target feature matrix to the decoding module through the timing transfer module; passing the output feature matrix of each convolutional layer in the encoding module to each transposed convolutional layer of the decoding module through the cross-layer connection module; and decoding the target feature matrix and the output feature matrix through the decoding module to obtain the target audio signal corresponding to the first noisy frequency data.

[0009] In an exemplary embodiment, the target neural network structure is trained using the following steps: obtaining the training sample set with noise; extracting features from the training samples with noise in the training sample set with noise to obtain a training feature set; and training the initial target neural network structure based on the training feature set to obtain the target neural network structure.

[0010] In one exemplary embodiment, obtaining the noisy frequency training sample set includes: obtaining the preset audio signal; obtaining the target noise signal; mixing and constructing the preset audio signal and the target noise signal to obtain the noisy frequency training samples; and determining the noisy frequency training sample set based on the noisy frequency training samples and the preset audio signal.

[0011] In an exemplary embodiment, acquiring the preset audio signal includes: collecting third band noise frequency data in a target environment, and performing noise reduction processing on the third band noise frequency data to obtain processed third band noise frequency data; scoring the processed third band noise frequency data to obtain a first target score; if the first target score is greater than a first threshold, then determining the processed third band noise frequency data as the preset audio signal.

[0012] In an exemplary embodiment, acquiring the target noise signal includes: collecting initial noise data in a target environment; scoring the initial noise data to obtain a second target score; and determining the initial noise data as the target noise signal if the second target score is greater than a second threshold.

[0013] In an exemplary embodiment, training an initial target neural network structure based on the training feature set to obtain the target neural network structure includes: inputting the training feature set into the initial target neural network structure to obtain a predicted audio signal corresponding to each training sample with noise; calculating a loss function based on the predicted audio signal and the preset audio signal to obtain a target loss function; and training the initial target neural network structure based on the target loss function to obtain the target neural network structure.

[0014] According to another embodiment of the present invention, an audio noise reduction device is provided, comprising: a first acquisition unit, configured to acquire first noisy audio data at the current time and second noisy audio data at a historical time; a first extraction unit, configured to perform feature extraction based on the first noisy audio data and the second noisy audio data to obtain target feature information; and an input unit, configured to input the target feature information into a target neural network structure to obtain a target audio signal corresponding to the first noisy audio data, wherein the target neural network structure is trained by a noisy audio training sample set, the noisy audio training sample set consists of multiple noisy audio training samples and a preset audio signal corresponding to each noisy audio training sample, and each noisy audio training sample is obtained through the preset audio signal and the target noise signal.

[0015] In an exemplary embodiment, the target neural network structure comprises an encoding module, a timing propagation module, a cross-layer connection module, and a decoding module. The cross-layer connection module is used to connect the convolutional layer in the encoding module and the transposed convolutional layer in the decoding module. The convolutional kernel in the cross-layer connection module is determined by the number of layers of the data output by the encoding module.

[0016] In an exemplary embodiment, the first extraction unit includes: a first transformation module, configured to perform a short-time Fourier transform on each frame of the first noisy frequency data to obtain first time-frequency domain information corresponding to the first noisy frequency data; a second transformation module, configured to perform a short-time Fourier transform on each frame of the second noisy frequency data to obtain second time-frequency domain information corresponding to the second noisy frequency data; and a splicing module, configured to splice the first time-frequency domain information and the second time-frequency domain information to obtain the target feature information.

[0017] In an exemplary embodiment, the input unit includes: an encoding module for encoding the target feature information to obtain a target feature matrix; a first transmission module for transmitting the target feature matrix to the decoding module via the timing transmission module; a second transmission module for transmitting the output feature matrix of each convolutional layer in the encoding module to each transposed convolutional layer of the decoding module via the cross-layer connection module; and a decoding module for decoding the target feature matrix and the output feature matrix to obtain the target audio signal corresponding to the first noisy frequency data.

[0018] In an exemplary embodiment, the target neural network structure is trained using the following steps: a second acquisition unit, configured to acquire the noisy frequency training sample set; a second extraction unit, configured to extract features from the noisy frequency training samples in the noisy frequency training sample set to obtain a training feature set; and a training unit, configured to train the initial target neural network structure based on the training feature set to obtain the target neural network structure.

[0019] In an exemplary embodiment, the second acquisition unit includes: a first acquisition module for acquiring the preset audio signal; a second acquisition module for acquiring the target noise signal; a processing module for mixing and constructing the preset audio signal and the target noise signal to obtain the noisy frequency training samples; and a determination module for determining the noisy frequency training sample set based on the noisy frequency training samples and the preset audio signal.

[0020] In an exemplary embodiment, the first acquisition module includes: a noise reduction submodule, configured to acquire third band noise frequency data in a target environment and perform noise reduction processing on the third band noise frequency data to obtain processed third band noise frequency data; a first scoring submodule, configured to score the processed third band noise frequency data to obtain a first target score; and a first determination submodule, configured to determine that the processed third band noise frequency data is the preset audio signal if the first target score is greater than a first threshold.

[0021] In an exemplary embodiment, the second acquisition module includes: an acquisition submodule, configured to acquire initial noise data in a target environment; a second scoring submodule, configured to score the initial noise data to obtain a second target score; and a second determination submodule, configured to determine the initial noise data as the target noise signal if the second target score is greater than a second threshold.

[0022] In an exemplary embodiment, the training unit includes: an input module for inputting the training feature set into the initial target neural network structure to obtain a predicted audio signal corresponding to each training sample with noise; a calculation module for calculating a loss function based on the predicted audio signal and the preset audio signal to obtain a target loss function; and a training module for training the initial target neural network structure based on the target loss function to obtain the target neural network structure.

[0023] According to yet another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0024] According to yet another embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0025] In this invention, feature extraction is performed on the first band of noise frequency data at the current moment and the second band of noise frequency data at historical moments. This obtains both the audio features of the current moment and the audio features of the historical moments. The target feature information is then input into a target neural network structure, which performs noise reduction processing to obtain the corresponding target audio signal. Since the target neural network structure is trained using a preset audio signal and a target noise signal, it has the advantage of better approximation of real-world data compared to existing neural network models that directly use data obtained from an anechoic chamber as clean data. Therefore, it solves the problem of low noise reduction accuracy caused by training the noise reduction model with clean audio signals in related technologies. This improves the noise reduction accuracy of the target neural network structure, thereby effectively enhancing the call quality of voice calls. Attached Figure Description

[0026] Figure 1 This is a hardware structure block diagram of a mobile terminal according to an embodiment of the present invention;

[0027] Figure 2This is a flowchart of an audio noise reduction method according to an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of the target neural network structure according to an embodiment of the present invention;

[0029] Figure 4 This is a flowchart of an optional audio noise reduction method according to an embodiment of the present invention;

[0030] Figure 5 This is a flowchart illustrating the training process of the target neural network structure according to an embodiment of the present invention;

[0031] Figure 6 This is a schematic diagram of an audio noise reduction device according to an embodiment of the present invention. Detailed Implementation

[0032] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0034] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for an audio noise reduction method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0035] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the audio noise reduction method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0036] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0037] This embodiment provides an audio noise reduction method that runs on the aforementioned mobile terminal. Figure 2 This is a flowchart of an audio noise reduction method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0038] Step S202: Obtain the first band noise frequency data at the current time and the second band noise frequency data at historical times;

[0039] Specifically, during audio noise reduction, the system collects the first noisy frequency data at the current moment and the second noisy frequency data from previous moments. For example, if it is necessary to process the noisy frequency data of the current 10 frames, then the system collects the first noisy audio data of the current 10 frames and obtains the second noisy frequency data from the previous 10 frames.

[0040] Step S204: Extract features based on the first and second band noise frequency data to obtain target feature information;

[0041] Specifically, the feature information of the first and second band-noise frequency data is extracted. The features of the noisy signal in the time-frequency domain can be obtained by performing a short-time Fourier transform on each frame of the first and second band-noise frequency data, thereby obtaining the aforementioned target feature information.

[0042] Step S206: Input the target feature information into the target neural network structure to obtain the target audio signal corresponding to the first noisy frequency data. The target neural network structure is trained by the noisy frequency training sample set. The noisy frequency training sample set consists of multiple noisy frequency training samples and a preset audio signal corresponding to each noisy frequency training sample. Each noisy frequency training sample is obtained by the preset audio signal and the target noise signal.

[0043] Specifically, the target feature information is input into the target neural network structure, and audio noise reduction is performed through the target neural network structure to output a clean audio signal corresponding to the first noise frequency data (i.e., the target audio signal mentioned above). The target neural network structure is trained using a noisy frequency training sample set. Each noisy frequency training sample is obtained by using a preset audio signal (i.e., a clean audio signal in a real environment) and the target noise signal.

[0044] It should be noted that the target neural network structure can be a U-net network structure. The U-net network structure can extract the features of the audio signal better and more accurately, thereby improving the noise reduction effect.

[0045] Through the above steps, since the target neural network structure is trained using preset audio signals and target noise signals, it has the advantage of being closer to real-world data than the neural network models in existing technologies that directly use data obtained from an anechoic chamber as pure data. This solves the problem in related technologies where training the noise reduction model with pure audio signals results in low noise reduction accuracy, thus improving the noise reduction accuracy of the target neural network structure and effectively enhancing the call quality of voice calls.

[0046] It should be noted that the entities performing the above steps can be base stations, terminals, etc., but are not limited to these.

[0047] The target neural network structure is an important part of the audio noise reduction in this embodiment. Therefore, in the audio noise reduction method provided in this embodiment, the target neural network structure consists of an encoding module, a timing propagation module, a cross-layer connection module, and a decoding module. The cross-layer connection module is used to connect the convolutional layer in the encoding module and the transposed convolutional layer in the decoding module. The convolutional kernel in the cross-layer connection module is determined by the number of layers of the data output by the encoding module.

[0048] Specifically, such as Figure 3The target neural network structure shown consists of an encoding module (ENCODE), a temporal propagation module, a cross-layer connection module, and a decoding module (DECODE). The encoding module (ENCODE) comprises three concatenated CONV layers, a BN layer, and an ACTIVATION layer. The decoding module comprises three concatenated TRANSCONV layers, a BN layer, and an ACTIVATION layer. The temporal propagation module consists of a single-layer GRU network, and the cross-layer connection module consists of a single CONV layer. The cross-layer connection module symmetrically connects the output of the ACTIVATE layer of the ENCODE to the input of the TRANSCONV layer within the DECODE module. The size of the convolutional kernel corresponding to the CONV used in the cross-layer connection module needs to be determined based on the input dimension of the TRANSCONV layers in different layers, which is determined by the output dimension of the encoding module. By using the cross-layer connection module, the original single-dimensional information propagation is transformed into multi-dimensional information propagation, enabling faster model convergence without significantly increasing the model size, thus achieving better noise reduction.

[0049] In an optional embodiment, the decompilation module: the convolution kernel size used in the CONV layer is [3, 3], the convolution stride is set to [2, 1], and the number of convolutions is [16, 32, 64]; the activation function used in ACTIVATION is RELU, and its formula is as follows: RELU(x)=max(x,0);

[0050] Decoding module: The TRANSCONV layers all use convolution kernel sizes of [3, 1], convolution strides of [2, 1], and the number of convolutions of [16, 32, 64]; the ACTIVATION layer uses the ReLU activation function.

[0051] Timing delivery module: GRU is:

[0052] Where h(x) is the output of the convolutional module, and w, b, v, and c are all learnable parameters. For element-wise product, σ(·) is the sigmoid function;

[0053] Cross-layer connectivity module: Its CONV layer uses convolutional kernel sizes of [[5, 1], [3, 1], [1, 1]], convolutional strides of [1, 1], and the number of convolutions of [16, 32, 64]. The cross-layer connectivity model is not limited to using different convolutional kernel sizes in its design.

[0054] In this embodiment, extracting the feature information of the audio signal is crucial. Therefore, in the audio noise reduction method provided in this embodiment, feature extraction based on the first and second band noise frequency data to obtain the target feature information includes: performing a short-time Fourier transform on each frame of the first band noise frequency data to obtain the first time-frequency domain information corresponding to the first band noise frequency data; performing a short-time Fourier transform on each frame of the second band noise frequency data to obtain the second time-frequency domain information corresponding to the second band noise frequency data; and concatenating the first and second time-frequency domain information to obtain the target feature information.

[0055] Specifically, a short-time Fourier transform is performed on each frame of the first band-noise frequency data to obtain its feature information in the time-frequency domain. Similarly, a short-time Fourier transform is performed on each frame of the second band-noise frequency data to obtain its feature information in the time-frequency domain. Then, the first and second time-frequency domain information are concatenated to obtain the target feature information. Through these steps, the feature information of the audio signal can be obtained more accurately.

[0056] In the audio noise reduction method provided in this embodiment, inputting target feature information into a target neural network structure to obtain the target audio signal corresponding to the first noisy frequency data includes: encoding the target feature information through an encoding module to obtain a target feature matrix; passing the target feature matrix to a decoding module through a timing transfer module; passing the output feature matrix of each convolutional layer in the encoding module to each transposed convolutional layer in the decoding module through a cross-layer connection module; and decoding the target feature matrix and the output feature matrix through the decoding module to obtain the target audio signal corresponding to the first noisy frequency data.

[0057] Specifically, the encoding module in the target neural network structure encodes the target feature information and outputs the target feature matrix. The timing transfer module passes the target feature matrix to the decoding module. At the same time, the cross-layer connection module transmits the output feature matrix of each convolutional layer in the encoding module to each transposed convolutional layer of the decoding module. The decoding module decodes the target feature matrix and the output feature matrix to obtain the clean audio signal corresponding to the first noisy frequency data.

[0058] The above model structure takes into account multi-frame and multi-dimensional information in the cross-layer connection module, which has better performance and model training convergence speed compared to the cross-layer connection module that only uses the current frame.

[0059] In an alternative embodiment, the following can be employed: Figure 4The flowchart shown illustrates the audio noise reduction process. It acquires the noisy frequency signal at the current moment and the noisy frequency signal at a historical moment, and performs a short-time Fourier transform on both signals to obtain the time-frequency domain signal of the current moment and the time-frequency domain signal of the historical frame. This time-frequency domain signal is then concatenated with the time-frequency domain signal of the historical frame to construct the signal features input to the network. These constructed signal features are then input into the target neural network structure to obtain the clean audio signal output by the target neural network structure.

[0060] In the audio noise reduction method provided in this embodiment, the target neural network structure is trained using the following steps: obtaining a training sample set of noisy frequencies; extracting features from the noisy frequency training samples in the training sample set of noisy frequencies to obtain a training feature set; and training the initial target neural network structure based on the training feature set to obtain the target neural network structure.

[0061] Specifically, a training sample set with noise is obtained. The training sample set with noise consists of multiple training samples with noise and a preset audio signal corresponding to each training sample with noise. Each training sample with noise is obtained by using a preset clean audio signal and a target noise signal. By performing a short-time Fourier transform on each training sample with noise, a training feature set of the training sample set with noise is obtained. Finally, the initial target neural network structure is trained using the training feature set to obtain the target neural network structure.

[0062] To improve the training effect on the initial target neural network structure, the audio noise reduction method provided in this embodiment includes obtaining a training sample set with noise by: obtaining a preset audio signal; obtaining a target noise signal; mixing and constructing the preset audio signal and the target noise signal to obtain training samples with noise; and determining the training sample set with noise based on the training samples with noise and the preset audio signal.

[0063] Specifically, the preset audio signal and the target noise signal are both collected from a real environment. The preset audio signal and the target noise signal are mixed and constructed to obtain noisy frequency training samples, i.e., data augmentation processing is performed on the preset audio signal and the target noise signal. The mixed and constructed preset audio signal and the target noise signal are then used as noisy frequency training samples to obtain a noisy frequency training sample set. Compared to existing technologies that directly use data obtained from an anechoic chamber as clean data, the above method for obtaining a noisy frequency training sample set has the advantage of better reflecting real-world scenarios. This allows the trained model to better reflect real-world application scenarios, thereby achieving better noise reduction results.

[0064] In an optional embodiment, data augmentation can be performed using different signal-to-noise ratios, different impulse response convolutions, different channel filtering designs, and different loudspeaker distortion response designs. These data augmentation methods can be used in combination or individually.

[0065] In the audio noise reduction method provided in this embodiment, obtaining the preset audio signal includes: acquiring third-band noise frequency data in the target environment, and performing noise reduction processing on the third-band noise frequency data to obtain processed third-band noise frequency data; scoring the processed third-band noise frequency data to obtain a first target score; if the first target score is greater than a first threshold, then determining the processed third-band noise frequency data as the preset audio signal.

[0066] Acquiring the target noise signal includes: collecting initial noise data in the target environment; scoring the initial noise data to obtain a second target score; if the second target score is greater than a second threshold, then the initial noise data is determined to be the target noise signal.

[0067] Specifically, third-band noise frequency data is collected in a real voice call environment (i.e., the target environment mentioned above). Then, the third-band noise frequency data is processed by noise reduction methods in the prior art to obtain processed third-band noise frequency data. The processed third-band noise frequency data is then sent to an existing voice scoring system for scoring. When the score obtained by the scoring system is greater than the set threshold (i.e., the first threshold mentioned above), the training data is used as the clean voice data for formal training (i.e., the preset audio signal mentioned above).

[0068] It should be noted that the first threshold can be set according to the voice scoring system and actual needs. When the full score is 5 points, the first threshold can be set at around 3.5.

[0069] For the target noise signal: Initial noise data is collected in a real voice call environment (i.e., the target environment mentioned above), and then the initial noise data is input into an existing noise scoring system for scoring. When the score obtained by the scoring system is greater than the set threshold (i.e., the second threshold mentioned above), the training noise data is used as the formal training noise data (i.e., the target noise signal mentioned above).

[0070] It should be noted that the second threshold can be set according to the noise scoring system and actual needs. When the full score is 5 points, the first threshold can be set at around 3.5.

[0071] In summary, by denoising and scoring the noisy signals collected in real-world environments, we can obtain the clean signals needed for subsequent training. Compared to existing technologies that directly use data obtained from anechoic chambers as clean data, this method has the advantage of being closer to real-world scenarios, enabling the trained model to better reflect actual application scenarios and thus achieving better noise reduction results.

[0072] To train the initial target neural network structure more effectively, the audio noise reduction method provided in this embodiment trains the initial target neural network structure based on the training feature set to obtain the target neural network structure. This includes: inputting the training feature set into the initial target neural network structure to obtain the predicted audio signal corresponding to each training sample with noise; calculating the loss function based on the predicted audio signal and the preset audio signal to obtain the target loss function; and training the initial target neural network structure based on the target loss function to obtain the target neural network structure.

[0073] Specifically, the aforementioned training feature set is input into the initial target neural network structure, which outputs a denoised predicted audio signal. Then, a loss function is calculated on the predicted audio signal and the preset audio signal (i.e., the clean audio signal), thereby using the target loss function to train the initial target neural network structure to obtain the target neural network structure.

[0074] In an optional embodiment, the predicted audio signal and the clean audio signal are simultaneously fed into a LOSS function to calculate the error. The LOSS function is designed as follows:

[0075]

[0076]

[0077] LOSS(est,ref) = 0.5 * LOSS abs (est,ref)+0.5*LOSS ri (est,ref)

[0078] Where k is the number of frequency points in the audio signal, n is the number of frames in the time dimension, and est and ref represent the time-frequency domain representations of the model output signal and the clean signal, respectively; where (·) r ,(·) i ,(·) abs These represent taking the real part, imaginary part, and absolute value of the signal, respectively.

[0079] The LOSS function described above is used to implement the training and optimization process of the initial target neural network structure.

[0080] In an alternative embodiment, the following can be employed: Figure 5 The flowchart described implements the model training process. Step 1. Data Augmentation: The speech data and noise data in the training set are screened to obtain qualified training data. The training data is then mixed using data augmentation and transformed from the time domain to the time-frequency domain using short-time Fourier transform.

[0081] Step 2. Feature extraction: The time-frequency domain signal at the current moment is concatenated with the time-frequency domain signals of historical frames to construct the signal features of the network input;

[0082] Step 3. Network Training: Construct the UNET network structure, in which each CONV layer in ENCODE and DECODE is connected through a custom cross-layer connection; the size of the CONV convolution kernel needs to be determined according to the number of frames of historical signal required by each CONV layer in ENCODE.

[0083] Step 4. Input the signal features constructed in Step 1 into the UNET network constructed in Step 2, and obtain the network output;

[0084] Step 5. Feed the network output and clean signal into the loss function for calculation, and train the model using the Adam optimization algorithm (a gradient descent algorithm);

[0085] Step 6. Finally, use the trained model to reduce noise in the test signal.

[0086] In the audio noise reduction method provided in this embodiment, feature extraction is performed on the first band noise frequency data at the current moment and the second band noise frequency data at historical moments. This obtains both the audio features at the current moment and the audio features at historical moments, fully considering the multi-frame, multi-dimensional audio information from history. The target feature information is input into the target neural network structure, and noise reduction is performed through the target neural network structure to obtain the corresponding target audio signal. Since the target neural network structure is trained using a preset audio signal and a target noise signal, it has the advantage of better approximation of real-world scene data compared to the neural network models in existing technologies that directly use data obtained from an anechoic chamber as clean data. Therefore, it can solve the problem of low noise reduction accuracy caused by training the noise reduction model with a clean audio signal in related technologies, thereby improving the noise reduction accuracy of the target neural network structure and effectively improving the call quality of voice calls.

[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0088] This embodiment also provides an audio noise reduction device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0089] Figure 6 This is a structural block diagram of an audio noise reduction device according to an embodiment of the present invention, such as... Figure 6 As shown, the device includes: a first acquisition unit 602, a first extraction unit 604, and an input unit 606.

[0090] The first acquisition unit 602 is used to acquire the first band noise frequency data at the current moment and the second band noise frequency data at historical moments;

[0091] The first extraction unit 604 is used to extract features based on the first noise-frequency data and the second noise-frequency data to obtain target feature information.

[0092] The input unit 606 is used to input target feature information into the target neural network structure to obtain the target audio signal corresponding to the first noisy frequency data. The target neural network structure is trained by the noisy frequency training sample set, which consists of multiple noisy frequency training samples and a preset audio signal corresponding to each noisy frequency training sample. Each noisy frequency training sample is obtained by the preset audio signal and the target noise signal.

[0093] Optionally, in the audio noise reduction device provided in this embodiment, the target neural network structure consists of an encoding module, a timing transfer module, a cross-layer connection module, and a decoding module. The cross-layer connection module is used to connect the convolutional layer in the encoding module and the transposed convolutional layer in the decoding module. The convolutional kernel in the cross-layer connection module is determined by the number of layers of the data output by the encoding module.

[0094] Optionally, in the audio noise reduction device provided in this embodiment, the first extraction unit includes: a first transformation module, used to perform a short-time Fourier transform on each frame signal in the first noisy frequency data to obtain first time-frequency domain information corresponding to the first noisy frequency data; a second transformation module, used to perform a short-time Fourier transform on each frame signal in the second noisy frequency data to obtain second time-frequency domain information corresponding to the second noisy frequency data; and a splicing module, used to splice the first time-frequency domain information and the second time-frequency domain information to obtain target feature information.

[0095] Optionally, in the audio noise reduction device provided in this embodiment, the input unit includes: an encoding module for encoding target feature information to obtain a target feature matrix; a first transmission module for transmitting the target feature matrix to the decoding module via a timing transmission module; a second transmission module for transmitting the output feature matrix of each convolutional layer in the encoding module to each transposed convolutional layer of the decoding module via a cross-layer connection module; and a decoding module for decoding the target feature matrix and the output feature matrix to obtain the target audio signal corresponding to the first noisy frequency data.

[0096] Optionally, in the audio noise reduction device provided in this embodiment, the target neural network structure is trained using the following steps: a second acquisition unit is used to acquire a training sample set with noise; a second extraction unit is used to extract features from the training samples with noise in the training sample set with noise to obtain a training feature set; and a training unit is used to train the initial target neural network structure based on the training feature set to obtain the target neural network structure.

[0097] Optionally, in the audio noise reduction device provided in this embodiment, the second acquisition unit includes: a first acquisition module for acquiring a preset audio signal; a second acquisition module for acquiring a target noise signal; a processing module for mixing and constructing the preset audio signal and the target noise signal to obtain a training sample with noise; and a determination module for determining a training sample set with noise based on the training sample with noise and the preset audio signal.

[0098] Optionally, in the audio noise reduction device provided in this embodiment, the first acquisition module includes: a noise reduction submodule, used to acquire third-band noise frequency data in a target environment and perform noise reduction processing on the third-band noise frequency data to obtain processed third-band noise frequency data; a first scoring submodule, used to score the processed third-band noise frequency data to obtain a first target score; and a first determination submodule, used to determine that the processed third-band noise frequency data is a preset audio signal if the first target score is greater than a first threshold.

[0099] Optionally, in the audio noise reduction device provided in this embodiment, the second acquisition module includes: an acquisition submodule, used to acquire initial noise data in the target environment; a second scoring submodule, used to score the initial noise data to obtain a second target score; and a second determination submodule, used to determine the initial noise data as a target noise signal if the second target score is greater than a second threshold.

[0100] Optionally, in the audio noise reduction device provided in this embodiment, the training unit includes: an input module for inputting a training feature set into an initial target neural network structure to obtain a predicted audio signal corresponding to each training sample with noise; a calculation module for calculating a loss function based on the predicted audio signal and a preset audio signal to obtain a target loss function; and a training module for training the initial target neural network structure based on the target loss function to obtain a target neural network structure.

[0101] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0102] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0103] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0104] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0105] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0106] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0107] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0108] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An audio noise reduction method, characterized by, The method comprises: obtaining first noisy audio data of a current moment and second noisy audio data of a historical moment; performing feature extraction on the first noisy audio data and the second noisy audio data to obtain target feature information; inputting the target feature information into a target neural network structure to obtain a target audio signal corresponding to the first noisy audio data, wherein the target neural network structure is trained by a noisy audio training sample set, the noisy audio training sample set is composed of a plurality of noisy audio training samples and a preset audio signal corresponding to each noisy audio training sample, and each noisy audio training sample is obtained by the preset audio signal and a target noise signal; wherein the target neural network structure comprises an encoding module, a time sequence transmission module, a cross-layer connection module and a decoding module, wherein the cross-layer connection module is used to connect a convolution layer in the encoding module and a transposed convolution layer in the decoding module, and a convolution kernel in the cross-layer connection module is determined by the number of layers of data output by the encoding module; inputting the target feature information into the target neural network structure to obtain the target audio signal corresponding to the first noisy audio data comprises: performing encoding processing on the target feature information by the encoding module to obtain a target feature matrix; transmitting the target feature matrix to the decoding module through the time sequence transmission module; transmitting the output feature matrix of each convolution layer in the encoding module to each transposed convolution layer in the decoding module through the cross-layer connection module; and performing decoding processing on the target feature matrix and the output feature matrix by the decoding module to obtain the target audio signal corresponding to the first noisy audio data.

2. The method of claim 1, wherein, performing feature extraction on the first noisy audio data and the second noisy audio data to obtain target feature information comprises: performing short-time Fourier transform on each frame of signal in the first noisy audio data to obtain first time-frequency domain information corresponding to the first noisy audio data; performing short-time Fourier transform on each frame of signal in the second noisy audio data to obtain second time-frequency domain information corresponding to the second noisy audio data; performing splicing processing on the first time-frequency domain information and the second time-frequency domain information to obtain the target feature information.

3. The method of claim 1, wherein, The target neural network structure is trained by the following steps: obtaining the noisy audio training sample set; performing feature extraction on the noisy audio training samples in the noisy audio training sample set to obtain a training feature set; training an initial target neural network structure according to the training feature set to obtain the target neural network structure.

4. The method of claim 3, wherein, obtaining the noisy audio training sample set comprises: obtaining the preset audio signal; obtaining the target noise signal; mixing and constructing the preset audio signal and the target noise signal to obtain the noisy audio training sample; determining the noisy audio training sample set according to the noisy audio training sample and the preset audio signal.

5. The method of claim 4, wherein, obtaining the preset audio signal comprises: Third noisy audio data is collected in a target environment, and the third noisy audio data is denoised to obtain processed third noisy audio data; The processed third noisy audio data is scored to obtain a first target score; If the first target score is greater than a first threshold, the processed third noisy audio data is determined to be the preset audio signal.

6. The method of claim 4, wherein, Obtaining the target noise signal includes: Initial noise data is collected in a target environment; The initial noise data is scored to obtain a second target score; If the second target score is greater than a second threshold, the initial noise data is determined to be the target noise signal.

7. The method of claim 3, wherein, Training an initial target neural network structure according to the training feature set to obtain the target neural network structure includes: The training feature set is input into the initial target neural network structure to obtain a predicted audio signal corresponding to each noisy audio training sample; A target loss function is calculated according to the predicted audio signal and the preset audio signal; The initial target neural network structure is trained according to the target loss function to obtain the target neural network structure.

8. An audio noise reduction device, comprising: It includes: A first acquisition unit configured to acquire first noisy audio data at a current time and second noisy audio data at a historical time; A first extraction unit configured to extract features from the first noisy audio data and the second noisy audio data to obtain target feature information; An input unit configured to input the target feature information into a target neural network structure to obtain a target audio signal corresponding to the first noisy audio data, wherein the target neural network structure is trained by a noisy audio training sample set, the noisy audio training sample set is composed of a plurality of noisy audio training samples and a preset audio signal corresponding to each noisy audio training sample, and each noisy audio training sample is obtained by the preset audio signal and a target noise signal; The target neural network structure is composed of an encoding module, a time sequence transmission module, a cross-layer connection module, and a decoding module, wherein the cross-layer connection module is used to connect a convolution layer in the encoding module and a transposed convolution layer in the decoding module, and a convolution kernel in the cross-layer connection module is determined by the number of layers of data output by the encoding module; The input unit includes an encoding module configured to encode the target feature information by the encoding module to obtain a target feature matrix, a first transmission module configured to transmit the target feature matrix to the decoding module by the time sequence transmission module, a second transmission module configured to transmit an output feature matrix of each convolution layer in the encoding module to each transposed convolution layer in the decoding module by the cross-layer connection module, and a decoding module configured to decode the target feature matrix and the output feature matrix by the decoding module to obtain the target audio signal corresponding to the first noisy audio data.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the steps of the audio noise reduction method in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the audio noise reduction method in any one of claims 1 to 7.

Images