Audio noise reduction method and related device

Abstract

The invention discloses an audio noise reduction method and a related device, and relates to the technical field of audio noise reduction, and the audio noise reduction method comprises the steps: determining an ideal ratio mask corresponding to a previous audio frame of a target audio frame according to a denoised audio frame corresponding to the previous audio frame of the target audio frame, the target audio frame is an audio frame to be de-noised in the multi-channel audio with noise; determining a beam parameter corresponding to the target audio frame according to an ideal ratio mask corresponding to a previous audio frame of the target audio frame; according to the beam parameter corresponding to the target audio frame, performing beam forming processing on the target audio frame to obtain an enhanced single-channel audio signal corresponding to the target audio frame; and performing noise reduction processing on the enhanced single-channel audio signal corresponding to the target audio frame by using an audio noise reduction model obtained by pre-training to obtain a noise-reduced audio frame corresponding to the target audio frame. According to the audio noise reduction method disclosed by the invention, the noise in the audio can be effectively suppressed, and a relatively good noise reduction effect is achieved.

Description

Technical Field

[0001] The present application relates to the technical field of audio noise reduction, and in particular to an audio noise reduction method and related devices. Background Art

[0002] In many scenarios, microphone arrays are used to collect audio signals. Due to many factors, the audio signals collected by microphone arrays usually contain noise. In order to improve the audio quality, it is usually necessary to perform noise reduction processing on the audio to suppress the noise in the audio and thus highlight the target speech in the audio.

[0003] Most of the current audio noise reduction methods are traditional noise reduction methods (such as using filtering algorithms to reduce audio noise). Although traditional noise reduction methods can reduce the noise in the audio to a certain extent, the noise reduction effect is poor and cannot meet the needs of some application scenarios. Summary of the invention

[0004] In view of this, the present application provides an audio noise reduction method and related device, which are used to solve the problem that the current audio noise reduction method has poor noise reduction effect. The technical solution is as follows:

[0005] The first aspect of the present application provides an audio noise reduction method, comprising:

[0006] Determine, according to a denoised audio frame corresponding to a previous audio frame of a target audio frame, an ideal ratio mask corresponding to the previous audio frame of the target audio frame, wherein the target audio frame is an audio frame to be denoised in noisy multi-channel audio;

[0007] Determining beam parameters corresponding to the target audio frame according to an ideal ratio mask corresponding to a previous audio frame of the target audio frame;

[0008] Performing beamforming processing on the target audio frame according to the beam parameters corresponding to the target audio frame to obtain an enhanced single-channel audio signal corresponding to the target audio frame;

[0009] Using a pre-trained audio noise reduction model, noise reduction processing is performed on the enhanced single-channel audio signal corresponding to the target audio frame to obtain a noise-reduced audio frame corresponding to the target audio frame.

[0010] In a possible implementation, determining, according to a noise-reduced audio frame corresponding to an audio frame preceding the target audio frame, an ideal ratio mask corresponding to the audio frame preceding the target audio frame includes:

[0011] An ideal ratio mask corresponding to the previous audio frame of the target audio frame is determined according to a frequency domain amplitude spectrum of a denoised audio frame corresponding to the previous audio frame of the target audio frame and an amplitude spectrum of an enhanced single-channel audio signal obtained by performing beamforming processing on the previous audio frame of the target audio frame.

[0012] In a possible implementation, determining the beam parameter corresponding to the target audio frame according to an ideal ratio mask corresponding to a previous audio frame of the target audio frame includes: determining a speech existence probability of the target audio frame according to an ideal ratio mask corresponding to a previous audio frame of the target audio frame;

[0013] A speech covariance matrix of the target audio frame is determined according to an ideal ratio mask corresponding to an audio frame preceding the target audio frame and a speech covariance matrix of an audio frame preceding the target audio frame.

[0014] In a possible implementation manner, performing beamforming processing on the target audio frame according to the beam parameter corresponding to the target audio frame to obtain an enhanced single-channel audio signal corresponding to the target audio frame includes:

[0015] Distinguishing the target audio frame into a speech frame and a noise frame;

[0016] If the target audio frame is a speech frame, a relative transfer function in the beamforming algorithm is updated according to the beam parameters corresponding to the target audio frame; if the target audio frame is a noise frame, an adaptive noise canceller in the beamforming algorithm is updated according to the beam parameters corresponding to the target audio frame to obtain an updated beamforming algorithm;

[0017] The updated beamforming algorithm is used to perform beamforming processing on the target audio frame to obtain an enhanced single-channel audio signal corresponding to the target audio frame.

[0018] In a possible implementation, the distinguishing the target audio frame as a speech frame and a noise frame includes:

[0019] The target audio frame is distinguished as a speech frame or a noise frame according to ideal ratio masks respectively corresponding to each audio frame of a forward adjacent frame sequence of the target audio frame, wherein the forward adjacent frame sequence of the target audio frame is a frame sequence consisting of N consecutive audio frames before the target audio frame, and N is an integer greater than 1.

[0020] In a possible implementation, the distinguishing the target audio frame as a speech frame or a noise frame according to the ideal ratio masks respectively corresponding to the audio frames in the forward adjacent frame sequence of the target audio frame includes:

[0021] If the ideal ratio masks respectively corresponding to the audio frames in the forward adjacent frame sequence of the target audio frame are all less than or equal to the set threshold, the target audio frame is determined to be a noise frame;

[0022] If the ideal ratio mask corresponding to at least one audio frame in the forward adjacent frame sequence of the target audio frame is greater than the set threshold, the target audio frame is determined to be a speech frame.

[0023] In a possible implementation manner, the enhanced single-channel audio signal corresponding to the target audio frame is a frequency domain signal;

[0024] The method of using the pre-trained audio noise reduction model to perform noise reduction processing on the enhanced single-channel audio signal corresponding to the target audio frame to obtain a noise-reduced audio frame corresponding to the target audio frame includes:

[0025] Obtaining an amplitude spectrum or a logarithmic amplitude spectrum and a phase spectrum according to the enhanced single-channel audio signal corresponding to the target audio frame;

[0026] Inputting the acquired amplitude spectrum or logarithmic amplitude spectrum into a pre-trained audio denoising model to obtain a denoised amplitude spectrum output by the audio denoising model;

[0027] An inverse short-time Fourier transform is performed on the frequency domain signal composed of the phase spectrum and the noise-reduced amplitude spectrum to obtain a noise-reduced audio frame corresponding to the target audio frame.

[0028] In a possible implementation, the audio noise reduction model is trained by using training noisy multi-channel audio and training clean multi-channel audio corresponding to the training noisy multi-channel audio, and the training noisy multi-channel audio is obtained by superimposing noise on the training clean multi-channel audio;

[0029] The training process of the audio noise reduction model includes:

[0030] Determining an ideal ratio mask corresponding to each audio frame of the training noisy multi-channel audio according to the training clean multi-channel audio and the noise;

[0031] Determining beam parameters corresponding to each audio frame of the training noisy multi-channel audio according to an ideal ratio mask corresponding to each audio frame of the training noisy multi-channel audio;

[0032] According to the beam parameters corresponding to each audio frame of the training noisy multi-channel audio, the training noisy multi-channel audio and the training clean multi-channel audio are respectively subjected to beamforming processing frame by frame to obtain enhanced single-channel audio signals corresponding to the training noisy multi-channel audio and the training clean multi-channel audio, respectively;

[0033] The audio denoising model is trained with the goal of making the denoised audio signal obtained by performing frame-by-frame denoising on the enhanced single-channel audio signal corresponding to the training noisy multi-channel audio be close to the enhanced single-channel audio signal corresponding to the training clean multi-channel audio.

[0034] A second aspect of the present application provides an audio noise reduction device, comprising: an ideal ratio mask determination module, a beam parameter determination module, a beam forming processing module, and an audio noise reduction module;

[0035] The ideal ratio mask determination module is used to determine the ideal ratio mask corresponding to the previous audio frame of the target audio frame according to the denoised audio frame corresponding to the previous audio frame of the target audio frame, wherein the target audio frame is an audio frame to be denoised in the noisy multi-channel audio;

[0036] The beam parameter determination module is used to determine the beam parameter corresponding to the target audio frame according to the ideal ratio mask corresponding to the previous audio frame of the target audio frame;

[0037] The beamforming processing module is used to perform beamforming processing on the target audio frame according to the beam parameters corresponding to the target audio frame to obtain an enhanced single-channel audio signal corresponding to the target audio frame;

[0038] The audio noise reduction module is used to use a pre-trained audio noise reduction model to perform noise reduction processing on the enhanced single-channel audio signal corresponding to the target audio frame to obtain a noise-reduced audio frame corresponding to the target audio frame.

[0039] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0040] The memory is used to store computer programs;

[0041] The processor is used to execute the computer program so that the electronic device can implement the steps of any one of the above-mentioned audio noise reduction methods.

[0042] A fourth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of any one of the above-mentioned audio noise reduction methods.

[0043] A fifth aspect of the present application provides a computer program product, comprising computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of any one of the above-mentioned audio noise reduction methods.

[0044] By means of the above technical scheme, the audio denoising method provided by the present application first determines the ideal ratio mask corresponding to the previous audio frame of the target audio frame according to the denoised audio frame corresponding to the previous audio frame of the target audio frame, and then determines the beam parameters corresponding to the target audio frame according to the ideal ratio mask corresponding to the previous audio frame of the target audio frame, and then performs beamforming processing on the target audio frame according to the beam parameters corresponding to the target audio frame to obtain an enhanced single-channel audio signal corresponding to the target audio frame, and finally uses the pre-trained audio denoising model to perform denoising processing on the enhanced single-channel audio signal corresponding to the target audio frame to obtain a denoised audio frame corresponding to the target audio frame. The audio noise reduction method provided by the present application is a noise reduction method that combines adaptive beamforming with an audio noise reduction model. The method determines the beam parameters corresponding to the target audio frame by means of an ideal ratio mask determined based on the noise-reduced audio frame corresponding to the previous audio frame of the target audio frame. Since the ideal ratio mask is determined based on the noise-reduced audio frame corresponding to the previous audio frame, relatively accurate beam parameters can be determined based on the ideal ratio mask. Furthermore, beamforming processing is performed on the target audio frame based on the relatively accurate beam parameters to obtain an audio signal with significantly improved signal quality. The audio signal with significantly improved signal quality is further subjected to noise reduction by the audio noise reduction model to obtain an audio signal with further improved signal quality. The audio noise reduction method provided by the present application fully combines the advantages of adaptive beamforming and audio noise reduction models, can effectively suppress noise in the audio, and has a good noise reduction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0046] Figure 1 A schematic diagram of a system architecture involved in this application;

[0047] Figure 2 A schematic diagram of a hardware structure of a terminal provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a hardware structure of a server provided in an embodiment of the present application;

[0049] Figure 4 A flowchart of an audio noise reduction method provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of performing noise reduction on the t-th audio frame provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of the structure of an audio noise reduction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0053] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0054] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0055] In one possible implementation, Figure 1 As shown, the system architecture involved in the present application may include a terminal 101 and a server 102, and the terminal 101 may interact with the server 102 via a network (wired network or wireless network). The server 102 may include one or more servers ( Figure 1 The terminal 101 can obtain the noisy multi-channel audio, transmit the noisy multi-channel audio to the server 102 through the network, and the server 102 uses the audio noise reduction method provided in the present application to reduce the noise of the noisy multi-channel audio, and transmits the noise reduction result to the terminal 101 through the network.

[0056] In another possible implementation, the system architecture involved in the present application may include a terminal. The terminal has strong data processing capabilities. The terminal may obtain noisy multi-channel audio and use the audio noise reduction method provided in the present application to reduce noise on the noisy multi-channel audio.

[0057] Next, the product form of the above-mentioned terminal is described.

[0058] The above-mentioned terminal can be a mobile phone, a tablet computer, a wearable device, a robot, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiments of the present application do not impose any restrictions on this.

[0059] Figure 2 An optional hardware structure diagram of a terminal is shown.

[0060] refer to Figure 2 As shown, the terminal may include a radio frequency unit 210, a memory 220, an input unit 230, a display unit 240, a camera 250 (optional), an audio circuit 260 (optional), a speaker 261 (optional), a microphone 262 (optional), an earphone jack 263 (optional), a processor 270, an external interface 280, a power supply 290 and other components. Those skilled in the art will appreciate that Figure 2 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0061] The input unit 230 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the terminal. Specifically, the input unit 230 may include a touch screen 231 (optional) and / or other input devices 232. The touch screen 231 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 270, and can receive and execute the command sent by the processor 270; the touch signal at least includes the contact point coordinate information. The touch screen 231 can provide an input interface and an output interface between the terminal and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch screen 231, the input unit 230 can also include other input devices. Specifically, other input devices 232 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0062] The display unit 240 may be used to display information input by a user or information provided to a user, various menus of the terminal, an interactive interface, file display, and / or playback of any multimedia file.

[0063] The memory 220 can be used to store instructions and data. The memory 220 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc. The instruction storage area can store software units such as operating systems, applications, and instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; it provides the processor 270 with hardware, software, and data resources including management of computing and processing equipment, and supports control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0064] The processor 270 is the control center of the terminal. It uses various interfaces and lines to connect various parts of the entire terminal. By running or executing instructions stored in the memory 220 and calling data stored in the memory 220, it executes various functions of the terminal and processes data, thereby controlling the terminal as a whole. Optionally, the processor 270 may include one or more processing units; preferably, the processor 270 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 270. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 270 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 220, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0065] Among them, the memory 220 can be used to store software codes related to SQL statement generation methods and data analysis methods, the processor 270 can execute the software codes in the memory 220, and can also schedule other units (such as the above-mentioned input unit 230 and display unit 240) to achieve corresponding functions.

[0066] The radio frequency unit 210 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 270 for processing; in addition, the designed uplink data is sent to the base station. Generally, the radio frequency unit 210 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LowNoiseAmplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 210 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General PacketRadio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0067] In the embodiment of the present application, the radio frequency unit 210 can send data to other devices, and can also receive data sent by other devices. It should be understood that the radio frequency unit 210 is optional, and can be replaced by other communication interfaces, such as a network port.

[0068] The terminal also includes a power supply 290 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 270 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.

[0069] The terminal also includes an external interface 280, which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal to communicate with other devices, or to connect a charger to charge the terminal.

[0070] Although not shown, the terminal may further include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here.

[0071] Next, the product form of the above server is described.

[0072] Figure 3 A structural diagram of the above server is provided, such as Figure 3As shown, the server may include a bus 301, a processor 302, a communication interface 303, and a memory 304. The processor 302, the memory 304, and the communication interface 303 communicate with each other via the bus 301.

[0073] The bus 301 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0074] The processor 302 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0075] The memory 304 may include a volatile memory, such as a random access memory (RAM). The memory 304 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0076] The memory 304 may be used to store software codes related to the SQL statement generation method and the data analysis method. The processor 302 may call the software codes stored in the memory 304 or schedule other units to implement corresponding functions.

[0077] The processors in the above-mentioned terminals and servers (for example, processor 270 and processor 302) may be hardware circuits (such as application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, digital signal processors (DSP), microprocessors or microcontrollers, etc.), or combinations of these hardware circuits. For example, the processor may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0078] In view of the poor noise reduction effect of the current audio noise reduction method, the inventor of this case conducted research and came up with a noise reduction scheme based on adaptive beamforming and neural network model during the research process. The noise reduction process of this scheme is: estimate the key beam parameters based on the neural network model, and perform beamforming processing on the audio to be noise-reduced according to the key beam parameters to obtain the noise-reduced signal. The inventor of this case studied the above-mentioned noise reduction scheme and found that the noise reduction effect of the above-mentioned scheme is subject to the ability of the adaptive beamforming algorithm itself. Although it can achieve more accurate directional enhancement of the signal, its noise reduction performance is still unsatisfactory in sudden noise and low signal-to-noise ratio environments. In view of the fact that the noise reduction effect of the above-mentioned scheme is still not ideal, the inventor of this case continued to study. Through continuous research, he finally proposed a better audio noise reduction method, which can effectively suppress the noise in the audio. Next, the audio noise reduction method provided by this application is introduced through the following embodiments.

[0079] See also Figure 4 , shows a schematic flow chart of an audio noise reduction method provided in an embodiment of the present application, the method may include:

[0080] Step S401: determining an ideal ratio mask (IRM) corresponding to the previous audio frame of the target audio frame according to the denoised audio frame corresponding to the previous audio frame of the target audio frame.

[0081] The target audio frame is an audio frame to be denoised in the noisy multi-channel audio, and the noisy multi-channel audio is collected by a microphone array.

[0082] like Figure 5As shown, assuming that the target audio frame is the t-th audio frame of the noisy multi-channel audio, the ideal ratio mask IRM(t-1) corresponding to the t-1-th audio frame is determined according to the denoised audio frame corresponding to the t-1-th audio frame of the noisy multi-channel audio.

[0083] Step S402: determining beam parameters corresponding to the target audio frame according to an ideal ratio mask corresponding to an audio frame preceding the target audio frame.

[0084] The beam parameters are parameters based on which the beam forming process is performed, and the beam parameters corresponding to the target audio frame may include: the speech existence probability and the speech covariance matrix of the target audio frame.

[0085] Assuming that the target audio frame is the t-th audio frame of the noisy multi-channel audio, the speech presence probability SPP(t) and the speech covariance matrix R of the t-th audio frame are determined according to the ideal ratio mask IRM(t-1) corresponding to the t-1-th audio frame of the noisy multi-channel audio. SS (t).

[0086] Step S403: performing beamforming processing on the target audio frame according to the beam parameters corresponding to the target audio frame to obtain an enhanced single-channel audio signal corresponding to the target audio frame.

[0087] like Figure 5 As shown, assuming that the target audio frame is the t-th audio frame of the noisy multi-channel audio, then according to the speech existence probability SPP(t) of the t-th audio frame and the speech covariance matrix R SS (t), using the set beamforming algorithm, beamforming processing is performed on the t-th audio frame to obtain an enhanced single-channel audio signal corresponding to the t-th audio frame. The beamforming processing can improve the signal-to-noise ratio of the audio signal.

[0088] Beamforming combines multi-channel microphone signals, suppresses non-target direction signals, and enhances target direction signals, thereby achieving focused sound pickup in a specific direction. It can effectively improve the signal-to-noise ratio of the audio signal and also play a role in noise reduction.

[0089] In this embodiment, the set beamforming algorithm may be a GSC algorithm (generalized sidelobe cancellation algorithm). Of course, this embodiment is not limited thereto. The set beamforming algorithm may also be an MVDR algorithm (minimum variance distortionless response algorithm), an LCMV algorithm (linear constrained minimum variance algorithm), and the like.

[0090] In addition, it should be noted that the ideal ratio mask corresponding to an audio frame is a vector, and the elements in the vector are the ideal ratio masks corresponding to the audio frame at each frequency point, that is, the ideal ratio masks corresponding to an audio frame at each frequency point constitute the ideal ratio mask corresponding to the audio frame. If the target audio frame is the first audio frame of noisy multi-channel audio, the ideal ratio mask is initialized to a vector of all 1s, and beamforming is performed on the first audio frame according to the initialized ideal ratio mask.

[0091] Step S404: using the pre-trained audio noise reduction model, perform noise reduction processing on the enhanced single-channel audio signal corresponding to the target audio frame to obtain a noise-reduced audio frame corresponding to the target audio frame.

[0092] like Figure 5 As shown, after obtaining the enhanced single-channel audio signal corresponding to the target audio frame, the pre-trained audio noise reduction model is further used to perform noise reduction processing on the enhanced single-channel audio signal corresponding to the target audio frame to obtain a noise-reduced audio frame corresponding to the target audio frame.

[0093] The audio noise reduction model is trained using training data in a pre-constructed training data set, each piece of training data in the training data set includes a training noisy multi-channel audio and a training clean multi-channel audio corresponding to the training noisy multi-channel audio, and the training noisy multi-channel audio is obtained by superimposing noise on the training clean multi-channel audio. The training process of the audio noise reduction model will be introduced in subsequent embodiments.

[0094] Assuming that the target audio frame is the t-th audio frame of the noisy multi-channel audio, after obtaining the denoised audio frame corresponding to the t-th audio frame, the ideal ratio mask IRM(t) corresponding to the t-th audio frame can be determined according to the denoised audio frame corresponding to the t-th audio frame, and the beam parameters SPP(t+1) and R corresponding to the t+1-th audio frame of the noisy multi-channel audio can be determined according to the ideal ratio mask IRM(t) corresponding to the t-th audio frame. SS (t+1), according to the beam parameters SPP(t+1) and R corresponding to the t+1th audio frame SS (t+1) Perform beamforming processing on the t+1th audio frame to obtain an enhanced single-channel audio signal corresponding to the t+1th audio frame, use the pre-trained audio denoising model to perform denoising processing on the enhanced single-channel audio signal corresponding to the t+1th audio frame to obtain a denoised audio frame corresponding to the t+1th audio frame, and so on, until all audio frames of the noisy multi-channel audio are processed.

[0095] The audio denoising method provided in the embodiment of the present application first determines an ideal ratio mask corresponding to the previous audio frame of the target audio frame according to the denoised audio frame corresponding to the previous audio frame of the target audio frame, then determines the beam parameters corresponding to the target audio frame according to the ideal ratio mask corresponding to the previous audio frame of the target audio frame, then performs beamforming processing on the target audio frame according to the beam parameters corresponding to the target audio frame to obtain an enhanced single-channel audio signal corresponding to the target audio frame, and finally uses a pre-trained audio denoising model to perform denoising processing on the enhanced single-channel audio signal corresponding to the target audio frame to obtain a denoised audio frame corresponding to the target audio frame. The audio noise reduction method provided in the embodiment of the present application is a noise reduction method that combines adaptive beamforming with an audio noise reduction model. The method determines the beam parameters corresponding to the target audio frame with the help of an ideal ratio mask determined based on the noise-reduced audio frame corresponding to the previous audio frame of the target audio frame. Since the ideal ratio mask is determined based on the noise-reduced audio frame corresponding to the previous audio frame, relatively accurate beam parameters can be determined based on the ideal ratio mask. Furthermore, beamforming processing is performed on the target audio frame based on the relatively accurate beam parameters to obtain an audio signal with significantly improved signal quality. The audio signal with significantly improved signal quality is further subjected to noise reduction by the audio noise reduction model to obtain an audio signal with further improved signal quality. The audio noise reduction method provided in the embodiment of the present application fully combines the advantages of adaptive beamforming and audio noise reduction models, can effectively suppress noise in audio, and has a good noise reduction effect.

[0096] In another embodiment of the present application, the specific implementation process of "Step S401: determining an ideal ratio mask corresponding to the previous audio frame of the target audio frame according to the denoised audio frame corresponding to the previous audio frame of the target audio frame" in the above embodiment is introduced.

[0097] In a possible implementation, the process of determining an ideal ratio mask corresponding to the previous audio frame of the target audio frame based on the denoised audio frame corresponding to the previous audio frame of the target audio frame may include: determining the ideal ratio mask corresponding to the previous audio frame of the target audio frame based on the frequency domain amplitude spectrum of the denoised audio frame corresponding to the previous audio frame of the target audio frame and the amplitude spectrum of the enhanced single-channel audio signal obtained by performing beamforming processing on the previous audio frame of the target audio frame.

[0098] Assuming that the target audio frame is the t-th audio frame of the noisy multi-channel audio, the ideal ratio mask IRM(t-1) corresponding to the t-1-th audio frame is determined according to the frequency domain amplitude spectrum of the denoised audio frame corresponding to the t-1-th audio frame of the noisy multi-channel audio and the amplitude spectrum of the enhanced single-channel audio signal obtained by beamforming the t-1-th audio frame.

[0099] More specifically, the ideal ratio mask IRM(t-1) corresponding to the t-1th audio frame can be calculated as shown in the following formula (1):

[0100] (1)

[0101] in, represents the frequency domain amplitude spectrum of the denoised audio frame corresponding to the t-1th audio frame, It represents the difference between the amplitude spectrum of the enhanced single-channel audio signal obtained by performing beamforming processing on the t-1th audio frame and the frequency domain amplitude spectrum of the denoised audio frame corresponding to the t-1th audio frame. It should be noted that the above formula is the calculation method of IRM(t-1) when the GSC algorithm is used for beamforming processing in the denoising process. When other algorithms are used for beamforming processing, the calculation method of IRM(t-1) remains unchanged.

[0102] In another embodiment of the present application, the specific implementation process of "determining the beam parameters corresponding to the target audio frame according to the ideal ratio mask corresponding to the previous audio frame of the target audio frame" in the above embodiment is introduced.

[0103] The process of determining the beam parameters corresponding to the target audio frame according to the ideal ratio mask corresponding to the previous audio frame of the target audio frame may include: determining the probability of speech existence in the target audio frame according to the ideal ratio mask corresponding to the previous audio frame of the target audio frame; determining the speech covariance matrix of the target audio frame according to the ideal ratio mask corresponding to the previous audio frame of the target audio frame and the speech covariance matrix of the previous audio frame of the target audio frame.

[0104] Assuming that the target audio frame is the t-th audio frame of the noisy multi-channel audio, the speech presence probability SPP(t) of the t-th audio frame is determined according to the ideal ratio mask IRM(t-1) corresponding to the t-1-th audio frame, and the speech covariance matrix R of the t-1-th audio frame is calculated according to the ideal ratio mask IRM(t-1) corresponding to the t-1-th audio frame. SS (t-1), determine the speech covariance matrix R of the t-th audio frame SS (t).

[0105] More specifically, the speech presence probability SPP(t) of the t-th audio frame can be calculated as shown in equation (2), and the speech covariance matrix R of the t-th audio frame can be calculated as shown in equation (3): SS (t):

[0106] (2)

[0107] (3)

[0108] Among them, α u and α s is the smoothing coefficient, X mic The raw noisy audio signal acquired for the primary microphone of a microphone array (a microphone array includes multiple microphones, one of which is usually designated as the primary microphone).

[0109] In another embodiment of the present application, the implementation process of "step S403: performing beamforming processing on the target audio frame according to the beam parameters corresponding to the target audio frame to obtain an enhanced single-channel audio signal corresponding to the target audio frame" in the above embodiment is introduced.

[0110] In a possible implementation, the process of performing beamforming processing on the target audio frame according to the beam parameters corresponding to the target audio frame to obtain the enhanced single-channel audio signal corresponding to the target audio frame may include:

[0111] Step a1: distinguish the target audio frame from a speech frame and a noise frame.

[0112] In a possible implementation, the target audio frame can be distinguished as a speech frame or a noise frame according to the ideal ratio masks corresponding to the audio frames in the forward adjacent frame sequence of the target audio frame, wherein the forward adjacent frame sequence of the target audio frame is the N consecutive audio frames before the target audio frame, N is an integer greater than 1, and the specific value of N can be determined according to the actual application scenario.

[0113] Specifically, according to the ideal ratio masks corresponding to each audio frame in the forward adjacent frame sequence of the target audio frame, the process of distinguishing the target audio frame as a speech frame or a noise frame may include: if the ideal ratio masks corresponding to each audio frame in the forward adjacent frame sequence of the target audio frame are all less than or equal to the set threshold, the target audio frame is determined to be a noise frame; if the ideal ratio mask corresponding to at least one audio frame in the forward adjacent frame sequence of the target audio frame is greater than the set threshold, the target audio frame is determined to be a speech frame. It should be noted that if the number of audio frames before the target audio frame is less than N, the target audio frame is determined to be a speech frame. The above embodiment mentioned that the ideal ratio mask corresponding to an audio frame includes the ideal ratio masks corresponding to the audio frame at multiple frequency points. If the average of the ideal ratio masks corresponding to the audio frame at multiple frequency points is less than or equal to the set threshold, the ideal ratio mask corresponding to the audio frame is determined to be less than or equal to the set threshold.

[0114] Exemplarily, N is 5, assuming that the target audio frame is the 8th audio frame of the noisy multi-channel audio, then the forward adjacent frame sequence of the target audio frame is a frame sequence consisting of the 7th audio frame, the 6th audio frame, the 5th audio frame, the 4th audio frame, and the 3rd audio frame of the noisy multi-channel audio. If the ideal ratio masks corresponding to the 7th audio frame, the 6th audio frame, the 5th audio frame, the 4th audio frame, and the 3rd audio frame of the noisy multi-channel audio are respectively less than or equal to the set threshold, then the 8th audio frame is determined to be a noise frame; if at least one of the 7th audio frame, the 6th audio frame, the 5th audio frame, the 4th audio frame, and the 3rd audio frame of the noisy multi-channel audio has an ideal ratio mask corresponding to it that is greater than the set threshold, then the 8th audio frame is determined to be a speech frame.

[0115] Step a2: If the target audio frame is a speech frame, the relative transfer function in the beamforming algorithm is updated according to the beam parameters corresponding to the target audio frame; if the target audio frame is a noise frame, the adaptive noise eliminator in the beamforming algorithm is updated according to the beam parameters corresponding to the target audio frame to obtain an updated beamforming algorithm.

[0116] The relative transfer function and the adaptive noise eliminator are two parts of the beamforming algorithm. The relative transfer function is the embodiment of the amplitude difference and phase difference of the microphone signal of the microphone array relative to the main microphone signal. The adaptive noise eliminator is an algorithm that uses adaptive filtering to eliminate noise. The purpose of updating the relative transfer function and the adaptive noise eliminator is to track the changes in speech and noise in time during the change of the audio signal to achieve adaptive noise reduction.

[0117] Step a3: Use the updated beamforming algorithm to perform beamforming processing on the target audio frame to obtain an enhanced single-channel audio signal corresponding to the target audio frame.

[0118] In another embodiment of the present application, the specific implementation process of "Step S404: using the pre-trained audio noise reduction model to perform noise reduction processing on the enhanced single-channel audio signal corresponding to the target audio frame, to obtain a noise-reduced audio frame corresponding to the target audio frame" is introduced.

[0119] In a possible implementation, the process of performing noise reduction processing on the enhanced single-channel audio signal corresponding to the target audio frame by using the pre-trained audio noise reduction model to obtain the noise-reduced audio frame corresponding to the target audio frame may include:

[0120] Step b1: Obtain an amplitude spectrum or a logarithmic amplitude spectrum and a phase spectrum according to the enhanced single-channel audio signal corresponding to the target audio frame.

[0121] It should be noted that the enhanced single-channel audio signal obtained by beamforming the target audio frame is a frequency domain signal, that is, the enhanced single-channel audio signal corresponding to the target audio frame is a frequency domain signal, and the amplitude spectrum and phase spectrum of the enhanced single-channel audio signal corresponding to the target audio frame can be extracted. Optionally, after obtaining the amplitude spectrum, the obtained amplitude spectrum can be logarithmized to obtain a logarithmic amplitude spectrum.

[0122] Step b2: input the acquired amplitude spectrum or logarithmic amplitude spectrum into a pre-trained audio denoising model to obtain a denoised amplitude spectrum output by the audio denoising model.

[0123] The acquired amplitude spectrum or logarithmic amplitude spectrum is input into a pre-trained audio denoising model, and the audio denoising model performs denoising on the input amplitude spectrum or logarithmic amplitude spectrum, and outputs the denoised amplitude spectrum.

[0124] Step b3: performing an inverse short-time Fourier transform on the frequency domain signal consisting of the phase spectrum and the noise-reduced amplitude spectrum obtained in step b1 to obtain a noise-reduced audio frame corresponding to the target audio frame.

[0125] An inverse short-time Fourier transform is performed on the frequency domain signal composed of the phase spectrum obtained in step b1 and the denoised amplitude spectrum output by the audio denoising model to obtain a time domain audio signal, that is, a denoised audio frame corresponding to the target audio frame.

[0126] The above embodiment mentioned that the above audio denoising model is obtained by training with noisy multi-channel audio and training clean multi-channel audio corresponding to the noisy multi-channel audio. Next, the training process of the audio denoising model is introduced.

[0127] Before introducing the training process of the audio denoising model, the process of obtaining the training noisy multi-channel audio and the training clean multi-channel audio corresponding to the training noisy multi-channel audio is first introduced.

[0128] The process of obtaining training noisy multi-channel audio and training clean multi-channel audio corresponding to the training noisy multi-channel audio may include: first, using a microphone array (such as a 6-channel microphone array) to collect clean multi-channel audio (the clean multi-channel audio here can be understood as audio that is not pure, and usually also includes some noise) and noise (such as scattered noise, point source noise, etc.); processing the collected clean multi-channel audio through voice activity detection (VAD) to obtain clean multi-channel audio after VAD processing; simulating room impulse responses of different scenarios through the image source method; convolving the clean multi-channel audio after VAD processing with the room impulse response, and obtaining the audio as the training clean multi-channel audio, superimposing noise (such as scattered noise, point source noise) on the training clean multi-channel audio to obtain training noisy multi-channel audio, wherein the training clean multi-channel audio and the noise can be mixed according to a certain signal-to-noise ratio to obtain the training noisy multi-channel audio.

[0129] Next, we will introduce the training process of the audio denoising model.

[0130] The training process of the audio denoising model includes:

[0131] Step c1: Determine an ideal ratio mask corresponding to each audio frame of the training noisy multi-channel audio according to the training clean multi-channel audio and noise.

[0132] The training clean multi-channel audio and noise in step c1 are signals used to construct training noisy multi-channel audio.

[0133] The ideal ratio mask IRM(i,f) corresponding to the i-th audio frame of the training noisy multi-channel audio at the frequency point f can be calculated by the following formula:

[0134] IRM(i,f)=|S(i,f)| 2 / (|S(i,f)| 2 +|N(i,f)| 2 ) (4)

[0135] Among them, |S(i,f)| represents the frequency domain magnitude spectrum of the i-th audio frame of the training clean multi-channel audio, and |N(i,f)| represents the frequency domain magnitude spectrum of the i-th audio frame of the noise.

[0136] The ideal ratio masks corresponding to the i-th audio frame of the training noisy multi-channel audio at each frequency point constitute the ideal ratio mask IRM(i) corresponding to the i-th audio frame of the training noisy multi-channel audio.

[0137] Step c2: determining beam parameters corresponding to each audio frame of the training noisy multi-channel audio according to the ideal ratio mask corresponding to each audio frame of the training noisy multi-channel audio.

[0138] For the i-th audio frame of the training noisy multi-channel audio: determine the beam parameters corresponding to the i-th audio frame according to the ideal ratio mask corresponding to the i-th audio frame.

[0139] Step c3: According to the beam parameters corresponding to each audio frame of the training noisy multi-channel audio, beamforming processing is performed on the training noisy multi-channel audio and the training clean multi-channel audio frame by frame to obtain enhanced single-channel audio signals corresponding to the training noisy multi-channel audio and the training clean multi-channel audio, respectively.

[0140] For the i-th audio frame of the training noisy multi-channel audio: the i-th audio frame is distinguished as a speech frame or a noise frame. If the i-th audio frame is a speech frame, the relative transfer function in the beamforming algorithm is updated according to the beam parameters corresponding to the i-th audio frame. If the i-th audio frame is a noise frame, the adaptive noise canceller in the beamforming algorithm is updated according to the beam parameters corresponding to the i-th audio frame to obtain an updated beamforming algorithm. The updated beamforming algorithm is used to perform beamforming processing on the i-th audio frame of the training noisy multi-channel audio to obtain an enhanced single-channel audio signal corresponding to the i-th audio frame of the training noisy multi-channel audio. At the same time, the updated beamforming algorithm is used to perform beamforming processing on the i-th audio frame of the training clean multi-channel audio to obtain an enhanced single-channel audio signal corresponding to the i-th audio frame of the training clean multi-channel audio.

[0141] Step c4, training the audio denoising model with the goal of making the denoised audio signal obtained by performing frame-by-frame denoising on the enhanced single-channel audio signal corresponding to the training noisy multi-channel audio be close to the enhanced single-channel audio signal corresponding to the training clean multi-channel audio.

[0142] Specifically, an amplitude spectrum or a logarithmic amplitude spectrum is obtained according to the enhanced single-channel audio signal corresponding to the training noisy multi-channel audio, and the amplitude spectrum (or logarithmic amplitude spectrum) of the enhanced single-channel audio signal corresponding to the training clean multi-channel audio is obtained, and the amplitude spectrum or logarithmic amplitude spectrum obtained according to the enhanced single-channel audio signal corresponding to the training noisy multi-channel audio is input into the audio denoising model to obtain the denoised amplitude spectrum output by the audio denoising model, and a first loss function (such as a mean square error loss function) is determined according to the denoised amplitude spectrum output by the audio denoising model (the logarithm of which can be taken to obtain the logarithmic amplitude spectrum) and the amplitude spectrum (or logarithmic amplitude spectrum) of the enhanced single-channel audio signal corresponding to the training clean multi-channel audio, and the parameters of the audio denoising model are updated according to the first loss function.

[0143] In another possible implementation, in addition to determining the first loss function based on the denoised amplitude spectrum output by the audio denoising model (the logarithm of which can be taken to obtain the logarithmic amplitude spectrum) and the amplitude spectrum (or logarithmic amplitude spectrum) of the enhanced single-channel audio signal corresponding to the training clean multi-channel audio, an inverse short-time Fourier transform may be performed on the frequency domain signal composed of the denoised amplitude spectrum output by the audio denoising model and the phase spectrum of the enhanced single-channel audio signal corresponding to the training noisy multi-channel audio, and an inverse short-time Fourier transform may be performed on the enhanced single-channel audio signal corresponding to the training clean multi-channel audio to obtain two time domain signals, and a second loss function (such as a scale-invariant signal-to-noise ratio loss function) is calculated based on the two time domain signals, and then the parameters of the audio denoising model are updated based on the first loss function and the second loss.

[0144] Of course, it is also possible to only determine the second loss function mentioned above, and then update the parameters of the audio denoising model according to the second loss function.

[0145] The audio denoising model is trained multiple times iteratively using the training data in the training data set according to the above training method until the training end conditions are met (for example, the model converges, the set number of training iterations is reached, etc.).

[0146] The audio noise reduction model in this embodiment may be, but is not limited to, FSMN (Feedforward Sequential Memory Networks). FSMN is improved over traditional DNN (Deep Neural Networks) by adding a memory module to the hidden layer to store node information at past moments. Therefore, relevant information of the time series can be effectively utilized. Compared with the RNN network, no feedback connection is required, so the causality of the network can be maintained.

[0147] The audio noise reduction method provided in the embodiment of the present application is a noise reduction scheme that combines adaptive beamforming with an audio noise reduction model (neural network). When performing noise reduction on a target audio frame, the scheme uses the audio noise reduction model to determine an ideal ratio mask corresponding to the previous audio frame based on the noise reduction result of the previous audio frame, and then determines the beam parameters corresponding to the target audio frame according to the ideal ratio mask corresponding to the previous audio frame, and then performs beamforming processing on the target audio frame according to the beam parameters corresponding to the target audio frame. The audio signal obtained by the beamforming processing is transmitted to the audio noise reduction model for deep post-filtering processing, and the processing result For noise reduction of the next audio frame, a closed-loop system is established in this way, which promotes close cooperation between adaptive beamforming and neural network noise reduction. Since the ideal ratio mask corresponding to the previous audio frame is determined based on the noise reduction result of the previous audio frame, relatively accurate beam parameters can be determined based on the ideal ratio mask corresponding to the previous audio frame. Then, beamforming processing is performed based on relatively accurate beam parameters to obtain an audio signal with significantly improved signal quality. The audio signal with significantly improved signal quality is further subjected to noise reduction by an audio noise reduction model to obtain an audio signal with further improved signal quality. The audio noise reduction method provided in the embodiment of the present application fully combines the advantages of adaptive beamforming and neural network technology, and can significantly improve the audio noise reduction effect.

[0148] The above describes the audio noise reduction method provided in the embodiments of the present application. The following describes a device corresponding to the above audio noise reduction method.

[0149] See also Figure 6 , Figure 6 This is a structural schematic diagram of an audio noise reduction device provided in an embodiment of the present application. The audio noise reduction device may include: an ideal ratio mask determination module 601, a beam parameter determination module 602, a beamforming processing module 603 and an audio noise reduction module 604.

[0150] The ideal ratio mask determination module 601 is used to determine the ideal ratio mask corresponding to the previous audio frame of the target audio frame according to the denoised audio frame corresponding to the previous audio frame of the target audio frame, wherein the target audio frame is an audio frame to be denoised in the noisy multi-channel audio.

[0151] The beam parameter determination module 602 is used to determine the beam parameters corresponding to the target audio frame according to the ideal ratio mask corresponding to the previous audio frame of the target audio frame.

[0152] The beamforming processing module 603 is used to perform beamforming processing on the target audio frame according to the beam parameters corresponding to the target audio frame to obtain an enhanced single-channel audio signal corresponding to the target audio frame.

[0153] The audio noise reduction module 604 is used to use the pre-trained audio noise reduction model to perform noise reduction processing on the enhanced single-channel audio signal corresponding to the target audio frame to obtain a noise-reduced audio frame corresponding to the target audio frame.

[0154] In a possible implementation, when the ideal ratio mask determining module 601 determines the ideal ratio mask corresponding to the previous audio frame of the target audio frame according to the noise-reduced audio frame corresponding to the previous audio frame of the target audio frame, it is specifically used to:

[0155] An ideal ratio mask corresponding to the previous audio frame of the target audio frame is determined according to a frequency domain amplitude spectrum of a denoised audio frame corresponding to the previous audio frame of the target audio frame and an amplitude spectrum of an enhanced single-channel audio signal obtained by performing beamforming processing on the previous audio frame of the target audio frame.

[0156] In a possible implementation, when the beam parameter determination module 602 determines the beam parameter corresponding to the target audio frame according to the ideal ratio mask corresponding to the previous audio frame of the target audio frame, it is specifically used to: determine the speech existence probability of the target audio frame according to the ideal ratio mask corresponding to the previous audio frame of the target audio frame;

[0157] A speech covariance matrix of the target audio frame is determined according to an ideal ratio mask corresponding to an audio frame preceding the target audio frame and a speech covariance matrix of an audio frame preceding the target audio frame.

[0158] In a possible implementation, when the beamforming processing module 603 performs beamforming processing on the target audio frame according to the beam parameters corresponding to the target audio frame to obtain the enhanced single-channel audio signal corresponding to the target audio frame, it is specifically used to:

[0159] Distinguish the target audio frame into a speech frame and a noise frame;

[0160] If the target audio frame is a speech frame, a relative transfer function in the beamforming algorithm is updated according to the beam parameters corresponding to the target audio frame; if the target audio frame is a noise frame, an adaptive noise canceller in the beamforming algorithm is updated according to the beam parameters corresponding to the target audio frame to obtain an updated beamforming algorithm;

[0161] The updated beamforming algorithm is used to perform beamforming processing on the target audio frame to obtain an enhanced single-channel audio signal corresponding to the target audio frame.

[0162] In a possible implementation, when the beamforming processing module 603 distinguishes the target audio frame from the speech frame and the noise frame, it is specifically used to:

[0163] According to the ideal ratio masks corresponding to each audio frame of the forward adjacent frame sequence of the target audio frame, the target audio frame is distinguished as a speech frame or a noise frame, wherein the forward adjacent frame sequence of the target audio frame is a frame sequence composed of N consecutive audio frames before the target audio frame, and N is an integer greater than 1.

[0164] In a possible implementation, the beamforming processing module 603 is specifically used to distinguish the target audio frame from the speech frame and the noise frame according to the ideal ratio masks corresponding to the audio frames of the forward adjacent frame sequence of the target audio frame:

[0165] If the ideal ratio masks corresponding to the audio frames in the forward adjacent frame sequence of the target audio frame are all less than or equal to the set threshold, the target audio frame is determined to be a noise frame;

[0166] If the ideal ratio mask corresponding to at least one audio frame in the forward adjacent frame sequence of the target audio frame is greater than the set threshold, the target audio frame is determined to be a speech frame.

[0167] The enhanced single-channel audio signal corresponding to the target audio frame is a frequency domain signal.

[0168] In a possible implementation, the audio noise reduction module 604 performs noise reduction processing on the enhanced single-channel audio signal corresponding to the target audio frame using the pre-trained audio noise reduction model to obtain a noise-reduced audio frame corresponding to the target audio frame, specifically for:

[0169] Obtaining an amplitude spectrum or a logarithmic amplitude spectrum and a phase spectrum according to the enhanced single-channel audio signal corresponding to the target audio frame;

[0170] Input the acquired amplitude spectrum or logarithmic amplitude spectrum into a pre-trained audio denoising model to obtain a denoised amplitude spectrum output by the audio denoising model;

[0171] An inverse short-time Fourier transform is performed on the frequency domain signal composed of the phase spectrum and the noise-reduced amplitude spectrum to obtain a noise-reduced audio frame corresponding to the target audio frame.

[0172] In a possible implementation, the audio noise reduction model is trained by using training noisy multi-channel audio and training clean multi-channel audio corresponding to the training noisy multi-channel audio, and the training noisy multi-channel audio is obtained by superimposing noise on the training clean multi-channel audio;

[0173] The audio noise reduction device may further include: a model training module. The model training module is used to:

[0174] Determining an ideal ratio mask corresponding to each audio frame of the training noisy multi-channel audio according to the training clean multi-channel audio and the noise;

[0175] Determining beam parameters corresponding to each audio frame of the training noisy multi-channel audio according to an ideal ratio mask corresponding to each audio frame of the training noisy multi-channel audio;

[0176] According to the beam parameters corresponding to each audio frame of the training noisy multi-channel audio, the training noisy multi-channel audio and the training clean multi-channel audio are respectively subjected to beamforming processing frame by frame to obtain enhanced single-channel audio signals corresponding to the training noisy multi-channel audio and the training clean multi-channel audio, respectively;

[0177] The audio denoising model is trained with the goal of making the denoised audio signal obtained by performing frame-by-frame denoising on the enhanced single-channel audio signal corresponding to the training noisy multi-channel audio be close to the enhanced single-channel audio signal corresponding to the training clean multi-channel audio.

[0178] The audio noise reduction device provided in the embodiment of the present application first determines an ideal ratio mask corresponding to the previous audio frame of the target audio frame according to the denoised audio frame corresponding to the previous audio frame of the target audio frame, then determines the beam parameters corresponding to the target audio frame according to the ideal ratio mask corresponding to the previous audio frame of the target audio frame, then performs beamforming processing on the target audio frame according to the beam parameters corresponding to the target audio frame to obtain an enhanced single-channel audio signal corresponding to the target audio frame, and finally uses a pre-trained audio noise reduction model to perform noise reduction processing on the enhanced single-channel audio signal corresponding to the target audio frame to obtain a denoised audio frame corresponding to the target audio frame. The audio noise reduction device provided in the embodiment of the present application is a noise reduction device that combines adaptive beamforming with an audio noise reduction model. The device accurately determines the beam parameters corresponding to the target audio frame by means of an ideal ratio mask determined according to the denoised audio frame corresponding to the previous audio frame of the target audio frame, and then performs beamforming processing on the target audio frame according to the beam parameters corresponding to the target audio frame. By performing beamforming processing on the target audio frame, an audio signal with directionality enhancement and improved signal-to-noise ratio can be obtained. The audio signal is further denoised by the audio noise reduction model to obtain an audio signal with directionality enhancement and higher signal-to-noise ratio. It can be seen that the audio noise reduction device provided in the present application can effectively suppress noise in the audio and has a good noise reduction effect.

[0179] An embodiment of the present application also provides an electronic device, which may include: at least one processor, at least one communication interface, at least one memory and at least one communication bus.

[0180] In the embodiment of the present application, the number of the processor, the communication interface, the memory, and the communication bus is at least one, and the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0181] The processor may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application;

[0182] The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0183] The memory stores a program, and the processor can call the program stored in the memory, where the program is used to implement the steps of the audio noise reduction method provided in the above embodiment.

[0184] An embodiment of the present application also provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the audio noise reduction method provided in the above embodiment.

[0185] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of the audio noise reduction method provided in the above embodiment.

[0186] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0187] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0188] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0189] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. An audio noise reduction method, characterized in that: include: Determine, according to a denoised audio frame corresponding to a previous audio frame of a target audio frame, an ideal ratio mask corresponding to the previous audio frame of the target audio frame, wherein the target audio frame is an audio frame to be denoised in noisy multi-channel audio; Determining beam parameters corresponding to the target audio frame according to an ideal ratio mask corresponding to an audio frame preceding the target audio frame; Performing beamforming processing on the target audio frame according to the beam parameters corresponding to the target audio frame to obtain an enhanced single-channel audio signal corresponding to the target audio frame; Using a pre-trained audio noise reduction model, noise reduction processing is performed on the enhanced single-channel audio signal corresponding to the target audio frame to obtain a noise-reduced audio frame corresponding to the target audio frame.

2. The speech noise reduction method according to claim 1, characterized in that: The step of determining, according to the noise-reduced audio frame corresponding to the previous audio frame of the target audio frame, the ideal ratio mask corresponding to the previous audio frame of the target audio frame comprises: An ideal ratio mask corresponding to the previous audio frame of the target audio frame is determined according to a frequency domain amplitude spectrum of a denoised audio frame corresponding to the previous audio frame of the target audio frame and an amplitude spectrum of an enhanced single-channel audio signal obtained by performing beamforming processing on the previous audio frame of the target audio frame.

3. The speech noise reduction method according to claim 1, characterized in that: The step of determining the beam parameter corresponding to the target audio frame according to the ideal ratio mask corresponding to the previous audio frame of the target audio frame comprises: determining the speech existence probability of the target audio frame according to the ideal ratio mask corresponding to the previous audio frame of the target audio frame; A speech covariance matrix of the target audio frame is determined according to an ideal ratio mask corresponding to an audio frame preceding the target audio frame and a speech covariance matrix of an audio frame preceding the target audio frame.

4. The speech noise reduction method according to claim 1, characterized in that: The step of performing beamforming processing on the target audio frame according to the beam parameters corresponding to the target audio frame to obtain an enhanced single-channel audio signal corresponding to the target audio frame includes: Distinguishing the target audio frame into a speech frame and a noise frame; If the target audio frame is a speech frame, a relative transfer function in the beamforming algorithm is updated according to the beam parameters corresponding to the target audio frame; if the target audio frame is a noise frame, an adaptive noise canceller in the beamforming algorithm is updated according to the beam parameters corresponding to the target audio frame to obtain an updated beamforming algorithm; The updated beamforming algorithm is used to perform beamforming processing on the target audio frame to obtain an enhanced single-channel audio signal corresponding to the target audio frame.

5. The speech noise reduction method according to claim 4, characterized in that: The distinguishing the target audio frame between a speech frame and a noise frame includes: The target audio frame is distinguished as a speech frame or a noise frame according to ideal ratio masks respectively corresponding to each audio frame of a forward adjacent frame sequence of the target audio frame, wherein the forward adjacent frame sequence of the target audio frame is a frame sequence consisting of N consecutive audio frames before the target audio frame, and N is an integer greater than 1.

6. The speech noise reduction method according to claim 5, characterized in that: The step of distinguishing a speech frame from a noise frame on the target audio frame according to the ideal ratio masks respectively corresponding to the audio frames in the forward adjacent frame sequence of the target audio frame comprises: If the ideal ratio masks respectively corresponding to the audio frames in the forward adjacent frame sequence of the target audio frame are all less than or equal to the set threshold, the target audio frame is determined to be a noise frame; If the ideal ratio mask corresponding to at least one audio frame in the forward adjacent frame sequence of the target audio frame is greater than the set threshold, the target audio frame is determined to be a speech frame.

7. The speech noise reduction method according to claim 1, characterized in that: The enhanced single-channel audio signal corresponding to the target audio frame is a frequency domain signal; The method of using the pre-trained audio noise reduction model to perform noise reduction processing on the enhanced single-channel audio signal corresponding to the target audio frame to obtain a noise-reduced audio frame corresponding to the target audio frame includes: Obtaining an amplitude spectrum or a logarithmic amplitude spectrum and a phase spectrum according to the enhanced single-channel audio signal corresponding to the target audio frame; Inputting the acquired amplitude spectrum or logarithmic amplitude spectrum into a pre-trained audio denoising model to obtain a denoised amplitude spectrum output by the audio denoising model; An inverse short-time Fourier transform is performed on the frequency domain signal composed of the phase spectrum and the noise-reduced amplitude spectrum to obtain a noise-reduced audio frame corresponding to the target audio frame.

8. The speech noise reduction method according to claim 1, characterized in that: The audio noise reduction model is trained by using training noisy multi-channel audio and training clean multi-channel audio corresponding to the training noisy multi-channel audio, wherein the training noisy multi-channel audio is obtained by superimposing noise on the training clean multi-channel audio; The training process of the audio noise reduction model includes: Determining an ideal ratio mask corresponding to each audio frame of the training noisy multi-channel audio according to the training clean multi-channel audio and the noise; Determining beam parameters corresponding to each audio frame of the training noisy multi-channel audio according to an ideal ratio mask corresponding to each audio frame of the training noisy multi-channel audio; According to the beam parameters corresponding to each audio frame of the training noisy multi-channel audio, the training noisy multi-channel audio and the training clean multi-channel audio are respectively subjected to beamforming processing frame by frame to obtain enhanced single-channel audio signals corresponding to the training noisy multi-channel audio and the training clean multi-channel audio, respectively; The audio denoising model is trained with the goal of making the denoised audio signal obtained by performing frame-by-frame denoising on the enhanced single-channel audio signal corresponding to the training noisy multi-channel audio be close to the enhanced single-channel audio signal corresponding to the training clean multi-channel audio.

9. An audio noise reduction device, characterized in that: include: An ideal ratio mask determination module, a beam parameter determination module, a beam forming processing module and an audio noise reduction module; The ideal ratio mask determination module is used to determine the ideal ratio mask corresponding to the previous audio frame of the target audio frame according to the denoised audio frame corresponding to the previous audio frame of the target audio frame, wherein the target audio frame is an audio frame to be denoised in the noisy multi-channel audio; The beam parameter determination module is used to determine the beam parameter corresponding to the target audio frame according to the ideal ratio mask corresponding to the previous audio frame of the target audio frame; The beamforming processing module is used to perform beamforming processing on the target audio frame according to the beam parameters corresponding to the target audio frame to obtain an enhanced single-channel audio signal corresponding to the target audio frame; The audio noise reduction module is used to use a pre-trained audio noise reduction model to perform noise reduction processing on the enhanced single-channel audio signal corresponding to the target audio frame to obtain a noise-reduced audio frame corresponding to the target audio frame.

10. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the steps of the audio noise reduction method as described in any one of claims 1 to 8.

11. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the audio noise reduction method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the steps of the audio noise reduction method as described in any one of claims 1 to 8.

Images