Audio noise-reduction processing method and apparatus, storage medium, and electronic device

Abstract

An audio noise-reduction processing method and apparatus, a storage medium, and an electronic device. The method comprises: obtaining an audio signal to be processed (S202); performing frequency-domain conversion processing on said audio signal to obtain a noisy frequency-domain representation corresponding to said audio signal (S204); dividing the noisy frequency-domain representation into N noisy frequency bands, and respectively inputting the N noisy frequency bands into N noise-reduction branches in an audio processing network to obtain N branch mask estimation results, wherein an i-th noise-reduction branch in the audio processing network is used for processing an i-th noisy frequency band among the N noisy frequency bands, so as to obtain an i-th branch mask estimation result corresponding to the i-th noisy frequency band (S206); using the noisy frequency-domain representation to modulate the N branch mask estimation results to obtain N audio frequency-domain representations (S208); and performing time-domain conversion processing on the N audio frequency-domain representations to obtain a voice signal having no noise signal interference in the audio signal (S210). The method solves the technical problem of inaccuracy of an audio noise-reduction processing result.

Description

Audio noise reduction processing method and device, storage medium and electronic device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 30, 2023, with application number 2023111128506 and application name “Audio noise reduction processing method and device, storage medium and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of audio processing technology, and in particular to audio noise reduction technology. Background Art

[0003] Today, to achieve speech enhancement and noise reduction for noisy audio signals, the audio signal is typically sampled at a single sampling rate and then further processed based on the specific application scenario. For example, if a full-band speech enhancement method is used to process a wideband signal, the audio signal needs to be upsampled and the high-frequency components set to zero, but this introduces unnecessary additional computational effort. If a wideband speech enhancement method is used to process a full-band signal, the audio signal needs to be downsampled, but this loses high-frequency information.

[0004] That is to say, the audio noise reduction method provided by the related art has the problem of inaccurate noise reduction processing results.

[0005] To address the above-mentioned problems, no effective solutions have been proposed so far.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide an audio noise reduction processing method and device, a storage medium, and an electronic device to at least solve the technical problem of inaccurate processing results of audio noise reduction processing.

[0008] According to one aspect of an embodiment of the present application, a method for audio noise reduction processing is provided, which is performed by an electronic device and includes:

[0009] Acquiring an audio signal to be processed, wherein the audio signal includes a speech signal to be recognized that has been interfered with by a noise signal;

[0010] Performing frequency domain conversion processing on the audio signal to obtain a noisy frequency domain representation corresponding to the audio signal;

[0011] Dividing the noisy frequency domain representation into N noisy frequency bands, and inputting the N noisy frequency bands into N noise reduction branches in the audio processing network respectively, to obtain N branch mask estimation results, wherein the i-th noise reduction branch in the audio processing network is used to process the i-th noisy frequency band among the N noisy frequency bands to obtain the i-th branch mask estimation result corresponding to the i-th noisy frequency band, the N noise reduction branches have the same signal processing structure, i is a natural number greater than or equal to 1 and less than or equal to N, and N is a natural number greater than 1;

[0012] Using the noisy frequency domain representation, modulate the N branch mask estimation results to obtain N speech frequency domain representations;

[0013] The N speech frequency domain representations are transformed into the time domain to obtain a speech signal without interference from noise signals in the audio signal.

[0014] According to another aspect of the embodiments of the present application, there is also provided an audio noise reduction processing device, comprising:

[0015] an acquiring unit, configured to acquire an audio signal to be processed, wherein the audio signal includes a speech signal to be recognized that has been interfered with by a noise signal;

[0016] An extraction unit, configured to perform frequency domain conversion processing on the audio signal to obtain a noisy frequency domain representation corresponding to the audio signal;

[0017] an input unit, configured to divide the noisy frequency domain representation into N noisy frequency bands, and input the N noisy frequency bands into N noise reduction branches in the audio processing network, respectively, to obtain N branch mask estimation results, wherein the i-th noise reduction branch in the audio processing network is configured to process the i-th noisy frequency band of the N noisy frequency bands to obtain the i-th branch mask estimation result corresponding to the i-th noisy frequency band, the N noise reduction branches having the same signal processing structure, i being a natural number greater than or equal to 1 and less than or equal to N, and N being a natural number greater than 1;

[0018] A modulation unit, configured to modulate the N branch mask estimation results using the noisy frequency domain representation to obtain N voice frequency domain representations;

[0019] The conversion unit is used to perform time domain conversion processing on the N audio domain representations of speech to obtain a speech signal without interference from a noise signal in the audio signal.

[0020] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the above-mentioned audio noise reduction processing method when running.

[0021] According to another aspect of the embodiments of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described audio noise reduction processing method.

[0022] According to another aspect of the embodiments of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the audio noise reduction processing method through the computer program.

[0023] In an embodiment of the present application, an audio signal to be processed is obtained, wherein the audio signal contains a speech signal to be identified that has been interfered with by a noise signal. Then, the audio signal is subjected to frequency domain conversion processing to obtain a noisy frequency domain representation corresponding to the audio signal. Next, the noisy frequency domain representation is divided into N noisy frequency bands, and the N noisy frequency bands are respectively input into N noise reduction branches in the audio processing network to obtain N branch mask estimation results, wherein the i-th noise reduction branch in the audio processing network is used to process the i-th noisy frequency band among the N noisy frequency bands to obtain the i-th branch mask estimation result corresponding to the i-th noisy frequency band, and the N noise reduction branches have the same signal processing structure, i is a natural number greater than or equal to 1 and less than or equal to N, and N is a natural number greater than 1. Furthermore, the N branch mask estimation results are modulated using the noisy frequency domain representation to obtain N speech frequency domain representations. Thus, the N speech frequency domain representations are subjected to time domain conversion processing to obtain a speech signal without noise signal interference in the audio signal. In other words, in an embodiment of the present application, multiple noise reduction branches are used to perform noise reduction processing on multiple noisy frequency domain segments corresponding to the audio signal, respectively, to obtain branch mask estimation results corresponding to different noisy frequency segments, and then the branch mask estimation results are used to modulate the noisy frequency domain representation, and the modulated speech signal is subjected to time domain conversion processing, thereby obtaining a speech signal that is not interfered with by the noise signal. Compared with the prior art that uses an audio signal processing model for a fixed sampling rate, the embodiment of the present application simultaneously uses multiple noise reduction branches to perform noise reduction processing on multiple different noisy frequency segments, and uses the obtained branch mask estimation results to generate the corresponding speech signal, which can directly obtain a speech signal without noise signal interference in the required frequency segment, thereby avoiding the interference caused by the introduction of intermediate links when using an audio signal processing model with a fixed sampling rate to perform noise reduction processing on the audio signal, and thus avoiding the occurrence of inaccurate noise reduction processing results. Thus, the technical effect of improving the accuracy of audio signal noise reduction processing is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] FIG1 is a schematic diagram of an application environment of an optional audio noise reduction processing method according to an embodiment of the present application;

[0026] FIG2 is a flowchart of an optional audio noise reduction processing method according to an embodiment of the present application;

[0027] FIG3 is a schematic diagram of an optional audio noise reduction processing method according to an embodiment of the present application;

[0028] FIG4 is a schematic diagram of another optional audio noise reduction processing method according to an embodiment of the present application;

[0029] FIG5 is a schematic diagram of another optional audio noise reduction processing method according to an embodiment of the present application;

[0030] FIG6 is a schematic diagram of another optional audio noise reduction processing method according to an embodiment of the present application;

[0031] FIG7 is a schematic diagram of another optional audio noise reduction processing method according to an embodiment of the present application;

[0032] FIG8 is a schematic diagram of another optional audio noise reduction processing method according to an embodiment of the present application;

[0033] FIG9 is a schematic diagram of another optional audio noise reduction processing method according to an embodiment of the present application;

[0034] FIG10 is a schematic diagram of another optional audio noise reduction processing method according to an embodiment of the present application;

[0035] FIG11 is a schematic diagram of another optional audio noise reduction processing method according to an embodiment of the present application;

[0036] FIG12 is a schematic diagram of another optional audio noise reduction processing method according to an embodiment of the present application;

[0037] FIG13 is a schematic diagram of another optional audio noise reduction processing method according to an embodiment of the present application;

[0038] FIG14 is a schematic structural diagram of another optional audio noise reduction processing device according to an embodiment of the present application;

[0039] FIG15 is a schematic structural diagram of another optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0042] According to one aspect of an embodiment of the present application, an audio noise reduction processing method is provided. Optionally, as an optional implementation, the audio noise reduction processing method can be, but is not limited to, applied in the environment shown in FIG1 . As shown in FIG1 , a terminal device 102 includes a memory 104 (for storing various data generated during the operation of the terminal device 102), a processor 106 (for processing and calculating the aforementioned various data), and a display 108. The terminal device 102 can exchange data with a server 112 via a network 110. The server 112 is connected to a database 114, which is used to store various data.

[0043] Furthermore, the specific application process of the above method in the environment shown in FIG1 is shown in the following steps:

[0044] Steps S102 to S104 are executed, and the terminal device 102 obtains an audio signal to be processed, wherein the audio signal includes a speech signal to be recognized that has been interfered with by a noise signal, and sends the audio signal to the server 112 via the network 110 .

[0045] Then, steps S106-S112 are executed. Server 112 performs frequency domain conversion processing on the audio signal to obtain a noisy frequency domain representation corresponding to the audio signal. Server 112 divides the noisy frequency domain representation into N noisy frequency bands and inputs each of the N noisy frequency bands into N noise reduction branches within the audio processing network to obtain N branch mask estimation results. The i-th noise reduction branch in the audio processing network is used to process the i-th noisy frequency band to obtain the i-th branch mask estimation result corresponding to the i-th noisy frequency band. The N noise reduction branches have the same signal processing structure, where i is a natural number greater than or equal to 1 and less than or equal to N, and N is a natural number greater than 1. Server 112 uses the noisy frequency domain representation to modulate the N branch mask estimation results to obtain N speech frequency domain representations. Server 112 performs time domain conversion processing on the N speech frequency domain representations to obtain a speech signal from the audio signal without noise interference.

[0046] Then, step S114 is executed, and the server 112 sends the voice signal to the terminal device 102 through the network 110.

[0047] In an embodiment of the present application, an audio signal to be processed is obtained, wherein the audio signal contains a speech signal to be identified that has been interfered with by a noise signal. Then, the audio signal is subjected to frequency domain conversion processing to obtain a noisy frequency domain representation corresponding to the audio signal. Next, the noisy frequency domain representation is divided into N noisy frequency bands, and the N noisy frequency bands are respectively input into N noise reduction branches in the audio processing network to obtain N branch mask estimation results, wherein the i-th noise reduction branch in the audio processing network is used to process the i-th noisy frequency band among the N noisy frequency bands to obtain the i-th branch mask estimation result corresponding to the i-th noisy frequency band, and the N noise reduction branches have the same signal processing structure, i is a natural number greater than or equal to 1 and less than or equal to N, and N is a natural number greater than 1. Furthermore, the N branch mask estimation results are modulated using the noisy frequency domain representation to obtain N speech frequency domain representations. Thus, the N speech frequency domain representations are subjected to time domain conversion processing to obtain a speech signal without noise signal interference in the audio signal. In other words, in the embodiments of the present application, multiple noise reduction branches are used to perform noise reduction processing on multiple noisy frequency domain segments corresponding to the audio signal, thereby obtaining a speech signal that is not interfered with by the noise signal. This avoids the problem of inaccurate noise reduction results caused by the use of an audio signal processing model for a fixed sampling rate in the prior art. This achieves the technical effect of improving the accuracy of the noise reduction processing of the audio signal.

[0048] Optionally, in this embodiment, the above-mentioned terminal device can be a terminal device configured with a target client, which can include but is not limited to at least one of the following: a mobile phone (such as an Android phone, an iOS phone, etc.), a laptop computer, a tablet computer, a PDA, an MID (Mobile Internet Devices), a PAD, a desktop computer, a smart TV, etc. The target client can be a video client, an instant messaging client, a browser client, an education client, etc. The above-mentioned network can include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network and a wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that realize wireless communication. The above-mentioned server can be a single server, or it can be a server cluster composed of multiple servers, or a cloud server. The above is only an example, and this embodiment does not impose any limitation on this.

[0049] Optionally, as an optional implementation, as shown in FIG2 , the audio noise reduction processing method may be performed by an electronic device, and the method includes:

[0050] S202, obtaining an audio signal to be processed, wherein the audio signal includes a speech signal to be recognized that has been interfered with by a noise signal;

[0051] S204, performing frequency domain conversion processing on the audio signal to obtain a noisy frequency domain representation corresponding to the audio signal;

[0052] S206: Divide the noisy frequency domain representation into N noisy frequency bands, and input the N noisy frequency bands into N noise reduction branches in the audio processing network, respectively, to obtain N branch mask estimation results, wherein the i-th noise reduction branch in the audio processing network is used to process the i-th noisy frequency band among the N noisy frequency bands to obtain the i-th branch mask estimation result corresponding to the i-th noisy frequency band, the N noise reduction branches have the same signal processing structure, i is a natural number greater than or equal to 1 and less than or equal to N, and N is a natural number greater than 1;

[0053] S208, using the noisy frequency domain representation, modulate the N branch mask estimation results to obtain N voice frequency domain representations;

[0054] S210 , performing time domain conversion processing on the N audio domain representations of speech to obtain a speech signal without interference from a noise signal in the audio signal.

[0055] It should be noted that the above-mentioned audio noise reduction processing method can be applied to, but is not limited to, audio signal noise reduction processing scenarios for voice calls, video calls, video conferences, cameras, smart home appliances, etc. Assuming that the above-mentioned audio noise reduction processing method is applied to the audio signal noise reduction processing scenario for voice calls, the above-mentioned method can be used to perform noise reduction processing on the audio signal collected during the voice call to obtain a voice signal that is not interfered with by noise. Assuming that the above-mentioned audio noise reduction processing method is applied to the audio signal noise reduction processing scenario for cameras, the above-mentioned method can be used to perform noise reduction processing on the audio signal collected by the camera to obtain a voice signal that is not interfered with by noise. Assuming that the above-mentioned audio noise reduction processing method is applied to the audio signal noise reduction processing scenario for smart home appliances, the above-mentioned method can be used to perform noise reduction processing on the audio signal collected by smart home appliances to obtain a voice signal that is not interfered with by noise.

[0056] Furthermore, the above-mentioned audio signal can be used, but is not limited to, to indicate the original voice signal collected by the terminal device, which includes useless noise and the voice signal to be extracted. For example, assuming that the above-mentioned audio noise reduction processing method is applied to the audio signal noise reduction processing scenario of a voice call, and the current user object A is talking to user object B through terminal device a, then the above-mentioned audio signal can be the audio signal of the environment where user object A is located, which is collected by terminal device a, and the audio signal includes the noise in the environment where user object A is located and the voice uttered by user object A.

[0057] Optionally, in this embodiment, before performing the frequency domain conversion processing on the audio signal to obtain the noisy frequency domain representation corresponding to the audio signal, the audio signal may be subjected to frame segmentation and windowing to prevent spectrum leakage. For example, the audio signal may be segmented into multiple frames of short signals of a fixed length (i.e., 1024) with a frame length of 1024 and a frame shift of 512 (i.e., the overlap length between two adjacent frames is 512). Specifically, the 1024 sampling points starting from the starting audio point of the audio signal may be used as the first frame of the short signal. Then, by shifting 512 sampling points backward, the 1024 sampling points starting from the 513th sampling point of the audio signal may be used as the second frame of the short signal, and so on, until all sampling points in the audio signal are evenly framed into corresponding short signals. A Hamming window is then used to modulate each frame of the audio signal to prevent spectrum leakage.

[0058] It should be noted that the windowing processing method used for the windowing processing of the audio signal is not limited to the Hamming window, and other methods may also be used, such as a rectangular window, a Hanning window, etc., which is not limited in this embodiment.

[0059] Furthermore, the above-mentioned frequency domain conversion processing of the audio signal to obtain a noisy frequency domain representation corresponding to the audio signal may include, but is not limited to, performing a discrete cosine transform (DCT) operation on the audio signal after frame and window processing to obtain the noisy frequency domain characteristics of the audio signal. It should be noted that the process of performing frame and window processing and discrete cosine transform operations on the audio signal is actually the process of performing a short-time discrete cosine transform (SDCT) on the audio signal.

[0060] It should be noted that after performing frame windowing processing on the audio signal, other methods can also be used to obtain a noisy frequency domain representation corresponding to the audio signal, such as using a short-time Fourier transform (STFT) method. In addition, in this embodiment, after performing frame windowing processing on the audio signal, the audio signal can also be converted into other acoustic features for analysis, such as an amplitude spectrum, a power spectrum, a mel spectrum, etc. This is not limited in any way in this embodiment.

[0061] Specifically, 1) The Short-Time Fourier Transform (STFT) is a mathematical transformation related to the Fourier transform that is used to determine the frequency and phase of a localized sinusoidal wave in a time-varying signal. Its core logic involves selecting a window function that localizes time and frequency. Assuming the analysis window function g(t) is stationary (pseudo-stationary) over a short time interval, the window function is shifted so that f(t) and g(t) are stationary signals within different finite time widths, thereby calculating the power spectrum at each moment.

[0062] 2) The discrete cosine transform (DCT) is a transform related to the Fourier transform. It is similar to the discrete Fourier transform (DFT), but uses only real numbers. The DCT is equivalent to a DFT of approximately twice the length. This DCT is performed on a real even function (because the Fourier transform of a real even function is still a real even function). Some transformations require shifting the input or output by half a unit. The basic principle of the DCT formula is to convert a time-domain signal x(n) of length N into a frequency-domain signal X(k) of length N, where k represents the frequency. The DCT formula is: X(k) = ∑[n = 0, N-1]x(n)cos[(π / N)(n+1 / 2)k], where k = 0, 1, 2, ..., N-1. This formula can be regarded as a Fourier transform based on the cosine function, which is used to decompose the time domain signal into the weighted sum of a series of cosine functions to obtain the frequency domain signal.

[0063] 3) The amplitude spectrum is a graph plotting signal amplitude against frequency (angular frequency). In a frequency-domain description of a signal, frequency is used as the independent variable, and the amplitudes of the signal's individual frequency components are used as the dependent variable. This frequency function is called the amplitude spectrum, which describes the distribution of the signal's amplitude over frequency. For frequency-domain descriptions of random signals, the power spectrum is often used, which describes the distribution of the signal's energy over frequency.

[0064] 4) Power spectrum, short for power spectral density function, is defined as the signal power per unit frequency band. It shows how signal power varies with frequency, that is, how the signal power is distributed in the frequency domain. The power spectrum shows how signal power varies with frequency.

[0065] 5) Mel spectrum is a spectrum converted from frequency to Mel scale. Mel spectrum can adapt to human hearing and is widely used in the field of speech.

[0066] Further, assuming that the sampling rate of the above audio signal is 48 kHz, the sampling rate range of the noisy frequency domain representation is [0 kHz, 24 kHz]. The above-mentioned division of the noisy frequency domain representation into N noisy frequency bands may include, but is not limited to: dividing the noisy frequency domain representation into a noisy low frequency band [0, 8 kHz] and a noisy high frequency band (8 kHz, 24 kHz], or dividing the noisy frequency domain representation into a noisy frequency band of [0, 8 kHz], a noisy frequency band of (8 kHz, 16 kHz] and a noisy frequency band of (16 kHz, 24 kHz]. In this embodiment, no limitation is imposed on this.

[0067] It should be noted that, assuming that the above-mentioned N noisy frequency bands include: a noisy low frequency band [0, 8kHz] and a noisy high frequency band (8kHz, 24kHz], then the above-mentioned audio processing network including N noise reduction branches can be but is not limited to including: a two-way audio processing network for modeling and analyzing audio based on the low frequency band [0, 8kHz] and the high frequency band (8kHz, 24kHz] respectively. Assuming that the above-mentioned N noisy frequency bands include: a noisy frequency band of [0, 8kHz], a noisy frequency band of (8kHz, 16kHz] and a noisy frequency band of (16kHz, 24kHz], then the above-mentioned audio processing network including N noise reduction branches can be but is not limited to including: a three-way audio processing network for modeling and analyzing audio based on [0, 8kHz], (8kHz, 16kHz] and (16kHz, 24kHz] respectively.

[0068] Optionally, in an embodiment of the present application, the above-mentioned audio processing network may, but is not limited to, adopt an encoder-decoder (Encoder-Decoder) interactive structure. Specifically, assuming that the above-mentioned N noisy frequency bands include: a noisy low frequency band [0, 8kHz] and a noisy high frequency band (8kHz, 24kHz], then the above-mentioned audio processing network includes two branches, namely a low-frequency branch obtained by training based on the audio information of the low frequency band [0, 8kHz], and a high-frequency branch obtained by training based on the audio information of the high frequency band (8kHz, 24kHz]. In addition, the above-mentioned audio processing network also includes a gating structure for transmitting the information of the low-frequency branch to the high-frequency branch. Among them, the above-mentioned low-frequency branch and the above-mentioned high-frequency branch can adopt an (Encoder-Decoder) interactive structure. It should be noted that the above-mentioned audio processing network can also adopt other structures, not limited to Encoder-Decoder, and no limitation is made to this in this embodiment.

[0069] It should be understood that in the embodiment of the present application, the branch mask estimation result is obtained by processing the corresponding noisy frequency band by the noise reduction branch in the audio processing network. The branch mask estimation result can be used to indicate the validity of the features at each position in the noisy frequency band. For example, for a certain noisy frequency band, the position corresponding to the noise signal in its corresponding branch mask estimation result will be indicated as invalid, while the position corresponding to the voice signal emitted by the user will be indicated as valid.

[0070] Optionally, in this embodiment, the above-mentioned use of the noisy frequency domain representation to modulate the N branch mask estimation results to obtain N voice frequency domain representations may include, but is not limited to, performing a cross product operation on the noisy frequency domain representation and the N branch mask estimation results to obtain voice frequency domain representations of the N branch masks. The modulation processing based on the noisy frequency domain representation and the N branch mask estimation results can essentially be understood as using the branch mask estimation results to enhance the features corresponding to the voice signal emitted by the user in the noisy frequency domain representation and eliminate or weaken the noise features in the noisy frequency domain representation, thereby achieving denoising processing in the frequency domain dimension.

[0071] Optionally, in this embodiment, the aforementioned time-domain conversion processing of the N voice-frequency representations to obtain a speech signal free of noise interference in the audio signal may include, but is not limited to, performing a short-time cosine transform on each voice-frequency representation to obtain a corresponding noise-free speech signal. Of course, in practical applications, other time-domain conversion processing methods may also be used to convert the voice-frequency representation into a corresponding time-domain speech signal. The time-domain conversion processing method used should correspond to the frequency-domain conversion processing method used above. This embodiment of the application does not impose any limitation on the time-domain conversion processing method used.

[0072] As an optional implementation, taking the above-mentioned noisy frequency domain representation as divided into two noisy frequency bands, namely the noisy low frequency band [0, 8kHz] and the noisy high frequency band (8kHz, 24kHz] as an example, the above-mentioned method is illustrated by the following steps as shown in FIG3:

[0073] Obtain an audio signal including noise, perform short-time cosine transform processing on the audio signal to obtain the frequency domain feature X of the audio signal k , which is the above-mentioned noisy frequency domain representation. Next, the frequency domain feature X k Perform segmentation processing to obtain the noisy low-frequency band and high-frequency noise Then, the noisy low frequency band Input to the low frequency noise reduction branch 302 for processing to obtain the mask estimation result corresponding to the noisy low frequency band Furthermore, based on and Modulation processing is performed to obtain a modulation result, and inverse short-time cosine transform processing is performed on the modulation result to obtain a wideband speech signal that is not interfered with by noise.

[0074] The high frequency band with noise The data is input to the high-frequency noise reduction branch 304 for processing, and the guidance information obtained by modulating the data in the low-frequency noise reduction branch 302 using the gate structure 306 is used to assist the operation of the high-frequency noise reduction branch 304 to obtain the mask estimation result corresponding to the noisy high-frequency band. Then, the mask estimation result and mask estimation results Perform splicing processing, based on the mask estimation result after splicing processing and X k Modulation processing is performed to obtain a modulation result, and inverse short-time cosine transform processing is performed on the modulation result to obtain a full-band speech signal that is not interfered with by noise.

[0075] In an embodiment of the present application, an audio signal to be processed is obtained, wherein the audio signal includes a speech signal to be identified that has been interfered with by a noise signal. Then, the audio signal is subjected to frequency domain conversion processing to obtain a noisy frequency domain representation corresponding to the audio signal. Next, the noisy frequency domain representation is divided into N noisy frequency bands, and the N noisy frequency bands are respectively input into N noise reduction branches within an audio processing network to obtain N branch mask estimation results, wherein the i-th noise reduction branch in the audio processing network is used to process the i-th noisy frequency band of the N noisy frequency bands to obtain the i-th branch mask estimation result corresponding to the i-th noisy frequency band. The N noise reduction branches have the same signal processing structure, i is a natural number greater than or equal to 1 and less than or equal to N, and N is a natural number greater than 1. Furthermore, the N branch mask estimation results are modulated using the noisy frequency domain representation to obtain N speech frequency domain representations. Thus, time domain conversion processing is performed on the N speech frequency domain representations to obtain a speech signal extracted from the audio signal without interference from the noise signal. In other words, in an embodiment of the present application, multiple noise reduction branches are used to perform noise reduction processing on multiple noisy frequency domain segments corresponding to the audio signal, respectively, to obtain branch mask estimation results corresponding to different noisy frequency segments, and then the branch mask estimation results are used to modulate the noisy frequency domain representation, and the modulated speech signal is subjected to time domain conversion processing, thereby obtaining a speech signal that is not interfered with by the noise signal. Compared with the prior art that uses an audio signal processing model for a fixed sampling rate, the embodiment of the present application simultaneously uses multiple noise reduction branches to perform noise reduction processing on multiple different noisy frequency segments, and uses the obtained branch mask estimation results to generate the corresponding speech signal, which can directly obtain a speech signal without noise signal interference in the required frequency segment, thereby avoiding the interference caused by the introduction of intermediate links when using an audio signal processing model with a fixed sampling rate to perform noise reduction processing on the audio signal, and thus avoiding the occurrence of inaccurate noise reduction processing results. Thus, the technical effect of improving the accuracy of audio signal noise reduction processing is achieved.

[0076] As an optional solution, N noisy frequency bands are input into N noise reduction branches in the audio processing network respectively, and N branch mask estimation results are obtained, including:

[0077] In the i-th noise reduction branch, the following operations are performed on the i-th noisy frequency band:

[0078] S1, transform the feature dimension of the i-th noisy frequency band to obtain the i-th noisy feature vector with the target feature length;

[0079] S2, performing denoising on the i-th noisy feature vector to obtain the i-th denoising result;

[0080] S3, performing an inverse transformation of the characteristic dimension on the i-th denoising result to obtain the i-th branch processing result having the same characteristic length as the i-th noisy frequency band;

[0081] S4, performing a mask estimation operation on the i-th branch processing result to obtain an i-th branch mask estimation result corresponding to the i-th noisy frequency band.

[0082] Optionally, in this embodiment, it is assumed that the above-mentioned noisy frequency domain representation is divided into two noisy frequency bands, namely a noisy low-frequency band and a noisy high-frequency band, wherein the frequency of the above-mentioned noisy high-frequency band is higher than the frequency of the above-mentioned noisy low-frequency band, and the sampling points corresponding to the above-mentioned noisy high-frequency band are more than the sampling points corresponding to the above-mentioned noisy low-frequency band. For the noisy low-frequency band, performing a feature dimension transformation on the noisy low-frequency band to obtain a noisy feature vector with a target feature length may include, but is not limited to: performing a feature dimension transformation on the noisy low-frequency band, and amplifying the feature vector obtained after the dimension change to obtain a noisy feature vector with a target feature length. For the noisy high-frequency band, performing a feature dimension transformation on the noisy high-frequency band to obtain a noisy feature vector with a target feature length may include, but is not limited to: performing a feature dimension transformation on the noisy high-frequency band, and compressing the feature vector obtained after the dimension change to obtain a noisy feature vector with a target feature length. In other words, in this embodiment, the feature dimension of the noisy feature vectors corresponding to each noisy frequency band is processed respectively so that the noisy feature vectors corresponding to each frequency band have the same feature length, ensuring that information interaction can be performed between the noise reduction branches corresponding to each frequency band.

[0083] Furthermore, it should be noted that the network structure of each of the N noise reduction branches may be, but is not limited to being, completely identical. Specifically, the noise reduction branches may include, but are not limited to:

[0084] 1) The fully connected feature dimension transformation Dense input layer is used to adjust the dimension of the noisy feature vector corresponding to the noisy frequency band;

[0085] 2) Encoder module, used to reduce the frequency domain feature dimension of the noisy feature vector while keeping the time domain feature dimension of the noisy feature vector unchanged to reduce the amount of computation. Specifically, the Encoder module can be, but is not limited to, composed of layers of EncConv2d modules. Specifically, the EncConv2d module consists of a convolution layer (i.e., two-dimensional convolution (Conv2d)), a normalization layer (i.e., normalization (BatchNorm)), and an activation layer (i.e., activation function (PReLU)). The convolution kernel size (Kernel Size) of each EncConv2d layer is (5, 2), indicating a frequency domain field of view of 5 and a time domain field of view of 2. That is, the analysis and processing of the signal features of each frame will refer to the signal of the previous frame, which can be regarded as a streaming convolution structure, ensuring the causal nature of the network. The stride of the convolution can be, but is not limited to, set to (2, 1), that is, the frequency domain stride of the convolution is 2 and the time domain stride is 1. This can halve the number of signal frequency domain features layer by layer, while keeping the dimension of time domain features unchanged, thus maintaining the time domain continuity of the information and reducing the amount of computation.

[0086] 3) An extraction module, used to extract temporal information from the encoder output. Specifically, the extraction module can be a recurrent neural network (RNN) composed of a stack of gated recurrent units (GRUs), or other types of neural networks, such as a residual convolution and attention (RA) mechanism, a two-layer long short-term memory (LSTM) neural network, etc., which are not limited in this embodiment. RNN is a type of neural network with short-term memory capabilities. In an RNN, neurons can not only receive information from other neurons, but also receive information from themselves, forming a network structure with loops. Compared with feedforward neural networks, RNNs are more consistent with the structure of biological neural networks. RNNs have been widely used in tasks such as speech recognition, language modeling, and natural language generation. RA is a neural network-based attention model used to process images of variable size and orientation. RA aims to mimic the attention mechanism of the human visual system, that is, focusing the eye on different parts of the image at different time points to enable more in-depth processing. LSTM is a variant of recurrent neural network (RNN) that is suitable for many modeling tasks involving time series or sequence data. Its basic structure consists of three gates: input gate, forget gate, and output gate, as well as a memory unit.

[0087] 4) Decoding Decoder module, used to restore the frequency domain feature number of the noisy feature vector. Specifically, the Decoder module can be, but is not limited to, composed of a stack of DecTConv2d modules. The DecTConv2d module is highly similar to the EncConv2d module, including: a transposed convolution layer (i.e., a transposed convolution network (ConvTranspose2d)) corresponding to the convolution layer (i.e., two-dimensional convolution (Conv2d)) in EncConv2d, a normalization layer (i.e., normalization (BatchNorm)), and an activation layer (i.e., activation function (PReLU)). The number of DecTConv2d layers included in the Decoder is the same as the number of EncConv2d layers included in the Encoder. The parameters of each layer of DecTConv2d are also the same as the parameters of the EncConv2d of the corresponding layer. In addition, the output of each layer of Encoder can be used as the influencing parameter of the corresponding layer in the Decoder by using a skip connection method, thereby realizing the layer-by-layer restoration of the signal feature dimension.

[0088] 5) The fully connected feature dimension transformation Dense output layer is used to restore the dimension of the noisy feature vector corresponding to the noisy frequency band.

[0089] As an optional embodiment, it is assumed that the above-mentioned noisy frequency domain representation is divided into two noisy frequency bands, namely a noisy low frequency band [0, 8 kHz] and a noisy high frequency band (8 kHz, 24 kHz). Taking the i-th noisy frequency band as the noisy low frequency band as an example, the above-mentioned method is illustrated by the following steps:

[0090] The Dense input layer is used to transform the feature dimension of the noisy low-frequency band, and the feature vector corresponding to the original feature length of 342 is amplified to a noisy feature vector with a feature length of 512, where the original feature length corresponding to the noisy low-frequency band is determined based on the number of frequency points corresponding to the noisy low-frequency band. Next, the noisy feature vector is denoised using the Encoder module, the Extraction module, and the Decoder module to obtain a denoised result. Then, the Dense output layer is used to perform an inverse feature dimension transformation on the denoised result with a feature length of 512 to obtain a branch processing result with a feature length of 342. Furthermore, a mask estimation operation is performed on the branch processing result with a feature length of 342 to obtain a mask estimation result that matches the noisy low-frequency band.

[0091] As an optional embodiment, it is assumed that the above-mentioned noisy frequency domain representation is divided into two noisy frequency bands, namely a noisy low frequency band [0, 8kHz] and a noisy high frequency band (8kHz, 24kHz). Taking the above-mentioned i-th noisy frequency band as the noisy high frequency band as an example, the above-mentioned method is illustrated by the following steps:

[0092] The Dense input layer is used to transform the feature dimension of the noisy high-frequency band, compressing the feature vector corresponding to the noisy high-frequency band with an original feature length of 682 to a noisy feature vector with a feature length of 512. The original feature length corresponding to the noisy high-frequency band is determined based on the number of frequency points corresponding to the noisy high-frequency band. Next, the noisy feature vector is denoised using the Encoder module, the Extraction module, and the Decoder module to obtain a denoised result. The Dense output layer is then used to inversely transform the feature dimension of the denoised result with a feature length of 512 to obtain a branch processing result with a feature length of 682. Furthermore, a mask estimation operation is performed on the branch processing result with a feature length of 682 to obtain a mask estimation result that matches the noisy high-frequency band.

[0093] In an embodiment of the present application, the following operations are performed on the i-th noisy frequency band in the i-th noise reduction branch: the feature dimension of the i-th noisy frequency band is transformed to obtain the i-th noisy feature vector with the target feature length. Then, the i-th noisy feature vector is subjected to noise reduction processing to obtain the i-th noise reduction result; the feature dimension of the i-th noise reduction result is inversely transformed to obtain the i-th branch processing result with the same feature length as the i-th noisy frequency band. Furthermore, a mask estimation operation is performed on the i-th branch processing result to obtain the i-th branch mask estimation result corresponding to the i-th noisy frequency band. In other words, in an embodiment of the present application, by performing a dimensionality transformation process on the noisy frequency band, the feature lengths of the noisy feature vectors corresponding to different noisy frequency bands can be unified, so that in the subsequent noise reduction processing, the noisy feature vectors corresponding to each noisy frequency band can interact, provide guidance for the processing of other noise reduction branches, and thus help improve the noise reduction effect, that is, ensure that the obtained branch mask estimation result more accurately distinguishes between valid speech signals and invalid noise signals.

[0094] Voice signal

[0095] As an optional solution, denoising is performed on the i-th noisy feature vector, and the i-th denoising result includes:

[0096] S1, through the encoding network built based on the streaming convolution structure, encodes the i-th noisy feature vector to obtain the i-th encoding result;

[0097] S2, through the recurrent neural network built based on the gated recurrent unit, analyzes the i-th encoding result and obtains the i-th intermediate result with timing information;

[0098] S3, decodes the i-th intermediate result through a decoding network built based on a streaming convolutional structure to obtain the i-th denoising result, wherein the sub-network in the decoding network is obtained by adjusting the sub-network in the encoding network.

[0099] It should be noted that the encoding network constructed based on the streaming convolution structure can be used, but is not limited to, to indicate the above-mentioned Encoder module. Specifically, the Encoder module can be used, but is not limited to, to reduce the frequency domain feature dimension of the noisy feature vector, but keep the time domain feature dimension of the noisy feature vector unchanged to reduce the amount of calculation. Specifically, the Encoder module can be, but is not limited to, stacked layers of EncConv2d modules. The structure of the EncConv2d module is shown in Figure 4, which consists of a convolution layer 402 (i.e., two-dimensional convolution (Conv2d)), a normalization layer 404 (i.e., normalization (BatchNorm)), and an activation layer 406 (i.e., activation function (PReLU)). Among them, the convolution kernel size (Kernel Size) of each layer of EncConv2d is (5, 2), indicating that the frequency domain field of view is 5 and the time domain field of view is 2, that is, the analysis and processing of the signal features of each frame will refer to the signal of the previous frame, which can be regarded as a streaming convolution structure, ensuring the causality of the network. The stride of the convolution can be, but is not limited to, set to (2, 1), that is, the stride of the convolution in the frequency domain is 2 and the stride in the time domain is 1. This can reduce the number of frequency domain features of the signal by half layer by layer, while keeping the dimension of the time domain features unchanged, thus maintaining the time domain continuity of the information and reducing the amount of computation. For example, assuming that the above encoding network is an Encoder module, the structure of the above encoding network can be, but is not limited to, as shown in FIG5 , consisting of t EncConv2d modules, where t is a positive integer greater than 2.

[0100] It should be noted that the aforementioned recurrent neural network constructed based on gated recurrent units can be used, but is not limited to, to indicate the aforementioned extraction module. Specifically, the extraction module can be a recurrent neural network (RNN) composed of a stack of gated recurrent units (GRUs), which is used to extract the timing information from the output of the encoder module.

[0101] It should be noted that the above-mentioned decoding network constructed based on the streaming convolution structure can be used, but is not limited to, to indicate the above-mentioned Decoder module. Specifically, it is used to restore the frequency domain feature number of the noisy feature vector. The Decoder module can be, but is not limited to, composed of a stack of DecTConv2d modules. The DecTConv2d structure is highly similar to EncConv2d, including: a transposed convolution layer (i.e., a transposed convolution network (ConvTranspose2d)) corresponding to the convolution layer (i.e., two-dimensional convolution (Conv2d)) in EncConv2d, a normalization layer (i.e., normalization (BatchNorm)), and an activation layer (i.e., activation function (PReLU)). The number of DecTConv2d modules included in the Decoder is the same as the number of EncConv2d modules included in the Encoder. The parameters of each DecTConv2d module are also the same as the parameters of the corresponding EncConv2 module. In addition, a skip connection approach can be used to use the output of each EncConv2d as the influencing parameter of the corresponding DecTConv2d in the Decoder module, thereby achieving layer-by-layer restoration of signal feature dimensions. For example, assuming that the above decoding network is the above Decoder module, the above decoding network structure can be, but is not limited to, as shown in Figure 6, consisting of t DecTConv2d modules, where t is a positive integer greater than 2.

[0102] As an optional embodiment, taking the above-mentioned noisy frequency domain representation as being divided into two noisy frequency bands, namely a noisy low frequency band [0, 8kHz] and a noisy high frequency band (8kHz, 24kHz], as an example, the above-mentioned method is illustrated by the following steps:

[0103] The encoder module in the low-frequency noise reduction branch encodes the noisy feature vector corresponding to the noisy low-frequency band to obtain a first encoding result. The RNN neural network in the low-frequency noise reduction branch then analyzes the first encoding result to obtain a first intermediate result containing timing information. Furthermore, the decoder module in the low-frequency noise reduction branch decodes the first intermediate result to obtain a first noise reduction result.

[0104] Next, the encoder module in the high-frequency noise reduction branch encodes the noisy feature vector corresponding to the noisy high-frequency band to obtain a second encoding result. The RNN neural network in the high-frequency noise reduction branch then analyzes the second encoding result to obtain a second intermediate result containing timing information. Furthermore, the decoder module in the high-frequency noise reduction branch decodes the second intermediate result to obtain a second noise reduction result.

[0105] In an embodiment of the present application, the i-th noisy feature vector is encoded by an encoding network constructed based on a streaming convolution structure to obtain the i-th encoding result. Then, the i-th encoding result is analyzed by a recurrent neural network constructed based on a gated recurrent unit to obtain the i-th intermediate result carrying timing information. Furthermore, the i-th intermediate result is decoded by a decoding network constructed based on a streaming convolution structure to obtain the i-th noise reduction result. In other words, in an embodiment of the present application, encoding and decoding are performed accordingly using an encoding network and a decoding network constructed based on a streaming convolution structure, so as to maintain the time domain continuity of the information. In addition, processing using a recurrent neural network constructed based on a gated recurrent unit can effectively obtain the timing information therein. Based on the above structure, the information in the noisy feature vector can be accurately and comprehensively referenced to achieve noise reduction processing. Speech signal

[0106] As an optional solution, the i-th noisy feature vector is encoded by an encoding network built based on a streaming convolutional structure to obtain the i-th encoding result, including:

[0107] The i-th noisy feature vector is encoded through M encoding sub-networks with a connection relationship in the encoding network to obtain the i-th encoding result, wherein each encoding sub-network includes a convolution layer, a normalization layer and an activation layer. When the noisy feature vector of each frame is convolved in the convolution layer, the noisy feature vector of the adjacent previous frame is referenced. M is a natural number greater than or equal to 2;

[0108] The decoding network built based on the streaming convolutional structure decodes the i-th intermediate result to obtain the i-th denoising result, including:

[0109] The i-th intermediate result is decoded by M decoding sub-networks having a connection relationship in the decoding network to obtain the i-th denoising result, wherein each decoding sub-network includes a transposed convolution layer, a normalization layer and an activation layer associated with the convolution layer, wherein a skip connection is set between the k-th encoding sub-network and the M-(k-1)-th decoding sub-network, and k is a natural number greater than or equal to 1 and less than or equal to M.

[0110] It should be noted that, taking the encoding network as an Encoder module as an example, the encoding subnetwork can be, but is not limited to, used to indicate the EncConv2d module that constitutes the Encoder module. Specifically, the EncConv2d module consists of a convolution layer (i.e., two-dimensional convolution (Conv2d)), a normalization layer (i.e., normalization (BatchNorm)), and an activation layer (i.e., activation function (PReLU)). The convolution kernel size (Kernel Size) of each EncConv2d layer is (5, 2), indicating a frequency domain field of view of 5 and a time domain field of view of 2. That is, the analysis and processing of each frame's signal features will refer to the signal of the previous frame, which can be regarded as a streaming convolution structure, ensuring the causal nature of the network. The stride (Stride) of the convolution can be, but is not limited to, set to (2, 1), i.e., the frequency domain stride of the convolution is 2 and the time domain stride is 1. This can halve the number of signal frequency domain features layer by layer, while keeping the time domain feature dimensionality unchanged, maintaining the time domain continuity of the information while reducing the computational complexity.

[0111] Furthermore, taking the above-mentioned decoding network as a Decoder module as an example, the above-mentioned decoding subnetwork can be used to indicate, but is not limited to, the DecTConv2d module used to constitute the Decoder module. The DecTConv2d structure is highly similar to EncConv2d, including: a transposed convolution layer (i.e., a transposed convolution network (ConvTranspose2d)) corresponding to the convolution layer (i.e., two-dimensional convolution (Conv2d)) in EncConv2d, a normalization layer (i.e., normalization (BatchNorm)) and an activation layer (i.e., activation function (PReLU)). The number of DecTConv2d layers included in the Decoder is the same as the number of EncConv2d layers included in the Encoder. The parameters of each layer of DecTConv2d are also the same as the parameters of the EncConv2d of the corresponding layer. In addition, the output of each layer of the Encoder can be used as the influencing parameter of the corresponding layer in the Decoder by using a skip connection method, thereby realizing the layer-by-layer restoration of the signal feature dimension.

[0112] As an optional implementation, the above-mentioned noisy frequency domain representation is divided into two noisy frequency bands, namely the noisy low frequency band [0, 8kHz] and the noisy high frequency band (8kHz, 24kHz]. Taking the noisy frequency band currently being processed as the noisy low frequency band as an example, it is assumed that the encoding network in the low-frequency noise reduction branch is an Encoder module, and the Encoder module is composed of 3 layers of EncConv2d modules. The decoding network in the low-frequency noise reduction branch is a Decoder module, and the Decoder module is composed of 3 layers of DecTConv2d modules. The above method is illustrated by the following steps shown in Figure 7:

[0113] Execute step S702 to obtain the noisy low-frequency band. Then, execute step S704 to input the noisy low-frequency band into the fully connected feature dimension transformation input layer (i.e., the Dense input layer). The Dense input layer amplifies the original feature vector corresponding to the noisy low-frequency band, which has a feature length of 342, to a noisy feature vector with a feature length of 512.

[0114] Then execute step S706, input the noisy feature vector with a feature length of 512 into the encoding network, and use EncConv2d-1 in the encoding network to convert the noisy feature vector with a frequency domain feature dimension of 512 into an encoding result with a frequency domain feature length of 256; use EncConv2d-2 to convert the encoding result with a frequency domain feature length of 256 into an encoding result with a frequency domain feature length of 128; and use EncConv2d-3 to convert the encoding result with a frequency domain feature length of 128 into an encoding result with a frequency domain feature length of 64.

[0115] Then execute step S708, input the encoding result with a frequency domain feature length of 64 into the RNN neural network, so as to use the RNN to obtain the timing information in the above encoding result, and then obtain the intermediate result with a frequency domain feature length of 64 carrying the timing information.

[0116] Then execute step S710, input the intermediate result with a frequency domain feature length of 64 carrying timing information into the decoding network, and convert the intermediate result with a frequency domain feature dimension of 64 into a denoising result with a frequency domain feature length of 128 through DecTConv2d-1 in the decoding network module, and use the output of EncConv2d-3 to affect the calculation of DecTConv2d-1; convert the denoising result with a frequency domain feature length of 128 into a denoising result with a frequency domain feature length of 256 through DecTConv2d-2, and use the output of EncConv2d-2 to affect the calculation of DecTConv2d-2; convert the denoising result with a frequency domain feature length of 256 into a denoising result with a frequency domain feature length of 512 through DecTConv2d-3, and use the output of EncConv2d-1 to affect the calculation of DecTConv2d-3.

[0117] Steps S712-S714 are further performed to input the denoising result with a frequency domain feature length of 512 into the fully connected feature dimensionality transformation output layer (i.e., the Dense output layer). The Dense output layer performs dimensionality reduction processing on the denoising result with a frequency domain feature length of 512 to obtain a denoising result with a feature length of 342. A mask estimation operation is performed on the denoising result with a feature length of 342 to obtain a mask estimation result corresponding to the noisy low-frequency band.

[0118] Specifically, the mask estimation operation may include, but is not limited to, dividing the noise reduction result by the noisy high-frequency band, i.e., the mask estimation result corresponding to the noisy low-frequency band = the noise reduction result with a feature length of 342 / the noisy low-frequency band. Alternatively, other methods may be used to perform the mask estimation operation on the noise reduction result, which is not limited in this embodiment.

[0119] As an optional implementation, the above-mentioned noisy frequency domain representation is divided into two noisy frequency bands, namely the noisy low frequency band [0, 8kHz] and the noisy high frequency band (8kHz, 24kHz]. Taking the noisy frequency band currently being processed as the noisy high frequency band as an example, it is assumed that the encoding network in the high-frequency noise reduction branch is an Encoder module, and the Encoder module is composed of 3 layers of EncConv2d modules. The decoding network in the high-frequency noise reduction branch is a Decoder module, and the Decoder module is composed of 3 layers of DecTConv2d modules. The above method is illustrated by the following steps:

[0120] The noisy high frequency band is input into the Dense input layer, and the original feature vector corresponding to the noisy high frequency band with a feature length of 682 is compressed into a noisy feature vector with a feature length of 512 through the Dense input layer.

[0121] Then, the noisy feature vector with a feature length of 512 is input into the encoding network, and the noisy feature vector with a frequency domain feature dimension of 512 is converted into an encoding result with a frequency domain feature length of 256 through EncConv2d-4 in the encoding network; the encoding result with a frequency domain feature length of 256 is converted into an encoding result with a frequency domain feature length of 128 through EncConv2d-5; and the encoding result with a frequency domain feature length of 128 is converted into an encoding result with a frequency domain feature length of 64 through EncConv2d-6.

[0122] Next, the encoding result with a frequency domain feature length of 64 is input into the RNN neural network to obtain the timing information in the above encoding result, and then obtain the intermediate result with a frequency domain feature length of 64 carrying the timing information.

[0123] Furthermore, the intermediate result with a frequency domain feature length of 64 carrying timing information is input into the decoding network, and the intermediate result with a frequency domain feature dimension of 64 is converted into a denoising result with a frequency domain feature length of 128 through DecTConv2d-4 in the decoding network, and the output of EncConv2d-6 is used to affect the calculation of DecTConv2d-4; the denoising result with a frequency domain feature length of 128 is converted into a denoising result with a frequency domain feature length of 256 through DecTConv2d-5, and the output of EncConv2d-5 is used to affect the calculation of DecTConv2d-5; the denoising result with a frequency domain feature length of 256 is converted into a denoising result with a frequency domain feature length of 512 through DecTConv2d-6, and the output of EncConv2d-4 is used to affect the calculation of DecTConv2d-6.

[0124] Next, the denoising result with a frequency domain feature length of 512 is input into the Dense output layer, and the denoising result with a frequency domain feature length of 512 is dimensionally restored through the Dense output layer to obtain a denoising result with a feature length of 682.

[0125] Furthermore, a mask estimation operation is performed on the denoising result with a feature length of 682 to obtain a mask estimation result corresponding to the noisy high frequency band.

[0126] Specifically, the mask estimation operation may include, but is not limited to, dividing the noise reduction result by the noisy high-frequency band, i.e., the mask estimation result corresponding to the noisy high-frequency band = the noise reduction result with a feature length of 682 / the noisy high-frequency band. Alternatively, other methods may be used to perform the mask estimation operation on the noise reduction result, which is not limited in this embodiment.

[0127] In an embodiment of the present application, an encoding network constructed using a streaming convolutional structure encodes the i-th noisy feature vector to obtain the i-th encoding result. A decoding network constructed based on the streaming convolutional structure decodes the i-th intermediate result to obtain the i-th noise reduction result. Furthermore, by setting a jump connection between the encoding subnetwork and the decoding subnetwork, the output result of the decoding subnetwork is made more accurate, thereby achieving the technical effect of improving the accuracy of the noise reduction processing of the audio signal.

[0128] As an optional solution, when the i-th intermediate result is decoded by the M decoding sub-networks having a connection relationship in the decoding network to obtain the i-th denoising result, the method further includes:

[0129] S1, when the i-th noise reduction branch is not the first noise reduction branch, the output results corresponding to the M coding sub-networks in the coding network within the i-th noise reduction branch and the M gating processing results associated with the i-1-th noise reduction branch are respectively weighted and summed to obtain M decoding reference results, wherein the j-th gating processing result is obtained by processing the output result of the j-th coding sub-network in the i-1-th noise reduction branch through the j-th information transmission gating structure in the audio processing network, and the convolution layer in each information transmission gating structure includes at least two convolution structures, and j is a natural number greater than or equal to 1 and less than or equal to M;

[0130] S2: Input each of the M decoding reference results into a corresponding decoding sub-network in the M decoding sub-networks in the i-th noise reduction branch.

[0131] It should be noted that the above-mentioned noisy frequency domain representation is assumed to be divided into two noisy frequency bands, namely a noisy low-frequency band and a noisy high-frequency band. A gating structure is also provided between the low-frequency denoising branch for processing the noisy low-frequency band and the high-frequency denoising branch for processing the noisy high-frequency band. The above-mentioned gating structure is used to use the modulated output of the encoding network in the low-frequency denoising branch as guidance information, and to act together with the output of the encoding network in the high-frequency denoising branch on the decoding network in the high-frequency denoising branch. Specifically, the above-mentioned gating structure can be, but is not limited to, composed of two two-dimensional convolutions (Conv2d), a layer of normalization (BatchNorm), and a layer of activation function (PReLU), as shown in Figure 8. The input of the gating structure is the output of each encoding sub-network (i.e., EncConv2d module) included in the encoding network, and the output of the gating structure is the gated processing result corresponding to the output of each encoding sub-network (i.e., EncConv2d module) included in the encoding network. Taking Figure 8 as an example, assuming that the encoding network includes three EncConv2d modules, namely EncConv2d-1, EncConv2d-2, and EncConv2d-3, the input of the gating structure is output result 1 output by EncConv2d-1, output result 2 output by EncConv2d-2, and output result 3 output by EncConv2d-3. The output of the gating structure is gated processing result 1 obtained by calculating output result 1 through the gating structure, gated processing result 2 obtained by calculating output result 2 through the gating structure, and gated processing result 3 obtained by calculating output result 3 through the gating structure.

[0132] As an optional implementation, the above-mentioned noisy frequency domain representation is divided into two noisy frequency bands, namely the noisy low frequency band [0, 8kHz] and the noisy high frequency band (8kHz, 24kHz] as an example. Assume that the above-mentioned M is 3, assume that the encoding network is an Encoder module, and the decoding network is a Decoder module. The above method is illustrated by the following steps shown in Figure 9:

[0133] Execute step S902 to obtain a noisy low-frequency band, and then execute step S904 to process the noisy low-frequency band through the low-frequency noise reduction branch to obtain a first processing result. Specifically, the noisy low-frequency band is input to the fully connected feature dimension transformation (i.e., Dense) input layer 1 of the low-frequency noise reduction branch, and the feature dimension transformation is performed on the noisy low-frequency band to obtain a first noisy feature vector with a target feature length. Then, the first noisy feature vector is input to the encoding network 1 of the low-frequency noise reduction branch, and then the first noisy feature vector is encoded using EncConv2d-1, EncConv2d-2, and EncConv2d-3 in the encoding network 1. Then, the first encoding result output by the encoding network 1 is input to the RNN1 neural network of the low-frequency noise reduction branch. The first encoding result is processed using the RNN1 neural network to obtain a first intermediate result carrying timing information. Then, the first intermediate result is input to the decoding network 1 of the low-frequency noise reduction branch. The first intermediate result is then decoded sequentially using DecTConv2d-1, DecTConv2d-2, and DecTConv2d-3 in decoding network 1. Simultaneously, the output of EncConv2d-3, the low-frequency noise reduction branch, is input into DecTConv2d-1, the output of EncConv2d-2, and the output of EncConv2d-1 into DecTConv2d-3. The output of EncConv2d-3 influences the calculation of DecTConv2d-1, the output of EncConv2d-2 influences the calculation of DecTConv2d-2, and the output of EncConv2d-1 influences the calculation of DecTConv2d-3. This results in the first noise reduction result output by decoding network 1. Next, the first denoising result is input into the fully connected feature dimension transformation (i.e., Dense) output layer 1 of the low-frequency denoising branch to restore the feature dimension of the first denoising result to the original dimension corresponding to the noisy low-frequency band, thereby obtaining the first processing result.

[0134] Next, step S906 is executed to perform a mask estimation operation on the first processing result to obtain a first mask estimation result corresponding to the noisy low-frequency band.

[0135] Then, step S908 is performed to calculate the output of encoding network 1 using the gating structure to obtain its corresponding gating processing result. Specifically, the output of EncConv2d-1 is input into the gating structure to obtain gating processing result 1; the output of EncConv2d-2 is input into the gating structure to obtain gating processing result 2; and the output of EncConv2d-3 is input into the gating structure to obtain gating processing result 3.

[0136] Further, step S910 is executed to obtain a noisy high-frequency band. Then, step S912 is executed to process the noisy high-frequency band through the high-frequency noise reduction branch to obtain a second processing result. Specifically, the noisy high-frequency band is input to the fully connected feature dimension transformation (i.e., Dense) input layer 2 of the high-frequency noise reduction branch. The feature dimension of the noisy high-frequency band is transformed to obtain a second noisy feature vector with a target feature length. Then, the second noisy feature vector is input to the encoding network 2 of the high-frequency noise reduction branch, and the second noisy feature vector is encoded using EncConv2d-4, EncConv2d-5, and EncConv2d-6 in the encoding network 2 in sequence. Then, the second encoding result output by the encoding network 2 is input to the RNN2 neural network of the high-frequency noise reduction branch. The second intermediate result carrying timing information is obtained using RNN2. At the same time, an exclusive-OR operation is performed on the gating result 1 and the output result of EncConv2d-4 to obtain decoding reference result 1. An exclusive-OR operation is performed on the gating result 2 and the output result of EncConv2d-5 to obtain decoding reference result 2. An exclusive-OR operation is performed on the gating result 3 and the output result of EncConv2d-6 to obtain decoding reference result 3. Next, the second intermediate result is input into decoding network 2 of the high-frequency noise reduction branch. It is then decoded sequentially using DecTConv2d-4, DecTConv2d-5, and DecTConv2d-6 in decoding network 2. Meanwhile, decoding reference result 3 is input into DecTConv2d-4, decoding reference result 2 is input into DecTConv2d-5, and decoding reference result 1 is input into DecTConv2d-6. The decoding reference result 3 influences the calculation of DecTConv2d-4, the decoding reference result 2 influences the calculation of DecTConv2d-5, and the decoding reference result 1 influences the calculation of DecTConv2d-6. The second denoising result is then output by decoding network 2. Next, the second denoising result is input to the fully connected feature dimensionality transformation (i.e., Dense) output layer 2 of the high-frequency denoising branch to restore the feature dimensions of the second denoising result to the original dimensions corresponding to the noisy high-frequency band, thereby obtaining the second processing result.

[0137] Then, step S914 is executed to perform a mask estimation operation on the second processing result to obtain a second mask estimation result corresponding to the noisy high frequency band.

[0138] In this embodiment of the present application, the M gating processing results associated with the (i-1)th noise reduction branch are used to process the output results corresponding to each of the M encoding sub-networks in the encoding network within the (i)th noise reduction branch. This improves the accuracy of the output results of the decoding sub-network in the (i)th noise reduction branch, thereby achieving the technical effect of improving the accuracy of the noise reduction processing results of the audio signal.

[0139] As an optional solution, the noisy frequency domain representation is used to modulate the N branch mask estimation results to obtain N voice frequency domain representations, including:

[0140] Splicing the N branch mask estimation results to obtain a spliced ​​expression;

[0141] The concatenated expression is modulated using the noisy frequency domain representation to obtain the full-band speech frequency domain representation.

[0142] Optionally, in this embodiment, the above-mentioned modulation processing of the concatenated expression using the noisy frequency domain representation to obtain the full-band voice frequency domain representation may include, but is not limited to: performing a cross multiplication operation on the noisy frequency domain representation and the concatenated expression to obtain the full-band voice frequency domain representation.

[0143] As an optional embodiment, it is assumed that the above-mentioned noisy frequency domain representation is divided into two noisy frequency bands, namely the noisy low frequency band and the noisy high frequency band. Assume that the mask estimation result corresponding to the noisy low frequency band is the first mask estimation result The mask estimation result corresponding to the noisy high frequency band is the second mask estimation result Then the above-mentioned splicing of the N branch mask estimation results to obtain a splicing expression may include, but is not limited to: the first mask estimation result With the second mask estimation result Splicing is performed to obtain spliced ​​expression (m k ),Right now Specifically, the above estimation result of the first mask With the second mask estimation result The splicing method may include but is not limited to: Splice to At the end of, or Splice to At the end of , etc., there is no limitation on this in this embodiment.

[0144] In an embodiment of the present application, the mask estimation results of N branches are spliced ​​to obtain a spliced ​​expression. Then, the spliced ​​expression is modulated using the noisy frequency domain representation to obtain a full-band speech frequency domain representation. In other words, in an embodiment of the present application, multiple noise reduction branches are used to perform noise reduction processing on multiple noisy frequency domain segments corresponding to the audio signal. The mask estimation results obtained through the N branches are then spliced ​​and calculated and transformed to obtain a full-band estimation result of the speech signal, which helps to obtain an accurate full-band speech signal without noise interference. This avoids the problem of inaccurate noise reduction processing results caused by the use of an audio signal processing model for a fixed sampling rate in the prior art to perform noise reduction processing on the audio signal. This achieves the technical effect of improving the accuracy of noise reduction processing of the audio signal.

[0145] As an optional solution, performing time domain conversion processing on the N speech frequency domain representations to obtain a speech signal in the audio signal includes:

[0146] The full-band speech signal is estimated by performing time-domain conversion on the full-band speech signal representation.

[0147] Furthermore, the above-mentioned time domain conversion processing of the full-band speech audio domain representation may be used, but is not limited to, to indicate that an Inverse Short Time Discrete Transform (ISDCT) processing is performed on the full-band speech audio domain representation.

[0148] As an optional implementation, assume that the noise frequency domain representation is X k , the above-mentioned noisy frequency domain representation X k Divide into two noisy frequency bands, namely the noisy low frequency band and the noisy high frequency band. Assume that the mask estimation result corresponding to the noisy low frequency band is the first mask estimation result The mask estimation result corresponding to the noisy high frequency band is the second mask estimation result The above method is illustrated by the following steps:

[0149] Estimation result of the first mask With the second mask estimation result Splicing is performed to obtain spliced ​​expression (m k ),Right now

[0150] Then, the spliced ​​expression (m k ) and the frequency domain representation of audio signals with noise (X k ) performs cross multiplication to obtain full-band spectrum estimation Right now,

[0151] Then, Perform Inverse Short Time Discrete Transform (ISDCT) processing to obtain the full-band estimation result of the speech signal (ie, a full-band speech signal without the presence of noise).

[0152] In an embodiment of the present application, the mask estimation results of N branches are spliced ​​to obtain a spliced ​​expression. Then, the spliced ​​expression is modulated using the noisy frequency domain representation to obtain a full-band speech frequency domain representation. In other words, in an embodiment of the present application, multiple noise reduction branches are used to perform noise reduction processing on multiple noisy frequency domain segments corresponding to the audio signal. The mask estimation results obtained through the N branches are then spliced ​​and calculated to obtain a full-band estimation result of the speech signal, which helps to obtain an accurate full-band speech signal without noise interference. This avoids the problem of inaccurate noise reduction processing results caused by the use of an audio signal processing model for a fixed sampling rate in the prior art to perform noise reduction processing on the audio signal. This achieves the technical effect of improving the accuracy of noise reduction processing of the audio signal.

[0153] As an optional solution, before concatenating the N branch mask estimation results to obtain the concatenated expression, the following steps are further included:

[0154] S1, using the i-th branch mask estimation result, modulate the noisy frequency domain representation of the i-th noisy frequency band to obtain the i-th voice frequency domain representation;

[0155] S2, performing time domain conversion processing on the i-th frequency domain representation of the speech signal to obtain the i-th frequency band estimation result of the speech signal.

[0156] As an optional implementation, assume that the noise frequency domain representation is X k , the above-mentioned noisy frequency domain representation (X k ) is divided into two noisy frequency bands, namely the noisy low frequency band and noisy high frequency band Assume that the mask estimation result corresponding to the noisy low frequency band is the first mask estimation result The mask estimation result corresponding to the noisy high frequency band is the second mask estimation result The above method is illustrated by the following steps:

[0157] For the low-frequency band with noise, With the first mask estimation result Perform a cross product calculation to obtain the spectrum estimate corresponding to the noisy low-frequency band Right now, Then, the spectrum corresponding to the noisy low-frequency band is estimated Perform Inverse Short Time Discrete Transform (ISDCT) processing to obtain the low-frequency band estimation result (i.e., a wideband speech signal in the absence of noise).

[0158] For the high-frequency band with noise, With the second mask estimation result Perform cross multiplication to obtain the spectrum estimate corresponding to the noisy high frequency band Right now, Then, the spectrum corresponding to the noisy high frequency band is estimated Perform Inverse Short Time Discrete Transform (ISDCT) processing to obtain high frequency band estimation results

[0159] In an embodiment of the present application, the noisy frequency domain representation of the i-th noisy frequency band is modulated using the i-th branch mask estimation result corresponding to the i-th noisy frequency band to obtain the i-th voice frequency domain representation. The i-th voice frequency domain representation is then converted into the time domain to obtain the i-th frequency band estimation result of the speech signal. In this way, the noisy frequency domain representation of the corresponding noisy frequency band can be accurately modulated based on the corresponding branch mask estimation result, thereby obtaining a noise-free speech signal corresponding to the noisy frequency band, thereby improving the denoising effect.

[0160] As an optional solution, before performing frequency domain conversion processing on the audio signal to obtain a noisy frequency domain representation corresponding to the audio signal, the method further includes:

[0161] Sampling the audio signal at a target sampling rate to obtain sampled audio data;

[0162] The sampled audio data is subjected to time-domain frame processing to obtain a processed audio signal.

[0163] It should be noted that the target sampling rate can be preset according to actual needs. Specifically, the target sampling rate can be, but is not limited to, 48 kHz, 44.1 kHz, etc., and this is not limited in this embodiment.

[0164] Furthermore, the time-domain framing of the sampled audio data may include, but is not limited to, frame-framing and windowing modulation of the audio data to prevent spectrum leakage. For example, the audio data may be segmented into multiple frames of short signals of fixed length, with a single frame consisting of 1024 sampling points (i.e., a frame length of 1024) and a frame shift of 512 (i.e., an overlap length of 512 between each two adjacent frames). Each frame of the audio data is then modulated using a Hamming window to obtain a modulated audio signal, thereby preventing spectrum leakage.

[0165] It should be noted that the windowing processing method used for the windowing processing of the audio data is not limited to the Hamming window, and other methods may also be used, such as a rectangular window, a Hanning window, etc., which is not limited in this embodiment.

[0166] As an optional embodiment, the above method is illustrated by the following steps:

[0167] The audio signal is sampled at a sampling rate of 48 kHz to obtain sampled audio data, and then the sampled audio data is subjected to frame-based windowing and modulation processing to obtain a processed audio signal.

[0168] In an embodiment of the present application, an audio signal is sampled at a target sampling rate to obtain sampled audio data. The sampled audio data is then subjected to time-domain framing processing to obtain a processed audio signal. Subsequently, denoising processing is performed on the audio signal after time-domain framing processing, thereby achieving the technical effect of improving the accuracy of the audio signal noise reduction processing.

[0169] As an optional solution, before obtaining the audio signal to be processed, the following steps are further included:

[0170] Obtain speech dataset and noise dataset;

[0171] Mix the speech data set and the noise data set to obtain a sample noise frequency signal;

[0172] The initialized audio processing network is trained using the sample noisy frequency signal until the loss function of the audio processing network reaches a convergence condition, where the loss function is used to calculate the difference between the speech signal in the speech dataset and the candidate reference speech signal identified by the audio processing network during training from the sample noisy frequency signal.

[0173] Optionally, in this embodiment, the speech data set may be, but is not limited to, used to indicate a pure speech set that does not contain noise. The noise data set may be, but is not limited to, used to indicate a useless noisy speech set.

[0174] As an optional implementation, assume that the speech data set is sn , the noise data set is d n The above method is illustrated by the following steps:

[0175] Get the speech dataset as s n , the noise data set is d n Then, for the speech dataset s n and the noise dataset d n Mix and get the sample noise frequency signal x n Then, x n Input to the initialized audio processing network and obtain the output result (i.e., a speech dataset that does not contain noise) to train the initialized audio processing network until the loss function of the audio processing network reaches a predetermined threshold, wherein the expression of the loss function can be, but is not limited to, as follows:

[0176] Among them, the above s n For a speech dataset that does not contain noise, the above is the output of the audio processing network during training. It should be noted that the above loss function can be, but is not limited to, any one of the mean square error (MSE) loss function, mean absolute error (MAE) loss function, and scale invariant signal-to-noise ratio (SI-SNR), and is not limited to any of the above in this embodiment.

[0177] In the embodiment of the present application, the initialized audio processing network is pre-trained with rich sample information to obtain a trained audio processing network. This audio processing network is then used to perform noise reduction processing on the audio signal, thereby achieving the technical effect of improving the accuracy of the noise reduction processing.

[0178] As an optional embodiment, the above-mentioned audio noise reduction processing method is illustrated by the following steps shown in FIG10:

[0179] Get the audio signal x including noise n , for x n The target audio signal performs short-time cosine transform processing on the audio signal to obtain the frequency domain features X of the audio signal. N Next, the frequency domain feature X k Perform segmentation processing to obtain the noisy low-frequency band and high-frequency noise

[0180] Then, the noisy low frequency band The input is sent to the low-frequency noise reduction branch and processed in sequence through the fully connected feature dimension transformation input layer, encoding network, RNN network, and fully connected feature dimension transformation output layer to obtain the mask estimation result corresponding to the noisy low-frequency band. Furthermore, and Perform a cross multiplication operation to obtain the calculation result, and perform a mask estimation operation on the calculation result to obtain the spectrum estimation corresponding to the noisy low frequency band And then Perform inverse short-time cosine transform processing to obtain a wideband speech signal that is not affected by noise Among them, the output result of EncConv2d in the encoding network is used to assist the calculation of DecTConv2d in the decoding network.

[0181] The high frequency band with noise The input is sent to the high-frequency noise reduction branch and processed in sequence through the fully connected feature dimension transformation input layer, encoding network, RNN network, and fully connected feature dimension transformation output layer to obtain the mask estimation result corresponding to the noisy high-frequency band. Then, the mask estimation result and mask estimation results Perform splicing to obtain the spliced ​​mask estimation result m k The mask estimation result m after splicing k With X k Perform a cross multiplication operation to obtain the operation result, perform a mask estimation operation on the operation result, and obtain the spectrum estimation corresponding to the full-band speech signal right The result of the operation is processed by inverse short-time cosine transform to obtain a full-band speech signal without noise interference. Among them, the guidance information after modulating the output of EncConv2d in the encoding network in the low-frequency noise reduction branch and the output of EncConv2d in the encoding network in the high-frequency noise reduction branch are used to assist the operation of DecTConv2d in the decoding network in the high-frequency noise reduction branch.

[0182] In this embodiment, multiple noise reduction branches are used to perform noise reduction processing on multiple noisy frequency domain segments corresponding to the audio signal, thereby obtaining a speech signal that is not interfered with by the noise signal. This avoids the problem of inaccurate noise reduction results caused by the use of audio signal processing models for fixed sampling rates in the prior art. This achieves the technical effect of improving the accuracy of audio signal noise reduction processing.

[0183] As another optional embodiment, the system framework adopted by the above-mentioned audio noise reduction processing method is illustrated by the following steps:

[0184] 1) Pre-processing and feature extraction module: for noisy speech signal x n Resampling is performed to resample all audio data of different sampling rates to 48kHz. After the resampling operation is completed, the long audio signal is then subjected to time domain frame division and windowing. According to the single frame length of 1024 and the frame shift of 512 (overlap 512), the original audio signal is divided into multiple frames of short signals with fixed lengths, and each frame signal is modulated using a Hamming window to prevent spectrum leakage. After the frame division and windowing operation is completed, the modulated signal is subjected to a discrete cosine transform operation to extract the frequency domain features and obtain the noisy speech signal x. n Frequency domain representation of X k The combination of audio signal framing and windowing with cosine transform is also called short-time cosine transform. k After that, it is divided into the frequency points below 8kHz. It can be regarded as the cosine spectrum of a broadband signal, consisting of frequencies above 8kHz. Bandwidth is twice as much.

[0185] 2) Neural Network Forward Inference Module: This module utilizes a dual-path encoder-decoder interaction structure to model and analyze the low-frequency band (0, 8kHz) and high-frequency band (8kHz, 24kHz) of the audio. The network model consists of three main components: a low-frequency branch, a high-frequency branch, and a gate structure that transmits low-frequency information to the high-frequency branch. The low-frequency branch and the high-frequency branch are structurally symmetrical, each consisting of four components: a fully connected feature dimension transformation layer (Dense), an encoder module, a recurrent neural network (RNN) module, and a decoder module. The Dense input layer transforms the low-frequency and high-frequency features entering the two branches, reducing the low-frequency feature length from 342 to 512 and compressing the high-frequency feature length from 682 to 512. This unifies the feature lengths of the high-frequency and low-frequency branches, facilitating interaction. The Dense output layer transforms the feature dimensions back to their original dimensions. The encoder consists primarily of stacked layers of EncConv2d modules, built around two-dimensional convolution (Conv2d) with batch normalization (BatchNorm) and the Pre-ReLU activation function. Each EncConv2d layer uses a kernel size of (5, 2), indicating a frequency-domain field of view of 5 and a time-domain field of view of 2. The analysis and processing of each frame's signal features references the previous frame, creating a streaming convolutional architecture that ensures network causality. The stride of the convolution is (2, 1), which halves the number of frequency-domain features layer by layer while preserving the dimensionality of the time-domain features. This maintains temporal continuity of the information while reducing computational complexity. The decoder consists primarily of stacked DecTConv2d modules, which are highly similar in structure to EncConv2d, except that the convolutional structure is replaced with a transposed convolutional network (ConvTranspose2d). The number of layers of the Decoder is the same as that of the Encoder, and the parameters of each layer of DecTConv2d are the same as those of the corresponding EncConv2d. The output of the Encoder is used as the input of the Decoder by skip connection, realizing the layer-by-layer restoration of the signal feature dimension. Between the Encoder and Decoder modules, a recurrent neural network module RNN composed of stacked GRU (Gated Recurrent Units) is used for the analysis and extraction of time series information. The main function of the information transmission module is to modulate the output of the low-frequency branch Encoder as guidance information together with the output of the high-frequency branch Encoder to the high-frequency branch Decoder. The module uses a gated structure to realize information extraction. The final output target of the two branches is the short-time cosine transform mask estimation of the signal. The low-frequency branch mask estimation value is The high-frequency branch mask estimate is

[0186] 3) Post-processing speech generation module: After obtaining the short-time cosine transform masks of the low-frequency and high-frequency signal components, the short-time cosine spectrum of the original noisy speech is modulated to obtain the short-time cosine spectrum estimates of the low-frequency and high-frequency components, respectively. The expressions are as follows:

[0187] The estimated value of the broadband pure speech signal can be obtained by inverse short-time cosine transform of the low-frequency cosine spectrum. Combining the low-frequency and high-frequency cosine spectra and performing iSDCT can obtain an estimate of the full-band pure speech signal.

[0188] It should be noted that in this embodiment, a design pattern of a band-separated speech enhancement and noise reduction model is given, which solves the noise suppression problem of broadband signals and full-band signals at the same time without introducing additional computational complexity. A band-separated noise reduction system based on an Encoder-Decoder dual-path interactive structure is proposed. By modeling and analyzing the low-frequency and high-frequency bands of the noisy frequency respectively, the noise components in each band can be effectively suppressed. Conventional speech enhancement and noise reduction solutions only model and analyze signals with one sampling rate, while this embodiment can utilize a dual-path structure to process broadband signals (16kHz) and full-band signals (48kHz), using one system to adapt to two different application scenarios.

[0189] Furthermore, 1000 sets of test data with a signal-to-noise ratio range of [-10, 30] dB in 2 dB increments were used to obtain the test results of this embodiment. The perceptual speech quality parameter (PESQ), the scale-invariant signal-to-noise ratio parameter (SI-SNR), and the simulated subjective audio quality perception parameter (DNSMOS) were selected as performance evaluation metrics to determine the test results. Specifically, Figure 11 shows the PESQ test results, Figure 12 shows the SI-SNR test results, and Figure 13 shows the MOS_OVL test results.

[0190] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0191] According to another aspect of the embodiments of the present application, an audio noise reduction processing device for implementing the above-mentioned audio noise reduction processing method is also provided. As shown in FIG14 , the device includes:

[0192] An acquiring unit 1402 is configured to acquire a speech signal to be processed, wherein the speech signal includes a speech signal to be recognized that has been interfered with by a noise signal;

[0193] The extraction unit 1404 is configured to perform frequency domain conversion processing on the speech signal to obtain a noisy frequency domain representation corresponding to the speech signal;

[0194] An input unit 1406 is configured to divide the noisy frequency domain representation into N noisy frequency bands, and input the N noisy frequency bands into N noise reduction branches within the audio processing network, respectively, to obtain N branch mask estimation results, wherein the i-th noise reduction branch in the audio processing network is configured to process the i-th noisy frequency band of the N noisy frequency bands to obtain the i-th branch mask estimation result corresponding to the i-th noisy frequency band, the N noise reduction branches having the same signal processing structure, i being a natural number greater than or equal to 1 and less than or equal to N, and N being a natural number greater than 1;

[0195] The modulation unit 1408 is configured to modulate the N branch mask estimation results using the noisy frequency domain representation to obtain N voice frequency domain representations;

[0196] The conversion unit 1410 is configured to perform time domain conversion processing on the N audio domain representations of speech to obtain a speech signal free from noise interference from the speech signal.

[0197] Optionally, the input unit includes:

[0198] An execution module is used to perform the following operations on the i-th noisy frequency band in the i-th denoising branch: perform feature dimension transformation on the i-th noisy frequency band to obtain the i-th noisy feature vector with a target feature length; perform denoising processing on the i-th noisy feature vector to obtain the i-th denoising result; perform feature dimension inverse transformation on the i-th denoising result to obtain the i-th branch processing result with the same feature length as the i-th noisy frequency band; perform mask estimation operation on the i-th branch processing result to obtain the i-th branch mask estimation result corresponding to the i-th noisy frequency band.

[0199] Optionally, the above-mentioned execution module is also used to encode the i-th noisy feature vector through an encoding network constructed based on a streaming convolution structure to obtain an i-th encoding result; analyze the i-th encoding result through a recurrent neural network constructed based on a gated recurrent unit to obtain an i-th intermediate result carrying timing information; and decode the i-th intermediate result through a decoding network constructed based on a streaming convolution structure to obtain an i-th denoising result, wherein the sub-network in the decoding network is obtained by adjusting the sub-network in the encoding network.

[0200] Optionally, the above-mentioned execution module is also used to encode the i-th noisy feature vector through M encoding sub-networks with a connection relationship in the encoding network to obtain the i-th encoding result, wherein each encoding sub-network includes a convolution layer, a normalization layer and an activation layer. When convolution processing is performed on the noisy feature vector of each frame in the convolution layer, the adjacent noisy feature vector of the previous frame will be referenced, and M is a natural number greater than or equal to 2; through M decoding sub-networks with a connection relationship in the decoding network, the i-th intermediate result is decoded to obtain the i-th denoising result, wherein each decoding sub-network includes a transposed convolution layer, a normalization layer and an activation layer associated with the convolution layer, wherein a skip connection is set between the k-th encoding sub-network and the M-(k-1)-th decoding sub-network, and k is a natural number greater than or equal to 1 and less than or equal to M.

[0201] Optionally, the above-mentioned execution module is also used to, when the i-th noise reduction branch is not the first noise reduction branch, perform weighted summation processing on the output results corresponding to each of the M coding sub-networks in the coding network within the i-th noise reduction branch, and the M gating processing results associated with the i-1-th noise reduction branch, to obtain M decoding reference results, wherein the j-th gating processing result is obtained by processing the output result of the j-th coding sub-network in the i-1-th noise reduction branch through the j-th information transmission gating structure in the audio processing network, and the convolution layer in each information transmission gating structure includes at least two convolution structures, and j is a natural number greater than or equal to 1 and less than or equal to M; each of the M decoding reference results is respectively input into the corresponding decoding sub-network in the M decoding sub-networks in the i-th noise reduction branch.

[0202] Optionally, the modulation unit includes:

[0203] A splicing module, used for splicing N branch mask estimation results to obtain a spliced ​​expression;

[0204] The modulation module is used to modulate the concatenated expression using the noisy frequency domain representation to obtain the full-band speech frequency domain representation.

[0205] Optionally, the conversion unit is further configured to perform time domain conversion processing on the full-band audio domain representation of the speech signal to obtain a full-band estimation result of the speech signal.

[0206] Optionally, the modulation unit further includes:

[0207] A first modulation module is configured to modulate the noisy frequency domain representation of the i-th noisy frequency band using the i-th branch mask estimation result to obtain the i-th voice frequency domain representation;

[0208] The conversion module is used to perform time domain conversion processing on the i-th voice frequency domain representation to obtain the i-th frequency band estimation result of the speech signal.

[0209] Optionally, the above device further includes:

[0210] The sampling unit is used to sample the speech signal according to the target sampling rate to obtain the sampled audio data;

[0211] The processing unit is used to perform time-domain frame processing on the sampled audio data to obtain a processed audio signal.

[0212] Optionally, the above device further includes:

[0213] A first acquisition unit, configured to acquire a speech dataset and a noise dataset;

[0214] A mixing unit is used to mix the speech data set and the noise data set to obtain a sample noise frequency signal;

[0215] A training unit is used to train the initialized audio processing network using the sample noisy frequency signal until the loss function of the audio processing network reaches a convergence condition, wherein the loss function is used to calculate the difference between the speech signal in the speech dataset and the candidate reference speech signal recognized by the audio processing network during training from the sample noisy frequency signal.

[0216] For a specific embodiment, please refer to the example shown in the above-mentioned audio noise reduction processing method, and this embodiment will not be described in detail here.

[0217] According to another aspect of the embodiments of the present application, an electronic device for implementing the above-mentioned audio noise reduction processing method is also provided. This embodiment is described using the electronic device as a terminal. As shown in Figure 15, the electronic device includes a memory 1502 and a processor 1504. The memory 1502 stores a computer program, and the processor 1504 is configured to execute the steps of any of the above-mentioned method embodiments using the computer program.

[0218] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0219] Optionally, in this embodiment, the above-mentioned processor can be configured to execute the audio noise reduction processing method provided in the embodiment of the present application through a computer program.

[0220] Alternatively, those skilled in the art will appreciate that the structure shown in FIG15 is merely illustrative, and the electronic device may also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile internet device (MID), a PAD, or other terminal device. FIG15 does not limit the structure of the electronic device. For example, the electronic device may include more or fewer components (such as a network interface, etc.) than those shown in FIG15 , or may have a configuration different from that shown in FIG15 .

[0221] Memory 1502 can be used to store software programs and modules, such as program instructions / modules corresponding to the audio noise reduction processing method and apparatus in the embodiments of the present application. Processor 1504 executes the software programs and modules stored in memory 1502 to perform various functional applications and data processing, thereby implementing the aforementioned audio noise reduction processing method. Memory 1502 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some embodiments, memory 1502 may further include memory remotely located from processor 1504, which can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. Memory 1502 can specifically, but is not limited to, be used to store information such as the target audio signal. As an example, as shown in FIG. 15 , memory 1502 may, but is not limited to, include the acquisition unit 1402, extraction unit 1404, input unit 1406, modulation unit 1408, and conversion unit 1410 of the aforementioned audio noise reduction processing apparatus. In addition, it may also include but is not limited to other module units in the above-mentioned audio noise reduction processing device, which will not be repeated in this example.

[0222] Optionally, the transmission device 1506 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one embodiment, the transmission device 1506 includes a network interface controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In one embodiment, the transmission device 1506 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0223] In addition, the electronic device further includes: a connection bus 1508 for connecting various module components in the electronic device.

[0224] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting multiple nodes via network communication. The nodes may form a point-to-point network, and any computing device, such as a server, terminal, or other electronic device, may become a node in the blockchain system by joining the point-to-point network.

[0225] According to one aspect of the present application, a computer program product is provided, comprising a computer program / instructions containing program code for executing the above-described method. In such an embodiment, the computer program can be downloaded and installed from a network via a communication component and / or installed from a removable medium. When the computer program is executed by a central processing unit, the various functions provided in the embodiments of the present application are performed.

[0226] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned audio noise reduction processing method.

[0227] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0228] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above-mentioned computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application.

[0229] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0230] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, and can be electrical or other forms.

[0231] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0232] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0233] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An audio noise reduction processing method, performed by an electronic device, comprising: Acquire an audio signal to be processed, wherein the audio signal includes a speech signal to be recognized that has been interfered by a noise signal; Performing frequency domain conversion processing on the audio signal to obtain a noisy frequency domain representation corresponding to the audio signal; Dividing the noisy frequency domain representation into N noisy frequency bands, and inputting the N noisy frequency bands into N noise reduction branches in the audio processing network respectively, to obtain N branch mask estimation results, wherein the i-th noise reduction branch in the audio processing network is used to process the i-th noisy frequency band among the N noisy frequency bands to obtain the i-th branch mask estimation result corresponding to the i-th noisy frequency band, the N noise reduction branches have the same signal processing structure, i is a natural number greater than or equal to 1 and less than or equal to N, and N is a natural number greater than 1; Using the noisy frequency domain representation, modulate the N branch mask estimation results to obtain N speech frequency domain representations; The N audio domain representations of speech are subjected to time domain conversion processing to obtain the speech signal without interference of noise signal in the audio signal.

2. The method according to claim 1, wherein the N noisy frequency bands are respectively input into N noise reduction branches in the audio processing network to obtain N branch mask estimation results, comprising: In the i-th noise reduction branch, the following operations are performed on the i-th noisy frequency band: Performing feature dimension transformation on the i-th noisy frequency band to obtain an i-th noisy feature vector having a target feature length; Performing denoising on the i-th noisy feature vector to obtain an i-th denoising result; Performing an inverse transformation of the characteristic dimension on the i-th denoising result to obtain an i-th branch processing result having the same characteristic length as the i-th noisy frequency band; A mask estimation operation is performed on the i-th branch processing result to obtain the i-th branch mask estimation result corresponding to the i-th noisy frequency band.

3. The method according to claim 1 or 2, performing denoising on the i-th noisy feature vector to obtain the i-th denoising result, comprising: By using a coding network constructed based on a streaming convolution structure, the i-th noisy feature vector is encoded to obtain an i-th encoding result; By means of a recurrent neural network constructed based on a gated recurrent unit, the i-th encoding result is analyzed to obtain an i-th intermediate result carrying timing information; The i-th intermediate result is decoded by a decoding network constructed based on a streaming convolution structure to obtain the i-th denoising result, wherein the sub-network in the decoding network is obtained by adjusting the sub-network in the encoding network.

4. The method according to any one of claims 1 to 3, characterized in that: The encoding network constructed based on the streaming convolution structure is used to encode the i-th noisy feature vector to obtain the i-th encoding result, including: Through the M encoding sub-networks having a connection relationship in the encoding network, the i-th noisy feature vector is processed Row encoding processing is performed to obtain the i-th encoding result, wherein each of the encoding subnetworks includes a convolution layer, a normalization layer, and an activation layer. When the noisy feature vector of each frame is convolved in the convolution layer, the noisy feature vector of the adjacent previous frame is referenced, and M is a natural number greater than or equal to 2; The decoding network constructed based on the streaming convolution structure is used to decode the i-th intermediate result to obtain the i-th denoising result, including: The i-th intermediate result is decoded by M decoding sub-networks having a connection relationship in the decoding network to obtain the i-th denoising result, wherein each of the decoding sub-networks includes a transposed convolution layer, a normalization layer and an activation layer associated with the convolution layer, Among them, a jump connection is set between the kth encoding subnetwork and the M-(k-1)th decoding subnetwork, and k is a natural number greater than or equal to 1 and less than or equal to M.

5. The method according to any one of claims 1 to 4, when the decoding process is performed on the i-th intermediate result through the M decoding sub-networks having a connection relationship in the decoding network to obtain the i-th denoising result, further comprising: In the case where the i-th denoising branch is not the first denoising branch, weighted summing is performed on the output results corresponding to each of the M coding sub-networks in the coding network in the i-th denoising branch, and the M gating processing results associated with the i-1-th denoising branch, to obtain M decoding reference results, wherein the j-th gating processing result is obtained by processing the output result of the j-th coding sub-network in the i-1-th denoising branch through the j-th information transmission gating structure in the audio processing network, and each convolution layer in the information transmission gating structure includes at least two convolution structures, and j is a natural number greater than or equal to 1 and less than or equal to M; Each of the M decoding reference results is input into a corresponding decoding sub-network in the M decoding sub-networks in the i-th noise reduction branch.

6. The method according to any one of claims 1 to 5, wherein the step of using the noisy frequency domain representation to modulate the N branch mask estimation results to obtain N speech frequency domain representations comprises: splicing the N branch mask estimation results to obtain a spliced ​​expression; The concatenated expression is modulated using the noisy frequency domain representation to obtain a full-band speech frequency domain representation.

7. The method according to any one of claims 1 to 6, wherein the step of performing time domain conversion processing on the N speech frequency domain representations to obtain the speech signal in the audio signal comprises: The full-band audio-domain representation of the speech is subjected to time-domain conversion processing to obtain a full-band estimation result of the speech signal.

8. The method according to any one of claims 1 to 7, before concatenating the N branch mask estimation results to obtain a concatenated expression, further comprising: Using the i-th branch mask estimation result, modulate the noisy frequency domain representation of the i-th noisy frequency band to obtain the i-th speech frequency domain representation; The i-th frequency domain representation of the speech is subjected to time domain conversion processing to obtain an i-th frequency band estimation result of the speech signal.

9. The method according to any one of claims 1 to 8, before performing frequency domain conversion processing on the audio signal to obtain a noisy frequency domain representation corresponding to the target audio signal, further comprising: Sampling the audio signal at a target sampling rate to obtain sampled audio data; The sampled audio data is subjected to time-domain frame processing to obtain a processed audio signal.

10. The method according to any one of claims 1 to 9, before obtaining the audio signal to be processed, further comprising: Get speech data set and noise data set; Mixing the speech data set and the noise data set to obtain a sample noise-containing frequency signal; The initialized audio processing network is trained using the sample noisy audio signal until a loss function of the audio processing network reaches a convergence condition, wherein the loss function is used to calculate the difference between the speech signal in the speech data set and the candidate reference speech signal identified by the audio processing network in training from the sample noisy audio signal.

11. An audio noise reduction processing device, characterized in that: include: An acquisition unit, configured to acquire an audio signal to be processed, wherein the audio signal includes a speech signal to be recognized that has been interfered by a noise signal; An extraction unit, configured to perform frequency domain conversion processing on the audio signal to obtain a noisy frequency domain representation corresponding to the audio signal; An input unit, used for dividing the noisy frequency domain representation into N noisy frequency bands, and inputting the N noisy frequency bands into N noise reduction branches in the audio processing network respectively, to obtain N branch mask estimation results, wherein the i-th noise reduction branch in the audio processing network is used to process the i-th noisy frequency band among the N noisy frequency bands to obtain the i-th branch mask estimation result corresponding to the i-th noisy frequency band, the N noise reduction branches have the same signal processing structure, i is a natural number greater than or equal to 1 and less than or equal to N, and N is a natural number greater than 1; A modulation unit, configured to modulate the N branch mask estimation results using the noisy frequency domain representation to obtain the N speech frequency domain representations; The conversion unit is used to perform time domain conversion processing on the N audio domain representations of speech to obtain the speech signal without interference of noise signal in the audio signal.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program is executed by a processor to perform the method described in any one of claims 1 to 10.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 10 through the computer program.

Images