A method and apparatus for training a speech enhancement network, and an electronic device

Abstract

Embodiments of the present application provide a speech enhancement network training method and device and electronic equipment, and relate to the technical field of speech processing, which is used to suppress non-stationary noise in audio while suppressing stationary noise in audio. The method comprises: obtaining a training sample set, wherein the training sample set comprises noisy audio data and noise-free audio data; extracting first feature data of an i-th noisy audio frame in the training sample set; performing a denoising operation on M sub-bands included in the i-th noisy audio frame to obtain an i-th first denoised audio frame according to the first feature data of the i-th noisy audio frame and second feature data of an (i-1)-th noisy audio frame; and adjusting network parameters of the speech enhancement network according to a first loss value of the i-th first denoised audio frame and a noise-free audio frame corresponding to the i-th noisy audio frame.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a training method, apparatus and electronic device for a speech enhancement network. Background Technology

[0002] In virtual reality (VR) or augmented reality (AR) devices, audio information, besides video, is another crucial element significantly impacting the user's immersive experience. This is especially true in VR / AR devices with call or voice interaction capabilities, where audio quality is paramount. High-quality voice enhancement technology can significantly improve the user experience and provide a better foundation for spatial audio technology. Furthermore, in conference call scenarios, the low signal-to-noise ratio (SNR) of long-distance voice pickup (meaning the voice energy is typically lower than the ambient noise) makes high-performance long-distance voice pickup a significant technical challenge in current conference call systems.

[0003] Based on the above scenarios, improving noise reduction capabilities has become a major research direction for existing technologies. However, the suppression capabilities of existing technologies for non-stationary noise such as impulse noise are usually not ideal; that is, there is a lack of a method in existing technologies that can suppress both stationary and non-stationary noise. Summary of the Invention

[0004] This application provides a training method, apparatus, and electronic device for a speech enhancement network, which can suppress both stationary and non-stationary noise in audio.

[0005] In a first aspect, embodiments of this application provide a method for training a speech enhancement network, comprising:

[0006] Obtain a training sample set, which includes noise-added frequency data and noise-free frequency data. The noise-added frequency data is obtained by adding noise to the noise-free frequency data. The noise-free frequency data includes N noise-free frequency frames, and the noise-added frequency data includes N noise-added frequency frames that correspond one-to-one with the N noise-free frequency frames; N is a positive integer. Extract the first feature data of the i-th noise-added frequency frame from the training sample set. The first feature data is used to describe the spectral characteristics of the i-th noise-added frequency frame; i is a positive integer less than or equal to N. Based on the first feature data of the i-th noise-added frequency frame and the first feature data of the (i-1)-th noise-added frequency frame... The second feature data is obtained by performing denoising operations on the M sub-bands included in the i-th noisy frequency frame to obtain the i-th first denoised frequency frame; the second feature data is obtained by the speech enhancement network during the denoising operation on the (i-1)-th noisy frequency frame; the second feature data is used to describe the feature correlation between each sub-band of the M sub-bands of the (i-1)-th noisy frequency frame and the other M-1 sub-bands, where M is a positive integer; the network parameters of the speech enhancement network are adjusted according to the first loss value of the i-th first denoised frequency frame and the noise-free frequency frame corresponding to the i-th noisy frequency frame.

[0007] Based on the above scheme, since the first feature data is used to describe the spectral characteristics of the i-th noisy frequency frame, the first feature data of the i-th noisy frequency frame can better preserve the speech signal. Performing denoising operations on the M sub-bands of the i-th noisy frequency frame can suppress stationary noise in the i-th noisy frequency frame. Since the second feature data shows the feature correlation between each sub-band of the M sub-bands of the (i-1)-th noisy frequency frame and the other M-1 sub-bands, the feature data output by each sub-band can contain important global information about that sub-band from the other sub-bands. Therefore, by using the second feature data of the previous frame, non-stationary noise spanning multiple frequency bands can be suppressed while suppressing stationary noise.

[0008] In one possible implementation, the speech enhancement network includes a first neural network and a second neural network. Based on the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame, denoising operations are performed on the M sub-frequency bands included in the i-th noisy frequency frame to obtain an i-th first denoised frequency frame. This includes: using the first neural network, performing stationary noise removal based on the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame to obtain third feature data of the M sub-frequency bands in the i-th noisy frequency frame; using the second neural network, performing non-stationary noise removal based on the third feature data of the M sub-frequency bands in the i-th noisy frequency frame to obtain second feature data of the i-th noisy frequency frame; and converting the second feature data of the i-th noisy frequency frame into the i-th first denoised frequency frame.

[0009] Based on the above scheme, the second feature data output by the second neural network is fed back into the input of the first neural network in the next frame, so that the feature data input to the first neural network of each sub-frequency band takes into account the important global information of other sub-frequency bands for that sub-frequency band, enabling the first neural network to better suppress non-stationary noise.

[0010] In one possible implementation, the first feature data includes one of the following: log power spectrum (LPS), phase difference, filter bank-based feature Fbank, and Mel-frequency cepstral coefficients (MFCC).

[0011] Based on the above scheme, the first feature data can be different feature data, making the training method of the speech enhancement network provided in this application embodiment more flexible.

[0012] In one possible implementation, the method further includes: extracting fourth feature data of the i-th noisy frequency frame; the fourth feature data is used to describe the spectral characteristics of the i-th noisy frequency frame, and the fourth feature data includes one of LPS, phase difference, Fbank, and MFCC that is different from the first feature data.

[0013] Based on the above scheme, the fourth feature data can be different feature data, making the training method of the speech enhancement network provided in this application embodiment more flexible.

[0014] In one possible implementation, the speech enhancement network further includes a third neural network, and the method further includes: using the third neural network to perform non-stationary noise removal based on the first feature data of the i-th noisy frequency frame to obtain the fifth feature data of the i-th noisy frequency frame; or, using the third neural network to perform non-stationary noise removal based on the fourth feature data of the i-th noisy frequency frame to obtain the fifth feature data of the i-th noisy frequency frame; the step of using the first neural network to perform stationary noise removal based on the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame to obtain the third feature data of M sub-frequency bands in the i-th noisy frequency frame includes: using the first neural network to perform stationary noise removal based on the first feature data of the i-th noisy frequency frame, the second feature data of the (i-1)-th noisy frequency frame, and the fifth feature data of the i-th noisy frequency frame to obtain the third feature data of M sub-frequency bands in the i-th noisy frequency frame.

[0015] Based on the above scheme, the second feature data output by the second neural network is fed back into the input of the first neural network in the next frame. This ensures that the feature data input to the first neural network for each sub-frequency band takes into account the important global information of other sub-frequency bands for that sub-frequency band, enabling the first neural network to better suppress non-stationary noise. Furthermore, since the full-band network has a strong ability to eliminate non-stationary noise, the third neural network can further suppress non-stationary noise.

[0016] In one possible implementation, obtaining the second feature data of the i-th noisy frequency frame by performing non-stationary noise removal based on the third feature data of the M sub-frequency bands in the i-th noisy frequency frame using the second neural network includes: performing non-stationary noise removal based on the third feature data of the M sub-frequency bands in the i-th noisy frequency frame and the first feature data of the i-th noisy frequency frame using the second neural network to obtain the second feature data of the i-th noisy frequency frame; or, performing non-stationary noise removal based on the third feature data of the M sub-frequency bands in the i-th noisy frequency frame and the fourth feature data of the i-th noisy frequency frame using the second neural network to obtain the second feature data of the i-th noisy frequency frame.

[0017] Based on the above scheme, inputting the first or fourth feature data of the i-th noisy frequency frame into the second neural network can avoid excessive signal cancellation. Furthermore, it makes the training method of the speech enhancement network provided in this application embodiment more flexible and scalable.

[0018] In one possible implementation, the method further includes: converting the third feature data of the i-th noisy frequency frame into the i-th second denoised frequency frame; determining the i-th third denoised frequency frame using the fifth feature data of the i-th noisy frequency frame; adjusting the network parameters of the speech enhancement network based on the first loss value of the i-th first denoised frequency frame and the noise-free frequency frame corresponding to the i-th noisy frequency frame includes: adjusting the network parameters of the speech enhancement network based on the weighted result of the first loss value, the second loss value, and the third loss value; the first loss value is determined based on the i-th first denoised frequency frame and the noise-free frequency frame corresponding to the i-th noisy frequency frame, the second loss value is determined based on the noise-free frequency frame corresponding to the i-th noisy frequency frame and the i-th second denoised frequency frame, and the third loss value is determined based on the noise-free frequency frame corresponding to the i-th noisy frequency frame and the i-th third denoised frequency frame.

[0019] Based on the above scheme, when adjusting the network parameters of the speech enhancement network by weighting the first loss value, the second loss value, and the third loss value, the second loss value and the third loss value can play a supervisory role, making the noise reduction effect of the speech enhancement network better.

[0020] Secondly, embodiments of this application provide an audio processing method based on a speech enhancement network, comprising: acquiring noise reduction frequency data; the noise reduction frequency data includes P noise reduction frequency frames; P is a positive integer; extracting first feature data of the j-th noise reduction frequency frame from the noise reduction frequency data, the first feature data being used to describe the spectral characteristics of the j-th noise reduction frequency frame; j is a positive integer less than or equal to P; and, through a speech enhancement network, processing the j-th noise reduction frequency frame according to the first feature data of the j-th noise reduction frequency frame and the second feature data of the (j-1)-th noise reduction frequency frame. The audio frame undergoes a denoising operation to obtain the audio denoising result of the j-th noise-reducing frequency frame; the second feature data is obtained by the speech enhancement network during the denoising operation of the (j-1)-th noise-reducing frequency frame; the second feature data is used to describe the feature correlation between each sub-frequency band in the M sub-frequency bands of the (j-1)-th noise-reducing frequency frame and the other M-1 sub-frequency bands, where M is a positive integer; the speech enhancement network is trained using the training method described in the first aspect; based on the audio denoising results of each noise-reducing frequency frame in the P noise-reducing frequency frames, the denoised audio data is obtained.

[0021] Thirdly, embodiments of this application provide a training apparatus for a speech enhancement network, comprising: an acquisition unit, configured to acquire a training sample set, the training sample set including noise-added frequency data and noise-free frequency data, wherein the noise-added frequency data is obtained by adding noise to the noise-free frequency data; the noise-free frequency data includes N noise-free frequency frames, and the noise-added frequency data includes N noise-added frequency frames corresponding one-to-one with the N noise-free frequency frames; N is a positive integer;

[0022] The processing unit is configured to perform the following processing: extracting first feature data of the i-th noisy frequency frame in the training sample set, wherein the first feature data describes the spectral characteristics of the i-th noisy frequency frame; i is a positive integer less than or equal to N; performing denoising operations on the M sub-bands included in the i-th noisy frequency frame according to the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame to obtain the i-th first denoised frequency frame; the second feature data is obtained by the speech enhancement network during the denoising operation on the (i-1)-th noisy frequency frame; the second feature data describes the feature correlation between each sub-band of the M sub-bands of the (i-1)-th noisy frequency frame and the other M-1 sub-bands, where M is a positive integer; and adjusting the network parameters of the speech enhancement network according to the first loss value of the i-th first denoised frequency frame and the noise-free frequency frame corresponding to the i-th noisy frequency frame.

[0023] In one possible implementation, the speech enhancement network includes a first neural network and a second neural network; the processing unit performs denoising operations on the M sub-bands included in the i-th noisy frequency frame according to the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame to obtain the i-th first denoised frequency frame, and is used to: perform stationary noise removal through the first neural network according to the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame to obtain the third feature data of the M sub-bands in the i-th noisy frequency frame; perform non-stationary noise removal through the second neural network according to the third feature data of the M sub-bands in the i-th noisy frequency frame to obtain the second feature data of the i-th noisy frequency frame; and convert the second feature data of the i-th noisy frequency frame into the i-th first denoised frequency frame.

[0024] In one possible implementation, the first feature data includes one of logarithmic power spectrum, phase difference, filter bank-based feature Fbank, and Mel-frequency cepstral coefficients (MFCC).

[0025] In one possible implementation, the processing unit is further configured to: extract fourth feature data of the i-th noisy frequency frame; the fourth feature data is used to describe the spectral characteristics of the i-th noisy frequency frame, and the fourth feature data includes one of LPS, phase difference, Fbank, and MFCC that is different from the first feature data.

[0026] In one possible implementation, the speech enhancement network further includes a third neural network, and the processing unit is further configured to: perform non-stationary noise removal based on the first feature data of the i-th noisy frequency frame using the third neural network to obtain the fifth feature data of the i-th noisy frequency frame; or, perform non-stationary noise removal based on the fourth feature data of the i-th noisy frequency frame using the third neural network to obtain the fifth feature data of the i-th noisy frequency frame; the processing unit, using the first neural network, performs stationary noise removal based on the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame to obtain the third feature data of M sub-frequency bands in the i-th noisy frequency frame, and is configured to: perform stationary noise removal based on the first feature data of the i-th noisy frequency frame, the second feature data of the (i-1)-th noisy frequency frame, and the fifth feature data of the i-th noisy frequency frame to obtain the third feature data of M sub-frequency bands in the i-th noisy frequency frame.

[0027] In one possible implementation, the processing unit obtains the second feature data of the i-th noisy frequency frame by performing non-stationary noise removal based on the third feature data of the M sub-frequency bands in the i-th noisy frequency frame using the second neural network. This is used to: obtain the second feature data of the i-th noisy frequency frame by performing non-stationary noise removal based on the third feature data of the M sub-frequency bands in the i-th noisy frequency frame and the first feature data of the i-th noisy frequency frame using the second neural network; or, obtain the second feature data of the i-th noisy frequency frame by performing non-stationary noise removal based on the third feature data of the M sub-frequency bands in the i-th noisy frequency frame and the fourth feature data of the i-th noisy frequency frame using the second neural network.

[0028] In one possible implementation, the processing unit is further configured to: convert the third feature data of the i-th noisy frequency frame into the i-th second denoised frequency frame; determine the i-th third denoised frequency frame from the fifth feature data of the i-th noisy frequency frame; the processing unit adjusts the network parameters of the speech enhancement network according to the first loss value of the i-th first denoised frequency frame and the noise-free frequency frame corresponding to the i-th noisy frequency frame, configured to: adjust the network parameters of the speech enhancement network according to the weighted result of the first loss value, the second loss value, and the third loss value; the first loss value is determined based on the i-th first denoised frequency frame and the noise-free frequency frame corresponding to the i-th noisy frequency frame, the second loss value is determined based on the noise-free frequency frame corresponding to the i-th noisy frequency frame and the i-th second denoised frequency frame, and the third loss value is determined based on the noise-free frequency frame corresponding to the i-th noisy frequency frame and the i-th third denoised frequency frame.

[0029] Fourthly, embodiments of this application provide an audio processing apparatus based on a speech enhancement network, comprising: an acquisition unit, configured to acquire noise reduction frequency data; the noise reduction frequency data includes P noise reduction frequency frames; P is a positive integer;

[0030] The processing unit is configured to extract the first feature data of the j-th noise-to-be-reduced frequency frame from the noise-to-be-reduced frequency data, wherein the first feature data describes the spectral characteristics of the j-th noise-to-be-reduced frequency frame; j is a positive integer less than or equal to P; through a speech enhancement network, based on the first feature data of the j-th noise-to-be-reduced frequency frame and the second feature data of the (j-1)-th noise-to-be-reduced frequency frame, a denoising operation is performed on the j-th noise-to-be-reduced frequency frame to obtain the audio denoising result of the j-th noise-to-be-reduced frequency frame; the second feature data is obtained during the denoising operation performed by the speech enhancement network on the (j-1)-th noise-to-be-reduced frequency frame; the second feature data describes the feature correlation between each of the M sub-frequency bands of the (j-1)-th noise-to-be-reduced frequency frame and the other M-1 sub-frequency bands, where M is a positive integer; the speech enhancement network is trained using the training method described in the first aspect; and based on the audio denoising results of each of the P noise-to-be-reduced frequency frames, denoised audio data is obtained.

[0031] Fifthly, embodiments of this application provide an electronic device, including:

[0032] Memory, used to store computer instructions;

[0033] A processor, connected to the memory, is configured to execute computer instructions in the memory, and, in executing the computer instructions, implement the method as described in any one of the first aspects.

[0034] Sixthly, embodiments of this application provide a computer-readable storage medium, comprising:

[0035] The computer-readable storage medium stores computer instructions that, when executed on a computer, cause the computer to perform the method as described in any one of the first aspects.

[0036] For the various aspects of the second to sixth aspects mentioned above, and the technical effects that each aspect may achieve, please refer to the above description of the technical effects that can be achieved for the first aspect or the various possible solutions in the first aspect, which will not be repeated here. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application.

[0038] Figure 1 A schematic diagram of the system structure to which the training method of the speech enhancement network provided in the embodiments of this application is applicable;

[0039] Figure 2 An exemplary flowchart illustrating a training method for a speech enhancement network provided in an embodiment of this application;

[0040] Figure 3 This is one of the flowcharts illustrating the noise reduction operation via a speech enhancement network provided in an embodiment of this application.

[0041] Figure 4A This is one of the schematic diagrams of the speech enhancement network structure provided in the embodiments of this application;

[0042] Figure 4B This is one of the flowcharts illustrating the noise reduction operation via a speech enhancement network provided in an embodiment of this application.

[0043] Figure 4C A schematic diagram of the input feature data of the first neural network provided in the embodiments of this application;

[0044] Figure 4D This is one of the flowcharts illustrating the noise reduction operation via a speech enhancement network provided in an embodiment of this application.

[0045] Figure 5A This is one of the schematic diagrams of the speech enhancement network structure provided in the embodiments of this application;

[0046] Figure 5B This is one of the flowcharts illustrating the noise reduction operation via a speech enhancement network provided in an embodiment of this application.

[0047] Figure 6 A schematic diagram illustrating the process of a self-attention network performing denoising operations according to an embodiment of this application;

[0048] Figure 7A This is one of the schematic diagrams of the speech enhancement network structure provided in the embodiments of this application;

[0049] Figure 7B This is one of the flowcharts illustrating the noise reduction operation via a speech enhancement network provided in an embodiment of this application.

[0050] Figure 7C This is one of the flowcharts illustrating the noise reduction operation via a speech enhancement network provided in an embodiment of this application.

[0051] Figure 7D This is one of the schematic diagrams of the speech enhancement network structure provided in the embodiments of this application;

[0052] Figure 7E This is one of the flowcharts illustrating the noise reduction operation via a speech enhancement network provided in an embodiment of this application.

[0053] Figure 8 This is a schematic diagram illustrating the noise reduction effect provided in an embodiment of this application;

[0054] Figure 9A This is one of the flowcharts illustrating the noise reduction operation via a speech enhancement network provided in an embodiment of this application.

[0055] Figure 9B This is one of the flowcharts illustrating the noise reduction operation via a speech enhancement network provided in an embodiment of this application.

[0056] Figure 10 An exemplary flowchart of an audio processing method based on a speech enhancement network provided in this application embodiment;

[0057] Figure 11 A schematic diagram of a training device for a speech enhancement network provided in an embodiment of this application;

[0058] Figure 12 A schematic diagram of an audio processing device based on a speech enhancement network provided in an embodiment of this application;

[0059] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0060] To facilitate understanding of the technical solutions provided in the embodiments of this application, the following describes the technical terms involved in the embodiments of this application.

[0061] (1) Filter bank (Fbank): It is a frequency domain feature based on cepstral extraction. Because it is more in line with auditory response characteristics, it has become a commonly used audio feature in speech recognition.

[0062] (2) Mel-Frequency Cepstral Coefficient (MFCC): These are the coefficients that make up the Mel-Frequency Cepstral, which is a linear transformation of the logarithmic energy spectrum based on the nonlinear Mel scale of the sound frequency.

[0063] (3) Stationary noise refers to noise whose statistical characteristics do not change over time.

[0064] (4) Non-stationary noise refers to noise whose statistical characteristics change over time.

[0065] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0066] The terms "first" and "second" in the embodiments of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The term "multiple" in this application can mean at least two, for example, two, three, or more; the embodiments of this application do not impose any limitations.

[0067] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0068] Currently, in VR / AR devices with call or voice interaction functions, the quality of sound information is particularly important. High-quality voice enhancement technology can significantly improve the user experience and provide a better foundation for spatial audio technology. Furthermore, in conference call scenarios, ultra-long-distance voice pickup typically has a low signal-to-noise ratio, meaning the energy of the voice is usually lower than the ambient noise floor. This makes high-performance long-distance voice pickup a technical challenge in current conference call systems. Based on these scenarios, improving noise reduction capabilities has become a major research direction for existing technologies. However, the suppression capabilities of existing technologies for non-stationary noise such as impulse noise are generally not ideal; that is, there is a lack of methods in current technologies that can suppress both stationary and non-stationary noise.

[0069] In view of this, embodiments of this application provide a training method for a speech enhancement network. This method performs denoising operations on the M sub-bands of the i-th noisy frequency frame based on the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame. This allows the network parameters of the speech enhancement network to be adjusted based on the first loss value of the i-th denoised frequency frame and the corresponding noise-free frequency frame, thereby training the speech enhancement network.

[0070] Using the above method, since the first feature data is used to describe the spectral characteristics of the i-th noisy frequency frame, the first feature data of the i-th noisy frequency frame can better preserve the speech signal. Performing denoising operations on the M sub-bands of the i-th noisy frequency frame can suppress stationary noise in the i-th noisy frequency frame. Since the second feature data shows the feature correlation between each sub-band of the (i-1)-th noisy frequency frame and the other M-1 sub-bands, the feature data output by each sub-band can contain important global information about that sub-band from the other sub-bands. Therefore, using the second feature data from the previous frame, non-stationary noise spanning multiple frequency bands can be suppressed while suppressing stationary noise.

[0071] See Figure 1This is a schematic diagram of a system architecture applicable to the training method of the speech enhancement network provided in this application embodiment. The system architecture can be a server 100, including a memory 110, a processor 120, and a communication interface 130. The communication interface 130 can be used to acquire noise-added frequency data and noise-free frequency data from the training sample set. The processor 120 is the control center of the server 100, connecting various parts of the server 100 through various interfaces and lines. It executes various functions of the server 100 and processes data by running or executing software programs or modules stored in the memory 110 and calling data stored in the memory 110. Optionally, the processor 120 may include one or more processing units. The memory 110 can be a high-speed random access memory or a non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. It should be noted that the above... Figure 1 The structure shown is merely an example, and the embodiments of this application are not limited thereto.

[0072] See Figure 2 An exemplary flowchart of a training method for a speech enhancement network provided in this application embodiment may include the following steps:

[0073] S201, Obtain the training sample set.

[0074] The training sample set includes noisy frequency data and noise-free frequency data. The noisy frequency data is obtained by adding noise to the noise-free frequency data. The noise-free frequency data includes N noise-free frequency frames, and the noisy frequency data includes N noisy frequency frames that correspond one-to-one with the N noise-free frequency frames. N is a positive integer.

[0075] In some embodiments, the noise-free frequency data included in the training sample set may be a continuous segment of noise-free frequency data comprising N frames, serving as N noise-free frequency frames. By adding noise to this segment of noise-free frequency data, a continuous segment of noise-added frequency data comprising N frames is obtained, serving as N noise-added frequency frames corresponding one-to-one with the N noise-free frequency frames.

[0076] In other embodiments, the noise-free frequency data included in the training sample set may also be K segments of noise-free frequency data, with the sum of the number of frames in the K segments being N. These K segments of noise-free frequency data are then used as N noise-free frequency frames. The number of frames in each noise-free frequency data segment can be the same or different, and K is an integer greater than 1. Noise is then added to each of the K segments of noise-free frequency data to obtain K segments of noise-added frequency data, with the sum of the number of frames in the K segments of noise-added frequency data being N. These N noise-added frequency frames correspond one-to-one with the N noise-free frequency frames.

[0077] S202, extract the first feature data of the i-th noisy frequency frame in the training sample set.

[0078] In this embodiment of the application, the first feature data is used to describe the spectral characteristics of the i-th noisy frequency frame. i is a positive integer less than or equal to N.

[0079] In some embodiments, the first feature data may be obtained by feature extraction from the i-th noisy frequency frame, and may include one of the following: Log Power Spectrum (LPS), phase difference, Fbank, and MFCC. It should be understood that the feature extraction method can be selected based on actual circumstances and experience, and this application does not limit it.

[0080] In some embodiments, when the noise-added frequency data and noise-free frequency data in the training sample set are acquired by a microphone array, the first feature data may also include spatial features such as the sinusoidal phase difference (sinIPD) and cosine phase difference (cosIPD) between the microphones.

[0081] Optionally, the first feature data may also include feature data for describing other features of the noisy frequency frame, such as frequency features, etc., which are not limited in this application.

[0082] S203, based on the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame, perform denoising operations on the M sub-bands included in the i-th noisy frequency frame to obtain the i-th first denoised frequency frame.

[0083] Where M is a positive integer, the second feature data of the (i-1)th noisy frequency frame is obtained during the denoising operation performed on the (i-1)th noisy frequency frame by the speech enhancement network. The second feature data is used to describe the feature correlation between each of the M sub-frequency bands of the (i-1)th noisy frequency frame and the other M-1 sub-frequency bands.

[0084] See Figure 3 This is one of the flowcharts illustrating a denoising operation performed via a speech enhancement network according to an embodiment of this application. The server can input the first feature data of the i-th noisy frequency frame obtained in S202 and the second feature data of the (i-1)-th noisy frequency frame into the speech enhancement network. After performing the denoising operation, the speech enhancement network can obtain the second feature data of the i-th noisy frequency frame. The speech enhancement network can then process the second feature data to obtain the i-th first denoised frequency frame. The second feature data of the i-th noisy frequency frame can be used to input the first feature data of the (i+1)-th noisy frequency frame into the speech enhancement network together with the first feature data of the (i+1)-th noisy frequency frame during the denoising operation.

[0085] S204, adjust the network parameters of the speech enhancement network according to the first loss value of the noise-free frequency frame corresponding to the i-th first denoised frequency frame and the i-th noise-added frequency frame.

[0086] In some embodiments, the first loss value may be obtained according to a preset loss function. For example, the first loss value may be the mean square error (MSE) loss based on the ideal amplitude mask (IAM) algorithm, which satisfies formula (1).

[0087]

[0088] In the formula, MSE i This is used to represent the first loss value, where i represents the frame number corresponding to the noise-added frequency frame, and f represents the f-th frequency point. IAM (i,f) Let represent the ideal amplitude audio masking corresponding to the i-th noise-added frequency frame, satisfying formula (2). Let represent the predicted audio masking result (Mask) of the speech enhancement network, where the predicted audio masking result is obtained based on the i-th first denoised frequency frame.

[0089]

[0090] In the formula, S(t,f) represents the amplitude spectrum of the noise-free frequency frame corresponding to the i-th noise-added frequency frame, and Y(t,f) represents the amplitude spectrum of the i-th noise-added frequency frame.

[0091] It should be noted that the preset loss function can also be a frequency domain masking loss function based on (Phase Shift Mask, PSM) or a time domain loss function such as scale-invariant signal-to-noise ratio (Si-SDR). This application does not limit this.

[0092] In one possible implementation, the server can terminate training and obtain the trained speech enhancement network when a preset end-of-training condition is met. For example, the end-of-training condition could be that the first loss value reaches a preset threshold. This application does not limit the end-of-training condition.

[0093] Based on the above scheme, since the first feature data is used to describe the spectral characteristics of the i-th noisy frequency frame, the first feature data of the i-th noisy frequency frame can better preserve the speech signal. Performing denoising operations on the M sub-bands of the i-th noisy frequency frame can suppress stationary noise in the i-th noisy frequency frame. Since the second feature data shows the feature correlation between each sub-band of the M sub-bands of the (i-1)-th noisy frequency frame and the other M-1 sub-bands, the feature data output by each sub-band can contain important global information about that sub-band from the other sub-bands. Therefore, by using the second feature data of the previous frame, non-stationary noise spanning multiple frequency bands can be suppressed while suppressing stationary noise.

[0094] Below, in order to more clearly understand the solution proposed in the embodiments of this application, a training method for a speech enhancement network provided by this application will be introduced in conjunction with specific embodiments.

[0095] In one possible implementation, see [link to relevant documentation]. Figure 4A This is one of the schematic diagrams of the speech enhancement network structure provided in the embodiments of this application, such as... Figure 4A As shown, the speech enhancement network includes a first neural network and a second neural network. The first neural network performs stationary noise removal based on the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame to obtain the third feature data of M sub-frequency bands in the i-th noisy frequency frame. The second neural network performs non-stationary noise removal based on the third feature data of the M sub-frequency bands in the i-th noisy frequency frame to obtain the second feature data of the i-th noisy frequency frame. The second feature data of the i-th noisy frequency frame is then converted into the i-th first denoised frequency frame.

[0096] In some embodiments, the server can concatenate the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame as the input feature data of the first neural network, and divide it into M sub-bands, determining the input feature data of each sub-band. Then, the input feature data of each sub-band is input into the first neural network for smooth noise removal to obtain the third feature data of each sub-band. Finally, the third feature data of each sub-band are merged to obtain the third feature data of the M sub-bands in the i-th noisy frequency frame.

[0097] For example, assuming M is 16, the first feature data of the i-th noisy frequency frame includes LPS, sinIPD, and cosIPD of the 256-band frequency. The first feature data and the second feature data of the (i-1)-th noisy frequency frame are then concatenated as the input feature data of the first neural network. The dimension of the input feature data of the first neural network can be [256, 4]. The input feature data of the first neural network can then be divided into 16 sub-bands, with the dimension of the input feature data for each sub-band being [16, 4]. The first neural network can perform stationary noise removal on each of the 16 sub-bands. Since the 16 sub-bands share a set of network parameters, the third feature data of each sub-band is obtained. The third feature data of each sub-band is then merged to obtain the third feature data of the 16 sub-bands with a dimension of [16, 16].

[0098] It should be noted that the sub-band division method can be selected according to the actual situation, and this application does not limit it.

[0099] See Figure 4B This is one of the flowcharts illustrating noise reduction operations via a speech enhancement network provided in an embodiment of this application. The server can first determine, for example... Figure 4C The input feature data of the first neural network shown includes the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame. For example... Figure 4B As shown, when performing denoising on the i-th noisy frequency frame, the server can divide the input feature data of the first neural network into M sub-bands and determine the input feature data of each sub-band. Then, the input feature data of each sub-band is input into the first neural network for stationary noise removal, resulting in the third feature data of each sub-band. The third feature data of each sub-band is then merged to obtain the third feature data of the M sub-bands in the i-th noisy frequency frame. The third feature data of the M sub-bands in the i-th noisy frequency frame can then be input into the second neural network for non-stationary noise removal, resulting in the second feature data of the i-th noisy frequency frame. The second neural network can also process the second feature data of the i-th noisy frequency frame to output the i-th first denoised frequency frame. The second feature data of the i-th noisy frequency frame can be used to input the first feature data of the (i+1)-th noisy frequency frame into the first neural network during denoising.

[0100] based on Figure 4A The speech enhancement network shown is as follows: Figure 4DAs shown, when performing denoising on the i-th noisy frequency frame, the first neural network can also convert the third feature data of each sub-band into the i-th second denoised frequency frame, and determine the second loss value based on the noise-free frequency frame corresponding to the i-th noisy frequency frame and the i-th second denoised frequency frame. The server can adjust the network parameters of the first neural network and the second neural network according to the weighted result of the first loss value and the second loss value, and end the training when a preset end-training condition is reached, thus obtaining the trained speech enhancement network. For example, the preset end-training condition can be that the weighted result of the first loss value and the second loss value reaches a preset threshold. The method for determining the first loss value can be found in the relevant description in the above method embodiments, and will not be repeated here.

[0101] It should be noted that the calculation method for the second loss value can be found in the calculation method for the first loss value, and will not be repeated here.

[0102] In one possible implementation, see [link to relevant documentation]. Figure 5A This is one of the schematic diagrams of a speech enhancement network structure provided in an embodiment of this application. For example... Figure 5A As shown, a speech enhancement network can include L first neural networks and second neural networks. See also Figure 5B This is one of the flowcharts illustrating noise reduction operations using a speech enhancement network, as provided in an embodiment of this application. Figure 5B As shown, for any one of the M sub-bands, the server can input the input feature data of the sub-band into the first neural network corresponding to that sub-band according to the preset correspondence between the sub-bands and the first neural network, to obtain the third feature data of that sub-band. Here, L is a positive integer; the case where L equals 1 can be seen in... Figures 4A-4D The relevant descriptions in the illustrated embodiments will not be repeated here.

[0103] In one example, L can be equal to M. In this case, the correspondence between the preset sub-bands and the first neural network can be one-to-one between M sub-bands and L first neural networks. That is, when the server inputs the input feature data of each sub-band into the first neural network for smooth noise removal, it can input the input feature data of each sub-band into the first neural network corresponding to that sub-band.

[0104] For example, when both L and M are equal to 4, the server can input the feature data of sub-band 1 into the first neural network A corresponding to sub-band 1, input the feature data of sub-band 2 into the first neural network B corresponding to sub-band 2, input the feature data of sub-band 3 into the first neural network C corresponding to sub-band 3, and input the feature data of sub-band 4 into the first neural network D corresponding to sub-band 4.

[0105] In another example, L can be less than M. In this case, the predefined correspondence between the sub-bands and the first neural network can be that multiple sub-bands out of M sub-bands correspond to one sub-band out of L first neural networks. That is, the server can input the input feature data of multiple sub-bands out of M sub-bands into one of the L first neural networks. For example, assuming L equals 2 and M equals 4, the server can input the feature data of sub-band 1 and sub-band 2 into first neural network A respectively, and input the feature data of sub-band 3 and sub-band 4 into first neural network B respectively. The server can also input the feature data of sub-band 1, sub-band 2, and sub-band 3 into first neural network A respectively, and input the feature data of sub-band 4 into first neural network B respectively.

[0106] It should be understood that the above-mentioned pre-set sub-band correspondence with the first neural network can be set according to the actual situation, and this application does not limit it.

[0107] It should be noted that the first neural network can be a sub-bank module used for denoising each sub-band. The network structure of the first neural network can be set according to actual conditions or experience, and this application does not limit it.

[0108] Since non-stationary noise typically spans multiple frequency bands, subband networks alone cannot suppress it. Therefore, a second neural network is needed to establish the correlation between each of the M subbands and the other M-1 subbands to suppress non-stationary noise in the noise-added frequency frames.

[0109] In one example, the second neural network could be a self-attention module. Assume the dimension of the third feature data input to the self-attention network is [16, 64], meaning 16 sub-bands, each with a feature dimension of 64. For example... Figure 6 As shown, in the self-attention network, the third feature data can be processed through three fully-connected layers (FC) to obtain three parts: query matrix (Q), key matrix (K), and value matrix (V). The second feature data output by the self-attention network satisfies formula (3).

[0110]

[0111] In the formula, Attention(Q,K,V) represents the second feature data output by the self-attention network, and d k This represents the feature dimension of each sub-band. The softmax function is used for normalization.

[0112] By performing matrix multiplication using the transposes of Q and K, the similarity calculation of each feature vector in Q with the remaining feature vectors yields an att_a matrix with a feature dimension of [16, 16]. Multiplying the att_a matrix by V ensures that each of the 16 feature vectors in the second feature data output by the self-attention network considers the weighted similarity of other feature vectors. In other words, the feature data of each sub-band in the second feature data considers the important feature information of the other M-1 sub-bands for that sub-band.

[0113] In another possible implementation, see Figure 7A This is one of the schematic diagrams of a speech enhancement network structure provided in an embodiment of this application. For example... Figure 7A As shown, the speech enhancement network can include a first neural network, a second neural network, and a third neural network. The third neural network can be used to perform full-band non-stationary noise removal processing on the feature data obtained from feature extraction of the i-th noisy frequency frame.

[0114] In some embodiments, the server can use a third neural network to perform non-stationary noise removal based on the first feature data of the i-th noisy frequency frame to obtain the fifth feature data of the i-th noisy frequency frame. Alternatively, the server can use the third neural network to perform non-stationary noise removal based on the fourth feature data of the i-th noisy frequency frame to obtain the fifth feature data of the i-th noisy frequency frame. Then, the server can use a first neural network to perform stationary noise removal based on the first feature data of the i-th noisy frequency frame, the second feature data of the (i-1)-th noisy frequency frame, and the fifth feature data of the i-th noisy frequency frame to output the third feature data of the M sub-bands in the i-th noisy frequency frame.

[0115] In this embodiment, the fourth feature data can be obtained by feature extraction from the i-th noisy frequency frame. The fourth feature data can also be used to describe the spectral characteristics of the i-th noisy frequency frame, and includes one of LPS, phase difference, Fbank, and MFCC that differs from the first feature data. It should be understood that the feature extraction method can be selected based on actual conditions and experience, and this application does not limit it.

[0116] See Figure 7B This is one of the flowcharts illustrating noise reduction operations using a speech enhancement network, as provided in an embodiment of this application. Figure 7BAs shown, when performing denoising on the i-th noisy frequency frame, the server can input the first feature data of the i-th noisy frequency frame into the third neural network. After performing non-stationary noise removal, the fifth feature data of the i-th noisy frequency frame can be obtained. Then, the first feature data of the i-th noisy frequency frame, the second feature data of the (i-1)-th noisy frequency frame, and the fifth feature data of the i-th noisy frequency frame are concatenated as the input feature data of the first neural network. Then, the input feature data for each sub-band is determined and input into the first neural network respectively. The output of the first neural network, the input of the second neural network, and the output of the second neural network can be found in [reference needed]. Figures 4A-4D The relevant descriptions in the illustrated embodiments will not be repeated here.

[0117] See Figure 7C This is one of the flowcharts illustrating noise reduction operations using a speech enhancement network, as provided in an embodiment of this application. Figure 7C As shown, when performing denoising on the i-th noisy frequency frame, the server can input the fourth feature data of the i-th noisy frequency frame into the third neural network. After performing non-stationary noise removal, the fifth feature data of the i-th noisy frequency frame can be obtained. Then, the first feature data of the i-th noisy frequency frame, the second feature data of the (i-1)-th noisy frequency frame, and the fifth feature data of the i-th noisy frequency frame are concatenated as the input feature data of the first neural network. Then, the input feature data for each sub-band is determined and input into the first neural network respectively. The output of the first neural network, the input of the second neural network, and the output of the second neural network can be found in [reference needed]. Figures 4A-4D The relevant descriptions in the illustrated embodiments will not be repeated here.

[0118] It should be noted that the third neural network can be a full-band network used for non-stationary noise removal across the entire frequency band of noisy frames. The network structure of the third neural network includes, but is not limited to, Temporal Convolutional Network (TCN), Full-Functional (FC), Gate Recurrent Unit (GRU), etc., and this application does not limit it.

[0119] In another possible implementation, see Figure 7D This is one of the schematic diagrams of a speech enhancement network structure provided in an embodiment of this application. For example... Figure 7D As shown, the speech enhancement network may also include a first neural network, L second neural networks, and a third neural network. The process of denoising using the L second neural networks can be found in [reference needed]. Figure 5A and Figure 5B The relevant descriptions in the illustrated embodiments will not be repeated here.

[0120] Based on such Figure 7AThe speech enhancement network structure shown is as follows: Figure 7E As shown, when performing denoising on the i-th noisy frequency frame, the third neural network can also convert the fifth feature data of the i-th noisy frequency frame into the i-th third denoised frequency frame. The third loss value is determined based on the noise-free frequency frame corresponding to the i-th noisy frequency frame and the i-th third denoised frequency frame. The server can adjust the network parameters of the first, second, and third neural networks based on the weighted result of the first, second, and third loss values, that is, adjust the network parameters of the speech enhancement network based on the weighted result of the first, second, and third loss values ​​to obtain a trained speech enhancement network. The weighted result of the first, second, and third loss values ​​satisfies formula (4). The methods for determining the first and second loss values ​​can be found in the relevant descriptions in the above method embodiments, and will not be repeated here.

[0121] Loss=αLoss1+βLoss2+γLoss3 Formula (4)

[0122] In the formula, Loss represents the weighted sum of the first, second, and third loss values; Loss1 represents the first loss value; α represents the weight of the first loss value; Loss2 represents the second loss value; β represents the weight of the second loss value; and Loss3 represents the third loss value; γ represents the weight of the third loss value.

[0123] It should be noted that the calculation method for the third loss value can be found in the calculation method for the first loss value, and will not be repeated here.

[0124] Based on the above scheme, the second feature data output by the second neural network is fed back into the input of the first neural network in the next frame. This ensures that the feature data input to the first neural network for each sub-frequency band takes into account the important global information of other sub-frequency bands for that sub-frequency band, enabling the first neural network to better suppress non-stationary noise. Furthermore, since the full-band network has a strong ability to eliminate non-stationary noise, a third neural network can further suppress non-stationary noise. Figure 8 As shown, the solution implemented in this application can more effectively suppress non-stationary noise such as impact noise.

[0125] In one possible implementation, the fifth feature data input to the first neural network can be replaced with enhanced features of the fifth feature data. These enhanced features can be obtained by processing the i-th denoised frequency frame obtained from the third neural network. For example, the enhanced features can be obtained by extracting the LPS after performing a Fast Fourier Transform on the i-th denoised frequency frame.

[0126] In another possible implementation, the fifth feature data input to the first neural network can be replaced with the audio masking result (Mask) corresponding to the i-th third denoised frequency frame obtained by the third neural network.

[0127] In one possible implementation, the third feature data input to the second neural network can be replaced with enhanced features of the third feature data. The enhanced features of the fifth feature data can be obtained by processing the i-th denoised frequency frame obtained from the first neural network. For example, the enhanced features of the third feature data can be obtained by extracting the LPS after performing a Fast Fourier Transform on the i-th denoised frequency frame.

[0128] In another possible implementation, the third feature data input to the second neural network can be replaced with the audio masking result (Mask) corresponding to the i-th second denoised frequency frame obtained by the first neural network.

[0129] In one possible implementation, see [link to relevant documentation]. Figure 9A This is one of the flowcharts illustrating noise reduction operations using a speech enhancement network, as provided in an embodiment of this application. Figure 9A As shown, the feature data input to the second neural network can be the input feature data of the second neural network, which includes the third feature data of M sub-frequency bands in the i-th noisy frequency frame and the first feature data of the i-th noisy frequency frame. That is, through the second neural network, non-stationary noise removal is performed based on the third feature data of the M sub-frequency bands in the i-th noisy frequency frame and the first feature data of the i-th noisy frequency frame to obtain the second feature data of the i-th noisy frequency frame. It should be noted that... Figure 9A The input feature data of the first neural network is only an example. The input feature data of the first neural network can be any of the above method embodiments, and will not be described again here.

[0130] In another possible implementation, see Figure 9B This is one of the flowcharts illustrating noise reduction operations using a speech enhancement network, as provided in an embodiment of this application. Figure 9B As shown, the feature data input to the second neural network can include the third feature data of the M sub-bands in the i-th noisy frequency frame and the fourth feature data of the i-th noisy frequency frame. That is, through the second neural network, non-stationary noise removal is performed based on the third feature data of the M sub-bands in the i-th noisy frequency frame and the fourth feature data of the i-th noisy frequency frame to obtain the second feature data of the i-th noisy frequency frame. It should be noted that... Figure 9B The input feature data of the first neural network is only an example. The input feature data of the first neural network can be any of the above method embodiments, and will not be described again here.

[0131] In one possible implementation, the audio data can also be processed using the speech enhancement model trained using the methods described above. See [link to relevant documentation]. Figure 10 An exemplary flowchart of an audio processing method based on a speech enhancement network provided in this application embodiment.

[0132] S1001, acquire the noise frequency data to be reduced.

[0133] The noise reduction frequency data includes P noise reduction frequency frames. P is a positive integer.

[0134] S1002, extract the first feature data of the j-th noise reduction frequency frame in the noise reduction frequency data.

[0135] The first feature data describes the spectral characteristics of the j-th noise-to-reduce frequency frame. j is a positive integer less than or equal to P. The extraction method can be found in [link to relevant documentation]. Figure 2 The relevant descriptions in the method embodiments shown will not be repeated here.

[0136] S1003, through the speech enhancement network, based on the first feature data of the j-th noise reduction frequency frame and the second feature data of the (j-1)-th noise reduction frequency frame, a denoising operation is performed on the j-th noise reduction frequency frame to obtain the audio denoising result of the j-th noise reduction frequency frame.

[0137] The second feature data is obtained during the denoising operation of the speech enhancement network on the (j-1)th noise-reducing frequency frame. The second feature data describes the feature correlation between each of the M sub-frequency bands in the (j-1)th noise-reducing frequency frame and the other M-1 sub-frequency bands, where M is a positive integer. The speech enhancement network can be trained using the aforementioned speech enhancement network training method.

[0138] S1004: Based on the audio noise reduction results of each of the P noise reduction frequency frames, obtain the noise-reduced audio data.

[0139] In one example, the audio denoising result of each noise-reducing frequency frame can be multiplied by the spectrum of the noise-reducing frequency data, and then subjected to inverse Fourier transform to obtain the denoised audio data. It should be understood that the audio denoising result can be IAM or PSM, and this application does not limit it. Furthermore, the method described above for obtaining the denoised audio data is merely exemplary; other methods in the prior art can also be referred to, and this application does not limit them.

[0140] Based on the same concept of the training method for the aforementioned speech enhancement network, see [link to relevant documentation]. Figure 11 This application provides a training apparatus 1100 for a speech enhancement network, which includes an acquisition unit 1101 and a processing unit 1102. In one scenario:

[0141] The acquisition unit 1101 is used to acquire a training sample set, which includes noise-added frequency data and noise-free frequency data. The noise-added frequency data is obtained by adding noise to the noise-free frequency data. The noise-free frequency data includes N noise-free frequency frames, and the noise-added frequency data includes N noise-added frequency frames that correspond one-to-one with the N noise-free frequency frames. N is a positive integer.

[0142] Processing unit 1102 is configured to perform the following processing: extracting first feature data of the i-th noisy frequency frame in the training sample set, wherein the first feature data describes the spectral characteristics of the i-th noisy frequency frame; i is a positive integer less than or equal to N; performing denoising operations on the M sub-bands included in the i-th noisy frequency frame according to the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame to obtain the i-th first denoised frequency frame; the second feature data is obtained by the speech enhancement network during the denoising operation on the (i-1)-th noisy frequency frame; the second feature data describes the feature correlation between each sub-band of the M sub-bands of the (i-1)-th noisy frequency frame and the other M-1 sub-bands, wherein M is a positive integer; and adjusting the network parameters of the speech enhancement network according to the first loss value of the i-th first denoised frequency frame and the noise-free frequency frame corresponding to the i-th noisy frequency frame.

[0143] In one possible implementation, the speech enhancement network includes a first neural network and a second neural network; the processing unit 1102 performs denoising operations on the M sub-bands included in the i-th noisy frequency frame according to the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame to obtain an i-th first denoised frequency frame, and is used to: perform stationary noise removal through the first neural network according to the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame to obtain the third feature data of the M sub-bands in the i-th noisy frequency frame; perform non-stationary noise removal through the second neural network according to the third feature data of the M sub-bands in the i-th noisy frequency frame to obtain the second feature data of the i-th noisy frequency frame; and convert the second feature data of the i-th noisy frequency frame into the i-th first denoised frequency frame.

[0144] In one possible implementation, the first feature data includes one of logarithmic power spectrum, phase difference, filter bank-based feature Fbank, and Mel-frequency cepstral coefficients (MFCC).

[0145] In one possible implementation, the processing unit 1102 is further configured to: extract fourth feature data of the i-th noisy frequency frame; the fourth feature data is used to describe the spectral characteristics of the i-th noisy frequency frame, and the fourth feature data includes one of LPS, phase difference, Fbank, and MFCC that is different from the first feature data.

[0146] In one possible implementation, the speech enhancement network further includes a third neural network, and the processing unit 1102 is further configured to: perform non-stationary noise removal based on the first feature data of the i-th noisy frequency frame using the third neural network to obtain the fifth feature data of the i-th noisy frequency frame; or, perform non-stationary noise removal based on the fourth feature data of the i-th noisy frequency frame using the third neural network to obtain the fifth feature data of the i-th noisy frequency frame; the processing unit 1102 performs stationary noise removal based on the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame using the first neural network to obtain the third feature data of M sub-frequency bands in the i-th noisy frequency frame, and is configured to: perform stationary noise removal based on the first feature data of the i-th noisy frequency frame, the second feature data of the (i-1)-th noisy frequency frame and the fifth feature data of the i-th noisy frequency frame to obtain the third feature data of M sub-frequency bands in the i-th noisy frequency frame using the first neural network.

[0147] In one possible implementation, the processing unit 1102 obtains the second feature data of the i-th noisy frequency frame by performing non-stationary noise removal based on the third feature data of the M sub-frequency bands in the i-th noisy frequency frame using the second neural network. This is used to: obtain the second feature data of the i-th noisy frequency frame by performing non-stationary noise removal based on the third feature data of the M sub-frequency bands in the i-th noisy frequency frame and the first feature data of the i-th noisy frequency frame using the second neural network; or, obtain the second feature data of the i-th noisy frequency frame by performing non-stationary noise removal based on the third feature data of the M sub-frequency bands in the i-th noisy frequency frame and the fourth feature data of the i-th noisy frequency frame using the second neural network.

[0148] In one possible implementation, the processing unit 1102 is further configured to: convert the third feature data of the i-th noisy frequency frame into the i-th second denoised frequency frame; determine the i-th third denoised frequency frame from the fifth feature data of the i-th noisy frequency frame; the processing unit 1102 adjusts the network parameters of the speech enhancement network according to the first loss value of the i-th first denoised frequency frame and the noise-free frequency frame corresponding to the i-th noisy frequency frame, configured to: adjust the network parameters of the speech enhancement network according to the weighted result of the first loss value, the second loss value and the third loss value; the first loss value is determined based on the i-th first denoised frequency frame and the noise-free frequency frame corresponding to the i-th noisy frequency frame, the second loss value is determined based on the noise-free frequency frame corresponding to the i-th noisy frequency frame and the i-th second denoised frequency frame, and the third loss value is determined based on the noise-free frequency frame corresponding to the i-th noisy frequency frame and the i-th third denoised frequency frame.

[0149] Based on the same concept of the above-mentioned audio processing method based on speech enhancement networks, see [link to relevant documentation]. Figure 12 This application provides an audio processing apparatus 1200 based on a speech enhancement network, which includes an acquisition unit 1201 and a processing unit 1202. In one scenario:

[0150] Acquisition unit 1201 is used to acquire noise frequency data to be reduced; the noise frequency data to be reduced includes P noise frequency frames to be reduced; P is a positive integer;

[0151] Processing unit 1202 is used to extract the first feature data of the j-th noise-to-be-reduced frequency frame from the noise-to-be-reduced frequency data. The first feature data is used to describe the spectral characteristics of the j-th noise-to-be-reduced frequency frame; j is a positive integer less than or equal to P. Through a speech enhancement network, based on the first feature data of the j-th noise-to-be-reduced frequency frame and the second feature data of the (j-1)-th noise-to-be-reduced frequency frame, a denoising operation is performed on the j-th noise-to-be-reduced frequency frame to obtain the audio denoising result of the j-th noise-to-be-reduced frequency frame. The second feature data is obtained during the denoising operation of the speech enhancement network on the (j-1)-th noise-to-be-reduced frequency frame. The second feature data is used to describe the feature correlation between each of the M sub-frequency bands of the (j-1)-th noise-to-be-reduced frequency frame and the other M-1 sub-frequency bands, where M is a positive integer. The speech enhancement network is trained using the training method described above. Based on the audio denoising results of each of the P noise-to-be-reduced frequency frames, denoised audio data is obtained.

[0152] Based on the same concept as the above method, see [link to relevant documentation]. Figure 13This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device includes at least one processor 1302 and a memory 1301 connected or coupled to the at least one processor 1302. In addition, the electronic device may also include a communication interface 1303. The electronic device can interact with other devices through the communication interface 1303.

[0153] For example, the communication interface 1303 can be a transceiver, circuit, bus, module, pin, or other type of communication interface. When the electronic device is a chip-based device or circuit, the communication interface 1303 in the electronic device can also be an input / output circuit, capable of inputting information (or receiving information) and outputting information (or sending information). The processor can be an integrated processor, a microprocessor, an integrated circuit, or a logic circuit, and the processor can determine the output information based on the input information.

[0154] The coupling in this application embodiment is an indirect coupling or communication connection between devices, units, or modules, which can be electrical, mechanical, or other forms, used for information interaction between devices, units, or modules. The processor 1302 may operate in conjunction with the memory 1301 and the communication interface 1303. This application does not limit the specific connection medium between the processor 1302, the memory 1301, and the communication interface 1303.

[0155] Optional, see Figure 13 The processor 1302, the memory 1301, and the communication interface 1303 are interconnected via a bus. This bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 13 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0156] In this embodiment, memory 1301, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 1301 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory 1301 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. Memory 1301 in this embodiment may also be a circuit or any other device capable of implementing storage functions for storing instructions, computer programs, and / or data.

[0157] In this embodiment, the processor 1302 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in this embodiment. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the speech enhancement network training method disclosed in this embodiment can be directly implemented by a hardware processor, or implemented using a combination of hardware and software modules within the processor.

[0158] By designing and programming the processor 1302, the code corresponding to the training method of the speech enhancement network described in the foregoing embodiments can be embedded into the chip, so that the chip can execute the steps of the aforementioned training method of the speech enhancement network when it runs. How to design and program the processor 1302 is a well-known technique to those skilled in the art, and will not be described in detail here.

[0159] In one or more embodiments, the memory 1301 stores instructions that can be executed by at least one processor 1302. The at least one processor 1302 can implement various steps of the training method for the speech enhancement network described above, or can also implement various steps of the audio processing method based on the speech enhancement network, by calling the instructions or computer program stored in the memory 1301.

[0160] This application also provides a computer-readable storage medium storing computer instructions that, when executed on a computer, enable the computer to implement the various steps of the training method for the speech enhancement network described above, or to implement the various steps of the audio processing method based on the speech enhancement network described above.

[0161] In some embodiments, various aspects of the training method for the speech enhancement network and various aspects of the audio processing method based on the speech enhancement network provided in this application can also be implemented in the form of a computer program product, which includes program code. When the computer program product is run on an electronic device, the program code is used to cause the electronic device to perform the various steps of the training method for the speech enhancement network described above in this specification, or it can also be used to cause the electronic device to perform the various steps of the audio processing method based on the speech enhancement network described above in this specification.

[0162] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0163] While specific embodiments of this application have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this application is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this application, and all such changes and modifications fall within the scope of protection of this application. Although preferred embodiments of this application have been described, those skilled in the art, once they understand the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0164] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A training method for a speech enhancement network, characterized in that, The method includes: Obtain a training sample set, which includes noise-added frequency data and noise-free frequency data. The noise-added frequency data is obtained by adding noise to the noise-free frequency data. The noise-free frequency data includes N noise-free frequency frames, and the noise-added frequency data includes N noise-added frequency frames that correspond one-to-one with the N noise-free frequency frames. N is a positive integer. Extract the first feature data of the i-th noisy frequency frame in the training sample set. The first feature data is used to describe the spectral characteristics of the i-th noisy frequency frame; i is a positive integer less than or equal to N. Based on the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame, denoising operations are performed on the M sub-frequency bands included in the i-th noisy frequency frame to obtain the i-th first denoised frequency frame; the second feature data is obtained by the speech enhancement network during the denoising operation on the (i-1)-th noisy frequency frame; the second feature data is used to describe the feature correlation between each sub-frequency band in the M sub-frequency bands of the (i-1)-th noisy frequency frame and the other M-1 sub-frequency bands, where M is a positive integer; The network parameters of the speech enhancement network are adjusted according to the first loss value of the noise-free frame corresponding to the i-th first denoised frequency frame and the i-th noise-added frequency frame; The speech enhancement network includes a first neural network and a second neural network; The step of performing denoising operations on the M sub-bands included in the i-th noisy frequency frame to obtain the i-th first denoised frequency frame based on the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame includes: The second neural network is used to remove non-stationary noise from the M sub-bands in the i-th noisy frequency frame to obtain the second feature data of the i-th noisy frequency frame; the third feature data of the M sub-bands in the i-th noisy frequency frame is obtained through the first neural network. Convert the second feature data of the i-th noisy frequency frame into the i-th denoised frequency frame; The step of obtaining the second feature data of the i-th noisy frequency frame by performing non-stationary noise removal based on the third feature data of the M sub-frequency bands in the i-th noisy frequency frame through the second neural network includes: The second neural network is used to perform non-stationary noise removal based on the third feature data of the M sub-bands in the i-th noisy frequency frame and the first feature data of the i-th noisy frequency frame to obtain the second feature data of the i-th noisy frequency frame; Alternatively, through the second neural network, non-stationary noise removal is performed based on the third feature data of the M sub-bands in the i-th noisy frequency frame and the fourth feature data of the i-th noisy frequency frame to obtain the second feature data of the i-th noisy frequency frame; the fourth feature data of the i-th noisy frequency frame is pre-extracted.

2. The method according to claim 1, characterized in that, The third feature data of the M sub-bands in the i-th noisy frequency frame are obtained in the following way: Using the first neural network, based on the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame, smooth noise removal is performed to obtain the third feature data of M sub-bands in the i-th noisy frequency frame.

3. The method according to claim 2, characterized in that, The first feature data includes one of the following: log power spectrum (LPS), phase difference, filter bank-based feature Fbank, and Mel-frequency cepstral coefficients (MFCC).

4. The method according to claim 3, characterized in that, The method further includes: Extract the fourth feature data of the i-th noisy frequency frame; the fourth feature data is used to describe the spectral characteristics of the i-th noisy frequency frame, and the fourth feature data includes one of LPS, phase difference, Fbank, and MFCC that is different from the first feature data.

5. The method according to claim 4, characterized in that, The speech enhancement network further includes a third neural network, and the method further includes: The third neural network is used to perform non-stationary noise removal based on the first feature data of the i-th noisy frequency frame to obtain the fifth feature data of the i-th noisy frequency frame; Alternatively, the third neural network can be used to perform non-stationary noise removal based on the fourth feature data of the i-th noisy frequency frame to obtain the fifth feature data of the i-th noisy frequency frame; The step of obtaining third feature data of M sub-bands in the i-th noisy frequency frame by performing smooth noise removal through the first neural network based on the first feature data of the i-th noisy frequency frame and the second feature data of the (i-1)-th noisy frequency frame includes: Using the first neural network, based on the first feature data of the i-th noisy frequency frame, the second feature data of the (i-1)-th noisy frequency frame, and the fifth feature data of the i-th noisy frequency frame, smooth noise removal is performed to obtain the third feature data of M sub-bands in the i-th noisy frequency frame.

6. The method according to claim 5, characterized in that, The method further includes: The third feature data of the i-th noisy frequency frame is converted into the i-th denoised frequency frame; The fifth feature data of the i-th noisy frequency frame is used to determine the i-th denoised frequency frame; The step of adjusting the network parameters of the speech enhancement network based on the first loss value of the noise-free frequency frame corresponding to the i-th first denoised frequency frame and the i-th added noise frequency frame includes: The network parameters of the speech enhancement network are adjusted based on the weighted result of the first loss value, the second loss value, and the third loss value. The first loss value is determined based on the noise-free frequency frame corresponding to the i-th first denoised frequency frame and the i-th noise-added frequency frame. The second loss value is determined based on the noise-free frequency frame corresponding to the i-th noise-added frequency frame and the i-th second denoised frequency frame. The third loss value is determined based on the noise-free frequency frame corresponding to the i-th noise-added frequency frame and the i-th third denoised frequency frame.

7. An audio processing method based on a speech enhancement network, characterized in that, include: Acquire the noise frequency data to be reduced; The noise frequency data to be reduced includes P noise frequency frames to be reduced; P is a positive integer; Extract the first feature data of the j-th noise reduction frequency frame from the noise reduction frequency data. The first feature data is used to describe the spectral characteristics of the j-th noise reduction frequency frame. j is a positive integer less than or equal to P; The speech enhancement network performs a denoising operation on the j-th noise-to-be-reduced frequency frame based on the first feature data of the j-th noise-to-be-reduced frequency frame and the second feature data of the (j-1)-th noise-to-be-reduced frequency frame, thereby obtaining the audio denoising result of the j-th noise-to-be-reduced frequency frame. The second feature data is obtained during the denoising operation performed by the speech enhancement network on the (j-1)-th noise-to-be-reduced frequency frame. The second feature data is used to describe the feature correlation between each of the M sub-frequency bands of the (j-1)-th noise-to-be-reduced frequency frame and the other M-1 sub-frequency bands, where M is a positive integer. The speech enhancement network is trained using the training method described in any one of claims 1-6. Based on the audio denoising results of each of the P noise reduction frequency frames, the denoised audio data is obtained.

8. An electronic device, characterized in that, include: Memory, used to store computer instructions; A processor, connected to the memory, is configured to execute computer instructions in the memory, and, in executing the computer instructions, implement the method as claimed in any one of claims 1 to 6 or the method as claimed in claim 7.

9. A computer-readable storage medium, characterized in that, include: The computer-readable storage medium stores computer instructions that, when executed on a computer, cause the computer to perform the method as claimed in any one of claims 1 to 6 or the method as claimed in claim 7.

Images