Audio signal processing method and device

Abstract

The present disclosure relates to an audio signal processing method and device, which comprises: acquiring a near-end collected audio signal, a far-end reference audio signal, and a first near-end audio signal obtained by performing linear echo elimination on the near-end collected audio signal; performing time-frequency transformation on the near-end collected audio signal, the far-end reference audio signal, and the first near-end audio signal, respectively, to obtain a frequency-domain near-end collected audio signal, a frequency-domain far-end reference audio signal, and a first frequency-domain near-end audio signal; performing deep learning noise reduction on the amplitude of the first frequency-domain near-end audio signal, to obtain a second frequency-domain near-end audio signal; performing nonlinear echo elimination on the second frequency-domain near-end audio signal based on the frequency-domain far-end reference audio signal, the frequency-domain near-end collected audio signal, the first frequency-domain near-end audio signal, and the second frequency-domain near-end audio signal, to obtain a frequency-domain near-end audio enhancement signal; and performing time-frequency inverse transformation on the frequency-domain near-end audio enhancement signal, to obtain a near-end audio enhancement signal.

Description

Technical Field

[0001] The present disclosure relates to the technical field of signal processing, and more specifically, to an audio signal processing method and device. Background Art

[0002] Acoustic Echo Cancellation (AEC) is one of the important technologies in real-time communication and is the key to ensuring the audio and video experience. Echo cancellation technology refers to eliminating the far-end signal from the audio signal collected by the near-end microphone and retaining only the near-end signal. The audio signal collected by the near-end microphone includes the near-end signal and the far-end signal played through the near-end speaker. Echo cancellation technology generally includes linear echo cancellation and nonlinear echo cancellation.

[0003] In the related art, echo cancellation is generally combined with traditional noise reduction such as filtering to achieve the effects of noise reduction and echo cancellation. However, the audio signal processed by this method has poor echo cancellation and noise reduction effects and poor sound quality. Summary of the invention

[0004] The present disclosure provides an audio signal processing method and device to at least solve the above-mentioned problems in the related art, and may not solve any of the above-mentioned problems.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for processing an audio signal is provided, comprising: acquiring a near-end collected audio signal, a far-end reference audio signal, and a first near-end audio signal obtained by performing linear echo elimination on the near-end collected audio signal; performing time-frequency transformation on the near-end collected audio signal, the far-end reference audio signal, and the first near-end audio signal, respectively, to obtain a frequency-domain near-end collected audio signal, a frequency-domain far-end reference audio signal, and a first frequency-domain near-end audio signal; performing deep learning noise reduction on the amplitude of the first frequency-domain near-end audio signal to obtain a second frequency-domain near-end audio signal; performing nonlinear echo elimination on the second frequency-domain near-end audio signal based on the frequency-domain far-end reference audio signal, the frequency-domain near-end collected audio signal, the first frequency-domain near-end audio signal, and the second frequency-domain near-end audio signal to obtain a frequency-domain near-end audio enhancement signal; and performing time-frequency inverse transformation on the frequency-domain near-end audio enhancement signal to obtain a near-end audio enhancement signal.

[0006] Optionally, the deep learning denoising is performed on the amplitude of the first frequency-domain near-end audio signal to obtain the second frequency-domain near-end audio signal, including: performing deep learning denoising on the amplitude of the first frequency-domain near-end audio signal through a trained denoising neural network model to obtain the amplitude of the second frequency-domain near-end audio signal; and obtaining the second frequency-domain near-end audio signal based on the amplitude of the second frequency-domain near-end audio signal and the phase of the first frequency-domain near-end audio signal.

[0007] Optionally, the amplitude of the first frequency-domain near-end audio signal is subjected to deep learning noise reduction by means of a trained noise reduction neural network model to obtain the amplitude of the second frequency-domain near-end audio signal, including: inputting the amplitude of the first frequency-domain near-end audio signal into the trained noise reduction neural network model to obtain a first signal amplitude ratio, wherein the first signal amplitude ratio is a predicted value of the ratio of the amplitude of the second frequency-domain near-end audio signal to the amplitude of the first frequency-domain near-end audio signal; and obtaining the amplitude of the second frequency-domain near-end audio signal according to the first signal amplitude ratio and the amplitude of the first frequency-domain near-end audio signal, wherein the amplitude of the second frequency-domain near-end audio signal is the product of the first signal amplitude ratio and the amplitude of the first frequency-domain near-end audio signal.

[0008] Optionally, based on the frequency domain far-end reference audio signal, the frequency domain near-end collected audio signal, the first frequency domain near-end audio signal and the second frequency domain near-end audio signal, nonlinear echo elimination is performed on the second frequency domain near-end audio signal to obtain a frequency domain near-end audio enhancement signal, including: correlating the frequency domain far-end reference audio signal with the frequency domain near-end collected audio signal and the second frequency domain near-end audio signal in each frequency band to obtain a second signal amplitude ratio of each frequency band; and performing nonlinear echo elimination on the second frequency domain near-end audio signal according to the second signal amplitude ratio, the first frequency domain near-end audio signal and the second frequency domain near-end audio signal to obtain a frequency domain near-end audio enhancement signal.

[0009] Optionally, the second frequency-domain near-end audio signal is subjected to nonlinear echo elimination based on the second signal amplitude ratio, the first frequency-domain near-end audio signal and the second frequency-domain near-end audio signal to obtain a frequency-domain near-end audio enhancement signal, including: obtaining the product of the second signal amplitude ratio and the amplitude of the first frequency-domain near-end audio signal as a reference amplitude; obtaining the minimum value between the reference amplitude and the amplitude of the second frequency-domain near-end audio signal as the amplitude of the frequency-domain near-end audio enhancement signal; and obtaining the frequency-domain near-end audio enhancement signal based on the amplitude of the frequency-domain near-end audio enhancement signal and the phase of the first frequency-domain near-end audio signal.

[0010] Optionally, the second signal amplitude ratio is obtained by the following formula:

[0011] Mask(n,k)=min{1-RCr(n,k),1-RY p r(n,k)};

[0012] Wherein, Mask(n,k) is the amplitude ratio of the second signal, RCr(n,k) is the mutual correlation coefficient between the frequency domain far-end reference audio signal and the frequency domain near-end collected audio signal, RY p r(n,k) is the mutual correlation coefficient between the frequency domain far-end reference audio signal and the second frequency domain near-end audio signal, n is the frame sequence number, k is the center frequency sequence number, 0<n≤N, 0<k≤K, N is the total number of frames, and K is the total number of frequency bands.

[0013] According to a second aspect of an embodiment of the present disclosure, there is provided an audio signal processing device, comprising: a signal acquisition unit, configured to: acquire a near-end collected audio signal, a far-end reference audio signal, and a first near-end audio signal obtained after performing linear echo elimination on the near-end collected audio signal; a frequency domain transformation unit, configured to: perform time-frequency transformation on the near-end collected audio signal, the far-end reference audio signal and the first near-end audio signal, respectively, to obtain a frequency domain near-end collected audio signal, a frequency domain far-end reference audio signal and a first frequency domain near-end audio signal; and a deep noise reduction unit. , configured to: perform deep learning noise reduction on the amplitude of the first frequency domain near-end audio signal to obtain a second frequency domain near-end audio signal; a nonlinear elimination unit, configured to: perform nonlinear echo elimination on the second frequency domain near-end audio signal based on the frequency domain far-end reference audio signal, the frequency domain near-end collected audio signal, the first frequency domain near-end audio signal and the second frequency domain near-end audio signal to obtain a frequency domain near-end audio enhancement signal; a time domain transformation unit, configured to: perform time-frequency inverse transformation on the frequency domain near-end audio enhancement signal to obtain a near-end audio enhancement signal.

[0014] Optionally, the deep noise reduction unit is configured to: perform deep learning noise reduction on the amplitude of the first frequency domain near-end audio signal through a trained noise reduction neural network model to obtain the amplitude of the second frequency domain near-end audio signal; and obtain the second frequency domain near-end audio signal based on the amplitude of the second frequency domain near-end audio signal and the phase of the first frequency domain near-end audio signal.

[0015] Optionally, the deep noise reduction unit is configured to: input the amplitude of the first frequency-domain near-end audio signal into the trained noise reduction neural network model to obtain a first signal amplitude ratio, wherein the first signal amplitude ratio is a predicted value of the ratio of the amplitude of the second frequency-domain near-end audio signal to the amplitude of the first frequency-domain near-end audio signal; obtain the amplitude of the second frequency-domain near-end audio signal based on the first signal amplitude ratio and the amplitude of the first frequency-domain near-end audio signal, wherein the amplitude of the second frequency-domain near-end audio signal is the product of the first signal amplitude ratio and the amplitude of the first frequency-domain near-end audio signal.

[0016] Optionally, the nonlinear elimination unit is configured to: correlate the frequency domain far-end reference audio signal with the frequency domain near-end collected audio signal and the second frequency near-end audio signal in each frequency band to obtain a second signal amplitude ratio of each frequency band; and perform nonlinear echo elimination on the second frequency domain near-end audio signal according to the second signal amplitude ratio, the first frequency domain near-end audio signal and the second frequency domain near-end audio signal to obtain a frequency domain near-end audio enhancement signal.

[0017] Optionally, the nonlinear elimination unit is configured to: obtain the product of the second signal amplitude ratio and the amplitude of the first frequency domain near-end audio signal as a reference amplitude; obtain the minimum value between the reference amplitude and the amplitude of the second frequency domain near-end audio signal as the amplitude of the frequency domain near-end audio enhancement signal; and obtain the frequency domain near-end audio enhancement signal based on the amplitude of the frequency domain near-end audio enhancement signal and the phase of the first frequency domain near-end audio signal.

[0018] Optionally, the second signal amplitude ratio is obtained by the following formula:

[0019] Mask(n,k)=min{1-RCr(n,k),1-RY p r(n,k)};

[0020] Wherein, Mask(n,k) is the amplitude ratio of the second signal, RCr(n,j) is the mutual correlation coefficient between the frequency domain far-end reference audio signal and the frequency domain near-end collected audio signal, RY p r(n,k) is the mutual correlation coefficient between the frequency domain far-end reference audio signal and the second frequency domain near-end audio signal, n is the frame sequence number, k is the center frequency sequence number, 0<n≤N, 0<k≤K, N is the total number of frames, and K is the total number of frequency bands.

[0021] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the audio signal processing method according to the present disclosure.

[0022] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the audio signal processing method according to the present disclosure.

[0023] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which implement the audio signal processing method according to the present disclosure when executed by at least one processor.

[0024] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0025] According to the audio signal processing method and device disclosed in the present invention, linear echo cancellation is first performed on the near-end collected audio signal, then deep learning noise reduction is performed on it, and then nonlinear echo cancellation is performed. The echo signal in the far-end reference audio signal is eliminated by linear echo cancellation and nonlinear echo cancellation to obtain the final near-end audio enhancement signal. The deep learning noise reduction and echo cancellation are combined to fully utilize the good performance of deep learning noise reduction. Compared with the combination of traditional noise reduction technology and echo cancellation used in related technologies, better echo cancellation effect and noise reduction effect can be obtained, bringing about an improvement in sound quality.

[0026] In addition, according to the audio signal processing method and device disclosed in the present invention, deep learning noise reduction can be used to handle situations that are difficult to solve with traditional noise reduction, such as the elimination of quasi-steady-state or non-steady-state noise and the problem that traditional noise reduction is not effective in eliminating noise.

[0027] In addition, according to the audio signal processing method and device disclosed in the present invention, in the process of nonlinear echo elimination, by obtaining the signal amplitude ratio based on the cross-correlation coefficient, a frequency domain near-end audio enhancement signal is finally obtained, which can achieve the effect of eliminating echo and noise at the same time.

[0028] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0030] Figure 1 Detailed description is given of an overall framework of an audio signal processing method according to an exemplary embodiment of the present disclosure.

[0031] Figure 2 is a flowchart of an audio signal processing method according to an exemplary embodiment of the present disclosure.

[0032] Figure 3 is a block diagram of an audio signal processing apparatus according to an exemplary embodiment of the present disclosure.

[0033] Figure 4 is a block diagram of an electronic device 400 according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0036] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0037] In real-time communication, audio signals may contain noise and echoes. Based on this, acoustic echo cancellation (AEC) has become one of the important technologies in real-time communication and is the key to ensuring the audio and video experience. Echo cancellation technology refers to eliminating the far-end signal in the audio signal collected by the near-end microphone and retaining only the near-end signal. The audio signal collected by the near-end microphone may include the near-end signal and the audio signal played by the far-end signal through the near-end speaker. It should be noted that the near-end can be called the local end and the far-end can be called the other end. Echo cancellation technology generally includes linear echo cancellation and nonlinear echo cancellation.

[0038] Linear echo cancellation can eliminate far-end signals through adaptive filtering, but there will often be residual far-end signals. Combining linear echo cancellation with nonlinear echo cancellation will improve the elimination effect of far-end signals. However, under normal circumstances, the signal-to-noise ratio of the signal collected at the near end will affect the effect of nonlinear echo cancellation. In related technologies, the audio signal that has undergone linear echo cancellation is denoised using traditional methods such as Wiener filtering to obtain a relatively clean audio signal, and then nonlinear echo cancellation is performed for correlation processing. However, the audio signal processed by this method is limited by the effect of traditional noise reduction technology, and the noise reduction effect is not good, especially when the noise is non-steady, which affects the echo cancellation effect and causes poor sound quality.

[0039] In order to solve the problems existing in the above-mentioned related technologies, the present disclosure proposes an audio signal processing method and device, which first performs linear echo elimination on the near-end collected audio signal, then performs deep learning noise reduction on it, and then performs nonlinear echo elimination to obtain the final near-end audio enhancement signal. The deep learning noise reduction and echo cancellation are combined. Compared with the combination of traditional noise reduction technology and echo cancellation used in the related technology, better echo cancellation effect and noise reduction effect can be obtained, bringing about an improvement in sound quality.

[0040] Next, we will refer to Figures 1 to 4 The audio signal processing method and device according to the present disclosure are described in detail.

[0041] Figure 1 is an overall framework diagram of an audio signal processing method according to an exemplary embodiment of the present disclosure. Figure 1 , the near-end collected audio signal can be linearly echo eliminated to obtain a first near-end audio signal, the first frequency domain near-end audio signal can be denoised by deep learning to obtain a second frequency domain near-end audio signal, and the second frequency domain near-end audio signal can be nonlinearly echo eliminated to obtain a frequency domain near-end audio enhancement signal. It should be noted that in this process, the echo signal in the far-end reference audio signal can be eliminated by linear echo elimination and nonlinear echo elimination.

[0042] After knowing the overall framework of the exemplary embodiment of the present disclosure, the audio signal processing method of the exemplary embodiment of the present disclosure is described below through specific method steps.

[0043] Figure 2 is a flowchart of an audio signal processing method according to an exemplary embodiment of the present disclosure.

[0044] Reference Figure 2 In step 201, a near-end collected audio signal, a far-end reference audio signal, and a first near-end audio signal obtained by performing linear echo cancellation on the near-end collected audio signal may be obtained.

[0045] According to an exemplary embodiment of the present disclosure, the near-end collected audio signal is an audio signal collected by a near-end microphone, and the far-end reference audio signal is an audio signal transmitted from the far-end without being played by a near-end speaker.

[0046] The audio signal collected by the near-end microphone may include, but is not limited to, a near-end user voice signal collected by the near-end microphone and an audio signal collected by the near-end microphone and transmitted from the far end via a network link and played by a near-end speaker.

[0047] In step 202, time-frequency transformation may be performed on the near-end collected audio signal, the far-end reference audio signal and the first near-end audio signal to obtain a frequency-domain near-end collected audio signal, a frequency-domain far-end reference audio signal and a first frequency-domain near-end audio signal.

[0048] According to an exemplary embodiment of the present disclosure, the time-frequency transform may be, but is not limited to, a short-time Fourier transform (STFT). The following description takes STFT as an example.

[0049] According to an exemplary embodiment of the present disclosure, the near-end collected audio signal, the far-end reference audio signal and the first near-end audio signal with a time length of T can be respectively expressed as c(t), rt) and y(t) in the time domain, where t is time and 0t≤T. After short-time Fourier transform, c(t), rt) and y(t) can be expressed in the frequency domain by the following equations (1)-(3):

[0050] C(n,k)=STFT(c(t)); (1)

[0051] R(n,k)=STFT(r(t)); (2)

[0052] Y(n,k)=STFT(y(t)); (3)

[0053] Wherein, c(t) is the near-end collected audio signal, rt) is the far-end reference audio signal, y(t) is the first near-end audio signal, STFT(x) represents short-time Fourier transform of x, C(n,k) is the frequency domain near-end collected audio signal, R(n,k) is the frequency domain far-end reference audio signal, Y(n,k) is the first frequency domain near-end audio signal, n is the frame sequence number, k is the center frequency sequence number, 0<n≤N, 0<k≤K, N is the total number of frames, and K is the total number of frequency bands.

[0054] In step 203, deep learning noise reduction may be performed on the amplitude of the first frequency domain near-end audio signal to obtain a second frequency domain near-end audio signal.

[0055] According to an exemplary embodiment of the present disclosure, deep learning noise reduction can be implemented by performing noise reduction through a neural network model. In this case, first, the amplitude of the near-end audio signal in the first frequency domain can be subjected to deep learning noise reduction through a trained noise reduction neural network model to obtain the amplitude of the near-end audio signal in the second frequency domain. Then, the near-end audio signal in the second frequency domain can be obtained according to the amplitude of the near-end audio signal in the second frequency domain and the phase of the near-end audio signal in the first frequency domain.

[0056] According to an exemplary embodiment of the present disclosure, the amplitude of the first frequency domain near-end audio signal can first be input into a trained denoising neural network model to obtain a first signal amplitude ratio, wherein the first signal amplitude ratio is a predicted value of the ratio of the amplitude of the second frequency domain near-end audio signal to the amplitude of the first frequency domain near-end audio signal. Then, the amplitude of the second frequency domain near-end audio signal can be obtained based on the first signal amplitude ratio and the amplitude of the first frequency domain near-end audio signal, wherein the amplitude of the second frequency domain near-end audio signal is the product of the first signal amplitude ratio and the amplitude of the first frequency domain near-end audio signal. For example, the trained denoising neural network model can be a neural network model based on deep learning, including, but not limited to, a convolutional neural network model.

[0057] According to an exemplary embodiment of the present disclosure, the denoising neural network model to be trained completes its training by training a training data set, wherein the training data set may include an amplitude data set of a noisy frequency signal.

[0058] According to an exemplary embodiment of the present disclosure, the second frequency domain near-end audio signal can be obtained by the following formula (4):

[0059] Y(n,k) p =Predict(MagY(n,k))*Phase(Y(n,k)); (4)

[0060] Among them, Y(n,k) pis the second frequency domain near-end audio signal, n is the frame sequence number, k is the center frequency sequence number, 0<n≤N, 0<k≤K, N is the total number of frames, K is the total number of frequency bands, Y(n,k) is the first frequency domain near-end audio signal, MagY(n,k) is the amplitude of the first frequency domain near-end audio signal, Predict(MagY(n,k)) represents deep learning denoising of the amplitude of the first frequency domain near-end audio signal, and Phase(Y(n,k)) is the phase of the first frequency domain near-end audio signal.

[0061] Back to Figure 2 In step 204, based on the frequency domain far-end reference audio signal, the frequency domain near-end collected audio signal, the first frequency domain near-end audio signal and the second frequency domain near-end audio signal, nonlinear echo cancellation can be performed on the second frequency domain near-end audio signal to obtain a frequency domain near-end audio enhancement signal.

[0062] According to an exemplary embodiment of the present disclosure, firstly, the frequency domain far-end reference audio signal can be correlated with the frequency domain near-end collected audio signal and the second frequency domain near-end audio signal in each frequency band to obtain the second signal amplitude ratio of each frequency band. Then, according to the second signal amplitude ratio, the first frequency domain near-end audio signal and the second frequency domain near-end audio signal, the second frequency domain near-end audio signal can be subjected to nonlinear echo cancellation to obtain the frequency domain near-end audio enhancement signal.

[0063] According to an exemplary embodiment of the present disclosure, firstly, a cross-correlation coefficient between a frequency domain far-end reference audio signal and a frequency domain near-end collected audio signal, as well as a cross-correlation coefficient between a frequency domain far-end reference audio signal and a second frequency domain near-end audio signal can be obtained. Then, based on the two cross-correlation coefficients, a second signal amplitude ratio of each frequency band can be obtained.

[0064] For example, the mutual correlation coefficient between the frequency domain far-end reference audio signal and the frequency domain near-end collected audio signal can be obtained by the following formula (5):

[0065] RCr(n,k)=Xcorr(R(n,k),C(n,k)); (5)

[0066] Wherein, RCr(n,k) is the mutual correlation coefficient between the frequency domain far-end reference audio signal and the frequency domain near-end collected audio signal, Xcorr(a,b) represents the mutual correlation coefficient between a and b in each frequency band obtained by cross-correlating a and b in each frame and each frequency band through a cross-correlation function, R(n,k) is the frequency domain far-end reference audio signal, C(n,k) is the frequency domain near-end collected audio signal, n is the frame sequence number, k is the center frequency sequence number, 0<n≤N, 0<k≤K, N is the total number of frames, and K is the total number of frequency bands.

[0067] For example, the cross-correlation coefficient between the frequency domain far-end reference audio signal and the second frequency domain near-end audio signal may be obtained by the following formula (6):

[0068] RY p r(n,k)=Xcorr(R(n,k),Y(n,k) p ); (6)

[0069] Among them, RY p r(n,k) is the cross-correlation coefficient between the frequency domain far-end reference audio signal and the second frequency domain near-end audio signal, Xcorr(a,b) represents the cross-correlation coefficient between a and b in each frame and each frequency band obtained by cross-correlation function, R(n,k) is the frequency domain far-end reference audio signal, Y(n,k) p is a second frequency domain near-end audio signal, n is a frame sequence number, k is a center frequency sequence number, 0<n≤N, 0<k≤K, N is a total number of frames, and K is a total number of frequency bands.

[0070] For example, the second signal amplitude ratio is obtained by the following formula (7):

[0071] Mask(n,k)=min{1-RCr(n,k),1-RY p r(n,k)}; (7)

[0072] Among them, Mask(n,k) is the second signal amplitude ratio, RCr(n,k) is the cross-correlation coefficient between the frequency domain far-end reference audio signal and the frequency domain near-end collected audio signal, RY p r(n,k) is the correlation coefficient between the frequency domain far-end reference audio signal and the second frequency domain near-end audio signal, n is the frame sequence number, k is the center frequency sequence number, 0<n≤N, 0<k≤K, N is the total number of frames, and K is the total number of frequency bands.

[0073] According to an exemplary embodiment of the present disclosure, first, the product of the second signal amplitude ratio and the amplitude of the first frequency domain near-end audio signal can be obtained as a reference amplitude. Then, the minimum value of the reference amplitude and the amplitude of the second frequency domain near-end audio signal can be obtained as the amplitude of the frequency domain near-end audio enhancement signal. Finally, the frequency domain near-end audio enhancement signal can be obtained according to the amplitude of the frequency domain near-end audio enhancement signal and the phase of the first frequency domain near-end audio signal.

[0074] For example, the frequency domain near-end audio enhancement signal can be obtained by the following formula (8):

[0075] Y(n,k) out =min{MagY(n,k)*Mask(n,k),MagY(n,k) p}*Phase(Y(n,k)); (8)

[0077] Among them, Y(n,k) outis the frequency domain near-end audio enhancement signal, n is the frame sequence number, k is the center frequency sequence number, 0<n≤N, 0<k≤K, N is the total number of frames, K is the total number of frequency bands, Y(n,k) is the first frequency domain near-end audio signal, MagY(n,k) is the amplitude of the first frequency domain near-end audio signal, Mask(n,k) is the second signal amplitude ratio, MagY(n,k) p is the amplitude of the near-end audio signal in the second frequency domain, Y(n,k) p is the second frequency-domain near-end audio signal, and Phase(Y(n,k)) is the phase of the first frequency-domain near-end audio signal.

[0078] Back to Figure 2 In step 205, the frequency domain near-end audio enhancement signal may be subjected to an inverse time-frequency transform to obtain a near-end audio enhancement signal.

[0079] According to an exemplary embodiment of the present disclosure, the inverse time-frequency transform may be, but is not limited to, a short-time inverse Fourier transform (ISTFT). In this case, the near-end audio enhancement signal may be obtained by the following formula (9):

[0080] y(t) out =ISTFT(Y(n,k) out ); (9)

[0081] Among them, y(t) out is the near-end audio enhancement signal, ISTFT(x) represents the short-time inverse Fourier transform of x, Y(n,k) out is the frequency domain near-end audio enhancement signal, t is time, 0<t≤T, T is the time length, n is the frame sequence number, k is the center frequency sequence number, 0<n≤N, 0<k≤K, N is the total number of frames, and K is the total number of frequency bands.

[0082] Figure 3 is a block diagram of an audio signal processing apparatus according to an exemplary embodiment of the present disclosure. Figure 3 According to an exemplary embodiment of the present disclosure, the audio signal processing device 300 may include a signal acquisition unit 301, a frequency domain transformation unit 302, a deep noise reduction unit 303, a nonlinear elimination unit 304 and a time domain transformation unit 305.

[0083] The signal acquisition unit 301 may acquire a near-end collected audio signal, a far-end reference audio signal, and a first near-end audio signal obtained by performing linear echo cancellation on the near-end collected audio signal.

[0084] According to an exemplary embodiment of the present disclosure, the near-end collected audio signal is an audio signal collected by a near-end microphone, and the far-end reference audio signal is an audio signal transmitted from the far-end without being played by a near-end speaker.

[0085] The audio signal collected by the near-end microphone may include, but is not limited to, a near-end user voice signal collected by the near-end microphone and an audio signal collected by the near-end microphone and transmitted from the far end via a network link and played by a near-end speaker.

[0086] The frequency domain transform unit 302 can perform time-frequency transform on the near-end collected audio signal, the far-end reference audio signal and the first near-end audio signal respectively to obtain a frequency domain near-end collected audio signal, a frequency domain far-end reference audio signal and a first frequency domain near-end audio signal.

[0087] According to an exemplary embodiment of the present disclosure, the time-frequency transform may be, but is not limited to, a short-time Fourier transform (STFT). The following description takes STFT as an example.

[0088] According to an exemplary embodiment of the present disclosure, the near-end collected audio signal, the far-end reference audio signal and the first near-end audio signal with a time length of T can be respectively expressed as c(t), r(t) and y(t) in the time domain, where t is time and 0<t≤T. After short-time Fourier transform, c(t), r(t) and y(t) can be expressed in the frequency domain by the above equations (1)-(3).

[0089] The deep noise reduction unit 303 may perform deep learning noise reduction on the amplitude of the first frequency domain near-end audio signal to obtain a second frequency domain near-end audio signal.

[0090] According to an exemplary embodiment of the present disclosure, deep learning noise reduction can be implemented by noise reduction through a neural network model. In this case, the deep noise reduction unit 303 can first perform deep learning noise reduction on the amplitude of the first frequency domain near-end audio signal through a trained noise reduction neural network model to obtain the amplitude of the second frequency domain near-end audio signal. The deep noise reduction unit 303 can then obtain the second frequency domain near-end audio signal based on the amplitude of the second frequency domain near-end audio signal and the phase of the first frequency domain near-end audio signal.

[0091] According to an exemplary embodiment of the present disclosure, the deep noise reduction unit 303 may first input the amplitude of the first frequency domain near-end audio signal into the trained noise reduction neural network model to obtain a first signal amplitude ratio, wherein the first signal amplitude ratio is a predicted value of the ratio of the amplitude of the second frequency domain near-end audio signal to the amplitude of the first frequency domain near-end audio signal. The deep noise reduction unit 303 may then obtain the amplitude of the second frequency domain near-end audio signal based on the first signal amplitude ratio and the amplitude of the first frequency domain near-end audio signal, wherein the amplitude of the second frequency domain near-end audio signal is the product of the first signal amplitude ratio and the amplitude of the first frequency domain near-end audio signal. For example, the trained noise reduction neural network model may be a neural network model based on deep learning (DeepLearning), including, but not limited to, a convolutional neural network model.

[0092] According to an exemplary embodiment of the present disclosure, the denoising neural network model to be trained completes its training by training a training data set, wherein the training data set may include an amplitude data set of a noisy frequency signal.

[0093] According to an exemplary embodiment of the present disclosure, the second frequency domain near-end audio signal can be obtained by the above formula (4).

[0094] Back to Figure 3 The nonlinear elimination unit 304 can perform nonlinear echo elimination on the second frequency domain near-end audio signal based on the frequency domain far-end reference audio signal, the frequency domain near-end collected audio signal, the first frequency domain near-end audio signal and the second frequency domain near-end audio signal to obtain a frequency domain near-end audio enhancement signal.

[0095] According to an exemplary embodiment of the present disclosure, the nonlinear elimination unit 304 may first correlate the frequency domain far-end reference audio signal with the frequency domain near-end collected audio signal and the second frequency near-end audio signal in each frequency band to obtain the second signal amplitude ratio of each frequency band. The nonlinear elimination unit 304 may then perform nonlinear echo elimination on the second frequency domain near-end audio signal according to the second signal amplitude ratio, the first frequency domain near-end audio signal and the second frequency domain near-end audio signal to obtain the frequency domain near-end audio enhancement signal.

[0096] According to an exemplary embodiment of the present disclosure, the nonlinear elimination unit 304 may first obtain a cross-correlation coefficient between a frequency domain far-end reference audio signal and a frequency domain near-end collected audio signal, and a cross-correlation coefficient between a frequency domain far-end reference audio signal and a second frequency domain near-end audio signal. The nonlinear elimination unit 304 may then obtain a second signal amplitude ratio of each frequency band based on the two cross-correlation coefficients.

[0097] For example, the mutual correlation coefficient between the frequency domain far-end reference audio signal and the frequency domain near-end collected audio signal can be obtained through the above formula (5).

[0098] For example, the cross-correlation coefficient between the frequency domain far-end reference audio signal and the second frequency domain near-end audio signal may be obtained by using the above formula (6).

[0099] For example, the second signal amplitude ratio is obtained by the above formula (7). According to an exemplary embodiment of the present disclosure, the nonlinear elimination unit 304 may first obtain the product of the second signal amplitude ratio and the amplitude of the first frequency domain near-end audio signal as the reference amplitude. The nonlinear elimination unit 304 may then obtain the minimum value of the reference amplitude and the amplitude of the second frequency domain near-end audio signal as the amplitude of the frequency domain near-end audio enhancement signal. The nonlinear elimination unit 304 may finally obtain the frequency domain near-end audio enhancement signal according to the amplitude of the frequency domain near-end audio enhancement signal and the phase of the first frequency domain near-end audio signal.

[0100] For example, the frequency domain near-end audio enhancement signal can be obtained by the above formula (8).

[0101] Back to Figure 3 The time domain transform unit 305 may perform an inverse time-frequency transform on the frequency domain near-end audio enhancement signal to obtain a near-end audio enhancement signal.

[0102] According to an exemplary embodiment of the present disclosure, the inverse time-frequency transform may be, but is not limited to, an inverse short-time Fourier transform (ISTFT). In this case, the near-end audio enhancement signal may be obtained by the above formula (9).

[0103] Figure 4 is a block diagram of an electronic device 400 according to an exemplary embodiment of the present disclosure.

[0104] Reference Figure 4 The electronic device 400 includes at least one memory 401 and at least one processor 402, wherein the at least one memory 401 stores a set of computer executable instructions, and when the computer executable instruction set is executed by the at least one processor 402, an audio signal processing method according to an exemplary embodiment of the present disclosure is executed.

[0105] As an example, the electronic device 400 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 400 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 400 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0106] In the electronic device 400, the processor 402 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0107] The processor 402 may execute instructions or codes stored in the memory 401, wherein the memory 401 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0108] The memory 401 may be integrated with the processor 402, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 401 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 401 and the processor 402 may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 402 can read files stored in the memory.

[0109] In addition, the electronic device 400 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 400 may be connected to each other via a bus and / or a network.

[0110] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to perform the audio signal processing method according to the exemplary embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0111] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided. Instructions in the computer program product may be executed by a processor of a computer device to implement the audio signal processing method according to an exemplary embodiment of the present disclosure.

[0112] According to the audio signal processing method and device disclosed in the present invention, linear echo cancellation is first performed on the near-end collected audio signal, then deep learning noise reduction is performed on it, and then nonlinear echo cancellation is performed. The echo signal in the far-end reference audio signal is eliminated by linear echo cancellation and nonlinear echo cancellation to obtain the final near-end audio enhancement signal. The deep learning noise reduction and echo cancellation are combined to fully utilize the good performance of deep learning noise reduction. Compared with the combination of traditional noise reduction technology and echo cancellation used in related technologies, better echo cancellation effect and noise reduction effect can be obtained, bringing about an improvement in sound quality.

[0113] In addition, according to the audio signal processing method and device disclosed in the present invention, deep learning noise reduction can be used to handle situations that are difficult to solve with traditional noise reduction, such as the elimination of quasi-steady-state or non-steady-state noise and the problem that traditional noise reduction is not effective in eliminating noise.

[0114] In addition, according to the audio signal processing method and device disclosed in the present invention, in the process of nonlinear echo elimination, by obtaining the signal amplitude ratio based on the cross-correlation coefficient, a frequency domain near-end audio enhancement signal is finally obtained, which can achieve the effect of eliminating echo and noise at the same time.

[0115] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0116] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An audio signal processing method, characterized in that: include: Acquire a near-end collected audio signal, a far-end reference audio signal, and a first near-end audio signal obtained by performing linear echo cancellation on the near-end collected audio signal; Performing time-frequency transformation on the near-end collected audio signal, the far-end reference audio signal and the first near-end audio signal respectively to obtain a frequency-domain near-end collected audio signal, a frequency-domain far-end reference audio signal and a first frequency-domain near-end audio signal; Performing deep learning noise reduction on the amplitude of the first frequency domain near-end audio signal to obtain a second frequency domain near-end audio signal; Correlating the frequency domain far-end reference audio signal with the frequency domain near-end collected audio signal and the second frequency domain near-end audio signal in each frequency band to obtain a mutual correlation coefficient between the frequency domain far-end reference audio signal and the frequency domain near-end collected audio signal, and a mutual correlation coefficient between the frequency domain far-end reference audio signal and the second frequency domain near-end audio signal, and obtaining a second signal amplitude ratio in each frequency band based on the obtained mutual correlation coefficient; performing nonlinear echo cancellation on the second frequency-domain near-end audio signal according to the second signal amplitude ratio, the first frequency-domain near-end audio signal, and the second frequency-domain near-end audio signal to obtain a frequency-domain near-end audio enhancement signal; Performing an inverse time-frequency transform on the frequency-domain near-end audio enhancement signal to obtain a near-end audio enhancement signal.

2. The audio signal processing method according to claim 1, characterized in that: The performing deep learning noise reduction on the amplitude of the first frequency domain near-end audio signal to obtain a second frequency domain near-end audio signal includes: Performing deep learning noise reduction on the amplitude of the near-end audio signal in the first frequency domain by using the trained noise reduction neural network model to obtain the amplitude of the near-end audio signal in the second frequency domain; The second frequency-domain near-end audio signal is obtained according to the amplitude of the second frequency-domain near-end audio signal and the phase of the first frequency-domain near-end audio signal.

3. The audio signal processing method according to claim 2, characterized in that: The step of performing deep learning noise reduction on the amplitude of the first frequency domain near-end audio signal by using the trained noise reduction neural network model to obtain the amplitude of the second frequency domain near-end audio signal includes: Inputting the amplitude of the first frequency-domain near-end audio signal into the trained denoising neural network model to obtain a first signal amplitude ratio, wherein the first signal amplitude ratio is a predicted value of a ratio of the amplitude of the second frequency-domain near-end audio signal to the amplitude of the first frequency-domain near-end audio signal; The amplitude of the second frequency-domain near-end audio signal is obtained according to the first signal amplitude ratio and the amplitude of the first frequency-domain near-end audio signal, wherein the amplitude of the second frequency-domain near-end audio signal is the product of the first signal amplitude ratio and the amplitude of the first frequency-domain near-end audio signal.

4. The audio signal processing method according to claim 1, characterized in that: The performing nonlinear echo cancellation on the second frequency-domain near-end audio signal according to the second signal amplitude ratio, the first frequency-domain near-end audio signal, and the second frequency-domain near-end audio signal to obtain a frequency-domain near-end audio enhancement signal includes: Acquire a product of the second signal amplitude ratio and the amplitude of the first frequency-domain near-end audio signal as a reference amplitude; Acquire a minimum value between the reference amplitude and the amplitude of the second frequency-domain near-end audio signal as the amplitude of the frequency-domain near-end audio enhancement signal; A frequency domain near-end audio enhancement signal is obtained according to the amplitude of the frequency domain near-end audio enhancement signal and the phase of the first frequency domain near-end audio signal.

5. The audio signal processing method according to claim 1 or 4, characterized in that: The second signal amplitude ratio is obtained by the following formula: Mask(n,k)=min{1-RCr(n,k),1-RY p r(n,k)}; Wherein, Mask(n,k) is the second signal amplitude ratio, RCr(n,k) is the cross-correlation coefficient between the frequency-domain far-end reference audio signal and the frequency-domain near-end collected audio signal, and RY p r(n,k) is the cross-correlation coefficient between the frequency-domain far-end reference audio signal and the second frequency-domain near-end audio signal, n is the frame sequence number, k is the center frequency sequence number, 0 < n ≤ N, 0 < k ≤ K, N is the total number of frames, and K is the total number of frequency bands.

6. An audio signal processing device, characterized in that: include: The signal acquisition unit is configured to: acquire a near-end collected audio signal, a far-end reference audio signal, and a first near-end audio signal obtained by performing linear echo cancellation on the near-end collected audio signal; A frequency domain transform unit is configured to: perform time-frequency transform on the near-end collected audio signal, the far-end reference audio signal and the first near-end audio signal respectively to obtain a frequency domain near-end collected audio signal, a frequency domain far-end reference audio signal and a first frequency domain near-end audio signal; A deep noise reduction unit is configured to: perform deep learning noise reduction on the amplitude of the first frequency domain near-end audio signal to obtain a second frequency domain near-end audio signal; The nonlinear elimination unit is configured to: correlate the frequency domain far-end reference audio signal with the frequency domain near-end collected audio signal and the second frequency domain near-end audio signal in each frequency band to obtain a mutual correlation coefficient between the frequency domain far-end reference audio signal and the frequency domain near-end collected audio signal, and a mutual correlation coefficient between the frequency domain far-end reference audio signal and the second frequency domain near-end audio signal, and obtain a second signal amplitude ratio of each frequency band based on the obtained mutual correlation coefficient; perform nonlinear echo elimination on the second frequency domain near-end audio signal according to the second signal amplitude ratio, the first frequency domain near-end audio signal and the second frequency domain near-end audio signal to obtain a frequency domain near-end audio enhancement signal; The time domain transform unit is configured to: perform time-frequency inverse transform on the frequency domain near-end audio enhancement signal to obtain a near-end audio enhancement signal.

7. The audio signal processing device according to claim 6, characterized in that: The Deep Denoising Unit is configured as: Performing deep learning noise reduction on the amplitude of the near-end audio signal in the first frequency domain by using the trained noise reduction neural network model to obtain the amplitude of the near-end audio signal in the second frequency domain; The second frequency-domain near-end audio signal is obtained according to the amplitude of the second frequency-domain near-end audio signal and the phase of the first frequency-domain near-end audio signal.

8. The audio signal processing device according to claim 7, characterized in that: The Deep Denoising Unit is configured as: Inputting the amplitude of the first frequency-domain near-end audio signal into the trained denoising neural network model to obtain a first signal amplitude ratio, wherein the first signal amplitude ratio is a predicted value of a ratio of the amplitude of the second frequency-domain near-end audio signal to the amplitude of the first frequency-domain near-end audio signal; The amplitude of the second frequency-domain near-end audio signal is obtained according to the first signal amplitude ratio and the amplitude of the first frequency-domain near-end audio signal, wherein the amplitude of the second frequency-domain near-end audio signal is the product of the first signal amplitude ratio and the amplitude of the first frequency-domain near-end audio signal.

9. The audio signal processing device according to claim 6, characterized in that: The nonlinear elimination unit is configured as: Acquire a product of the second signal amplitude ratio and the amplitude of the first frequency-domain near-end audio signal as a reference amplitude; Acquire a minimum value between the reference amplitude and the amplitude of the second frequency-domain near-end audio signal as the amplitude of the frequency-domain near-end audio enhancement signal; A frequency domain near-end audio enhancement signal is obtained according to the amplitude of the frequency domain near-end audio enhancement signal and the phase of the first frequency domain near-end audio signal.

10. The audio signal processing device according to claim 6 or 9, characterized in that: The second signal amplitude ratio is obtained by the following formula: Mask(n,k)=min{1-RCr(n,k),1-RY p r(n,k)}; Among them, Mask(n,k) is the second signal amplitude ratio, RCr(n,k) is the cross-correlation coefficient between the frequency-domain far-end reference audio signal and the frequency-domain near-end collected audio signal, and RY p r(n,k) is the cross-correlation coefficient between the frequency-domain far-end reference audio signal and the second frequency-domain near-end audio signal, n is the frame sequence number, k is the center frequency sequence number, 0 < n ≤ N, 0 < k ≤ K, N is the total number of frames, and K is the total number of frequency bands.

11. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to perform the audio signal processing method according to any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the audio signal processing method according to any one of claims 1 to 5.

13. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by at least one processor, the audio signal processing method according to any one of claims 1 to 5 is implemented.

Images