Audio processing method, electronic equipment and computer readable storage medium

Abstract

The invention discloses an audio processing method, electronic equipment and a computer readable storage medium. The audio processing method comprises the following steps: acquiring a near-end audio signal and a far-end audio signal; respectively inputting the near-end audio signal and the far-end audio signal into a neural network model to enable the neural network model to output filtering parameters in real time, and updating the filtering parameters of the adaptive filter; respectively inputting the far-end audio signal and the echo cancellation signal into an adaptive filter, so that the adaptive filter outputs a predicted linear echo signal; and obtaining an echo cancellation signal based on the near-end audio signal and the predicted linear echo signal. Based on the above mode, the accuracy of echo cancellation can be improved.

Description

Technical Field

[0001] The present application relates to the field of audio technology, and in particular to an audio processing method, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the prior art, in a vehicle-mounted space, for example, when a user is making a call, the audio signal emitted by a distant speaker device will be synchronously received by the sound receiving device, resulting in an echo phenomenon, which causes the quality of the voice call made by the user through the sound receiving device to be poor. Therefore, it is usually necessary to set a module for echo cancellation in the audio device in the vehicle-mounted space to improve the quality of the voice call.

[0003] The defect of the prior art is that the existing modules for echo cancellation are usually unable to adapt to complex and changeable acoustic environments. That is, when the reception and amplification conditions of the audio signals in the vehicle space change differently, the effect of the echo cancellation will also change accordingly, sometimes the effect is good and sometimes the effect is poor. It is impossible to change the relevant parameters of the echo cancellation according to different acoustic environments, so that the accuracy of each echo cancellation cannot be ensured, which easily makes the quality of voice calls unstable. Therefore, how to improve the accuracy of echo cancellation has become a technical problem that needs to be solved urgently in this field. Summary of the invention

[0004] The main technical problem solved by this application is how to improve the accuracy of echo cancellation.

[0005] In order to solve the above technical problems, the first technical solution adopted by the present application is: an audio processing method, applied to an audio processing device, the audio processing device includes an echo cancellation module, the echo cancellation module includes a neural network model and an adaptive filter; the audio processing method includes: obtaining a near-end audio signal and a far-end audio signal; inputting the near-end audio signal and the far-end audio signal into the neural network model respectively, so that the neural network model outputs filtering parameters in real time and updates the filtering parameters of the adaptive filter; inputting the far-end audio signal into the adaptive filter, so that the adaptive filter outputs a predicted linear echo signal; and obtaining an echo cancellation signal based on the near-end audio signal and the predicted linear echo signal.

[0006] Among them, the echo cancellation module also includes a nonlinear echo processing module; based on the near-end audio signal and the predicted linear echo signal, an echo cancellation signal is obtained, including: subtracting the predicted linear echo signal from the near-end audio signal to obtain a transition echo cancellation signal; the near-end audio signal, the far-end audio signal and the transition echo cancellation signal are respectively input into the nonlinear echo processing module, so that the nonlinear echo processing module outputs an echo cancellation signal.

[0007] The audio processing method further includes: obtaining room impulse response data, clean audio samples and noisy audio samples; obtaining noisy audio samples based on the clean audio samples and the noisy audio samples; using the first noisy audio sample as a sample of a far-end audio signal; convolving the far-end audio signal with the room impulse response data to obtain a convolution result, and performing weighted summation of the convolution result and the second noisy audio sample to obtain a sample of a near-end audio signal; using the second noisy audio sample as a sample of an echo cancellation signal; and training a neural network model based on samples of the far-end audio signal, samples of the near-end audio signal and samples of the echo cancellation signal.

[0008] The neural network model is trained based on samples of the far-end audio signal, samples of the near-end audio signal and samples of the echo cancellation signal, including: inputting samples of the far-end audio signal and samples of the near-end audio signal into the neural network model respectively so that the adaptive filter outputs the echo cancellation signal to be adjusted; obtaining a loss function based on samples of the echo cancellation signal to be adjusted and the echo cancellation signal; and updating the parameters of the neural network model based on the loss function and the gradient descent method.

[0009] Among them, the audio processing device also includes a noise reduction module, which includes a feature fusion layer, a feature compression layer, a sub-band division layer, a feature enhancement layer and a mask layer; the audio processing method includes: inputting the echo cancellation signal into the feature fusion layer so that the feature fusion layer outputs a first signal; inputting the first signal into the feature compression layer so that the feature compression layer outputs a second signal; inputting the second signal into the sub-band division layer so that the sub-band division layer outputs multiple third signals respectively used to characterize the corresponding sub-band features; two third signals adjacent in the frequency domain have an overlapping area in the frequency domain; inputting the third signal into the feature enhancement layer so that the feature enhancement layer outputs a fourth signal; inputting the echo cancellation signal and the fourth signal into the mask layer so that the mask layer outputs a noise reduction signal.

[0010] Among them, the feature enhancement layer includes a coding layer, an enhancement layer and a decoding layer; inputting the third signal into the feature enhancement layer so that the feature enhancement layer outputs a fourth signal, including: inputting the third signal into the coding layer so that the coding layer outputs a coded signal; inputting the coded signal into the enhancement layer so that the enhancement layer outputs an enhanced signal; inputting the enhancement signal into the decoding layer so that the decoding layer outputs a decoded signal, recorded as the fourth signal.

[0011] Before the echo cancellation signal is input into the feature fusion layer, the audio processing method further includes: using the initial noise reduction module as a teacher model; performing knowledge distillation on the teacher model to obtain a student model, recorded as a noise reduction model.

[0012] The audio processing device further includes a gain control module; the echo cancellation signal or the noise reduction signal obtained by performing noise reduction based on the echo cancellation signal is input into the gain control module for gain processing to obtain an output audio signal.

[0013] In order to solve the above technical problems, the second technical solution adopted in the present application is: an electronic device, comprising: a memory and a processor; the memory is used to store program instructions, and the processor is used to execute the program instructions to implement the above method.

[0014] In order to solve the above technical problems, the third technical solution adopted in the present application is: a computer-readable storage medium, which stores program instructions, and the above method is implemented when the program instructions are executed by a processor.

[0015] The beneficial effect of the present application is that, different from the prior art, in the technical solution of the present application, a near-end audio signal and a far-end audio signal are obtained, and the near-end audio signal and the far-end audio signal are respectively input into a neural network model so that the neural network model outputs filter parameters in real time, and the filter parameters of the adaptive filter are updated, and the far-end audio signal is input into the adaptive filter so that the adaptive filter outputs a predicted linear echo signal, and an echo cancellation signal is obtained based on the near-end audio signal and the predicted linear echo signal. Based on the above method, by setting a neural network model, the near-end audio signal and the far-end audio signal currently received can be processed and analyzed in real time, and the filter parameters that are most suitable for the acoustic environment can be output. Based on the filter parameters generated in real time, the adaptive filter can update its filter parameters in real time to improve its echo cancellation capability for the acoustic environment, that is, based on the setting of the neural network model, the adaptive filter can have a good echo cancellation capability regardless of the acoustic environment it is facing, thereby improving the accuracy of echo cancellation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0017] Figure 1 is a structural schematic diagram of an embodiment of the audio processing device of the present application;

[0018] Figure 2 It is a structural diagram of an embodiment of the echo cancellation module of the present application;

[0019] Figure 3 It is a flowchart of an embodiment of the audio processing method of the present application;

[0020] Figure 4 is a flowchart of another embodiment of the audio processing method of the present application;

[0021] Figure 5 It is a structural schematic diagram of an embodiment of the noise reduction module of the present application;

[0022] Figure 6 This is a schematic diagram of a process flow of an embodiment of knowledge distillation of the present application;

[0023] Figure 7 It is a schematic diagram of the structure of an embodiment of the electronic device of the present application;

[0024] Figure 8 It is a structural schematic diagram of an embodiment of the computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0025] The present application is further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present application, but are not intended to limit the scope of the present application. Similarly, the following examples are only some embodiments of the present application rather than all embodiments, and all other embodiments obtained by those of ordinary skill in the art without making creative work are within the scope of protection of the present application.

[0026] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0027] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "install", "set", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or connected through an intermediate medium. For ordinary technicians in this field, the above-mentioned specific meanings belonging to this application can be connected according to specific circumstances.

[0028] This application proposes an audio processing method, which is applied to an audio processing device, see Figures 1 to 3 , Figure 1 is a structural diagram of an embodiment of the audio processing device of the present application, Figure 2 is a schematic structural diagram of an embodiment of the echo cancellation module of the present application, Figure 3 It is a flowchart of an embodiment of the audio processing method of the present application.

[0029] like Figure 1 As shown, in one example, in a vehicle space or other type of space, a near-end audio signal received by a sound receiving device located near the user can be input into an echo cancellation module 1, and the near-end audio signal generally includes an original audio signal generated by the user's direct voice and an echo audio signal generated by a speaker device located far from the user. In addition, a far-end audio signal currently received by the speaker device located far from the user can also be input into the echo cancellation module 1, and the far-end audio signal is generally an audio signal based on which the speaker device speaks.

[0030] The echo cancellation module 1 can perform echo cancellation on the near-end audio signal based on the received far-end audio signal and the near-end audio signal, and then input the audio signal after echo cancellation into the noise reduction module 2. The noise reduction module 2 can perform noise reduction on the received audio signal, and then input the audio signal after noise reduction into the gain control module 3. The gain control module 3 can perform gain control on the received audio signal so that the volume of the output audio signal output by the gain control module 3 is within an appropriate range. The gain value can be greater than 1, less than 1, or even equal to 1, depending on the actual situation.

[0031] Based on the above method, the obtained output audio signal can be successfully echo-cancelled, noise-reduced and gain-controlled, and the final output audio signal can better restore the original information of the user's voice and provide a better voice call effect.

[0032] like Figure 2 As shown, the audio processing device includes an echo cancellation module 1 , and the echo cancellation module 1 includes a neural network model 11 and an adaptive filter 12 .

[0033] like Figure 3 As shown, the audio processing method includes:

[0034] Step S11: Acquire a near-end audio signal and a far-end audio signal.

[0035] The near-end audio signal may specifically be an audio signal generated by a receiving device when a user speaks, and may generally include an original audio signal generated by the user's direct voice and an echo audio signal generated by a speaker located at a remote end from the user.

[0036] The far-end audio signal may specifically be a speaker audio signal input into a speaker device for speaker sound.

[0037] Step S12: inputting the near-end audio signal and the far-end audio signal into the neural network model 11 respectively, so that the neural network model 11 outputs the filtering parameters in real time, and updates the filtering parameters of the adaptive filter 12.

[0038] The adaptive filter 12 may specifically be a filter that performs adaptive filtering based on a frequency domain block Kalman algorithm, and the above-mentioned filtering parameters may specifically refer to static parameters alpha and beta in the frequency domain block Kalman algorithm.

[0039] The neural network model 11 can analyze and process the near-end audio signal and the far-end audio signal, and output the filtering parameters adapted to the acoustic environment to the adaptive filter 12. The adaptive filter 12 can update its original filtering parameters in real time to the latest filtering parameters received and output by the neural network model 11, so that the frequency domain block Kalman algorithm therein can be more adapted to the acoustic environment, so as to provide better linear echo cancellation effect in any acoustic environment.

[0040] Step S13: inputting the far-end audio signal into the adaptive filter 12, so that the adaptive filter 12 outputs a predicted linear echo signal.

[0041] The signal output by the adaptive filter 12 is a predicted linear echo signal, and an echo cancellation signal can be obtained and output by subtracting the predicted linear echo signal from the near-end audio signal.

[0042] The adaptive filter 12 may perform adaptive filtering processing based on the far-end audio signal to estimate a predicted linear echo signal.

[0043] Step S14: obtaining an echo cancellation signal based on the near-end audio signal and the predicted linear echo signal.

[0044] The predicted linear echo signal may be subtracted from the near-end audio signal to obtain and output an echo cancellation signal.

[0045] Specifically, in one example, Figure 2 As shown, the calculation process of the adaptive filter 12 is as follows:

[0046] The Kalman state parameter P and the predicted echo path H of the adaptive filter 12 are initialized, and the static parameters alpha and beta of the adaptive filter 12 are given.

[0047] First, Fourier transform is performed on the near-end audio signal and the far-end audio signal to obtain the transformed near-end audio signal Y and far-end audio signal S:

[0048] Y = STFT(y)

[0049] S = STFT(s)

[0050] Second, based on the predicted echo path H t The first echo cancellation signal E is obtained, where * is a cross product or a vector multiplication:

[0051] E=SY*H t

[0052] Third, calculate the Kalman gain, where () T is the conjugate transpose of the vector, conj() is the conjugate of the vector, P t is the Kalman parameter, G is the Kalman gain:

[0053]

[0054] Fourth, update the next Kalman state parameter P t+1 and the next predicted echo path H t+1 :

[0055] P t+1 =beta*G*Y*P t +H t *(H t ) T

[0056] H t+1 =H t +E*G

[0057] Fifth, based on the above method, the static parameters alpha and beta of the adaptive filter 12 can be continuously updated based on the filter parameters currently received and sent by the neural network model 11, so that the adaptive filter 12 can perform adaptive filtering processing in the state that best suits the acoustic environment, and continuously update the next Kalman state parameter P t+1 and the next predicted echo path H t+1 , to output a predicted linear echo signal, to generate a first echo cancellation signal E that cancels the linear echo, which can be used as the echo cancellation signal later, or the first echo cancellation signal E can be subjected to other echo cancellation processing to obtain a second echo cancellation signal F, which is used as the echo cancellation signal, which is not limited here.

[0058] In summary, by setting the neural network model 11, the accuracy of the predicted linear echo signal output by the adaptive filter 12 in different acoustic environments can be effectively improved, thereby improving the accuracy and stability of echo cancellation.

[0059] Different from the prior art, in the technical solution of the present application, a near-end audio signal and a far-end audio signal are obtained, and the near-end audio signal and the far-end audio signal are respectively input into a neural network model so that the neural network model outputs filter parameters in real time, and the filter parameters of the adaptive filter are updated. The far-end audio signal is input into the adaptive filter so that the adaptive filter outputs a predicted linear echo signal, and an echo cancellation signal is obtained based on the near-end audio signal and the predicted linear echo signal. Based on the above method, by setting a neural network model, the near-end audio signal and the far-end audio signal currently received can be processed and analyzed in real time, and the filter parameters that are most suitable for the acoustic environment can be output. Based on the filter parameters generated in real time, the adaptive filter can update its filter parameters in real time to improve its echo cancellation capability for the acoustic environment. That is, based on the setting of the neural network model, the adaptive filter can have a good echo cancellation capability regardless of the acoustic environment it is facing, thereby improving the accuracy of echo cancellation.

[0060] In one embodiment, if Figure 2 As shown, the echo cancellation module 1 further includes a nonlinear echo processing module 13 .

[0061] Based on the near-end audio signal and the predicted linear echo signal, an echo cancellation signal is obtained, including:

[0062] The predicted linear echo signal is subtracted from the near-end audio signal to obtain a transition echo cancellation signal.

[0063] The near-end audio signal, the far-end audio signal and the transition echo cancellation signal are respectively input into the nonlinear echo processing module 13, so that the nonlinear echo processing module 13 outputs an echo cancellation signal.

[0064] Specifically, after being processed by the adaptive filter 12 to achieve linear echo cancellation of the near-end audio signal, a transition echo cancellation signal can be obtained, that is, the first echo cancellation signal E described in the previous embodiment. The nonlinear echo processing module 13 can perform nonlinear echo cancellation on the first echo cancellation signal E to obtain a final echo cancellation signal, that is, the second echo cancellation signal F described in the previous embodiment.

[0065] In one example, if Figure 2 As shown, the calculation process of the nonlinear echo processing module 13 is as follows:

[0066] First, Fourier transform is performed on the near-end audio signal, the far-end audio signal and the first echo cancellation signal (transition echo cancellation signal) respectively to obtain the transformed near-end audio signal Y, the far-end audio signal S and the first echo cancellation signal E, where cohes is the similarity between the first echo cancellation signal and the near-end audio signal, and cohys is the similarity between the first echo cancellation signal and the far-end audio signal:

[0067] E = STFT(e)

[0068] S = STFT(s)

[0069] Y = STFT(y)

[0070]

[0071]

[0072] Second, the nonlinear echo residual mask is calculated.

[0073] When cohes is smaller, it means that the echo residue in the first echo cancellation signal is smaller. When cohys is larger, it means that the echo proportion in the near-end audio signal of the current frame is larger. Therefore, the mask value is as follows, where min() is the minimum value operation:

[0074] mask=min(cohes,1-cohys)

[0075] Third, the echo residual mask mask is cross-multiplied with the first echo cancellation signal E to obtain a second echo cancellation signal F, and then the second echo cancellation signal F is inversely Fourier transformed to obtain a second echo cancellation signal f(n), which is the echo cancellation signal described in the above embodiment:

[0076] F=mask*E

[0077] f(n)=ISTFT(F)

[0078] At this point, the related operations of linear echo cancellation and nonlinear echo cancellation of the near-end audio signal are completed, and an echo-cancelled signal is obtained.

[0079] Based on the above method, the echo cancellation operation can be further improved, the echo residue in the output echo cancellation signal can be reduced or eliminated as much as possible, and the accuracy of echo cancellation is improved.

[0080] For an example, see Figure 4 , Figure 4 is a flow chart of another embodiment of the audio processing method of the present application. Figure 4 As shown, the audio processing method includes:

[0081] Step S21: Acquire a near-end audio signal and a far-end audio signal.

[0082] Step S22: inputting the near-end audio signal and the far-end audio signal into the neural network model 11 respectively, so that the neural network model 11 outputs the filtering parameters in real time, and updates the filtering parameters of the adaptive filter 12.

[0083] Step S23: inputting the far-end audio signal into the adaptive filter 12, so that the adaptive filter 12 outputs a predicted linear echo signal.

[0084] Step S24: obtaining an echo cancellation signal based on the near-end audio signal and the predicted linear echo signal.

[0085] The above steps S21-S24 correspond to steps S11-S14 in the previous embodiment and will not be repeated here.

[0086] Step S25: Based on the noise reduction module, perform noise reduction processing on the echo cancellation signal to obtain a noise reduction signal.

[0087] Step S26: Based on the gain control module, the noise reduction signal is subjected to gain processing to obtain an output audio signal.

[0088] Based on the above steps S21-S24, step S25 and step S26, the near-end audio signal can be echo eliminated, noise reduced and gain controlled in sequence to complete the audio processing of the near-end audio signal. The obtained output audio signal reduces or eliminates the echo and noise compared to the near-end audio signal, and makes its volume more appropriate, thereby improving the quality of the output audio signal and improving the user's call experience.

[0089] In one embodiment, the audio processing method further includes:

[0090] Get room impulse response data, clean audio samples, and noisy audio samples.

[0091] Based on the clean audio samples and the noisy audio samples, noisy audio samples are obtained.

[0092] The first noisy frequency sample is taken as the sample of the far-end audio signal.

[0093] The far-end audio signal is convolved with the room impulse response data to obtain a convolution result, and the convolution result is weightedly summed with the second noisy frequency sample to obtain a sample of the near-end audio signal.

[0094] The second noisy frequency sample is used as the echo cancellation signal sample.

[0095] The neural network model 11 is trained based on samples of the far-end audio signal, samples of the near-end audio signal, and samples of the echo cancellation signal.

[0096] Specifically, the neural network model 11 may include an encoder, a recurrent neural network (RNN) and a feedforward neural network (FNN) connected in sequence. The near-end audio signal and the far-end audio signal after Fourier transformation may be input into the encoder, and then processed by the recurrent neural network and the feedforward neural network to obtain the output result of the filtering parameter, wherein the recurrent neural network can establish a long-distance dependency relationship, which plays an important role in identifying the echo environment. The neural network model 11 may also be other types of neural networks, which are not limited here.

[0097] Training the neural network model 11 based on the samples of the far-end audio signal, the samples of the near-end audio signal, and the samples of the echo cancellation signal may specifically include:

[0098] The samples of the far-end audio signal and the samples of the near-end audio signal are respectively input into the neural network model 11, so that the adaptive filter 12 outputs the echo cancellation signal to be adjusted.

[0099] A loss function is obtained based on the samples of the echo cancellation signal to be adjusted and the echo cancellation signal.

[0100] Based on the loss function and the gradient descent method, the parameters of the neural network model 11 are updated.

[0101] In one example, the training process for the neural network model 11 is as follows:

[0102] First, a data set of room impulse response (RIR) of the space to be echo canceled (such as the vehicle space) is constructed, wherein the room impulse response can be calculated based on parameters such as the size of the vehicle space, the radio equipment and the speaker equipment, or the room impulse response corresponding to the vehicle space provided by the AEC-challenge database can be obtained, or the room impulse response can be obtained in other ways, which are not limited here.

[0103] Second, construct a dataset of noisy audio samples. First, construct a dataset of clean audio samples. Clean audio samples can be obtained by convolving the single-channel clean audio signal in the DNS-Challenge dataset with the room impulse response. Then construct a dataset of noisy audio samples. This can be constructed by obtaining noisy audio samples in the DNS-Challenge noise dataset and the MUSAN dataset. Finally, any clean audio sample and any noisy audio sample are weighted and summed at a preset signal-to-noise ratio to obtain a noisy audio sample.

[0104] Third, construct samples of near-end audio signals and samples of user voice signals, wherein the samples of user voice signals are samples of echo cancellation signals, which can be understood as audio signals in which the echo is theoretically completely eliminated and only the voice signal portion of the user input to the sound receiving device is retained. Two noisy frequency samples can be randomly selected, one as a sample of the far-end audio signal and the other as a sample of the user voice signal. The echo signal is obtained by convolving the sample of the far-end audio signal with the room impulse response, and then the echo signal and the user voice signal are weighted and summed according to the preset signal-to-echo ratio to obtain a sample of the near-end audio signal.

[0105] The neural network model 11 can be trained in groups based on each group of far-end audio signal samples, near-end audio signal samples and user voice signal samples. For example, the far-end audio signal samples and near-end audio signal samples can be Fourier transformed into the time-frequency domain respectively, and then respectively input into the neural network model 11. The neural network model 11 outputs corresponding filter parameters based on the input samples, and the filter parameters of the adaptive filter 12 are updated so that the adaptive filter 12 outputs the echo cancellation signal to be adjusted. The MSE loss function can be used to reflect the distinction between the echo cancellation signal to be adjusted and the user voice signal sample. Based on the distinction, the network parameters of the neural network model 11 are updated by the gradient descent method, and the adaptive filter 12 outputs the echo cancellation signal to be adjusted. The cycle is repeated until the value of the MSE loss function reaches a preset condition, such as being less than a preset value or within a preset range, and the training of a group of samples is completed. By completing the training of multiple groups of samples, the neural network model 11 can be equipped with the ability to output the filter parameters that are most suitable for the current acoustic environment for echo cancellation.

[0106] The formula for the MSE loss function is as follows:

[0107]

[0108] Among them, MSE is the value of the MSE loss function, N is the number of signal sampling points, x(n) is the user's voice signal, and x(n) ′ is the first echo cancellation signal.

[0109] Based on the above method, the neural network model 11 can be effectively trained so that the neural network model 11 has the ability to recognize the characteristics of the current acoustic environment and output the filtering parameters that are most suitable for the current acoustic environment, further improving the accuracy of echo cancellation.

[0110] In one embodiment, see Figure 5 , Figure 5 is a structural diagram of an embodiment of the noise reduction module of the present application, such as Figure 5As shown, the audio processing device also includes a noise reduction module 2, which includes a feature fusion layer 21, a feature compression layer 22, a sub-band division layer 23, a feature enhancement layer 24 and a mask layer 25.

[0111] The audio processing method may further include:

[0112] The echo cancellation signal is input into the feature fusion layer 21 so that the feature fusion layer 21 outputs a first signal.

[0113] The first signal is input into the feature compression layer 22 so that the feature compression layer 22 outputs a second signal.

[0114] The second signal is input to the sub-band division layer 23, so that the sub-band division layer 23 outputs a plurality of third signals respectively used to characterize the characteristics of the corresponding sub-bands. Two third signals adjacent in the frequency domain have an overlapping area in the frequency domain.

[0115] The third signal is input to the feature enhancement layer 24 so that the feature enhancement layer 24 outputs a fourth signal.

[0116] The echo cancellation signal and the fourth signal are input to the mask layer 25 so that the mask layer 25 outputs a noise reduction signal.

[0117] Specifically, the feature enhancement layer 24 may include a coding layer 241 , an enhancement layer 242 and a decoding layer 243 .

[0118] Inputting the third signal into the feature enhancement layer 24 so that the feature enhancement layer 24 outputs a fourth signal may specifically include:

[0119] The third signal is input to the coding layer 241 so that the coding layer 241 outputs a coded signal.

[0120] The encoded signal is input to the enhancement layer 242 so that the enhancement layer 242 outputs an enhanced signal.

[0121] The enhanced signal is input to the decoding layer 243, so that the decoding layer 243 outputs a decoded signal, which is recorded as a fourth signal.

[0122] In one example, the noise reduction process may be specifically as follows:

[0123] First, the echo cancellation signal can be Fourier transformed to obtain a complex spectrum echo cancellation signal as the input signal of the feature fusion layer. Specifically, the Fourier transform can be a short-time Fourier transform. Assuming that the window length of the short-time Fourier transform is 512 and the frame shift size is 256, its frequency dimension is 257. After the input signal is input into the feature fusion layer for feature fusion and then into the feature compression layer for feature compression, the frequency dimension will be compressed from 257 to 129.

[0124] Second, given the compressed feature feat, feat is a tensor, that is, the shape is (B, C, T, 129), where B is the batch, C is the number of channels, T is the time dimension, and 129 is the compressed frequency dimension. feat can be expanded into a feature map according to the local window k and the sliding step size s to perform sub-band division, and the divided sub-band feature feat_sub is obtained, that is, the shape is (B, C*k, T, 129).

[0125] Taking k=3 and s=1 as an example, the frequency range calculation process corresponding to each sub-band is as follows:

[0126] 1. Calculate the frequency resolution:

[0127]

[0128] Where Δf is the frequency resolution, f_s is the sampling rate, and N is the number of sampling points. Taking the sampling rate as 16K Hz (16000 Hz) as an example, Δf = 16000 / 512 = 31.25 Hz. Assume that after feature compression processing, the frequency points can still be evenly distributed within the original frequency range.

[0129] 2. Calculate the frequency range covered by each sub-band:

[0130] Δf sub =Δf*k

[0131] Where Δf sub is the frequency range covered by a single subband. In this example, k is 3, then Δf sub =31.25*3=93.75Hz.

[0132] 3. In the above calculation process, in this example, the frequency range of each sub-band is divided into Δf sub , and the specific division range of each sub-band also depends on the frequency starting point of each sub-band, as shown in the following formula:

[0133] (is)*Δf~(is)*Δf+Δf sub

[0134] In this example, s is 1, and the above is the calculation formula for the frequency range corresponding to the i-th sub-band, for example:

[0135] Frequency range of the first subband:

[0136] (1-1)*31.25~(1-1)*31.25+93.75Hz=0~93.75Hz

[0137] As can be seen from the above, the frequency range of the first sub-band is 0 to 93.75 Hz.

[0138] Frequency range of the second subband:

[0139] (2-1)*31.25~(2-1)*31.25+93.75Hz=31.25~125Hz

[0140] As can be seen from the above, the frequency range of the second sub-band is 31.25~125Hz.

[0141] By analogy, the frequency range of each sub-band can be calculated until all sub-bands can cover all frequency points. The signal corresponding to each of the above sub-bands is the above third signal.

[0142] Based on the above method, when dividing the sub-bands, the information in the frequency range of the adjacent sub-bands can be used to enhance the information in the frequency range of the sub-band, thereby achieving the enhancement of each sub-band information, and based on the information in each sub-band, the effect of reducing noise and retaining useful information can be better achieved, thereby improving the reliability of the noise reduction process. Moreover, some noise may only affect certain parts of the spectrum. Through sub-band division, the influence of these noises on the overall processing results can be reduced, and through the setting of overlapping frequency domain sub-bands, the noise reduction module 2 can focus on fine-grained spectral features, further improving the reliability of the noise reduction process.

[0143] Third, the third signal obtained by sub-band division can be input into the encoding layer for encoding, and then the encoded signal obtained after encoding is input into the enhancement layer composed of the dual-path recurrent neural network (DPRNN) for feature transformation to enhance the features, and then the enhanced signal obtained after feature enhancement is input into the decoding layer for decoding, and the decoded signal is output, that is, the fourth signal is output to the mask layer. It should be noted that the grouped convolution kernel grouped transposed convolution structure is adopted in both the encoding layer and the decoding layer, and the grouped GRU and grouped Linear structures are adopted in the dual-path recurrent neural network layer, which further optimizes the processing efficiency and improves the noise reduction efficiency.

[0144] Fourth, the mask layer obtains the echo cancellation signal before inputting into the noise reduction module 2 and the fourth signal obtained by processing the echo cancellation signal by the noise reduction module 2, and performs a dot product between the echo cancellation signal and the fourth signal to obtain the final output noise reduction signal, thereby achieving noise reduction.

[0145] Optionally, before inputting the echo cancellation signal into the feature fusion layer 21, the audio processing method may further include:

[0146] The initial denoising module is used as the teacher model.

[0147] The teacher model is subjected to knowledge distillation to obtain the student model, which is denoised as the denoising model.

[0148] Specifically, see Figure 6 , Figure 6 This is a schematic diagram of the process of an embodiment of the knowledge distillation of the present application. Figure 6 As shown in the figure, Noisy input is the noisy signal introduced during the knowledge distillation process.

[0149] Teacher ANS model is a teacher noise reduction model, and the initially constructed noise reduction module model described in the above embodiment can be used as the teacher noise reduction model.

[0150] Solf Enhancement is the denoising result output by the teacher denoising model, that is, its enhanced signal, which serves as a soft label during offline distillation training.

[0151] The Student ANS model is the student denoising model, which is what this knowledge distillation aims to obtain. The denoising module model initially constructed is simplified to obtain the model of denoising module 2 that is ultimately used for denoising processing.

[0152] Hold Enhancement is the denoising result output by the student denoising model, that is, its enhanced signal, which serves as a hard label during offline distillation training.

[0153] Offline Distillation Loss is an offline distillation loss function, which can be used to measure the difference between the teacher denoising model and the student denoising model, so as to adjust the network parameters of the student denoising model according to the difference.

[0154] Clean label is the pure Ground-truth label, and Rawloss is the original loss function. The denoising ability of the student denoising model can be measured by the original loss function, so as to adjust the network parameters of the student denoising model according to the ability.

[0155] The above is an example of a response-based distillation method. In other examples, the initial noise reduction module can also be used as a teacher model to perform knowledge distillation in other ways to obtain a student noise reduction model for use as noise reduction module 2. While maintaining the noise reduction performance of noise reduction module 2, its computational complexity can be reduced, so that it can still operate efficiently in a resource-constrained vehicle system, thereby improving the noise reduction efficiency.

[0156] In one example, when training the model of the noise reduction module 2, the training data can be preprocessed, that is, the training samples are first processed by the echo cancellation module 1 and then input into the noise reduction module 2 for model training. This can simulate the processing flow in the actual scene and enhance the model's adaptability to various acoustic interferences, thereby improving the generalization performance of the model.

[0157] Samples of reverberation-type acoustic environment characteristics may also be added to the above training samples, thereby improving the ability of the noise reduction module 2 to remove late reverberation in the vehicle and further improving the noise reduction capability.

[0158] In one embodiment, if Figure 1 As shown, the audio processing device further includes a gain control module 3 .

[0159] The audio processing method may further include:

[0160] The echo cancellation signal or the noise reduction signal obtained by performing noise reduction based on the echo cancellation signal is input into the gain control module 3 for gain processing to obtain an output audio signal.

[0161] Specifically, since the echo cancellation module 1 and the noise reduction module 2 described in the foregoing embodiments can effectively optimize the processing of echoes and noise compared to traditional technologies, the volume level of the audio signal input to the gain control module 3 can be kept similar or the same as the audio signal input to the receiving device when the user speaks, thereby retaining the natural volume perception during the call. Therefore, the gain control module 3 of the present application can multiply the audio signal input therein by the corresponding gain. Since there is no residual or very little echo and noise remaining in the audio signal, the possibility of unnecessary echo or noise amplification is reduced, and it can be ensured that the output audio signal will not be too loud or too soft at this gain. The user can adjust the gain value of the gain processing according to his or her own needs, thereby improving the comfort brought to the user by the output audio signal and improving the reliability of audio processing.

[0162] This application also proposes an electronic device, see Figure 7 , Figure 7 is a schematic diagram of the structure of an embodiment of the electronic device of the present application, such as Figure 7 As shown, the electronic device 30 includes a processor 31 , a memory 32 and a bus 33 .

[0163] The processor 31 and the memory 32 are connected to the bus 33 respectively. The memory 32 stores program instructions. The processor 31 is used to execute the program instructions to implement the audio processing method in the above embodiment.

[0164] In this embodiment, the processor 31 may also be referred to as a CPU (Central Processing Unit). The processor 31 may be an integrated circuit chip having signal processing capabilities. The processor 31 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 31 may also be any conventional processor, etc.

[0165] Different from the prior art, in the technical solution of the present application, a near-end audio signal and a far-end audio signal are obtained, and the near-end audio signal and the far-end audio signal are respectively input into a neural network model so that the neural network model outputs filter parameters in real time, and the filter parameters of the adaptive filter are updated. The far-end audio signal is input into the adaptive filter so that the adaptive filter outputs a predicted linear echo signal, and an echo cancellation signal is obtained based on the near-end audio signal and the predicted linear echo signal. Based on the above method, by setting a neural network model, the near-end audio signal and the far-end audio signal currently received can be processed and analyzed in real time, and the filter parameters that are most suitable for the acoustic environment can be output. Based on the filter parameters generated in real time, the adaptive filter can update its filter parameters in real time to improve its echo cancellation capability for the acoustic environment. That is, based on the setting of the neural network model, the adaptive filter can have a good echo cancellation capability regardless of the acoustic environment it is facing, thereby improving the accuracy of echo cancellation.

[0166] This application also proposes a computer-readable storage medium, see Figure 8 , Figure 8 is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of the present application. Figure 8 As shown, the computer-readable storage medium 40 stores program instructions 41 thereon, and when the program instructions 41 are executed by a processor (not shown), the audio processing method in the above embodiment is implemented.

[0167] The computer-readable storage medium 40 of this embodiment can be, but is not limited to, a USB flash drive, an SD card, a PD optical drive, a mobile hard disk, a large-capacity floppy drive, a flash memory, a multimedia memory card, a server, a storage unit in an FPGA or an ASIC, etc.

[0168] Different from the prior art, in the technical solution of the present application, a near-end audio signal and a far-end audio signal are obtained, and the near-end audio signal and the far-end audio signal are respectively input into a neural network model so that the neural network model outputs filter parameters in real time, and the filter parameters of the adaptive filter are updated. The far-end audio signal is input into the adaptive filter so that the adaptive filter outputs a predicted linear echo signal, and an echo cancellation signal is obtained based on the near-end audio signal and the predicted linear echo signal. Based on the above method, by setting a neural network model, the near-end audio signal and the far-end audio signal currently received can be processed and analyzed in real time, and the filter parameters that are most suitable for the acoustic environment can be output. Based on the filter parameters generated in real time, the adaptive filter can update its filter parameters in real time to improve its echo cancellation capability for the acoustic environment. That is, based on the setting of the neural network model, the adaptive filter can have a good echo cancellation capability regardless of the acoustic environment it is facing, thereby improving the accuracy of echo cancellation.

[0169] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradicting each other.

[0170] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0171] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0172] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (which can be a personal computer, server, network device or other system that can fetch instructions from the instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0173] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An audio processing method, characterized in that: Applied to an audio processing device, the audio processing device includes an echo cancellation module, the echo cancellation module includes a neural network model and an adaptive filter; The audio processing method comprises: Acquire near-end audio signals and far-end audio signals; Inputting the near-end audio signal and the far-end audio signal into the neural network model respectively, so that the neural network model outputs filtering parameters in real time, and updates the filtering parameters of the adaptive filter; Inputting the far-end audio signal into the adaptive filter so that the adaptive filter outputs a predicted linear echo signal; An echo cancellation signal is obtained based on the near-end audio signal and the predicted linear echo signal.

2. The audio processing method according to claim 1, characterized in that: The echo cancellation module also includes a nonlinear echo processing module; The obtaining the echo cancellation signal based on the near-end audio signal and the predicted linear echo signal comprises: Subtracting the predicted linear echo signal from the near-end audio signal to obtain a transition echo cancellation signal; The near-end audio signal, the far-end audio signal and the transition echo cancellation signal are respectively input into the nonlinear echo processing module, so that the nonlinear echo processing module outputs the echo cancellation signal.

3. The audio processing method according to claim 1 or 2, characterized in that: The audio processing method further comprises: Get room impulse response data, clean audio samples, and noisy audio samples; Based on the clean audio sample and the noisy audio sample, obtain a noisy audio sample; Using the first noisy audio sample as a sample of the far-end audio signal; Convolving the far-end audio signal with the room impulse response data to obtain a convolution result, and performing weighted summation of the convolution result and the second noisy frequency sample to obtain a sample of the near-end audio signal; Using the second noisy frequency sample as a sample of the echo cancellation signal; The neural network model is trained based on samples of the far-end audio signal, samples of the near-end audio signal, and samples of the echo cancellation signal.

4. The audio processing method according to claim 3, characterized in that: The training of the neural network model based on the samples of the far-end audio signal, the samples of the near-end audio signal and the samples of the echo cancellation signal comprises: Inputting samples of the far-end audio signal and samples of the near-end audio signal into the neural network model respectively, so that the adaptive filter outputs a to-be-adjusted echo cancellation signal; Obtaining a loss function based on the sample of the echo cancellation signal to be adjusted and the sample of the echo cancellation signal; Based on the loss function and the gradient descent method, the parameters of the neural network model are updated.

5. The audio processing method according to claim 1 or 2, characterized in that: The audio processing device further comprises a noise reduction module, wherein the noise reduction module comprises a feature fusion layer, a feature compression layer, a sub-band division layer, a feature enhancement layer and a mask layer; The audio processing method further comprises: Inputting the echo cancellation signal into the feature fusion layer so that the feature fusion layer outputs a first signal; Inputting the first signal into the feature compression layer so that the feature compression layer outputs a second signal; Inputting the second signal into the sub-band division layer, so that the sub-band division layer outputs a plurality of third signals respectively used to characterize the characteristics of the corresponding sub-bands; wherein two adjacent third signals in the frequency domain have an overlapping area in the frequency domain; Inputting the third signal into the feature enhancement layer so that the feature enhancement layer outputs a fourth signal; The echo cancellation signal and the fourth signal are input into a mask layer so that the mask layer outputs a noise reduction signal.

6. The audio processing method according to claim 5, characterized in that: The feature enhancement layer includes a coding layer, an enhancement layer and a decoding layer; The step of inputting the third signal into the feature enhancement layer so that the feature enhancement layer outputs a fourth signal comprises: Inputting the third signal into the coding layer so that the coding layer outputs a coded signal; inputting the encoded signal into the enhancement layer so that the enhancement layer outputs an enhanced signal; The enhanced signal is input into the decoding layer so that the decoding layer outputs a decoded signal, which is recorded as the fourth signal.

7. The audio processing method according to claim 5, characterized in that: Before inputting the echo cancellation signal into the feature fusion layer, the audio processing method further includes: Use the initial denoising module as the teacher model; The teacher model is subjected to knowledge distillation to obtain a student model, which is referred to as a denoising model.

8. The audio processing method according to claim 1 or 2, characterized in that: The audio processing device also includes a gain control module; The audio processing method further comprises: The echo cancellation signal or the noise reduction signal obtained by performing noise reduction based on the echo cancellation signal is input into the gain control module for gain processing to obtain an output audio signal.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store program instructions, and the processor is used to execute the program instructions to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images