Audio processing method, electronic device, and computer-readable storage medium

Abstract

This application discloses an audio processing method, an electronic device, and a computer-readable storage medium. The audio processing method includes: acquiring a near-end audio signal and a far-end audio signal; inputting the near-end audio signal and the far-end audio signal into a neural network model respectively, so that the neural network model outputs filtering parameters in real time to update the filtering parameters of an adaptive filter; inputting the far-end audio signal and the echo cancellation signal into the adaptive filter respectively, so that the adaptive filter outputs a predicted linear echo signal; and obtaining an echo cancellation signal based on the near-end audio signal and the predicted linear echo signal. Based on the above method, the accuracy of echo cancellation can be improved.

Description

Technical Field

[0001] This application relates to the field of audio technology, and in particular to audio processing methods, electronic devices, and computer-readable storage media. Background Technology

[0002] In existing technologies, within the vehicle's interior, such as when a user is making a call, the radio device may simultaneously receive audio signals emitted by a distant speaker, creating an echo. This results in poor voice call quality for the user using the radio device. Therefore, it is usually necessary to install an echo cancellation module in the audio device within the vehicle's interior to improve the quality of voice calls.

[0003] The drawback of existing technologies is that existing echo cancellation modules are generally unable to adapt to complex and changing acoustic environments. That is, when the sound reception and speaker conditions of the audio signal in the vehicle space change, the echo cancellation effect will also change accordingly, sometimes good and sometimes bad. It is impossible to adapt the relevant parameters of echo cancellation according to different acoustic environments, thus failing to ensure the accuracy of echo cancellation every time, which easily leads to unstable voice call quality. Therefore, how to improve the accuracy of echo cancellation has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0004] The main technical problem addressed in this application is how to improve the accuracy of echo cancellation.

[0005] To address the aforementioned technical problems, the first technical solution adopted in this application is: an audio processing method applied to an audio processing device, the audio processing device including an echo cancellation module, the echo cancellation module including a neural network model and an adaptive filter; the audio processing method includes: acquiring a near-end audio signal and a far-end audio signal; inputting the near-end audio signal and the far-end audio signal into the neural network model respectively, so that the neural network model outputs filtering parameters in real time to update the filtering parameters of the adaptive filter; inputting the far-end audio signal into the adaptive filter, so that the adaptive filter outputs a predicted linear echo signal; and obtaining an echo cancellation signal based on the near-end audio signal and the predicted linear echo signal.

[0006] The echo cancellation module also includes a nonlinear echo processing module; based on the near-end audio signal and the predicted linear echo signal, an echo cancellation signal is obtained, including: subtracting the predicted linear echo signal from the near-end audio signal to obtain a transitional echo cancellation signal; the near-end audio signal, the far-end audio signal and the transitional echo cancellation signal are respectively input into the nonlinear echo processing module so that the nonlinear echo processing module outputs an echo cancellation signal.

[0007] The audio processing method further includes: acquiring room impulse response data, clean audio samples, and noisy audio samples; obtaining noisy frequency samples based on the clean audio samples and noisy audio samples; using the first noisy frequency sample as a sample of the far-end audio signal; convolving the far-end audio signal with the room impulse response data to obtain a convolution result, and then weighting and summing the convolution result with the second noisy frequency sample to obtain a sample of the near-end audio signal; using the second noisy frequency sample as a sample of the echo cancellation signal; and training a neural network model based on the far-end audio signal sample, the near-end audio signal sample, and the echo cancellation signal sample.

[0008] The process involves training a neural network model based on samples of far-end audio signals, near-end audio signals, and echo cancellation signals. This includes: inputting samples of far-end and near-end audio signals into the neural network model to make the adaptive filter output the echo cancellation signal to be adjusted; obtaining a loss function based on the echo cancellation signal to be adjusted and the echo cancellation signal samples; and updating the parameters of the neural network model based on the loss function and the gradient descent method.

[0009] The audio processing device further includes a noise reduction module, which comprises a feature fusion layer, a feature compression layer, a subband partitioning layer, a feature enhancement layer, and a masking layer. The audio processing method includes: inputting an echo cancellation signal into the feature fusion layer to output a first signal; inputting the first signal into the feature compression layer to output a second signal; inputting the second signal into the subband partitioning layer to output multiple third signals, each representing a feature of a corresponding subband; two adjacent third signals in the frequency domain having an overlapping region; inputting the third signals into the feature enhancement layer to output a fourth signal; and inputting the echo cancellation signal and the fourth signal into the masking layer to output a noise reduction signal.

[0010] The feature enhancement layer includes an encoding layer, an enhancement layer, and a decoding layer. Inputting a third signal into the feature enhancement layer to make the feature enhancement layer output a fourth signal includes: inputting the third signal into the encoding layer to make the encoding layer output an encoded signal; inputting the encoded signal into the enhancement layer to make the enhancement layer output an enhanced signal; and inputting the enhanced signal into the decoding layer to make the decoding layer output a decoded signal, denoted as the fourth signal.

[0011] The audio processing method further includes, before inputting the echo cancellation signal into the feature fusion layer, using the initial noise reduction module as the teacher model and performing knowledge distillation on the teacher model to obtain the student model, denoted as the noise reduction model.

[0012] The audio processing device also includes a gain control module; the echo cancellation signal or the noise-reduced signal obtained by noise reduction based on the echo cancellation signal is input into the gain control module for gain processing to obtain the output audio signal.

[0013] To solve the above-mentioned technical problems, the second technical solution adopted in this application is: an electronic device, including: a memory and a processor; the memory is used to store program instructions, and the processor is used to execute the program instructions to implement the above-mentioned method.

[0014] To solve the above-mentioned technical problems, the third technical solution adopted in this application is: a computer-readable storage medium storing program instructions, which implement the above-mentioned method when executed by a processor.

[0015] The beneficial effects of this application are as follows: Unlike existing technologies, the technical solution of this application acquires near-end and far-end audio signals, inputs them separately into a neural network model, and enables the neural network model to output filtering parameters in real time. This updates the filtering parameters of the adaptive filter. The far-end audio signal is then input into the adaptive filter, causing it to output a predicted linear echo signal. Based on the near-end audio signal and the predicted linear echo signal, an echo cancellation signal is obtained. Based on this method, by setting up a neural network model, the currently received near-end and far-end audio signals can be processed and analyzed in real time, outputting filtering parameters most suitable for the acoustic environment. Based on these real-time generated filtering parameters, the adaptive filter can update its filtering parameters in real time to improve its echo cancellation capability for different acoustic environments. In other words, based on the setting of the neural network model, the adaptive filter can possess good echo cancellation capability regardless of the acoustic environment, improving the accuracy of echo cancellation. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the structure of an embodiment of the audio processing device of this application;

[0018] Figure 2 This is a schematic diagram of the structure of an embodiment of the echo cancellation module of this application;

[0019] Figure 3 This is a flowchart illustrating an embodiment of the audio processing method of this application;

[0020] Figure 4 This is a flowchart illustrating another embodiment of the audio processing method of this application;

[0021] Figure 5 This is a schematic diagram of the structure of one embodiment of the noise reduction module of this application;

[0022] Figure 6 This is a schematic flowchart of an embodiment of knowledge distillation according to this application;

[0023] Figure 7 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0024] Figure 8 This is a schematic diagram of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0025] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the application. Similarly, the following embodiments are only some, not all, embodiments of the present application, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present application.

[0026] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0027] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "setting," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or a connection through an intermediate medium. Those skilled in the art will understand the specific meanings of the above terms within the context of this application.

[0028] This application proposes an audio processing method, applied to an audio processing device, see [link to relevant documentation]. Figures 1 to 3 , Figure 1 This is a schematic diagram of the structure of an embodiment of the audio processing device of this application. Figure 2 This is a schematic diagram of the structure of an embodiment of the echo cancellation module of this application. Figure 3 This is a flowchart illustrating an embodiment of the audio processing method of this application.

[0029] like Figure 1 As shown, in one example, located in a vehicle space or other type of space, the near-end audio signal received by the receiving device located near the user can be input into the echo cancellation module 1. The near-end audio signal typically includes the original audio signal generated by the user's direct voice and the echo audio signal generated by the speaker located far from the user. In addition, the far-end audio signal currently received by the speaker located far from the user can also be input into the echo cancellation module 1. The far-end audio signal is typically the audio signal on which the speaker is based when it is speaking.

[0030] The echo cancellation module 1 can cancel the echo of the near-end audio signal based on the received far-end audio signal and near-end audio signal. Then, the echo-cancelled audio signal is input to the noise reduction module 2, which can reduce the noise of the received audio signal. Then, the noise-reduced audio signal is input to the gain control module 3, which can control the gain of the received audio signal so that the volume of the output audio signal output by the gain control module 3 is within a suitable range. The gain value can be greater than 1, less than 1, or even equal to 1, depending on the actual situation.

[0031] Based on the above method, the obtained output audio signal can be successfully echo-cancelled, noise-reduced, and gain-controlled. The final output audio signal can better reproduce the original information of the user's voice and provide a better voice call effect.

[0032] like Figure 2 As shown, the audio processing device includes an echo cancellation module 1, which includes a neural network model 11 and an adaptive filter 12.

[0033] like Figure 3 As shown, the audio processing methods include:

[0034] Step S11: Acquire the near-end audio signal and the far-end audio signal.

[0035] Specifically, the near-end audio signal can be the audio signal generated by the recording device when the user speaks. It can typically include the original audio signal generated by the user speaking directly and the echo audio signal generated by the speaker located at the far end of the user.

[0036] The remote audio signal can specifically be the speaker audio signal input to the speaker device for amplification.

[0037] Step S12: Input the near-end audio signal and the far-end audio signal into the neural network model 11 respectively, so that the neural network model 11 outputs the filtering parameters in real time and updates the filtering parameters of the adaptive filter 12.

[0038] Specifically, the adaptive filter 12 can be a filter that performs adaptive filtering based on the frequency domain block Kalman algorithm, and the above-mentioned filtering parameters can specifically refer to the static parameters alpha and beta in the frequency domain block Kalman algorithm.

[0039] The neural network model 11 can analyze and process the near-end audio signal and the far-end audio signal, and output filtering parameters adapted to the acoustic environment to the adaptive filter 12. The adaptive filter 12 can update its original filtering parameters in real time to the latest filtering parameters output by the neural network model 11, so that the frequency domain block Kalman algorithm within it can be better adapted to the acoustic environment, so as to provide a good linear echo cancellation effect in any acoustic environment.

[0040] Step S13: Input the far-end audio signal into the adaptive filter 12 so that the adaptive filter 12 outputs a predicted linear echo signal.

[0041] The signal output by the adaptive filter 12 is a predicted linear echo signal, which can be obtained and output as an echo cancellation signal by subtracting the predicted linear echo signal from the near-end audio signal.

[0042] The adaptive filter 12 can perform adaptive filtering based on the far-end audio signal to estimate the predicted linear echo signal.

[0043] Step S14: Obtain the echo cancellation signal based on the near-end audio signal and the predicted linear echo signal.

[0044] Among them, the predicted linear echo signal can be subtracted from the near-end audio signal to obtain and output the echo cancellation signal.

[0045] Specifically, in one example, such as Figure 2 As shown, the calculation process of adaptive filter 12 is as follows:

[0046] Initialize the Kalman state parameters P and predicted echo path H of the adaptive filter 12, and give the static parameters alpha and beta of the adaptive filter 12.

[0047] First, perform Fourier transforms on the near-end audio signal and the far-end audio signal respectively to obtain the transformed near-end audio signal Y and the far-end audio signal S:

[0048] Y = STFT(y)

[0049] S = STFT(s)

[0050] Second, based on the predicted echo path H t The first echo-cancelled signal E is obtained, where * represents the cross product or vector multiplication:

[0051] E = SY * H t

[0052] Third, calculate the Kalman gain, where, () T For the conjugate transpose of a vector, conj() is the conjugate of a vector, and P is the conjugate of a vector. t Here are the Kalman parameters, and G is the Kalman gain:

[0053]

[0054] Fourth, update the next Kalman state parameter P. t+1 and the next predicted echo path H t+1 :

[0055] P t+1 =beta*G*Y*P t +H t *(H t ) T

[0056] H t+1 =H t +E*G

[0057] Fifth, based on the above method, the static parameters alpha and beta of the adaptive filter 12 can be continuously updated based on the currently received filtering parameters sent by the neural network model 11, so that the adaptive filter 12 can perform adaptive filtering processing in the state most suitable to the acoustic environment and continuously update the next Kalman state parameter P. t+1 and the next predicted echo path H t+1 The output predicts the linear echo signal and generates a first echo cancellation signal E to eliminate the linear echo. This first echo cancellation signal E can then be used as the echo cancellation signal, or the first echo cancellation signal E can be further processed to obtain a second echo cancellation signal F, which can then be used as the echo cancellation signal. This is not limited here.

[0058] In summary, by setting the neural network model 11, the accuracy of the predicted linear echo signal output by the adaptive filter 12 under different acoustic environments can be effectively improved, thereby improving the accuracy and stability of echo cancellation.

[0059] Unlike existing technologies, the technical solution of this application acquires near-end and far-end audio signals, respectively inputs them into a neural network model, and enables the neural network model to output filtering parameters in real time. This updates the filtering parameters of the adaptive filter. The far-end audio signal is then input into the adaptive filter, causing it to output a predicted linear echo signal. Based on the near-end audio signal and the predicted linear echo signal, an echo cancellation signal is obtained. Based on this method, by setting up a neural network model, the currently received near-end and far-end audio signals can be processed and analyzed in real time, outputting filtering parameters best suited to the acoustic environment. Based on these real-time generated filtering parameters, the adaptive filter can update its filtering parameters in real time to improve its echo cancellation capability for different acoustic environments. In other words, based on the setting of the neural network model, the adaptive filter can have good echo cancellation capability regardless of the acoustic environment, improving the accuracy of echo cancellation.

[0060] In one embodiment, such as Figure 2 As shown, the echo cancellation module 1 also includes a nonlinear echo processing module 13.

[0061] Based on the near-end audio signal and the predicted linear echo signal, the echo cancellation signal is obtained, including:

[0062] The predicted linear echo signal is subtracted from the near-end audio signal to obtain the transition echo cancellation signal.

[0063] The near-end audio signal, the far-end audio signal, and the transition echo cancellation signal are respectively input into the nonlinear echo processing module 13 so that the nonlinear echo processing module 13 outputs the echo cancellation signal.

[0064] Specifically, after the adaptive filter 12 processes the near-end audio signal to achieve linear echo cancellation, a transitional echo cancellation signal can be obtained, which is the first echo cancellation signal E described in the previous embodiment. The nonlinear echo processing module 13 can perform nonlinear echo cancellation on the first echo cancellation signal E to obtain the final echo cancellation signal, which is the second echo cancellation signal F described in the previous embodiment.

[0065] In one example, such as Figure 2 As shown, the calculation process of the nonlinear echo processing module 13 is as follows:

[0066] First, Fourier transforms are performed on the near-end audio signal, the far-end audio signal, and the first echo cancellation signal (transition echo cancellation signal) respectively to obtain the transformed near-end audio signal Y, the far-end audio signal S, and the first echo cancellation signal E, where cohes is the similarity between the first echo cancellation signal and the near-end audio signal, and cohys is the similarity between the first echo cancellation signal and the far-end audio signal.

[0067] E = STFT(e)

[0068] S = STFT(s)

[0069] Y = STFT(y)

[0070]

[0071]

[0072] Second, calculate the nonlinear echo residual mask.

[0073] The smaller the cohes value, the smaller the echo residue in the first echo cancellation signal; the larger the cohes value, the larger the proportion of echo in the near-end audio signal of the current frame. Therefore, the mask values ​​are as follows, where min() is the minimum value operation:

[0074] mask = min(cohes, 1 - cohys)

[0075] Third, the echo residual mask is cross-multiplied with the first echo cancellation signal E to obtain the second echo cancellation signal F. Then, the second echo cancellation signal F is subjected to inverse Fourier transform to obtain the second echo cancellation signal f(n), which is the echo cancellation signal described in the previous embodiment:

[0076] F = mask * E

[0077] f(n) = ISTFT(F)

[0078] This completes the linear and nonlinear echo cancellation of the near-end audio signal, resulting in an echo-cancelled signal.

[0079] Based on the above methods, the echo cancellation operation can be further improved, and the echo residue in the output echo cancellation signal can be reduced or eliminated as much as possible, thereby improving the accuracy of echo cancellation.

[0080] For example, see Figure 4 , Figure 4 This is a flowchart illustrating another embodiment of the audio processing method of this application, as shown below. Figure 4 As shown, the audio processing methods include:

[0081] Step S21: Acquire the near-end audio signal and the far-end audio signal.

[0082] Step S22: Input the near-end audio signal and the far-end audio signal into the neural network model 11 respectively, so that the neural network model 11 outputs the filtering parameters in real time and updates the filtering parameters of the adaptive filter 12.

[0083] Step S23: Input the far-end audio signal into the adaptive filter 12 so that the adaptive filter 12 outputs a predicted linear echo signal.

[0084] Step S24: Obtain the echo cancellation signal based on the near-end audio signal and the predicted linear echo signal.

[0085] The steps S21-S24 above correspond to steps S11-S14 in the previous embodiment, and will not be repeated here.

[0086] Step S25: Based on the noise reduction module, perform noise reduction processing on the echo cancellation signal to obtain a noise-reduced signal.

[0087] Step S26: Based on the gain control module, the noise reduction signal is processed to obtain the output audio signal.

[0088] Based on the above steps S21-S24, S25 and S26, echo cancellation, noise reduction and gain control can be performed on the near-end audio signal in sequence to complete the audio processing of the near-end audio signal. Compared with the near-end audio signal, the output audio signal has reduced or eliminated echo and noise, and its volume is more appropriate, which improves the quality of the output audio signal and improves the user's call experience.

[0089] In one embodiment, the audio processing method further includes:

[0090] Acquire room impulse response data, clean audio samples, and noisy audio samples.

[0091] Based on clean audio samples and noisy audio samples, noisy frequency samples are obtained.

[0092] The first noisy frequency sample is used as the sample of the far-end audio signal.

[0093] The far-end audio signal is convolved with the room impulse response data to obtain the convolution result. The convolution result is then weighted and summed with a second noisy frequency sample to obtain a sample of the near-end audio signal.

[0094] The second noisy frequency sample is used as the sample for the echo cancellation signal.

[0095] The neural network model 11 is trained based on samples of far-end audio signals, near-end audio signals, and echo cancellation signals.

[0096] Specifically, the neural network model 11 may include an encoder, a recurrent neural network (RNN), and a feedforward neural network (FNN) connected in sequence. The Fourier-transformed near-end and far-end audio signals can be input into the encoder, and then processed by the recurrent neural network and the feedforward neural network to obtain the output of the filtering parameters. The recurrent neural network can establish long-distance dependencies, playing a crucial role in recognizing echo environments. The neural network model 11 can also be other types of neural networks, which are not limited here.

[0097] The neural network model 11 is trained based on samples of far-end audio signals, near-end audio signals, and echo-cancellation signals. Specifically, this may include:

[0098] Samples of the far-end audio signal and samples of the near-end audio signal are respectively input into the neural network model 11 so that the adaptive filter 12 outputs the echo cancellation signal to be adjusted.

[0099] The loss function is obtained based on samples of the echo cancellation signal to be adjusted and the echo cancellation signal.

[0100] The parameters of neural network model 11 are updated based on the loss function and gradient descent method.

[0101] In one example, the training process for neural network model 11 is as follows:

[0102] First, construct a dataset of room impulse response (RIR) for the space to be echo-cancelled (such as a vehicle space). The room impulse response can be calculated based on parameters such as the size of the vehicle space, the radio equipment, and the speaker equipment. Alternatively, the room impulse response corresponding to the vehicle space can be obtained from the AEC-challenge database. Other methods can also be used to obtain the room impulse response, which are not limited here.

[0103] Second, construct a dataset with noisy frequency samples. First, construct a dataset of clean audio samples by convolving a single-channel clean audio signal from the DNS-Challenge dataset with the room impulse response. Then, construct a dataset of noisy audio samples by obtaining noisy audio samples from the DNS-Challenge and MUSAN datasets. Finally, sum any clean audio sample and any noisy audio sample using a preset signal-to-noise ratio to obtain a dataset with noisy frequency samples.

[0104] Third, construct samples of the near-end audio signal and the user's voice signal. The user's voice signal sample is the echo-cancellation signal sample, which can be understood as the audio signal that theoretically completely eliminates echoes while retaining only the speech signal portion input from the user's receiver. Two noisy frequency samples can be randomly selected, one as the far-end audio signal sample and the other as the user's voice signal sample. The echo signal is obtained by convolving the far-end audio signal sample with the room impulse response. Then, the echo signal and the user's voice signal are weighted and summed according to a preset signal-to-back ratio to obtain the near-end audio signal sample.

[0105] The neural network model 11 can be trained in groups based on samples of far-end audio signals, near-end audio signals, and user voice signals. For example, the samples of far-end audio signals and near-end audio signals can be Fourier transformed to the time-frequency domain and then input into the neural network model 11. The neural network model 11 outputs corresponding filtering parameters based on the input samples, and updates the filtering parameters of the adaptive filter 12 so that the adaptive filter 12 outputs the echo cancellation signal to be adjusted. The difference between the echo cancellation signal to be adjusted and the user voice signal samples can be reflected by the MSE loss function. Based on this difference, the network parameters of the neural network model 11 are updated using the gradient descent method, and the adaptive filter 12 outputs the echo cancellation signal to be adjusted. This process is repeated until the value of the MSE loss function reaches a preset condition, such as being less than a preset value or within a preset range, thus completing the training of one set of samples. By completing the training of multiple sets of samples, the neural network model 11 can be equipped with the ability to output the filtering parameters that are most suitable for the current acoustic environment for echo cancellation.

[0106] The formula for the MSE loss function is shown below:

[0107]

[0108] Where MSE is the value of the MSE loss function, N is the number of signal sampling points, and x(n) is the user's voice signal. ′ This is to cancel the signal for the first echo.

[0109] Based on the above method, the neural network model 11 can be effectively trained, enabling the neural network model 11 to recognize the characteristics of the current acoustic environment and output the filtering parameters that best fit the current acoustic environment, thereby further improving the accuracy of echo cancellation.

[0110] In one embodiment, see Figure 5 , Figure 5 This is a schematic diagram of the structure of one embodiment of the noise reduction module of this application, as shown below. Figure 5As shown, the audio processing device also includes a noise reduction module 2, which includes a feature fusion layer 21, a feature compression layer 22, a sub-band segmentation layer 23, a feature enhancement layer 24, and a masking layer 25.

[0111] Audio processing methods may also include:

[0112] The echo cancellation signal is input to the feature fusion layer 21 so that the feature fusion layer 21 outputs the first signal.

[0113] The first signal is input into the feature compression layer 22 so that the feature compression layer 22 outputs the second signal.

[0114] The second signal is input to the subband division layer 23, so that the subband division layer 23 outputs multiple third signals, each used to characterize the features of the corresponding subband. Two adjacent third signals in the frequency domain have overlapping regions.

[0115] The third signal is input into the feature enhancement layer 24 so that the feature enhancement layer 24 outputs the fourth signal.

[0116] The echo cancellation signal and the fourth signal are input into the mask layer 25 so that the mask layer 25 outputs the noise reduction signal.

[0117] Specifically, the feature enhancement layer 24 may include an encoding layer 241, an enhancement layer 242, and a decoding layer 243.

[0118] The third signal is input to the feature enhancement layer 24 so that the feature enhancement layer 24 outputs a fourth signal, which may specifically include:

[0119] The third signal is input into the coding layer 241 so that the coding layer 241 outputs the coded signal.

[0120] The encoded signal is input into the enhancement layer 242 so that the enhancement layer 242 outputs the enhancement signal.

[0121] The enhancement signal is input to the decoding layer 243 so that the decoding layer 243 outputs a decoding signal, denoted as the fourth signal.

[0122] In one example, the noise reduction process can be specifically described as follows:

[0123] First, the echo-cancelling signal can be subjected to a Fourier transform to obtain a complex-spectrum echo-cancelling signal, which serves as the input signal for the feature fusion layer. Specifically, this Fourier transform can be a short-time Fourier transform. Assuming the window length of the short-time Fourier transform is 512 and the frame shift is 256, its frequency dimension is 257. After inputting the input signal into the feature fusion layer for feature fusion, and then into the feature compression layer for feature compression, the frequency dimension will be compressed from 257 to 129.

[0124] Second, given the compressed feature *feat*, which is a tensor with shape (B, C, T, 129), where B is the batch size, C is the number of channels, T is the time dimension, and 129 is the compressed frequency dimension, *feat* can be expanded into a feature map according to a local window *k* and a sliding step size *s* to perform sub-band division, resulting in the sub-band feature *feat_sub*, with shape (B, C*k, T, 129).

[0125] Taking k=3 and s=1 as an example, the calculation process for the frequency range of each sub-band is as follows:

[0126] 1. Calculate frequency resolution:

[0127]

[0128] Where Δf is the frequency resolution, f_s is the sampling rate, and N is the number of sampling points. Taking a sampling rate of 16kHz (16000Hz) as an example, Δf = 16000 / 512 = 31.25Hz. It is assumed that after feature compression, the frequency points will still be uniformly distributed within the original frequency range.

[0129] 2. Calculate the frequency range covered by each sub-band:

[0130] Δf sub =Δf*k

[0131] Where, Δf sub For the frequency range covered by a single sub-band, in this example, k is 3, then Δf sub =31.25*3=93.75Hz.

[0132] 3. In the above calculation process, in this example, the frequency range of each sub-band is divided into Δf. sub The specific range of each sub-band also depends on the starting point of the frequency of each sub-band, as shown in the following formula:

[0133] (is)*Δf~(is)*Δf+Δf sub

[0134] In this example, s is 1, and the above is the calculation formula for the frequency range corresponding to the i-th sub-band, for example:

[0135] The frequency range of the first sub-band:

[0136] (1-1)*31.25~(1-1)*31.25+93.75Hz=0~93.75Hz

[0137] As can be seen above, the frequency range of the first sub-band is 0 to 93.75 Hz.

[0138] The frequency range of the second sub-band:

[0139] (2-1)*31.25~(2-1)*31.25+93.75Hz=31.25~125Hz

[0140] As can be seen above, the frequency range of the second sub-band is 31.25 to 125 Hz.

[0141] By analogy, the frequency range of each sub-band can be calculated until all sub-bands can cover all frequency points. The signal corresponding to each sub-band is the third signal mentioned above.

[0142] Based on the above method, when dividing subbands, information within the frequency range of adjacent subbands can be used to enhance the information within the frequency range of the current subband, thereby enhancing the information of each subband. This allows for better noise reduction while retaining useful information based on the information in each subband, improving the reliability of the noise reduction process. Furthermore, some noise may only affect certain parts of the spectrum; subband division can reduce the impact of this noise on the overall processing result. Additionally, by setting overlapping frequency domain subbands, the noise reduction module 2 can focus on fine-grained spectral features, further improving the reliability of the noise reduction process.

[0143] Third, the third signal obtained after sub-band division can be input into the encoding layer for encoding. The encoded signal is then input into the enhancement layer, which is composed of a Dual-Path Recurrent Neural Network (DPRNN), for feature transformation to enhance the features. Finally, the enhanced signal is input into the decoding layer for decoding, outputting the decoded signal, which is also the fourth signal, to the masking layer. It should be noted that both the encoding and decoding layers employ grouped convolutional kernels and grouped transposed convolutional structures. Furthermore, the dual-path recurrent neural network layer uses grouped GRU and grouped Linear structures to further optimize processing efficiency and improve noise reduction efficiency.

[0144] Fourth, the mask layer obtains the echo cancellation signal before the input noise reduction module 2 and the fourth signal obtained by the noise reduction module 2 after processing the echo cancellation signal. The echo cancellation signal and the fourth signal are multiplied to obtain the final output noise reduction signal, thereby achieving noise reduction.

[0145] Optionally, before inputting the echo cancellation signal into the feature fusion layer 21, the audio processing method may further include:

[0146] The initial noise reduction module is used as the teacher model.

[0147] The student model is obtained by knowledge distillation of the teacher model, and is denoted as the noise reduction model.

[0148] Specifically, see Figure 6 , Figure 6 This is a schematic flowchart of an embodiment of the knowledge distillation of this application, as shown below. Figure 6 As shown, the Noisy input is the noisy signal introduced during the knowledge distillation process.

[0149] The Teacher ANS model is a teacher noise reduction model, and the initial noise reduction module model described in the previous embodiment can be used as this teacher noise reduction model.

[0150] Solf Enhancement is the denoising result output by the teacher denoising model, i.e., its enhanced signal, which serves as a soft label during offline distillation training.

[0151] The Student ANS model is the student noise reduction model, which is the model to be obtained in this knowledge distillation. It is a simplified version of the initial noise reduction module model, resulting in the final noise reduction module 2 model used for noise reduction processing.

[0152] Hold Enhancement is the denoising result output by the student denoising model, i.e., its enhanced signal, which serves as a hard label during offline distillation training.

[0153] Offline Distillation Loss is an offline distillation loss function that can be used to measure the difference between the teacher's denoising model and the student's denoising model, so that the network parameters of the student's denoising model can be adjusted according to the difference.

[0154] Clean label is a clean ground-truth label, and Rawloss is the original loss function. The original loss function can be used to measure the denoising ability of the student denoising model, and the network parameters of the student denoising model can be adjusted according to this ability.

[0155] The above is an example of a response-based distillation method. In other examples, the initial noise reduction module can also be used as a teacher model to perform knowledge distillation in other ways to obtain a student noise reduction model, which can be used as noise reduction module 2. This can reduce the computational complexity of noise reduction module 2 while maintaining its noise reduction performance, enabling it to operate efficiently in resource-constrained vehicle systems and improving noise reduction efficiency.

[0156] In one example, when training the model for noise reduction module 2, the training data can be preprocessed. That is, the training samples are first processed by echo cancellation module 1 and then input into noise reduction module 2 for model training. This can simulate the processing flow in the actual scene and enhance the model's adaptability to various acoustic interferences, thereby improving the model's generalization performance.

[0157] Furthermore, samples of reverberation-related acoustic environmental characteristics can be added to the above training samples to improve the ability of noise reduction module 2 to remove late-stage reverberation in the vehicle, thereby further enhancing noise reduction capabilities.

[0158] In one embodiment, such as Figure 1 As shown, the audio processing device also includes a gain control module 3.

[0159] Audio processing methods may also include:

[0160] The echo cancellation signal or the noise-reduced signal obtained by noise reduction based on the echo cancellation signal is input to the gain control module 3 for gain processing to obtain the output audio signal.

[0161] Specifically, since the echo cancellation module 1 and noise reduction module 2 described in the previous embodiments can effectively optimize the processing of echo and noise compared to traditional technologies, the audio signal of the input gain control module 3 can maintain a similar or the same volume level as the audio signal input to the radio device when the user speaks, thus preserving the natural volume perception during the call. Therefore, the gain control module 3 of this application can multiply the input audio signal by a corresponding gain. Since there is no longer any residual or very little echo and noise in the audio signal, the possibility of unnecessary echo or noise amplification is reduced, ensuring that the output audio signal will not be too loud or too soft at this gain. The user can adjust the gain value of the gain processing according to their own needs, improving the comfort of the output audio signal and improving the reliability of audio processing.

[0162] This application also proposes an electronic device, see [link to document]. Figure 7 , Figure 7 This is a schematic diagram of an embodiment of the electronic device of this application, as shown below. Figure 7 As shown, the electronic device 30 includes a processor 31, a memory 32, and a bus 33.

[0163] The processor 31 and the memory 32 are respectively connected to the bus 33. The memory 32 stores program instructions, and the processor 31 is used to execute the program instructions to implement the audio processing method in the above embodiment.

[0164] In this embodiment, processor 31 can also be referred to as CPU (Central Processing Unit). Processor 31 may be an integrated circuit chip with signal processing capabilities. Processor 31 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The general-purpose processor can be a microprocessor, or processor 31 can be any conventional processor.

[0165] Unlike existing technologies, the technical solution of this application acquires near-end and far-end audio signals, respectively inputs them into a neural network model, and enables the neural network model to output filtering parameters in real time. This updates the filtering parameters of the adaptive filter. The far-end audio signal is then input into the adaptive filter, causing it to output a predicted linear echo signal. Based on the near-end audio signal and the predicted linear echo signal, an echo cancellation signal is obtained. Based on this method, by setting up a neural network model, the currently received near-end and far-end audio signals can be processed and analyzed in real time, outputting filtering parameters best suited to the acoustic environment. Based on these real-time generated filtering parameters, the adaptive filter can update its filtering parameters in real time to improve its echo cancellation capability for different acoustic environments. In other words, based on the setting of the neural network model, the adaptive filter can have good echo cancellation capability regardless of the acoustic environment, improving the accuracy of echo cancellation.

[0166] This application also proposes a computer-readable storage medium, see [link to relevant documentation] Figure 8 , Figure 8 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application, as shown below. Figure 8 As shown, the computer-readable storage medium 40 stores program instructions 41 thereon, which, when executed by a processor (not shown), implement the audio processing method in the above embodiments.

[0167] In this embodiment, the computer-readable storage medium 40 may be, but is not limited to, a USB flash drive, SD card, PD optical drive, portable hard drive, high-capacity floppy drive, flash memory, multimedia memory card, storage unit in a server, FPGA, or ASIC, etc.

[0168] Unlike existing technologies, the technical solution of this application acquires near-end and far-end audio signals, respectively inputs them into a neural network model, and enables the neural network model to output filtering parameters in real time. This updates the filtering parameters of the adaptive filter. The far-end audio signal is then input into the adaptive filter, causing it to output a predicted linear echo signal. Based on the near-end audio signal and the predicted linear echo signal, an echo cancellation signal is obtained. Based on this method, by setting up a neural network model, the currently received near-end and far-end audio signals can be processed and analyzed in real time, outputting filtering parameters best suited to the acoustic environment. Based on these real-time generated filtering parameters, the adaptive filter can update its filtering parameters in real time to improve its echo cancellation capability for different acoustic environments. In other words, based on the setting of the neural network model, the adaptive filter can have good echo cancellation capability regardless of the acoustic environment, improving the accuracy of echo cancellation.

[0169] In the description of this application, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0170] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0171] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0172] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (which may be a personal computer, server, network device, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0173] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An audio processing method, characterized in that, An audio processing device is used, the audio processing device including an echo cancellation module, the echo cancellation module including a neural network model and an adaptive filter; The audio processing method includes: Acquire near-end audio signals and far-end audio signals, wherein the near-end audio signals include the original audio signals generated by the user's direct vocalization and the echo audio signals generated by the speaker located at the user's far end; The near-end audio signal and the far-end audio signal are respectively input into the neural network model so that the neural network model outputs filtering parameters in real time to update the filtering parameters of the adaptive filter; The far-end audio signal is input into the adaptive filter so that the adaptive filter outputs a predicted linear echo signal; Based on the near-end audio signal and the predicted linear echo signal, an echo cancellation signal is obtained; The echo cancellation module also includes a nonlinear echo processing module; The process of obtaining the echo cancellation signal based on the near-end audio signal and the predicted linear echo signal includes: Subtracting the predicted linear echo signal from the near-end audio signal yields the transition echo cancellation signal, which is the first echo cancellation signal. The near-end audio signal, the far-end audio signal, and the transition echo cancellation signal are respectively input into the nonlinear echo processing module, so that the nonlinear echo processing module outputs the echo cancellation signal, including: Fourier transforms are performed on the near-end audio signal, the far-end audio signal, and the first echo cancellation signal, respectively, to obtain the transformed near-end audio signal, the transformed far-end audio signal, and the transformed first echo cancellation signal. The nonlinear echo residual mask is calculated based on the similarity between the transformed first echo cancellation signal and the transformed near-end audio signal, and the similarity between the transformed first echo cancellation signal and the transformed far-end audio signal. The nonlinear echo residual mask is cross-multiplied with the transformed first echo cancellation signal to obtain the second echo cancellation signal, and the second echo cancellation signal is subjected to inverse Fourier transform to obtain the echo cancellation signal.

2. The audio processing method according to claim 1, characterized in that, The audio processing method further includes: Acquire room impulse response data, clean audio samples, and noisy audio samples; Based on the clean audio sample and the noisy audio sample, a noisy frequency sample is obtained; The first noisy frequency sample is used as a sample of the far-end audio signal; The far-end audio signal is convolved with the room impulse response data to obtain a convolution result, and the convolution result is weighted and summed with the second noisy frequency sample to obtain a sample of the near-end audio signal; The second noisy frequency sample is used as a sample of the echo cancellation signal; The neural network model is trained based on samples of the far-end audio signal, samples of the near-end audio signal, and samples of the echo cancellation signal.

3. The audio processing method according to claim 2, characterized in that, The neural network model is trained using samples of the far-end audio signal, samples of the near-end audio signal, and samples of the echo cancellation signal, including: The samples of the far-end audio signal and the samples of the near-end audio signal are respectively input into the neural network model so that the adaptive filter outputs the echo cancellation signal to be tuned; Based on the samples of the echo cancellation signal to be adjusted and the echo cancellation signal, the loss function is obtained; The parameters of the neural network model are updated based on the loss function and gradient descent method.

4. The audio processing method according to claim 1, characterized in that, The audio processing device further includes a noise reduction module, which includes a feature fusion layer, a feature compression layer, a sub-band segmentation layer, a feature enhancement layer, and a masking layer. The audio processing method further includes: The echo cancellation signal is input into the feature fusion layer so that the feature fusion layer outputs a first signal; The first signal is input to the feature compression layer so that the feature compression layer outputs the second signal; The second signal is input to the subband division layer so that the subband division layer outputs a plurality of third signals, each used to characterize the features of the corresponding subband; wherein, two adjacent third signals in the frequency domain have an overlapping region in the frequency domain; The third signal is input into the feature enhancement layer so that the feature enhancement layer outputs a fourth signal; The echo cancellation signal and the fourth signal are input into the mask layer so that the mask layer outputs a noise reduction signal.

5. The audio processing method according to claim 4, characterized in that, The feature enhancement layer includes an encoding layer, an enhancement layer, and a decoding layer; The step of inputting the third signal into the feature enhancement layer to cause the feature enhancement layer to output a fourth signal includes: The third signal is input into the coding layer so that the coding layer outputs a coded signal. The encoded signal is input into the enhancement layer so that the enhancement layer outputs an enhanced signal; The enhancement signal is input to the decoding layer so that the decoding layer outputs a decoding signal, denoted as the fourth signal.

6. The audio processing method according to claim 4, characterized in that, Before inputting the echo cancellation signal into the feature fusion layer, the audio processing method further includes: The initial noise reduction module is used as the teacher model; The teacher model is subjected to knowledge distillation to obtain the student model, which is denoted as the noise reduction model.

7. The audio processing method according to claim 1, characterized in that, The audio processing device also includes a gain control module; The audio processing method further includes: The echo cancellation signal or the noise-reduced signal obtained by noise reduction based on the echo cancellation signal is input into the gain control module for gain processing to obtain the output audio signal.

8. An electronic device, characterized in that, include: Memory and processor; The memory is used to store program instructions, and the processor is used to execute the program instructions to implement the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 7.

Images