Echo cancellation method and device
Through the target knowledge distillation end-to-end echo cancellation model, processing proximal and distal speech is generated, and the compression model is solved, which solves the problem of poor echo cancellation performance in extremely low signal-to-return ratio and delay unstable environments, and improves voice interaction performance and robustness sex.
Patent Information
- Application Number
- CN202411307928.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-18
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-18
AI Technical Summary
In the case of extremely low signal-to-return ratio, unstable delay, and variable echo propagation paths in the prior art, the echo cancellation performance is poor, resulting in insufficient voice interaction performance and robustness.
The target knowledge distillation end-to-end echo cancellation model is adopted, and pre-processed by obtaining proximal and distal voices and inputting them into the knowledge distillation end-to-end echo cancellation model, determine the joint loss function, and adjust the model to generate knowledge distillation The model is compressed and the target knowledge distillation model is generated to achieve echo cancellation.
In the case of extremely low signal-to-return ratio, unstable delay, and variable echo propagation paths, acoustic echo cancellation is effectively realized, the voice interaction performance and robustness of terminal devices can be improved, and it can be deployed on the low-computing end side.
Smart Images

Figure CN119091901A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and more particularly to a method and device for echo elimination. Background Art
[0002] Acoustic Echo Cancellation (AEC) is widely used in conference systems, in-car voice platforms, smart speaker interactions, etc. The traditional linear echo cancellation algorithm has strong robustness when the acoustic echo path is linear and time-invariant, and has small nonlinear distortion, corresponding to low sound quality loss. However, when the delay fluctuations of far-end and near-end voices in the time domain are large, the nonlinear distortion of the speaker is significant, the acoustic echo propagation path is variable, and the noise interference is large, the performance of the above traditional linear echo cancellation algorithm is poor.
[0003] In the prior art, two new solutions have been proposed to replace the common linear echo cancellation algorithm. Solution 1 combines signal processing and neural networks. Specifically, linear echo suppression is first completed in the signal processing process, and then nonlinear echo suppression, noise reduction, reverberation, automatic gain control and other functions are completed through neural networks. Solution 2 is to establish an end-to-end multi-task learning network architecture such as DeepVQE, which can simultaneously complete acoustic echo cancellation, noise suppression, automatic gain control and other functions, with the characteristics of low latency, strong operability and simple link. However, when the signal-to-return ratio reaches below -30dB, the voice interaction performance and robustness of the above two solutions are poor.
[0004] In summary, how to better implement acoustic echo cancellation and improve the voice interaction performance and robustness of terminal devices under the conditions of extremely low signal-to-echo ratio, unstable delay, and variable echo propagation path is a problem that needs to be solved at present. Summary of the invention
[0005] In view of this, an embodiment of the present invention provides a method and device for echo cancellation, which adopts a target knowledge distillation end-to-end echo cancellation model to better realize acoustic echo cancellation under conditions of extremely low signal-to-echo ratio, unstable delay, and variable echo propagation path, thereby improving the voice interaction performance and robustness of the terminal device.
[0006] In a first aspect, an embodiment of the present invention provides a method for echo cancellation, the method comprising:
[0007] Acquire a first near-end voice and a first far-end voice;
[0008] Preprocessing the first near-end speech and the first far-end speech to generate a preprocessed second near-end speech and a second far-end speech;
[0009] Inputting the second near-end speech and the second far-end speech into the knowledge distillation end-to-end echo cancellation model to be trained respectively, and determining a joint loss function;
[0010] Adjusting the knowledge distillation end-to-end echo cancellation model to be trained according to the joint loss function to generate a knowledge distillation end-to-end echo cancellation model;
[0011] The knowledge distillation end-to-end echo cancellation model is compressed to generate a target knowledge distillation end-to-end echo cancellation model, wherein the target knowledge distillation end-to-end echo cancellation model is used to implement echo cancellation.
[0012] Optionally, the method further includes:
[0013] The target knowledge distillation end-to-end echo cancellation model is deployed to the terminal side.
[0014] Optionally, preprocessing the first near-end speech and the first far-end speech to generate a preprocessed second near-end speech and a second far-end speech specifically includes:
[0015] The first near-end speech and the first far-end speech are pre-emphasized, power frequency interference is removed, windowed, and short-time Fourier transformed to generate the second near-end speech and the second far-end speech.
[0016] Optionally, the step of inputting the second near-end speech and the second far-end speech into the knowledge distillation end-to-end echo cancellation model to be trained to determine the joint loss function specifically includes:
[0017] Inputting the second near-end speech and the second far-end speech into a teacher branch and a student branch of a knowledge distillation end-to-end echo cancellation model to be trained, respectively, and obtaining training loss functions of the teacher branch and the student branch, respectively;
[0018] A joint loss function is determined according to the training loss function of the teacher branch and the training loss function of the student branch.
[0019] Optionally, the step of inputting the second near-end speech and the second far-end speech into a teacher branch in a knowledge distillation end-to-end echo cancellation model to be trained, and obtaining a training loss function of the teacher branch, specifically includes:
[0020] Extracting features of the second near-end speech and the second far-end speech to obtain features of the second near-end speech and features of the second far-end speech;
[0021] Determining a first delay loss and a first speech enhancement loss of the teacher branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech;
[0022] A training loss function for the teacher branch is determined based on the first delay loss and the first speech enhancement loss.
[0023] Optionally, determining the first delay loss of the teacher branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech specifically includes:
[0024] Inputting the features of the second near-end speech and the features of the second far-end speech into the first time delay detection module of the teacher branch to generate a first predicted near-end speech signal of the first time delay detection module;
[0025] A first delay penalty for the teacher branch is determined according to the first predicted near-end speech signal.
[0026] Optionally, determining the first speech enhancement loss of the teacher branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech specifically includes:
[0027] Inputting the features of the second near-end speech and the features of the second far-end speech into the first time delay detection module of the teacher branch to generate a first predicted near-end speech signal;
[0028] Inputting the first predicted near-end speech signal into a coding layer, a bottleneck layer, a decoding layer, a complex spectrum convolution mapping layer and a data frame filter to generate a first predicted target speech signal;
[0029] A first speech enhancement loss for the teacher branch is determined according to the first predicted target speech signal.
[0030] Optionally, determining the training loss function of the teacher branch according to the first delay loss and the first speech enhancement loss specifically includes:
[0031] determining a first product of the first delay loss and a first weight, and a second product of the first speech enhancement loss and a second weight;
[0032] The sum of the first product and the second product is determined as the training loss function of the teacher branch.
[0033] Optionally, the step of inputting the second near-end speech and the second far-end speech into a student branch in a knowledge distillation end-to-end echo cancellation model to be trained, and obtaining a training loss function of the student branch, specifically includes:
[0034] Extracting features of the second near-end speech and the second far-end speech to obtain features of the second near-end speech and features of the second far-end speech;
[0035] Determining a second delay loss and a second speech enhancement loss of the student branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech;
[0036] A training loss function for losing the student branch is determined according to the second delay loss and the second speech enhancement loss.
[0037] Optionally, determining the second delay loss of the student branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech specifically includes:
[0038] Inputting the features of the second near-end speech and the features of the second far-end speech into the second time delay detection module of the student branch to generate a second predicted near-end speech signal;
[0039] A second delay penalty for the student branch is determined based on the second predicted near-end speech signal.
[0040] Optionally, determining the first speech enhancement loss of the student branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech specifically includes:
[0041] Inputting the features of the second near-end speech and the features of the second far-end speech into the second time delay detection module of the student branch to generate a second predicted near-end speech signal of the second time delay detection module;
[0042] Inputting the second predicted near-end speech signal into a coding layer, a bottleneck layer, a decoding layer, a complex spectrum convolution mapping layer and a data frame filter to generate a second predicted target speech signal;
[0043] A second speech enhancement loss for the student branch is determined according to the second predicted target speech signal.
[0044] Optionally, determining the training loss function of the student branch according to the second delay loss and the second speech enhancement loss specifically includes:
[0045] determining a third product of the second delay loss and a third weight, and a fourth product of the second speech enhancement loss and a fourth weight;
[0046] A sum of the third product and the fourth product is determined as a training loss function of the student branch.
[0047] Optionally, determining a joint loss function according to the training loss function of the teacher branch and the training loss function of the student branch specifically includes:
[0048] Determine the product of the training loss function of the student branch and a setting parameter, wherein the setting parameter represents the KL divergence between the teacher branch and the student branch;
[0049] The sum of the product and the training loss function of the teacher branch is determined as the joint loss function.
[0050] Optionally, compressing the knowledge distillation end-to-end echo cancellation model to generate a target knowledge distillation end-to-end echo cancellation model specifically includes:
[0051] Online quantization of the knowledge distillation end-to-end echo cancellation model is performed to generate a knowledge distillation end-to-end echo cancellation model with a Torch model structure;
[0052] The knowledge distillation end-to-end echo cancellation model of the Torch model structure is converted to ONNX by an open neural network to generate a knowledge distillation end-to-end echo cancellation model of the ONNX model structure;
[0053] The knowledge distillation end-to-end echo cancellation model of the ONNX model structure is passed through the MNN inference engine to generate the target knowledge distillation end-to-end echo cancellation model.
[0054] In a second aspect, an embodiment of the present invention provides a method for echo cancellation, the method comprising:
[0055] Get the signal to be processed;
[0056] The signal to be processed is input into a target knowledge distillation end-to-end echo cancellation model to generate a target signal, wherein the target knowledge distillation end-to-end echo cancellation model is deployed on the terminal side to implement echo cancellation. In a third aspect, an embodiment of the present invention provides an echo cancellation device, the device comprising:
[0057] An acquisition unit, configured to acquire a first near-end voice and a first far-end voice;
[0058] A processing unit, configured to preprocess the first near-end speech and the first far-end speech to generate a preprocessed second near-end speech and a second far-end speech;
[0059] A determination unit, configured to input the second near-end speech and the second far-end speech into the knowledge distillation end-to-end echo cancellation model to be trained, respectively, and determine a joint loss function;
[0060] A generating unit, configured to adjust the knowledge distillation end-to-end echo cancellation model to be trained according to the joint loss function to generate a knowledge distillation end-to-end echo cancellation model;
[0061] The generating unit is further used to perform model compression on the knowledge distillation end-to-end echo cancellation model to generate a target knowledge distillation end-to-end echo cancellation model, wherein the target knowledge distillation end-to-end echo cancellation model is used to implement echo cancellation.
[0062] Optionally, the device further comprises:
[0063] A deployment unit is used to deploy the target knowledge distillation end-to-end echo cancellation model to the terminal side.
[0064] Optionally, the processing unit is specifically used for:
[0065] The first near-end speech and the first far-end speech are pre-emphasized, power frequency interference is removed, windowed, and short-time Fourier transformed to generate the second near-end speech and the second far-end speech.
[0066] Optionally, the determining unit is specifically configured to:
[0067] Inputting the second near-end speech and the second far-end speech into a teacher branch and a student branch of a knowledge distillation end-to-end echo cancellation model to be trained, respectively, and obtaining training loss functions of the teacher branch and the student branch, respectively;
[0068] A joint loss function is determined according to the training loss function of the teacher branch and the training loss function of the student branch.
[0069] Optionally, the determining unit is further configured to:
[0070] Extracting features of the second near-end speech and the second far-end speech to obtain features of the second near-end speech and features of the second far-end speech;
[0071] Determining a first delay loss and a first speech enhancement loss of the teacher branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech;
[0072] A training loss function for the teacher branch is determined based on the first delay loss and the first speech enhancement loss.
[0073] Optionally, the determination unit is further configured to: input the feature of the second near-end speech and the feature of the second far-end speech into the first time delay detection module of the teacher branch to generate a first predicted near-end speech signal of the first time delay detection module;
[0074] A first delay penalty for the teacher branch is determined according to the first predicted near-end speech signal.
[0075] Optionally, the determining unit is further configured to: input the feature of the second near-end speech and the feature of the second far-end speech into the first time delay detection module of the teacher branch to generate a first predicted near-end speech signal;
[0076] Inputting the first predicted near-end speech signal into a coding layer, a bottleneck layer, a decoding layer, a complex spectrum convolution mapping layer and a data frame filter to generate a first predicted target speech signal;
[0077] A first speech enhancement loss for the teacher branch is determined according to the first predicted target speech signal.
[0078] Optionally, the determining unit is further configured to: determine a first product of the first delay loss and a first weight, and a second product of the first speech enhancement loss and a second weight;
[0079] The sum of the first product and the second product is determined as the training loss function of the teacher branch.
[0080] Optionally, the determining unit is further configured to: perform feature extraction on the second near-end speech and the second far-end speech to obtain features of the second near-end speech and features of the second far-end speech;
[0081] Determining a second delay loss and a second speech enhancement loss of the student branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech;
[0082] A training loss function for losing the student branch is determined according to the second delay loss and the second speech enhancement loss.
[0083] Optionally, the determining unit is further configured to: input the feature of the second near-end speech and the feature of the second far-end speech into the second time delay detection module of the student branch to generate a second predicted near-end speech signal;
[0084] A second delay penalty for the student branch is determined based on the second predicted near-end speech signal.
[0085] Optionally, the determining unit is further configured to: input the feature of the second near-end speech and the feature of the second far-end speech into the second time delay detection module of the student branch to generate a second predicted near-end speech signal of the second time delay detection module;
[0086] Inputting the second predicted near-end speech signal into a coding layer, a bottleneck layer, a decoding layer, a complex spectrum convolution mapping layer and a data frame filter to generate a second predicted target speech signal;
[0087] A second speech enhancement loss for the student branch is determined according to the second predicted target speech signal.
[0088] Optionally, the determining unit is further configured to: determine a third product of the second delay loss and a third weight, and a fourth product of the second speech enhancement loss and a fourth weight;
[0089] A sum of the third product and the fourth product is determined as a training loss function of the student branch.
[0090] Optionally, the determining unit is further used to: determine the product of the training loss function of the student branch and a set parameter, wherein the set parameter represents the KL divergence between the teacher branch and the student branch;
[0091] The sum of the product and the training loss function of the teacher branch is determined as the joint loss function.
[0092] Optionally, the generating unit is specifically used for:
[0093] Online quantization of the knowledge distillation end-to-end echo cancellation model is performed to generate a knowledge distillation end-to-end echo cancellation model with a Torch model structure;
[0094] The knowledge distillation end-to-end echo cancellation model of the Torch model structure is converted to ONNX by an open neural network to generate a knowledge distillation end-to-end echo cancellation model of the ONNX model structure;
[0095] The knowledge distillation end-to-end echo cancellation model of the ONNX model structure is passed through the MNN inference engine to generate the target knowledge distillation end-to-end echo cancellation model.
[0096] Optionally, the acquisition unit is further used to: acquire a signal to be processed;
[0097] The generating unit is further used for: inputting the signal to be processed into a target knowledge distillation end-to-end echo cancellation model to generate a target signal.
[0098] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement a method as described in the first aspect, any possibility of the first aspect, or any one of the second aspect.
[0099] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, they implement a method as described in the first aspect, any possibility of the first aspect, or any method as described in the second aspect.
[0100] In an embodiment of the present invention, a first near-end speech and a first far-end speech are obtained; the first near-end speech and the first far-end speech are preprocessed to generate a preprocessed second near-end speech and a second far-end speech; the second near-end speech and the second far-end speech are respectively input into the knowledge distillation end-to-end echo cancellation model to be trained to determine a joint loss function; the knowledge distillation end-to-end echo cancellation model to be trained is adjusted according to the joint loss function to generate a knowledge distillation end-to-end echo cancellation model; the knowledge distillation end-to-end echo cancellation model is compressed to generate a target knowledge distillation end-to-end echo cancellation model, wherein the target knowledge distillation end-to-end echo cancellation model is used to achieve echo cancellation. Through the above method, a target knowledge distillation end-to-end echo cancellation model is generated, and the target knowledge distillation end-to-end echo cancellation model can better achieve acoustic echo cancellation under the conditions of extremely low signal-to-echo ratio, unstable delay, and variable echo propagation path, thereby improving the voice interaction performance and robustness of the terminal device. BRIEF DESCRIPTION OF THE DRAWINGS
[0101] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:
[0102] Figure 1 is a flow chart of a method for echo cancellation in an embodiment of the present invention;
[0103] Figure 2 is a schematic diagram of a knowledge distillation end-to-end echo cancellation model to be trained in an embodiment of the present invention;
[0104] Figure 3 is a flow chart of a method for obtaining a training loss function of the teacher branch in an embodiment of the present invention;
[0105] Figure 4 is a structural diagram of a teacher branch in an embodiment of the present invention;
[0106] Figure 5 is a structural diagram of a training loss function for generating a teacher branch in an embodiment of the present invention;
[0107] Figure 6 is a flow chart of a method for obtaining a training loss function of the student branch in an embodiment of the present invention;
[0108] Figure 7 is a schematic diagram of the structure of a student branch in an embodiment of the present invention;
[0109] Figure 8 is a structural diagram of a training loss function for generating a student branch in an embodiment of the present invention;
[0110] Fig. 9 is a structural diagram of a first TDE module in an embodiment of the present invention;
[0111] Fig.10 is a flow chart of a method for model compression in an embodiment of the present invention;
[0112] Fig.11 is a flow chart of another echo cancellation method in an embodiment of the present invention;
[0113] Fig.12 is a flow chart of another echo cancellation method in an embodiment of the present invention;
[0114] Fig.13 is a schematic diagram of a system structure in an embodiment of the present invention;
[0115] Fig.14 is a schematic diagram of an echo cancellation device in an embodiment of the present invention;
[0116] Fig.15 is a schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION
[0117] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, some specific details are described in detail. It is possible for those skilled in the art to fully understand the present application without the description of these details. In order to avoid confusing the essence of the present application, known methods, processes, flows, components and circuits are not described in detail.
[0118] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.
[0119] Unless the context clearly requires otherwise, the words "include", "comprising" and similar words throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, the meaning is "including but not limited to".
[0120] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.
[0121] In the prior art, in order to solve the problem that the traditional linear acoustic echo cancellation (LAEC) algorithm and the neural network acoustic echo cancellation (NNAEC) algorithm have poor performance in scenarios with large delay fluctuations, significant nonlinear distortion of the speaker, variable acoustic echo propagation paths, and large noise interference, two new solutions are proposed to replace the common linear echo cancellation algorithms, as follows:
[0122] Solution 1: Combination of signal processing and neural network. Specifically, linear echo suppression is first completed in the signal processing process, and then nonlinear echo suppression, noise reduction, dereverberation, automatic gain control and other functions are completed through the neural network; for example, Gated Convolutional FT-LSTM neural network (GFTNN), Band-split Recurrent Neural Network (BSRNN), and Neural Kalman filtering (NKF); wherein the GFTNN and the BSRNN can also be called an integrated end-to-end model of echo cancellation, noise suppression and dereverberation.
[0123] Solution 2: Establish an end-to-end multi-task learning network architecture to simultaneously complete functions such as Acoustic Echo Cancellation (AEC), noise suppression (NS), and Automatic Gain Control (AGC). It has the characteristics of low latency, strong operability, and simple links; for example, the Deep Variational Quantum Eigensolver (DEEPVQE) and a Repairing and Denoising Network for Speech Signal Improvement (RadNET).
[0124] However, in the process of voice interaction in smart speakers, large conference screens, and cars, there are often situations such as playing music, videos, and text-to-speech (TTS) at a high volume, resulting in a signal-to-return ratio of less than -30dB. At this time, the voice interaction performance of both solutions is poor; and the above solution increases the number of delay taps in the frequency domain block adaptive filter to align the far-end voice and the near-end voice, which increases the computational complexity and causes divergence when the delay jitters, resulting in poor robustness; furthermore, the model parameters of the above solution are huge and cannot be implemented on the low-computing power side. Therefore, how to solve the problem of full-duplex in the case of extremely low signal-to-return ratio, unstable delay, and variable echo propagation path, better realize acoustic echo cancellation, improve the voice interaction performance and robustness of terminal equipment, and can be installed and deployed on the low-computing power side, is a problem that needs to be solved at present.
[0125] In an embodiment of the present invention, the signal-to-echo ratio refers to the ratio of the power of the near-end voice signal received from the terminal side to the power of the echo signal received from the speaker, wherein the echo signal is generated by playing the voice of the user on the opposite side of the terminal side through the speaker on the terminal side; the full-duplex is a communication mode in which data can be transmitted simultaneously in two directions, that is, both communicating parties can send and receive data at the same time, and the transmission in these two directions does not interfere with each other. For example, real-time communication (RTC), live broadcast and microphone connection and other scenarios are all full-duplex communications.
[0126] In order to solve the above problem, an echo cancellation method is proposed in the embodiment of the present invention. Figure 1 As shown, the method includes:
[0127] Step S101: Acquire a first near-end voice and a first far-end voice.
[0128] Specifically, the first near-end speech and the first far-end speech are both training data. Before obtaining the first near-end speech and the first far-end speech, it is necessary to construct a simulation data set, and then obtain the first near-end speech and the first far-end speech in the simulation data set.
[0129] In an embodiment of the present invention, the near-end voice and the far-end voice are affected by factors such as sampling clock asynchrony, network delay instability and sound effect algorithm processing. The near-end voice and the far-end voice are not strictly aligned in the time dimension and have certain volatility, wherein the sound effect algorithm processing refers to Dolby sound effects, surround sound effects, etc.
[0130] In a possible implementation, when constructing the simulation data set, data collection is first performed, and the collected data can come from public data sets Librispeech, Aishell, AEC-CHALLENGE and recorded echo data, OPENSIL and 100,000 simulated room impulse responses, and DNS-CHALLENGE and recorded data; wherein the public data sets Librispeech and Aishell are used as clean speech, the AEC-CHALLENGE and recorded echo data are used as far-end speech of synthetic data, the OPENSIL and 100,000 simulated room impulse responses are used as transfer functions, and the noise data comes from DNS-CHALLENGE and recorded data; the collected data are synthesized into 500 hours of training data in a signal-to-return ratio range of [-35,15] and a signal-to-noise ratio range of [-5,15] dB, wherein the number of simulated room impulse responses, signal-to-return ratio range, signal-to-noise ratio range, and training data duration are all determined according to actual conditions, and are only illustrative here.
[0131] In an embodiment of the present invention, while obtaining the first near-end speech and the first far-end speech, the target near-end speech and the clean speech are also obtained, wherein the target near-end speech and the clean speech are used to calculate the joint loss function of the knowledge distillation end-to-end echo cancellation model to be trained.
[0132] Step S102: preprocess the first near-end speech and the first far-end speech to generate a preprocessed second near-end speech and a second far-end speech.
[0133] Specifically, the first near-end speech and the first far-end speech are pre-emphasized, power frequency interference is removed, windowed, and short-time Fourier transform (STFT) is performed on the first near-end speech and the first far-end speech to generate the second near-end speech and the second far-end speech.
[0134] In a possible implementation, the parameters used in the short-time Fourier transform are shown in Table 1, which are as follows:
[0135] Table 1
[0136] parameter Numeric Sampling rate 16000 Window length 32ms(512 points) Frame Shift 8ms(128 points) Fast Fourier Transform (FFT) length 512 points
[0137] The window function used in the short-time Fourier transform is the Hanning window.
[0138] Step S103: input the second near-end speech and the second far-end speech into the knowledge distillation end-to-end echo cancellation model to be trained respectively, and determine a joint loss function.
[0139] In an embodiment of the present invention, the knowledge distillation end-to-end echo cancellation model to be trained includes a teacher model and a student model, which may also be referred to as a teacher branch and a student branch; wherein knowledge distillation is a machine learning technology used to transfer the knowledge in a complex, well-trained teacher model to a smaller, simpler student model, and the knowledge distillation may also be referred to as teacher-student learning (TS-Learning).
[0140] In a possible implementation, the step of inputting the second near-end speech and the second far-end speech into the knowledge distillation end-to-end echo cancellation model to be trained to determine the joint loss function specifically includes: inputting the second near-end speech and the second far-end speech into the teacher branch and the student branch of the knowledge distillation end-to-end echo cancellation model to be trained, respectively, and obtaining the training loss functions of the teacher branch and the student branch respectively; determining the joint loss function according to the training loss function of the teacher branch and the training loss function of the student branch, and performing joint training; specifically as follows Figure 2 As shown, Figure 2 The schematic diagram of the knowledge distillation end-to-end echo cancellation model to be trained includes a teacher branch and a student branch. The first near-end speech and the first far-end speech are preprocessed respectively, and the preprocessing includes pre-emphasis, removal of power frequency interference, windowing processing and STFT. The preprocessed second near-end speech and the second far-end speech are input into the teacher branch and the student branch of the knowledge distillation end-to-end echo cancellation model to be trained respectively. The outputs obtained by the student branch and the teacher branch are respectively subjected to inverse short-time Fourier transform (ISTFT) to generate the training loss function L of the teacher branch. dt And the training loss function L of the student branch kd , determine the joint loss function according to the training loss function of the teacher branch and the training loss function of the student branch.
[0141] The following two specific embodiments are used to respectively describe in detail the training loss functions for obtaining the teacher branch and the student branch, as follows: Specific embodiment 1:
[0143] The second near-end speech and the second far-end speech are respectively input into the teacher branch of the knowledge distillation end-to-end echo cancellation model to be trained, and the training loss function of the teacher branch is obtained, such as Figure 3 As shown, specifically including:
[0144] Step S301: extract features of the second near-end speech and the second far-end speech to obtain features of the second near-end speech and features of the second far-end speech.
[0145] In a possible implementation, feature extraction is performed on the second near-end speech and the second far-end speech through an Equivalent Rectangular Bandwidth (ERB) filter to obtain features of the second near-end speech and features of the second far-end speech.
[0146] Step S302: Determine a first delay loss and a first speech enhancement loss of the teacher branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech.
[0147] Specifically, the features of the second near-end speech and the features of the second far-end speech are input into the first time delay detection (Time Delay Estimation, TDE) module of the teacher branch to generate a first predicted near-end speech signal of the first time delay detection module; the first delay loss of the teacher branch is determined according to the first predicted near-end speech signal. At the same time, the generated first predicted near-end speech signal is input into the encoder layer, the bottleneck layer, the decoder layer, the complex convolving mask block (CCM) layer and the data frame filter (Data Frame filter, DF filter) to generate a first predicted target speech signal; the first speech enhancement loss of the teacher branch is determined according to the first predicted target speech signal.
[0148] In one possible implementation, a first delay loss is determined based on the first predicted near-end speech signal and a pre-acquired target near-end speech; and at the same time, a first speech enhancement loss is determined based on the first predicted target speech signal and a pre-acquired clean speech.
[0149] In the embodiment of the present invention, the structural diagram of the teacher branch is as follows: Figure 4As shown, it is assumed that the second near-end speech Mic and the second far-end speech Ref are input into the first feature extraction module 401 of the teacher branch, and then sequentially input into the first TDE module 402, the first coding layer 403, the first bottleneck layer 404, the first decoding layer 405, the first complex spectrum convolution mapping CCM layer 406 and the first data framework filter 407, wherein the output of the first TDE module is subjected to an inverse short-time Fourier transform to generate a first predicted near-end speech signal; the output of the first data framework filter is subjected to an inverse short-time Fourier transform to generate a first predicted target speech signal; wherein the first coding layer can also be referred to as a first coding module, the first bottleneck layer can also be referred to as a first bottleneck module, the first decoding layer can also be referred to as a first decoding module, and the first complex spectrum convolution mapping CCM layer can also be referred to as a first complex spectrum convolution mapping CCM module.
[0150] In a possible implementation, the network structure of the first coding layer and the first decoding layer refers to the network open source structures such as the Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement (DCCRN) and DEEPVQE for phase-aware speech enhancement; wherein the first coding layer is composed of 4 small coding modules, each of which is composed of 2 convolutions (Conv2d), regularization (Batch Norm), exponential linear units (ELU) and grouped temporal convolution blocks (GT-Conv blocks); wherein the first CCM layer is composed of two stages, by convolving the output features of the first decoding layer with the input second near-end speech Mic, the convolution output is divided into three weight components to construct a complex mask, each component is the weight of a 120° rotation vector in the complex plane, and the enhanced complex domain mask feature is obtained; finally, the output of the first CCM layer is filtered by the first data frame filter DF The filter performs deep convolution filtering, and then performs an inverse short-time Fourier transform (ISTFT) on the output of the first data framework filter to generate a first predicted target speech signal.
[0151] In the embodiment of the present invention, a schematic diagram of determining the training loss function of the teacher branch is as follows: Figure 5 As stated in Figure 4On the basis of the method, two inverse short time Fourier transforms (ISTFT) are added, and a first delay loss (i.e., a first TDE Loss) is determined according to the first predicted near-end speech signal (Est Mic) and the target near-end speech (Target Mic), a first speech enhancement loss (i.e., a first Se Loss) is determined according to the first predicted target speech signal (Est Speech) and the pre-acquired clean speech (Clean Speech), and a schematic diagram of the training loss function of the teacher branch is determined according to the above-mentioned first delay loss and the first speech enhancement loss.
[0152] In a possible implementation, it is assumed that the clean speech is s(n), and the corresponding frequency domain signal is S(k, f); the target near-end speech (Target Mic) of the first TDE module is m(n), and the corresponding frequency domain signal is M(k, f); the first predicted near-end speech signal (Est MIC) of the first TDE module is The corresponding frequency domain signal is The first predicted target speech signal (EST SPEECH) predicted by the teacher branch is The corresponding frequency domain signal is The first delay loss L is determined by using the loss function mean square error (MSE) and scale-invariant source-to-noise ratio (SISNR) in the field of commonly used speech noise reduction and separation. 1 With the first speech enhancement loss L 2 , as follows:
[0153]
[0154]
[0155] Herein, the α can be set to 0.5.
[0156] Step S303: Determine a training loss function of the teacher branch according to the first delay loss and the first speech enhancement loss.
[0157] Specifically, determine the first delay loss L 1 The first product with the first weight, and the first speech enhancement loss L 2 and a second product of a second weight; determining the sum of the first product and the second product as the training loss function L of the teacher branch dt .
[0158] Assume that the first weight is 0.3, the second weight is 0.7, and the training loss function L of the teacher branch is dt The formula is as follows:
[0159] L dt =0.3*L 1 +0.7*L 2 Specific embodiment 2:
[0161] The second near-end speech and the second far-end speech are respectively input into the student branch of the knowledge distillation end-to-end echo cancellation model to be trained, and the training loss function of the student branch is obtained, such as Figure 6 As shown, specifically including:
[0162] Step S601: extract features of the second near-end speech and the second far-end speech to obtain features of the second near-end speech and features of the second far-end speech.
[0163] In a possible implementation, feature extraction is performed on the second near-end speech and the second far-end speech through an Equivalent Rectangular Bandwidth (ERB) filter to obtain features of the second near-end speech and features of the second far-end speech.
[0164] Step S602: Determine a second delay loss and a second speech enhancement loss of the student branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech.
[0165] Specifically, the features of the second near-end speech and the features of the second far-end speech are input into the second time delay detection (Time Delay Estimation, TDE) module of the student branch to generate a second predicted near-end speech signal of the second time delay detection module; the second delay loss of the student branch is determined according to the second predicted near-end speech signal. At the same time, the generated second predicted near-end speech signal is input into the encoder layer, the bottleneck layer, the decoder layer, the complex convolving mask block (CCM) layer and the data frame filter (Data Frame filter, DF filter) to generate a second predicted target speech signal; the second speech enhancement loss of the student branch is determined according to the second predicted target speech signal.
[0166] In one possible implementation, a second delay loss is determined based on the second predicted near-end speech signal and a pre-acquired target near-end speech; and at the same time, a second speech enhancement loss is determined based on the second predicted target speech signal and a pre-acquired clean speech.
[0167] In the embodiment of the present invention, the structural diagram of the student branch is as follows: Figure 7 As shown, it is assumed that the second near-end speech Mic and the second far-end speech Ref are input into the second feature extraction module 701 of the student branch, and then sequentially input into the second TDE module 702, the second coding layer 703, the second bottleneck layer 704, the second decoding layer 705, the second complex spectrum convolution mapping CCM layer 706 and the second data framework filter 707, wherein the output of the second TDE module is subjected to an inverse short-time Fourier transform to generate a second predicted near-end speech signal; the output of the second data framework filter is subjected to an inverse short-time Fourier transform to generate a second predicted target speech signal; wherein the second coding layer can also be called a second coding module, the second bottleneck layer can also be called a second bottleneck module, the second decoding layer can also be called a second decoding module, and the second complex spectrum convolution mapping CCM layer can also be called a second complex spectrum convolution mapping CCM module.
[0168] In a possible implementation, the network structure of the second coding layer and the second decoding layer refers to the network open source structures such as the Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement (DCCRN) and DEEPVQE for phase-aware speech enhancement; wherein the coding layer is composed of 4 small coding modules, each of which is composed of 2 convolutions (Conv2d), regularization (BatchNorm), exponential linear units (ELU) and grouped temporal convolution blocks (GT-Conv blocks); wherein the CCM layer is composed of two stages, by convolving the output features of the decoding layer and the input second near-end speech Mic, the convolution output is divided into three weight components to construct a complex mask, each component is the weight of a 120° rotation vector in the complex plane, and the enhanced complex domain mask feature is obtained; finally, the output of the CCM layer is filtered through the second data frame filter DF The filter performs deep convolution filtering, and then performs an inverse short-time Fourier transform (ISTFT) on the output of the second data framework filter to generate a second predicted target speech signal.
[0169] In the embodiment of the present invention, a schematic diagram of determining the training loss function of the student branch is as follows: Figure 8 As stated in Figure 7 On the basis of the above, two inverse short time Fourier transforms (ISTFT) are added, as well as a schematic diagram of determining the second delay loss (i.e., the second TDE Loss) according to the second predicted near-end speech signal and the target near-end speech, determining the second speech enhancement loss (i.e., the second Se Loss) according to the second predicted target speech signal and the pre-acquired clean speech, and determining the training loss function of the student branch according to the above-mentioned second delay loss and the second speech enhancement loss.
[0170] In a possible implementation, it is assumed that the clean speech is s(n), and the corresponding frequency domain signal is S(k, f); the target near-end speech (Target Mic) of the second TDE module is m(n), and the corresponding frequency domain signal is M(k, f); the second predicted near-end speech signal (Est MIC) of the second TDE module is The corresponding frequency domain signal is The second predicted target speech signal (EST SPEECH) predicted by the teacher branch is The corresponding frequency domain signal is The second delay loss L is determined by using the loss function mean square error (MSE) and scale-invariant source-to-noise ratio (SISNR) in the field of commonly used speech noise reduction and separation. 1 With the second speech enhancement loss L 2 , as follows:
[0171]
[0172]
[0173] Herein, the α can be set to 0.5.
[0174] Step S603: Determine a training loss function for losing the student branch according to the second delay loss and the second speech enhancement loss.
[0175] Specifically, the second delay loss L is determined 1 The third product with the third weight, and the second speech enhancement loss L 2 and a fourth product of a fourth weight; determining the sum of the third product and the fourth product as the training loss function L of the student branch kt .
[0176] Assume that the third weight is 0.3, the fourth weight is 0.7, and the training loss function L of the student branch is kt The formula is as follows:
[0177] L kt =0.3*L 1 +0.7*L 2
[0178] In the embodiment of the present invention, the L of the teacher branch 1 and the student branch of L 1 It can be the same value or different data. The L of the teacher branch 2 and the student branch of L 2 It can be the same value or different data, depending on the actual situation.
[0179] In the embodiment of the present invention, the training loss function L of the teacher branch determined according to the above specific embodiment 1 is dt and the training loss function L of the student branch determined in the second specific embodiment kt , and jointly determine the joint loss function, including:
[0180] Determine the product of the training loss function of the student branch and a set parameter, wherein the set parameter represents the KL (Kullback-Leibler) divergence between the teacher branch and the student branch; and determine the sum of the product and the training loss function of the teacher branch as the joint loss function.
[0181] In a possible implementation, the formula of the joint loss function is as follows:
[0182] L=L dt +γL kd
[0183] Wherein, the γ is a setting parameter, which is determined according to the KL divergence between the teacher branch and the student branch.
[0184] In the embodiment of the present invention, the network structures of the teacher branch (i.e., the teacher model) and the student branch (i.e., the student model) are completely consistent. The teacher model is mainly responsible for training noisy echo data with a delay within a set range, for example, the set range is 500 milliseconds (ms); the student model is jointly trained with the teacher model using the same data during the training process. When the performance of the teacher model is determined to be optimal through objective indicator evaluation data, the decision space of the teacher model is mapped (init) to the decision space of the student model by means of distillation learning, wherein the decision space includes model parameters; the teacher model and The student model is trained and the parameters are updated at the same time, which helps to narrow the performance gap between the teacher model and the student model. When training the teacher model and the student model, data with a larger delay dynamic range and simulation data in the AEC-Challenge can also be used for simultaneous training, which can improve the robustness of the teacher model and the student model under conditions of large delay jitter. Moreover, through the knowledge distillation between the teacher branch and the student branch, the performance of the end-to-end model can be effectively improved. The joint training method will reduce the problem of excessive nonlinear distortion of speech caused by the noise reduction process of the end-to-end model, and has the advantages of low latency and better speech recovery.
[0185] In a possible implementation, the first TDE module and the second TDE module have the same structure. Taking the first TDE module as an example, the specific structure is as follows: Fig. 9 As shown, it includes: determining a feature X of the first near-end speech mic ∈R c×t×f and the feature X of the first remote speech ref ∈R c×t×f , where R represents the data set, c represents the number of channels, t represents the time dimension, and f represents the frequency dimension; the convolutional network (Conv) is used to perform point-to-point convolution processing on the two features at the channel layer, and the feature after processing is Q mic ∈R h×t×f and K ref ∈R h×t×f , where h represents the number of current convolutional layers; then, the Q mic ∈R h×t×f and K ref ∈R h×t×f Processing, K ref ∈R h×t×f Unfold in the time domain, and increase the time length to K_Delay ref ∈R h×t×d×f , where d is the delay fluctuation range, which can be set to 120 frames; at the same time, for Q mic ∈R h×t×fPerform a transpose operation to generate Q mic ∈R h×f×t ; Q mic ∈R h×f×t and K ref ∈R c×t×d×f Perform matrix multiplication (matmul) operation on the frequency dimension; obtain the difference feature K on the implicit time delay dimension ref ∈R h×d×t , the K ref ∈R h×d×t Input to the Softmax layer to calculate the delay probability distribution D∈R d×t , then the Q mic ∈R h×t×f and the D∈R d×t Perform weighted calculation to obtain the proximal alignment mapping feature X mic ∈R h×t×d×f , then perform the Max operation on the delayed channel dimension, and then perform the deconvolution operation to obtain the aligned output X mic_align ∈R c×t×f , which is the output of the first TDE module; the first TDE module adds a multi-layer attention mechanism and convolution operation to align the signal delay and directly outputs X mic_align The signal is used to calculate the training loss function of the teacher branch and has strong robustness.
[0186] In the embodiment of the present invention, the second TDE module has the same structure as the first TDE module, and the training loss function of the student branch is generated according to the above process, which also has strong robustness.
[0187] Step S104: adjusting the knowledge distillation end-to-end echo cancellation model to be trained according to the joint loss function to generate a knowledge distillation end-to-end echo cancellation model.
[0188] Step S105: compress the knowledge distillation end-to-end echo cancellation model to generate a target knowledge distillation end-to-end echo cancellation model, wherein the target knowledge distillation end-to-end echo cancellation model is used to implement echo cancellation.
[0189] Specifically, the model compression process is as follows: Fig.10 As shown, including:
[0190] Step S1001: Online quantization is performed on the knowledge distillation end-to-end echo cancellation model to generate a knowledge distillation end-to-end echo cancellation model with a Torch model structure.
[0191] Specifically, the knowledge distillation end-to-end echo cancellation model is online quantized through QAT, and the weights are quantized to INT_8 or Float_16 to generate a knowledge distillation end-to-end echo cancellation model with a Torch model structure.
[0192] Step S1002: The knowledge distillation end-to-end echo cancellation model of the Torch model structure is converted to ONNX through an open neural network to generate a knowledge distillation end-to-end echo cancellation model of the ONNX model structure.
[0193] Step S1003: The knowledge distillation end-to-end echo cancellation model of the ONNX model structure is passed through the MNN inference engine to generate the target knowledge distillation end-to-end echo cancellation model.
[0194] Through the above embodiment, first, in the data simulation process, the delays of the near-end signal and the far-end signal are randomly set, and then the delay size is predicted by the time delay detection module, and the filling is completed; finally, the echo cancellation training is completed by the target knowledge distillation end-to-end echo cancellation model, and the echo cancellation performance and delay detection performance of the target knowledge distillation end-to-end echo cancellation model are guaranteed to the greatest extent.
[0195] In the embodiment of the present invention, after step S105, the method further includes other steps, specifically: Fig.11 As shown, the details are as follows:
[0196] Step S106: deploy the target knowledge distillation end-to-end echo cancellation model to the terminal side.
[0197] Specifically, the target knowledge distillation end-to-end echo cancellation model is deployed on the Advanced RISC Machines (ARM) or Digital Signal Processing (DSP) platform on the terminal side, and then the entire voice link is integrated through engineering integration to form an end-to-end voice signal streaming processing system. Since the parameters of the target knowledge distillation end-to-end echo cancellation model are relatively small, it can be deployed on the terminal side with low computing power.
[0198] In an embodiment of the present invention, the target knowledge distillation end-to-end echo cancellation model is deployed to the terminal side, which can realize functions such as delay detection, echo cancellation, speech noise reduction, and dereverberation, and complete the processing flow of the speech signal.
[0199] In a possible implementation, after completing step S106 to deploy the target knowledge distillation end-to-end echo cancellation model to the terminal side, the method further includes other steps, specifically, Fig.12 As shown:
[0200] Step S107: Obtain the signal to be processed.
[0201] Wherein, the signal to be processed is a voice signal.
[0202] Step S108: input the signal to be processed into the target knowledge distillation end-to-end echo cancellation model to generate a target signal.
[0203] Specifically, by inputting the signal to be processed into the target knowledge distillation end-to-end echo cancellation model, the target signal can be directly generated from the signal to be processed, thereby improving the delay detection, echo cancellation, voice noise reduction, and dereverberation of the signal to be processed under conditions of extremely low signal-to-back ratio, unstable delay, and variable echo propagation path, thereby improving the voice interaction performance and robustness of the terminal device.
[0204] In the embodiment of the present invention, the system structure of training the target knowledge distillation end-to-end echo cancellation model is briefly described. Fig.13 As shown, it includes: a data collection and simulation data set construction module 1301, a model training module 1302 and a model compression and terminal deployment module 1303. This is only an exemplary description, and the specific system structure is constructed according to the actual situation.
[0205] In an embodiment of the present invention, a device for echo cancellation is provided, such as Fig.14 As shown, it specifically includes: an acquisition unit 1401, a processing unit 1402, a determination unit 1403 and a generation unit 1404;
[0206] Among them, the acquisition unit 1401 is used to acquire a first near-end speech and a first far-end speech; the processing unit 1402 is used to preprocess the first near-end speech and the first far-end speech to generate a preprocessed second near-end speech and a second far-end speech; the determination unit 1403 is used to input the second near-end speech and the second far-end speech into the knowledge distillation end-to-end echo cancellation model to be trained respectively, and determine the joint loss function; the generation unit 1404 is used to adjust the knowledge distillation end-to-end echo cancellation model to be trained according to the joint loss function to generate a knowledge distillation end-to-end echo cancellation model; the generation unit 1404 is also used to perform model compression on the knowledge distillation end-to-end echo cancellation model to generate a target knowledge distillation end-to-end echo cancellation model, wherein the target knowledge distillation end-to-end echo cancellation model is used to achieve echo cancellation.
[0207] Furthermore, the device also includes:
[0208] A deployment unit is used to deploy the target knowledge distillation end-to-end echo cancellation model to the terminal side.
[0209] Furthermore, the processing unit is specifically used for:
[0210] The first near-end speech and the first far-end speech are pre-emphasized, power frequency interference is removed, windowed, and short-time Fourier transformed to generate the second near-end speech and the second far-end speech.
[0211] Furthermore, the determining unit is specifically configured to:
[0212] Inputting the second near-end speech and the second far-end speech into a teacher branch and a student branch of a knowledge distillation end-to-end echo cancellation model to be trained, respectively, and obtaining training loss functions of the teacher branch and the student branch, respectively;
[0213] A joint loss function is determined according to the training loss function of the teacher branch and the training loss function of the student branch.
[0214] Furthermore, the determining unit is further configured to:
[0215] Extracting features of the second near-end speech and the second far-end speech to obtain features of the second near-end speech and features of the second far-end speech;
[0216] Determining a first delay loss and a first speech enhancement loss of the teacher branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech;
[0217] A training loss function for the teacher branch is determined based on the first delay loss and the first speech enhancement loss.
[0218] Furthermore, the determination unit is specifically further used to: input the features of the second near-end speech and the features of the second far-end speech into the first time delay detection module of the teacher branch to generate a first predicted near-end speech signal of the first time delay detection module;
[0219] A first delay penalty for the teacher branch is determined according to the first predicted near-end speech signal.
[0220] Furthermore, the determination unit is specifically further used to: input the features of the second near-end speech and the features of the second far-end speech into the first time delay detection module of the teacher branch to generate a first predicted near-end speech signal;
[0221] Inputting the first predicted near-end speech signal into a coding layer, a bottleneck layer, a decoding layer, a complex spectrum convolution mapping layer and a data frame filter to generate a first predicted target speech signal;
[0222] A first speech enhancement loss for the teacher branch is determined according to the first predicted target speech signal.
[0223] Further, the determining unit is specifically further used to: determine a first product of the first delay loss and a first weight, and a second product of the first speech enhancement loss and a second weight;
[0224] The sum of the first product and the second product is determined as the training loss function of the teacher branch.
[0225] Furthermore, the determining unit is specifically further used to: perform feature extraction on the second near-end speech and the second far-end speech to obtain features of the second near-end speech and features of the second far-end speech;
[0226] Determining a second delay loss and a second speech enhancement loss of the student branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech;
[0227] A training loss function for losing the student branch is determined according to the second delay loss and the second speech enhancement loss.
[0228] Furthermore, the determining unit is specifically further used to: input the features of the second near-end speech and the features of the second far-end speech into the second time delay detection module of the student branch to generate a second predicted near-end speech signal;
[0229] A second delay penalty for the student branch is determined based on the second predicted near-end speech signal.
[0230] Furthermore, the determining unit is specifically further used to: input the features of the second near-end speech and the features of the second far-end speech into the second time delay detection module of the student branch to generate a second predicted near-end speech signal of the second time delay detection module;
[0231] Inputting the second predicted near-end speech signal into a coding layer, a bottleneck layer, a decoding layer, a complex spectrum convolution mapping layer and a data frame filter to generate a second predicted target speech signal;
[0232] A second speech enhancement loss for the student branch is determined according to the second predicted target speech signal.
[0233] Further, the determining unit is specifically further used to: determine a third product of the second delay loss and a third weight, and a fourth product of the second speech enhancement loss and a fourth weight;
[0234] A sum of the third product and the fourth product is determined as a training loss function of the student branch.
[0235] Furthermore, the determination unit is specifically used to: determine the product of the training loss function of the student branch and a set parameter, wherein the set parameter represents the KL divergence between the teacher branch and the student branch;
[0236] The sum of the product and the training loss function of the teacher branch is determined as the joint loss function.
[0237] Furthermore, the generating unit is specifically used for:
[0238] Online quantization of the knowledge distillation end-to-end echo cancellation model is performed to generate a knowledge distillation end-to-end echo cancellation model with a Torch model structure;
[0239] The knowledge distillation end-to-end echo cancellation model of the Torch model structure is converted to ONNX by an open neural network to generate a knowledge distillation end-to-end echo cancellation model of the ONNX model structure;
[0240] The knowledge distillation end-to-end echo cancellation model of the ONNX model structure is passed through the MNN inference engine to generate the target knowledge distillation end-to-end echo cancellation model.
[0241] Furthermore, the acquisition unit is further used to: acquire a signal to be processed;
[0242] The generating unit is further used for: inputting the signal to be processed into the target knowledge distillation end-to-end echo cancellation model to generate a target signal.
[0243] Fig.15 Schematic diagram of the structure of the electronic device in the embodiment of the present invention. Fig.15 As shown, it includes a general computer hardware structure, which at least includes a processor 1501 and a memory 1502. The processor 1501 and the memory 1502 are connected via a bus 1503. The memory 1502 is suitable for storing instructions or programs executable by the processor 1501. The processor 1501 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 1501 executes the instructions stored in the memory 1502, thereby executing the method flow of the embodiment of the present invention as described above to realize the processing of data and the control of other devices. The bus 1503 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 1504 and the display device and the input / output (I / O) device 1505. The input / output (I / O) device 1505 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 1505 is connected to the system through an input / output (I / O) controller 1506.
[0244] Among them, the instructions stored in the memory 1502 are executed by at least one processor 1501 to achieve: obtaining a first near-end speech and a first far-end speech; preprocessing the first near-end speech and the first far-end speech to generate a preprocessed second near-end speech and a second far-end speech; inputting the second near-end speech and the second far-end speech into the knowledge distillation end-to-end echo cancellation model to be trained respectively, and determining a joint loss function; adjusting the knowledge distillation end-to-end echo cancellation model to be trained according to the joint loss function to generate a knowledge distillation end-to-end echo cancellation model; compressing the knowledge distillation end-to-end echo cancellation model to generate a target knowledge distillation end-to-end echo cancellation model, wherein the target knowledge distillation end-to-end echo cancellation model is used to achieve echo cancellation.
[0245] Specifically, the electronic device includes: one or more processors 1501 and a memory 1502, Fig.15 Take a processor 1501 as an example. The processor 1501 and the memory 1502 may be connected via a bus or other means. Fig.15 The example of the bus connection is taken as an example. The memory 1502 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The processor 1501 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions and modules stored in the memory 1502, that is, the above-mentioned method for determining echo cancellation is implemented.
[0246] The memory 1502 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store a list of options, etc. In addition, the memory 1502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1502 may optionally include a memory remotely arranged relative to the processor 1501, and these remote memories may be connected to an external device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0247] One or more modules are stored in the memory 1502, and when executed by one or more processors 1501, the echo cancellation method in any of the above method embodiments is executed.
[0248] As will be appreciated by those skilled in the art, various aspects of the embodiments of the present invention may be implemented as a system, method or computer program product. Therefore, various aspects of the embodiments of the present invention may take the form of a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.), or an implementation that combines software aspects with hardware aspects, which may generally be referred to herein as a "circuit," "module," or "system." In addition, various aspects of the embodiments of the present invention may take the form of a computer program product implemented in one or more computer-readable media having a computer-readable program code implemented thereon.
[0249] Any combination of one or more computer-readable media can be utilized. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, (but not limited to) electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or apparatuses, or any suitable combination of the foregoing. More specific examples (non-exhaustive enumeration) of computer-readable storage media will include the following: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of an embodiment of the present invention, a computer-readable storage medium can be any tangible medium that can contain or store a program used by an instruction execution system, device or apparatus or a program used in conjunction with an instruction execution system, device or apparatus.
[0250] A computer readable signal medium may include a propagated digital signal having a computer readable program code implemented therein, such as in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any of the following computer readable media: not a computer readable storage medium, and may communicate, propagate, or transmit a program used by or in conjunction with an instruction execution system, device, or apparatus.
[0251] Program code embodied on a computer readable medium may be transmitted using any appropriate medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0252] The computer program code for performing operations for various aspects of the embodiments of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++, etc., and conventional process programming languages such as "C" programming language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer as a stand-alone software package; partially on the user's computer and partially on a remote computer; or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., by using the Internet of an Internet service provider).
[0253] The flowchart legend and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present invention described above describe various aspects of the embodiment of the present invention.It will be understood that each block of the flowchart legend and / or block diagram and the combination of the blocks in the flowchart legend and / or block diagram can be implemented by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that the instructions (executed by the processor of the computer or other programmable data processing device) create a device for implementing the function / action specified in the flowchart and / or block diagram block or block.
[0254] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing device, or other apparatus to operate in a particular manner, so that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions / actions specified in the flowchart and / or block diagram blocks or blocks.
[0255] The computer program instructions may also be loaded onto a computer, other programmable data processing device or other apparatus so that a series of operable steps are performed on the computer, other programmable device or other apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the functions / actions specified in the flowchart and / or block diagram block or blocks.
[0256] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
[0257] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portals for users to choose to authorize or refuse. Users' refusal to process personal information other than the necessary information required for basic functions will not affect the user's use of basic functions.
[0258] The embodiment of the present invention discloses A1, a method for echo cancellation, the method comprising:
[0259] Acquire a first near-end voice and a first far-end voice;
[0260] Preprocessing the first near-end speech and the first far-end speech to generate a preprocessed second near-end speech and a second far-end speech;
[0261] Inputting the second near-end speech and the second far-end speech into the knowledge distillation end-to-end echo cancellation model to be trained respectively, and determining a joint loss function;
[0262] Adjusting the knowledge distillation end-to-end echo cancellation model to be trained according to the joint loss function to generate a knowledge distillation end-to-end echo cancellation model;
[0263] The knowledge distillation end-to-end echo cancellation model is compressed to generate a target knowledge distillation end-to-end echo cancellation model, wherein the target knowledge distillation end-to-end echo cancellation model is used to implement echo cancellation.
[0264] A2. The method according to A1, further comprising:
[0265] The target knowledge distillation end-to-end echo cancellation model is deployed to the terminal side.
[0266] A3. According to the method described in A1, preprocessing the first near-end speech and the first far-end speech to generate the preprocessed second near-end speech and the second far-end speech specifically includes:
[0267] The first near-end speech and the first far-end speech are pre-emphasized, power frequency interference is removed, windowed, and short-time Fourier transformed to generate the second near-end speech and the second far-end speech.
[0268] A4. According to the method described in A1, the step of inputting the second near-end speech and the second far-end speech into the knowledge distillation end-to-end echo cancellation model to be trained to determine the joint loss function specifically includes:
[0269] Inputting the second near-end speech and the second far-end speech into a teacher branch and a student branch of a knowledge distillation end-to-end echo cancellation model to be trained, respectively, and obtaining training loss functions of the teacher branch and the student branch, respectively;
[0270] A joint loss function is determined according to the training loss function of the teacher branch and the training loss function of the student branch.
[0271] A5. The method according to A4, wherein the second near-end speech and the second far-end speech are respectively input into a teacher branch in the knowledge distillation end-to-end echo cancellation model to be trained, and the training loss function of the teacher branch is obtained, specifically comprising:
[0272] Extracting features of the second near-end speech and the second far-end speech to obtain features of the second near-end speech and features of the second far-end speech;
[0273] Determining a first delay loss and a first speech enhancement loss of the teacher branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech;
[0274] A training loss function for the teacher branch is determined based on the first delay loss and the first speech enhancement loss.
[0275] A6. The method according to A5, wherein determining the first delay loss of the teacher branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech specifically comprises:
[0276] Inputting the features of the second near-end speech and the features of the second far-end speech into the first time delay detection module of the teacher branch to generate a first predicted near-end speech signal of the first time delay detection module;
[0277] A first delay penalty for the teacher branch is determined according to the first predicted near-end speech signal.
[0278] A7. The method according to A5, wherein determining the first speech enhancement loss of the teacher branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech, specifically comprises:
[0279] Inputting the features of the second near-end speech and the features of the second far-end speech into the first time delay detection module of the teacher branch to generate a first predicted near-end speech signal;
[0280] Inputting the first predicted near-end speech signal into a coding layer, a bottleneck layer, a decoding layer, a complex spectrum convolution mapping layer and a data frame filter to generate a first predicted target speech signal;
[0281] A first speech enhancement loss for the teacher branch is determined according to the first predicted target speech signal.
[0282] A8. The method according to A5, wherein determining the training loss function of the teacher branch according to the first delay loss and the first speech enhancement loss specifically comprises:
[0283] determining a first product of the first delay loss and a first weight, and a second product of the first speech enhancement loss and a second weight;
[0284] The sum of the first product and the second product is determined as the training loss function of the teacher branch.
[0285] A9. The method according to A4, wherein the second near-end speech and the second far-end speech are respectively input into a student branch of a knowledge distillation end-to-end echo cancellation model to be trained, and a training loss function of the student branch is obtained, specifically comprising:
[0286] Extracting features of the second near-end speech and the second far-end speech to obtain features of the second near-end speech and features of the second far-end speech;
[0287] Determining a second delay loss and a second speech enhancement loss of the student branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech;
[0288] A training loss function for losing the student branch is determined according to the second delay loss and the second speech enhancement loss.
[0289] A10. The method according to A9, wherein determining the second delay loss of the student branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech specifically comprises:
[0290] Inputting the features of the second near-end speech and the features of the second far-end speech into the second time delay detection module of the student branch to generate a second predicted near-end speech signal;
[0291] A second delay penalty for the student branch is determined based on the second predicted near-end speech signal.
[0292] A11. The method according to A9, wherein determining the first speech enhancement loss of the student branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech, specifically comprises:
[0293] Inputting the features of the second near-end speech and the features of the second far-end speech into the second time delay detection module of the student branch to generate a second predicted near-end speech signal of the second time delay detection module;
[0294] Inputting the second predicted near-end speech signal into a coding layer, a bottleneck layer, a decoding layer, a complex spectrum convolution mapping layer and a data frame filter to generate a second predicted target speech signal;
[0295] A second speech enhancement loss for the student branch is determined according to the second predicted target speech signal.
[0296] A12. The method according to A9, wherein determining the training loss function of the student branch according to the second delay loss and the second speech enhancement loss specifically comprises:
[0297] determining a third product of the second delay loss and a third weight, and a fourth product of the second speech enhancement loss and a fourth weight;
[0298] A sum of the third product and the fourth product is determined as a training loss function of the student branch.
[0299] A13. According to the method described in A4, determining the joint loss function based on the training loss function of the teacher branch and the training loss function of the student branch specifically includes:
[0300] Determine the product of the training loss function of the student branch and a setting parameter, wherein the setting parameter represents the KL divergence between the teacher branch and the student branch;
[0301] The sum of the product and the training loss function of the teacher branch is determined as the joint loss function.
[0302] A14. According to the method described in A1, compressing the knowledge distillation end-to-end echo cancellation model to generate a target knowledge distillation end-to-end echo cancellation model specifically includes:
[0303] Online quantization of the knowledge distillation end-to-end echo cancellation model is performed to generate a knowledge distillation end-to-end echo cancellation model with a Torch model structure;
[0304] The knowledge distillation end-to-end echo cancellation model of the Torch model structure is converted to ONNX by an open neural network to generate a knowledge distillation end-to-end echo cancellation model of the ONNX model structure;
[0305] The knowledge distillation end-to-end echo cancellation model of the ONNX model structure is passed through the MNN inference engine to generate the target knowledge distillation end-to-end echo cancellation model.
[0306] The embodiment of the present invention discloses B1, a method for echo cancellation, the method comprising:
[0307] Get the signal to be processed;
[0308] The signal to be processed is input into a target knowledge distillation end-to-end echo cancellation model to generate a target signal, wherein the target knowledge distillation end-to-end echo cancellation model is deployed on the terminal side to realize echo cancellation.
[0309] The embodiment of the present invention discloses C1, a device for echo cancellation, the device comprising:
[0310] An acquisition unit, configured to acquire a first near-end voice and a first far-end voice;
[0311] A processing unit, configured to preprocess the first near-end speech and the first far-end speech to generate a preprocessed second near-end speech and a second far-end speech;
[0312] A determination unit, configured to input the second near-end speech and the second far-end speech into the knowledge distillation end-to-end echo cancellation model to be trained, respectively, and determine a joint loss function;
[0313] A generating unit, configured to adjust the knowledge distillation end-to-end echo cancellation model to be trained according to the joint loss function to generate a knowledge distillation end-to-end echo cancellation model;
[0314] The generating unit is further used to perform model compression on the knowledge distillation end-to-end echo cancellation model to generate a target knowledge distillation end-to-end echo cancellation model, wherein the target knowledge distillation end-to-end echo cancellation model is used to implement echo cancellation.
[0315] An embodiment of the present invention discloses D1, an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement a method as described in any one of A1-A14 and B1.
[0316] The embodiment of the present invention discloses E1, a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of A1-A14 and B1 is implemented.
Claims
1. A method for echo cancellation, characterized in that: The method comprises: Acquire a first near-end voice and a first far-end voice; Preprocessing the first near-end speech and the first far-end speech to generate a preprocessed second near-end speech and a second far-end speech; Inputting the second near-end speech and the second far-end speech into the knowledge distillation end-to-end echo cancellation model to be trained respectively, and determining a joint loss function; Adjusting the knowledge distillation end-to-end echo cancellation model to be trained according to the joint loss function to generate a knowledge distillation end-to-end echo cancellation model; The knowledge distillation end-to-end echo cancellation model is compressed to generate a target knowledge distillation end-to-end echo cancellation model, wherein the target knowledge distillation end-to-end echo cancellation model is used to implement echo cancellation.
2. The method according to claim 1, characterized in that The method further comprises: The target knowledge distillation end-to-end echo cancellation model is deployed to the terminal side.
3. The method according to claim 1, characterized in that The preprocessing of the first near-end speech and the first far-end speech to generate the preprocessed second near-end speech and the second far-end speech specifically includes: The first near-end speech and the first far-end speech are pre-emphasized, power frequency interference is removed, windowed, and short-time Fourier transformed to generate the second near-end speech and the second far-end speech.
4. The method according to claim 1, characterized in that: The step of inputting the second near-end speech and the second far-end speech into the knowledge distillation end-to-end echo cancellation model to be trained to determine the joint loss function specifically includes: Inputting the second near-end speech and the second far-end speech into a teacher branch and a student branch of a knowledge distillation end-to-end echo cancellation model to be trained, respectively, and obtaining training loss functions of the teacher branch and the student branch, respectively; A joint loss function is determined according to the training loss function of the teacher branch and the training loss function of the student branch.
5. The method according to claim 4, characterized in that The step of inputting the second near-end speech and the second far-end speech into a teacher branch in the knowledge distillation end-to-end echo cancellation model to be trained, and obtaining a training loss function of the teacher branch, specifically includes: Extracting features of the second near-end speech and the second far-end speech to obtain features of the second near-end speech and features of the second far-end speech; Determining a first delay loss and a first speech enhancement loss of the teacher branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech; A training loss function for the teacher branch is determined based on the first delay loss and the first speech enhancement loss.
6. The method according to claim 5, characterized in that The determining the first delay loss of the teacher branch according to the characteristics of the second near-end speech and the characteristics of the second far-end speech specifically includes: Inputting the features of the second near-end speech and the features of the second far-end speech into the first time delay detection module of the teacher branch to generate a first predicted near-end speech signal of the first time delay detection module; A first delay penalty for the teacher branch is determined according to the first predicted near-end speech signal.
7. A method for echo cancellation, characterized in that: The method comprises: Get the signal to be processed; The signal to be processed is input into a target knowledge distillation end-to-end echo cancellation model to generate a target signal, wherein the target knowledge distillation end-to-end echo cancellation model is deployed on the terminal side to realize echo cancellation.
8. An echo cancellation device, characterized in that: The device comprises: An acquisition unit, configured to acquire a first near-end voice and a first far-end voice; A processing unit, configured to preprocess the first near-end speech and the first far-end speech to generate a preprocessed second near-end speech and a second far-end speech; A determination unit, configured to input the second near-end speech and the second far-end speech into the knowledge distillation end-to-end echo cancellation model to be trained, respectively, and determine a joint loss function; A generating unit, configured to adjust the knowledge distillation end-to-end echo cancellation model to be trained according to the joint loss function to generate a knowledge distillation end-to-end echo cancellation model; The generating unit is further used to perform model compression on the knowledge distillation end-to-end echo cancellation model to generate a target knowledge distillation end-to-end echo cancellation model, wherein the target knowledge distillation end-to-end echo cancellation model is used to implement echo cancellation.
9. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer readable storage The medium stores a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Model training method, echo cancellation method, system and device and storage medium
CN114530160A
Server, terminal equipment and model compression method
CN117892778A
Nonlinear echo cancellation method and system for telephone communication
CN118381854A
Acoustic echo cancellation methods and systems
GB202101965D0
Cited By
Method, device and equipment for echo cancellation and noise reduction in panoramic sound scene
CN120526788A
Hybrid echo cancellation method and system based on depth filtering and frequency band weighting loss
CN121687087A
