Echo cancellation model training method, echo cancellation method, equipment, medium and product

Abstract

The invention discloses an echo cancellation model training method, an echo cancellation method, equipment, a medium and a product, and relates to the technical field of machine learning, and the method comprises the steps: obtaining training data, a to-be-trained echo cancellation model and a pre-constructed dynamic selection network, the dynamic selection network comprises a fusion model and a plurality of pre-trained teacher models, the fusion model is used for fusing the output of each teacher model, and each teacher model at least comprises an echo cancellation task model and a voice retention task model; inputting the training data into an echo cancellation model to obtain a student echo cancellation result, and inputting the training data into a dynamic selection network to obtain a fusion echo cancellation result; calculating knowledge distillation loss based on the student echo cancellation result and the fusion echo cancellation result; and optimizing the echo cancellation model based on the knowledge distillation loss. According to the invention, the echo cancellation model can give consideration to both echo cancellation and human voice retention.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to an echo cancellation model training method, an echo cancellation method, a device, a medium and a product. Background Art

[0002] When a local terminal is talking to a remote terminal, an acoustic echo is usually generated. That is, the sound played by the speaker is picked up by the microphone and sent back to the remote terminal, so that the user of the remote terminal hears his or her own echo signal.

[0003] Echo cancellation models based on deep learning can be used for echo cancellation. However, these models have high parameter count and computational complexity, and are highly dependent on training data. In actual scenarios, in order to improve the applicability of the model, especially to enable the model to be deployed for echo cancellation on devices with low computing resources and low storage resources, knowledge distillation is usually used to reduce the size of the model while maintaining performance.

[0004] However, traditional knowledge distillation methods usually use a single teacher model to guide the student model, which is unable to fully capture and transfer diverse knowledge. This is particularly unfavorable for maintaining the robustness of the model in complex acoustic environments, resulting in the student echo cancellation model obtained by distillation having a single echo cancellation performance and difficulty in balancing echo cancellation and voice retention. Summary of the invention

[0005] The main purpose of this application is to provide an echo cancellation model training method, an echo cancellation method, a device, a medium and a product, aiming to solve the technical problem that the current echo cancellation model is difficult to balance echo cancellation and human voice preservation.

[0006] To achieve the above object, the present application provides an echo cancellation model training method, the echo cancellation model training method comprising the following steps:

[0007] Acquire training data, an echo cancellation model to be trained, and a pre-built dynamic selection network, wherein the dynamic selection network includes a fusion model and a plurality of pre-trained teacher models, the fusion model is used to fuse the outputs of the teacher models, and each teacher model includes at least an echo cancellation task model and a voice retention task model;

[0008] Inputting the training data into the echo cancellation model to obtain a student echo cancellation result, and inputting the training data into the dynamic selection network to obtain a fusion echo cancellation result;

[0009] Calculating knowledge distillation loss based on the student echo cancellation result and the fusion echo cancellation result;

[0010] The echo cancellation model is optimized based on the knowledge distillation loss, and the step of inputting the training data into the echo cancellation model to obtain the student echo cancellation result and inputting the training data into the dynamic selection model to obtain the fused echo cancellation result is returned to execute, until the preset training end condition is met, and the trained echo cancellation model is obtained.

[0011] In one embodiment, the step of calculating the knowledge distillation loss based on the student echo cancellation result and the fusion echo cancellation result includes:

[0012] Inputting the student echo cancellation result into a preset self-supervised learning model to obtain the student deep features, and inputting the fused echo cancellation result into the self-supervised learning model to obtain the teacher deep features, wherein the self-supervised learning model includes a large-scale self-supervised pre-trained WavLM model for full-stack speech processing;

[0013] The student deep features and the teacher deep features are input into a first preset loss function to obtain the knowledge distillation loss.

[0014] In one embodiment, after the step of calculating the knowledge distillation loss based on the student echo cancellation result and the fusion echo cancellation result, the method further includes:

[0015] Obtaining the current iteration number of the echo cancellation model;

[0016] If the current number of iterations meets a preset condition, freezing the model parameters of the fusion model, and executing the step of optimizing the echo cancellation model based on the knowledge distillation loss;

[0017] If the current number of iterations does not meet a preset condition, the model parameters of the echo cancellation model are frozen, and the model parameters of the fusion model are optimized.

[0018] In one embodiment, the step of optimizing the model parameters of the fusion model includes:

[0019] Obtaining a quadratic gradient of a second preset loss function, and optimizing the fusion model based on the quadratic gradient;

[0020] Among them, the second preset loss function is a loss function used when calculating the hard label loss of the echo cancellation model, and the hard label loss is a loss value between the student echo cancellation result and the training label corresponding to the training data.

[0021] In one embodiment, the preset condition is that the current iteration number is an odd number.

[0022] In one embodiment, the fusion model includes an encoder, a self-attention network, a deep learning Mamba network based on sequence modeling, a decoder and an output layer connected in sequence.

[0023] In addition, to achieve the above object, the present application also provides an echo cancellation method, which includes the following steps:

[0024] Acquire an original signal to be echo-cancelled and a target echo-cancellation model, and input the original signal into the target echo-cancellation model to obtain an echo-cancellation result;

[0025] The target echo cancellation model is an echo cancellation model trained by the echo cancellation model training method as described above.

[0026] In addition, to achieve the above-mentioned purpose, the present application also provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the echo cancellation model training method and / or the echo cancellation method as described above.

[0027] In addition, to achieve the above-mentioned purpose, the present application also provides a readable storage medium, which is a computer-readable storage medium, and a computer program is stored on the computer-readable storage medium. The computer program is executed by a processor to implement the steps of the echo cancellation model training method and / or the echo cancellation method as described above.

[0028] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned echo cancellation model training method and / or echo cancellation method.

[0029] One or more technical solutions proposed in this application have at least the following technical effects:

[0030] Acquire training data, an echo cancellation model to be trained, and a pre-constructed dynamic selection network, wherein the dynamic selection network includes a fusion model and multiple pre-trained teacher models, the fusion model is used to fuse the outputs of each of the teacher models, and each of the teacher models includes at least an echo cancellation task model and a voice retention task model; input the training data into the echo cancellation model to obtain a student echo cancellation result, and input the training data into the dynamic selection network to obtain a fused echo cancellation result; calculate the knowledge distillation loss based on the student echo cancellation result and the fused echo cancellation result; optimize the echo cancellation model based on the knowledge distillation loss, return to execute the steps of inputting the training data into the echo cancellation model to obtain a student echo cancellation result, and inputting the training data into the dynamic selection model to obtain a fused echo cancellation result, until the preset training end condition is met, and obtain the trained echo cancellation model. In this way, the embodiment of the present application fuses the output results of multiple pre-trained teacher models through a fusion model, calculates the knowledge distillation loss through the fused echo elimination result obtained after fusion and the student elimination result output by the echo cancellation model, and optimizes the echo cancellation model based on this knowledge distillation loss. At this time, the echo cancellation model also indicates the student model in the distillation process, so as to achieve the purpose of multiple teacher models guiding the student model, instead of using a single teacher model to guide the student model, so that the student model can fully capture and learn the diverse knowledge of multiple teacher models, and because the teacher model includes at least a voice preservation task model and an echo cancellation task model, the student model can fully capture and learn the teacher model's knowledge in voice preservation and echo cancellation, so that the student model can take into account both echo cancellation and voice preservation, that is, the echo cancellation model can take into account both echo cancellation and voice preservation. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0033] Figure 1 This is a flow chart of the first embodiment of the echo cancellation model training method of the present application;

[0034] Figure 2 This is a schematic diagram of the fusion model structure involved in an embodiment of the echo cancellation model training method of the present application;

[0035] Figure 3 A schematic diagram of the structure of a teacher model involved in an embodiment of an echo cancellation model training method of the present application;

[0036] Figure 4 A schematic diagram of the structure of an echo cancellation model involved in an embodiment of an echo cancellation model training method of the present application;

[0037] Figure 5 A brief flowchart of the echo cancellation model training method of this application;

[0038] Figure 6 This is a schematic diagram of the structure of the echo cancellation model training device of the present application;

[0039] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the echo cancellation model training method device in the embodiment of the present application.

[0040] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0041] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0042] Acoustic Echo Cancellation (AEC) technology plays a vital role in audio communication. Its purpose is to eliminate the coupling echo of the far-end signal between the local speaker and microphone, thereby improving the communication quality. Current acoustic echo cancellation (AEC) methods can be divided into traditional adaptive filter algorithms and deep learning-based algorithms. In traditional adaptive filter methods, the main techniques used are least mean square, normalized least mean square, proportional normalized least mean square and subband adaptive filter. These methods adapt to the changes in the echo path by adjusting the filter weights, and have low computational complexity, but have limitations in dealing with nonlinear distortion or long-term dependence. Deep learning-based AEC methods use models such as deep feedforward neural networks, convolutional neural networks and recurrent neural networks to enhance the echo cancellation effect and achieve excellent performance. However, these models have high parameter volume and computational complexity, and are also highly dependent on training data. In addition, hybrid models combine linear filters with neural networks to balance robustness and adaptability.

[0043] In actual scenarios, especially on low-power embedded devices and mobile terminals, the deployment of AEC models is usually limited by computing resources and storage space. Although deep neural networks have excellent performance, they are difficult to apply under the requirements of low latency and low power consumption. Therefore, under the conditions of limited computing, storage and other resources, how to design an efficient AEC model is still a challenge that needs to be overcome.

[0044] In recent years, knowledge distillation (KD) technology has been widely used in the field of deep learning to reduce the size of the model and maintain performance. However, in the field of echo cancellation, knowledge distillation technology faces significant challenges in ensuring both the amount of echo removal and the sound quality of the near-end speaker, especially in an environment with low parameters and low computing resources. Traditional knowledge distillation methods usually use a single teacher model to guide the student model, which cannot fully capture and transfer diverse knowledge, which is particularly unfavorable for maintaining robustness in complex acoustic environments. Therefore, there is a need for an acoustic echo cancellation framework that can effectively integrate the advantages of multiple teacher models and is suitable for low-parameter, low-computing resource conditions to achieve the best balance between echo cancellation and voice preservation.

[0045] Based on this, the main solution of the present application is: obtaining training data, an echo cancellation model to be trained, and a pre-constructed dynamic selection network, wherein the dynamic selection network includes a fusion model and multiple pre-trained teacher models, the fusion model is used to fuse the outputs of each of the teacher models, and each of the teacher models includes at least an echo cancellation task model and a voice retention task model; inputting the training data into the echo cancellation model to obtain a student echo cancellation result, and inputting the training data into the dynamic selection network to obtain a fused echo cancellation result; calculating the knowledge distillation loss based on the student echo cancellation result and the fused echo cancellation result; optimizing the echo cancellation model based on the knowledge distillation loss, returning to execute the steps of inputting the training data into the echo cancellation model to obtain a student echo cancellation result, and inputting the training data into the dynamic selection model to obtain a fused echo cancellation result, until the preset training end condition is met, and the trained echo cancellation model is obtained.

[0046] The present application fuses the output results of multiple pre-trained teacher models through a fusion model, calculates the knowledge distillation loss through the fused echo elimination result obtained after fusion and the student elimination result output by the echo cancellation model, and optimizes the echo cancellation model based on this knowledge distillation loss. At this time, the echo cancellation model is also the student model in the instruction distillation process, so as to achieve the purpose of multiple teacher models guiding the student model, instead of using a single teacher model to guide the student model, so that the student model can fully capture and learn the diverse knowledge of multiple teacher models, and because the teacher model includes at least a human voice preservation task model and an echo cancellation task model, the student model can fully capture and learn the teacher model's knowledge in human voice preservation and echo cancellation, so that the student model can take into account both echo cancellation and human voice preservation, that is, the echo cancellation model can take into account both echo cancellation and human voice preservation.

[0047] It should be noted that the execution subject of each embodiment of the echo cancellation model training method of the present application can be a computing service device with data processing, model communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device that can realize the above functions, such as headphones, AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, AR helmets, VR helmets, smart glasses, etc. The embodiments of the echo cancellation model training method of the present application do not make specific restrictions on this.

[0048] Based on this, the present application proposes a first embodiment of the echo cancellation model training method, please refer to Figure 1 As shown, the echo cancellation model training method includes the following steps S10 to S50:

[0049] Step S10, obtaining training data, an echo cancellation model to be trained, and a pre-constructed dynamic selection network, wherein the dynamic selection network includes a fusion model and a plurality of pre-trained teacher models, the fusion model is used to fuse the outputs of the teacher models, and each teacher model includes at least an echo cancellation task model and a voice retention task model;

[0050] At least two teacher models may be pre-trained on different training data sets in advance so that they can capture common features and patterns in the data. Each teacher model has different performance on different tasks, and the teacher models include but are not limited to echo cancellation task models and voice preservation task models, such as dual-speaker balance task models.

[0051] It should be noted that, in the present embodiment, defining a model as a certain task model means that the performance of the model on the task satisfies certain outstanding performance conditions, that is, the model performs outstandingly on the task. For example, the echo cancellation task model means that the model performs outstandingly on the echo cancellation task and can well eliminate the echo in the signal. The voice preservation task model means that the model performs outstandingly on the voice preservation task and can well preserve the echo in the signal. The dual-speaker balance task model means that the model performs outstandingly on the dual-speaker balance task and can well balance echo cancellation and voice preservation in both dual-speaker and single-speaker scenarios.

[0052] Further, relevant personnel may pre-set the conditions for evaluating whether the model performs outstandingly in a certain performance. For example, in a specific implementation, the performance of the model can be evaluated using the AECMOS (Acoustic Echo Control Mean Opinion Score) indicator to obtain echo mos (Echo Mean Opinion Score) and deg mos (Degradation Mean Opinion Score). If echo mos is greater than 3 and less than or equal to 5, it is determined that the model performs outstandingly in the echo cancellation task. If deg mos is greater than 3 and less than or equal to 5, it is determined that the model performs outstandingly in the human voice cancellation task. If echomos is greater than 1 and less than or equal to 3 and deg mos is greater than 1 and less than or equal to 3, it is determined that the model performs outstandingly in the double-talk balance task.

[0053] Furthermore, in order to pre-train a teacher model with outstanding performance in each performance, relevant personnel may pre-train the teacher model using a corresponding training data set in advance to obtain a teacher model with outstanding performance in each performance.

[0054] For the convenience of subsequent elaboration and explanation, exemplary, the three models of the pre-trained teacher model, namely the echo cancellation task model, the human voice retention task model and the double-talk balance task, are used as examples to explain the various embodiments of the present application. In a specific embodiment, in order to pre-train the echo cancellation task model, the echo-dominated scene can be simulated, the signal strength of the near-end speech can be reduced, the near-end speech can be mixed with different background noises, various noise environments can be simulated, and nonlinear distortion, reverberation effects and dynamic echo changes can be added to the mixed signal to obtain speech with noisy echoes. The teacher model is pre-trained with speech with noisy echoes as input and clean speech as a label to obtain the echo cancellation task model. To obtain the human voice preservation task model through pre-training, it is possible to ensure that near-end speech is dominant, reduce the reverberation and nonlinear distortion added to the noise, give priority to generating signals with high speech fidelity, avoid excessive delays and signal distortion, and let the model focus on restoring the naturalness and clarity of speech. In other words, the teacher model can be pre-trained with a mixed signal of speech and echo with a signal-to-noise ratio greater than a certain threshold as input and clean speech as label to obtain the human voice preservation task model. To obtain the dual-speaking balance task model through pre-training, it is possible to balance the distribution of dual-speaking and single-speaking scenarios, generate mixed signals of speech and echo in multiple scenarios including silence, near-end single-speaking, far-end single-speaking and dual-speaking, annotate the scene categories through voice activity detection, and combine the generalization ability of the diverse reverberation and noise enhancement data sets. In other words, the teacher model can be pre-trained with mixed signals in multiple scenarios as input and clean speech and scene categories as labels to obtain the dual-speaking balance task model.

[0055] After the teacher models are obtained through pre-training, these teacher models are subsequently used to guide the learning of the student models. In this embodiment, the student model is the echo cancellation model to be trained.

[0056] Considering that knowledge distillation uses the output of a teacher model to guide the student model, and multiple teacher models correspond to multiple outputs, in order to make these multiple outputs can be effectively used to guide the student model, this embodiment uses a fusion model to fuse the outputs of multiple teacher models to fuse the multiple outputs into one output, so that the principle of knowledge distillation can still be used to guide the student model based on the fused output.

[0057] Relevant personnel may construct a fusion model in advance. For example, in one specific implementation, the fusion model is MetaNet (Meta Networks, meta-learning network).

[0058] It should be noted that the output of each teacher model is connected to the input of the fusion model respectively, so that the output of each teacher model is received through the fusion model and the output of each teacher model is fused.

[0059] Step S20, inputting the training data into the echo cancellation model to obtain a student echo cancellation result, and inputting the training data into the dynamic selection network to obtain a fusion echo cancellation result;

[0060] After obtaining the training data, the training data is input into the echo cancellation model and the dynamic selection network respectively to obtain the student echo cancellation result representing the output of the student model and the fusion echo cancellation result representing the output of the teacher model.

[0061] It should be noted that the student echo cancellation result may specifically be the final output of the echo cancellation model, and the fusion echo cancellation result may specifically be the final output of the dynamic selection network, that is, the final output of the fusion model.

[0062] Step S30, calculating the knowledge distillation loss based on the student echo cancellation result and the fusion echo cancellation result;

[0063] The knowledge distillation loss is calculated based on the student echo elimination results and the fusion echo elimination results. Specifically, a loss function (hereinafter referred to as the first preset loss function) can be set in advance, and the student echo elimination results and the fusion echo elimination results are input into the first preset loss function to calculate the knowledge distillation loss.

[0064] It should be noted that the first preset loss function can be any loss function set in advance, such as a cross loss function, a logarithmic loss function, etc., and this embodiment does not impose any specific restrictions on this.

[0065] Step S40: optimizing the echo cancellation model based on the knowledge distillation loss.

[0066] If the preset training end condition is met, step S50 is executed: obtaining the trained echo cancellation model.

[0067] If the preset training end condition is not met, the process returns to step S20 until the preset training end condition is met.

[0068] It is easy to understand that during knowledge distillation, the knowledge distillation loss (also called soft label loss or category probability distillation loss) between the output of the teacher network and the student network is calculated, and the loss of the student network on the true label is usually calculated, which is called hard label loss or standard loss. Specifically, the overall loss function of knowledge distillation is usually a combination of these two losses. Based on this, this embodiment can also calculate the hard label loss between the student echo cancellation result and the training label, and optimize the echo cancellation model based on the knowledge distillation loss and the hard label loss, where the training label is the true label of the training data.

[0069] The echo cancellation model is optimized based on the knowledge distillation loss and the hard label loss. Specifically, the knowledge distillation loss and the hard label loss can be reduced as the optimization target, and the model parameters in the echo cancellation model can be updated. For example, the gradient value corresponding to the model parameter in the echo cancellation model can be calculated using the gradient descent algorithm according to the sum of the knowledge distillation loss and the hard label loss, and the model parameters in the echo cancellation model can be updated according to the gradient value, thereby achieving the purpose of optimizing the echo cancellation model.

[0070] The training end condition may be a condition set in advance, such as reaching a predetermined number of iterations, the loss function value being lower than a predetermined threshold, computing resources being exhausted, reaching a time limit, the accuracy reaching a predetermined threshold, etc. This embodiment does not impose specific restrictions on this.

[0071] This embodiment fuses the output results of multiple pre-trained teacher models through a fusion model, calculates the knowledge distillation loss through the fused echo elimination result obtained after fusion and the student elimination result output by the echo elimination model, and optimizes the echo elimination model based on this knowledge distillation loss. At this time, the echo elimination model is also the student model in the instruction distillation process, so as to achieve the purpose of multiple teacher models guiding the student model, instead of using a single teacher model to guide the student model, so that the student model can fully capture and learn the diverse knowledge of multiple teacher models, and because the teacher model includes at least a human voice preservation task model and an echo elimination task model, the student model can fully capture and learn the teacher model's knowledge in human voice preservation and echo elimination, so that the student model can take into account both echo elimination and human voice preservation, that is, the echo elimination model can take into account both echo elimination and human voice preservation. The echo elimination model can maintain efficient echo elimination capabilities and human voice preservation effects with low computational complexity and low parameter quantity. It can improve user experience without significantly increasing the number of model parameters, especially in mobile devices and low-power scenarios. It also improves the adaptability of the echo cancellation model to complex acoustic environments and enhances the robustness and generalization ability of the model.

[0072] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated later. On this basis, the step of calculating the knowledge distillation loss based on the student echo cancellation result and the fusion echo cancellation result includes:

[0073] Step A10, inputting the student echo cancellation result into a preset self-supervised learning model to obtain the student deep features, and inputting the fused echo cancellation result into the self-supervised learning model to obtain the teacher deep features, wherein the self-supervised learning model includes a large-scale self-supervised pre-trained WavLM model for full-stack speech processing;

[0074] It should be noted that relevant personnel can also set other self-supervised learning models (Self-Supervised Learning, SSL) as the preset self-supervised learning model, and this embodiment does not impose specific restrictions on this. Exemplarily, the preset self-supervised learning model is WavLM (Large-Scale Self-Supervised Pre-Training for FullStack Speech Processing, large-scale self-supervised pre-training for full-stack speech processing) model as an example to illustrate and explain the various embodiments of this application.

[0075] Furthermore, the WavLM model may specifically be a model with frozen parameters, that is, the model parameters of the WavLM model are not updated during the training process of the echo cancellation model.

[0076] The student echo cancellation results and the fused echo cancellation results are respectively input into the WavLM model to extract the deep features in the echo cancellation results and the fused echo cancellation results respectively, and the deep features extracted from the student echo cancellation results are determined as the student deep features, and the deep features extracted from the fused echo cancellation results are determined as the teacher deep features.

[0077] Furthermore, the deep features (including student deep features and teacher deep features) can be specifically the output of one or more layers of the network in the WavLM model. For example, in a specific embodiment, the deep features are the outputs of the N-10th to Nth layers of the WavLM model, where N is the number of layers of the last layer of the WavLM model, or the total number of network layers of the WavLM model.

[0078] Furthermore, the deep features may specifically include one or more of semantic information (such as the meaning of words or phrases in the audio), speaker characteristics (such as voiceprint, speaker identity), emotional characteristics (such as joy, anger and other emotions in the voice), and contextual information (such as the context of the voice, interrogative sentences, declarative sentences, etc.), which shall be subject to the actual output of the WavLM model, and this embodiment does not impose any specific restrictions on this.

[0079] Step A20, inputting the student deep features and the teacher deep features into a first preset loss function to obtain the knowledge distillation loss.

[0080] In this embodiment, the student echo cancellation result and the fused echo cancellation result are input into the WavLM model, the WavLM model is used to extract deep features, and the knowledge distillation loss is calculated based on the extracted deep features. It can be understood that the deep features extracted by the WavLM model can better represent the depth information of the features, that is, it reflects the difference between the echo cancellation model and the teacher model at a deeper level, and then the knowledge distillation loss is calculated based on the deep features to optimize the echo cancellation model, which can further improve the learning effect of the echo cancellation model.

[0081] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first and second embodiments can be referred to the above introduction, and will not be repeated later. On this basis, after the step of calculating the knowledge distillation loss based on the student echo cancellation result and the fusion echo cancellation result, the method further includes:

[0082] Step B10, obtaining the current iteration number of the echo cancellation model;

[0083] Step B20, if the current number of iterations meets a preset condition, freezing the model parameters of the fusion model, and executing the step of optimizing the echo cancellation model based on the knowledge distillation loss;

[0084] When the current number of iterations meets the preset conditions, the model parameters of the fusion model are frozen, that is, the model parameters of the fusion model are not updated, and the model parameters of the echo cancellation model are updated in this iteration.

[0085] The preset condition can be any condition set by relevant personnel based on actual needs, and is not specifically limited here. In a preferred embodiment, the preset condition is that the current number of iterations is an odd number, so as to alternately update the model parameters of the echo cancellation model and the fusion model in sequence, thereby ensuring balanced training and joint learning optimization of the echo cancellation model and the fusion model.

[0086] Step B30: If the current number of iterations does not meet a preset condition, the model parameters of the echo cancellation model are frozen, and the model parameters of the fusion model are optimized.

[0087] When the current number of iterations does not meet the preset conditions, the model parameters of the echo cancellation model are frozen, that is, the model parameters of the echo cancellation model are not updated, and the model parameters of the fusion model are updated in this iteration.

[0088] Considering that the student echo cancellation result output by the echo cancellation model will be subjected to knowledge distillation calculation with the fusion echo cancellation result output by the dynamic selection network, and then gradient backpropagation is performed. If the fusion model is updated together with the echo cancellation model, the output result of the dynamic selection network may temporarily fail to align with the output result of the echo cancellation model with excellent performance. Based on this, in this embodiment, the echo cancellation model and the fusion model update the model parameters alternately to make the training of the fusion model and the echo cancellation model more stable and easier to converge.

[0089] Furthermore, this embodiment updates the model parameters of the fusion model so that the fusion model can automatically select the teacher output that is most suitable for the student model learning, dynamically adapt to different acoustic environments, effectively integrate the knowledge of multiple teacher models, and optimize the learning process of the student model, thereby achieving high-quality acoustic echo cancellation even in a resource-constrained environment.

[0090] In a possible implementation, the step of optimizing the model parameters of the fusion model includes:

[0091] Step C10, obtaining the quadratic gradient of a second preset loss function, and optimizing the fusion model based on the quadratic gradient; wherein the second preset loss function is a loss function used to calculate the hard label loss of the echo cancellation model, and the hard label loss is a loss value between the student echo cancellation result and the training label corresponding to the training data.

[0092] Similar to the first preset loss function, the second preset loss function can be any loss function set in advance, such as a cross loss function, a logarithmic loss function, etc., and this embodiment does not impose any specific restrictions on this. For example, in a specific implementation, the second preset loss function is a mean square error (MSE) loss. Assume that the student echo cancellation result is The training label is y, then the MSE loss function is defined as: Among them, N is the amount of training data, and i are the student echo prediction results and the corresponding values ​​of the training labels for the i-th training data.

[0093] Let the second preset loss function be L MSE , first calculate the first gradient, then take its second derivative to get the second gradient.

[0094] In this embodiment, the quadratic gradient of the second preset loss function is used to update the fusion model, and based on the quadratic gradient method to update the model parameters, not only the gradient information is used in the model parameter update process, but also the Hessian matrix of the loss function, that is, the second-order derivative, which reflects the curvature of the objective function. This enables the optimizer to adjust the step size and direction according to the "degree of curvature" of the function when updating the parameters. For example, in steep areas, the Hessian matrix can guide the optimizer to use a smaller step size, while a larger step size can be used in flat areas to avoid the problem of over- or under-updating the parameters, thereby making the training of the fusion model more stable and easier to converge.

[0095] Based on the first embodiment, the second embodiment and / or the third embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those of the first embodiment, the second embodiment and the third embodiment can be referred to the above introduction, and will not be described in detail later. Figure 2 As shown, the fusion model includes an encoder, a self-attention network, a deep learning Mamba network based on sequence modeling, a decoder and an output layer ( Figure 2 not shown).

[0096] The encoder is used to convert the input data into a more compact and easier to process internal representation. Relevant personnel can pre-set the specific structure of the encoder. For example, in one specific embodiment, the encoder includes a first convolutional layer, a second convolutional layer, and an input projection layer connected in sequence. The kernel size of the convolutional layer is (3, 5), the padding is (1, 2), and the number of output channels is consistent with the input. After each layer of convolution, a ReLU (Rectified Linear Unit) activation function is connected to enhance the ability to represent nonlinear features. The input projection layer inputs the convolved features into the linear projection layer, and the feature dimension is the same as the input.

[0097] The self-attention network is used to capture the long-distance dependencies of the sequence and extract the high-level features of the input data. The relevant personnel can pre-set the specific structure of the self-attention network. For example, in a specific embodiment, the self-attention network includes a position encoder and a Transformer encoder connected in sequence. The position encoder uses two-dimensional position coding to encode the time position information into the features to enhance the model's sensitivity to time and frequency changes. The position coding is generated by sine and cosine functions, and the frequency axes of time and frequency are encoded at different scales. The Transformer encoder is an encoder stack composed of 4 layers of encoders connected in sequence. Each layer of encoder has 8 attention heads. The encoder uses a self-attention mechanism to capture global dependencies in the sequence.

[0098] The Mamba network is used to predict sequence data, and relevant personnel can pre-set various network parameters of the Mamba network. For example, in a specific implementation, the Mamba network is used as a mixer class, the convolution parameter d_conv=4, the state dimension d_state=16, and the expansion factor expand=4. The network includes forward and backward blocks, and the forward and backward feature representations are combined in a bidirectional model.

[0099] The decoder is used to generate the target sequence, and the relevant personnel can pre-set the specific structure of the decoder. For example, in a specific embodiment, the decoder includes an output projection layer and a deconvolution layer connected in sequence. The output projection layer re-projects the output of the Mamba network to the d_model dimension, and the output is further optimized through the residual connection. The deconvolution layer includes two deconvolution operations, the deconvolution kernel size is (3, 5), and the padding is (1, 2). The output after deconvolution is normalized by softmax to generate the final weight matrix for synthesizing the output signal.

[0100] The output layer is used to fuse the outputs of each teacher model based on the weight matrix output by the decoder to obtain the fused echo prediction result. Specifically, the output signal of the teacher model can be used as input, and the teacher amplitude is weighted by the weights generated by the encoder, decoder and Mamba network. The weighted amplitude is combined with the phase information of the teacher model to form a complex signal in the frequency domain, and then summed in the teacher dimension and inverse Fourier transformed to generate the final output audio signal.

[0101] For example, assuming that the three pre-trained teacher models are teacher model 1, teacher model 2, and teacher model 3, respectively, after the training data are input into the three teacher models, the outputs are output as output signals 1, 2, and 3, and the output signals 1, 2, and 3 are input into the fusion model. After the encoder, decoder, and Mamba network, the weight matrix generated is [weight 1, weight 2, weight 3], where weight 1 is the weight corresponding to output signal 1, weight 2 is the weight corresponding to output signal 2, and weight 3 is the weight corresponding to output signal 3. Then, in the output layer, the signal amplitudes of output signals 1, 2, and 3 are weighted and summed based on weight 1, weight 2, and weight 3 to obtain weighted signal amplitudes. The weighted signal amplitudes are combined with the phases of output signals 1, 2, and 3 to obtain complex signals 1, 2, and 3 in the frequency domain. Then, the three complex signals are summed and inverse STFT (Short-Time Fourier Transform) is performed to obtain the fusion echo cancellation result in the time domain.

[0102] Further, in a specific embodiment, referring to Figure 3As shown, each teacher model consists of an encoder, a bottleneck layer, an alignment layer, and a decoder.

[0103] The encoder includes a proximal signal branch and a distal signal branch. The proximal signal is input to the proximal signal branch, and the distal signal is input to the distal signal branch. The proximal branch contains five encoding blocks, while the distal signal branch contains two encoding blocks, followed by an alignment layer. The alignment layer uses a multi-head attention mechanism to align the proximal features with the distal features in time. After alignment, the distal and proximal features are connected together and input into the fifth encoding block.

[0104] Each encoding block consists of a downsampling convolution layer, a batch normalization layer (BatchNorm, BN), an ELU (Exponential Linear Unit) activation function, and a residual block (Residual Block, RB). The first encoding block uses 16 filters, the second encoding block uses 64 filters, the third encoding block uses 32 filters, and the fourth to seventh encoding blocks each use 64 filters. The kernel size of the downsampling convolution layer is 4×3, and the step size is 1×2. It is mainly used to reduce the frequency dimension on the frequency axis. All convolution operations are causal convolutions, that is, no look-ahead operation is performed to ensure that only past inputs are relied on.

[0105] In the alignment layer, the proximal feature output by the fourth encoding block is X M ∈R c×t×f , the remote feature output by the second encoding block is X F ∈R c×t×f These features are processed through point-wise convolutional layers to generate query (Q) and key (K) tensors, denoted as Q∈R h ×t×f and K∈R h×f×f , where h is the number of similar channels, t is time, f is frequency, and c is the feature channel. Then K is expanded on the time axis to form a delay dimension, which becomes Among them, d max is the maximum echo delay (in time frames). Then the dot product of the query and the expanded key is performed on the frequency axis to obtain The dot product result is fed into a convolutional layer with a kernel size of 5×3, padding of 3×1, and a stride of 1. The convolutional layer uses a filter to merge h similar channels into one attention head. Finally, a softmax operation is performed on the delay axis to output the delay probability distribution. Then, the distal feature X F The aligned far-end features are obtained by performing weighted summation on the time axis by delay probability weights, and the aligned far-end features are input into the fifth encoding block.

[0106] The bottleneck layer is located between the encoder and the decoder, and consists of a Mamba network and a linear projection layer connected in sequence. The input features are first flattened in the channel and frequency dimensions to obtain X∈R t×(c·f) , and then input to the Mamba network, and then passed to the linear projection layer. The output of the linear projection layer is resized to X∈R c×t×f Compared with LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit), Mamba network can achieve higher performance with similar parameters, so it is used as a bottleneck layer to improve model performance. Using linear projection after the recurrent layer (i.e. Mamba network) can reduce the number of hidden units, thereby improving performance and training stability.

[0107] The decoder consists of five decoding blocks. Except for the fifth decoding block, the first to fourth decoding blocks each contain a skip connection block, a residual block, a sub-pixel convolution block, batch normalization, and an ELU activation function. The last decoding block contains a skip connection block, a residual block, and a sub-pixel convolution block. The number of filters in the decoder only changes in the sub-pixel convolution block, and the tensor shape remains unchanged in the other modules. The number of filters in the sub-pixel convolution block in the first to fifth decoding blocks is 64, 64, 64, 32, and 32, respectively. The size of each convolution kernel is 4×3, the stride is 1, and all convolution operations are causal convolutions.

[0108] In addition, residual blocks are added to the encoding block and the decoding block to increase the capacity of the network without hindering the flow of gradients. The residual block contains convolutional layers, batch normalization, and ELU activation functions, which are defined as follows: Y = X + ELU (BatchNorm (Conv2D (X))), where X and Y represent the input and output tensors, respectively, BatchNorm represents batch normalization, Conv2D represents two-dimensional convolution, and ELU is the activation function. The convolutional layer has the same number of filters as the input, a kernel size of 4×3, a stride of 1, and causal padding to ensure that the input shape remains unchanged.

[0109] Further, in a specific embodiment, referring to Figure 4 As shown in Figure 1, the echo cancellation model consists of an encoder, a bottleneck layer, and a decoder connected in sequence. Compared with the teacher model, the alignment layer is removed and the feature dimensions of the decoder and encoder are reduced to reduce the amount of computation and complexity.

[0110] The basic structure of the encoder remains the same as the teacher model, but the third and fourth encoding blocks each use 32 filters, and the fifth to seventh encoding blocks each use 48 filters. The first encoding block uses 16 filters, and the second encoding block uses 32 filters.

[0111] The basic structure of the decoder remains the same as the teacher model, but the number of filters in the decoder changes only in the sub-pixel convolution block, and the tensor shape of the other modules remains unchanged. The number of filters in the sub-pixel convolution block is 48, 48, 48, 32, and 32, respectively, and the size of each convolution kernel is 4×3 with a stride of 1. All convolution operations are causal convolutions.

[0112] Compared with the teacher model, the echo cancellation model has a basically identical model structure, but the number of model parameters is significantly reduced. For example, when the teacher model and the echo cancellation model are designed with the above parameters, the number of teacher model parameters is approximately 1.01M, and the number of student model parameters is approximately 0.26M (260K). The number of parameters is significantly reduced, so that the echo cancellation model can be deployed on devices with lower computing power for echo cancellation.

[0113] For example, to help understand the technical concept or technical principle of the echo cancellation model training method after combining this embodiment with the first embodiment, the second embodiment and the third embodiment, please refer to Figure 5 As shown, the training method of the echo cancellation model includes:

[0114] Step 1: Input the training data into the pre-trained teacher model 1, teacher model 2 and teacher model 3 respectively, use the pre-trained teacher model to perform echo elimination on the training data, and then obtain the fused echo elimination result through the MetaNet fusion model.

[0115] Step 2: Input the training data into the echo cancellation model to be trained, that is, the student model, and output the student echo cancellation result.

[0116] Step 3: Use the clean speech as the training label and compare it with the student echo cancellation result estimated by the student model based on the first loss function L MSE Calculate hard label loss;

[0117] Step 4: The fusion echo cancellation result selected by the MetaNet fusion model is used as the label, and the student echo cancellation result estimated by the student model is input into the WavLM model respectively, and the student deep features and teacher deep features are obtained accordingly. Based on their deep features and the second loss function L kd Calculate knowledge distillation loss;

[0118] Step 5: Update the parameters of the student model by back-propagating according to the two losses obtained;

[0119] Step 6: Use clean speech as label and compare it with the first loss function L MSE The quadratic gradient is used to update the parameters of the MetaNet fusion model.

[0120] It should be noted that, in different training rounds, only step five or step six is ​​executed, and in a specific implementation manner, step five and step six may be executed alternately in the manner of executing step five in the 1st to 100th training rounds, executing step six in the 101st training round, executing step five in the 102nd to 201st training rounds, executing step six in the 202nd training round..., that is, the student model and the MetaNet fusion model are updated alternately at an alternating frequency of updating the student model 100 times and updating the MetaNet fusion model 1 time.

[0121] It should be noted that the above examples are only used to assist in understanding the present application and do not constitute a limitation on the echo cancellation model training method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0122] In addition, the embodiment of the present application also provides an echo cancellation method, which includes the following steps:

[0123] Acquire an original signal to be echo-cancelled and a target echo-cancellation model, and input the original signal into the target echo-cancellation model to obtain an echo-cancellation result;

[0124] The target echo cancellation model is an echo cancellation model trained by the echo cancellation model training method described in any of the above embodiments.

[0125] The execution subject of each embodiment of the echo cancellation method can be a computing service device with data processing, model communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device that can realize the above functions, such as headphones, AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, AR helmets, VR helmets, smart glasses, etc. This embodiment does not make specific restrictions on this. Furthermore, the execution subjects of the echo cancellation method of the present application and the echo cancellation model training method may be the same or different, and no specific restrictions are made here.

[0126] It is easy to understand that after the original signal to be echo-cancelled is acquired, the original signal is input into the target echo cancellation model to obtain an echo cancellation result, which may specifically be the output of the target echo cancellation model.

[0127] In addition, the present application also provides an echo cancellation model training device, referring to Figure 6 As shown, the echo cancellation model training device includes:

[0128] An acquisition module 10 is used to acquire training data, an echo cancellation model to be trained, and a pre-built dynamic selection network, wherein the dynamic selection network includes a fusion model and a plurality of pre-trained teacher models, the fusion model is used to fuse the outputs of each of the teacher models, and each of the teacher models includes at least an echo cancellation task model and a voice retention task model;

[0129] An input module 20, used for inputting the training data into the echo cancellation model to obtain a student echo cancellation result, and inputting the training data into the dynamic selection network to obtain a fusion echo cancellation result;

[0130] A calculation module 30, configured to calculate the knowledge distillation loss based on the student echo cancellation result and the fusion echo cancellation result;

[0131] The optimization module 40 is used to optimize the echo cancellation model based on the knowledge distillation loss until a preset training end condition is met, thereby obtaining the trained echo cancellation model.

[0132] In one embodiment, the calculation module 30 is further used for:

[0133] Inputting the student echo cancellation result into a preset self-supervised learning model to obtain the student deep features, and inputting the fused echo cancellation result into the self-supervised learning model to obtain the teacher deep features, wherein the self-supervised learning model includes a large-scale self-supervised pre-trained WavLM model for full-stack speech processing;

[0134] The student deep features and the teacher deep features are input into a first preset loss function to obtain the knowledge distillation loss.

[0135] In one embodiment, the optimization module 40 is further configured to:

[0136] Obtaining the current iteration number of the echo cancellation model;

[0137] If the current number of iterations meets a preset condition, freezing the model parameters of the fusion model, and optimizing the echo cancellation model based on the knowledge distillation loss;

[0138] If the current number of iterations does not meet a preset condition, the model parameters of the echo cancellation model are frozen, and the model parameters of the fusion model are optimized.

[0139] In one embodiment, the optimization module 40 is further configured to:

[0140] Obtaining a quadratic gradient of a second preset loss function, and optimizing the fusion model based on the quadratic gradient;

[0141] Among them, the second preset loss function is a loss function used when calculating the hard label loss of the echo cancellation model, and the hard label loss is a loss value between the student echo cancellation result and the training label corresponding to the training data.

[0142] In one embodiment, the preset condition is that the current iteration number is an odd number.

[0143] In one embodiment, the fusion model includes an encoder, a self-attention network, a deep learning Mamba network based on sequence modeling, a decoder and an output layer connected in sequence.

[0144] In addition, an embodiment of the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the echo cancellation model training method and / or the echo cancellation method as described above.

[0145] refer to Figure 7 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may also include, but are not limited to, mobile terminals such as headphones, AR glasses, VR glasses, AR helmets, VR helmets, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0146] like Figure 7As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0147] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the model through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0148] The electronic device provided in the embodiment of the present application adopts the echo cancellation model training method and / or echo cancellation method in the above embodiment, which can solve the technical problem that the current echo cancellation model is difficult to balance echo cancellation and voice preservation. Compared with the prior art, the beneficial effects of the electronic device provided in the present application are the same as the beneficial effects of the echo cancellation model training method and / or echo cancellation method provided in the above embodiment, and other technical features in the electronic device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0149] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0150] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0151] In addition, to achieve the above-mentioned purpose, an embodiment of the present application also provides a readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the echo cancellation model training method and / or echo cancellation method in the above-mentioned embodiment.

[0152] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0153] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0154] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device: obtains training data, an echo cancellation model to be trained, and a pre-constructed dynamic selection network, wherein the dynamic selection network includes a fusion model and multiple pre-trained teacher models, and the fusion model is used to fuse the outputs of each of the teacher models, and each of the teacher models includes at least an echo cancellation task model and a voice retention task model; inputs the training data into the echo cancellation model to obtain a student echo cancellation result, and inputs the training data into the dynamic selection network to obtain a fused echo cancellation result; calculates the knowledge distillation loss based on the student echo cancellation result and the fused echo cancellation result; optimizes the echo cancellation model based on the knowledge distillation loss, and returns to execute the steps of inputting the training data into the echo cancellation model to obtain a student echo cancellation result, and inputting the training data into the dynamic selection model to obtain a fused echo cancellation result, until the preset training end condition is met, and the trained echo cancellation model is obtained.

[0155] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of model, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0156] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0157] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of a module does not, in some cases, constitute a limitation on the module itself.

[0158] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned echo cancellation model training method and / or echo cancellation method, and can solve the technical problem that the current echo cancellation model is difficult to balance echo cancellation and human voice preservation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the echo cancellation model training method and / or echo cancellation method provided by the above-mentioned embodiments, and will not be elaborated here.

[0159] In addition, an embodiment of the present application also proposes a computer program product, including a computer program, which, when executed by a processor, implements the steps of the echo cancellation model training method and / or the echo cancellation method as described above.

[0160] The specific implementation of the computer program product of the present application is basically the same as the above-mentioned echo cancellation model training and / or echo cancellation method embodiments, and will not be repeated here.

[0161] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0162] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0163] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software sensor, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, including a number of instructions for a terminal device (which can be a mobile phone, computer, server or model device, etc.) to execute the methods described in each embodiment of the present application.

[0164] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An echo cancellation model training method, characterized in that: The echo cancellation model training method comprises the following steps: Acquire training data, an echo cancellation model to be trained, and a pre-built dynamic selection network, wherein the dynamic selection network includes a fusion model and a plurality of pre-trained teacher models, the fusion model is used to fuse the outputs of the teacher models, and each teacher model includes at least an echo cancellation task model and a voice retention task model; Inputting the training data into the echo cancellation model to obtain a student echo cancellation result, and inputting the training data into the dynamic selection network to obtain a fusion echo cancellation result; Calculating knowledge distillation loss based on the student echo cancellation result and the fusion echo cancellation result; The echo cancellation model is optimized based on the knowledge distillation loss, and the step of inputting the training data into the echo cancellation model to obtain the student echo cancellation result and inputting the training data into the dynamic selection model to obtain the fused echo cancellation result is returned to execute, until the preset training end condition is met, and the trained echo cancellation model is obtained.

2. The echo cancellation model training method according to claim 1, characterized in that: The step of calculating the knowledge distillation loss based on the student echo cancellation result and the fusion echo cancellation result includes: Inputting the student echo cancellation result into a preset self-supervised learning model to obtain the student deep features, and inputting the fused echo cancellation result into the self-supervised learning model to obtain the teacher deep features, wherein the self-supervised learning model includes a large-scale self-supervised pre-trained WavLM model for full-stack speech processing; The student deep features and the teacher deep features are input into a first preset loss function to obtain the knowledge distillation loss.

3. The echo cancellation model training method according to claim 1, characterized in that: After the step of calculating the knowledge distillation loss based on the student echo cancellation result and the fusion echo cancellation result, the method further includes: Obtaining the current iteration number of the echo cancellation model; If the current number of iterations meets a preset condition, freezing the model parameters of the fusion model, and executing the step of optimizing the echo cancellation model based on the knowledge distillation loss; If the current number of iterations does not meet a preset condition, the model parameters of the echo cancellation model are frozen, and the model parameters of the fusion model are optimized.

4. The echo cancellation model training method according to claim 3, characterized in that: The step of optimizing the model parameters of the fusion model comprises: Obtaining a quadratic gradient of a second preset loss function, and optimizing the fusion model based on the quadratic gradient; Among them, the second preset loss function is a loss function used when calculating the hard label loss of the echo cancellation model, and the hard label loss is a loss value between the student echo cancellation result and the training label corresponding to the training data.

5. The echo cancellation model training method according to claim 3, characterized in that: The preset condition is that the current number of iterations is an odd number.

6. The echo cancellation model training method according to any one of claims 1 to 5, characterized in that: The fusion model includes an encoder, a self-attention network, a deep learning Mamba network based on sequence modeling, a decoder and an output layer connected in sequence.

7. An echo cancellation method, characterized in that: The echo cancellation method comprises the following steps: Acquire an original signal to be echo-cancelled and a target echo-cancellation model, and input the original signal into the target echo-cancellation model to obtain an echo-cancellation result; The target echo cancellation model is an echo cancellation model trained by the echo cancellation model training method according to any one of claims 1 to 6.

8. An electronic device, characterized in that: The electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that: The readable storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, which implements the steps of the method according to any one of claims 1 to 7 when executed by a processor.

Images