Echo cancellation and noise reduction method and device

Abstract

The embodiment of the invention discloses an echo cancellation and noise reduction method and device. In the embodiment of the invention, a first near-end voice and a first far-end voice are obtained, a fourth near-end voice and a fourth far-end voice are generated through preprocessing, delay detection and short-time Fourier transform, then linear echo cancellation is carried out, and an error signal of linear echo cancellation and a linear echo signal are generated; inputting the fourth near-end voice, the fourth far-end voice, the linear echo cancelled error signal and the linear echo signal into a to-be-trained two-stage neural network multi-task fusion model, and determining a joint loss function; and generating a two-stage neural network multi-task fusion model according to the joint loss function, and then performing model compression to generate a target two-stage neural network multi-task fusion model for realizing echo cancellation and noise reduction. Through the above method, under the condition of an extremely low signal-to-return ratio, acoustic echo cancellation and noise reduction are realized, and the voice interaction and communication performance of the terminal device is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more particularly to a method and device for echo cancellation and noise reduction. Background Art

[0002] Intelligent voice technology is mainly used in voice communication and interactive systems, such as smart tablets, smart speakers, mobile conference screens, etc. The above voice communication and interactive systems have problems such as noise interference, large residual far-end echo, and long pickup distance, which seriously restrict the performance of voice communication and recognition systems.

[0003] In the prior art, two solutions have been proposed to improve the performance of voice communication and recognition systems. Solution 1 is to combine signal processing and neural networks. Specifically, linear echo suppression is first completed in the signal processing process, and then nonlinear echo suppression, noise reduction, reverberation, automatic gain control and other functions are completed through neural networks. Solution 2 is to establish an end-to-end multi-task learning network architecture such as DeepVQE, which can simultaneously complete acoustic echo cancellation, noise suppression, automatic gain control and other functions, with the characteristics of low latency, strong operability and simple link. However, when the signal-to-return ratio reaches below -30dB, the voice interaction and communication performance of the above two solutions are poor.

[0004] In summary, how to better achieve acoustic echo cancellation and noise reduction under extremely low signal-to-echo ratio and improve the voice interaction and communication performance of terminal equipment is a problem that needs to be solved at present. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and device for echo cancellation and noise reduction, which adopts a target two-stage neural network multi-task fusion model to better achieve acoustic echo cancellation and noise reduction under extremely low signal-to-echo ratio conditions, thereby improving the voice interaction and communication performance of terminal equipment.

[0006] In a first aspect, an embodiment of the present invention provides a method for echo cancellation and noise reduction, the method comprising:

[0007] Acquire a first near-end voice and a first far-end voice;

[0008] Preprocessing the first near-end speech and the first far-end speech to generate a preprocessed second near-end speech and a second far-end speech;

[0009] Performing delay detection on the second near-end speech and the second far-end speech to generate aligned third near-end speech and third far-end speech;

[0010] Performing short-time Fourier transform on the third near-end speech and the third far-end speech to generate a fourth near-end speech and a fourth far-end speech;

[0011] Performing linear echo cancellation on the fourth near-end voice and the fourth far-end voice to generate a linear echo cancellation error signal and a linear echo signal;

[0012] Inputting the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal into a two-stage neural network multi-task fusion model to be trained, and determining a joint loss function;

[0013] Adjust the two-stage neural network multi-task fusion model to be trained according to the joint loss function to generate a two-stage neural network multi-task fusion model;

[0014] The two-stage neural network multi-task fusion model is compressed to generate a target two-stage neural network multi-task fusion model, wherein the two-stage neural network multi-task fusion model is used to achieve echo cancellation and noise reduction.

[0015] Optionally, the method further includes:

[0016] The two-stage neural network multi-task fusion model is deployed to the terminal side.

[0017] Optionally, preprocessing the first near-end speech and the first far-end speech to generate a preprocessed second near-end speech and a second far-end speech specifically includes:

[0018] The first near-end speech and the first far-end speech are pre-emphasized, power frequency interference is removed, and windowing is performed to generate the second near-end speech and the second far-end speech.

[0019] Optionally, inputting the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal into a two-stage neural network multi-task fusion model to be trained to determine a joint loss function specifically includes:

[0020] Inputting the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal into the first stage of the two-stage neural network multi-task fusion model to be trained, generating a first-stage training loss function, and generating a first enhanced near-end speech and a first enhanced linear echo cancellation signal, wherein the first stage of the two-stage neural network multi-task fusion model is used for preliminary noise reduction, preliminary echo cancellation, and removal of late reverberation;

[0021] Inputting the fourth near-end speech, the error signal of the linear echo cancellation, the first enhanced near-end speech and the first enhanced linear echo cancellation signal into the second stage of the two-stage neural network multi-task fusion model to be trained, generating a second stage training loss function, wherein the second stage of the two-stage neural network multi-task fusion model is used to remove residual noise, residual echo and early reverberation;

[0022] A joint loss function is determined according to the first-stage training loss function and the second-stage training loss function.

[0023] Optionally, the step of inputting the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal into the first stage of the two-stage neural network multi-task fusion model to be trained to generate a first-stage training loss function specifically includes:

[0024] Extracting and concatenating features of the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal to obtain a first speech feature;

[0025] Inputting the first speech feature into a first encoding layer, a first bottleneck layer, and a first decoding layer to generate first intermediate speech data;

[0026] Inputting the first intermediate speech data and the fourth near-end speech into a first complex spectrum convolution mapping layer to generate the first enhanced near-end speech, and inputting the first intermediate speech data and the fourth far-end speech into a second complex spectrum convolution mapping layer to generate the first enhanced linear echo cancellation signal;

[0027] Inputting the first enhanced near-end speech and the first enhanced linear echo cancellation signal into a first fusion module to generate first-stage output features;

[0028] The first-stage training loss function is determined according to the first-stage output features.

[0029] Optionally, determining the first-stage training loss function according to the first-stage output features specifically includes:

[0030] Performing an inverse short-time Fourier transform on the output features of the first stage to generate the first stage target speech;

[0031] The first-stage training loss function is determined based on the first-stage target speech and the pre-acquired clean speech.

[0032] Optionally, the step of inputting the fourth near-end speech, the error signal of the linear echo cancellation, the first enhanced near-end speech, and the first enhanced linear echo cancellation signal into the second stage of the two-stage neural network multi-task fusion model to be trained to generate a second-stage training loss function specifically includes:

[0033] Extracting and concatenating features of the fourth near-end speech, the error signal of the linear echo cancellation, the first enhanced near-end speech, and the first enhanced linear echo cancellation signal to obtain a second speech feature;

[0034] Inputting the second speech feature into a second encoding layer, a second bottleneck layer, and a second decoding layer to generate second intermediate speech data;

[0035] Inputting the second intermediate speech data and the first enhanced near-end speech into a third complex spectrum convolution mapping layer to generate a second enhanced near-end speech, and inputting the second intermediate speech data and the first enhanced linear echo cancellation signal into a fourth complex spectrum convolution mapping layer to generate a second enhanced linear echo cancellation signal;

[0036] Inputting the second enhanced near-end speech and the second enhanced linear echo cancellation signal into a second fusion module to generate second-stage output features;

[0037] The second-stage training loss function is determined according to the second-stage output features.

[0038] Optionally, determining the second-stage training loss function according to the second-stage output features specifically includes:

[0039] Performing an inverse short-time Fourier transform on the output features of the second stage to generate a second stage target speech;

[0040] The second-stage training loss function is determined according to the second-stage target speech and the pre-acquired clean speech.

[0041] Optionally, determining a joint loss function according to the first-stage training loss function and the second-stage training loss function specifically includes:

[0042] Obtaining a first weight of the first-stage training loss function and a second weight of the first-stage training loss function;

[0043] Determine a first product of the first weight and the first stage training loss function, and a second product of the second weight and the second stage training loss function;

[0044] A sum of the first product and the second product is determined as the joint loss function.

[0045] Optionally, compressing the two-stage neural network multi-task fusion model to generate a target two-stage neural network multi-task fusion model specifically includes:

[0046] The two-stage neural network multi-task fusion model is quantized online to generate a two-stage neural network multi-task fusion model with a Torch model structure;

[0047] The two-stage neural network multi-task fusion model of the Torch model structure is converted to ONNX by an open neural network to generate a two-stage neural network multi-task fusion model of the ONNX model structure;

[0048] The two-stage neural network multi-task fusion model of the ONNX model structure is passed through the MNN inference engine to generate the target two-stage neural network multi-task fusion model.

[0049] In a second aspect, an embodiment of the present invention provides a method for echo cancellation and noise reduction, the method comprising:

[0050] Get the signal to be processed;

[0051] The signal to be processed is input into a target two-stage neural network multi-task fusion model to generate a target signal, wherein the target two-stage neural network multi-task fusion model is deployed on the terminal side to realize echo cancellation and noise reduction.

[0052] In a third aspect, an embodiment of the present invention provides a device for echo cancellation and noise reduction, the device comprising:

[0053] An acquisition unit, configured to acquire a first near-end voice and a first far-end voice;

[0054] A processing unit, configured to preprocess the first near-end speech and the first far-end speech to generate a preprocessed second near-end speech and a second far-end speech;

[0055] A detection unit, configured to perform delayed detection on the second near-end speech and the second far-end speech to generate an aligned third near-end speech and a third far-end speech;

[0056] The processing unit is further used to perform short-time Fourier transform on the third near-end speech and the third far-end speech to generate a fourth near-end speech and a fourth far-end speech respectively;

[0057] A generating unit, configured to perform linear echo cancellation on the fourth near-end voice and the fourth far-end voice, and generate a linear echo cancellation error signal and a linear echo signal;

[0058] A determination unit, configured to input the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal into a two-stage neural network multi-task fusion model to be trained, and determine a joint loss function;

[0059] The generating unit is further used to adjust the two-stage neural network multi-task fusion model to be trained according to the joint loss function to generate a two-stage neural network multi-task fusion model;

[0060] The generating unit is also used to compress the two-stage neural network multi-task fusion model to generate a target two-stage neural network multi-task fusion model, wherein the two-stage neural network multi-task fusion model is used to achieve echo cancellation and noise reduction.

[0061] Optionally, the device further comprises:

[0062] A deployment unit is used to deploy the two-stage neural network multi-task fusion model to the terminal side.

[0063] Optionally, the processing unit is specifically used for:

[0064] The first near-end speech and the first far-end speech are pre-emphasized, power frequency interference is removed, and windowing is performed to generate the second near-end speech and the second far-end speech.

[0065] Optionally, the determining unit is specifically configured to:

[0066] Inputting the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal into the first stage of the two-stage neural network multi-task fusion model to be trained, generating a first-stage training loss function, and generating a first enhanced near-end speech and a first enhanced linear echo cancellation signal, wherein the first stage of the two-stage neural network multi-task fusion model is used for preliminary noise reduction, preliminary echo cancellation, and removal of late reverberation;

[0067] Inputting the fourth near-end speech, the error signal of the linear echo cancellation, the first enhanced near-end speech and the first enhanced linear echo cancellation signal into the second stage of the two-stage neural network multi-task fusion model to be trained, generating a second stage training loss function, wherein the second stage of the two-stage neural network multi-task fusion model is used to remove residual noise, residual echo and early reverberation;

[0068] A joint loss function is determined according to the first-stage training loss function and the second-stage training loss function.

[0069] Optionally, the determining unit is specifically configured to:

[0070] Extracting and concatenating features of the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal to obtain a first speech feature;

[0071] Inputting the first speech feature into a first encoding layer, a first bottleneck layer, and a first decoding layer to generate first intermediate speech data;

[0072] Inputting the first intermediate speech data and the fourth near-end speech into a first complex spectrum convolution mapping layer to generate the first enhanced near-end speech, and inputting the first intermediate speech data and the fourth far-end speech into a second complex spectrum convolution mapping layer to generate the first enhanced linear echo cancellation signal;

[0073] Inputting the first enhanced near-end speech and the first enhanced linear echo cancellation signal into a first fusion module to generate first-stage output features;

[0074] The first-stage training loss function is determined according to the first-stage output features.

[0075] Optionally, the determining unit is specifically configured to:

[0076] Performing an inverse short-time Fourier transform on the output features of the first stage to generate the first stage target speech;

[0077] The first-stage training loss function is determined based on the first-stage target speech and the pre-acquired clean speech.

[0078] Optionally, the determining unit is specifically configured to:

[0079] Extracting and concatenating features of the fourth near-end speech, the error signal of the linear echo cancellation, the first enhanced near-end speech, and the first enhanced linear echo cancellation signal to obtain a second speech feature;

[0080] Inputting the second speech feature into a second encoding layer, a second bottleneck layer, and a second decoding layer to generate second intermediate speech data;

[0081] Inputting the second intermediate speech data and the first enhanced near-end speech into a third complex spectrum convolution mapping layer to generate a second enhanced near-end speech, and inputting the second intermediate speech data and the first enhanced linear echo cancellation signal into a fourth complex spectrum convolution mapping layer to generate a second enhanced linear echo cancellation signal;

[0082] Inputting the second enhanced near-end speech and the second enhanced linear echo cancellation signal into a second fusion module to generate second-stage output features;

[0083] The second-stage training loss function is determined according to the second-stage output features.

[0084] Optionally, the determining unit is specifically configured to:

[0085] Performing an inverse short-time Fourier transform on the output features of the second stage to generate a second stage target speech;

[0086] The second-stage training loss function is determined according to the second-stage target speech and the pre-acquired clean speech.

[0087] Optionally, the determining unit is specifically configured to:

[0088] Obtaining a first weight of the first-stage training loss function and a second weight of the first-stage training loss function;

[0089] Determine a first product of the first weight and the first stage training loss function, and a second product of the second weight and the second stage training loss function;

[0090] A sum of the first product and the second product is determined as the joint loss function.

[0091] Optionally, the generating unit is specifically used for:

[0092] The two-stage neural network multi-task fusion model is quantized online to generate a two-stage neural network multi-task fusion model with a Torch model structure;

[0093] The two-stage neural network multi-task fusion model of the Torch model structure is converted to ONNX by an open neural network to generate a two-stage neural network multi-task fusion model of the ONNX model structure;

[0094] The two-stage neural network multi-task fusion model of the ONNX model structure is passed through the MNN inference engine to generate the target two-stage neural network multi-task fusion model.

[0095] Optionally, the acquisition unit is further used to: acquire a signal to be processed;

[0096] The generating unit is also used to: input the signal to be processed into a target two-stage neural network multi-task fusion model to generate a target signal.

[0097] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement a method as described in the first aspect or any possible embodiment of the first aspect.

[0098] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement a method as described in the first aspect or any possible embodiment of the first aspect.

[0099] In an embodiment of the present invention, a first near-end speech and a first far-end speech are acquired; the first near-end speech and the first far-end speech are preprocessed to generate a preprocessed second near-end speech and a second far-end speech; the second near-end speech and the second far-end speech are delayed detected to generate an aligned third near-end speech and a third far-end speech; the third near-end speech and the third far-end speech are short-time Fourier transformed to generate a fourth near-end speech and a fourth far-end speech respectively; the fourth near-end speech and the fourth far-end speech are linearly echo-cancelled to generate an error signal of linear echo cancellation and a linear echo signal; the fourth near-end speech, the fourth far-end speech, the error signal of linear echo cancellation and the linear echo signal are input into a two-stage neural network multi-task fusion model to be trained to determine a joint loss function; the two-stage neural network multi-task fusion model to be trained is adjusted according to the joint loss function to generate a two-stage neural network multi-task fusion model; the two-stage neural network multi-task fusion model is model compressed to generate a target two-stage neural network multi-task fusion model, wherein the two-stage neural network multi-task fusion model is used to realize echo cancellation and noise reduction. Through the above method, a target two-stage neural network multi-task fusion model is generated. The target two-stage neural network multi-task fusion model can achieve acoustic echo cancellation and noise reduction under extremely low signal-to-echo ratio conditions, thereby improving the voice interaction and communication performance of terminal equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0101] Figure 1 is a flow chart of a method for echo cancellation and noise reduction in an embodiment of the present invention;

[0102] Figure 2 This is a schematic diagram of a structure of preliminary data processing in an embodiment of the present invention;

[0103] Figure 3 is a flow chart of a method for obtaining a first-stage training loss function in an embodiment of the present invention;

[0104] Figure 4 is a structural schematic diagram of a first fusion module in an embodiment of the present invention;

[0105] Figure 5is a schematic diagram of the structure of the first stage in an embodiment of the present invention;

[0106] Figure 6 is a schematic diagram of a structure for obtaining a first-stage training loss function in an embodiment of the present invention;

[0107] Figure 7 is a flow chart of a method for training loss function in the second stage in an embodiment of the present invention;

[0108] Figure 8 is a schematic diagram of the structure of the second stage in an embodiment of the present invention;

[0109] Fig. 9 is a schematic diagram of a structure for obtaining a second-stage training loss function in an embodiment of the present invention;

[0110] Fig.10 is a structural schematic diagram of generating the joint loss function in an embodiment of the present invention;

[0111] Fig.11 is a flow chart of a method for model compression in an embodiment of the present invention;

[0112] Fig.12 is a flow chart of another echo cancellation method in an embodiment of the present invention;

[0113] Fig.13 is a flow chart of another echo cancellation method in an embodiment of the present invention;

[0114] Fig.14 is a schematic diagram of a system structure in an embodiment of the present invention;

[0115] Fig.15 is a schematic diagram of another device for echo cancellation and noise reduction in an embodiment of the present invention;

[0116] Fig.16 is a schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0117] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, some specific details are described in detail. It is possible for those skilled in the art to fully understand the present application without the description of these details. In order to avoid confusing the essence of the present application, known methods, processes, flows, components and circuits are not described in detail.

[0118] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0119] Unless the context clearly requires otherwise, the words "include", "comprising" and similar words throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, the meaning is "including but not limited to".

[0120] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0121] With the upgrade of the intelligent voice industry in terms of acoustic hardware, application scenarios, user groups, software functions, etc., full-duplex communication (i.e. voice communication) and interactive systems have many types of acoustic environmental noise, which are complex and changeable, prominent nonlinear distortion of speakers in high-volume playback scenarios, large delay fluctuation range between far-end and near-end voices, extremely low signal-to-return ratio, obvious distortion of far-end interactive voice, prominent reverberation, damage to voice clarity, large residual far-end echo, and long pickup distance, which seriously restrict the performance of voice communication and recognition systems. In the prior art, two solutions are proposed to improve the performance of voice communication and recognition systems.

[0122] Solution 1: Combination of signal processing and neural network. Specifically, linear echo suppression is first completed in the signal processing process, and then nonlinear echo suppression, noise reduction, dereverberation, automatic gain control and other functions are completed through the neural network; for example, Gated Convolutional FT-LSTM neural network (GFTNN), Band-split Recurrent Neural Network (BSRNN), and Neural Kalman filtering (NKF); wherein the GFTNN and the BSRNN can also be called an integrated end-to-end model of echo cancellation, noise suppression and dereverberation.

[0123] Solution 2: Establish an end-to-end multi-task learning network architecture to simultaneously complete functions such as acoustic echo cancellation (AEC), noise suppression (NS), and automatic gain control (AGC). It has the characteristics of low latency, strong operability, and simple links; for example, the deep variational quantum eigensolver (DEEPVQE) and a speech signal repair and denoising network (ARepairing and Denoising Network for Speech SignalImprovement, RadNET).

[0124] However, in the process of smart speakers, large conference screens, and in-car voice interaction, there are often situations such as playing music, videos, and text-to-speech (TTS) at high volume, resulting in a signal-to-return ratio of less than -30dB. At this time, the voice interaction performance of both solutions is poor; and the above solutions are also poorly robust and cannot be implemented on low-computing power terminals. Therefore, how to achieve better acoustic echo cancellation in full-duplex under extremely low signal-to-return ratios, improve the voice communication performance, voice recognition performance, and robustness of terminal devices, and install and deploy them on low-computing power terminals is a problem that needs to be solved at present.

[0125] In an embodiment of the present invention, the signal-to-echo ratio refers to the ratio of the power of the near-end voice signal received from the terminal side to the power of the echo signal received from the speaker, wherein the echo signal is generated by playing the voice of the user on the opposite side of the terminal side through the speaker on the terminal side; the full-duplex is a communication mode in which data can be transmitted simultaneously in two directions, that is, both communicating parties can send and receive data at the same time, and the transmission in these two directions does not interfere with each other. For example, real-time communication (RTC), live broadcast and microphone connection and other scenarios are all full-duplex communications.

[0126] In order to solve the above problems, an echo cancellation and noise reduction method is proposed in the embodiment of the present invention. Figure 1 As shown, the method includes:

[0127] Step S101: Acquire a first near-end voice and a first far-end voice.

[0128] Specifically, the first near-end speech x(n) and the first far-end speech r(n) are both training data. Before obtaining the first near-end speech and the first far-end speech, it is necessary to construct a simulation data set, and then obtain the first near-end speech and the first far-end speech in the simulation data set.

[0129] In a possible implementation, when constructing the simulation data set, data collection is first performed, and the collected data can come from public data sets Librispeech, Aishell, AEC-CHALLENGE and recorded echo data, OPENSIL and 100,000 simulated room impulse responses, and DNS-CHALLENGE and recorded data; wherein the public data sets Librispeech and Aishell are used as clean speech, the AEC-CHALLENGE and recorded echo data are used as far-end speech of synthetic data, the OPENSIL and 100,000 simulated room impulse responses are used as transfer functions, and the noise data comes from DNS-CHALLENGE and recorded data; the collected data are synthesized into 500 hours of training data in a signal-to-return ratio range of [-35,15] and a signal-to-noise ratio range of [-5,15] dB, wherein the number of simulated room impulse responses, signal-to-return ratio range, signal-to-noise ratio range, and training data duration are all determined according to actual conditions, and are only illustrative here.

[0130] In an embodiment of the present invention, clean speech is obtained while obtaining the first near-end speech and the first far-end speech, wherein the clean speech is used to calculate the joint loss function of the two-stage neural network multi-task fusion model to be trained.

[0131] Step S102: preprocess the first near-end speech and the first far-end speech to generate a preprocessed second near-end speech and a second far-end speech.

[0132] Specifically, the first near-end voice and the first far-end voice are pre-emphasized, power frequency interference is removed, and windowing is performed to generate a pre-processed second near-end voice and a second far-end voice.

[0133] Step S103: Delay detection is performed on the second near-end speech and the second far-end speech to generate an aligned third near-end speech and a third far-end speech.

[0134] Specifically, the delay of the second near-end speech and the second far-end speech is estimated and aligned by a generalized cross-correlation variation algorithm (eg, GCC-PHAT function), thereby generating an aligned third near-end speech and a third far-end speech.

[0135] In a possible implementation, the time delay estimation (TDE) can improve the convergence speed and echo cancellation capability of a subsequent linear acoustic echo cancellation (LAEC) algorithm.

[0136] In an embodiment of the present invention, the second near-end voice and the second far-end voice are affected by factors such as sampling clock asynchrony, network delay instability and sound effect algorithm processing. The second near-end voice and the second far-end voice are not strictly aligned in the time dimension and have certain volatility, wherein the sound effect algorithm processing refers to Dolby sound effects, surround sound effects, etc.

[0137] Step S104: Perform short-time Fourier transform on the third near-end speech and the third far-end speech to generate a fourth near-end speech and a fourth far-end speech respectively.

[0138] In a possible implementation, the parameters used in the short-time Fourier transform (STFT) are shown in Table 1, specifically as follows:

[0139] Table 1

[0140] parameter Numeric Sampling rate 16000 Window length 32ms(512 points) Frame Shift 8ms(128 points) Fast Fourier Transform (FFT) length 512 points

[0141] The window function used in the short-time Fourier transform is the Hanning window.

[0142] Step S105: Perform linear echo cancellation on the fourth near-end voice and the fourth far-end voice to generate a linear echo cancellation error signal and a linear echo signal.

[0143] Specifically, the fourth near-end speech and the fourth far-end speech are input into a linear echo cancellation LACE module, and a linear echo cancellation error signal and a linear echo signal are generated according to the filter in the LACE module, wherein the filter in the LACE module uses a frequency domain block Kalman filter (Frequency Domain Block Kalman Filter, FDBKF) and a power-normalized least mean squares (Power-Normalized Least Mean Squares, PNLMS) filter as a primary adaptive and secondary adaptive filter, respectively, and when the FDBKF diverges, the PNLMS is used as the filter of the FDBKF, thereby improving the overall echo cancellation performance.

[0144] In one possible implementation, the filter can also be a partitioned block frequency domain Kalman filter (Partioned Block Frequeney Domain Kalman Filte, PBFDKF), a partitioned block frequency domain adaptive filter (Partioned Block Frequeney Domain AdaPtive Filter, PBDAF), a partitioned frequency domain least mean square (Partitioned Frequency-domain Block least-mean-square, PFBLMS) filter or a recursive least-squares adaptive filter (Recursive Least-Squares, RLS); during the data simulation process, the above-mentioned filters can also be used to dynamically expand the effect of echo cancellation under different conditions, enhance the suppression of residual echo by nonlinear echo cancellation, and improve the robustness of the two-stage neural network multi-task fusion model.

[0145] In the embodiment of the present invention, it is assumed that r(n) is the far-end speech and x(n) is the near-end speech. When the far-end speech passes through the unknown echo path w(n), an echo signal y(n)=r(n)*w(n) is generated, and the signal received by the near-end microphone is d(n)=y(n)+x(n). The adaptive filter w^(n) in the near-end linear echo cancellation module will estimate the linear echo signal y^(n) with reference to the far-end speech r(n), and subtract it from the signal d(n) received by the near-end microphone to obtain an error signal e(n)=d(n)-y^(n), wherein the e(n) should be close to the near-end speech x(n); without considering the near-end speech, the smaller the value of the error signal is, the closer the echo path estimated by the adaptive filter is to the actual echo path.

[0146] In the embodiment of the present invention, the above steps S101 to S105 complete the preliminary processing of the data. The specific structural diagram is as follows: Figure 2 As shown, the first near-end speech x(n) and the first far-end speech r(n) are pre-processed respectively, and then delayed detection TDE is performed to generate the second near-end speech and the second far-end speech. The second near-end speech and the second far-end speech are short-time Fourier transform STFT respectively to generate the third near-end speech and the third far-end speech. The third near-end speech and the third far-end speech are input into the linear echo cancellation LACE module to generate the error signal of the linear echo cancellation and the linear echo signal.

[0147] Step S106: input the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal into a two-stage neural network multi-task fusion model to be trained to determine a joint loss function.

[0148] In a possible implementation, the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal are input into the first stage of the two-stage neural network multi-task fusion model to be trained to generate a first-stage training loss function, and generate a first enhanced near-end speech and a first enhanced linear echo cancellation signal, wherein the first stage of the two-stage neural network multi-task fusion model is used for preliminary noise reduction, preliminary echo cancellation, and removal of late reverberation; the fourth near-end speech, the error signal of the linear echo cancellation, the first enhanced near-end speech, and the first enhanced linear echo cancellation signal are input into the second stage of the two-stage neural network multi-task fusion model to be trained to generate a second-stage training loss function, wherein the second stage of the two-stage neural network multi-task fusion model is used to remove residual noise, residual echo, and early reverberation; and a joint loss function is determined according to the first-stage training loss function and the second-stage training loss function.

[0149] In an embodiment of the present invention, the first stage of the two-stage neural network multi-task fusion model to be trained is also used to reduce the discontinuous distortion and amplitude change problems of the fourth near-end speech and the fourth far-end speech. The first stage of the two-stage neural network multi-task fusion model to be trained can also be called a wake-up and automatic speech recognition (Automatic Speech Recognition, ASR) model. The traditional deep learning-based noise reduction model is damaging to the speech. When the speech is damaged, the speech content cannot be accurately recognized. In the first stage, only 80% of the noise is removed to facilitate adaptation of the wake-up and ASR models, that is, to facilitate accurate recognition of the speech content in the first stage; the second stage of the two-stage neural network multi-task fusion model to be trained is used to reduce the distortion and overfitting problems generated by the first-stage model, improve the sound quality of the output speech, and thereby improve the user's auditory experience during voice communication.

[0150] The following two specific embodiments are used to respectively describe in detail how to obtain the first-stage training loss function and the second-stage training loss function, as follows: Specific embodiment 1:

[0152] The fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation and the linear echo signal are input into the first stage of the two-stage neural network multi-task fusion model to be trained, and the first stage training loss function is generated, such as Figure 3 As shown, specifically including:

[0153] Step S301: extract and concatenate features of the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal to obtain a first speech feature.

[0154] Specifically, the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo elimination, and the linear echo signal are subjected to short-time Fourier transform to retain the real and imaginary parts of the above four signals, and feature splicing is performed on the channel dimension to determine that the feature dimension of the first speech feature is (B, T, C, F), wherein B represents the size of each batch, T represents the total number of frames of the training audio, C represents the number of channels of the feature, for example, the number of channels is 8, and F represents the number of frequency points, for example, the number of frequency points is 257. This is only an exemplary description and will be determined based on actual conditions.

[0155] Step S302: input the first speech feature into a first encoding layer, a first bottleneck layer and a first decoding layer to generate first intermediate speech data.

[0156] In a possible implementation, the network structures of the first coding layer and the first decoding layer refer to network open source structures such as the Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement (DCCRN), the Dual-Path Convolution Recurrent Network (DPCRN), the Grouped Temporal Convolutional Recurrent Network (GTCRN) and DEEPVQE for phase-aware speech enhancement; wherein, the first coding layer is composed of 4 small coding modules, each of which is composed of 2 convolutions (Conv2d), regularization (Batch Norm), exponential linear unit (ELU) units and grouped temporal convolution blocks (GT-Conv blocks).

[0157] Step S303: input the first intermediate voice data and the fourth near-end voice into a first complex spectrum convolution mapping layer to generate the first enhanced near-end voice, and input the first intermediate voice data and the fourth far-end voice into a second complex spectrum convolution mapping layer to generate the first enhanced linear echo cancellation signal.

[0158] Specifically, the first complex spectrum convolution mapping (Complex convolving The first CCM layer and the second CCM layer are composed of two stages. First, the first CCM layer is taken as an example for detailed description. The output features of the first decoding layer and the fourth near-end speech input are convolved, and the convolution output is divided into three weight components to construct a complex mask, each component is the weight of a 120° rotation vector in the complex plane, and the enhanced complex domain mask features are obtained; the complex domain mask features are calculated in the time and frequency dimensions by a multi-core convolution filter to generate the first enhanced near-end speech; secondly, the second CCM layer is taken as an example for detailed description. The output features of the first decoding layer and the fourth far-end speech input are convolved, and the convolution output is divided into three weight components to construct a complex mask, each component is the weight of a 120° rotation vector in the complex plane, and the enhanced complex domain mask features are obtained; the complex domain mask features are calculated in the time and frequency dimensions by a multi-core convolution filter to generate the first enhanced linear echo cancellation signal.

[0159] In a possible implementation manner, the first CCM layer may also be expressed as a first MIC CCM layer, and the second CCM layer may also be expressed as a first LAEC CCM layer.

[0160] Step S304: input the first enhanced near-end speech and the first enhanced linear echo cancellation signal into a first fusion module to generate first-stage output features.

[0161] Specifically, the first enhanced near-end speech output by the first MIC CCM layer is represented by X mic The first enhanced linear echo cancellation signal output by the first LAEC CCM layer is represented by X laec , the first enhanced near-end speech X mic The first enhanced linear echo cancellation signal X laec Fusion is performed on the channel dimension to generate fusion feature X concate =(X laec ,X mic) ; The fusion feature X concate Perform convolution and generate X after one convolution module fusion =conv1(X concate ), after the secondary convolution module generates w = sigmoid (conv2 (X concate )), wherein w is a representation of weight weight, and the parameters of the first convolution and the second convolution are the same; the first enhanced near-end speech is processed to generate The enhanced linear echo cancellation signal is processed to generate For the Y LAEC Perform average pooling and maximum pooling, and then concatenate to obtain the Y LAEC Adaptive weight Mask, and then determine Y MIC The adaptive weight is 1-Mask, The first stage output features output by the first fusion module are determined. The specific processing process diagram is as follows: Figure 4 shown.

[0162] Step S305: Determine the first-stage training loss function according to the first-stage output features.

[0163] Specifically, the output features of the first stage are subjected to an inverse short-time Fourier transform to generate a first-stage target speech; and the first-stage training loss function is determined based on the first-stage target speech and the pre-acquired clean speech.

[0164] In a possible implementation, it is assumed that the clean speech is s(n), and the corresponding frequency domain signal is S(k, f); the target speech of the first stage is t(n), and the corresponding frequency domain signal is T(k, f); the target speech of the first stage is the first near-end speech with 80% noise reduction, and the clean speech is the first near-end speech with complete noise reduction. This is only an exemplary description. The target speech of the first stage is The corresponding frequency domain signal is The loss function mean square error (MSE) and scale-invariant source-to-noise ratio (SISNR) in the commonly used speech noise reduction and separation fields are used to determine the first stage training loss function L1, as follows:

[0165]

[0166] Herein, the α can be set to 0.5.

[0167] In the embodiment of the present invention, the structural diagram of the first stage is as follows: Figure 5As shown, it is assumed that the fourth near-end speech Mic, the fourth far-end speech Far end, the error signal Error of the linear echo elimination and the linear echo signal Lref are input into the first feature extraction module 501 of the two-stage neural network multi-task fusion model to be trained, and then sequentially input into the first encoding layer 502, the first bottleneck layer 503, and the first decoding layer 504 to generate the first intermediate speech data, the first intermediate speech data and the fourth near-end speech are input into the first CCM layer 505 to generate the first enhanced near-end speech, and the first intermediate speech data and the fourth far-end speech are input into the second complex spectrum convolution CCM layer 506 to generate the first enhanced linear echo elimination signal; the first enhanced near-end speech and the first enhanced linear echo elimination signal are input into the first fusion module 507 to generate the first stage output features.

[0168] Among them, the first encoding layer can also be called a first encoding module, the first bottleneck layer can also be called a first bottleneck module, the first decoding layer can also be called a first decoding module, the first complex spectrum convolution mapping CCM layer can also be called a first complex spectrum convolution mapping CCM module, and the second complex spectrum convolution mapping CCM layer can also be called a second complex spectrum convolution mapping CCM module.

[0169] In the embodiment of the present invention, the schematic diagram of determining the first stage training loss function is as follows: Figure 6 As stated in Figure 5 On this basis, an inverse short-time Fourier transform ISTFT is added. Specifically, the output features of the first stage are subjected to an inverse short-time Fourier transform to generate the first-stage target speech FS SPEECH; the first-stage training loss function FS LOSS is determined according to the first-stage target speech FS SPEECH and the pre-acquired clean speech Clean Speech. Specific embodiment 2:

[0171] The fourth near-end speech, the error signal of the linear echo cancellation, the first enhanced near-end speech and the first enhanced linear echo cancellation signal are input into the second stage of the two-stage neural network multi-task fusion model to be trained, and the second stage training loss function is generated, such as Figure 7 As shown, specifically including:

[0172] Step S701: extract and concatenate features of the fourth near-end speech, the error signal of the linear echo cancellation, the first enhanced near-end speech, and the first enhanced linear echo cancellation signal to obtain second speech features.

[0173] Specifically, the fourth near-end speech, the error signal of the linear echo cancellation, the first enhanced near-end speech and the first enhanced linear echo cancellation signal are subjected to short-time Fourier transform to retain the real and imaginary parts of the above four signals, and feature splicing is performed on the channel dimension to determine that the feature dimension of the first speech feature is (B, T, C, F), wherein B represents the size of each batch, T represents the total number of frames of the training audio, C represents the number of channels of the feature, for example, the number of channels is 8, and F represents the number of frequency points, for example, the number of frequency points is 257. This is only an exemplary description and shall be determined according to actual conditions.

[0174] Step S702: input the second speech feature into a second encoding layer, a second bottleneck layer and a second decoding layer to generate second intermediate speech data.

[0175] In a possible implementation, the network structure of the second coding layer and the second decoding layer refers to network open source structures such as the Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement (DCCRN), the Dual-Path Convolution Recurrent Network (DPCRN), the Grouped Temporal Convolutional Recurrent Network (GTCRN) and DEEPVQE for phase-aware speech enhancement; wherein, the first coding layer is composed of 4 small coding modules, each of which is composed of 2 convolutions (Conv2d), regularization (Batch Norm), exponential linear unit (ELU) units and grouped temporal convolution blocks (GT-Conv blocks).

[0176] Step S703: input the second intermediate speech data and the first enhanced near-end speech into the third complex spectrum convolution mapping layer to generate a second enhanced near-end speech, and input the second intermediate speech data and the first enhanced linear echo cancellation signal into the fourth complex spectrum convolution mapping layer to generate a second enhanced linear echo cancellation signal.

[0177] Specifically, the third complex spectrum convolution mapping (Complex convolving The third CCM layer and the fourth CCM layer are both composed of two stages. First, the third CCM layer is taken as an example for detailed description. The complex mask is constructed by convolving the output features of the second decoding layer with the first enhanced near-end speech, and the convolution output is divided into three weight components. Each component is the weight of a 120° rotation vector in the complex plane to obtain the enhanced complex domain mask feature; the complex domain mask feature is calculated in the time and frequency dimensions by a multi-core convolution filter to generate the second enhanced near-end speech; secondly, the fourth CCM layer is taken as an example for detailed description. The output features of the second decoding layer are convolved with the first enhanced linear echo cancellation signal, and the convolution output is divided into three weight components to construct a complex mask. Each component is the weight of a 120° rotation vector in the complex plane to obtain the enhanced complex domain mask feature; the complex domain mask feature is calculated in the time and frequency dimensions by a multi-core convolution filter to generate the second enhanced linear echo cancellation signal.

[0178] In a possible implementation, the third CCM layer may also be expressed as a second MIC CCM layer, and the fourth CCM layer may also be expressed as a second LAEC CCM layer.

[0179] Step S704: input the second enhanced near-end speech and the second enhanced linear echo cancellation signal into a second fusion module to generate second-stage output features.

[0180] Specifically, the second enhanced near-end speech output by the second MIC CCM layer is represented by X mic The second enhanced linear echo cancellation signal output by the second LAEC CCM layer is represented by X laec , the second enhanced near-end speech X mic The second enhanced linear echo cancellation signal X laec Fusion is performed on the channel dimension to generate fusion feature X concate =(X laec ,X mic) ; The fusion feature X concate Perform convolution and generate X after one convolution fusion =conv1(X concate ), after secondary convolution, w = sigmoid (conv2 (X concate )), wherein w is a representation of weight weight, and the parameters of the first convolution and the second convolution are the same; the second enhanced near-end speech is processed to generate The enhanced linear echo cancellation signal is processed to generate For the Y LAEC Perform average pooling and maximum pooling, and then concatenate to obtain the Y LAEC Adaptive weight Mask, and then determine Y MIC The adaptive weight is 1-Mask, The second stage output feature output by the second fusion module is determined. The specific processing process diagram is the same as that described above. Figure 4 The same, only the input is different, which will not be described here.

[0181] Step S705: Determine the second-stage training loss function according to the second-stage output features.

[0182] Specifically, the second-stage output features are subjected to an inverse short-time Fourier transform to generate a second-stage target speech; and the second-stage training loss function is determined based on the second-stage target speech and the pre-acquired clean speech.

[0183] In a possible implementation, it is assumed that the clean speech is s(n), the corresponding frequency domain signal is S(k,f), and the target speech in the second stage is Corresponding frequency domain signal The loss function mean square error (MSE) and scale-invariant source-to-noise ratio (SISNR) in the commonly used speech noise reduction and separation fields are used to determine the first stage training loss function L1, as follows:

[0184]

[0185] Herein, the α can be set to 0.5.

[0186] In the embodiment of the present invention, the structural diagram of the second stage is as follows: Figure 8As shown, it is assumed that the fourth near-end speech Mic, the error signal Error of the linear echo cancellation, the first enhanced near-end speech Mic CCM and the first enhanced linear echo cancellation signal Lace CCM are input into the second feature extraction module 801 of the two-stage neural network multi-task fusion model to be trained, and then sequentially input into the second encoding layer 802, the second bottleneck layer 803, and the second decoding layer 804 to generate the second intermediate speech data, and the second intermediate speech data and the first enhanced near-end speech are input into the third CCM layer 805 to generate the second enhanced near-end speech, and the second intermediate speech data and the first enhanced linear echo cancellation signal are input into the fourth complex spectrum convolution CCM layer 806 to generate the second enhanced linear echo cancellation signal; the second enhanced near-end speech and the second enhanced linear echo cancellation signal are input into the second fusion module 807 to generate the second stage output features.

[0187] Among them, the second encoding layer can also be called a second encoding module, the second bottleneck layer can also be called a second bottleneck module, the second decoding layer can also be called a second decoding module, the third complex spectrum convolution mapping CCM layer can also be called a third complex spectrum convolution mapping CCM module, and the fourth complex spectrum convolution mapping CCM layer can also be called a fourth complex spectrum convolution mapping CCM module.

[0188] In the embodiment of the present invention, the schematic diagram of determining the second stage training loss function is as follows: Fig. 9 As stated in Figure 8 On the basis of the second stage, an inverse short time Fourier transform ISTFT is added. Specifically, the output features of the second stage are subjected to an inverse short time Fourier transform to generate the second stage target speech TS SPEECH; the second stage training loss function TS LOSS is determined according to the second stage target speech TS SPEECH and the pre-acquired clean speech Clean Speech.

[0189] In the embodiment of the present invention, the first-stage training loss function FS LOSS and the second-stage training loss function TS LOSS are determined through the above-mentioned specific embodiment 1 and specific embodiment 2.

[0190] Step S107: adjusting the two-stage neural network multi-task fusion model to be trained according to the joint loss function to generate a two-stage neural network multi-task fusion model.

[0191] Specifically, obtain a first weight of the first-stage training loss function and a second weight of the first-stage training loss function; determine a first product of the first weight and the first-stage training loss function, and a second product of the second weight and the second-stage training loss function; and determine the sum of the first product and the second product as the joint loss function.

[0192] In a possible implementation, assuming that the first weight is 0.3, the second weight is 0.7, the first stage training loss function is L1, and the first stage training loss function is L2, the joint loss function is as follows:

[0193] L loss =0.3*L1+0.7*L2

[0194] The values ​​of the first weight and the second weight are determined according to actual conditions, and are only illustrative here.

[0195] In the embodiment of the present invention, combined with the Figure 6 and stated Fig. 9 , determine the structural diagram of generating the joint loss function TOTALLOSS, as shown in Fig.10 shown.

[0196] Step S108: compress the two-stage neural network multi-task fusion model to generate a target two-stage neural network multi-task fusion model, wherein the two-stage neural network multi-task fusion model is used to achieve echo cancellation and noise reduction.

[0197] Specifically, the model compression process is as follows: Fig.11 As shown, including:

[0198] Step S1101: Online quantization is performed on the two-stage neural network multi-task fusion model to generate a two-stage neural network multi-task fusion model with a Torch model structure.

[0199] Specifically, the two-stage neural network multi-task fusion model is quantized online through QAT, the weights are quantized to INT_8 or Float_16, and a two-stage neural network multi-task fusion model of the Torch model structure is generated.

[0200] Step S1102: The two-stage neural network multi-task fusion model of the Torch model structure is converted to ONNX through an open neural network to generate a two-stage neural network multi-task fusion model of the ONNX model structure.

[0201] Step S1103: The two-stage neural network multi-task fusion model of the ONNX model structure is passed through the MNN inference engine to generate the target two-stage neural network multi-task fusion model.

[0202] Through the above embodiment, preliminary noise reduction, preliminary echo cancellation and late reverberation removal are performed in the first stage of the two-stage neural network multi-task fusion model to avoid speech distortion caused by excessive noise reduction and inability to accurately perform speech recognition. Then, the residual noise, residual echo and early reverberation are removed in the second stage to obtain a voice signal for voice communication, and then acoustic echo cancellation and noise reduction are achieved under extremely low signal-to-echo ratio conditions, thereby improving the voice interaction and communication performance of the terminal device. In addition, the two-stage neural network multi-task fusion model also has the characteristics of low latency, weak dependence on linear AEC, and greater signal-to-noise ratio improvement. Through model compression, quantization and other strategies, the model is deployed on the ARM platform using the MNN framework, so that the two-stage neural network multi-task fusion model has the characteristics of low latency and low computing power occupancy.

[0203] In the embodiment of the present invention, after step S108, the method further includes other steps, specifically: Fig.12 As shown, the details are as follows:

[0204] Step S109: deploy the two-stage neural network multi-task fusion model to the terminal side.

[0205] Specifically, the two-stage neural network multi-task fusion model is deployed on the Advanced RISC Machines (ARM) or Digital Signal Processing (DSP) platform on the terminal side, and then the entire voice link is integrated through engineering integration to form an end-to-end voice signal streaming processing system. Since the parameters of the target two-stage neural network multi-task fusion model are relatively small, it can be deployed on the terminal side with low computing power.

[0206] In an embodiment of the present invention, the target two-stage neural network multi-task fusion model is deployed to the terminal side, which can realize functions such as echo cancellation, speech noise reduction, nonlinear echo suppression, and dereverberation, and complete the processing flow of the speech signal.

[0207] In a possible implementation, after completing step S109 to deploy the target two-stage neural network multi-task fusion model to the terminal side, the method further includes other steps, specifically, Fig.13 As shown:

[0208] Step S110: Obtain a signal to be processed.

[0209] Wherein, the signal to be processed is a voice signal.

[0210] Step S111: input the signal to be processed into a target two-stage neural network multi-task fusion model to generate a target signal.

[0211] Specifically, by inputting the signal to be processed into the target two-stage neural network multi-task fusion model, the target signal can be directly generated from the signal to be processed, thereby improving the echo elimination, voice noise reduction, and dereverberation of the signal to be processed under conditions of extremely low signal-to-response ratio, unstable delay, and variable echo propagation path, thereby improving the voice interaction performance and robustness of the terminal device.

[0212] In the embodiment of the present invention, the system structure for training the target two-stage neural network multi-task fusion model is briefly described, as follows: Fig.14 As shown, it includes: a data collection and simulation data set construction module 1401, a model training module 1402 and a model compression and terminal deployment module 1403. This is only an exemplary description, and the specific system structure is constructed according to actual conditions.

[0213] In an embodiment of the present invention, a device for echo cancellation and noise reduction is provided, such as Fig.15 As shown, it specifically includes: an acquisition unit 1501, a processing unit 1502, a detection unit 1503, a generation unit 1504 and a determination unit 1505;

[0214] Among them, the acquisition unit 1501 is used to acquire a first near-end voice and a first far-end voice; the processing unit 1502 is used to pre-process the first near-end voice and the first far-end voice to generate a pre-processed second near-end voice and a second far-end voice; the detection unit 1503 is used to delay the detection of the second near-end voice and the second far-end voice to generate an aligned third near-end voice and a third far-end voice; the processing unit 1502 is also used to perform short-time Fourier transform on the third near-end voice and the third far-end voice to generate a fourth near-end voice and a fourth far-end voice respectively; the generation unit 1504 is used to perform linear echo cancellation on the fourth near-end voice and the fourth far-end voice to generate a linear echo cancellation The fourth near-end speech, the fourth far-end speech, the linear echo elimination error signal and the linear echo signal are input into the two-stage neural network multi-task fusion model to be trained to determine the joint loss function; the generation unit 1504 is also used to adjust the two-stage neural network multi-task fusion model to be trained according to the joint loss function to generate a two-stage neural network multi-task fusion model; the generation unit 1504 is also used to compress the two-stage neural network multi-task fusion model to generate a target two-stage neural network multi-task fusion model, wherein the two-stage neural network multi-task fusion model is used to realize echo elimination and noise reduction.

[0215] Furthermore, the device also includes:

[0216] A deployment unit is used to deploy the two-stage neural network multi-task fusion model to the terminal side.

[0217] Furthermore, the processing unit is specifically used for:

[0218] The first near-end speech and the first far-end speech are pre-emphasized, power frequency interference is removed, and windowing is performed to generate the second near-end speech and the second far-end speech.

[0219] Furthermore, the determining unit is specifically configured to:

[0220] Inputting the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal into the first stage of the two-stage neural network multi-task fusion model to be trained, generating a first-stage training loss function, and generating a first enhanced near-end speech and a first enhanced linear echo cancellation signal, wherein the first stage of the two-stage neural network multi-task fusion model is used for preliminary noise reduction, preliminary echo cancellation, and removal of late reverberation;

[0221] Inputting the fourth near-end speech, the error signal of the linear echo cancellation, the first enhanced near-end speech and the first enhanced linear echo cancellation signal into the second stage of the two-stage neural network multi-task fusion model to be trained, generating a second stage training loss function, wherein the second stage of the two-stage neural network multi-task fusion model is used to remove residual noise, residual echo and early reverberation;

[0222] A joint loss function is determined according to the first-stage training loss function and the second-stage training loss function.

[0223] Furthermore, the determining unit is specifically configured to:

[0224] Extracting and concatenating features of the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal to obtain a first speech feature;

[0225] Inputting the first speech feature into a first encoding layer, a first bottleneck layer, and a first decoding layer to generate first intermediate speech data;

[0226] Inputting the first intermediate speech data and the fourth near-end speech into a first complex spectrum convolution mapping layer to generate the first enhanced near-end speech, and inputting the first intermediate speech data and the fourth far-end speech into a second complex spectrum convolution mapping layer to generate the first enhanced linear echo cancellation signal;

[0227] Inputting the first enhanced near-end speech and the first enhanced linear echo cancellation signal into a first fusion module to generate first-stage output features;

[0228] The first-stage training loss function is determined according to the first-stage output features.

[0229] Furthermore, the determining unit is specifically configured to:

[0230] Performing an inverse short-time Fourier transform on the output features of the first stage to generate the first stage target speech;

[0231] The first-stage training loss function is determined based on the first-stage target speech and the pre-acquired clean speech.

[0232] Furthermore, the determining unit is specifically configured to:

[0233] Extracting and concatenating features of the fourth near-end speech, the error signal of the linear echo cancellation, the first enhanced near-end speech, and the first enhanced linear echo cancellation signal to obtain a second speech feature;

[0234] Inputting the second speech feature into a second encoding layer, a second bottleneck layer, and a second decoding layer to generate second intermediate speech data;

[0235] Inputting the second intermediate speech data and the first enhanced near-end speech into a third complex spectrum convolution mapping layer to generate a second enhanced near-end speech, and inputting the second intermediate speech data and the first enhanced linear echo cancellation signal into a fourth complex spectrum convolution mapping layer to generate a second enhanced linear echo cancellation signal;

[0236] Inputting the second enhanced near-end speech and the second enhanced linear echo cancellation signal into a second fusion module to generate second-stage output features;

[0237] The second-stage training loss function is determined according to the second-stage output features.

[0238] Furthermore, the determining unit is specifically configured to:

[0239] Performing an inverse short-time Fourier transform on the output features of the second stage to generate a second stage target speech;

[0240] The second-stage training loss function is determined according to the second-stage target speech and the pre-acquired clean speech.

[0241] Furthermore, the determining unit is specifically configured to:

[0242] Obtaining a first weight of the first-stage training loss function and a second weight of the first-stage training loss function;

[0243] Determine a first product of the first weight and the first stage training loss function, and a second product of the second weight and the second stage training loss function;

[0244] A sum of the first product and the second product is determined as the joint loss function.

[0245] Furthermore, the generating unit is specifically used for:

[0246] The two-stage neural network multi-task fusion model is quantized online to generate a two-stage neural network multi-task fusion model with a Torch model structure;

[0247] The two-stage neural network multi-task fusion model of the Torch model structure is converted to ONNX by an open neural network to generate a two-stage neural network multi-task fusion model of the ONNX model structure;

[0248] The two-stage neural network multi-task fusion model of the ONNX model structure is passed through the MNN inference engine to generate the target two-stage neural network multi-task fusion model.

[0249] Furthermore, the acquisition unit is further used to: acquire a signal to be processed;

[0250] The generating unit is also used to: input the signal to be processed into a target two-stage neural network multi-task fusion model to generate a target signal.

[0251] Fig.16 Schematic diagram of the structure of the electronic device in the embodiment of the present invention. Fig.16 As shown, it includes a general computer hardware structure, which at least includes a processor 1601 and a memory 1602. The processor 1601 and the memory 1602 are connected via a bus 1603. The memory 1602 is suitable for storing instructions or programs executable by the processor 1601. The processor 1601 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 1601 executes the instructions stored in the memory 1602, thereby executing the method flow of the embodiment of the present invention as described above to realize the processing of data and the control of other devices. The bus 1603 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 1604 and the display device and the input / output (I / O) device 1605. The input / output (I / O) device 1605 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 1605 is connected to the system through an input / output (I / O) controller 1606.

[0252] Among them, the instructions stored in the memory 1602 are executed by at least one processor 1601 to achieve: obtaining a first near-end voice and a first far-end voice; preprocessing the first near-end voice and the first far-end voice to generate a preprocessed second near-end voice and a second far-end voice; delaying the second near-end voice and the second far-end voice to generate an aligned third near-end voice and a third far-end voice; performing short-time Fourier transform on the third near-end voice and the third far-end voice to generate a fourth near-end voice and a fourth far-end voice; performing linear echo cancellation on the fourth near-end voice and the fourth far-end voice to generate a linear echo cancellation. The fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation and the linear echo signal are input into the two-stage neural network multi-task fusion model to be trained to determine the joint loss function; the two-stage neural network multi-task fusion model to be trained is adjusted according to the joint loss function to generate a two-stage neural network multi-task fusion model; the two-stage neural network multi-task fusion model is compressed to generate a target two-stage neural network multi-task fusion model, wherein the two-stage neural network multi-task fusion model is used to realize echo cancellation and noise reduction.

[0253] Specifically, the electronic device includes: one or more processors 1601 and a memory 1602, Fig.16 Take a processor 1601 as an example. The processor 1601 and the memory 1602 may be connected via a bus or other means. Fig.16 The example of the bus connection is taken as an example. The memory 1602 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The processor 1601 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions and modules stored in the memory 1602, that is, the above-mentioned method for determining echo cancellation and noise reduction is implemented.

[0254] The memory 1602 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store a list of options, etc. In addition, the memory 1602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1602 may optionally include a memory remotely arranged relative to the processor 1601, and these remote memories may be connected to an external device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0255] One or more modules are stored in the memory 1602 , and when executed by one or more processors 1601 , the echo cancellation and noise reduction method in any of the above method embodiments is executed.

[0256] As will be appreciated by those skilled in the art, various aspects of the embodiments of the present invention may be implemented as a system, method or computer program product. Therefore, various aspects of the embodiments of the present invention may take the form of a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.), or an implementation that combines software aspects with hardware aspects, which may generally be referred to herein as a "circuit," "module," or "system." In addition, various aspects of the embodiments of the present invention may take the form of a computer program product implemented in one or more computer-readable media having a computer-readable program code implemented thereon.

[0257] Any combination of one or more computer-readable media can be utilized. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, (but not limited to) electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or apparatuses, or any suitable combination of the foregoing. More specific examples (non-exhaustive enumeration) of computer-readable storage media will include the following: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of an embodiment of the present invention, a computer-readable storage medium can be any tangible medium that can contain or store a program used by an instruction execution system, device or apparatus or a program used in conjunction with an instruction execution system, device or apparatus.

[0258] A computer readable signal medium may include a propagated digital signal having a computer readable program code implemented therein, such as in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any of the following computer readable media: not a computer readable storage medium, and may communicate, propagate, or transmit a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0259] Program code embodied on a computer readable medium may be transmitted using any appropriate medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0260] The computer program code for performing operations for various aspects of the embodiments of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and conventional process programming languages ​​such as "C" programming language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer as a stand-alone software package; partially on the user's computer and partially on a remote computer; or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., by using the Internet of an Internet service provider).

[0261] The flowchart legend and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present invention described above describe various aspects of the embodiment of the present invention.It will be understood that each block of the flowchart legend and / or block diagram and the combination of the blocks in the flowchart legend and / or block diagram can be implemented by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that the instructions (executed by the processor of the computer or other programmable data processing device) create a device for implementing the function / action specified in the flowchart and / or block diagram block or block.

[0262] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing device, or other apparatus to operate in a particular manner, so that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions / actions specified in the flowchart and / or block diagram blocks or blocks.

[0263] The computer program instructions may also be loaded onto a computer, other programmable data processing device or other apparatus so that a series of operable steps are performed on the computer, other programmable device or other apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the functions / actions specified in the flowchart and / or block diagram block or blocks.

[0264] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0265] The embodiment of the present invention discloses A1, a method for echo cancellation and noise reduction, the method comprising:

[0266] Acquire a first near-end voice and a first far-end voice;

[0267] Preprocessing the first near-end speech and the first far-end speech to generate a preprocessed second near-end speech and a second far-end speech;

[0268] Performing delay detection on the second near-end speech and the second far-end speech to generate aligned third near-end speech and third far-end speech;

[0269] Performing short-time Fourier transform on the third near-end speech and the third far-end speech to generate a fourth near-end speech and a fourth far-end speech;

[0270] Performing linear echo cancellation on the fourth near-end voice and the fourth far-end voice to generate a linear echo cancellation error signal and a linear echo signal;

[0271] Inputting the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal into a two-stage neural network multi-task fusion model to be trained, and determining a joint loss function;

[0272] Adjust the two-stage neural network multi-task fusion model to be trained according to the joint loss function to generate a two-stage neural network multi-task fusion model;

[0273] The two-stage neural network multi-task fusion model is compressed to generate a target two-stage neural network multi-task fusion model, wherein the two-stage neural network multi-task fusion model is used to achieve echo cancellation and noise reduction.

[0274] A2. The method according to A1, further comprising:

[0275] The two-stage neural network multi-task fusion model is deployed to the terminal side.

[0276] A3. According to the method described in A1, preprocessing the first near-end speech and the first far-end speech to generate the preprocessed second near-end speech and the second far-end speech specifically includes:

[0277] The first near-end speech and the first far-end speech are pre-emphasized, power frequency interference is removed, and windowing is performed to generate the second near-end speech and the second far-end speech.

[0278] A4, according to the method described in A1, the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation and the linear echo signal are input into the two-stage neural network multi-task fusion model to be trained to determine the joint loss function, specifically including:

[0279] Inputting the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal into the first stage of the two-stage neural network multi-task fusion model to be trained, generating a first-stage training loss function, and generating a first enhanced near-end speech and a first enhanced linear echo cancellation signal, wherein the first stage of the two-stage neural network multi-task fusion model is used for preliminary noise reduction, preliminary echo cancellation, and removal of late reverberation;

[0280] Inputting the fourth near-end speech, the error signal of the linear echo cancellation, the first enhanced near-end speech and the first enhanced linear echo cancellation signal into the second stage of the two-stage neural network multi-task fusion model to be trained, generating a second stage training loss function, wherein the second stage of the two-stage neural network multi-task fusion model is used to remove residual noise, residual echo and early reverberation;

[0281] A joint loss function is determined according to the first-stage training loss function and the second-stage training loss function.

[0282] A5. The method according to A4, wherein the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal are input into the first stage of the two-stage neural network multi-task fusion model to be trained to generate a first-stage training loss function, specifically comprising:

[0283] Extracting and concatenating features of the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal to obtain a first speech feature;

[0284] Inputting the first speech feature into a first encoding layer, a first bottleneck layer, and a first decoding layer to generate first intermediate speech data;

[0285] Inputting the first intermediate speech data and the fourth near-end speech into a first complex spectrum convolution mapping layer to generate the first enhanced near-end speech, and inputting the first intermediate speech data and the fourth far-end speech into a second complex spectrum convolution mapping layer to generate the first enhanced linear echo cancellation signal;

[0286] Inputting the first enhanced near-end speech and the first enhanced linear echo cancellation signal into a first fusion module to generate first-stage output features;

[0287] The first-stage training loss function is determined according to the first-stage output features.

[0288] A6. The method according to A5, wherein determining the first-stage training loss function according to the first-stage output features specifically includes:

[0289] Performing an inverse short-time Fourier transform on the output features of the first stage to generate the first stage target speech;

[0290] The first-stage training loss function is determined based on the first-stage target speech and the pre-acquired clean speech.

[0291] A7, according to the method of A4, the fourth near-end speech, the error signal of the linear echo cancellation, the first enhanced near-end speech and the first enhanced linear echo cancellation signal are input into the second stage of the two-stage neural network multi-task fusion model to be trained to generate the second stage training loss function, specifically including:

[0292] Extracting and concatenating features of the fourth near-end speech, the error signal of the linear echo cancellation, the first enhanced near-end speech, and the first enhanced linear echo cancellation signal to obtain a second speech feature;

[0293] Inputting the second speech feature into a second encoding layer, a second bottleneck layer, and a second decoding layer to generate second intermediate speech data;

[0294] Inputting the second intermediate speech data and the first enhanced near-end speech into a third complex spectrum convolution mapping layer to generate a second enhanced near-end speech, and inputting the second intermediate speech data and the first enhanced linear echo cancellation signal into a fourth complex spectrum convolution mapping layer to generate a second enhanced linear echo cancellation signal;

[0295] Inputting the second enhanced near-end speech and the second enhanced linear echo cancellation signal into a second fusion module to generate second-stage output features;

[0296] The second-stage training loss function is determined according to the second-stage output features.

[0297] A8. The method according to A7, wherein determining the second-stage training loss function according to the second-stage output features specifically includes:

[0298] Performing an inverse short-time Fourier transform on the output features of the second stage to generate a second stage target speech;

[0299] The second-stage training loss function is determined according to the second-stage target speech and the pre-acquired clean speech.

[0300] A9. The method according to A4, wherein determining the joint loss function according to the first-stage training loss function and the second-stage training loss function specifically includes:

[0301] Obtaining a first weight of the first-stage training loss function and a second weight of the first-stage training loss function;

[0302] Determine a first product of the first weight and the first stage training loss function, and a second product of the second weight and the second stage training loss function;

[0303] A sum of the first product and the second product is determined as the joint loss function.

[0304] A10. According to the method described in A1, compressing the two-stage neural network multi-task fusion model to generate a target two-stage neural network multi-task fusion model specifically includes:

[0305] The two-stage neural network multi-task fusion model is quantized online to generate a two-stage neural network multi-task fusion model with a Torch model structure;

[0306] The two-stage neural network multi-task fusion model of the Torch model structure is converted to ONNX by an open neural network to generate a two-stage neural network multi-task fusion model of the ONNX model structure;

[0307] The two-stage neural network multi-task fusion model of the ONNX model structure is passed through the MNN inference engine to generate the target two-stage neural network multi-task fusion model.

[0308] The embodiment of the present invention discloses B1, a method for echo cancellation and noise reduction, the method further comprising:

[0309] Get the signal to be processed;

[0310] The signal to be processed is input into a target two-stage neural network multi-task fusion model to generate a target signal, wherein the target two-stage neural network multi-task fusion model is deployed on the terminal side to realize echo cancellation and noise reduction.

[0311] The embodiment of the present invention discloses C1, a device for echo cancellation and noise reduction, the device comprising:

[0312] An acquisition unit, configured to acquire a first near-end voice and a first far-end voice;

[0313] A processing unit, configured to preprocess the first near-end speech and the first far-end speech to generate a preprocessed second near-end speech and a second far-end speech;

[0314] A detection unit, configured to perform delayed detection on the second near-end speech and the second far-end speech to generate an aligned third near-end speech and a third far-end speech;

[0315] The processing unit is further used to perform short-time Fourier transform on the third near-end speech and the third far-end speech to generate a fourth near-end speech and a fourth far-end speech respectively;

[0316] A generating unit, configured to perform linear echo cancellation on the fourth near-end voice and the fourth far-end voice, and generate a linear echo cancellation error signal and a linear echo signal;

[0317] A determination unit, configured to input the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal into a two-stage neural network multi-task fusion model to be trained, and determine a joint loss function;

[0318] The generating unit is further used to adjust the two-stage neural network multi-task fusion model to be trained according to the joint loss function to generate a two-stage neural network multi-task fusion model;

[0319] The generating unit is also used to compress the two-stage neural network multi-task fusion model to generate a target two-stage neural network multi-task fusion model, wherein the two-stage neural network multi-task fusion model is used to achieve echo cancellation and noise reduction.

[0320] An embodiment of the present invention discloses D1, an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement a method as described in any one of A1-A10 and B1.

[0321] The embodiment of the present invention discloses E1, a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of A1-A10 and B1 is implemented.

Claims

1. A method for echo cancellation and noise reduction, It is characterized in that The method comprises: Acquire a first near-end voice and a first far-end voice; Preprocessing the first near-end speech and the first far-end speech to generate a preprocessed second near-end speech and a second far-end speech; Performing delay detection on the second near-end speech and the second far-end speech to generate aligned third near-end speech and third far-end speech; Performing short-time Fourier transform on the third near-end speech and the third far-end speech to generate a fourth near-end speech and a fourth far-end speech; Performing linear echo cancellation on the fourth near-end voice and the fourth far-end voice to generate a linear echo cancellation error signal and a linear echo signal; Inputting the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal into a two-stage neural network multi-task fusion model to be trained, and determining a joint loss function; Adjust the two-stage neural network multi-task fusion model to be trained according to the joint loss function to generate a two-stage neural network multi-task fusion model; The two-stage neural network multi-task fusion model is compressed to generate a target two-stage neural network multi-task fusion model, wherein the two-stage neural network multi-task fusion model is used to achieve echo cancellation and noise reduction.

2. The method according to claim 1, It is characterized in that The method further comprises: The two-stage neural network multi-task fusion model is deployed to the terminal side.

3. The method according to claim 1, It is characterized in that The preprocessing of the first near-end speech and the first far-end speech to generate the preprocessed second near-end speech and the second far-end speech specifically includes: The first near-end speech and the first far-end speech are pre-emphasized, power frequency interference is removed, and windowing is performed to generate the second near-end speech and the second far-end speech.

4. The method according to claim 1, It is characterized in that The step of inputting the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal into a two-stage neural network multi-task fusion model to be trained to determine a joint loss function specifically includes: Inputting the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal into the first stage of the two-stage neural network multi-task fusion model to be trained, generating a first-stage training loss function, and generating a first enhanced near-end speech and a first enhanced linear echo cancellation signal, wherein the first stage of the two-stage neural network multi-task fusion model is used for preliminary noise reduction, preliminary echo cancellation, and removal of late reverberation; Inputting the fourth near-end speech, the error signal of the linear echo cancellation, the first enhanced near-end speech and the first enhanced linear echo cancellation signal into the second stage of the two-stage neural network multi-task fusion model to be trained, generating a second stage training loss function, wherein the second stage of the two-stage neural network multi-task fusion model is used to remove residual noise, residual echo and early reverberation; A joint loss function is determined according to the first-stage training loss function and the second-stage training loss function.

5. The method according to claim 4, It is characterized in that The step of inputting the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal into the first stage of the two-stage neural network multi-task fusion model to be trained, and generating a first-stage training loss function, specifically includes: Extracting and concatenating features of the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal to obtain a first speech feature; Inputting the first speech feature into a first encoding layer, a first bottleneck layer, and a first decoding layer to generate first intermediate speech data; Inputting the first intermediate speech data and the fourth near-end speech into a first complex spectrum convolution mapping layer to generate the first enhanced near-end speech, and inputting the first intermediate speech data and the fourth far-end speech into a second complex spectrum convolution mapping layer to generate the first enhanced linear echo cancellation signal; Inputting the first enhanced near-end speech and the first enhanced linear echo cancellation signal into a first fusion module to generate first-stage output features; The first-stage training loss function is determined according to the first-stage output features.

6. The method according to claim 5, It is characterized in that The determining the first-stage training loss function according to the first-stage output features specifically includes: Performing an inverse short-time Fourier transform on the output features of the first stage to generate the first stage target speech; The first-stage training loss function is determined based on the first-stage target speech and the pre-acquired clean speech.

7. A method for echo cancellation and noise reduction, It is characterized in that The method further comprises: Get the signal to be processed; The signal to be processed is input into a target two-stage neural network multi-task fusion model to generate a target signal, wherein the target two-stage neural network multi-task fusion model is deployed on the terminal side to realize echo cancellation and noise reduction.

8. An echo cancellation and noise reduction device, It is characterized in that The device comprises: An acquisition unit, configured to acquire a first near-end voice and a first far-end voice; A processing unit, configured to preprocess the first near-end speech and the first far-end speech to generate a preprocessed second near-end speech and a second far-end speech; A detection unit, configured to perform delayed detection on the second near-end speech and the second far-end speech to generate an aligned third near-end speech and a third far-end speech; The processing unit is further used to perform short-time Fourier transform on the third near-end speech and the third far-end speech to generate a fourth near-end speech and a fourth far-end speech respectively; A generating unit, configured to perform linear echo cancellation on the fourth near-end voice and the fourth far-end voice, and generate a linear echo cancellation error signal and a linear echo signal; A determination unit, configured to input the fourth near-end speech, the fourth far-end speech, the error signal of the linear echo cancellation, and the linear echo signal into a two-stage neural network multi-task fusion model to be trained, and determine a joint loss function; The generating unit is further used to adjust the two-stage neural network multi-task fusion model to be trained according to the joint loss function to generate a two-stage neural network multi-task fusion model; The generating unit is also used to compress the two-stage neural network multi-task fusion model to generate a target two-stage neural network multi-task fusion model, wherein the two-stage neural network multi-task fusion model is used to achieve echo cancellation and noise reduction.

9. An electronic device comprising a memory and a processor, It is characterized in that The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that The computer readable storage The medium stores a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images