Echo cancellation method and system based on heterogeneous multi-scale convolution circulation network

Abstract

The invention discloses an echo cancellation method and system based on a heterogeneous multi-scale convolution circulation network, and the method comprises the steps: firstly extracting a related feature mode based on a multi-frame context, constructing a related encoder, and then constructing a frequency spectrum encoder according to a single-frame extraction frequency spectrum mode; then forming a heterogeneous multi-scale encoder by adopting a related encoder and a frequency spectrum encoder, and obtaining multi-channel encoding characteristics; according to the invention, a function of extracting different features from an input mixed signal spectrum and a reference signal spectrum by using a heterogeneous double-channel encoder is realized, and in order to adapt to a time delay condition between a reference signal and a mixed signal, correlation encoding is adopted, and correlation feature expression is extracted from a multi-frame spectrum; meanwhile, in order to enhance the spectrum resolution of the current frame, spectrum coding is adopted to extract spectrum mode expression from the current frame, the function of further adopting a channel attention feature soft selection mechanism to fuse correlation feature coding and spectrum coding is achieved for the two-way features, and the method is suitable for being widely popularized and used.

Description

Technical Field

[0001] The present invention relates to the technical field of echo processing, and in particular to an echo cancellation method and system based on a heterogeneous multi-scale convolutional recurrent network. Background Art

[0002] Echo cancellation is an ongoing challenge in the communications field, and the presence of echo greatly affects many voice backend tasks, such as automatic speech recognition and blind source separation; if the near-end microphone picks up the voice from the speaker and transmits it back to the far end, this will cause the far-end user to hear their own voice, thus generating an echo.

[0003] At present, echoes are often mixed with reverberation and noise, which increases the difficulty of the echo cancellation task; traditional methods mainly use normalized least mean square (NLMS) adaptive filters and eliminate echoes by estimating the echo path between the speaker and the microphone; however, these adaptive filters require parameter adjustment to balance the convergence speed and the echo tracking ability of the filter, and the filter takes a period of time to converge in a remote single-talk scenario. If there is a long-term double-talk scenario, the adaptive filter is easy to diverge; therefore, it is necessary to design an echo cancellation method and system based on a heterogeneous multi-scale convolutional recurrent network. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art, and to effectively solve the problem that the traditional method mainly applies the normalized least mean square NLMS adaptive filter, and eliminates the echo by estimating the echo path between the speaker and the microphone; however, these adaptive filters require parameter adjustment to balance the convergence speed and the echo tracking ability of the filter, and the filter needs a period of time to converge in a remote single-talking scene. If there is a long-term double-talking scene, the adaptive filter is very likely to diverge. Provided is an echo cancellation method and system based on a heterogeneous multi-scale convolutional recurrent network, which realizes the function of extracting different features from the input mixed signal spectrum and the reference signal spectrum using a heterogeneous two-way encoder, and adopts correlation coding to extract correlation feature expression from multiple frame spectra to adapt to the time delay between the reference signal and the mixed signal, and adopts spectrum coding to extract spectrum mode expression from the current frame in order to enhance the spectrum resolution of the current frame, and also realizes the function of further adopting the channel attention feature soft selection mechanism to fuse the correlation feature coding and spectrum coding for the two-way feature, and can reconstruct the coding structure in the convolutional recurrent model and further propose a multi-way feature fusion mechanism, which can stably and effectively improve the effect of the model while ensuring the model parameter amount and algorithm complexity.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] The echo cancellation method based on heterogeneous multi-scale convolutional recurrent network includes the following steps:

[0007] Step A, extracting relevant feature patterns based on multi-frame contexts to construct a correlation encoder, then extracting spectrum patterns based on a single frame to construct a spectrum encoder, and then using the correlation encoder and the spectrum encoder to form a heterogeneous multi-scale encoder to obtain multi-path coding features;

[0008] Step B, construct a channel attention fusion mechanism based on multi-channel encoding features;

[0009] Step C: construct a long memory network based on the channel attention fusion mechanism, and then use the long memory network to capture the inter-frame temporal dependency features and intra-frame spectral harmonic features;

[0010] Step D, constructing a complex filtering paradigm according to the time-dependent features between frames and the harmonic features of the spectrum within a frame, and then using the complex filtering paradigm to reconstruct the signal and output the complex spectrum of the sample speech;

[0011] Step E: Use inverse Fourier transform to restore the time domain waveform of the complex spectrum of the sample speech, and then use the overlap-add method to synthesize the entire speech segment to complete the echo cancellation operation.

[0012] The echo cancellation method based on heterogeneous multi-scale convolutional recurrent network mentioned above, step A, based on multi-frame context extraction of relevant feature patterns to construct a relevant encoder, then according to the single frame extraction of spectrum patterns to construct a spectrum encoder, then the relevant encoder and the spectrum encoder are used to form a heterogeneous multi-scale encoder to obtain multi-path coding features, wherein the relevant encoder is used to extract correlation features from multi-frame input spectra and obtain global spectrum features, the relevant encoder is composed of a stacked 4-layer convolutional network, each layer of convolution in the relevant encoder has a length of 3 in the time dimension and a length of 5 in the frequency dimension, the first two layers of convolution in the relevant encoder are used to downsample the frequency, the output size of the relevant encoder is half of the input size, and the last two layers of convolution in the relevant encoder are equal-size mappings; the spectrum encoder is used to extract spectrum patterns from a single-frame mixed spectrum and obtain local features, the spectrum encoder is composed of a 4-layer convolutional network, the convolution size of the spectrum encoder in the time dimension is 1, and the specific steps of constructing the heterogeneous multi-scale encoder are as follows,

[0013] Step A1: transform the mixed signal spectrum X mic and the reference signal spectrum X ref The concatenation is done in the channel dimension and then input into the related encoder to extract features. The first convolution layer of the related encoder is X i＝0 (t) As shown in formula (1),

[0014]

[0015] Among them, Φi and * represent the convolution kernel parameters and convolution calculation of the i-th layer of the relevant encoder, respectively. LN represents the normalized LayerNorm. The subsequent layers i=1…3 take the output of the i-1th layer as input, i.e., X i =PReLU(LN(X i-1 *Φ i-1 ));

[0016] Step A2: transform the mixed signal spectrum X mic The input spectrum encoder calculates the spectrum features, as shown in formula (2),

[0017] X i＝0 (t) = PReLU(LN(X mic (t)*Θ i )), i∈[0,1,2,3](2)

[0018] Among them, Θ i Represents the convolution kernel parameters of the ith layer of the spectral encoder.

[0019] The aforementioned echo cancellation method based on heterogeneous multi-scale convolutional recurrent network, step B, constructs a channel attention fusion mechanism according to multi-channel coding features, wherein each channel in the output feature spectrum of the correlation encoder and the spectrum encoder is taken as a subband, and then the subband global feature attention and local feature attention are applied to the input dual-channel feature spectrum for feature fusion. The global feature is passed through the global pooling layer to calculate the average value of all frequency points of each subband, and then the average value of all frequency points of each subband is sent to the sequence composed of two layers of point-by-point convolution for feature mapping. The local feature is sent to the sequence composed of two layers of point-by-point convolution after transposing the subband and frequency point dimensions to calculate the components of the frequency points on different channels. The feature fusion specifically adds the global features and the local features and uses Sigmoid to calculate the weights, and then fuses the two-channel feature spectra in a soft selection manner to complete the construction of the channel attention fusion mechanism. The specific steps are as follows,

[0020] Step B1, add relevant features X point by point corr and spectral feature X spec Thus, the input characteristic spectrum of the fusion module is obtained As shown in formula (3),

[0021] X=X corr +X spec (3);

[0022] Step B2: Input the input feature spectrum X into the pooling layer Calculate the statistical features of each sub-band and then feed it into two layers of point-by-point convolution sequence As shown in formula (4),

[0023]

[0024] Among them, θ1 and θ2 represent the two-layer point-by-point convolution parameters, and the global feature

[0025]

[0026] Step B3: Input the feature spectrum X into the two-layer point-by-point convolution sequence to calculate the local features

[0027]

[0028] Step B4, add the global features and local features in a broadcast form to restore the feature map, and then send it to the Sigmoid function σ to calculate the channel weight W, as shown in formula (5),

[0029]

[0030] Step B5: Use soft selection to select features and fuse related features X corr and spectral feature X spec , as shown in formula (6),

[0031] Z=WWX corr +(IW)WX spec (6)

[0032] Among them, Z represents Batch dimension fusion and I represents the identity matrix.

[0033] The echo cancellation method based on heterogeneous multi-scale convolutional recurrent network mentioned above, step C, builds a long memory network based on the channel attention fusion mechanism, and then uses the long memory network to capture the inter-frame time dependency features and intra-frame spectrum harmonic features, wherein specifically, a bidirectional LSTM layer is used to sequentially scan the inter-frame spectrum and extract the time dependency features between frequency points, and then a unidirectional LSTM layer is used to scan the time dimension and extract the time dependency features in the fusion features. The specific steps are as follows:

[0034] Step C1: Fuse the fusion feature time dimension and batch dimension Input the bidirectional LSTM layer, then output the intermediate features after fully connected FC and LN As shown in formula (7),

[0035] Z F =LN(FC(Bi-LSTM(Z)))+Z(7);

[0036] Step C2: The intermediate feature Z F Transpose Input unidirectional LSTM layer and obtain deep features As shown in formula (8),

[0037]

[0038] Among them, h t represents the hidden layer state at time t.

[0039] In the aforementioned echo cancellation method based on heterogeneous multi-scale convolutional recurrent network, step D constructs a complex filtering paradigm based on the time dependency characteristics between frames and the harmonic characteristics of the spectrum within a frame, and then uses the complex filtering paradigm to reconstruct the signal and output the complex spectrum of the sample speech. Specifically, the real and imaginary parts of the spectrum are masked and the phase response is introduced to improve the speech quality and intelligibility after echo cancellation.

[0040] An echo cancellation system based on a heterogeneous multi-scale convolutional recurrent network includes an encoder construction module, a channel attention fusion module, a long memory network construction module, a signal reconstruction module and a speech synthesis module. The encoder construction module is used to extract relevant feature patterns based on multi-frame contexts to construct a relevant encoder, and then extract spectrum patterns based on a single frame to construct a spectrum encoder. Then, the relevant encoder and the spectrum encoder are used to form a heterogeneous multi-scale encoder and obtain multi-path coding features.

[0041] The channel attention fusion module is used to construct a channel attention fusion mechanism according to multi-path coding features;

[0042] The long memory network building module is used to build a long memory network based on the channel attention fusion mechanism, and then use the long memory network to capture the inter-frame time dependency features and the intra-frame spectrum harmonic features;

[0043] The signal reconstruction module is used to construct a complex filtering paradigm according to the time dependency characteristics between frames and the harmonic characteristics of the spectrum within a frame, and then use the complex filtering paradigm to reconstruct the signal and output the complex spectrum of the sample speech;

[0044] The speech synthesis module is used to restore the time domain waveform by inverse Fourier transform on the complex spectrum of the sample speech, and then use the overlap-addition method to synthesize the entire speech segment and complete the echo cancellation operation.

[0045] The beneficial effects of the present invention are as follows: the echo cancellation method and system based on heterogeneous multi-scale convolutional recurrent network of the present invention first constructs a correlation encoder based on multi-frame context extraction of relevant feature patterns, then constructs a spectrum encoder based on single-frame extracted spectrum patterns, then uses the correlation encoder and the spectrum encoder to form a heterogeneous multi-scale encoder and obtains multi-path coding features, then constructs a channel attention fusion mechanism based on the multi-path coding features, then constructs a long memory network based on the channel attention fusion mechanism, then uses the long memory network to capture inter-frame time dependency features and intra-frame spectrum harmonic features, then constructs a complex filtering paradigm based on the inter-frame time dependency features and intra-frame spectrum harmonic features, then uses the complex filtering paradigm to reconstruct the signal and output the sample speech complex spectrum, finally uses inverse Fourier transform to restore the time domain waveform of the sample speech complex spectrum, and then uses The overlap-addition method synthesizes the entire speech segment and completes the echo cancellation operation; the echo cancellation method and system effectively realize the function of using a heterogeneous two-way encoder to extract different features from the input mixed signal spectrum and the reference signal spectrum, and in order to adapt to the time delay between the reference signal and the mixed signal, correlation coding is used to extract correlation feature expressions from multi-frame spectra, and in order to enhance the spectrum resolution of the current frame, spectrum coding is used to extract spectrum pattern expressions from the current frame, and the function of further using a channel attention feature soft selection mechanism to fuse correlation feature coding and spectrum coding for two-way features is realized; the present invention can reconstruct the coding structure in the convolutional loop model and further propose a multi-way feature fusion mechanism, which can stably and effectively improve the effect of the model while ensuring the model parameter quantity and algorithm complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flow chart of the echo cancellation method based on heterogeneous multi-scale convolutional recurrent network of the present invention;

[0047] Figure 2 It is a schematic diagram of the echo cancellation system based on a heterogeneous multi-scale convolutional recurrent network of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described below in conjunction with the accompanying drawings.

[0049] like Figure 1 As shown, the echo cancellation method based on heterogeneous multi-scale convolutional recurrent network of the present invention comprises the following steps:

[0050] Step A, construct a correlation encoder based on multi-frame context extraction of relevant feature patterns, then construct a spectrum encoder based on single-frame extracted spectrum patterns, then use the correlation encoder and the spectrum encoder to form a heterogeneous multi-scale encoder to obtain multi-path coding features, wherein the correlation encoder is used to extract correlation features from multi-frame input spectra and obtain global spectrum features, the correlation encoder is composed of a stacked 4-layer convolutional network, each convolution layer in the correlation encoder has a length of 3 in the time dimension and a length of 5 in the frequency dimension, the first two convolution layers in the correlation encoder are used to downsample the frequency, the output size of the correlation encoder is half of the input size, and the last two convolution layers in the correlation encoder are equal-size mappings; the spectrum encoder is used to extract spectrum patterns from a single-frame mixed spectrum and obtain local features, the spectrum encoder is composed of a 4-layer convolutional network, and the convolution size of the spectrum encoder in the time dimension is 1. The specific steps of constructing the heterogeneous multi-scale encoder are as follows,

[0051] Step A1: transform the mixed signal spectrum X mic and the reference signal spectrum X ref The concatenation is done in the channel dimension and then input into the related encoder to extract features. The first convolution layer of the related encoder is X i＝0 (t) As shown in formula (1),

[0052]

[0053] Among them, Φ i and * represent the convolution kernel parameters and convolution calculation of the i-th layer of the relevant encoder, respectively. LN represents the normalized LayerNorm. The subsequent layers i=1…3 take the output of the i-1th layer as input, i.e., X i =PReLU(LN(X i-1 *Φ i-1 ));

[0054] Step A2: transform the mixed signal spectrum X mic The input spectrum encoder calculates the spectrum features, as shown in formula (2),

[0055] X i＝0 (t) = PReLU(LN(X mic (t)*Θ i )), i∈[0,1,2,3](2)

[0056] Among them, Θ i Represents the convolution kernel parameters of the ith layer of the spectral encoder.

[0057] Step B, constructing a channel attention fusion mechanism based on multi-channel coding features, specifically, each channel in the output feature spectrum of the correlation encoder and the spectrum encoder is taken as a subband, and then the subband global feature attention and local feature attention are applied to the input dual-channel feature spectrum for feature fusion. The global feature is calculated after the global pooling layer for the average value of all frequency points of each subband, and then the average value of all frequency points of each subband is sent to the sequence composed of two layers of point-by-point convolution for feature mapping. The local feature is sent to the sequence composed of two layers of point-by-point convolution after transposing the subband and frequency point dimensions to calculate the components of the frequency points on different channels. The feature fusion specifically adds the global features and the local features and uses Sigmoid to calculate the weights, and then fuses the two-channel feature spectra in a soft selection manner to complete the construction of the channel attention fusion mechanism. The specific steps are as follows:

[0058] Step B1, add relevant features X point by point corr and spectral feature X spec Thus, the input characteristic spectrum of the fusion module is obtained As shown in formula (3),

[0059] X=X corr +X spec (3);

[0060] Step B2: Input the input feature spectrum X into the pooling layer Calculate the statistical features of each sub-band and then feed it into two layers of point-by-point convolution sequence As shown in formula (4),

[0061]

[0062] Among them, θ1 and θ2 represent the two-layer point-by-point convolution parameters, and the global feature

[0063]

[0064] Step B3: Input the feature spectrum X into the two-layer point-by-point convolution sequence to calculate the local features

[0065]

[0066] Step B4, add the global features and local features in a broadcast form to restore the feature map, and then send it to the Sigmoid function σ to calculate the channel weight W, as shown in formula (5),

[0067]

[0068] Step B5: Use soft selection to select features and fuse related features X corr and spectral feature X spec , as shown in formula (6),

[0069] Z=WWX corr +(IW)WX spec (6)

[0070] Among them, Z represents Batch dimension fusion and I represents the identity matrix.

[0071] Step C, build a long memory network based on the channel attention fusion mechanism, and then use the long memory network to capture the inter-frame time dependency features and intra-frame spectrum harmonic features. Specifically, a bidirectional LSTM layer is used to sequentially scan the inter-frame spectrum and extract the time dependency features between frequency points, and then a unidirectional LSTM layer is used to scan the time dimension and extract the time dependency features in the fusion features. The specific steps are as follows:

[0072] Step C1: Fuse the fusion feature time dimension and batch dimension Input the bidirectional LSTM layer, then output the intermediate features after fully connected FC and LN As shown in formula (7),

[0073] Z F =LN(FC(Bi-LSTM(Z)))+Z(7);

[0074] Step C2: The intermediate feature Z F Transpose Input unidirectional LSTM layer and obtain deep features As shown in formula (8),

[0075]

[0076] Among them, h t represents the hidden layer state at time t.

[0077] Step D, constructing a complex filtering paradigm based on the time-dependent characteristics between frames and the harmonic characteristics of the spectrum within a frame, and then using the complex filtering paradigm to reconstruct the signal and output the complex spectrum of the sample speech, specifically, calculating the masking of the real and imaginary parts of the spectrum and introducing the phase response to improve the speech quality and intelligibility after echo cancellation.

[0078] Step E: Use inverse Fourier transform to restore the time domain waveform of the complex spectrum of the sample speech, and then use the overlap-add method to synthesize the entire speech segment to complete the echo cancellation operation.

[0079] like Figure 2As shown, the echo cancellation system based on the heterogeneous multi-scale convolutional recurrent network includes an encoder construction module, a channel attention fusion module, a long memory network establishment module, a signal reconstruction module and a speech synthesis module, wherein the encoder construction module is used to construct a correlation encoder based on multi-frame context extraction of relevant feature patterns, and then construct a spectrum encoder based on a single-frame extracted spectrum pattern, and then the correlation encoder and the spectrum encoder are used to form a heterogeneous multi-scale encoder and obtain multi-path coding features;

[0080] The channel attention fusion module is used to construct a channel attention fusion mechanism according to multi-path coding features;

[0081] The long memory network building module is used to build a long memory network based on the channel attention fusion mechanism, and then use the long memory network to capture the inter-frame time dependency features and the intra-frame spectrum harmonic features;

[0082] The signal reconstruction module is used to construct a complex filtering paradigm according to the time dependency characteristics between frames and the harmonic characteristics of the spectrum within a frame, and then use the complex filtering paradigm to reconstruct the signal and output the complex spectrum of the sample speech;

[0083] The speech synthesis module is used to restore the time domain waveform by inverse Fourier transform on the complex spectrum of the sample speech, and then use the overlap-addition method to synthesize the entire speech segment and complete the echo cancellation operation.

[0084] To better describe the use effect of the present invention, a specific embodiment of the echo cancellation method and system of the present invention is introduced below. The embodiment is set to conduct comparative experiments on the public data sets AEC Challenge and DNSChallenge; the AEC data set contains 20 hours of recorded samples and 27 hours of simulated samples; and the DNS data set includes 544 hours of speech samples and 181 hours of noise samples.

[0085] In order to generate a training data set, this embodiment randomly samples from AEC and DNS, and synthesizes training sample pairs from signal-to-reply ratios randomly sampled from a uniform distribution of -15dB to 18dB, and finally generates 100 hours of double-talk samples and 30 hours of near-end single-talk samples; the performance indicators corresponding to the test set are shown in Table 1, where the comparison algorithm selects the real-time echo removal model DTLN-AEC and ICASSP baseline, which also publish results on the AEC data set, and the standard convolutional recurrent network CRN; from the performance indicators in Table 1, the model proposed in the present invention has significantly improved the echo removal indicators.

[0086] Table 1. Performance comparison table

[0087]

[0088] In summary, the echo cancellation method and system based on heterogeneous multi-scale convolutional recurrent network of the present invention first constructs a correlation encoder based on multi-frame context extraction of relevant feature patterns, then constructs a spectrum encoder based on single-frame extracted spectrum patterns, then uses the correlation encoder and the spectrum encoder to form a heterogeneous multi-scale encoder and obtains multi-path coding features, then constructs a channel attention fusion mechanism based on the multi-path coding features, then constructs a long memory network based on the channel attention fusion mechanism, and then uses the long memory network to capture inter-frame time dependency features and intra-frame spectrum harmonic features, then constructs a complex filtering paradigm based on the inter-frame time dependency features and intra-frame spectrum harmonic features, then uses the complex filtering paradigm to reconstruct the signal and output the sample speech complex spectrum, finally uses the inverse Fourier transform to restore the time domain waveform of the sample speech complex spectrum, and then uses the overlapping The addition method synthesizes the entire speech segment and completes the echo cancellation operation; the echo cancellation method and system effectively realize the function of using a heterogeneous two-way encoder to extract different features from the input mixed signal spectrum and the reference signal spectrum, and in order to adapt to the time delay between the reference signal and the mixed signal, correlation coding is used to extract correlation feature expressions from multi-frame spectra, and in order to enhance the spectrum resolution of the current frame, spectrum coding is used to extract spectrum pattern expressions from the current frame, and the function of further using a channel attention feature soft selection mechanism to fuse correlation feature coding and spectrum coding for two-way features is realized; the present invention can reconstruct the coding structure in the convolutional loop model and further propose a multi-way feature fusion mechanism, which can stably and effectively improve the effect of the model while ensuring the model parameter quantity and algorithm complexity.

[0089] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An echo cancellation method based on a heterogeneous multi-scale convolutional recurrent network, characterized in that: The following steps are included: Step A, extracting relevant feature patterns based on multi-frame contexts to construct a correlation encoder, then extracting spectrum patterns based on a single frame to construct a spectrum encoder, and then using the correlation encoder and the spectrum encoder to form a heterogeneous multi-scale encoder to obtain multi-path coding features; Step B, construct a channel attention fusion mechanism based on multi-channel encoding features; Step C: construct a long memory network based on the channel attention fusion mechanism, and then use the long memory network to capture the inter-frame temporal dependency features and intra-frame spectral harmonic features; Step D, constructing a complex filtering paradigm according to the time-dependent features between frames and the harmonic features of the spectrum within a frame, and then using the complex filtering paradigm to reconstruct the signal and output the complex spectrum of the sample speech; Step E: Use inverse Fourier transform to restore the time domain waveform of the complex spectrum of the sample speech, and then use the overlap-add method to synthesize the entire speech segment to complete the echo cancellation operation.

2. The echo cancellation method based on heterogeneous multi-scale convolutional recurrent network according to claim 1, characterized in that: Step A, construct a correlation encoder based on multi-frame context extraction of relevant feature patterns, then construct a spectrum encoder based on single-frame extracted spectrum patterns, then use the correlation encoder and the spectrum encoder to form a heterogeneous multi-scale encoder to obtain multi-path coding features, wherein the correlation encoder is used to extract correlation features from multi-frame input spectra and obtain global spectrum features, the correlation encoder is composed of a stacked 4-layer convolutional network, each convolution layer in the correlation encoder has a length of 3 in the time dimension and a length of 5 in the frequency dimension, the first two convolution layers in the correlation encoder are used to downsample the frequency, the output size of the correlation encoder is half of the input size, and the last two convolution layers in the correlation encoder are equal-size mappings; the spectrum encoder is used to extract spectrum patterns from a single-frame mixed spectrum and obtain local features, the spectrum encoder is composed of a 4-layer convolutional network, and the convolution size of the spectrum encoder in the time dimension is 1. The specific steps of constructing the heterogeneous multi-scale encoder are as follows, Step A1: transform the mixed signal spectrum X mic and the reference signal spectrum X ref The concatenation is done in the channel dimension and then input into the related encoder to extract features. The first convolution layer of the related encoder is X i＝0 (t) As shown in formula (1), Among them, Φ i and * represent the convolution kernel parameters and convolution calculation of the i-th layer of the relevant encoder, respectively. LN represents the normalized LayerNorm. The subsequent layers i=1…3 take the output of the i-1th layer as input, i.e., X i =PReLU(LN(X i-1 *Φ i-1 )); Step A2: transform the mixed signal spectrum X mic The input spectrum encoder calculates the spectrum features, as shown in formula (2), X i＝0 (t)6PReLU(LN(X). mic (t)*Θ i )),i∈[0,1,2,3](2) Among them, Θ i Represents the convolution kernel parameters of the ith layer of the spectral encoder.

3. The echo cancellation method based on heterogeneous multi-scale convolutional recurrent network according to claim 2, characterized in that: Step B, constructing a channel attention fusion mechanism based on multi-channel coding features, specifically, each channel in the output feature spectrum of the correlation encoder and the spectrum encoder is taken as a subband, and then the subband global feature attention and local feature attention are applied to the input dual-channel feature spectrum for feature fusion. The global feature is calculated after the global pooling layer for the average value of all frequency points of each subband, and then the average value of all frequency points of each subband is sent to the sequence composed of two layers of point-by-point convolution for feature mapping. The local feature is sent to the sequence composed of two layers of point-by-point convolution after transposing the subband and frequency point dimensions to calculate the components of the frequency points on different channels. The feature fusion specifically adds the global features and the local features and uses Sigmoid to calculate the weights, and then fuses the two-channel feature spectra in a soft selection manner to complete the construction of the channel attention fusion mechanism. The specific steps are as follows: Step B1, add relevant features X point by point corr and spectral feature X spec Thus, the input characteristic spectrum of the fusion module is obtained As shown in formula (3), X=X corr +X spec (3); Step B2: Input the input feature spectrum X into the pooling layer Calculate the statistical features of each sub-band and then feed it into two layers of point-by-point convolution sequence As shown in formula (4), Among them, θ1 and θ2 represent the two-layer point-by-point convolution parameters, and the global feature Step B3: Input the feature spectrum X into the two-layer point-by-point convolution sequence to calculate the local features Step B4, add the global features and local features in a broadcast form to restore the feature map, and then send it to the Sigmoid function σ to calculate the channel weight W, as shown in formula (5), Step B5: Use soft selection to select features and fuse related features X corr and spectral feature X spec , as shown in formula (6), Z=WWX corr +(IW)WX spec (6) Among them, Z represents Batch dimension fusion and I represents the identity matrix.

4. The echo cancellation method based on heterogeneous multi-scale convolutional recurrent network according to claim 3, characterized in that: Step C, build a long memory network based on the channel attention fusion mechanism, and then use the long memory network to capture the inter-frame time dependency features and intra-frame spectrum harmonic features. Specifically, a bidirectional LSTM layer is used to sequentially scan the inter-frame spectrum and extract the time dependency features between frequency points, and then a unidirectional LSTM layer is used to scan the time dimension and extract the time dependency features in the fusion features. The specific steps are as follows: Step C1: Fuse the fusion feature time dimension and batch dimension Input the bidirectional LSTM layer, then output the intermediate features after fully connected FC and LN As shown in formula (7), Step C2: The intermediate feature Z F Transpose Input unidirectional LSTM layer and obtain deep features As shown in formula (8), Among them, h t represents the hidden layer state at time t.

5. The echo cancellation method based on heterogeneous multi-scale convolutional recurrent network according to claim 4, characterized in that: Step D, constructing a complex filtering paradigm based on the time-dependent characteristics between frames and the harmonic characteristics of the spectrum within a frame, and then using the complex filtering paradigm to reconstruct the signal and output the complex spectrum of the sample speech, specifically, calculating the masking of the real and imaginary parts of the spectrum and introducing the phase response to improve the speech quality and intelligibility after echo cancellation.

6. An echo cancellation system based on a heterogeneous multi-scale convolutional recurrent network, wherein the echo cancellation process of the echo cancellation system is based on the echo cancellation method according to any one of claims 1 to 5, characterized in that: It includes an encoder construction module, a channel attention fusion module, a long memory network construction module, a signal reconstruction module and a speech synthesis module. The encoder construction module is used to extract relevant feature patterns based on multi-frame contexts to construct a relevant encoder, and then extract spectrum patterns based on a single frame to construct a spectrum encoder. Then, the relevant encoder and the spectrum encoder are used to form a heterogeneous multi-scale encoder and obtain multi-path coding features. The channel attention fusion module is used to construct a channel attention fusion mechanism according to multi-path coding features; The long memory network building module is used to build a long memory network based on the channel attention fusion mechanism, and then use the long memory network to capture the inter-frame time dependency features and the intra-frame spectrum harmonic features; The signal reconstruction module is used to construct a complex filtering paradigm according to the time dependency characteristics between frames and the harmonic characteristics of the spectrum within a frame, and then use the complex filtering paradigm to reconstruct the signal and output the complex spectrum of the sample speech; The speech synthesis module is used to restore the time domain waveform by inverse Fourier transform on the complex spectrum of the sample speech, and then use the overlap-addition method to synthesize the entire speech segment and complete the echo cancellation operation.

Images