Echo cancellation method and apparatus, electronic device, and storage medium

Abstract

The echo cancellation method, device, electronic equipment and storage medium are provided, wherein the echo cancellation method comprises: obtaining a first remote speech signal and a first near-end speech signal; performing feature extraction on the first remote speech signal and the first near-end speech signal respectively to obtain a first speech feature and a second speech feature; performing time delay alignment on the first remote speech signal and the first near-end speech signal based on the first speech feature and the second speech feature to obtain an aligned second remote speech signal and a second near-end speech signal; and performing echo cancellation on the second near-end speech signal based on the second remote speech signal to obtain a first target near-end speech signal after echo cancellation. Thus, the remote speech signal and the near-end speech signal can be accurately aligned, thereby improving the effect of echo cancellation, and the remote speech can be eliminated from the near-end mixed signal to achieve the purpose of echo cancellation.

Description

Technical Field

[0001] This disclosure relates to the field of communication technology, and in particular to an echo cancellation method, apparatus, electronic device and storage medium. Background Technology

[0002] In recent years, the development of the internet has driven the widespread application of VoIP (Voice over Internet Protocol) technologies such as internet telephony and video conferencing. At the same time, voice quality has become an increasing concern, with one of the most critical factors being echo. Because the sound emitted by the speaker is picked up by the microphone and transmitted back to the distant end, coupled with various delays, the speaker at the remote end can hear their own echo, severely impacting call quality.

[0003] In situations with severe room reverberation and complex environmental noise, relevant echo cancellation methods suffer from inaccurate time delay estimation, making it difficult to align far-end and near-end speech signals, thus affecting the effectiveness of echo cancellation. Summary of the Invention

[0004] This disclosure provides an echo cancellation method, apparatus, electronic device, and storage medium.

[0005] A first aspect of this disclosure provides an echo cancellation method, comprising: acquiring a first far-end speech signal and a first near-end speech signal; extracting features from the first far-end speech signal and the first near-end speech signal respectively to obtain a first speech feature and a second speech feature; performing time delay alignment on the first far-end speech signal and the first near-end speech signal based on the first speech feature and the second speech feature to obtain an aligned second far-end speech signal and a second near-end speech signal; and performing echo cancellation on the second near-end speech signal based on the second far-end speech signal to obtain a first target near-end speech signal after echo cancellation.

[0006] The echo cancellation method of this embodiment acquires a first far-end speech signal and a first near-end speech signal; extracts features from the first far-end speech signal and the first near-end speech signal respectively to obtain a first speech feature and a second speech feature; performs time delay alignment on the first far-end speech signal and the first near-end speech signal based on the first speech feature and the second speech feature to obtain an aligned second far-end speech signal and a second near-end speech signal; and performs echo cancellation on the second near-end speech signal based on the second far-end speech signal to obtain a first target near-end speech signal after echo cancellation. This embodiment extracts features from both the far-end and near-end speech signals respectively, and performs time delay alignment on the far-end and near-end speech signals using the extracted features. This accurately aligns the far-end and near-end speech signals, thereby improving the echo cancellation effect and effectively eliminating far-end speech from the near-end mixed signal to achieve the purpose of echo cancellation.

[0007] A second aspect of this disclosure provides an echo cancellation device, comprising: an acquisition module for acquiring a first far-end speech signal and a first near-end speech signal; a feature extraction module for extracting features from the first far-end speech signal and the first near-end speech signal respectively to obtain a first speech feature and a second speech feature; a time delay alignment module for performing time delay alignment on the first far-end speech signal and the first near-end speech signal based on the first speech feature and the second speech feature to obtain an aligned second far-end speech signal and a second near-end speech signal; and an echo cancellation module for performing echo cancellation on the second near-end speech signal based on the second far-end speech signal to obtain a first target near-end speech signal after echo cancellation.

[0008] A third aspect of this disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the echo cancellation method as described in the first aspect of the present disclosure.

[0009] A fourth aspect of this disclosure provides a computer-readable storage medium storing computer instructions for causing the computer to perform the echo cancellation method as described in the first aspect above.

[0010] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0011] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0012] Figure 1 A schematic flowchart of an echo cancellation method provided in an embodiment of this disclosure;

[0013] Figure 2 A schematic diagram of the principle of echo cancellation;

[0014] Figure 3 A schematic flowchart of an echo cancellation method provided in another embodiment of this disclosure;

[0015] Figure 4 This is a schematic diagram illustrating the principle of latency alignment in attention-based networks.

[0016] Figure 5 A schematic flowchart of an echo cancellation method provided in another embodiment of this disclosure;

[0017] Figure 6 A schematic flowchart of an echo cancellation method provided in another embodiment of this disclosure;

[0018] Figure 7 This is a schematic diagram of the training process of an attention mechanism network.

[0019] Figure 8 A schematic flowchart of an echo cancellation method provided in another embodiment of this disclosure;

[0020] Figure 9 This is a schematic diagram of the principle of state detection and divergence detection;

[0021] Figure 10 A schematic flowchart of an echo cancellation method provided in another embodiment of this disclosure;

[0022] Figure 11 This is a schematic diagram of the training process for a nonlinear echo cancellation model.

[0023] Figure 12 A schematic flowchart of an echo cancellation method provided in another embodiment of this disclosure;

[0024] Figure 13 This is a schematic diagram of the model training iteration process;

[0025] Figure 14 This is a schematic diagram of the structure of an echo cancellation device provided in another embodiment of the present disclosure;

[0026] Figure 15 A block diagram of an electronic device provided according to an embodiment of this disclosure. Detailed Implementation

[0027] Embodiments of this disclosure are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0028] The echo cancellation method, apparatus, electronic device, and storage medium of this disclosure are described below with reference to the accompanying drawings.

[0029] Figure 1 This is a schematic flowchart illustrating an echo cancellation method provided in an embodiment of this disclosure. Figure 1 As shown, the method includes the following steps:

[0030] S101, acquire the first far-end speech signal and the first near-end speech signal.

[0031] The echo cancellation method of this disclosure can be applied to call scenarios such as video calls and voice calls.

[0032] The first far-end voice signal is a signal generated by a far-end device, and the first near-end voice signal is a signal generated by a near-end device. The far-end device and the near-end device can be communication devices such as mobile phones, computers, tablets, and smart TVs, without any restrictions.

[0033] See Figure 2 After the first far-end speech signal x(n) is transmitted to the near-end device, it will be played out through the speaker of the near-end device. After being reflected by surrounding objects to form an echo y(n), it will be received by the microphone of the near-end device. At the same time, the speech s(n) and the ambient noise v(n) will be received by the microphone of the near-end device. After receiving the echo y(n), speech s(n) and noise v(n), the near-end device will generate the first near-end speech signal d(n).

[0034] Wherein, speech s(n) is the audio of the user speaking at the near end, excluding noise and echo.

[0035] S102, feature extraction is performed on the first far-end speech signal and the first near-end speech signal respectively to obtain the first speech feature and the second speech feature.

[0036] After acquiring the first far-end speech signal and the first near-end speech signal, the first far-end speech signal and the first near-end speech signal can be buffered by a buffer. The buffer length is the length of the signal. Assuming that the buffer length of the first far-end speech signal is M, the first far-end speech signal can be divided into M frames of far-end data. Assuming that the buffer length of the first near-end speech signal is N, the first near-end speech signal can be divided into N frames of near-end data.

[0037] For the first far-end speech signal, feature extraction can be performed on each frame of far-end data in the first far-end speech signal to obtain the first speech feature corresponding to each frame of far-end data; for the first near-end speech signal, feature extraction can be performed on each frame of near-end data in the first near-end speech signal to obtain the second speech feature corresponding to each frame of near-end data.

[0038] Among them, the first speech feature and the second speech feature can be high-dimensional features, which are used to calculate the similarity between the far-end data and the near-end data.

[0039] In some implementations, remote data and near-end data are input into a neural network, which then maps the remote data and near-end data into high-dimensional features, namely the first speech feature and the second speech feature.

[0040] S103, based on the first speech feature and the second speech feature, perform time delay alignment on the first far-end speech signal and the first near-end speech signal to obtain the aligned second far-end speech signal and the second near-end speech signal.

[0041] During the process from the first far-end voice signal being played out by the speaker of the near-end device to being received by the microphone of the near-end device and converted into a digital signal, there will be a time delay with the original first far-end signal. That is, there is a time delay between the echo signal in the first near-end voice signal and the first far-end voice signal. In order to eliminate the echo signal in the first near-end voice signal, the first far-end voice signal and the first near-end voice signal must be time-delay aligned.

[0042] In some implementations, the similarity between the first speech feature and the second speech feature can be calculated. Based on the similarity between the first speech feature and the second speech feature, similar far-end data and near-end data can be determined. The similar far-end data and near-end data can be aligned, thereby aligning the first far-end speech signal and the first near-end speech signal to obtain the aligned second far-end speech signal and the second near-end speech signal.

[0043] S104, based on the second far-end speech signal, perform echo cancellation on the second near-end speech signal to obtain the target near-end speech signal after echo cancellation.

[0044] After obtaining the aligned second distal speech signal and second proximal speech signal, echo estimation can be performed on the second distal speech signal to obtain the mixed echo signal in the second proximal speech signal. Based on the echo signal, linear echo cancellation and nonlinear echo cancellation are performed on the second proximal speech signal to obtain the target proximal speech signal after echo cancellation.

[0045] In this embodiment, a first far-end speech signal and a first near-end speech signal are acquired. Feature extraction is performed on both signals to obtain a first speech feature and a second speech feature. Based on these features, time delay alignment is performed on the far-end and near-end speech signals to obtain aligned second far-end and near-end speech signals. This embodiment extracts features from both far-end and near-end speech signals and performs time delay alignment using these extracted features. This accurately aligns the far-end and near-end speech signals, improving echo cancellation and effectively eliminating far-end speech from the near-end mixed signal, thus achieving echo cancellation.

[0046] Figure 3 This is a schematic flowchart of an echo cancellation method provided in an embodiment of the present disclosure, as shown below. Figure 3 As shown, the echo cancellation method also includes the following steps:

[0047] S301, acquire the first far-end speech signal and the first near-end speech signal.

[0048] For details on step S301, please refer to the relevant description of step S101, which will not be repeated here.

[0049] S302, the first far-end speech signal and the first near-end speech signal are input into the trained attention mechanism network, and feature extraction is performed on the first far-end speech signal and the first near-end speech signal respectively to obtain the first speech feature and the second speech feature.

[0050] See Figure 4 The first far-end speech signal and the first near-end speech signal can be input into the trained attention mechanism network. The feature extraction layer in the attention mechanism network will extract features from the first far-end speech signal and the first near-end speech signal respectively to obtain the corresponding first speech features and second speech features.

[0051] S303, based on the first speech feature and the second speech feature, the time delay of the first far-end speech signal and the first near-end speech signal is aligned to obtain the aligned second far-end speech signal and the second near-end speech signal.

[0052] For details on step S303, please refer to the relevant description in step S103, which will not be repeated here.

[0053] S304, based on the second far-end speech signal, perform echo cancellation on the second near-end speech signal to obtain the first target near-end speech signal after echo cancellation.

[0054] For details on step S303, please refer to the relevant description in step S104, which will not be repeated here.

[0055] In this embodiment, a first far-end speech signal and a first near-end speech signal are acquired. These signals are then input into a trained attention mechanism network. Feature extraction is performed on both signals to obtain a first speech feature and a second speech feature. Based on these features, time-delay alignment is performed on the far-end and near-end speech signals to obtain aligned second far-end and near-end speech signals. Finally, echo cancellation is performed on the near-end speech signal based on the second far-end speech signal to obtain a first target near-end speech signal. This embodiment uses an attention mechanism network to perform time-delay alignment on the far-end and near-end speech signals, which accurately aligns them, thereby improving the echo cancellation effect on the near-end speech signal.

[0056] Figure 5 This is a flowchart illustrating an echo cancellation method provided in one embodiment of the present disclosure. Based on the above embodiment, it further incorporates... Figure 5 The process of time delay alignment of the first far-end speech signal and the first near-end speech signal based on the first speech feature and the second speech feature is explained and described, including the following steps:

[0057] S501, obtain the similarity between the first speech feature and the second speech feature.

[0058] Optionally, see Figure 4 The first far-end speech signal and the first near-end speech signal are input into the trained attention mechanism network. The feature extraction layer in the attention mechanism network extracts features from the first far-end speech signal and the first near-end speech signal respectively to obtain the first speech feature and the second speech feature. The fully connected layer in the attention mechanism network processes the first speech feature and the second speech feature to obtain the similarity between the first speech feature and the second speech feature.

[0059] Among them, the fully connected layer in the attention mechanism network can calculate the similarity between the first speech feature and the second speech feature to obtain the similarity between the first speech feature and the second speech feature. Then, the initial similarity between the first speech feature and the second speech feature is normalized to obtain the normalized similarity. The calculation process can be found in the following formula (1).

[0060]

[0061] Where, α m,nLet be the normalized similarity between the first speech feature corresponding to the m-th frame of the first far-end speech signal and the second speech feature corresponding to the n-th frame of the first near-end speech signal, where m is the number of frames in the first far-end speech signal and n is the number of frames in the first near-end speech signal. Let F be the first speech feature corresponding to m frames of far-end data. For the second speech feature (N) of the nth frame of near-end data, Score represents the value of the second speech feature (N). and To perform similarity calculations, exp represents an exponential function with the natural constant e as its base.

[0062] S502, based on similarity, perform time delay alignment between the first far-end speech signal and the first near-end speech signal.

[0063] Optionally, the target first speech feature and target second speech feature corresponding to the maximum similarity are determined, and the target far-end data corresponding to the target first speech feature is aligned with the target near-end data corresponding to the target second speech feature.

[0064] See Figure 4 The first far-end speech signal includes M frames of far-end data, and the first near-end speech signal includes N frames of near-end data. The similarity between each frame of far-end data in the M frames of far-end data and one frame of near-end data can be calculated sequentially to obtain M similarity scores. Then, the maximum similarity score can be determined from the M similarity scores. The far-end data and near-end data corresponding to the maximum similarity score are the target far-end data and the target near-end data, and the target far-end data and the target near-end data can be aligned.

[0065] For example, by sequentially calculating the similarity between each frame of far-end data in the M frames and the first frame of near-end data, we can obtain M similarity scores, denoted as α. 1,1 α 2,1 , ..., α m,1 If α 2,1 To achieve the highest similarity, the second frame of far-end data in the first far-end speech signal is most similar to the first frame of near-end data in the first near-end speech signal. Therefore, the second frame of far-end data is the target far-end data, and the first frame of near-end data is the target near-end data. The second frame of far-end data in the first far-end speech signal can be aligned with the first near-end data in the first near-end speech signal.

[0066] In some implementations, the frame indices of the target far-end data and the target near-end data are obtained. Based on the frame indices, the time difference between the target far-end data and the target near-end data is determined. Based on the time difference, the target far-end data and the target near-end data are aligned.

[0067] Based on the time difference between the target far-end data and the target near-end data, the data buffers of the target far-end data and the target near-end data can be shifted respectively, thereby aligning the target far-end data and the target near-end data. In this way, each frame of near-end data in the first near-end speech signal can be aligned with each frame of far-end data in the first far-end speech signal, thereby aligning the first far-end speech signal and the first near-end speech signal.

[0068] Furthermore, the similarity α can be calculated using the following formula (2). m,n 32-bit quantization is performed to obtain T m,n Then, it calls an Application Programming Interface (API), such as WebReal-Time Communication (WebRTC), and uses WebRTC to determine the quantization result T. m,n The maximum similarity is determined, and the target far-end data and target near-end data corresponding to the maximum similarity are determined. Then, the data buffers of the target far-end data and the target near-end data are shifted to align the target far-end data and the target near-end data, thereby aligning the first far-end speech signal and the first near-end speech signal.

[0069] T m,n =int(α) m,n ·2 32 (2)

[0070] Among them, T m,n For similarity α m,n The 32-bit quantized value, the int() function is used to convert a number into an integer.

[0071] In this embodiment of the disclosure, the similarity between a first speech feature and a second speech feature is obtained, and based on the similarity, the first far-end speech signal and the first near-end speech signal are time-delay aligned. This embodiment of the disclosure determines similar far-end and near-end data by using the similarity between speech features, improving the accuracy of determining similar data from far-end and near-end speech signals, thereby enabling accurate alignment of far-end and near-end speech signals.

[0072] In summary, as Figure 4As shown, when aligning the first near-end speech signal with the first far-end speech signal in terms of time delay, the feature extraction layer of the attention mechanism network sequentially extracts features from the far-end data of each frame in the first near-end speech signal to obtain a first speech feature for each frame of far-end data. The feature extraction layer of the attention mechanism network then extracts features from the nth frame of the first near-end speech data to obtain the corresponding second speech feature. Next, the fully connected layer of the attention mechanism network calculates the similarity between each first speech feature and the second speech feature, and normalizes the calculated similarity. Then, the normalized similarity is quantized using 32 bits to obtain a 32-bit quantized value. Finally, WebRTC is called to determine the far-end data in the first near-end speech signal that is similar to the near-end data of the nth frame based on the 32-bit quantized value, and this far-end data is aligned with the near-end data of the nth frame. In this way, similar data in each frame of the first far-end speech signal and the first near-end speech signal can be aligned, thus aligning the first far-end speech signal and the first near-end speech signal.

[0073] Figure 6 This is a schematic flowchart of an echo cancellation method provided in an embodiment of the present disclosure, as shown below. Figure 6 As shown, the training process of the attention mechanism network includes the following steps:

[0074] S601, collect the first sample far-end speech signal and the first sample near-end speech signal.

[0075] See Figure 7 Two audio recordings from different speakers are randomly selected from a clean (noise-free) audio library. One audio recording is used as the far-end speech signal, which is the first sample far-end speech signal. The other audio recording is used as the near-end speech signal. A type of noise is randomly selected from a noise library, and the noise signal power is adjusted according to a set signal-to-noise ratio. The noise signal is then superimposed on the near-end speech signal. Two room impulse responses are then randomly selected from a room impulse response simulator. These room impulse responses can be artificially simulated or actually acquired, and the settings are determined based on the relative distances between the microphone, speaker, and speaker. These two room impulse responses are then superimposed on the near-end and far-end speech signals respectively, with the noise signal superimposed. The near-end and far-end speech signals are then superimposed according to a set power ratio (randomly set within a certain range, such as -10dB to 30dB). Finally, the superimposed signal is time-delayed to obtain the first sample near-end speech signal.

[0076] The system allows users to control whether to overlay near-end audio signals via a switch, simulating a real-world one-way conversation. Furthermore, within the same room configuration, the relative positions of the speaker and speaker relative to the microphone can be adjusted according to the application scenario. For example, with handheld devices, the distance between the speaker and microphone can be set relatively small; in large-screen conference scenarios, the distance can be set relatively large.

[0077] It should be noted that when selecting a room impulse response from the room impulse response simulator, it should be randomly selected from the same room configuration.

[0078] S602, input the first sample far-end speech signal and the first sample near-end speech signal into the initial attention mechanism network to generate the predicted similarity between the first sample far-end data of the first sample far-end speech signal and the first sample near-end data of the first sample near-end speech signal.

[0079] After the first sample far-end speech signal and the first sample near-end speech signal are input into the initial attention mechanism network, the feature extraction layer in the initial attention mechanism network extracts features from each frame of data in the first sample far-end speech signal and the first sample near-end speech signal, respectively, to obtain the first speech feature and the second speech feature. Then, the fully connected layer in the initial attention mechanism network processes the first speech feature and the second speech feature to obtain the similarity between the first speech feature and the second speech feature. This similarity is the predicted similarity between the first sample far-end data of the first sample far-end speech signal and the first sample near-end data of the first sample near-end speech signal.

[0080] S603, based on predicted similarity and label similarity, adjusts the model parameters of the initial attention mechanism network to obtain the trained attention mechanism network.

[0081] See Figure 7 The correlation between the far-end data of the first sample far-end signal and the near-end data of the first sample near-end speech signal can be calculated to obtain the correlation between the near-end data of the first sample near-end speech signal and the far-end speech signal of the first sample. This correlation is used as the label similarity. Then, the error between the label similarity and the predicted similarity is obtained. Based on this error, the model parameters of the initial attention mechanism network are adjusted to obtain the trained attention mechanism network.

[0082] In this embodiment, a first sample far-end speech signal and a first sample near-end speech signal are acquired. These two signals are then input into an initial attention mechanism network to generate a predicted similarity between a first speech feature and a second speech feature. Based on the predicted similarity and label similarity, the model parameters of the initial attention mechanism network are adjusted to obtain a trained attention mechanism network. Training the attention mechanism network in this embodiment improves the accuracy of similarity calculation, thereby improving the accuracy of time delay estimation and ultimately, the accuracy of time delay alignment.

[0083] Figure 8 This is a flowchart illustrating an echo cancellation method provided in one embodiment of the present disclosure. Based on the above embodiment, it further incorporates... Figure 8 The echo cancellation process for the second near-end speech signal is explained, including the following steps:

[0084] S801 performs echo estimation on the second far-end speech signal to obtain the echo signal.

[0085] Optionally, the second far-end speech signal is input into an adaptive filter to generate an echo signal.

[0086] The adaptive filter can employ any digital adaptive filtering algorithm or an adaptive algorithm that achieves similar functionality, such as the multi-delay filtering algorithm in Speex (an open-source speech algorithm library) or the block frequency domain adaptive filtering algorithm in WebRTC, etc. No restrictions are imposed here.

[0087] In some implementations, the echo signal is calculated using the following formula (3).

[0088]

[0089] in, For the nth frame, the echo signal with frequency component k, w k (l) represents the adaptive filter weight parameters for the k-th frequency component in the l-th frame, L is the length of the adaptive filter, l is the frame number index of the adaptive filter, and x k (nl) is the second far-end speech signal of the k-th frequency component in the nl-th frame.

[0090] In this embodiment of the disclosure, the voice call states between the remote device and the near device include far-end state, near-end state, dual-end state, and dual-silent state. The far-end state is when only the far end speaks, the near-end state is when only the near end speaks, the dual-end state is when both the far end and the near end speak simultaneously, and the dual-silent state is when neither the far end nor the near end speaks.

[0091] See Figure 9 The voice call status between the remote and near-end devices can be determined based on the energy of the second far-end voice signal and the third near-end voice signal, i.e., status detection. If the energy of the second far-end voice signal is greater than or equal to a first energy threshold, and the energy of the third near-end voice signal is less than the second energy threshold, then the voice call status between the remote and near-end devices is far-end talk status; if the energy of the second far-end voice signal is less than the first energy threshold, and the energy of the third near-end voice signal is greater than or equal to the second energy threshold, then the voice call status between the remote and near-end devices is near-end talk status; if the energy of the second far-end voice signal is greater than or equal to the first energy threshold, and the energy of the third near-end voice signal is greater than or equal to the second energy threshold, then the voice call status between the remote and near-end devices is two-way talk status; if the energy of the second far-end signal is less than the first energy threshold, and the energy of the third near-end voice signal is less than the second energy threshold, then the voice call status between the remote and near-end devices is two-way silent status.

[0092] It should be noted that the first and second energy thresholds can be set according to the actual situation, and no restrictions are imposed here.

[0093] When performing echo estimation on the second far-end speech signal using an adaptive filter, the voice call status between the far-end and near-end devices can be detected. If the voice call status between the far-end and near-end devices is far-reaching, then the weight parameter w of the adaptive filter needs to be updated. k (l); If the voice call status between the remote device and the near device is a non-far-talking state (near-talking state, two-way talking state, or two-way silent state), then do not update the weight parameters of the adaptive filter.

[0094] In some implementations, the update step size of the adaptive filter can be obtained, and the weight parameters of the adaptive filter can be updated based on the update step size. The updated weight parameters of the adaptive filter can be calculated using the following formula (4).

[0095]

[0096] Among them, w k (l, i+1) represents the adaptive filter weight parameters for the k-th frequency component in the l-th frame at the (i+1)-th update, w k (l, i) represents the adaptive filter weight parameters for the k-th frequency component in the l-th frame during the i-th update, x k * (nl) represents x k The conjugate of (nl), μ is the step size of the adaptive filter, and i is a natural number.

[0097] Optionally, the optimal step size of the adaptive filter is selected to update the weight parameters of the adaptive filter. The optimal step size of the adaptive filter can be calculated by the following formula (5).

[0098]

[0099] Where, μ opt (k,l) represents the optimal step size of the adaptive filter for the k-th frequency component in the l-th frame. Let represent the variance of the nonlinear (residual) echo signal of the k-th frequency component in the l-th frame. This represents the variance of the third near-end speech signal of the k-th frequency component in the l-th frame.

[0100] in, It can be calculated using the following formula (6).

[0101]

[0102] in, Let represent the variance of the echo signal of the k-th frequency component in the l-th frame. Leakage coefficient, It can be calculated using the following formula (7).

[0103]

[0104] Among them, R EY (k, l) represents the cross-correlation function between the third proximal speech signal and the echo signal, R YY (k, l) represents the autocorrelation function of the echo signal.

[0105] Further, see Figure 9 The adaptive filter can be tested for divergence by measuring the energy of the third and second near-end speech signals. If the energy of the third near-end speech signal is greater than that of the second near-end speech signal, the adaptive filter diverges; if the energy of the third near-end speech signal is less than that of the second near-end speech signal, the adaptive filter does not diverge. In response to the detection of adaptive filter divergence, the backup adaptive filter is switched to perform echo estimation on the second far-end speech signal.

[0106] S802, based on the echo signal, performs echo cancellation on the second near-end speech signal to obtain the first target near-end speech signal.

[0107] In some implementations, linear echo cancellation is performed on the second near-end speech signal based on the echo signal to obtain a third near-end speech signal after linear echo cancellation. Then, nonlinear echo cancellation is performed on the third near-end speech signal based on the echo signal to obtain a first target near-end speech signal after nonlinear echo cancellation.

[0108] The process of linear echo cancellation: The third near-end speech signal after linear echo cancellation can be calculated using the following formula (8).

[0109]

[0110] Among them, e k (n) represents the third near-end speech signal of the nth frame and the kth frequency component, d k (n) is the second near-end speech signal of the k-th frequency component in the nth frame.

[0111] The process of nonlinear echo cancellation is as follows: The echo signal and the third near-end speech signal are input into the trained target nonlinear echo cancellation model to generate the first spectral mask. Based on the first spectral mask and the third near-end speech signal, the first target near-end speech signal after nonlinear echo cancellation is obtained.

[0112] Among them, the target nonlinear echo cancellation model is a model obtained by further training the pre-trained nonlinear model.

[0113] Wherein, the first spectral mask refers to the user speech signal in the second near-end speech signal at each frequency point (frequency component) (see...). Figure 2 The ratio of the energy of s(n) in the second proximal speech signal to the energy of the second proximal speech signal.

[0114] In some implementations, the mask value of the first spectral mask at each frequency component is multiplied by the third near-end speech signal of the corresponding frequency component to obtain the first target near-end speech signal after eliminating nonlinear echoes.

[0115] In this embodiment, the microphone and nonlinear devices (such as amplifiers) in the near-end device introduce nonlinear echoes when processing near-end speech signals. At the same time, the adaptive filter also introduces residual echoes when performing echo estimation. Related technologies can effectively eliminate linear echoes and suppress nonlinear echoes to a certain extent, but the degree of suppression of nonlinear echoes is limited. This embodiment uses a trained nonlinear echo cancellation model to perform nonlinear echo cancellation on near-end speech signals, which can effectively eliminate nonlinear echoes in near-end speech signals and improve the effect of nonlinear echo cancellation.

[0116] Figure 10 This is a flowchart illustrating an echo cancellation method provided in one embodiment of the present disclosure. Based on the above embodiment, it further incorporates... Figure 10 The process of iteratively training the pre-trained nonlinear echo cancellation model to obtain the trained target nonlinear echo cancellation model is explained below, including the following steps:

[0117] S1001, Collect the second sample far-end speech signal and the second sample near-end speech signal.

[0118] In some implementations, see Figure 11 First, two different audio signals from different speakers are randomly selected from a clean (noise-free) audio library: Audio Signal 1 and Audio Signal 2. Audio Signal 1 serves as the second sample far-end speech signal, and Audio Signal 2 serves as the input near-end speech signal. Then, two room impulse responses are randomly selected from a room impulse response simulator: Room Impulse Response 1 and Room Impulse Response 2. The room impulse responses can be artificially simulated or actually acquired, and are set according to the relative distances between the microphone, speaker, and speaker. Room Impulse Response 1 is convolved with Audio Signal 1, and Room Impulse Response 2 is convolved with Audio Signal 2. Then, the power of the convolved Audio Signal 1 and Audio Signal 2 is adjusted according to a set power ratio (randomly set within a certain range, such as -10dB to 30dB). Finally, the adjusted Audio Signal 1 and Audio Signal 2 are superimposed to obtain a near-end speech signal mixed with echo.

[0119] Further, a noise is randomly selected from the noise library, and the power of the original audio signal 1 is adjusted according to the set signal-to-noise ratio (relative to the power of the near-end speech signal). Then, the original audio signal 1 and the noise signal with adjusted power are superimposed to obtain the near-end mixed speech signal. Finally, the near-end mixed speech signal is delayed according to a certain time size to obtain the second sample near-end speech signal.

[0120] Because the near-end speech signal of the second sample underwent time delay processing, there is a time delay between the acquired far-end speech signal of the second sample and the near-end speech signal of the second sample. Therefore, it is necessary to perform time delay alignment processing on the near-end and far-end signals of the second sample. (See...) Figure 11 The time delay between the second sample near-end speech signal and the second sample far-end signal can be estimated by using an attention mechanism network. Then, based on this time delay, the second sample near-end speech signal and the second sample far-end signal are buffered and shifted to align their time delays, resulting in aligned second sample near-end speech signals and second sample far-end speech signals.

[0121] In some implementations, the near-end speech signal with echo and noise are superimposed to obtain a near-end mixed signal, and then the near-end mixed signal is used as the second sample near-end speech signal. That is, no time delay processing is performed on the near-end mixed signal. In this case, there is no time delay between the second sample far-end speech signal and the second sample near-end speech signal, and there is no need to perform time delay alignment processing on the second sample far-end speech signal and the second sample near-end speech signal.

[0122] S1002, the second sample far-end speech signal is input into the adaptive filter to generate the target echo signal.

[0123] The target echo signal is obtained by echo estimation of the far-end speech signal of the second sample.

[0124] For details on the echo estimation process of the far-end speech signal of the second sample, please refer to the relevant introduction in step S801, which will not be repeated here.

[0125] S1003, based on the target echo signal, perform linear echo cancellation on the second sample near-end speech signal to obtain the second target near-end speech signal after linear echo cancellation.

[0126] The second target near-end speech signal is the speech signal after linear echo cancellation of the second sample near-end speech signal.

[0127] For a description of step S1003, please refer to the relevant introduction of linear echo cancellation in step S802, which will not be repeated here.

[0128] S1004, based on the target echo signal, the second target near-end speech signal and the target weight parameters of the adaptive filter, iteratively trains the pre-trained nonlinear echo cancellation model to obtain the trained target nonlinear echo cancellation model.

[0129] The target weight parameter of the adaptive filter is the weight parameter used by the adaptive filter for the next echo estimation.

[0130] The target echo signal and the second target near-end speech signal are processed based on the pre-trained nonlinear echo cancellation model to obtain the predicted near-end speech signal. Then, a loss function is obtained based on the predicted near-end speech signal and the second sample near-end speech signal. The pre-trained nonlinear echo cancellation model is adjusted based on the loss function, and the target weight parameters of the adaptive filter for the next training are obtained based on the predicted near-end speech signal. The next echo estimation is performed based on the target weight parameters, and the next model training is carried out. The entire iterative training process ends when the convergence of the loss function reaches the minimum value or the signal-to-noise ratio of the predicted near-end speech signal reaches the maximum value, and the trained target nonlinear echo cancellation model is obtained.

[0131] The predicted near-end speech signal is the speech signal after removing linear and nonlinear echoes from the second sample near-end speech signal.

[0132] Alternatively, the loss function can be expressed as the following formula (9):

[0133]

[0134] in, Let s(n) be the predicted near-end speech signal of n frames, and s(n) be the user speech signal (i.e., audio signal 2) of n frames. Indicates prediction of near-end speech signals The loss function between the near-end speech signal s(n) (i.e., audio signal 2) and N, where N represents the signal length.

[0135] In this embodiment, a second sample far-end speech signal and a second sample near-end speech signal are acquired. The second sample far-end speech signal is input into an adaptive filter to generate a target echo signal. Based on the target echo signal, linear echo cancellation is performed on the second sample near-end speech signal to obtain a second target near-end speech signal after linear echo cancellation. Based on the target echo signal, the second target near-end speech signal, and the target weight parameters of the adaptive filter, a nonlinear echo cancellation model is iteratively trained to obtain a trained target nonlinear echo cancellation model. This embodiment combines the adaptive filter and the nonlinear echo cancellation model for multiple iterative trainings, which can improve the nonlinear echo cancellation effect of the nonlinear echo cancellation model, thereby effectively eliminating nonlinear echoes in the near-end speech signal.

[0136] Figure 12 This is a flowchart illustrating an echo cancellation method according to an embodiment of the present disclosure. Based on the above embodiment, the process of iteratively training the pre-trained nonlinear echo cancellation model is explained, including the following steps:

[0137] S1201, the target echo signal and the near-end speech signal of the second target are input into the nonlinear echo cancellation model to generate the second spectral mask.

[0138] It should be noted that the nonlinear echo cancellation model in this embodiment is a pre-trained nonlinear echo cancellation model.

[0139] The second spectral mask refers to the ratio of the energy of the user speech signal s(n) (i.e., audio signal 2) to the energy of the second target near-end speech signal at each frequency point (frequency component).

[0140] For a description of step S1201, please refer to the relevant introduction in step S802, which will not be repeated here.

[0141] S1202, based on the second spectrum mask and the second target near-end speech signal, obtain the predicted near-end speech signal.

[0142] For a description of steps S1201 and S1202, please refer to the relevant introduction in step S802, which will not be repeated here.

[0143] S1203, based on the predicted near-end speech signal, obtains the target weight parameters of the adaptive filter for the next training.

[0144] Based on the energy of the second sample far-end speech signal and the second sample near-end speech signal, the voice call state between the far-end device and the near-end device is determined. In response to the voice call state being far-end speech, the target weight parameters of the adaptive filter are obtained based on the predicted near-end speech signal. In response to the voice call state being non-far-end speech, the current weight parameters of the adaptive filter are used as the target weight parameters.

[0145] See Figure 11 During the iterative training of the nonlinear echo cancellation model, state detection can be performed to determine the voice call state between the remote and near devices. If the voice call state is a far-field state, the target weight parameters of the adaptive filter for the next training are obtained based on the predicted near-field voice signal. If the voice call state is a non-far-field state (near-field state, dual-field state, or dual-silent state), the current weight parameters of the adaptive filter are used as the target weight parameters of the adaptive filter for the next training.

[0146] In some implementations, when the voice call between the remote device and the near device is in the far-reach state, the optimal step size of the adaptive filter for the next training can be calculated using the following formula (10).

[0147]

[0148] in, Let be the variance of the predicted near-end speech signal of the k-th frequency component in the l-th frame.

[0149] Furthermore, after calculating the optimal step size of the adaptive filter for the next training using the above formula (10), the target weight parameters of the adaptive filter for the next training can be calculated using the above formula (4).

[0150] In some implementations, the training speed of the nonlinear echo cancellation model is reduced because the input of the nonlinear echo cancellation model at the current moment depends on the optimal step size calculated at the previous moment during the iterative training process. In order to improve the training speed, the average step size (batch) can be used to correct the step size of the adaptive filter. That is, when each batch is trained, the power of the predicted near-end speech signal is averaged within the batch. Similarly, the power of the nonlinear echo of the adaptive filter within the batch is averaged. Then, the two average powers are calculated according to formula (10) to obtain the average optimal step size within the batch. The average optimal step size is used as the step size of the adaptive filter when training the next batch.

[0151] Assuming each batch has M samples, and the average nonlinear echo power of each sample is... The average predicted near-end speech signal power for each sample is The average optimal step size of M samples in a batch can be calculated using the following formula (11):

[0152]

[0153] Here, k represents the k-th batch, and each batch contains M samples. Therefore, in the k-th batch, the sample indices range from kM to kM+M-1.

[0154] Figure 13 A schematic diagram of the model training iteration process, such as... Figure 13 As shown, the optimal step size calculated in the (k-1)th batch is used for the adaptive filter calculation in the kth batch.

[0155] S1204, based on the target weight parameters, performs the next training of the nonlinear echo cancellation model.

[0156] The weight parameters of the adaptive filter are updated to the target weight parameters, and the second sample far-end signal used for the next training is input into the updated adaptive filter to generate the target echo signal for the next training. Based on the target echo signal, linear echo cancellation is performed on the second sample near-end speech signal used for the next training to obtain the second target near-end speech signal used for the next training. Based on the target echo signal and the second target near-end speech signal, the nonlinear echo cancellation model is trained for the next time.

[0157] Further, see Figure 11 The process of pre-training the nonlinear echo cancellation model is explained below:

[0158] like Figure 11As shown, a second sample far-end speech signal and a second sample near-end speech signal are collected for pre-training of the nonlinear echo cancellation model. If there is a time delay between the collected second sample far-end speech signal and the second sample near-end speech signal, the second sample far-end speech signal and the second sample near-end speech signal are time-delay aligned based on the attention mechanism network. The aligned second sample far-end speech signal is then input into an adaptive filter to perform echo estimation and obtain the echo signal.

[0159] Linear echo cancellation is performed on the second sample near-end speech signal based on the echo signal to obtain the second target near-end speech signal. The echo signal and the second target near-end speech signal are input into the nonlinear echo cancellation model to be trained to generate a spectral mask. The mask value at each frequency component of the spectral mask is multiplied with the second target near-end speech signal at the corresponding frequency component to obtain the predicted near-end speech signal. The model parameters of the nonlinear echo cancellation model to be trained are adjusted according to the error between the predicted near-end speech signal and the user speech signal (audio signal 2).

[0160] After adjusting the model parameters of the nonlinear echo cancellation model to be trained, the target step size of the adaptive filter for the next training is obtained based on the second target near-end speech signal, and the target weight parameters of the adaptive filter for the next training are obtained based on the target step size.

[0161] It should be noted that during the training process, the state of the voice call between the remote device and the near device can be detected. If the voice call between the remote device and the near device is detected as a far-field state, the weight parameters of the adaptive filter are obtained based on the second target near-field voice signal. If the voice call between the remote device and the near device is detected as a non-far-field state (near-field state, two-way state, or two-way silent state), the current weight parameters of the adaptive filter are used as the target weight parameters.

[0162] When training the nonlinear echo cancellation model to be trained for the next iteration, echo estimation is performed on the far-end speech signal of the second sample used for the next training based on the target weight parameters of the adaptive filter to obtain the target echo signal for the next training. Then, nonlinear echo cancellation is performed on the near-end speech signal of the second sample used for the next training based on the target echo signal to obtain the second target near-end speech signal for the next training. Based on the target echo signal and the second target near-end speech signal, the nonlinear echo cancellation model to be trained is trained for the next training. After the entire iterative training process is completed, the pre-trained nonlinear echo cancellation model is obtained.

[0163] In this embodiment, the target echo signal and the second target near-end speech signal are input into the nonlinear echo cancellation model to generate a second spectral mask. Based on the second spectral mask and the second target near-end speech signal, a predicted near-end speech signal is obtained. Based on the predicted near-end speech signal, target weight parameters for the adaptive filter used in the next training iteration are obtained. Based on the target weight parameters, the nonlinear echo cancellation model is trained for the next iteration. This embodiment enables iterative training of the nonlinear echo cancellation model across multiple rounds, ensuring sufficient training of the model and thereby improving its nonlinear echo cancellation performance.

[0164] To achieve the above embodiments, this disclosure also proposes an echo cancellation device. Figure 14 This is a schematic diagram of the structure of an echo cancellation device according to an embodiment of this disclosure. Figure 14 As shown, the echo cancellation device 1400 includes:

[0165] Acquisition module 1410 is used to acquire a first far-end speech signal and a first near-end speech signal;

[0166] The feature extraction module 1420 extracts features from the first far-end speech signal and the first near-end speech signal respectively to obtain the first speech features and the second speech features.

[0167] The time delay alignment module 1430 is used to perform time delay alignment on the first far-end speech signal and the first near-end speech signal based on the first speech feature and the second speech feature to obtain the aligned second far-end speech signal and the second near-end speech signal.

[0168] The echo cancellation module 1440 is used to perform echo cancellation on the second near-end speech signal based on the second far-end speech signal to obtain the first target near-end speech signal after echo cancellation.

[0169] In one embodiment of this disclosure, the time delay alignment module 1430 is further configured to obtain the similarity between the first speech feature and the second speech feature; and to perform time delay alignment on the first far-end speech signal and the first near-end speech signal based on the similarity.

[0170] In one embodiment of this disclosure, the time delay alignment module 1430 is further configured to determine the target first speech feature and the target second speech feature corresponding to the maximum similarity; and align the target far-end data corresponding to the target first speech feature with the target near-end data corresponding to the target second speech feature.

[0171] In one embodiment of this disclosure, the delay alignment module 1430 is further configured to acquire frame indices of target far-end data and target near-end data; determine the time difference between target far-end data and target near-end data based on the frame indices; and align the target far-end data and target near-end data based on the time difference.

[0172] In one embodiment of this disclosure, the echo cancellation module 1440 is further configured to perform echo estimation on the second far-end speech signal to obtain an echo signal; and perform echo cancellation on the second near-end speech signal based on the echo signal to obtain a first target near-end speech signal.

[0173] In one embodiment of this disclosure, the echo cancellation module 1440 is further configured to perform linear echo cancellation on the second near-end speech signal based on the echo signal to obtain a third near-end speech signal after linear echo cancellation; and perform nonlinear echo cancellation on the third near-end speech signal based on the echo signal to obtain a first target near-end speech signal after nonlinear echo cancellation.

[0174] In one embodiment of this disclosure, the time delay alignment module 1430 is further configured to input the first far-end speech signal and the first near-end speech signal into the pre-trained initial attention mechanism network; the feature extraction layer in the target attention mechanism network extracts features from the first far-end speech signal and the first near-end speech signal respectively to obtain the first speech feature and the second speech feature; the fully connected layer in the attention mechanism network processes the first speech feature and the second speech feature to obtain the similarity between the first speech feature and the second speech feature.

[0175] In one embodiment of this disclosure, the echo cancellation device 1400 further includes a first training module 1450, which is used to acquire a first sample far-end speech signal and a first sample near-end speech signal; input the first sample far-end speech signal and the first sample near-end speech signal into an attention mechanism network to generate a predicted similarity between a first speech feature and a second speech feature; and adjust the model parameters of the initial attention mechanism network based on the predicted similarity and the label similarity to obtain the trained attention mechanism network.

[0176] In one embodiment of this disclosure, the echo cancellation module 1440 is further configured to input the echo signal and the third near-end speech signal into a trained target nonlinear echo cancellation model to generate a first spectral mask; and to obtain a first target near-end speech signal based on the first spectral mask and the third near-end speech signal.

[0177] In one embodiment of this disclosure, the echo cancellation device 1400 further includes a second training module 1460, which is used to acquire a second sample far-end speech signal and a second sample near-end speech signal; input the second sample far-end speech signal into an adaptive filter to generate a target echo signal; perform linear echo cancellation on the second sample near-end speech signal based on the target echo signal to obtain a second target near-end speech signal after linear echo cancellation; and perform iterative training on a nonlinear echo cancellation model based on the target echo signal, the second target near-end speech signal, and the target weight parameters of the adaptive filter to obtain a trained target nonlinear echo cancellation model.

[0178] In one embodiment of this disclosure, the second training module 1460 is further configured to input the target echo signal and the second target near-end speech signal into the nonlinear echo cancellation model to generate a second spectral mask; obtain a predicted near-end speech signal based on the second spectral mask and the second target near-end speech signal; obtain target weight parameters for the adaptive filter used for the next training based on the predicted near-end speech signal; and perform the next model training on the nonlinear echo cancellation model based on the target weight parameters.

[0179] In one embodiment of this disclosure, the second training module 1460 is further configured to determine the voice call state between the remote device and the near device based on the energy of the second sample far-end speech signal and the second sample near-end speech signal; in response to the voice call state being a far-end speech state, to obtain the target weight parameters of the adaptive filter based on the predicted near-end speech signal; and in response to the voice call state being a non-far-end speech state, to use the current weight parameters of the adaptive filter as the target weight parameters.

[0180] In one embodiment of this disclosure, the second training module 1460 is further configured to update the weight parameters of the adaptive filter to target weight parameters; input the second sample far-end signal for the next training into the updated adaptive filter to generate a target echo signal for the next training; perform linear echo cancellation on the second sample near-end speech signal for the next training based on the target echo signal to obtain a second target near-end speech signal for the next training; and perform the next model training on the nonlinear echo cancellation model based on the target echo signal and the second target near-end speech signal.

[0181] It should be noted that the above explanation of the echo cancellation method embodiments also applies to the echo cancellation device of the present disclosure embodiments, and the specific process will not be repeated here.

[0182] In this embodiment, a first far-end speech signal and a first near-end speech signal are acquired. Feature extraction is performed on both signals to obtain a first speech feature and a second speech feature. Based on these features, time delay alignment is performed on the far-end and near-end speech signals to obtain aligned second far-end and near-end speech signals. This embodiment extracts features from both far-end and near-end speech signals and performs time delay alignment using these extracted features. This accurately aligns the far-end and near-end speech signals, improving echo cancellation and effectively eliminating far-end speech from the near-end mixed signal, thus achieving echo cancellation.

[0183] like Figure 15 The diagram shown is a block diagram of an electronic device for an echo cancellation method according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as intelligent voice interaction devices, personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0184] like Figure 15 As shown, the electronic device includes one or more processors 1501, a memory 1502, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processor 1501 can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple storage devices, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 15 Take a processor 1501 as an example.

[0185] The memory 1502 is the non-transitory computer-readable storage medium provided in this disclosure. The memory stores instructions executable by at least one processor to cause at least one processor to perform the echo cancellation method provided in this disclosure. The non-transitory computer-readable storage medium of this disclosure stores computer instructions for causing a computer to perform the echo cancellation method provided in this disclosure.

[0186] The memory 1502, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the echo cancellation method in the embodiments of this disclosure. The processor 1501 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 1502, thereby implementing the echo cancellation method in the above method embodiments.

[0187] Memory 1502 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application program required for at least one function; the data storage area may store data created by the use of the electronic device according to the echo cancellation method. Furthermore, memory 1502 may include high-speed random access memory and may also include non-transient memory, such as at least one disk storage device, flash memory device, or other non-transient solid-state storage device. In some embodiments, memory 1502 may optionally include memory remotely located relative to processor 1501, and these remote memories may be connected to the electronic device of the echo cancellation method via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0188] The electronic device for echo cancellation may further include an input device 1503 and an output device 1504. The processor 1501, memory 1502, input device 1503, and output device 1504 can be connected via a bus or other means. Figure 15 Taking the example of a connection between China and Israel via a bus.

[0189] Input device 1503 can receive input digital or character information, as well as key signal input related to user settings and function control of the electronic device for the echo cancellation method, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 1504 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0190] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0191] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0192] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0193] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0194] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability.

[0195] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0196] In the description of this specification, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise expressly and specifically defined.

[0197] Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present disclosure.

Claims

1. An echo cancellation method, characterized by, The method comprises: obtaining a first remote speech signal and a first near-end speech signal; performing feature extraction on the first remote speech signal and the first near-end speech signal respectively to obtain first speech features and second speech features; performing time delay alignment on the first remote speech signal and the first near-end speech signal based on the first speech features and the second speech features to obtain an aligned second remote speech signal and a second near-end speech signal; performing echo cancellation on the second near-end speech signal based on the second remote speech signal to obtain a first target near-end speech signal after echo cancellation; the method of performing echo cancellation on the second near-end speech signal based on the second remote speech signal to obtain a first target near-end speech signal after echo cancellation comprises: performing echo estimation on the second remote speech signal to obtain an echo signal; performing linear echo cancellation on the second near-end speech signal based on the echo signal to obtain a third near-end speech signal after linear echo cancellation; inputting the echo signal and the third near-end speech signal into a trained target nonlinear echo cancellation model to generate a first spectral mask; obtaining the first target near-end speech signal based on the first spectral mask and the third near-end speech signal; the training process of the target nonlinear echo cancellation model comprises: collecting a second sample remote speech signal and a second sample near-end speech signal; inputting the second sample remote speech signal into an adaptive filter to generate a target echo signal; performing linear echo cancellation on the second sample near-end speech signal based on the target echo signal to obtain a second target near-end speech signal after linear echo cancellation; iteratively training a pre-trained nonlinear echo cancellation model based on the target echo signal, the second target near-end speech signal, and target weight parameters of the adaptive filter to obtain a trained target nonlinear echo cancellation model; the method of iteratively training a pre-trained nonlinear echo cancellation model based on the target echo signal, the second target near-end speech signal, and target weight parameters of the adaptive filter comprises: inputting the target echo signal and the second target near-end speech signal into the nonlinear echo cancellation model to generate a second spectral mask; obtaining a predicted near-end speech signal based on the second spectral mask and the second target near-end speech signal; obtaining target weight parameters of the adaptive filter for next training based on the predicted near-end speech signal; performing next model training on the nonlinear echo cancellation model based on the target weight parameters.

2. The method of claim 1, wherein, the method of performing time delay alignment on the first remote speech signal and the first near-end speech signal based on the first speech features and the second speech features comprises: obtaining a similarity between the first speech features and the second speech features; performing time delay alignment on the first remote speech signal and the first near-end speech signal based on the similarity.

3. The method of claim 2, wherein, the method of performing time delay alignment on the first remote speech signal and the first near-end speech signal based on the similarity comprises: determine a target first speech feature and a target second speech feature corresponding to the maximum similarity; align target far-end data corresponding to the target first speech feature with target near-end data corresponding to the target second speech feature.

4. The method of claim 3, wherein, The aligning of the target far-end data corresponding to the target first speech feature with the target near-end data corresponding to the target second speech feature comprises: obtaining frame indices of the target far-end data and the target near-end data; determining a time difference between the target far-end data and the target near-end data according to the frame indices; aligning the target far-end data and the target near-end data according to the time difference.

5. The method of claim 1, wherein, After the first far-end speech signal and the first near-end speech signal are obtained, the method further comprises: inputting the first far-end speech signal and the first near-end speech signal into the trained attention mechanism network, and performing feature extraction on the first far-end speech signal and the first near-end speech signal respectively to obtain the first speech feature and the second speech feature; delay aligning the first far-end speech signal and the first near-end speech signal based on the first speech feature and the second speech feature to obtain an aligned second far-end speech signal and a second near-end speech signal; performing echo cancellation on the second near-end speech signal based on the second far-end speech signal to obtain the first target near-end speech signal.

6. The method of claim 2, wherein, The obtaining of the similarity between the first speech feature and the second speech feature comprises: inputting the first far-end speech signal and the first near-end speech signal into the trained attention mechanism network; performing feature extraction on the first far-end speech signal and the first near-end speech signal respectively by a feature extraction layer in the attention mechanism network to obtain the first speech feature and the second speech feature; processing the first speech feature and the second speech feature by a fully connected layer in the attention mechanism network to obtain the similarity between the first speech feature and the second speech feature.

7. The method of claim 6, wherein, The training process of the attention mechanism network comprises: collecting a first sample far-end speech signal and a first sample near-end speech signal; inputting the first sample far-end speech signal and the first sample near-end speech signal into an initial attention mechanism network to generate a predicted similarity between first sample far-end data of the first sample far-end speech signal and first sample near-end data of the first sample near-end speech signal; adjusting model parameters of the initial attention mechanism network based on the predicted similarity and a label similarity to obtain a trained attention mechanism network.

8. The method of claim 1, wherein, The obtaining of the target weight parameter of the adaptive filter for the next training comprises: determining a voice call state between a far-end device and a near-end device based on energies of the second sample far-end speech signal and the second sample near-end speech signal; in response to the voice call state being a far-talking state, obtaining the target weight parameter of the adaptive filter based on the predicted near-end speech signal; In response to the voice call state being a non-talking state, the current weight parameter of the adaptive filter is taken as the target weight parameter.

9. The method of claim 1, wherein, The next model training of the nonlinear echo cancellation model based on the target weight parameter comprises: updating the weight parameter of the adaptive filter to the target weight parameter; inputting a second sample remote signal for next training into the updated adaptive filter to generate a target echo signal for next training; performing linear echo cancellation on a second sample near-end speech signal for next training based on the target echo signal to obtain a second target near-end speech signal for next training; performing the next model training of the nonlinear echo cancellation model based on the target echo signal and the second target near-end speech signal.

10. An echo cancellation device, characterized by comprise: an acquisition module, configured to acquire a first remote speech signal and a first near-end speech signal; a feature extraction module, configured to perform feature extraction on the first remote speech signal and the first near-end speech signal respectively to obtain first speech features and second speech features; a time delay alignment module, configured to perform time delay alignment on the first remote speech signal and the first near-end speech signal based on the first speech features and the second speech features to obtain an aligned second remote speech signal and a second near-end speech signal; an echo cancellation module, configured to perform echo cancellation on the second near-end speech signal based on the second remote speech signal to obtain a first target near-end speech signal after echo cancellation; the echo cancellation module is specifically configured to perform echo estimation on the second remote speech signal to obtain an echo signal; perform linear echo cancellation on the second near-end speech signal based on the echo signal to obtain a third near-end speech signal after linear echo cancellation; input the echo signal and the third near-end speech signal into the trained target nonlinear echo cancellation model to generate a first spectral mask; acquire the first target near-end speech signal based on the first spectral mask and the third near-end speech signal; The device further comprises a second training module, and the second training module is configured to collect a second sample remote speech signal and a second sample near-end speech signal; input the second sample remote speech signal into an adaptive filter to generate a target echo signal; perform linear echo cancellation on the second sample near-end speech signal based on the target echo signal to obtain a second target near-end speech signal after linear echo cancellation; perform iterative training on the pre-trained nonlinear echo cancellation model based on the target echo signal, the second target near-end speech signal and a target weight parameter of the adaptive filter to obtain a trained target nonlinear echo cancellation model; the second training module is further configured to input the target echo signal and the second target near-end speech signal into the nonlinear echo cancellation model to generate a second spectral mask; acquire a predicted near-end speech signal based on the second spectral mask and the second target near-end speech signal; obtain target weight parameters of the adaptive filter for next time training based on the predicted near-end speech signal; perform next time model training on the nonlinear echo cancellation model based on the target weight parameters.

11. An electronic device, comprising: comprise: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

12. A computer readable storage medium having stored thereon computer instructions, wherein, the computer instructions are used to enable the computer to perform the method of any one of claims 1-9.

Images