A BNLMS speech signal echo cancellation method and system based on LSTM improvement

Abstract

The application relates to an improved BNLMS speech signal echo cancellation method and system based on an LSTM, which comprises the following steps: performing adaptive linear filtering processing on a far-end reference signal and a microphone signal to obtain a residual error signal; performing feature extraction on the residual error signal, the far-end reference signal and the microphone signal to obtain a time-frequency domain joint feature sequence; inputting the time-frequency domain joint feature sequence into a recurrent neural network model to perform nonlinear mapping and obtain a pure near-end speech estimation signal; taking a loss value between the pure near-end speech estimation signal and a pure near-end speech label as an optimization target, performing joint gradient updating on filter coefficients used for adaptive linear filtering processing and network weight parameters in the recurrent neural network model to obtain an optimal hybrid echo cancellation model; and performing echo cancellation through the optimal hybrid echo cancellation model. The application realizes joint suppression of linear and nonlinear echoes and reduces system processing delay.

Description

Technical Field

[0001] This invention belongs to the field of signal processing technology, and in particular relates to a BNLMS speech signal echo cancellation method and system based on LSTM improvement. Background Technology

[0002] Acoustic echo cancellation (AEC) is a core front-end processing technology in real-time voice communication systems. Its mechanism is that the voice signal played by the speaker of the remote device is re-acquired by the microphone of the near end through the acoustic path (including air propagation, room reflection, etc.), and superimposed with the voice signal of the near end user and transmitted back to the remote end, causing the remote user to hear the echo of his own voice.

[0003] Traditional AEC methods primarily rely on adaptive filtering techniques. These involve constructing filters that simulate the real acoustic path to process the far-end reference signal, generating an echo estimation signal, and then subtracting this signal from the mixed signal captured by the microphone to obtain a clean near-end speech signal. Among these methods, the Block Normalized Least Mean Square (BNLMS) algorithm is widely used due to its low computational complexity and good convergence performance. Meanwhile, deep learning methods, represented by Long Short-Term Memory (LSTM) networks, have also been introduced into the echo cancellation field due to their powerful modeling capabilities for nonlinear features, achieving good results in various scenarios.

[0004] In existing technologies, there are also hybrid schemes that cascade adaptive filtering algorithms with deep learning models. For example, the NLMS algorithm is first used to perform preliminary linear echo cancellation, and then the residual signal is input into a neural network for post-processing. However, the above schemes generally adopt a step-by-step training approach, that is, the deep learning model and the adaptive filter are trained independently and then combined. This results in a lack of a collaborative optimization mechanism between the two modules. The coefficient update of the adaptive filter depends only on its own error signal and cannot be subject to the reverse constraint of the final optimization target of the downstream deep learning module. As a result, the entire hybrid system cannot achieve globally optimal echo cancellation performance. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of the invention is to provide a BNLMS speech signal echo cancellation method and system based on LSTM improvement.

[0006] This invention provides an LSTM-based improved BNLMS speech signal echo cancellation method, comprising: S1. Perform adaptive linear filtering on the far-end reference signal and the microphone signal to obtain the residual error signal; S2. Perform feature extraction on the residual error signal, the far-end reference signal, and the microphone signal to obtain a joint time-frequency domain feature sequence; S3. Input the time-frequency domain joint feature sequence into a recurrent neural network model for nonlinear mapping to obtain a clean near-end speech estimation signal; S4. Using the loss value between the pure near-end speech estimation signal and the pure near-end speech label as the optimization objective, the filter coefficients used for adaptive linear filtering and the network weight parameters in the recurrent neural network model are jointly updated with gradients to obtain the optimal hybrid echo cancellation model. S5. Perform echo cancellation using the optimal hybrid echo cancellation model.

[0007] According to the present invention, a BNLMS speech signal echo cancellation method based on LSTM improvement is provided, wherein step S1 further includes: S11. Divide the remote reference signal into blocks according to the block length to obtain multiple remote signal blocks; S12. Based on the far-end signal block and the current filter coefficients, the microphone signal is subjected to block-domain normalized convolution using the BNLMS algorithm to obtain the linear echo estimation signal. S13. Subtract the microphone signal from the linear echo estimation signal to obtain the residual error signal.

[0008] According to the LSTM-based improved BNLMS speech signal echo cancellation method provided by the present invention, in step S12, the filter coefficient update formula of the BNLMS algorithm is as follows: in, This is the filter order index value. For the first The filter coefficient vector before the next block update For the first The filter coefficient vector updated in the next block. To update the step size, To prevent regularization constants with denominators of zero, For block error vector, For block input matrices, For block length, For the first Normalized energy factor during next block update; in, For the first Normalized energy factor during next block update The energy smoothing coefficient, The trace of the matrix, This indicates the matrix transpose.

[0009] According to the present invention, a BNLMS speech signal echo cancellation method based on LSTM improvement is provided, wherein step S2 further includes: S21. Perform a short-time Fourier transform on the residual error signal to obtain the amplitude spectrum and phase spectrum of the residual error signal; S22. Organize the amplitude spectrum into a two-dimensional feature matrix according to the frame index and frequency index to obtain the time-frequency domain joint feature sequence.

[0010] According to the LSTM-based improved BNLMS speech signal echo cancellation method provided by the present invention, step S3 further includes: S31. The time-frequency domain joint feature sequence is fed into the input layer for windowing processing to obtain a context feature vector composed of the current frame and historical frames. S32. The context feature vector is sequentially passed through the first and second LSTM hidden layers for gated temporal modeling to obtain the hidden state sequence. S33. The hidden state sequence is fed into the fully connected output layer, and the dimension is mapped by a linear activation function to obtain the time-frequency domain mask. S34. Multiply the time-frequency domain mask by the microphone signal spectrum to obtain a clean near-end speech spectrum estimation signal; S35. Based on the phase spectrum, the pure near-end speech spectrum estimation signal is reconstructed by inverse short-time Fourier transform to obtain the pure near-end speech estimation signal.

[0011] According to the LSTM-based improved BNLMS speech signal echo cancellation method provided by the present invention, step S4 further includes: S41. The loss is calculated by applying the mean square error function to the pure near-end speech estimation signal and the pure near-end speech label to obtain the loss value; S42. Based on the loss value, the gradients of the LSTM network weight parameters and BNLMS filter coefficients are calculated simultaneously using the backpropagation algorithm to obtain the joint gradient vector. S43. Input the joint gradient vector into the Adam optimizer to update the parameters, obtain the updated network weight parameters and filter coefficients, and continuously optimize the training process until the loss value converges to obtain the optimal hybrid echo cancellation model.

[0012] According to the LSTM-based improved BNLMS speech signal echo cancellation method provided by the present invention, in step S41, the expression of the mean square error function is: in, This represents the loss value of the mean squared error function. For sampling point index, This represents the total number of sampling points used to calculate the loss in the current training batch. For the first The true label value of clean near-end speech at each sampling point For the first The clean near-end speech estimate output by the LSTM module at each sampling point.

[0013] According to the LSTM-based improved BNLMS speech signal echo cancellation method provided by the present invention, step S5 further includes: S51. The real-time audio data collected by the microphone and the remote reference signal are processed by buffer block through the audio driver interface to obtain the remote signal block and the microphone signal block at the current moment. S52. The far-end signal block and the microphone signal block are fed into the optimal hybrid echo cancellation model for linear echo cancellation processing to obtain a real-time residual error signal block, and the real-time residual error signal block is stored in the LSTM input buffer. S53. The real-time residual error signal block is read from the LSTM input buffer through the LSTM module, and nonlinear mapping is performed on each sample in a sequential manner to obtain a real-time clean near-end speech estimation block, which is then sent to the output buffer for continuous audio output.

[0014] According to the LSTM-based improved BNLMS speech signal echo cancellation method provided by the present invention, in step S53, when outputting continuous audio, a pipelined parallel processing mechanism is adopted.

[0015] The present invention also provides an LSTM-based improved BNLMS speech signal echo cancellation system for performing an LSTM-based improved BNLMS speech signal echo cancellation method as described in any of the above claims, comprising: Filtering module: Used to perform adaptive linear filtering on the far-end reference signal and the microphone signal to obtain the residual error signal; Extraction module: used to extract features from the residual error signal, the far-end reference signal, and the microphone signal to obtain a joint time-frequency domain feature sequence; A mapping module, configured as a recurrent neural network model, is used to receive the joint time-frequency domain feature sequence and perform nonlinear mapping to obtain a clean near-end speech estimation signal; Training module: Used to perform joint gradient updates on the filter coefficients used for adaptive linear filtering and the network weight parameters in the recurrent neural network model with the loss value between the clean near-end speech estimation signal and the clean near-end speech label as the optimization objective, so as to obtain the optimal hybrid echo cancellation model; An output module is configured as the optimal hybrid echo cancellation model obtained by the training module for echo cancellation.

[0016] This invention provides an improved BNLMS speech signal echo cancellation method and system based on LSTM. Firstly, by cascading a BNLMS adaptive filtering module and an LSTM recurrent neural network module and introducing an end-to-end joint gradient update mechanism, the optimization direction of the filter coefficients is no longer limited to the constraints of their own error signal. Instead, it is simultaneously influenced by the backpropagation of the loss value between the final output of the downstream network and the clean near-end speech label. This allows the two modules to adapt to each other and converge collaboratively during training, achieving optimal echo cancellation performance of the entire hybrid system at the global level. This effectively overcomes the performance bottleneck caused by the lack of deep collaboration between modules in traditional step-by-step training schemes. Furthermore, the BNLMS module of this invention undertakes the task of fast cancellation of linear echo components through block-domain normalized convolution processing. This significantly compresses the linear components in the residual error signal input to the LSTM module. The learning objective of the LSTM module is thus greatly simplified from mapping the complex mixed signal to clean speech to mapping the linear residual to clean speech. This simplification of the objective allows for effective reduction in network size, significantly reducing the computational cost of the model while maintaining cancellation accuracy, making the system easier to deploy on resource-constrained embedded terminal devices. Furthermore, the introduction of the pipelined parallel processing mechanism in this invention allows the processing of the current block of data by the BNLMS module and the inference of the previous block of residual signal by the LSTM module to overlap in time. The total system delay is compressed from the sum of the serial delays of the two modules to the order of magnitude of the processing delay of a single module, ensuring the continuity and real-time performance of the audio output stream. In addition, this invention transforms the residual error signal into a joint time-frequency domain feature sequence through short-time Fourier transform, and combines time-frequency domain masking and inverse transform for speech reconstruction. This fully utilizes the sparsity of the speech signal in the time-frequency domain, further improving the suppression accuracy of nonlinear residual echoes and the naturalness of the reconstructed speech. Attached Figure Description

[0017] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. It is obvious that the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings.

[0018] Figure 1 A schematic flowchart of a BNLMS speech signal echo cancellation method based on LSTM improvement provided in an embodiment of the present invention; Figure 2This is a schematic diagram of a BNLMS speech signal echo cancellation system based on LSTM improvement, provided for an embodiment of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0020] Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts disclosed in this invention.

[0021] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The terms "installed," "connected," and "linked" should be interpreted broadly; for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with some aspects of the invention as detailed in the appended claims.

[0023] To better understand this invention, the research background of this invention will be explained in detail below.

[0024] Echo cancellation is an important technology in voice communication front-end processing. Echoes mainly occur in real-time audio and video calls when sound from the speaker is re-recorded into the microphone, causing a blurred signal. Therefore, voice echo cancellation technology has broad application prospects. In real-time voice communication systems, acoustic echo is a common problem that seriously affects call quality. Its mechanism is that the voice signal played by the speaker at the far-end device is re-recorded by the microphone at the near-end device after traveling through the acoustic path (including air propagation, room reflections, etc.), and transmitted back to the far-end along with the near-end user's voice signal, causing the far-end user to hear an echo of their own voice. To eliminate this annoying echo, acoustic echo cancellation (AEC) technology has emerged. Traditional AEC methods mainly rely on adaptive filtering techniques. The core idea is to construct a filter that can simulate the real acoustic path, process the far-end reference signal through this filter to generate an echo estimation signal, and then subtract this estimation signal from the mixed signal collected by the microphone to obtain a clean local voice signal.

[0025] Among numerous adaptive filtering algorithms, the Block Normalized Least Mean Square (BNLMS) algorithm is widely used due to its low computational complexity, simple implementation, and good convergence performance. However, the performance of traditional BNLMS algorithms is significantly limited when dealing with complex acoustic environments. First, these algorithms are inherently linear, assuming the acoustic path is a linear time-invariant (LTI) system. However, in practical applications, due to the nonlinear characteristics of audio hardware such as speakers and power amplifiers, as well as signal clipping at high sound pressure levels, the resulting echo signals often contain a large number of nonlinear components. Linear adaptive filters are powerless against these nonlinear distortions, resulting in noticeable nonlinear echoes remaining after eliminating linear echoes, severely impacting the final speech quality. Second, there is an inherent contradiction between the convergence speed and steady-state error of traditional algorithms. When the echo path changes abruptly (such as when a user moves a mobile device), the algorithm needs to converge quickly to track the new path, but this usually leads to a large steady-state error; conversely, pursuing low steady-state error slows down the convergence speed, making it unable to adapt to rapidly changing environments. Furthermore, in double-talk scenarios, where users at the near end and far end speak simultaneously, the near-end speech signal severely interferes with the coefficient update of the adaptive filter, causing the filter to diverge and its performance to drop sharply.

[0026] In recent years, with the rapid development of deep learning technology, researchers have begun to explore the use of deep neural networks (DNNs) to address the limitations of traditional acoustic echo cancellation methods. In particular, recurrent neural networks (RNNs) and their variants, such as long short-term memory (LSTM) networks and gated recurrent units (GRUs), have been widely applied to acoustic echo cancellation tasks due to their inherent advantages in processing sequential data. Deep learning models, especially LSTMs, can learn complex nonlinear mappings that traditional linear models cannot capture. By training on massive amounts of data, LSTM networks can learn the characteristics of various complex acoustic phenomena such as speaker nonlinearity, room reverberation, and background noise, thereby achieving joint suppression of linear and nonlinear echoes. Their performance significantly outperforms traditional methods in many scenarios.

[0027] However, applying deep learning models to real-time AEC systems also faces a series of severe challenges. First, there's the complexity and computational cost of the models. Achieving high-precision echo cancellation typically requires designing complex, parameter-intensive deep network models. These models require enormous computational resources during both training and inference, which is a huge burden for resource-constrained embedded devices (such as smartphones and smart speakers), making it difficult to meet real-time processing requirements. Second, there's the issue of acquiring and labeling training data. The performance of deep learning models is highly dependent on large-scale, high-quality training data. Building a dataset covering various acoustic environments, device types, speakers, and noise scenarios is a time-consuming and labor-intensive task. Furthermore, obtaining "clean" near-end speech for supervised learning as training labels is also a challenge. Finally, there's the issue of model generalization ability. A model trained on a specific dataset may show significant performance degradation when faced with entirely new scenarios not present in the training data (such as new types of speakers or new room acoustic characteristics), exhibiting poor generalization ability. Therefore, designing lightweight, efficient, and well-generalized deep learning AEC models is currently a core challenge in this field.

[0028] To combine the real-time performance of traditional adaptive filtering algorithms with the powerful nonlinear modeling capabilities of deep learning models, researchers have proposed various hybrid approaches. A common approach is to cascade the two: first, a lightweight adaptive filter (such as NLMS) is used for initial linear echo cancellation; then, the residual signal after cancellation is input into a deep learning model for fine-grained nonlinear echo and noise suppression. The advantage of this architecture is that the pre-processed adaptive filter handles most of the linear echo cancellation, significantly reducing the burden on the subsequent deep learning model, allowing for the use of a smaller, more efficient neural network model to achieve the desired performance. For example, some studies have used the output of NLMS as the input to subsequent neural networks with good results. However, existing hybrid methods still have some shortcomings. First, in terms of architectural design, most schemes are simply a superposition of functions, lacking exploration of deeper collaborative mechanisms between the two modules. For example, can the update process of the adaptive filtering module be optimized using the output of the deep learning module? Can the training of the deep learning module utilize the internal state information of the adaptive filtering module? These questions are rarely addressed in existing technologies. Secondly, regarding training methods, most solutions employ a step-by-step training approach, where the deep learning model is trained independently first, and then combined with a fixed adaptive filter. This training method cannot guarantee the global optimum of the entire system. A better approach would be end-to-end joint training of the entire hybrid system, allowing the two modules to adapt to each other and optimize collaboratively. Finally, in terms of real-time processing, how to efficiently schedule and execute the computational tasks of the two modules to minimize system latency is also a problem that requires careful design. For example, the BNLMS algorithm, due to its block processing characteristics, is very suitable for parallel computation, while the inference process of LSTM has sequential dependencies. Designing an efficient real-time processing framework that fully leverages the advantages of both algorithms is key to improving the overall system performance.

[0029] This invention proposes an algorithm for echo cancellation using LSTM cascaded with BNLMS, and the combination of the two has a better echo cancellation effect.

[0030] The embodiments of the present invention are described below with reference to the figures.

[0031] like Figure 1 As shown, this invention provides an LSTM-based improved BNLMS speech signal echo cancellation method, comprising: S1. Perform adaptive linear filtering on the far-end reference signal and the microphone signal to obtain the residual error signal.

[0032] Step S1 further includes: S11. Divide the remote reference signal into blocks according to the block length to obtain multiple remote signal blocks.

[0033] In step S11, the present invention uses continuously input remote reference signals According to the length of the fixed block It is divided into several remote signal blocks, each remote signal block containing These continuous sampling points form a discrete processing sequence in blocks, providing basic data units for subsequent block-domain filtering operations.

[0034] S12. Based on the far-end signal block and the current filter coefficients, the microphone signal is subjected to block-domain normalized convolution using the BNLMS algorithm to obtain the linear echo estimation signal.

[0035] In step S12, the filter coefficient update formula of the BNLMS algorithm is as follows: in, This is the filter order index value. For the first The filter coefficient vector before the next block update For the first The filter coefficient vector updated in the next block. To update the step size, To prevent regularization constants with denominators of zero, This is the block error vector. For block input matrices, For block length, For the first Normalized energy factor during next block update; in, For the first Normalized energy factor during next block update The energy smoothing coefficient, The trace of the matrix, This indicates the matrix transpose.

[0036] In step S12, for the first A remote signal block, the present invention will include the block within Each sampling point is organized into a block input matrix. Combined with the current filter coefficient vector The microphone signal is subjected to block-domain normalized convolution operation, and the corresponding linear echo estimation signal is output. Subsequently, the present invention updates the filter coefficients once: using the block error vector With block input matrix The product divided by the normalized energy factor With regularization constant The sum, multiplied by the step size Superimposed on the current coefficient vector The updated filter coefficient vector is obtained from the above. Among them, the normalized energy factor By inputting the block matrix The product of the transpose and itself Trace the path and then combine it with the normalized energy factor of the previous block. By smoothing coefficient The coefficients are obtained by performing an exponential weighted summation, which adaptively matches the magnitude of each coefficient update with the energy level of the current input signal, thus avoiding excessive gradient update magnitude that could lead to filter coefficient divergence when the input signal energy is high.

[0037] S13. Subtract the microphone signal from the linear echo estimation signal to obtain the residual error signal.

[0038] Furthermore, in S13, the microphone signal... Compared with the above linear echo estimation signal The difference is calculated to obtain the residual error signal. , The superposition of nonlinear echo components and near-end speech components still remains.

[0039] S2. Perform feature extraction on the residual error signal, the far-end reference signal, and the microphone signal to obtain a joint time-frequency domain feature sequence.

[0040] Step S2 further includes: S21. Perform a short-time Fourier transform on the residual error signal to obtain the amplitude spectrum and phase spectrum of the residual error signal.

[0041] Furthermore, the residual error signal Since it is a time-domain signal, when it is directly modeled nonlinearly in the time domain, the network has difficulty effectively distinguishing echo remnants from speech components at different frequencies. Therefore, in S21, this invention addresses the residual error signal... Short-Time Fourier Transform (STFT) processing is performed. The core operation of STFT is to divide the time-domain signal into several short-time frames according to the frame length and frame shift parameters. After multiplying each frame by a window function, a Discrete Fourier Transform is performed to output the complex spectrum of each frame at each frequency point. Subsequently, this invention extracts the amplitude spectrum and phase spectrum from the complex spectrum. The obtained amplitude spectrum reflects the energy distribution at each frequency point of each frame, and the phase spectrum records the phase information at each frequency point, which are stored in S35 for time-domain signal reconstruction.

[0042] S22. Organize the amplitude spectrum into a two-dimensional feature matrix according to the frame index and frequency index to obtain the time-frequency domain joint feature sequence.

[0043] Furthermore, in step S22, the present invention indexes the amplitude spectrum by frame. (Timeline) and Frequency Index Arranged along the frequency axis, organized as a two-dimensional feature matrix, i.e., a joint feature sequence in the instantaneous frequency domain. The rows correspond to frames at different times, and the columns correspond to different frequency components. This two-dimensional matrix is ​​used as the input data of the LSTM module and sent to the subsequent processing flow.

[0044] S3. Input the time-frequency domain joint feature sequence into the recurrent neural network model for nonlinear mapping to obtain a clean near-end speech estimation signal.

[0045] Step S3 further includes: S31. The time-frequency domain joint feature sequence is sent into the input layer for windowing processing to obtain a context feature vector composed of the current frame and historical frames.

[0046] Furthermore, in step S31, the present invention combines the time-frequency domain joint feature sequence. The data is fed into the input layer of the LSTM network, and each frame is indexed. Extract the current frame and history from the feature sequence. Frames are concatenated to form a context feature vector. Window size These are preset hyperparameters. This windowing process allows the network to simultaneously perceive spectral changes within a local time range while processing each frame.

[0047] S32. The context feature vector is sequentially passed through the first and second LSTM hidden layers for gated temporal modeling to obtain the hidden state sequence.

[0048] In step S32, the context feature vector The first and second LSTM hidden layers are input sequentially. Each LSTM unit contains three gate structures: a forget gate, an input gate, and an output gate. The forget gate determines which information from the previous cell state is retained, the input gate determines which information from the current input is written into the cell state, and the output gate determines which information from the current cell state is output as the hidden state. Each frame index... At each step, the LSTM unit receives the current input vector and the hidden state of the previous frame. After element-wise operations through three gates, the cell state is updated, and the hidden state of the current frame is output. After processing by two stacked LSTM layers, the output of the second layer constitutes a hidden state sequence. Each vector in this sequence comprehensively encodes the nonlinear spectral features of the current frame and its historical frames.

[0049] S33. The hidden state sequence is fed into the fully connected output layer, and the dimension is mapped by a linear activation function to obtain the time-frequency domain mask.

[0050] In step S33, the hidden state sequence is fed into the fully connected output layer. The fully connected layer performs a linear weighted summation operation on the hidden state vector at each time step, and maps it to an output vector with the same frequency dimension as the input through a linear activation function, i.e., frequency domain masking. Time-frequency domain masking is a joint feature sequence with the time-frequency domain. A real-valued matrix of the same dimension, where the value of each element reflects the estimated proportion of near-end speech components relative to residual echo components at the corresponding time and frequency point.

[0051] S34. Multiply the time-frequency domain mask by the microphone signal spectrum to obtain a clean near-end speech spectrum estimation signal.

[0052] In step S34, the present invention masks the time-frequency domain. With microphone signal spectrum Element-wise multiplication yields the pure near-end speech spectrum estimation signal. The microphone signal spectrum From microphone signal The complex spectrum obtained by STFT processing contains both amplitude and phase information. After multiplication with time-frequency domain masking, the echo residual components at the corresponding time-frequency points are suppressed, and the near-end speech components are preserved.

[0053] S35. Based on the phase spectrum, the pure near-end speech spectrum estimation signal is reconstructed by inverse short-time Fourier transform to obtain the pure near-end speech estimation signal.

[0054] Furthermore, in S35, the present invention combines the phase spectrum of the residual error signal retained in S21 with the pure near-end speech spectrum estimation signal obtained in S34. The amplitude information is combined, and each frame is subjected to inverse short-time Fourier transform (iSTFT) and then superimposed and added according to the frame shift parameter to reconstruct a clean near-end speech estimation signal in the time domain. .

[0055] Before performing end-to-end joint training of S4, a diverse audio dataset for training needs to be pre-constructed, specifically including the following steps S401-S403. Since real-world recording scenarios involve high acquisition costs and difficulty in controlling acoustic variables, this invention generates training data through acoustic simulation: a set of room impulse responses covering various acoustic environments is generated using preset room parameters driven by an acoustic simulation tool; then, a clean speech signal is convolved with the room impulse responses and nonlinear distortion is superimposed to obtain a simulated microphone signal. Then the simulated microphone signal Remote reference signal The filter coefficients output block by block during the BNLMS algorithm operation The organization is the training sample triplet. After forming a diverse audio dataset, the process proceeds to the joint gradient update process in S4.

[0056] S401. Using acoustic simulation tools, room impulse response generation processing is performed on preset room parameters to obtain a set of room impulse responses covering different reverberation times and room sizes.

[0057] In S401, this invention processes preset room parameters using an acoustic simulation tool. These preset room parameters include the room's length, width, and height dimensions, the spatial positions of the sound source and microphone, and the wall absorption coefficients. The acoustic simulation tool simulates the propagation, reflection, and absorption of sound waves within an enclosed space based on these parameters, outputting the Room Impulse Response (RIR). The RIR is a finite-length discrete-time sequence that describes the impulse response characteristics of the complete acoustic path from the sound source to the microphone. This invention generates a set of room impulse responses covering different reverberation times and room sizes by performing multiple combinations of the above parameters.

[0058] S402. Convolve the clean speech signal with the room impulse response set to obtain a simulated microphone signal containing linear echo components and nonlinear echo components.

[0059] In step S402, the present invention performs a linear convolution operation between the clean speech signal and each RIR sequence in the room impulse response set obtained in step S401 to obtain an echo signal after simulating acoustic path propagation. Subsequently, the present invention adds nonlinear distortion processing to the convolution result to simulate the nonlinear distortion components introduced by the speaker and power amplifier hardware in actual operation. Finally, the near-end speech signal is added to the above processing result to obtain a simulated microphone signal containing both linear and nonlinear echo components. .

[0060] S403, combine the simulated microphone signal, the corresponding far-end reference signal, and the filter coefficients obtained during the BNLMS algorithm operation. The samples are jointly organized into training sample triplets to obtain a diverse audio dataset for end-to-end joint training.

[0061] In S403, the present invention uses the simulated microphone signal obtained in S402. , corresponding remote reference signal And the filter coefficients output block by block during the BNLMS algorithm's operation on this set of data. After aligning the sampling points, they are packaged together into a single training sample triplet. Repeat steps S402 to S403 on all room impulse response sets generated by S401 and multiple sets of clean speech signals, summarizing all triples to obtain a diverse audio dataset covering various acoustic environments, device characteristics, and call scenarios. This dataset is directly input into the end-to-end joint training process of S4.

[0062] S4. Using the loss value between the pure near-end speech estimation signal and the pure near-end speech label as the optimization objective, perform joint gradient updates on the filter coefficients used for adaptive linear filtering and the network weight parameters in the recurrent neural network model to obtain the optimal hybrid echo cancellation model.

[0063] Step S4 further includes: S41. The loss is calculated by applying the mean square error function to the pure near-end speech estimation signal and the pure near-end speech label to obtain the loss value.

[0064] In step S41, the expression for the mean square error function is: in, This represents the loss value of the mean squared error function. The sampling point index is used as a dummy variable for summation. This represents the total number of sampling points used to calculate the loss in the current training batch. For the first The true label value of clean near-end speech at each sampling point For the first The clean near-end speech estimate output by the LSTM module at each sampling point.

[0065] Furthermore, in S41, the present invention uses the mean square error function to estimate the clean near-end speech signal output in S35. Corresponding pure near-end speech real tags Square the difference between each sample point, then apply the result to all samples in the current training batch. The loss value is obtained by averaging the squared differences of the sample points. .

[0066] It should be noted that the pure near-end voice is labeled with real tags. With microphone signal In Both refer to speech signals captured by near-end microphones, but their index meanings differ. In This is a time-domain sample point index, describing the first sample point in the real-time audio stream. The microphone mixed signal value at each sampling point. In To sum dummy variables, they are only used in the accumulation operation of the mean squared error function to iterate through the current training batch. Each sampling point essentially points to a different indexed representation of the same physical signal in different processing contexts.

[0067] S42. Based on the loss value, the gradients of the LSTM network weight parameters and BNLMS filter coefficients are calculated simultaneously using the backpropagation algorithm to obtain the joint gradient vector.

[0068] In step S42, the present invention uses the loss value obtained in S41. The gradient is propagated layer by layer backward along the computational graph using the backpropagation algorithm. Specifically, the gradient of the loss value with respect to the weights of the LSTM output layer is first calculated, then propagated forward sequentially to the gating parameters of each hidden layer of the LSTM, and then further forward to the filter coefficients of the BNLMS module. The gradients of all weight parameters of the LSTM network are concatenated with the gradients of the BNLMS filter coefficients to obtain a joint gradient vector. Ultimately, this invention ensures that the update direction of the BNLMS filter coefficients no longer depends solely on its own block error, but is also constrained by the final loss value.

[0069] S43. Input the joint gradient vector into the Adam optimizer to update the parameters, obtain the updated network weight parameters and filter coefficients, and continuously optimize the training process until the loss value converges to obtain the optimal hybrid echo cancellation model.

[0070] In step S43, the present invention inputs the joint gradient vector into the Adam optimizer. The Adam optimizer maintains a first-order momentum (exponentially weighted moving average of the gradient) and a second-order momentum (exponentially weighted moving average of the squared gradient) for each parameter component in the joint gradient vector. After adaptively scaling the first-order momentum with the second-order momentum, it is superimposed on the corresponding parameter to complete one parameter update, thus obtaining the updated LSTM network weight parameters and BNLMS filter coefficients. The present invention then repeats steps S41 to S43, traversing the entire training dataset for several rounds, continuously optimizing the training process until the loss value is reached. Once the signal converges to below a preset threshold, the corresponding LSTM network weight parameters and BNLMS filter coefficients are fixed to obtain the optimal hybrid echo cancellation model.

[0071] S5. Perform echo cancellation using the optimal hybrid echo cancellation model.

[0072] Step S5 further includes: S51. The real-time audio data acquired by the microphone and the remote reference signal are processed by buffering and block processing through the audio driver interface to obtain the remote signal block and the microphone signal block at the current moment.

[0073] In step S51, the present invention continuously reads the real-time audio data collected by the microphone through the audio driver interface. With remote reference signal The two samples are then stored in their respective input buffers. When the number of accumulated sample points in each buffer reaches the block length... At that time, the present invention retrieves from the buffer. Each of the continuous sampling points is organized into a remote signal block at the current moment. With microphone signal block Block numbering mark.

[0074] S52. The far-end signal block and the microphone signal block are fed into the optimal hybrid echo cancellation model for linear echo cancellation processing to obtain a real-time residual error signal block, and the real-time residual error signal block is stored in the LSTM input buffer.

[0075] In step S52, the present invention will remotely signal the block. With microphone signal block The signal is fed into the BNLMS module of the optimal hybrid echo cancellation model, and the linear echo estimation signal of the current block is calculated according to the block-domain normalized convolution operation process described in S12. and the microphone signal block and The difference is calculated to obtain the real-time residual error signal block. Then Stored in the LSTM input buffer, and simultaneously based on BNLMS filter coefficients Complete an online update and receive .

[0076] S53. The real-time residual error signal block is read from the LSTM input buffer through the LSTM module, and nonlinear mapping is performed on each sample in a sequential manner to obtain a real-time clean near-end speech estimation block, which is then sent to the output buffer for continuous audio output.

[0077] In step S53, the present invention reads the real-time residual error signal block from the LSTM input buffer. According to the processing flow described in S31 to S35, the frame index is used. frame by frame The process involves STFT transformation, context feature construction, LSTM-gated temporal modeling, time-frequency domain masking estimation, and iSTFT reconstruction to output a real-time clean near-end speech estimation block. The data is then fed into the output buffer, and finally, the data in the output buffer is sent to the audio playback device point by point.

[0078] In step S53, when outputting continuous audio, a pipelined parallel processing mechanism is used.

[0079] Furthermore, during the execution of steps S52-S53, this invention employs a pipelined parallel processing mechanism. Specifically, the BNLMS module processes the... While processing the first data block, the LSTM module simultaneously processes the second data block. The real-time residual error signal of the first data block is synchronously filled by the audio acquisition thread. The first data block, the audio playback thread synchronously outputs the first... The processing results of each data block are overlapped in time by the four threads. The total system latency is determined by the maximum processing latency of a single thread in the four parallel paths, rather than by the sequential sum of the latency of each thread.

[0080] like Figure 2 As shown, the present invention also provides a BNLMS speech signal echo cancellation system based on LSTM improvement, comprising: Filtering module 100: Used to perform adaptive linear filtering on the far-end reference signal and the microphone signal to obtain the residual error signal; Extraction module 200: used to extract features from the residual error signal, the far-end reference signal, and the microphone signal to obtain a joint time-frequency domain feature sequence; The mapping module 300 is configured as a recurrent neural network model to receive the joint time-frequency domain feature sequence and perform nonlinear mapping to obtain a clean near-end speech estimation signal. Training module 400: is used to perform joint gradient updates on the filter coefficients used for adaptive linear filtering and the network weight parameters in the recurrent neural network model with the loss value between the clean near-end speech estimation signal and the clean near-end speech label as the optimization objective, so as to obtain the optimal hybrid echo cancellation model; Output module 500 is configured as the optimal hybrid echo cancellation model obtained by the training module for echo cancellation.

[0081] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0082] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0083] This invention proposes an improved BNLMS speech signal echo cancellation algorithm based on LSTM. Specifically, it presents a hybrid echo cancellation system and method that combines a Block-wise Normalized Least Mean Square (BNLMS) adaptive filtering algorithm with a Long Short-Term Memory (LSTM) deep learning network. This aims to address the performance bottlenecks of traditional echo cancellation methods when handling nonlinear echoes, environmental noise, and complex acoustic path changes, while overcoming the challenges of pure deep learning models in terms of real-time performance, model complexity, and training data dependency. Applications of this invention include, but are not limited to, video conferencing, smart speakers, in-vehicle hands-free calling, IP phones, and various smart devices requiring high-quality full-duplex voice interaction.

[0084] This invention proposes a novel BNLMS-LSTM cascade architecture. Unlike the simple functional superposition in existing technologies, this architecture meticulously designs the functional positioning and signal flow of the two modules. The BNLMS module is positioned as a linear echo preprocessor, whose core task is to quickly and efficiently eliminate linear components in the signal, reducing the burden on the subsequent LSTM module. The LSTM module is positioned as a nonlinear residual signal refiner, focusing on handling nonlinear distortion and complex residuals that BNLMS cannot handle. This clear division of labor allows each module to function in its area of ​​expertise. More importantly, the architecture of this invention is not a simple cascade but lays the foundation for subsequent collaborative optimization. For example, the input to the LSTM is the residual from the BNLMS, which makes the learning objective of the LSTM very clear: learning the mapping from linear residuals to clean speech. This is much simpler than the task of learning directly from mixed signals to clean speech, and can therefore be achieved with a smaller and more efficient model.

[0085] Secondly, this invention proposes a unique joint training and optimization method. Traditional hybrid systems typically employ step-by-step training, where a deep learning model is trained independently first, and then combined with a fixed adaptive filter. This approach fails to achieve global optimization. This invention designs an end-to-end joint training process where the entire BNLMS-LSTM system is trained as a unified deep learning model. Training data includes far-end signals, microphone signals, and corresponding clean near-end speech labels. During training, the error between the system's final output (i.e., the LSTM output) and the clean speech labels is used to update the parameters of the LSTM network and the filter coefficients of the BNLMS module simultaneously through backpropagation. This joint optimization mechanism ensures that the coefficient updates of the BNLMS module no longer depend solely on its own error signal but are influenced by the final optimization objective of the LSTM module. Ultimately, the BNLMS module learns a filter state that provides the "most favorable" residual signal to the LSTM module, thereby achieving deep collaboration between the two modules and optimization of global performance.

[0086] Furthermore, this invention proposes an efficient real-time processing flow. Considering that the BNLMS algorithm is based on block processing, its computation process can be highly parallelized, while the inference process of LSTM has a natural sequential dependency. To minimize system latency, this invention designs a pipelined processing mechanism. Specifically, after the BNLMS module processes a data block and outputs a residual, the residual block can be immediately sent to the LSTM module for processing, while the BNLMS module can start processing the next input data block. This parallel pipelined processing method effectively hides some computational latency. In addition, this invention optimizes key parameters such as the block size of BNLMS and the network structure of LSTM (e.g., number of layers, number of hidden units) to achieve the best balance between model performance and computational efficiency. The entire real-time processing flow is designed to be highly modular, facilitating deployment and optimization on different hardware platforms (e.g., CPU, DSP, GPU, dedicated AI accelerators), ensuring that the system can achieve low-latency and high-efficiency operation on various terminal devices.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A BNLMS speech signal echo cancellation method based on LSTM improvement, characterized in that, include: S1. Perform adaptive linear filtering on the far-end reference signal and the microphone signal to obtain the residual error signal; S2. Perform feature extraction on the residual error signal, the far-end reference signal, and the microphone signal to obtain a joint time-frequency domain feature sequence; S3. Input the time-frequency domain joint feature sequence into a recurrent neural network model for nonlinear mapping to obtain a clean near-end speech estimation signal; S4. Using the loss value between the pure near-end speech estimation signal and the pure near-end speech label as the optimization objective, the filter coefficients used for adaptive linear filtering and the network weight parameters in the recurrent neural network model are jointly updated with gradients to obtain the optimal hybrid echo cancellation model. S5. Perform echo cancellation using the optimal hybrid echo cancellation model.

2. The BNLMS speech signal echo cancellation method based on LSTM improvement according to claim 1, characterized in that, Step S1 further includes: S11. Divide the remote reference signal into blocks according to the block length to obtain multiple remote signal blocks; S12. Based on the far-end signal block and the current filter coefficients, the microphone signal is subjected to block-domain normalized convolution using the BNLMS algorithm to obtain the linear echo estimation signal. S13. Subtract the microphone signal from the linear echo estimation signal to obtain the residual error signal.

3. The BNLMS speech signal echo cancellation method based on LSTM improvement according to claim 2, characterized in that, In step S12, the filter coefficient update formula of the BNLMS algorithm is: in, This is the filter order index value. For the first The filter coefficient vector before the next block update For the first The filter coefficient vector updated in the next block. To update the step size, To prevent regularization constants with denominators of zero, For block error vector, For block input matrices, For block length, For the first Normalized energy factor during next block update; in, For the first Normalized energy factor during next block update The energy smoothing coefficient, The trace of the matrix, This indicates the matrix transpose.

4. The BNLMS speech signal echo cancellation method based on LSTM improvement according to claim 1, characterized in that, Step S2 further includes: S21. Perform a short-time Fourier transform on the residual error signal to obtain the amplitude spectrum and phase spectrum of the residual error signal; S22. Organize the amplitude spectrum into a two-dimensional feature matrix according to the frame index and frequency index to obtain the time-frequency domain joint feature sequence.

5. The BNLMS speech signal echo cancellation method based on LSTM improvement according to claim 1, characterized in that, Step S3 further includes: S31. The time-frequency domain joint feature sequence is fed into the input layer for windowing processing to obtain a context feature vector composed of the current frame and historical frames. S32. The context feature vector is sequentially passed through the first and second LSTM hidden layers for gated temporal modeling to obtain the hidden state sequence. S33. The hidden state sequence is fed into the fully connected output layer, and the dimension is mapped by a linear activation function to obtain the time-frequency domain mask. S34. Multiply the time-frequency domain mask by the microphone signal spectrum to obtain a clean near-end speech spectrum estimation signal; S35. Based on the phase spectrum, the pure near-end speech spectrum estimation signal is reconstructed by inverse short-time Fourier transform to obtain the pure near-end speech estimation signal.

6. The BNLMS speech signal echo cancellation method based on LSTM improvement according to claim 1, characterized in that, Step S4 further includes: S41. The loss is calculated by applying the mean square error function to the pure near-end speech estimation signal and the pure near-end speech label to obtain the loss value; S42. Based on the loss value, the gradients of the LSTM network weight parameters and BNLMS filter coefficients are calculated simultaneously using the backpropagation algorithm to obtain the joint gradient vector. S43. Input the joint gradient vector into the Adam optimizer to update the parameters, obtain the updated network weight parameters and filter coefficients, and continuously optimize the training process until the loss value converges to obtain the optimal hybrid echo cancellation model.

7. The BNLMS speech signal echo cancellation method based on LSTM improvement according to claim 6, characterized in that, In step S41, the expression for the mean square error function is: in, This represents the loss value of the mean squared error function. For sampling point index, This represents the total number of sampling points used to calculate the loss in the current training batch. For the first The true label value of clean near-end speech at each sampling point For the first The clean near-end speech estimate output by the LSTM module at each sampling point.

8. The BNLMS speech signal echo cancellation method based on LSTM improvement according to claim 1, characterized in that, Step S5 further includes: S51. The real-time audio data collected by the microphone and the remote reference signal are processed by buffer block through the audio driver interface to obtain the remote signal block and the microphone signal block at the current moment. S52. The far-end signal block and the microphone signal block are fed into the optimal hybrid echo cancellation model for linear echo cancellation processing to obtain a real-time residual error signal block, and the real-time residual error signal block is stored in the LSTM input buffer. S53. The real-time residual error signal block is read from the LSTM input buffer through the LSTM module, and nonlinear mapping is performed on each sample in a sequential manner to obtain a real-time clean near-end speech estimation block, which is then sent to the output buffer for continuous audio output.

9. The BNLMS speech signal echo cancellation method based on LSTM improvement according to claim 8, characterized in that, In step S53, when outputting continuous audio, a pipelined parallel processing mechanism is used.

10. A BNLMS speech signal echo cancellation system based on LSTM improvement, used to perform the BNLMS speech signal echo cancellation method based on LSTM improvement as described in any one of claims 1 to 9, characterized in that, include: Filtering module: Used to perform adaptive linear filtering on the far-end reference signal and the microphone signal to obtain the residual error signal; Extraction module: used to extract features from the residual error signal, the far-end reference signal, and the microphone signal to obtain a joint time-frequency domain feature sequence; A mapping module, configured as a recurrent neural network model, is used to receive the joint time-frequency domain feature sequence and perform nonlinear mapping to obtain a clean near-end speech estimation signal; Training module: Used to perform joint gradient updates on the filter coefficients used for adaptive linear filtering and the network weight parameters in the recurrent neural network model with the loss value between the clean near-end speech estimation signal and the clean near-end speech label as the optimization objective, so as to obtain the optimal hybrid echo cancellation model; An output module is configured as the optimal hybrid echo cancellation model obtained by the training module for echo cancellation.

Images