Building talkback echo elimination method and system based on multi-level self-adaption and neural network fusion, and building talkback terminal

Abstract

The invention relates to a building intercom echo elimination method and system based on multistage self-adaption and neural network fusion and a building intercom terminal. The building intercom echo elimination method comprises the following steps: S1, estimating background noise power and loudspeaker nonlinear distortion indexes of a near end; s2, original parameters of a near-end echo cancellation system are selected according to scene classification, the echo cancellation system comprises an NLMS filter, a PAP filter and a Kalman filter which are subjected to parallel processing, the original parameters are dynamically adjusted according to the background noise power and the loudspeaker nonlinear distortion index, and the background noise power is obtained; filtering the near-end microphone signal according to the performance of each filter, and dynamically weighting and outputting the near-end microphone signal to obtain a preliminary echo cancellation signal; s3, multi-modal frequency domain features are extracted, the multi-modal frequency domain features are used for training a lightweight CNN + LSTM neural network, and a gain mask is extracted; and S4, applying the gain mask to the frequency domain signal and converting the frequency domain signal back to a time domain signal to obtain a net voice signal.

Description

Technical Field

[0001] This invention relates to the field of echo cancellation, specifically to a building intercom echo cancellation method, system, and building intercom terminal based on multi-level adaptive and neural network fusion. Background Technology

[0002] Building intercom systems are essential security and communication devices in modern buildings, primarily enabling two-way voice / video calls between the main unit at the building entrance and the user's indoor extension unit. The system connects via Ethernet, and both units are equipped with microphones for data capture and speakers for playback. Echoes can occur during building intercoms. The mechanism of this echo is as follows: when the voice signal captured by the main unit's microphone is transmitted over the network to the extension unit's speaker, due to acoustic coupling (airborne propagation), the extension unit's microphone will re-pick up this played sound (i.e., the echo) and transmit it back to the main unit via the network. This causes the user at the main unit to hear their own delayed echo, severely impacting call quality.

[0003] In existing technologies, the WebRTC AEC3 algorithm is generally used for echo cancellation. However, the WebRTC AEC3 algorithm has the following significant drawbacks when applied to building intercom systems: (1) Poor adaptability to architectural acoustic environments: A fixed 128-tap filter can only cover an impulse response of about 128ms (at a sampling rate of 16kHz), and cannot fully model the long reverberation tail of the doorway environment (T60 requires more than 256 taps to reach 1.5 seconds). In high reverberation environments, the ERLE (Echo Return Loss Enhancement) is only 28-32dB, and residual echoes are clearly perceptible. Fixed parameters cannot automatically adapt to changes in the acoustic environment caused by different building materials and space sizes.

[0004] (2) Insufficient nonlinear distortion handling capability: The algorithm is based on the assumption of a linear system. The speaker power of the intercom door station is high (3-5W), and significant nonlinear distortion (clipping, harmonic distortion) is generated when the volume is high. The linear filter cannot eliminate nonlinear echo components, resulting in residual echo of -25 to -30dB. It is easy to trigger howling (acoustic feedback) phenomenon.

[0005] (3) High false alarm rate in dual-talk detection: Using a single power ratio criterion, the power ratio calculation is distorted in high-noise doorway environments or under network jitter conditions. False alarms as dual-talk cause the filter to freeze incorrectly, and echo cancellation fails; false alarms as single-talk cause the filter to diverge, and near-end speech is canceled. The accuracy of dual-talk detection is only 84-86%, and the false alarm rate is 8-12%.

[0006] (4) Poor full-duplex performance: The conservative dual-talk detection strategy, in order to avoid misjudgment, leads to frequent voice truncation. The detection delay is 50-100ms, resulting in the loss of effective voice segments. Both parties cannot speak fluently at the same time, resulting in a poor user experience.

[0007] (5) Slow response to sudden environmental changes: When the acoustic environment changes (such as someone opening or closing a door at the entrance), it takes 3-5 seconds for the filter to reconverge. During the convergence process, the echo suppression performance drops significantly, affecting call continuity.

[0008] The purpose of this invention is to design a building intercom echo cancellation method, system, and building intercom terminal based on multi-level adaptive and neural network fusion to address the problems existing in the prior art. Summary of the Invention

[0009] To address the problems existing in the prior art, the present invention provides a building intercom echo cancellation method, system, and building intercom terminal based on multi-level adaptive and neural network fusion, which can effectively solve at least one of the problems existing in the prior art.

[0010] The technical solution of this invention is: A building intercom echo cancellation method based on multi-level adaptive and neural network fusion includes the following steps: S1. Estimate the reverberation time of the room where the near end is located based on the signals from the near end microphone and the far end microphone, and estimate the background noise power and speaker nonlinear distortion index of the near end. S2, classify the room where the near end is located according to the reverberation time, select the original parameters of the near end echo cancellation system according to the scene classification, the echo cancellation system includes parallel processing NLMS filter, PAP filter and Kalman filter, and dynamically adjust the original parameters according to the background noise power and the speaker nonlinear distortion index, filter the near end microphone signal according to the performance of each filter and dynamically weight the output to obtain the preliminary echo cancellation signal; S3, convert the initial echo cancellation signal into a frequency domain signal, extract multimodal frequency domain features, use the multimodal frequency domain features to train a lightweight CNN+LSTM neural network, and extract the gain mask; S4, apply the gain mask to the frequency domain signal and convert it back to the time domain signal to obtain the net speech signal.

[0011] Furthermore, in S1, estimating the reverberation time of the room where the near-end microphone is located based on the near-end microphone signal and the far-end microphone signal includes: The echo signal is estimated based on the near-end microphone signal, the difference between the near-end microphone signal and the echo signal is calculated to obtain the error signal, and the room impulse response RIR is obtained by autocorrelation and cross-correlation estimation of the far-end microphone signal and the error signal. The energy decay curve (EDC) of the room impulse response (RIR) is calculated, and then logarithmic transformation and linear fitting are performed to obtain the logarithmic decay curve. The reverberation time is obtained by fitting the logarithmic decay curve.

[0012] Furthermore, the room scene is classified according to the reverberation time, and the original parameters of the near-end echo cancellation system are selected according to the scene classification, including: If the reverberation time is greater than 0.8s, the scene is classified as a high reverberation scene, and the original parameters of the echo cancellation system are set as follows: filter length of 256 taps, NLMS step size of 0.2-0.4, and double talk threshold of 3.0dB. If the reverberation time is 0.4-0.8s, the scene is classified as a medium reverberation scene, and the original parameters of the echo cancellation system are set as follows: filter length is 192 taps, NLMS step size is 0.3-0.6, and double talk threshold is 2.0dB. If the reverberation time is less than 0.4s, the scene is classified as a low reverberation scene, and the original parameters of the echo cancellation system are set as follows: filter length of 128 taps, NLMS step size of 0.4-0.8, and double talk threshold of 1.0dB.

[0013] Furthermore, dynamically adjusting the original parameters based on the background noise power and the speaker nonlinear distortion index includes: If the background noise power is greater than the preset noise power threshold, increase the filter length and dual-talk threshold by 30% to 50%, and reduce the NLMS step size by 30% to 50%; conversely, if the background noise power is less than the preset noise power threshold, reduce the filter length and dual-talk threshold by 30% to 50%, and increase the NLMS step size by 30% to 50%; and / or, If the speaker's nonlinear distortion index is greater than the preset distortion threshold, the filter length and dual-talk threshold will be increased by 30% to 50%, and the NLMS step size will be reduced by 30% to 50%. Conversely, if the nonlinear distortion index is less than the preset distortion threshold, the filter length and dual-talk threshold will be reduced by 30% to 50%, and the NLMS step size will be increased by 30% to 50%.

[0014] Furthermore, based on the performance of each filter, the near-end microphone signal is filtered and dynamically weighted for output, resulting in a preliminary echo cancellation signal including: Near-end microphone signals are processed using NLMS filters, PAP filters, and Kalman filters, respectively. Calculate the echo suppression gain of the output signals of the NLMS filter, PAP filter, and Kalman filter respectively; The filters with higher echo suppression gains are assigned higher weights, and the outputs of each filter are fused according to the weights to obtain a preliminary echo cancellation signal.

[0015] Furthermore, filters with higher echo suppression gains are assigned higher weights, and their outputs are weighted and fused using the Softmax function based on the echo suppression gains of each filter.

[0016] Furthermore, after step S2 and before step S3, the following is executed: Calculate the energy ratio, cross-correlation, and spectral deviation of the far-end microphone signal and the near-end microphone signal; The energy ratio, the cross-correlation, and the spectral deviation are weighted and summed to obtain the fused dual-talk feature; If the fused dual-talk feature is greater than the preset dual-talk threshold, it is determined to be a dual-talk state and the parameters of each filter are frozen.

[0017] Further, extracting multimodal frequency domain features, training a CNN neural network with the multimodal frequency domain features to extract local frequency domain features, and training an LSTM neural network with the local frequency domain features to extract a gain mask includes: The amplitude spectrum, phase difference, logarithmic power spectrum, spectral flatness, and cross-correlation features with the reference signal are extracted from the frequency domain signal, and the amplitude spectrum, phase difference, logarithmic power spectrum, spectral flatness, and cross-correlation features with the reference signal are normalized. The normalized amplitude spectrum, phase difference, logarithmic power spectrum, spectral flatness, and cross-correlation characteristics with the reference signal are combined to form a multidimensional feature tensor. The multidimensional feature tensor is used to train a CNN neural network with the multimodal frequency domain features to extract local frequency domain features, and the local frequency domain features are used to train an LSTM neural network to extract a gain mask.

[0018] Furthermore, a building intercom echo cancellation system based on multi-level adaptive and neural network fusion is provided to implement the method, comprising the following modules: The acoustic scene recognition and preprocessing module is used to estimate the reverberation time of the room where the near end is located based on the signals from the near end microphone and the far end microphone, and to estimate the background noise power and nonlinear distortion index of the near end microphone. A multi-level adaptive filter parallel processing module is used to classify the scene of the room where the near end is located according to the reverberation time, select the original parameters of the near-end echo cancellation system according to the scene classification, the echo cancellation system includes parallel processing NLMS filter, PAP filter and Kalman filter, and dynamically adjust the original parameters according to the background noise power and the speaker nonlinear distortion index, filter the near-end microphone signal according to the performance of each filter and dynamically weight the output to obtain the preliminary echo cancellation signal; A lightweight neural network residual echo suppression module is used to convert the initial echo cancellation signal into a frequency domain signal, extract multimodal frequency domain features, use the multimodal frequency domain features to train a lightweight CNN+LSTM neural network, and extract a gain mask. The output module is used to apply the gain mask to the frequency domain signal and convert it back to the time domain signal to obtain the net speech signal.

[0019] Furthermore, a building intercom terminal is provided, which implements any of the methods described in the present invention during operation.

[0020] Therefore, the present invention provides the following effects and / or advantages: This application achieves acoustic scene recognition and preprocessing by real-time estimation of reverberation time T60 and scene classification, automatically adjusting the parameters of the echo cancellation algorithm to adapt to different acoustic environments. By dynamically adjusting the filter length and step size, the system's adaptability to different architectural environments is enhanced. Multi-level adaptive filters are used in parallel processing to address different environmental conditions, summarizing the advantages of multiple filters to improve echo suppression performance, especially in high-noise and complex environments. Finally, combined with neural network residual echo suppression, a lightweight CNN+LSTM neural network is introduced, using spectral masking to suppress residual echoes, thereby significantly improving the echo cancellation effect.

[0021] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained through the structures particularly pointed out in the description and the drawings.

[0022] It should be understood that the above summary and the following detailed description of the invention are exemplary and explanatory, and are intended to provide further explanation of the invention as claimed. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating one embodiment of the present invention.

[0024] Figure 2 This is the logic block diagram after dual-talk detection.

[0025] Figure 3 The following is a logic block diagram for steps S3 and S4. Detailed Implementation

[0026] To facilitate understanding by those skilled in the art, the present invention will now be described in further detail with reference to the embodiments: refer to Figure 1 A method for echo cancellation in building intercom systems based on multi-level adaptive and neural network fusion, characterized by the following steps: S1. Estimate the reverberation time of the room where the near end is located based on the signals from the near end microphone and the far end microphone, and estimate the background noise power and speaker nonlinear distortion index of the near end. S2, classify the room scene of the near end according to the reverberation time, select the parameters of the near end echo cancellation system according to the scene classification, establish NLMS filter, PAP filter and Kalman filter to process the near end microphone signal in parallel, and dynamically adjust the parameters of NLMS filter, PAP filter and Kalman filter according to the background noise power and the speaker nonlinear distortion index, and dynamically weight the output according to the performance of each filter to obtain the preliminary echo cancellation signal; S3, convert the initial echo cancellation signal into a frequency domain signal, extract multimodal frequency domain features, train the multimodal frequency domain features into a CNN neural network to extract local frequency domain features, and train the local frequency domain features into an LSTM neural network to extract a gain mask; S4, apply the gain mask to the frequency domain signal and convert it back to the time domain signal to obtain the net speech signal.

[0027] This application achieves acoustic scene recognition and preprocessing by real-time estimation of reverberation time T60 and scene classification, automatically adjusting the parameters of the echo cancellation algorithm to adapt to different acoustic environments. By dynamically adjusting the filter length and step size, the system's adaptability to different architectural environments is enhanced. Multi-level adaptive filters are used in parallel processing to address different environmental conditions, summarizing the advantages of multiple filters to improve echo suppression performance, especially in high-noise and complex environments. Finally, combined with neural network residual echo suppression, a lightweight CNN+LSTM neural network is introduced, using spectral masking to suppress residual echoes, thereby significantly improving the echo cancellation effect.

[0028] Specifically, in S1, estimating the reverberation time of the room where the near-end microphone is located based on the near-end microphone signal and the far-end microphone signal includes: The echo signal is estimated based on the near-end microphone signal, the difference between the near-end microphone signal and the echo signal is calculated to obtain the error signal, and the room impulse response RIR is obtained by autocorrelation and cross-correlation estimation of the far-end microphone signal and the error signal. The energy decay curve (EDC) of the room impulse response (RIR) is calculated, and then logarithmic transformation and linear fitting are performed to obtain the logarithmic decay curve. The reverberation time is obtained by fitting the logarithmic decay curve.

[0029] The room impulse response (RIR) refers to the characteristics of the sound signal received at the near-end microphone, including the direct path and all reflected paths. The basic methods for calculating the RIR are cross-correlation or adaptive filtering. Typically, in single-talk mode, the RIR is calculated by cross-correlation of the far-end reference signal and the near-end error signal. ,in This is the signal from the near-end microphone. Let (τ) be the error signal, and (τ) be the time delay.

[0030] After obtaining the room impulse response (RIR), the next step is to calculate the energy decay curve. The result is obtained by summing the energies of the RIR signals: , where M is the total length of the RIR and k is the time index.

[0031] Next to A logarithmic transformation is performed, and a portion of the decay curve is fitted using the linear least squares method to obtain the logarithmic decay curve, which represents the slope of the reverberation time T60. The reverberation time can be obtained by fitting this logarithmic decay curve.

[0032] Furthermore, the room scene is classified according to the reverberation time, and the original parameters of the near-end echo cancellation system are selected according to the scene classification, including: If the reverberation time is greater than 0.8s, the scene is classified as a high reverberation scene, and the original parameters of the echo cancellation system are set as follows: filter length of 256 taps, NLMS step size of 0.2-0.4, and double talk threshold of 3.0dB. If the reverberation time is 0.4-0.8s, the scene is classified as a medium reverberation scene, and the original parameters of the echo cancellation system are set as follows: filter length is 192 taps, NLMS step size is 0.3-0.6, and double talk threshold is 2.0dB. If the reverberation time is less than 0.4s, the scene is classified as a low reverberation scene, and the original parameters of the echo cancellation system are set as follows: filter length of 128 taps, NLMS step size of 0.4-0.8, and double talk threshold of 1.0dB.

[0033] In this embodiment, scene classification based on reverberation time T60 is a common method in the fields of acoustics and echo cancellation, and its specific principles will not be explained here. This is because reverberation time directly reflects the acoustic characteristics of a room and the duration of echoes. Environments with long reverberation times typically have longer echo durations; therefore, reverberation time is used to help determine the environment type, thereby automatically setting the original parameters of the echo cancellation system. Scene classification and specific parameter settings can be found in the table below.

[0034] Table 1. Scene Classification and Specific Parameter Settings

[0035] Furthermore, dynamically adjusting the original parameters based on the background noise power and the speaker nonlinear distortion index includes: If the background noise power is greater than the preset noise power threshold, increase the filter length and dual-talk threshold by 30% to 50%, and reduce the NLMS step size by 30% to 50%; conversely, if the background noise power is less than the preset noise power threshold, reduce the filter length and dual-talk threshold by 30% to 50%, and increase the NLMS step size by 30% to 50%; and / or, If the speaker's nonlinear distortion index is greater than the preset distortion threshold, the filter length and dual-talk threshold will be increased by 30% to 50%, and the NLMS step size will be reduced by 30% to 50%. Conversely, if the nonlinear distortion index is less than the preset distortion threshold, the filter length and dual-talk threshold will be reduced by 30% to 50%, and the NLMS step size will be increased by 30% to 50%.

[0036] In this embodiment, the speaker's nonlinear distortion index can be measured by calculating the clipping rate, which represents the proportion of the near-end speaker exceeding its maximum amplitude. When the clipping rate exceeds 5% or higher, it indicates significant clipping distortion in the speaker output, potentially requiring the activation of a nonlinear compensation mechanism to correct the distorted signal. Alternatively, the total harmonic distortion (THD) can be calculated to obtain the distortion ratio between the fundamental frequency and its harmonic components of the speaker signal. If the THD exceeds 10% or higher, it indicates severe nonlinear distortion in the speaker output, potentially requiring the activation of nonlinear compensation.

[0037] Having obtained the original parameters of each filter, we will now adaptively adjust these parameters based on the actual situation. For background noise power: The filter length is directly related to the echo duration the system can handle. In high-noise environments, due to the low signal-to-noise ratio, the filter needs to accommodate more noise components, which may require a longer filter length. However, in low-noise environments, a shorter filter is sufficient to handle the echo.

[0038] The filter step size is a key parameter controlling the filter update speed. In environments with strong background noise, an excessively large step size may lead to over-adjustment of the filter, resulting in unstable echo cancellation effects or even falsely canceling speech signals. Therefore, the step size needs to be dynamically adjusted according to the intensity of the background noise. In noisy environments, the filter step size is reduced to avoid over-adjusting the filter coefficients and prevent excessive noise interference. Reducing the step size helps to make the filter update smoother and avoids falsely canceling speech signals.

[0039] The duotalk threshold is used to determine whether duotalk (i.e., two people speaking simultaneously) exists. In environments with strong background noise, noise may interfere with duotalk detection, leading to false positives. Therefore, in noisy environments, the false positive rate of duotalk detection increases. The system will increase the duotalk threshold to reduce false positives in noisy conditions and avoid erroneously freezing the filter parameters.

[0040] In environments with severe nonlinear distortion, a nonlinear compensation mechanism needs to be activated, while in environments with less nonlinear distortion, the system uses a standard linear filter for echo cancellation. Regarding loudspeaker nonlinear distortion specifications: In environments with high loudspeaker nonlinear distortion specifications, the system requires finer filters to handle distorted signals due to the nonlinear distortion generated by the loudspeaker (such as clipping or harmonic distortion). In this case, the filter length needs to be increased to better accommodate long-term echoes and distorted signals.

[0041] Due to the complex echoes caused by speaker distortion, the system may require a smaller step size to ensure filter stability. A small step size can prevent the filter from updating too drastically, thereby reducing distortion and howling.

[0042] Two-way detection can be more prone to misjudging or losing voice signals. To avoid this, the system may need to set a higher two-way threshold to ensure that the system does not freeze filter updates due to misjudgment of two-way communication.

[0043] Furthermore, based on the performance of each filter, the near-end microphone signal is filtered and dynamically weighted for output, resulting in a preliminary echo cancellation signal including: Near-end microphone signals are processed using NLMS filters, PAP filters, and Kalman filters, respectively. Calculate the echo suppression gain of the output signals of the NLMS filter, PAP filter, and Kalman filter respectively; The filters with higher echo suppression gains are assigned higher weights, and the outputs of each filter are fused according to the weights to obtain a preliminary echo cancellation signal.

[0044] In this process, higher weights are assigned to filters with higher echo suppression gains, and the outputs of each filter are weighted and fused using the Softmax function based on their echo suppression gains.

[0045] In this step, multiple adaptive filters are used to process the signal, the performance of each filter is calculated, and then their outputs are dynamically weighted and fused based on the performance of each filter. The NLMS filter is used for fast convergence, making it particularly suitable for scenarios with rapidly changing environments. The PAP filter continuously optimizes when the environment is stable, making it suitable for relatively stable environments. The Kalman filter handles noisy environments through optimal estimation, making it suitable for high-noise or unstable signal scenarios. Given the uncertainties in the environment of the intercom system, including echo and variations in ambient temperature, volume, and noise, multiple filters can be used for parallel filtering. The echo suppression gain of each filter can then be calculated by comparing the power ratio of the original near-end microphone signal to the filtered signal.

[0046] In the weighted fusion process, weight setting is a crucial step in optimizing the echo cancellation system. The weights of each filter are dynamically calculated based on its echo suppression gain. The core idea is to determine the contribution of each filter to the final result based on its performance in the current environment, ensuring that the system can automatically adapt and optimize the echo cancellation effect in different scenarios. The larger the echo suppression gain, the better the echo suppression effect of the filter. The Softmax function calculates the weight of each filter by normalizing the ERLE of each filter. At the same time, the Softmax function can prevent any filter from having an excessively large weight in the weighting process, avoiding over-reliance on the output of any single filter.

[0047] Furthermore, the weights output by the Softmax function can be smoothed. Smoothed weights can avoid instability during the weighting process, maintaining a stable signal output. In echo cancellation systems, sudden environmental changes, such as a door being opened, people entering or exiting, or a sudden increase in background noise, can cause drastic fluctuations in filter performance, thus affecting its echo cancellation capability. For example, sudden noise or changes in the acoustic environment may cause the performance of some filters to deteriorate momentarily; in such cases, fluctuations in weights may lead to output instability.

[0048] Specifically, the weights from multiple time periods are arranged into a two-dimensional matrix. The rows of the matrix represent all weights within a given time period, and the columns represent the time progression. A nonlocal mean algorithm is used to smooth the two-dimensional matrix. Furthermore, the smoothing parameter of the nonlocal mean algorithm is reduced when there are sudden power changes in the near-end microphone signal. This is because reducing the smoothing parameter of the nonlocal mean algorithm during signal power changes helps make the system more sensitive to real-time changes, rather than relying on historical weights. This helps the system adapt to environmental changes more quickly and avoids response delays caused by over-smoothing.

[0049] Further, refer to Figure 2 After step S2 and before step S3, execute: Calculate the energy ratio, cross-correlation, and spectral deviation of the far-end microphone signal and the near-end microphone signal; The energy ratio, the cross-correlation, and the spectral deviation are weighted and summed to obtain the fused dual-talk feature; If the fused dual-talk feature is greater than the preset dual-talk threshold, it is determined to be a dual-talk state and the parameters of each filter are frozen.

[0050] In this embodiment, the power ratio is used to determine the energy ratio of the far-end signal and the near-end signal in the current microphone signal. If the energy of the far-end signal dominates, it indicates a one-way talk state; if the energy of the near-end signal increases and exceeds a certain threshold, it indicates a possible two-way talk state. Cross-correlation characteristics can be used to determine the correlation between the near-end microphone signal and the far-end reference signal. In a one-way talk state, the two signals should be highly correlated; while in a two-way talk state, the signal correlation decreases. Spectral deviation can be used to detect inconsistencies in the signal spectrum. In a two-way talk state, the spectra of the near-end signal and the far-end signal should have some differences, therefore the spectral deviation is relatively large.

[0051] The fused dual-talk feature DTD(n) = 0.4 × R_norm(n) + 0.3 × C_feature(n) + 0.3 × S(n), where R_norm(n) represents the energy ratio, C_feature(n) represents the cross-correlation, S(n) represents the spectral deviation, and 0.4, 0.3, and 0.3 represent the weights of the energy ratio, the cross-correlation, and the spectral deviation, respectively.

[0052] When dual-talk mode is detected, the echo cancellation system typically needs to freeze filter updates to avoid interference with the voice signal during dual-talk. Additionally, anti-jitter mechanisms, such as hysteresis thresholds, can be incorporated, ensuring that dual-talk mode is only truly switched to after a certain duration.

[0053] Further, refer to Figure 3Extracting multimodal frequency domain features, training a CNN neural network with the multimodal frequency domain features to extract local frequency domain features, and training an LSTM neural network with the local frequency domain features to extract a gain mask includes: The amplitude spectrum, phase difference, logarithmic power spectrum, spectral flatness, and cross-correlation features with the reference signal are extracted from the frequency domain signal, and the amplitude spectrum, phase difference, logarithmic power spectrum, spectral flatness, and cross-correlation features with the reference signal are normalized. The normalized amplitude spectrum, phase difference, logarithmic power spectrum, spectral flatness, and cross-correlation characteristics with the reference signal are combined to form a multidimensional feature tensor. The multidimensional feature tensor is used to train a CNN neural network with the multimodal frequency domain features to extract local frequency domain features, and the local frequency domain features are used to train an LSTM neural network to extract a gain mask.

[0054] Specifically, the calculation methods for amplitude spectrum, phase difference, logarithmic power spectrum, spectral flatness, and cross-correlation characteristics with the reference signal in this step are directly adopted from existing technologies, and these five characteristics are combined into a multidimensional feature tensor [257×5×5].

[0055] The multidimensional feature tensor is first processed by a CNN neural network to extract local spatial patterns. These multidimensional spectral features are then processed to learn their local patterns, particularly local features of the spectral information, such as activity patterns at certain frequency points. These patterns aid in echo recognition and suppression. After the CNN extracts the local features, they are fed into an LSTM neural network, which is used to capture long-term dependencies in time-series data. In speech signal processing, LSTM learns the temporal trends of the signal, enabling it to extract temporal dependencies from consecutive frames and identify change patterns in speech. The output of the LSTM process yields a gain mask, which is used to selectively suppress echoes (mask value close to 0) or preserve the speech signal (mask value close to 1).

[0056] Furthermore, a building intercom echo cancellation system based on multi-level adaptive and neural network fusion is provided to implement the method, comprising the following modules: The acoustic scene recognition and preprocessing module is used to estimate the reverberation time of the room where the near end is located based on the signals from the near end microphone and the far end microphone, and to estimate the background noise power and nonlinear distortion index of the near end microphone. A multi-level adaptive filter parallel processing module is used to classify the scene of the room where the near end is located according to the reverberation time, select the original parameters of the near-end echo cancellation system according to the scene classification, the echo cancellation system includes parallel processing NLMS filter, PAP filter and Kalman filter, and dynamically adjust the original parameters according to the background noise power and the speaker nonlinear distortion index, filter the near-end microphone signal according to the performance of each filter and dynamically weight the output to obtain the preliminary echo cancellation signal; A lightweight neural network residual echo suppression module is used to convert the initial echo cancellation signal into a frequency domain signal, extract multimodal frequency domain features, use the multimodal frequency domain features to train a lightweight CNN+LSTM neural network, and extract a gain mask. The output module is used to apply the gain mask to the frequency domain signal and convert it back to the time domain signal to obtain the net speech signal.

[0057] Furthermore, a building intercom terminal is provided, which implements any of the methods described in the present invention during operation.

[0058] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0059] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0060] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0061] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0062] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

Claims

1. A method for echo cancellation of a building intercom based on multi-stage adaptive and neural network fusion, characterized in that: The method comprises the following steps: S1, estimating the reverberation time of the room where the near-end is located according to the signals of the near-end microphone and the far-end microphone, and estimating the background noise power and the loudspeaker nonlinear distortion index of the near-end; S2, classifying the scene of the room where the near-end is located according to the reverberation time, selecting the original parameters of the echo cancellation system of the near-end according to the scene classification, the echo cancellation system comprising parallel processing NLMS filter, PAP filter and Kalman filter, and dynamically adjusting the original parameters according to the background noise power and the loudspeaker nonlinear distortion index, filtering the near-end microphone signal according to the performance of each filter and dynamically weighting the output to obtain a preliminary echo cancellation signal; S3, converting the preliminary echo cancellation signal into a frequency domain signal, extracting a multi-modal frequency domain feature, using the multi-modal frequency domain feature to train a lightweight CNN+LSTM neural network, and extracting a gain mask; S4, applying the gain mask to the frequency domain signal and converting it back to a time domain signal to obtain a clean speech signal.

2. The echo cancellation method for building intercom based on multi-stage adaptive and neural network fusion according to claim 1, characterized in that: In S1, the reverberation time of the room where the near-end is located is estimated according to the signals of the near-end microphone and the far-end microphone, which comprises: estimating an echo signal according to the near-end microphone signal, calculating the difference between the near-end microphone signal and the echo signal to obtain an error signal, and performing autocorrelation and cross-correlation estimation on the far-end microphone signal and the error signal to obtain the room impulse response RIR; calculating the energy decay curve EDC of the room impulse response RIR, then performing logarithmic transformation and linear fitting to obtain the logarithmic decay curve, and fitting the logarithmic decay curve to obtain the reverberation time.

3. The echo cancellation method for building intercom based on multi-stage adaptive and neural network fusion according to claim 1, characterized in that: According to the scene classification, the original parameters of the echo cancellation system of the near-end are selected, which comprises: if the reverberation time is greater than 0.8s, the scene is classified as a high-reverberation scene, and the original parameters of the echo cancellation system are set as: filter length of 256 taps, NLMS step of 0.2-0.4, and double-talk threshold of 3.0dB; if the reverberation time is 0.4-0.8s, the scene is classified as a medium-reverberation scene, and the original parameters of the echo cancellation system are set as: filter length of 192 taps, NLMS step of 0.3-0.6, and double-talk threshold of 2.0dB; if the reverberation time is less than 0.4s, the scene is classified as a low-reverberation scene, and the original parameters of the echo cancellation system are set as: filter length of 128 taps, NLMS step of 0.4-0.8, and double-talk threshold of 1.0dB.

4. The multi-stage adaptive and neural network fusion-based echo cancellation method for building intercom according to claim 3, characterized in that: The original parameters are dynamically adjusted according to the background noise power and the loudspeaker nonlinear distortion index, which comprises: if the background noise power is greater than the preset noise power threshold, the filter length and the double-talk threshold are increased by 30% to 50%, the NLMS step is reduced by 30% to 50%, otherwise the filter length and the double-talk threshold are reduced by 30% to 50%, and the NLMS step is increased by 30% to 50%; and / or, If the loudspeaker nonlinear distortion index is greater than the preset distortion threshold, the filter length and the double-talk threshold are increased by 30% to 50%, and the NLMS step size is reduced by 30% to 50%, and vice versa.

5. The echo cancellation method for building intercom based on multi-stage adaptive and neural network fusion according to claim 1, characterized in that: Filtering the near-end microphone signal according to the performance of each filter and dynamically weighting the output to obtain a preliminary echo cancellation signal includes: Processing the near-end microphone signal through NLMS filters, PAP filters, and Kalman filters respectively; Calculating the echo suppression gain of the signals output by the NLMS filters, PAP filters, and Kalman filters respectively; The higher the echo suppression gain of the filter, the higher the weight assigned to it, and the outputs of each filter are fused according to the weight to obtain a preliminary echo cancellation signal.

6. The multi-stage adaptive and neural network fusion based echo cancellation method for building intercom according to claim 5, characterized in that: In the process of assigning higher weights to filters with higher echo suppression gains, the outputs of each filter are weighted and fused according to the echo suppression gain of each filter through a Softmax function.

7. The method according to claim 1, wherein the method is a multi-stage adaptive and neural network fusion based intercom echo cancellation method. After step S2 and before step S3, the following steps are performed: Calculate the energy ratio, cross-correlation, and spectral deviation of the far-end microphone signal and the near-end microphone signal; Weight and add the energy ratio, cross-correlation, and spectral deviation to obtain a fused double-talk feature; If the fused double-talk feature is greater than the preset double-talk threshold, it is determined to be a double-talk state and the parameters of each filter are frozen.

8. The method according to claim 1, wherein the method is a multi-stage adaptive and neural network fusion based intercom echo cancellation method. Extracting multi-modal frequency domain features, training a CNN neural network using the multi-modal frequency domain features to extract local frequency domain features, and training an LSTM neural network using the local frequency domain features to extract a gain mask includes: Extracting amplitude spectrum, phase difference, log power spectrum, spectral flatness, and cross-correlation features with the reference signal from the frequency domain signal, and normalizing the amplitude spectrum, phase difference, log power spectrum, spectral flatness, and cross-correlation features with the reference signal; The normalized amplitude spectrum, phase difference, log power spectrum, spectral flatness, and cross-correlation features with the reference signal form a multi-dimensional feature tensor. The multi-dimensional feature tensor is used to train a CNN neural network using the multi-modal frequency domain features to extract local frequency domain features, and the local frequency domain features are used to train an LSTM neural network to extract a gain mask.

9. A multi-stage adaptive and neural network fusion based intercom echo cancellation system, characterized in that: A method for implementing any of claims 1-8 includes the following modules: An acoustic scene recognition and preprocessing module for estimating the reverberation time of the room where the near-end is located based on the near-end microphone signal and the far-end microphone signal, and estimating the background noise power and loudspeaker nonlinear distortion index of the near-end; A multi-stage adaptive filter parallel processing module for classifying the scene of the room where the near-end is located based on the reverberation time, selecting the original parameters of the echo cancellation system of the near-end based on the scene classification, the echo cancellation system including parallel processing NLMS filters, PAP filters, and Kalman filters, and dynamically adjusting the original parameters based on the background noise power and the loudspeaker nonlinear distortion index, filtering the near-end microphone signal according to the performance of each filter and dynamically weighting the output to obtain a preliminary echo cancellation signal; A lightweight neural network residual echo suppression module is configured to convert the preliminary echo cancellation signal into a frequency domain signal, extract multi-modal frequency domain features, use the multi-modal frequency domain features to train a lightweight CNN+LSTM neural network, and extract a gain mask. An output module is configured to apply the gain mask to the frequency domain signal and convert it back into a time domain signal to obtain a clean speech signal.

10. A door entry terminal, characterized by In operation, the method of any one of claims 1-8 is implemented.

Images