Intelligent echo reduction audio processing method and system

Abstract

The invention relates to an intelligent echo reduction audio processing method and system, and the method comprises the steps: obtaining a sound field signal and a loudspeaker reference signal, and carrying out the signal framing and short-time Fourier transform, and obtaining time-frequency spectrum audio data; performing linear echo cancellation processing on the time-frequency spectrum audio data, and dynamically adjusting a convergence rate to obtain a preliminary filtering signal; performing nonlinear residual echo cancellation on the preliminary filtering signal, and performing multi-reflection path generation to obtain a nonlinear cancellation signal; performing multi-mode residual suppression on the nonlinear cancellation signal, and performing frequency spectrum processing through sub-band division and joint judgment to obtain an echo suppression signal; and acquiring real-time environment sound field change data, and performing acoustic enhancement on the echo suppression signal to obtain target echo reduction audio output. According to the invention, basic echo components can be identified and filtered more accurately.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and in particular to an intelligent echo reduction audio processing method and system. Background Art

[0002] As an important medium for modern information exchange, audio communication systems have been widely used in remote conferencing, smart assistants, and voice control devices. With the rapid development of audio interaction technology and the continuous improvement of user experience requirements, how to effectively eliminate echo interference and improve audio quality and communication clarity has become one of the research focuses. Existing echo cancellation technologies often only use a single processing method, such as simple linear filtering or fixed parameter spectrum suppression, while ignoring the complexity of the actual acoustic environment and the comprehensive impact of multipath propagation on echo formation. This simplified method may result in poor echo cancellation effects, especially under conditions of nonlinear distortion and dynamic environmental changes, which significantly affects the overall performance of the audio system and user experience. Summary of the invention

[0003] The main purpose of the present invention is to provide an intelligent echo reduction audio processing method and system, which can more accurately identify and filter basic echo components, thereby improving the initial effect of echo cancellation.

[0004] To achieve the above object, the present invention provides an intelligent echo reduction audio processing method, comprising: Acquire the sound field signal and the speaker reference signal, perform signal framing and short-time Fourier transform, and obtain time-frequency spectrum audio data; Performing linear echo cancellation processing on the time-frequency spectrum audio data, and dynamically adjusting the convergence speed to obtain a preliminary filtered signal; performing nonlinear residual echo elimination on the preliminary filtered signal and performing multi-reflection path generation to obtain a nonlinear elimination signal; Performing multimodal residual suppression on the nonlinear elimination signal, performing spectrum processing through sub-band division and joint decision, and obtaining an echo suppression signal; Real-time environmental sound field change data is acquired, and the echo suppression signal is acoustically enhanced to obtain a target echo-reduced audio output.

[0005] Furthermore, the acquisition of the sound field signal and the speaker reference signal, signal framing and short-time Fourier transform, and obtaining time-frequency spectrum audio data include: Performing synchronous digital sampling on the sound field signal and the loudspeaker reference signal according to a preset sampling rate and bit depth to obtain a digital domain original signal; Divide the original digital domain signal into time frames according to preset frame length and frame shift parameters to obtain a frame sequence; Optimizing spectrum leakage of the frame sequence according to a preset Hanning window function to obtain a windowed signal frame; Performing a fast Fourier transform on the windowed signal frame to obtain complex frequency domain coefficients; Calculating the amplitude spectrum and the phase spectrum according to the complex frequency domain coefficients, and performing frequency band energy analysis to obtain a frequency band energy distribution matrix; Performing Mel frequency scale mapping on the frequency band energy distribution matrix to obtain Mel filter bank coefficients; The Mel filter bank coefficients and the complex frequency domain coefficients are used to construct three-dimensional features to obtain the time-frequency spectrum audio data.

[0006] Furthermore, the performing of linear echo cancellation processing on the time-frequency spectrum audio data and dynamically adjusting the convergence speed to obtain a preliminary filtered signal includes: Performing multi-sub-band decomposition on the time-frequency spectrum audio data to obtain multiple frequency sub-band signals; Performing echo path analysis on the multiple frequency sub-band signals to obtain a linear echo path signal; Performing instantaneous imbalance measurement calculation on the sound field signal and the loudspeaker reference signal to obtain a convergence state parameter; Adjust the step length parameter according to the convergence state parameter to obtain an optimized step length value; Performing upper and lower threshold setting mapping on the optimized step value to obtain a stable step parameter; Perform frequency domain filtering on the time-frequency spectrum audio data according to the stable step size parameter to obtain a linear echo estimation signal; Signal optimization is performed on the time-frequency spectrum audio data according to the linear echo estimation signal, and residual correlation compensation is performed to obtain the preliminary filtered signal.

[0007] Further, performing nonlinear residual echo elimination on the preliminary filtered signal and generating multiple reflection paths to obtain a nonlinear elimination signal includes: Performing nonlinear residual basis construction on the preliminary filtered signal to obtain a coupling characteristic matrix; Constructing a Volterra kernel expansion space according to the coupling characteristic matrix, and updating the preliminary filtering signal in real time to obtain optimized filter parameters; Perform multi-reflection path analysis based on the optimized filter parameters to obtain a set of adversarial reflection paths; Superimposing the countermeasure reflection path set and the preliminary filtered signal in time and frequency domains to obtain a multipath suppression signal; Performing sub-band energy distribution analysis on the multipath suppression signal, and generating a dynamic time-frequency masking matrix according to a preset threshold value; The multipath suppression signal is subjected to a point-by-point multiplication operation according to the dynamic time-frequency masking matrix to obtain the nonlinear elimination signal.

[0008] Furthermore, the Volterra kernel expansion space is constructed according to the coupling characteristic matrix, and the preliminary filtering signal is updated in real time to obtain optimized filter parameters, including: Performing singular value decomposition on the coupling feature matrix to obtain a reduced-dimensional feature space; Constructing separation components for the dimension-reduced feature space to obtain a separation vector group; Constructing the second-order expansion term of the Volterra kernel function according to the separation vector group, and performing weighted summation of the expansion term to obtain a second-order kernel expansion matrix; Constructing a third-order cross term based on the separation vector group and the second-order kernel expansion matrix, and performing a tensor product operation to obtain a third-order kernel expansion tensor; Performing an inner product operation on the second-order kernel expansion matrix and the third-order kernel expansion tensor to obtain a spatial mapping function; Performing nonlinear projection on the preliminary filtered signal according to the spatial mapping function to obtain a projection coefficient vector; The projection coefficient vector is optimized by least squares iteration to obtain the optimized filter parameters.

[0009] Furthermore, performing multimodal residual suppression on the nonlinear elimination signal, performing spectrum processing by sub-band division and joint decision, and obtaining an echo suppression signal, includes: performing sub-band decomposition on the nonlinear elimination signal to obtain a non-uniform sub-band representation; Performing multimodal feature extraction and integration on the non-uniform sub-band representation to obtain a multimodal feature set; Calculating cross entropy and mutual information according to the multimodal feature set to obtain a subband grouping structure; Adaptively grouping the non-uniform subband representations according to the subband grouping structure to obtain multiple subband sets; Performing decision rule evaluation on the multiple groups of subband sets respectively to obtain multiple suppression decision values; Performing joint decision processing on the multiple suppression decision values ​​to obtain a target suppression strategy; Performing differential processing on the multiple groups of sub-band sets according to the target suppression strategy to obtain a modal cooperative suppression signal; performing variational decomposition phase compensation on the modal cooperative suppression signal to obtain a phase coordination signal; Asymmetric singular value decomposition and reconstruction are performed on the phase coordination signal to obtain the echo suppression signal.

[0010] Furthermore, the acquiring of real-time environmental sound field change data, acoustically enhancing the echo suppression signal, and obtaining a target echo-reduced audio output includes: Acquire environmental sensor data, and perform spatial spectrum decomposition and sub-band analysis on the environmental sensor data according to the sound field signal to obtain spectrum characteristics of the environmental sound field; Calculating acoustic parameter changes on the ambient sound field spectrum characteristics and the ambient sensor data to obtain the real-time ambient sound field change data; Performing multi-stage cascade filtering on the echo suppression signal according to the real-time environmental sound field change data to obtain a preliminary enhanced signal; Performing phase correction and spectrum smoothing processing on the preliminary enhanced signal, adjusting the gain factor according to the real-time environmental sound field change data, and obtaining a frequency domain enhanced signal; Selectively suppressing residual noise on the frequency domain enhanced signal and the real-time ambient sound field change data to obtain a noise suppressed signal; Performing auditory frequency band energy adjustment on the noise suppression signal to obtain a time domain enhanced signal; The time domain enhanced signal is subjected to harmonic reconstruction and formant enhancement processing to obtain a target echo-reduced audio output.

[0011] Furthermore, the selectively suppressing residual noise of the frequency domain enhanced signal and the real-time environmental sound field change data to obtain a noise suppressed signal includes: Performing frequency band region division on the frequency domain enhanced signal to obtain a frequency band divided signal; Performing cross-correlation analysis on the frequency band division signal and the real-time environmental sound field change data to obtain residual noise distribution characteristics; Performing a band-selective masking analysis on the residual noise distribution characteristics to obtain a band noise marker signal; Performing Bayesian probability estimation on the frequency band noise marker signal to obtain a noise probability distribution diagram; Performing nonlinear suppression analysis on the frequency domain enhanced signal according to the noise probability distribution graph to obtain a preliminary noise suppression signal; Performing spectral subtraction balance on the preliminary noise suppression signal and the real-time environmental sound field change data to obtain a balanced suppression signal; Performing artifact threshold removal on the balance suppression signal to obtain an artifact removal coefficient; The artifact removal coefficients are subjected to wavelet reconstruction and phase preservation to obtain a noise suppression signal.

[0012] The present invention further provides an intelligent echo reduction audio processing system, which is applied to any one of the above-mentioned intelligent echo reduction audio processing methods, comprising: An acquisition module, which is used to acquire a sound field signal and a loudspeaker reference signal, and perform signal framing and short-time Fourier transform to obtain time-frequency spectrum audio data; An analysis module, the analysis module is used to perform linear echo cancellation processing on the time-frequency spectrum audio data and dynamically adjust the convergence speed to obtain a preliminary filtered signal; An association module, the association module is used to perform nonlinear residual echo elimination on the preliminary filtered signal and generate multiple reflection paths to obtain a nonlinear elimination signal; A processing module, the processing module is used to perform multi-modal residual suppression on the nonlinear elimination signal, perform spectrum processing through sub-band division and joint decision, and obtain an echo suppression signal; A control module is used to obtain real-time environmental sound field change data, acoustically enhance the echo suppression signal, and obtain a target echo-reduced audio output.

[0013] The present invention provides an intelligent echo reduction audio processing method and system, which has the following beneficial effects: By performing time-frequency analysis on the sound field signal and the speaker reference signal, combined with the dynamically adjusted linear echo cancellation technology, the basic echo components can be more accurately identified and filtered, thereby improving the initial effect of echo cancellation and providing a more reliable basis for subsequent processing. By introducing nonlinear residual echo cancellation and multi-reflection path generation mechanism, the complex echo path can be processed in a refined manner, which helps to solve the processing defects of traditional methods under nonlinear conditions and avoid sound quality distortion. The joint decision mechanism based on multimodal residual suppression and sub-band division can ensure that the system can operate efficiently under different acoustic environments and echo characteristics, reduce the impact of residual echo on communication quality, and improve the overall processing efficiency of the system. By acquiring the environmental sound field change data in real time and performing acoustic enhancement, a more intelligent adaptive processing strategy is formulated to achieve dynamic optimization of the entire system, thereby effectively improving audio clarity and reducing communication interference. And by comprehensively considering the complexity of the actual acoustic environment, the echo cancellation strategy can be flexibly adjusted according to the characteristics and demand changes of different application scenarios, making the system more adaptable to diverse audio interaction scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flow chart of an intelligent echo reduction audio processing method provided by the present invention; Figure 2 The present invention provides an intelligent echo reduction audio processing system structure diagram.

[0015] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0017] The present invention is further described below in conjunction with the accompanying drawings and specific implementation methods.

[0018] Reference Figure 1 As shown, the present invention provides 1. an intelligent echo reduction audio processing method, comprising: Step S1: Acquire a sound field signal and a loudspeaker reference signal, perform signal framing and short-time Fourier transform, and obtain time-frequency spectrum audio data; Step S2: performing linear echo cancellation processing on the time-frequency spectrum audio data and dynamically adjusting the convergence speed to obtain a preliminary filtered signal; Step S3: performing nonlinear residual echo elimination on the preliminary filtered signal and performing multi-reflection path generation to obtain a nonlinear elimination signal; Step S4: performing multi-modal residual suppression on the nonlinear elimination signal, performing spectrum processing through sub-band division and joint decision, and obtaining an echo suppression signal; Step S5: Acquire real-time environmental sound field change data, acoustically enhance the echo suppression signal, and obtain target echo-reduced audio output.

[0019] Based on the above steps, the detailed steps are as follows: Step S1: The microphone array captures the sound field signal containing the target speech and echo components, and the system obtains the reference signal played by the speaker. After preprocessing, the two signals are framed, usually with a frame length of 20-30ms, and there is an overlap of about 50% between frames to ensure the continuity of signal processing. The framed signal is multiplied by a window function such as a Hanning window or a Hamming window to reduce spectral leakage. Then, a fast Fourier transform (FFT) is performed on each frame of the windowed signal to map the time domain signal to the complex frequency domain to form a time-frequency spectrum representation. The time-frequency spectrum data contains amplitude spectrum and phase spectrum information. The amplitude spectrum reflects the energy distribution of each frequency component, and the phase spectrum retains the timing characteristics of the signal. This time-frequency representation enables the echo cancellation algorithm to process signal characteristics separately in different frequency bands, thereby improving the accuracy and efficiency of echo cancellation. Time-frequency domain processing also facilitates the subsequent analysis of acoustic environment characteristics, such as room impulse response and reverberation characteristics, and provides basic data for frequency domain analysis for subsequent steps.

[0020] Step S2: Based on the acquired time-frequency spectrum data, the linear echo cancellation process uses a frequency-domain-based adaptive algorithm, such as NLMS (normalized least mean square) or RLS (recursive least squares) algorithm, to estimate the linear transfer function from the speaker signal to the microphone. The adaptive filter continuously updates the filter coefficients to minimize the mean square error between the estimated echo and the actual echo. Dynamic adjustment of the convergence speed is the key optimization point of the process. The system intelligently adjusts the step size parameter by evaluating the signal-to-noise ratio, signal stability, and double-talk detection results. In a single-talk scenario (only far-end signal), a larger step size is used to accelerate convergence; in a double-talk scenario (far-end and near-end coexist), the step size is reduced to avoid the near-end speech being erroneously eliminated. The filter length is dynamically set according to the room reverberation time, usually covering an echo path of 50-300ms. The preliminary filtered signal output by the linear processing significantly reduces the main echo component, but may still contain nonlinear residual echo caused by factors such as speaker nonlinear distortion, microphone nonlinear response, and sampling clock drift. This signal is used as the input of the next stage of nonlinear processing to achieve progressive optimization of echo cancellation.

[0021] Step S3: The obtained preliminary filtered signal still contains residual echo introduced by speaker distortion, microphone nonlinearity and various nonlinear factors in the signal processing chain. Nonlinear residual echo cancellation uses deep learning-based models such as deep neural network (DNN), long short-term memory network (LSTM) or convolutional neural network (CNN) to construct nonlinear mapping functions. These models are trained with a large amount of echo data and can identify the spectral characteristics of residual echo. Multi-reflection path generation technology simulates the multipath propagation phenomenon of sound waves in complex acoustic environments. The system builds a reflection model based on the room geometry and material properties to generate multiple possible echo paths and their corresponding spectral characteristics. The nonlinear elimination process simultaneously considers the high-order harmonic characteristics of the speaker reference signal and captures the energy migration caused by nonlinear distortion through octave analysis. The residual echo estimator combines the state information of linear echo cancellation to accurately distinguish residual echo from near-end speech, especially to protect the near-end speech quality in the double-talk state. The nonlinear elimination signal obtained after processing greatly reduces various complex echo interferences, laying the foundation for subsequent multimodal residual suppression.

[0022] Step S4: Multimodal residual suppression technology integrates the features of three different characterization modes, namely time domain, frequency domain and perceptual domain, to form a more comprehensive description of residual echo. The subband division process decomposes the full-band signal into multiple critical bandwidth subbands, each of which is processed independently to adapt to the echo characteristics of different frequency regions. The joint decision mechanism integrates multiple features, including subband energy ratio, phase correlation, spectral smoothness and modulation spectrum, to accurately distinguish residual echo from near-end speech. Based on the decision results, the system applies spectral weighting technology to impose greater attenuation on time-frequency points with strong residual echo, while maintaining minimal intervention on areas dominated by near-end speech. Spectral processing adopts a Bayesian statistical framework, combined with a priori probability model to estimate the probability distribution of echo and speech, and achieves optimal suppression in the sense of minimum mean square error. In order to reduce music noise and speech distortion, the system introduces perceptual masking threshold control to ensure that spectrum modification does not exceed the range perceptible to the human ear. Spectral continuity and phase consistency are maintained during processing to avoid artificial traces. The output echo suppression signal has removed most of the echo components while maintaining the naturalness and clarity of the near-end speech.

[0023] Step S5: Real-time environmental sound field change data is obtained through a variety of sensors and signal analysis, including room acoustic parameter estimation, background noise characteristics analysis and user position tracking. The system monitors reverberation time changes, room mode resonance and acoustic impedance characteristics to capture environmental changes in a timely manner. Based on these data, the acoustic enhancement module applies dynamic equalization adjustment, adaptive gain control and spectrum shaping technology to compensate for speech distortion that may be introduced during echo cancellation. Special attention is paid to the restoration of the naturalness of speech listening, and the timbre characteristics of speech are maintained through harmonic reconstruction and formant enhancement. For low signal-to-noise ratio conditions, the system integrates noise suppression algorithms to further improve speech clarity. Terminal processing includes de-reverberation, dynamic range compression and psychoacoustic enhancement, so that the output audio can obtain the best listening experience on various playback devices. To reduce algorithm delay, the system adopts an inter-frame progressive processing strategy to achieve low-latency processing. The final echo-reduced audio output has extremely low echo residual, natural voice quality and stable audio characteristics, and can maintain excellent performance even when the acoustic environment changes significantly.

[0024] Step S6: The adaptive quality assessment module calculates a series of objective indicators, including echo return loss enhancement (ERLE), perceptual evaluation speech quality (PESQ) and speech intelligibility index (SII), to comprehensively evaluate the system's echo cancellation effect, speech quality preservation and overall listening experience. The system automatically fine-tunes the key parameters of each processing stage based on the evaluation results without user intervention. Long-term running data analysis identifies different usage scenario modes, such as conference room mode, home environment mode and mobile usage mode, and builds an optimized profile for each scenario. Parameter optimization adopts a hybrid strategy, combining rule-based expert systems and machine learning methods to strike a balance between achieving the best echo reduction effect and computing resource consumption. The system also has self-learning capabilities, and gradually improves the internal decision-making model by recording parameter combinations of successful processing cases. The robustness guarantee mechanism prevents the system from experiencing performance crashes due to extreme conditions, and sets parameter change upper limits and fallback mechanisms. This step enables the entire echo reduction system to form a closed-loop control, continuously evolve and adapt, and ensure that an excellent audio communication experience is always provided in the changing technical environment and the evolution of user needs.

[0025] The present invention provides an intelligent echo reduction audio processing method, which can more accurately identify and filter basic echo components by performing time-frequency analysis on sound field signals and loudspeaker reference signals, combined with dynamically adjusted linear echo cancellation technology, thereby improving the initial effect of echo cancellation and providing a more reliable basis for subsequent processing. By introducing nonlinear residual echo cancellation and multi-reflection path generation mechanism, the refined processing of complex echo paths is achieved, which helps to solve the processing defects of traditional methods under nonlinear conditions and avoid sound quality distortion. Based on the joint decision mechanism of multimodal residual suppression and sub-band division, it can ensure that the system can operate efficiently under different acoustic environments and echo characteristics, reduce the impact of residual echo on communication quality, and improve the overall processing efficiency of the system. By acquiring environmental sound field change data in real time and performing acoustic enhancement, a more intelligent adaptive processing strategy is formulated to achieve dynamic optimization of the entire system, thereby effectively improving audio clarity and reducing communication interference. And by comprehensively considering the complexity of the actual acoustic environment, the echo cancellation strategy can be flexibly adjusted according to the characteristics and demand changes of different application scenarios, so that the system is more adaptable to diverse audio interaction scenarios.

[0026] In one embodiment, a sound field signal and a loudspeaker reference signal are obtained, and signal framing and short-time Fourier transform are performed to obtain time-frequency spectrum audio data, including: At the beginning of signal processing, analog signals (sound field signals and speaker reference signals) need to be converted into digital signals. This process depends on two parameters: sampling rate and bit depth. The sampling rate specifies the number of times the signal is sampled per second, and the bit depth defines the accuracy of each sample value. Through these preset conditions, the sound field signal and the speaker reference signal are divided into equally spaced digital sampling points in time, and then converted into the original signal in the digital domain. After sampling, the signal is no longer a continuous analog waveform, but a time series composed of a series of discrete digital values, forming the basis of the digital signal.

[0027] The digitized signal needs to be further segmented to obtain more detailed spectrum information in time-frequency analysis. Time frame segmentation is performed by setting preset frame length and frame shift parameters. The frame length determines the duration of each signal segment, usually represented by a certain number of sampling points. The frame shift parameter sets the degree of overlap between consecutive frames and controls the sliding of signal frames on the time axis. The purpose of framing is to perform local analysis of the overall signal through small segments of the signal, so that the analysis can capture the details of the signal that change over time. These frame sequences become the basis for subsequent frequency domain processing.

[0028] Since directly performing Fourier transform on each frame of signal will introduce spectrum leakage problem, that is, the spectrum components of the signal will expand to adjacent frequency bands, affecting the analysis accuracy, the use of window function can reduce this effect. Hanning window function is a commonly used windowing technique, which optimizes spectrum leakage by applying lower weights to both ends of the signal frame and keeping higher weights in the middle of the signal frame. As a result of windowing the signal frame, the window function effectively reduces the spectrum aliasing caused by signal segmentation, making the frequency domain analysis more accurate.

[0029] Fast Fourier transform is a key step in converting time domain signals into frequency domain signals. Each windowed signal frame is processed by FFT, and the result is a complex frequency domain coefficient. The complex frequency domain coefficient contains the amplitude and phase information of the signal, where the real part represents the amplitude information of the signal and the imaginary part represents the phase information of the signal. FFT has high computational efficiency and can convert time domain signals into frequency domain signals in a relatively short time, which provides a basis for subsequent spectrum analysis.

[0030] The amplitude spectrum and phase spectrum of the signal can be extracted from the complex frequency domain coefficients. The amplitude spectrum reflects the energy of the signal at each frequency component and describes the distribution of the signal in the frequency dimension; the phase spectrum describes the phase information of the signal at different frequency components. The calculation of the amplitude spectrum and phase spectrum is based on the modulus and phase angle of the complex frequency domain coefficients, thereby obtaining the frequency component and phase information of the signal.

[0031] Band energy analysis refers to dividing the frequency domain signal into certain frequency bands and calculating the energy distribution in each frequency band. The band energy distribution matrix records the corresponding energy value in each frequency band. Band division is based on the spectral characteristics of the audio signal. Usually, the frequency domain is divided into several bands of equal width or set according to actual needs to analyze the energy distribution of different frequency components. The purpose of band energy analysis is to capture the energy information of different frequency bands, which is helpful for spectrum optimization and feature extraction.

[0032] The Mel frequency scale is a nonlinear frequency scale designed to mimic the auditory characteristics of the human ear. Through the mapping of the Mel frequency scale, the frequency band energy distribution matrix can be converted into a spectrum representation that conforms to the auditory characteristics of the human ear. The Mel frequency scale compresses the low-frequency part and stretches the high-frequency part to improve the frequency resolution in the frequency band that the human ear is more sensitive to. The Mel filter bank coefficients are obtained by mapping the frequency band energy distribution matrix through the Mel frequency scale, which reflects the spectral distribution of the signal under the Mel scale.

[0033] The Mel filter bank coefficients are combined with the complex frequency domain coefficients to construct a three-dimensional feature. A three-dimensional feature is a data set that combines time, frequency, and feature information, and is usually used for subsequent signal denoising, classification, or other processing tasks. By combining the Mel filter bank coefficients with the frequency domain coefficients, the characteristics of the signal can be fully described in the time-frequency domain to obtain a comprehensive, informative time-frequency audio data. This three-dimensional feature not only retains the temporal structure of the audio signal, but also captures the energy changes of the signal at different frequencies, thus providing strong support for further audio analysis.

[0034] This embodiment can effectively optimize spectrum leakage and improve the accuracy of frequency band energy analysis in signal processing through intelligent echo reduction audio processing of sound field signals and speaker reference signals, and realize spectrum representation that conforms to the auditory characteristics of the human ear through Mel frequency scale mapping, thereby improving the processing quality of audio signals. By adopting synchronous digital sampling and frame processing, it can better adapt to signal changes on different time scales, ensure that the signal is fully analyzed in the time-frequency domain, and provide accurate data support for subsequent noise reduction and signal enhancement. The use of Mel filter group coefficients can convert the spectrum characteristics of audio signals into a format that is more in line with the auditory characteristics of the human ear, effectively improving the natural perception of audio analysis. Through this processing method, the frequency characteristics of audio signals are more accurately characterized, reducing the errors caused by spectrum leakage and time-frequency resolution problems, making noise reduction processing more accurate and audio quality effectively improved. The application of this method in intelligent audio systems can improve the intelligence level of audio processing, especially in complex noise environments, and can significantly improve the clarity and restoration of audio signals.

[0035] In one embodiment, linear echo cancellation is performed on the time-frequency spectrum audio data, and the convergence speed is dynamically adjusted to obtain a preliminary filtered signal, including: The time-spectral audio data is divided in the frequency domain through a polyphase filter bank. The multi-subband decomposition adopts the critical bandwidth division criterion to divide the full frequency band into 32 non-uniformly distributed frequency sub-band signals. The bandwidth of each frequency sub-band signal follows the equivalent rectangular bandwidth (ERB) auditory model to ensure that the sub-band division matches the auditory characteristics of the human ear. The prototype filter of the polyphase filter bank adopts a cosine modulated filter with a 60 dB stopband attenuation, and a 15% overlap area is set between adjacent sub-bands to avoid frequency domain leakage. After processing, the original time-spectral audio data is converted into multiple frequency sub-band signals, each of which contains independent time-frequency energy distribution characteristics.

[0036] Multiple frequency sub-band signals are processed in parallel, and the frequency domain block least mean square (FDAF) algorithm is used to construct a linear echo path model. In each sub-band, the loudspeaker reference signal and the sound field signal are cross-correlated to calculate the filter impulse response of 256 points. The filter order is dynamically adjusted according to the sub-band center frequency. The low-frequency sub-band uses a 128-order filter to capture long-delay reflections, and the high-frequency sub-band uses a 64-order filter to reduce the computational complexity. The filter coefficient matrix is ​​updated once after processing 8 time-frequency frames through a block update method. A forgetting factor of 0.95 is introduced during the update process to balance tracking speed and stability. The linear echo path signal is generated by processing, which characterizes the transmission characteristics of the direct sound and early reflected sound from the loudspeaker to the microphone.

[0037] Based on the short-time coherence function of the sound field signal and the loudspeaker reference signal, the instantaneous misalignment measure is calculated. In the time-frequency domain, the amplitude coherence coefficient between the two signals is calculated for each frequency subband signal with a frame length of 20 ms. The amplitude coherence coefficient is calculated using the sliding window method with a window length of 5 frames. The convergence state parameter is generated by comparing the coherence change rate between the current frame and the historical frame. The convergence state parameter ranges from [0,1]. When the parameter value is lower than 0.3, it is judged as an insufficient convergence state, and when it is higher than 0.7, it is judged as an over-convergence state. This parameter reflects the tracking performance of the adaptive filter in real time and provides a quantitative basis for step size adjustment.

[0038] According to the nonlinear mapping relationship of the convergence state parameters, a piecewise proportional integral (PI) controller is used to generate the optimized step value. In the insufficient convergence area (parameter < 0.3), the integral gain is set to 0.8 and the proportional gain is set to 1.2 to speed up the convergence speed; in the stable convergence area (0.3 ≤ parameter ≤ 0.7), the integral gain is reduced to 0.2 and the proportional gain is set to 0.5 to maintain the convergence accuracy; in the over-convergence area (parameter > 0.7), the integral term is turned off and the proportional gain of 0.1 is enabled to prevent the filter from diverging. The saturation limiter mechanism is introduced during the adjustment process to ensure that the step value is always within the effective range of [0.001, 0.1]. The output frequency of the optimized step value is synchronized with the filter update period and is updated every 8 time-frequency frames.

[0039] The dynamic range of the optimized step value is compressed, and the hyperbolic tangent function is used to realize nonlinear mapping. The upper threshold of the step value is set to 0.05, and the lower threshold is set to 0.005. When the input step value exceeds the upper limit, the output value is fixed at 1.2 times the upper limit; when it is lower than the lower limit, the output value is fixed at 0.8 times the lower limit. Within the threshold range, the step value is scaled according to a logarithmic relationship, and the scaling formula is: stable step parameter = lower threshold + (upper threshold-lower threshold) × log10 (optimized step value / lower threshold); this process eliminates the sudden change interference of the step parameter and generates a stable step parameter with smooth transition characteristics.

[0040] The update process of the frequency domain block LMS algorithm is controlled by the stable step size parameter to filter the time-frequency spectrum audio data. In each frequency subband, the linear echo path signal is convolved with the speaker reference signal to generate a linear echo estimation signal. The convolution operation uses the overlap-preserve method, with a 50% frame overlap rate combined with the Hanning window function to avoid time domain aliasing. During the filtering process, the delay change of the echo path signal is monitored in real time. When the delay deviation exceeds 2ms, the filter coefficient reinitialization mechanism is triggered to prevent performance degradation caused by sudden changes in the environment.

[0041] After the linear echo estimation signal is subtracted from the original time-frequency spectrum audio data, the residual component is processed. Residual correlation compensation includes two parallel operations: amplitude compensation uses the subband energy matching method to calculate the energy ratio of the residual signal to the original signal in each frequency subband. When the energy ratio exceeds -15 dB, the residual signal amplitude is proportionally increased; phase compensation uses the least squares fitting in the complex domain to correct the phase offset of the residual signal. The compensated signal is reconstructed into a time domain waveform through a synthetic filter bank, and a preliminary filtered signal is output. This signal retains the target speech component while suppressing the linear echo component by at least 25dB.

[0042] This embodiment can effectively divide the full-band audio signal into 32 frequency sub-bands by introducing multi-sub-band decomposition technology into the intelligent echo reduction audio processing method. The bandwidth of each sub-band signal matches the auditory characteristics of the human ear, thereby improving the accuracy of echo cancellation and the naturalness of human hearing. The use of a multi-phase filter group ensures the accuracy of frequency domain division, reduces frequency domain leakage through a 15% sub-band overlap area, and improves frequency resolution. A linear echo path model is constructed by using a frequency domain block least mean square (FDAF) algorithm, and the filter order is dynamically adjusted in each sub-band, so that the low-frequency sub-band can capture long-delay reflections, while the high-frequency sub-band reduces the computational complexity, thereby achieving efficient and accurate echo path analysis. The block-based update of the filter coefficient matrix combined with the use of the forgetting factor ensures the stability and real-time performance of the algorithm.

[0043] In one embodiment, performing nonlinear residual echo cancellation on the preliminary filtered signal and generating multiple reflection paths to obtain a nonlinear cancellation signal includes: The signal that has been initially filtered is subjected to nonlinear residual echo elimination. The initially filtered signal is usually a signal that has been processed by a traditional echo suppression algorithm. Although it has been initially processed, it still contains certain residual echo components. In order to further eliminate these residual echoes, nonlinear residual echo elimination technology is used. This process analyzes the nonlinear characteristics of the signal, identifies and extracts those echo components, and continuously adjusts the parameters of the filter during the optimization process so that it can eliminate these echoes more accurately. In this way, the signal obtained is an echo signal that has been initially eliminated, which is usually called a nonlinear elimination signal.

[0044] After eliminating the echo signal, the coupling feature matrix is ​​constructed next. The coupling feature matrix extracts the key features of the signal by analyzing the relationship between the features of the preliminary filtered signal and the echo components. These features can help determine the performance of the echo components in the signal, and then guide the subsequent echo cancellation process. The method of constructing the coupling feature matrix is ​​usually to analyze the time domain and frequency domain features of the signal, combined with the delay and attenuation characteristics of the echo, to obtain the relevant feature information of the signal.

[0045] After obtaining the coupling characteristic matrix, the Volterra kernel expansion space is further constructed. The Volterra kernel expansion space is a mathematical tool for modeling signals through multi-order convolution operations. It can describe the nonlinear changes of signals during multiple reflections. In this space, the nonlinear characteristics of signals in complex propagation environments can be captured, and support can be provided for subsequent optimization filter design. By further processing the coupling characteristic matrix, a multi-dimensional expansion space is constructed. This space can describe the nonlinear changes of signals, thereby providing a basis for optimizing filter parameters.

[0046] After constructing the Volterra kernel expansion space, the next step is to update the preliminary filtered signal in real time to obtain more accurate filter parameters. This process adjusts the filter parameters in real time based on the features in the expansion space to eliminate more detailed echo components. At each moment, the filter is optimized according to the current signal state and the constructed expansion space, so that the filter response is more in line with actual needs and gradually approaches the optimal solution. The key to this step is to be able to adjust the filter in real time to ensure that the effect of echo cancellation is continuously optimized over time.

[0047] After the optimized filter processing, the echo component in the signal has been effectively suppressed, but in some complex environments, multipath propagation may exist. Multipath propagation refers to the fact that the sound signal reaches the receiving end through multiple different paths due to reflection, refraction, etc., resulting in the superposition of multiple reflected signals, further affecting the clarity of the voice. In order to effectively identify and suppress these reflected signals, it is necessary to analyze the multiple reflection paths.

[0048] By performing multi-reflection path analysis on the optimized signal, different reflection paths in the signal can be identified. The characteristics of these paths are mainly reflected in the signal delay, amplitude and attenuation mode. After identifying these paths, a "set of countermeasure reflection paths" can be accurately constructed. This set includes the reflection paths that have the greatest impact on the received signal. They have a high degree of interference with the signal and therefore need to be dealt with in detail.

[0049] After obtaining the set of countermeasure reflection paths, we need to perform time-frequency domain superposition on these paths and the preliminary filtered signal. Time-frequency domain superposition combines the characteristics of the signal in the time domain and frequency domain, and superimposes these reflection path signals so that these interference components can be fully suppressed in both frequency and time. Through time-frequency domain superposition, the impact of these multipath signals can be effectively weakened, thereby further improving the signal quality.

[0050] The resulting signal is called a multipath suppression signal, which has been superimposed in the time and frequency domains to remove most of the reflections and multipath interference. At this point, the remaining noise components in the signal will be mainly low-intensity and more consistent with the characteristics of natural speech.

[0051] In order to further improve the voice quality and suppress low-intensity noise that is difficult to eliminate by traditional methods, it is necessary to perform sub-band energy distribution analysis on the multipath suppression signal. The purpose of sub-band energy distribution analysis is to decompose the signal into several sub-band signals and identify which frequency bands in the signal have more prominent noise components by analyzing the energy distribution of each sub-band.

[0052] Based on the results of the sub-band energy distribution analysis, a dynamic time-frequency masking matrix is ​​generated. This matrix can dynamically adjust the degree of suppression of each sub-band according to the energy distribution at different times and frequencies. For those frequency bands with stronger noise, the masking matrix will give stronger suppression; for those frequency bands with weaker noise, it will give lighter suppression. This dynamic adjustment mechanism can remove noise to the maximum extent while ensuring speech clarity.

[0053] Based on the generated dynamic time-frequency masking matrix, the multipath suppression signal is multiplied point by point. The point-by-point product operation is to multiply each element in the time-frequency masking matrix with the corresponding frequency component of the multipath suppression signal, so as to accurately suppress the frequency of the signal. This operation can effectively suppress the residual noise and optimize the signal quality at all levels of the time domain and frequency domain. Through the point-by-point product operation, the final signal obtained is the nonlinear elimination signal. This signal has been processed at multiple levels, including echo cancellation, reflection path analysis, time-frequency domain suppression, etc., which has greatly improved the quality and clarity of the voice.

[0054] This embodiment effectively improves the quality of the audio signal by combining nonlinear residual echo elimination and multi-reflection path generation technology. By performing nonlinear residual echo elimination on the preliminary filtered signal, the echo component can be significantly removed, avoiding the problem of echo residue in the traditional echo elimination method, thereby improving the clarity of the speech. The construction of the nonlinear residual basis and the application of the Volterra kernel expansion space enable the method to accurately capture the complex nonlinear characteristics in the signal, further optimize the parameters of the filter, enable it to dynamically adapt to the echo changes in different environments, and provide a more efficient echo elimination effect. Through multi-reflection path analysis and the generation of adversarial reflection path sets, the interference of multi-path propagation on the audio signal can be effectively identified and eliminated. The time-frequency domain superposition technology further reduces multi-path interference and improves the overall quality of the signal. The generation of the dynamic time-frequency masking matrix accurately suppresses noise in different frequency bands, thereby ensuring the naturalness and clarity of the speech.

[0055] In one embodiment, a Volterra kernel expansion space is constructed according to the coupling characteristic matrix, and the preliminary filtering signal is updated in real time to obtain optimized filter parameters, including: The coupling feature matrix is ​​obtained by extracting features from the audio signal in the preprocessing stage. Every moment of the audio signal is analyzed to extract multi-dimensional feature data, which reflects the frequency, amplitude, phase and other information of the signal. The coupling feature matrix is ​​composed of these feature information, where each row represents the feature information within a time window, and each column represents the change of the feature in different time windows. The purpose of constructing the coupling feature matrix is ​​to comprehensively describe the complex characteristics of the signal and provide a complete data basis for subsequent processing.

[0056] By performing singular value decomposition (SVD) on the coupled feature matrix, the matrix is ​​decomposed into the product of three matrices: a left singular matrix, a diagonal matrix, and a right singular matrix. The goal of singular value decomposition is to find the principal components of the matrix, thereby achieving dimensionality reduction. In signal processing, dimensionality reduction can not only reduce computational complexity, but also help remove noise and unnecessary redundant information and retain the main effective features. After dimensionality reduction, a low-dimensional feature space is obtained, in which the main features of the signal are better represented.

[0057] In the reduced-dimensional feature space, separation components are constructed to obtain a separation vector group. The main task of this step is to further process the reduced-dimensional feature space and extract the most representative components of the audio signal. Through separation algorithms (such as independent component analysis ICA), the reduced-dimensional data set is decomposed into several independent components. Each component represents an independent feature in the signal, and the separation vector group is the collection of these components. The separation vector group will provide a basis for the subsequent kernel function construction.

[0058] The second-order expansion of the Volterra kernel function is further constructed using the separation vector group. The Volterra series is a polynomial model that contains the nonlinear relationships of various orders of the signal. When constructing the second-order expansion, the weighted summation method of the separation vector group is used to obtain a second-order kernel expansion matrix. This matrix represents the interaction of the second-order nonlinear features in the signal. In this way, the second-order nonlinear effects in the signal can be captured more accurately, providing a more effective mathematical model for echo cancellation.

[0059] Among them, the formula for constructing the second-order kernel expansion matrix includes: ; represents the second-order kernel expansion matrix, is the weighting coefficient, and are two components extracted from the separation vector group, Represents a tensor product operation. is the total number of separation vector groups, that is, the number of independent components extracted from the audio signal. i is the index variable used to traverse all separation vectors.

[0060] This formula is used to construct the second-order nonlinear feature matrix in the signal by weighted summation.

[0061] When continuing to construct the third-order kernel expansion tensor, the separation vector group and the second-order kernel expansion matrix are constructed by third-order cross terms. At this time, the separation vector group and the second-order kernel expansion matrix are combined by tensor product operation to generate a third-order kernel expansion tensor. Tensor product operation is an operation method between multidimensional arrays, which can combine features of different orders to further improve the expression ability of the model. Through the third-order kernel expansion tensor, more complex high-order nonlinear relationships in the signal can be captured, making the echo reduction process more accurate.

[0062] After obtaining the third-order kernel expansion tensor, an inner product operation is performed to construct a spatial mapping function. The function of the spatial mapping function is to map the audio signal to a new space in which the nonlinear characteristics of each order of the signal are fully expressed. Through the inner product operation, the relationship between the second-order kernel expansion matrix and the third-order kernel expansion tensor is converted into a spatial mapping function, which can effectively represent the high-order nonlinear characteristics of the signal. The mapped signal has a clearer structure in the new space, which helps to further optimize the filter parameters.

[0063] The preliminary filtered signal is nonlinearly projected using a spatial mapping function. The projection operation maps the signal from the original space to a new space, where the nonlinear characteristics of each order of the signal are enhanced. The result of this process is a projection coefficient vector that contains the main information of the signal in the new space. These coefficient vectors will serve as key inputs in subsequent optimization steps.

[0064] The projection coefficient vector will be optimized by least squares iteration. Least squares is a common optimization algorithm that continuously adjusts the filter parameters by minimizing the error between the predicted result and the actual result. Here, the goal of least squares iteration optimization is to find the optimal filter parameters to achieve the best echo cancellation effect. This process needs to be iterated continuously until it converges to the minimum error value. Finally, the optimized filter parameters will be used for signal echo reduction processing.

[0065] This embodiment introduces a coupling feature matrix into the audio processing method, which can comprehensively and accurately describe the frequency, amplitude, phase and other characteristics of the audio signal, provide a good data basis for subsequent processing, and thus improve the accuracy of processing. The coupling feature matrix is ​​reduced in dimension by singular value decomposition (SVD), which can effectively reduce the computational complexity, remove noise and redundant information, and ensure the retention and optimization of the main features. Independent component analysis (ICA) is used to separate the independent feature components of the audio signal, further enhancing the accuracy of signal processing. Second-order and third-order kernel expansion matrices and tensors are constructed, and the complex second-order and high-order nonlinear relationships in the signal are captured by tensor product operations, which can significantly improve the effect of echo reduction processing. The spatial mapping function constructed by the inner product operation maps the signal to a new feature space, so that the nonlinear characteristics are more fully expressed, thereby optimizing the adjustment of the filter parameters. In the final least squares iterative optimization process, the filter parameters are dynamically adjusted to ensure the optimization of the echo cancellation effect and improve the clarity of the audio signal. The entire method realizes efficient and accurate audio echo reduction processing through the organic combination of multiple steps, significantly improving the audio signal quality and user experience.

[0066] In one embodiment, multi-modal residual suppression is performed on the nonlinear elimination signal, and spectrum processing is performed through sub-band division and joint decision to obtain an echo suppression signal, including: The first step of signal processing is to perform sub-band decomposition. Sub-band decomposition is to divide the original audio signal into multiple sub-bands according to the frequency range to obtain a non-uniform sub-band representation. Through sub-band decomposition, the different frequency components of the signal are separated, making the processing of each frequency band more precise. Non-uniform sub-band representation means that the sub-band density in different frequency ranges is uneven, and more precise adjustments are made according to the characteristics of the frequency distribution, thereby improving the echo suppression effect.

[0067] The non-uniform subband representation will be integrated with multimodal feature extraction. The goal of this step is to extract the characteristic information of the signal from multiple dimensions, such as time domain, frequency domain, and phase information, to form a multimodal feature set. These features can reflect the multi-faceted information of the signal and help with subsequent echo suppression decisions. The multimodal feature set refers to a collection of multiple feature information, which is a different way of perceiving the signal state, which helps to achieve more accurate echo suppression processing.

[0068] According to the multimodal feature set, the cross entropy and mutual information calculations are performed to obtain the subband grouping structure. Cross entropy and mutual information are important mathematical tools to measure the similarity and dependency between signal features. Cross entropy reflects the information difference between different subbands, and mutual information is used to measure the degree of mutual dependence between different features. Through calculation, the correlation between each subband can be determined, thereby obtaining an optimized subband grouping structure. The subband grouping structure refers to the reasonable grouping of subbands after spectrum decomposition according to their similarity or information dependency.

[0069] After obtaining the subband grouping structure, the non-uniform subband representation is adaptively grouped according to the structure to form multiple subband sets. The adaptive grouping process automatically adjusts the grouping rules according to the specific characteristics of the signal, so that each subband set can better represent the actual situation of the signal. These adaptively grouped subband sets have a more effective suppression effect in the subsequent processing stage.

[0070] The decision rules are evaluated for multiple subband sets respectively to obtain multiple suppression decision values. At this stage, each subband set is evaluated according to the set decision rules to generate corresponding suppression decision values. The decision rules are based on the understanding of signal characteristics and processing goals, including the strength of the signal, noise characteristics, etc., to decide whether to further suppress the echo. The decision value is the result of processing the signal according to the preset rules, which determines whether and how to perform echo suppression.

[0071] All multiple suppression decision values ​​will be processed by joint decision to obtain the final target suppression strategy. The process of joint decision processing is to combine the suppression decisions of different sub-bands to form a unified suppression strategy. In this process, the suppression decision values ​​of each sub-band are involved, and the overall echo suppression effect is comprehensively evaluated to finally obtain the optimal suppression strategy. The target suppression strategy refers to the optimal echo suppression solution provided by the system for the entire audio signal, which can minimize the echo and improve the sound quality.

[0072] According to the target suppression strategy, the system performs differential processing on multiple groups of sub-band sets to obtain modal cooperative suppression signals. In this process, each sub-band set is adjusted in a targeted manner according to different suppression strategies. For example, for frequency bands with more obvious echoes, a stronger suppression strategy is adopted, while for frequency bands with less echoes, a lighter treatment is adopted. The purpose of this differentiated processing is to flexibly adjust according to the different degrees of echoes so as to maximize the elimination of echoes while maintaining sound quality.

[0073] The modal cooperative suppression signal is the result of differentiation processing. It integrates the characteristics of each sub-band and can effectively eliminate the echo component. Next, the modal cooperative suppression signal is subjected to variational decomposition phase compensation. Variational decomposition is a technique in signal processing that aims to decompose a signal into multiple components so that each component can be processed independently. In this process, the system compensates the phase of the signal to ensure that the phase of each frequency band of the signal remains coordinated, thereby avoiding sound quality loss due to phase imbalance.

[0074] The phase-coordinated signal is reconstructed by asymmetric singular value decomposition to obtain an echo-suppressed signal. Asymmetric singular value decomposition is an efficient signal reconstruction method that can remove noise and echo from complex signals while maintaining the original characteristics of the signal. During the reconstruction process, the system analyzes and adjusts the singular values ​​of the signal to ensure that the echo is effectively suppressed while maximizing the naturalness and clarity of the audio. The echo-suppressed signal is an audio signal obtained after a series of processing, in which the echo component is greatly reduced and the sound quality is significantly improved.

[0075] This embodiment uses multimodal residual suppression technology to perform fine sub-band decomposition and spectrum processing on the nonlinear elimination signal, effectively improving the accuracy and effect of echo suppression. The non-uniform representation and multimodal feature extraction of each sub-band can more comprehensively capture the multi-dimensional information of the signal, thereby realizing a more accurate echo suppression strategy. The introduction of cross entropy and mutual information calculation optimizes the sub-band grouping structure, making the adaptive grouping more intelligent, and further enhancing the system's ability to recognize and suppress echoes. Through the evaluation of the decision rules of multiple groups of sub-band sets, multiple suppression decision values ​​are obtained, and through joint decision processing, the optimality and consistency of the final suppression strategy are ensured. This processing method not only improves the accuracy of echo suppression, but also can perform differentiated processing according to the characteristics of different frequency bands, avoiding the loss of sound quality caused by excessive suppression. At the same time, the combination of variational decomposition phase compensation and asymmetric singular value decomposition reconstruction ensures the phase coordination of the signal and the restoration of the overall sound quality.

[0076] In one embodiment, real-time environmental sound field change data is acquired, and the echo suppression signal is acoustically enhanced to obtain a target echo-reduced audio output, including: Acquire environmental sensor data. Environmental sensor data includes audio signals collected by microphones, sound pressure level data, and directional data. Through multiple sensors, sound information at different locations and directions in the environment can be obtained. These data are preprocessed to remove interference signals and unnecessary noise, and then the short-time Fourier transform (STFT) is used to convert the time domain signal to the frequency domain to obtain spectrum information. According to the sensor position and the sound propagation model, the spectrum information is spatially decomposed to obtain the spectrum components in each direction, and sub-band analysis is performed to divide the spectrum into several sub-bands. The characteristics of each sub-band are analyzed to obtain detailed spectrum characteristics.

[0077] The acoustic parameter changes of the ambient sound field spectrum characteristics and the ambient sensor data are calculated to obtain real-time ambient sound field change data. Instantaneous parameters such as sound pressure level, frequency, phase, etc. are extracted from the ambient sensor data, and these instantaneous parameters are compared with the ambient sound field spectrum characteristics to analyze their change trends over time. Through the time series analysis method, the sound field changes in the future period are predicted, and the current and predicted sound field data are compared to calculate the real-time ambient sound field change data.

[0078] According to the real-time environmental sound field change data, the echo suppression signal is processed by multi-stage cascade filtering to obtain a preliminary enhanced signal. Using the real-time environmental sound field change data, the parameters of each filter are set, and the signal is preliminarily processed using the primary filter to filter out obvious echo components. Multiple filters are cascaded in sequence, and each filter optimizes the signal quality in turn, and finally outputs a preliminary enhanced signal, which has significantly reduced the echo.

[0079] The initial enhanced signal is subjected to phase correction and spectrum smoothing, and the gain factor is adjusted according to the real-time environmental sound field change data to obtain the frequency domain enhanced signal. The phase correction algorithm is used to adjust the phase of the initial enhanced signal to make the phase of the signal more consistent; the spectrum smoothing technology is applied to smooth the signal spectrum to reduce the sharp changes in the spectrum; according to the real-time environmental sound field change data, the gain factor is dynamically adjusted to optimize the signal strength and balance, and the frequency domain enhanced signal is output to make the signal spectrum smoother and the quality significantly improved.

[0080] The frequency domain enhanced signal and the real-time environmental sound field change data are combined to selectively suppress residual noise to obtain a noise suppression signal. The residual noise components in the frequency domain enhanced signal are identified, and the characteristics and change trends of the noise are analyzed using the real-time environmental sound field change data. The selective suppression algorithm is applied to select appropriate suppression strategies for different types of noise, and the noise suppression signal is output to significantly reduce the residual noise.

[0081] The noise suppression signal is subjected to auditory frequency band energy adjustment to obtain a time domain enhanced signal. The spectrum characteristics of the noise suppression signal are analyzed to determine the energy distribution of each frequency band. According to the auditory characteristics of the human ear, the energy of each frequency band is adjusted to make the signal more in line with auditory habits. The auditory masking effect is applied to optimize the frequency band energy distribution of the signal and output a time domain enhanced signal, which makes the signal sound more natural.

[0082] The time-domain enhanced signal is subjected to harmonic reconstruction and formant enhancement processing to obtain the target echo-reduced audio output. The fundamental frequency and harmonic components of the time-domain enhanced signal are analyzed, and the harmonic components of the signal are reconstructed using harmonic reconstruction technology to enhance the harmonic structure of the signal, identify the position and characteristics of the formant in the signal, and apply the formant enhancement algorithm to enhance the formant components of the signal, making the signal sound fuller and more layered, and finally outputting the target echo-reduced audio signal to achieve the best signal quality.

[0083] This embodiment can accurately capture various types of sound information in the environment through multi-level processing of environmental sensor data, and adjust the echo suppression signal in real time, effectively reducing the impact of environmental noise and echo, thereby improving the clarity and quality of the audio signal. Through spatial spectrum decomposition and sub-band analysis, the sound characteristics of different directions and frequency bands can be carefully analyzed, providing accurate spectrum feature support for subsequent audio processing, and effectively improving the system's adaptability to complex environmental sound fields. The use of multi-stage cascade filtering processing makes the suppression of echoes and noise more refined, further enhancing the quality of the signal. In addition, through processing steps such as phase correction, spectrum smoothing and gain adjustment, the smoothness and balance of the signal are significantly improved, which helps to improve the naturalness and auditory comfort of the sound output. Through the selective suppression of residual noise and the adjustment of auditory band energy, the interference of background noise is effectively reduced, so that the final output audio signal is more in line with the auditory characteristics of the human ear, and the delicacy and layering of the sound quality are improved.

[0084] In one embodiment, the frequency domain enhanced signal and the real-time ambient sound field change data are subjected to residual noise selective suppression to obtain a noise suppression signal, including: The frequency domain enhancement signal is extracted from the audio source signal and processed to enhance the main audio components in the signal, especially in a noisy environment, to ensure the clarity of important information (such as speech). After the frequency domain enhancement signal is divided into frequency band regions, the frequency components of the signal are divided into multiple independent frequency bands, each of which contains a signal in a specific frequency range. Through this division, the processing of different frequency components can be more refined, and the impact of noise on specific frequency bands can be reduced in a targeted manner. The goal of this step is to convert the signal into a frequency domain representation that is convenient for subsequent processing, allowing each frequency band to be analyzed and processed independently.

[0085] The signal after frequency band division is cross-correlated with the real-time environmental sound field change data. By calculating the correlation between the two signals (frequency band division signal and environmental sound field change data), cross-correlation analysis can reveal the impact of changes in the environmental sound field on the noise components in the signal. This analysis helps to identify which frequency bands may contain more complex noise components and the time variation pattern of the noise components. Through cross-correlation analysis, the distribution characteristics of the residual noise can be obtained, which will provide key information for subsequent noise suppression.

[0086] Based on the residual noise distribution characteristics obtained, a band-selective masking analysis is performed. This step determines which bands have more obvious noise by analyzing the intensity and distribution of the residual noise in different frequency bands, and performs masking on these bands. Masking refers to reducing or completely suppressing the noise component in a frequency band to improve the clarity of the signal. The result of the band-selective masking analysis is a band noise marker signal, which indicates the noise area that needs to be processed in the frequency band, which is helpful for subsequent noise reduction operations.

[0087] The band noise marker signal is used for Bayesian probability estimation, which combines the ambient sound field change data and the noise marker information to calculate the probability of each frequency band area being noise according to the Bayesian formula. This analysis process results in a noise probability distribution map, which shows the probability of noise in each frequency band. Based on this map, it can be further determined which frequency bands need to be focused on for noise reduction.

[0088] Based on the noise probability distribution map, the frequency domain enhanced signal is subjected to nonlinear suppression analysis. Nonlinear suppression is a common method in signal processing. It suppresses the more prominent noise components in the frequency band according to the noise probability of each frequency band in the noise probability distribution map. During the processing, the nonlinear suppression analysis can automatically adjust the suppression strength according to the noise intensity, and suppress the frequency bands with stronger noise more strongly, while retaining the frequency bands with weaker noise appropriately. Through this process, a preliminary noise suppression signal is obtained, that is, the noise has been partially suppressed, but a certain amount of audio information is still retained.

[0089] The preliminary noise suppression signal is further combined with the real-time environmental sound field change data to perform spectral subtraction and balancing. Spectral subtraction and balancing performs subtraction operations on the signal through spectrum analysis to reduce the abnormal components caused by noise in the signal. At the same time, it is combined with the real-time environmental sound field change data to adjust the balance parameters in the spectral subtraction process to make noise suppression more accurate. The signal after spectral subtraction and balancing is the balanced suppression signal, which has a stronger noise suppression effect than the preliminary noise suppression signal and retains more original audio content.

[0090] After obtaining the balanced suppression signal, the next step is to perform artifact threshold removal. Artifact removal is a processing method to remove unreal noise components in the signal. By setting the artifact removal threshold, those parts that appear to be noise but are actually signal errors are removed. The artifact removal coefficient obtained after artifact threshold removal represents the signal strength after artifact removal. At this stage, the invalid noise components in the signal have been effectively removed.

[0091] In this stage, a series of spectrum analyses (such as Fourier transform or wavelet transform) are performed on the signal to identify the sudden, discrete, extreme or irregularly changing components in the signal. These sudden changes are usually artifacts. By comparing with the waveform and spectrum characteristics of the real signal, the characteristics of the artifact can be extracted. The artifact threshold is usually set according to the statistical characteristics of the signal (such as mean, variance, peak, etc.). The purpose of setting the threshold is to distinguish artifacts from real noise or valid signals. This threshold can be a fixed value or dynamically adjusted according to the real-time changes of the signal. For example, if the spectral amplitude of a signal changes very drastically and is far away from its normal waveform, it may be judged as an artifact, and the part exceeding a certain amplitude threshold will be marked as an artifact.

[0092] Once the threshold is set, the artifacts are removed by the cleaning algorithm. In the frequency domain, each frequency component of the signal has an amplitude and phase information. When removing artifacts, the signal amplitude is first detected and compared with the set threshold to remove components with excessive amplitude or rapid changes (i.e. possible artifacts). For frequency components that exceed the threshold range, their amplitudes are usually set to zero, or their values ​​are reduced through smoothing to avoid excessive impact on the natural components of the signal.

[0093] After the artifacts are removed, the remaining signal is the artifact removal coefficient. At this point, the noise components of the signal (especially the artifact components) have been effectively removed. The artifact removal coefficients usually include the adjusted amplitude of each frequency component (the smoother amplitude after the artifacts are removed) and the phase information of the original signal. Through wavelet reconstruction or other signal reconstruction methods, these artifact-free amplitudes are combined with the original phase information to finally restore the time domain signal.

[0094] During the reconstruction of the signal after artifact removal, detailed adjustments are required to ensure that the artifact removal process does not introduce other distortions or unnatural changes. For example, the phase can be further smoothed to avoid unnatural phase jumps during reconstruction.

[0095] After the artifact removal coefficients are processed by wavelet reconstruction and phase preservation, the noise suppression signal is finally obtained. Wavelet reconstruction is to reconstruct the processed signal coefficients into a time domain signal through wavelet transform, while phase preservation ensures that the phase information of the signal will not be lost during the reconstruction process.

[0096] The combination of the frequency domain enhancement signal and the real-time environmental sound field change data of this embodiment can achieve efficient noise suppression and speech enhancement. By dividing the frequency band of the audio signal and cross-correlation analysis, the noise components in different frequency bands are accurately identified, so that the noise reduction process can be carried out in a targeted manner, avoiding the global suppression of the entire signal by traditional methods, thereby improving the audio quality and detail retention. By combining Bayesian probability estimation with nonlinear suppression technology, the intensity of noise suppression can be dynamically adjusted to ensure the balance between noise and signal, and avoid excessive suppression leading to audio distortion or information loss. Further spectral subtraction balance processing and artifact threshold removal technology effectively remove artifacts and invalid noise, ensuring the clarity and naturalness of the audio signal. Combined with wavelet reconstruction and phase retention processing, it is ensured that the signal will not lose important phase information during the reconstruction process, so that the noise suppression signal finally output not only retains the delicacy of the original audio, but also achieves efficient noise suppression effect.

[0097] Reference Figure 2 As shown, the present invention also provides an intelligent echo reduction audio processing system, which is applied to any one of the above-mentioned intelligent echo reduction audio processing methods, including: The acquisition module is used to obtain the sound field signal and the speaker reference signal, and perform signal framing and short-time Fourier transform to obtain time-frequency spectrum audio data; An analysis module, which is used to perform linear echo cancellation processing on the time-frequency spectrum audio data and dynamically adjust the convergence speed to obtain a preliminary filtered signal; The association module is used to perform nonlinear residual echo elimination on the preliminary filtered signal and generate multiple reflection paths to obtain a nonlinear elimination signal; A processing module, the processing module is used to perform multi-modal residual suppression on the nonlinear elimination signal, perform spectrum processing through sub-band division and joint decision, and obtain an echo suppression signal; The control module is used to obtain real-time environmental sound field change data, acoustically enhance the echo suppression signal, and obtain the target echo-reduced audio output.

[0098] The present invention provides an intelligent echo reduction audio processing system, which can more accurately identify and filter basic echo components by performing time-frequency analysis on sound field signals and loudspeaker reference signals, combined with dynamically adjusted linear echo cancellation technology, thereby improving the initial effect of echo cancellation and providing a more reliable basis for subsequent processing. By introducing nonlinear residual echo cancellation and multi-reflection path generation mechanism, the refined processing of complex echo paths is achieved, which helps to solve the processing defects of traditional methods under nonlinear conditions and avoid sound quality distortion. Based on the joint decision mechanism of multimodal residual suppression and sub-band division, it can ensure that the system can operate efficiently under different acoustic environments and echo characteristics, reduce the impact of residual echo on communication quality, and improve the overall processing efficiency of the system. By acquiring environmental sound field change data in real time and performing acoustic enhancement, a more intelligent adaptive processing strategy is formulated to achieve dynamic optimization of the entire system, thereby effectively improving audio clarity and reducing communication interference. And by comprehensively considering the complexity of the actual acoustic environment, the echo cancellation strategy can be flexibly adjusted according to the characteristics and demand changes of different application scenarios, so that the system is more adaptable to diverse audio interaction scenarios.

[0099] It should be noted that technicians in the relevant technical field can clearly understand that for the convenience and conciseness of description, the specific working process of the system and each module described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0100] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An intelligent echo reduction audio processing method, characterized in that: include: Acquire the sound field signal and the speaker reference signal, perform signal framing and short-time Fourier transform, and obtain time-frequency spectrum audio data; Performing linear echo cancellation processing on the time-frequency spectrum audio data, and dynamically adjusting the convergence speed to obtain a preliminary filtered signal; performing nonlinear residual echo elimination on the preliminary filtered signal and performing multi-reflection path generation to obtain a nonlinear elimination signal; Performing multimodal residual suppression on the nonlinear elimination signal, performing spectrum processing through sub-band division and joint decision, and obtaining an echo suppression signal; Real-time environmental sound field change data is acquired, and the echo suppression signal is acoustically enhanced to obtain a target echo-reduced audio output.

2. The intelligent echo reduction audio processing method according to claim 1, characterized in that: The step of acquiring the sound field signal and the loudspeaker reference signal, performing signal framing and short-time Fourier transform, and obtaining time-frequency spectrum audio data includes: Performing synchronous digital sampling on the sound field signal and the loudspeaker reference signal according to a preset sampling rate and bit depth to obtain a digital domain original signal; Divide the original digital domain signal into time frames according to preset frame length and frame shift parameters to obtain a frame sequence; Optimizing spectrum leakage of the frame sequence according to a preset Hanning window function to obtain a windowed signal frame; Performing a fast Fourier transform on the windowed signal frame to obtain complex frequency domain coefficients; Calculating the amplitude spectrum and the phase spectrum according to the complex frequency domain coefficients, and performing frequency band energy analysis to obtain a frequency band energy distribution matrix; Performing Mel frequency scale mapping on the frequency band energy distribution matrix to obtain Mel filter bank coefficients; The Mel filter bank coefficients and the complex frequency domain coefficients are used to construct three-dimensional features to obtain the time-frequency spectrum audio data.

3. The intelligent echo reduction audio processing method according to claim 1, characterized in that: The performing of linear echo cancellation processing on the time-frequency spectrum audio data and dynamically adjusting the convergence speed to obtain a preliminary filtered signal includes: Performing multi-sub-band decomposition on the time-frequency spectrum audio data to obtain multiple frequency sub-band signals; Performing echo path analysis on the multiple frequency sub-band signals to obtain a linear echo path signal; Performing instantaneous imbalance measurement calculation on the sound field signal and the loudspeaker reference signal to obtain a convergence state parameter; Adjust the step length parameter according to the convergence state parameter to obtain an optimized step length value; Performing upper and lower threshold setting mapping on the optimized step value to obtain a stable step parameter; Perform frequency domain filtering on the time-frequency spectrum audio data according to the stable step size parameter to obtain a linear echo estimation signal; Signal optimization is performed on the time-frequency spectrum audio data according to the linear echo estimation signal, and residual correlation compensation is performed to obtain the preliminary filtered signal.

4. The intelligent echo reduction audio processing method according to claim 1, characterized in that: The performing nonlinear residual echo elimination on the preliminary filtered signal and generating multiple reflection paths to obtain a nonlinear elimination signal includes: Performing nonlinear residual basis construction on the preliminary filtered signal to obtain a coupling characteristic matrix; Constructing a Volterra kernel expansion space according to the coupling characteristic matrix, and updating the preliminary filtering signal in real time to obtain optimized filter parameters; Perform multi-reflection path analysis based on the optimized filter parameters to obtain a set of adversarial reflection paths; Superimposing the countermeasure reflection path set and the preliminary filtered signal in time and frequency domains to obtain a multipath suppression signal; Performing sub-band energy distribution analysis on the multipath suppression signal, and generating a dynamic time-frequency masking matrix according to a preset threshold value; A point-by-point product operation is performed on the multipath suppression signal according to the dynamic time-frequency masking matrix to obtain the nonlinear elimination signal.

5. The intelligent echo reduction audio processing method according to claim 4, characterized in that: The step of constructing a Volterra kernel expansion space according to the coupling characteristic matrix and updating the preliminary filtering signal in real time to obtain optimized filter parameters includes: Performing singular value decomposition on the coupling feature matrix to obtain a reduced-dimensional feature space; Constructing separation components for the dimension-reduced feature space to obtain a separation vector group; Constructing the second-order expansion term of the Volterra kernel function according to the separation vector group, and performing weighted summation of the expansion term to obtain a second-order kernel expansion matrix; Constructing a third-order cross term based on the separation vector group and the second-order kernel expansion matrix, and performing a tensor product operation to obtain a third-order kernel expansion tensor; Performing an inner product operation on the second-order kernel expansion matrix and the third-order kernel expansion tensor to obtain a spatial mapping function; Performing nonlinear projection on the preliminary filtered signal according to the spatial mapping function to obtain a projection coefficient vector; The projection coefficient vector is optimized by least squares iteration to obtain the optimized filter parameters.

6. The intelligent echo reduction audio processing method according to claim 1, characterized in that: The performing multi-modal residual suppression on the nonlinear elimination signal, performing spectrum processing by sub-band division and joint decision, and obtaining an echo suppression signal, comprises: performing sub-band decomposition on the nonlinear elimination signal to obtain a non-uniform sub-band representation; Performing multimodal feature extraction and integration on the non-uniform sub-band representation to obtain a multimodal feature set; Calculating cross entropy and mutual information according to the multimodal feature set to obtain a subband grouping structure; Adaptively grouping the non-uniform subband representations according to the subband grouping structure to obtain multiple subband sets; Performing decision rule evaluation on the multiple groups of subband sets respectively to obtain multiple suppression decision values; Performing joint decision processing on the multiple suppression decision values ​​to obtain a target suppression strategy; Performing differential processing on the multiple groups of subband sets according to the target suppression strategy to obtain a modal cooperative suppression signal; performing variational decomposition phase compensation on the modal cooperative suppression signal to obtain a phase coordination signal; Asymmetric singular value decomposition and reconstruction are performed on the phase coordination signal to obtain the echo suppression signal.

7. The intelligent echo reduction audio processing method according to claim 1, characterized in that: The step of acquiring real-time environmental sound field change data, acoustically enhancing the echo suppression signal, and obtaining a target echo-reduced audio output includes: Acquire environmental sensor data, and perform spatial spectrum decomposition and sub-band analysis on the environmental sensor data according to the sound field signal to obtain spectrum characteristics of the environmental sound field; Calculating acoustic parameter changes on the ambient sound field spectrum characteristics and the ambient sensor data to obtain the real-time ambient sound field change data; Performing multi-stage cascade filtering on the echo suppression signal according to the real-time environmental sound field change data to obtain a preliminary enhanced signal; Performing phase correction and spectrum smoothing processing on the preliminary enhanced signal, adjusting the gain factor according to the real-time environmental sound field change data, and obtaining a frequency domain enhanced signal; Selectively suppressing residual noise on the frequency domain enhanced signal and the real-time ambient sound field change data to obtain a noise suppressed signal; Performing auditory frequency band energy adjustment on the noise suppression signal to obtain a time domain enhanced signal; The time domain enhanced signal is subjected to harmonic reconstruction and formant enhancement processing to obtain a target echo-reduced audio output.

8. The intelligent echo reduction audio processing method according to claim 7, characterized in that: The selectively suppressing residual noise of the frequency domain enhanced signal and the real-time environmental sound field change data to obtain a noise suppressed signal includes: Performing frequency band region division on the frequency domain enhanced signal to obtain a frequency band divided signal; Performing cross-correlation analysis on the frequency band division signal and the real-time environmental sound field change data to obtain residual noise distribution characteristics; Performing a band-selective masking analysis on the residual noise distribution characteristics to obtain a band noise marker signal; Performing Bayesian probability estimation on the frequency band noise marker signal to obtain a noise probability distribution diagram; Performing nonlinear suppression analysis on the frequency domain enhanced signal according to the noise probability distribution graph to obtain a preliminary noise suppression signal; Performing spectral subtraction balance on the preliminary noise suppression signal and the real-time environmental sound field change data to obtain a balanced suppression signal; Performing artifact threshold removal on the balance suppression signal to obtain an artifact removal coefficient; The artifact removal coefficients are subjected to wavelet reconstruction and phase preservation to obtain a noise suppression signal.

9. An intelligent echo reduction audio processing system, characterized in that: The intelligent echo reduction audio processing method applied to any one of claims 1 to 8 above comprises: An acquisition module, which is used to acquire a sound field signal and a loudspeaker reference signal, and perform signal framing and short-time Fourier transform to obtain time-frequency spectrum audio data; An analysis module, the analysis module is used to perform linear echo cancellation processing on the time-frequency spectrum audio data and dynamically adjust the convergence speed to obtain a preliminary filtered signal; An association module, the association module is used to perform nonlinear residual echo elimination on the preliminary filtered signal and generate multiple reflection paths to obtain a nonlinear elimination signal; A processing module, the processing module is used to perform multi-modal residual suppression on the nonlinear elimination signal, perform spectrum processing through sub-band division and joint decision, and obtain an echo suppression signal; A control module is used to obtain real-time environmental sound field change data, acoustically enhance the echo suppression signal, and obtain a target echo-reduced audio output.

Images