A small hardware-based deep learning echo suppression and noise reduction processing method

Abstract

The application discloses a small hardware acoustic cavity echo suppression and noise reduction processing method based on deep learning, and belongs to the technical field of speech signal processing and artificial intelligence. The application solves the problems of strong nonlinear echo in a small hardware acoustic cavity, failure of a traditional algorithm, and difficulty of a deep learning model in real-time operation on a limited hardware. The scheme is as follows: loudspeaker feedback current and voltage are collected by using an intelligent power amplifier; a nonlinear reference signal containing thermal compression and displacement effect is reconstructed through a physical acoustic cavity nonlinear simulator; after initial filtering through sub-band adaptive filtering, hardware state and acoustic characteristics are fused; a quantized light-weight deep neural network is used to predict a complex ratio mask; and finally, pure speech is reconstructed and output. The application significantly improves the nonlinear echo suppression capability, the model parameter quantity is less than 150K, and the model can be operated in real time on a low-power chip. In a very low signal-to-noise ratio noisy environment, the speech recognition word error rate is reduced by more than 30%.

Description

Technical Field

[0001] This invention belongs to the field of speech signal processing and artificial intelligence technology, specifically relating to a method for echo suppression and noise reduction of a small hardware acoustic cavity based on deep learning. Background Technology

[0002] With the widespread adoption of the Internet of Things (IoT), smart wearable devices (such as smart Bluetooth headsets and smart glasses), and miniature smart home terminals, voice-based human-machine interfaces have become a core control method. In these miniaturized and integrated hardware devices, due to extreme physical size limitations, the physical distance between the microphone and the speaker (loudspeaker) is typically only a few millimeters to a few centimeters, and they share the same miniature acoustic cavity. In this highly integrated miniature hardware acoustic cavity structure, the sound played by the speaker is directly fed back to the microphone at extremely high intensity through solid conduction and air coupling, creating a strong acoustic echo. Simultaneously, these devices are typically used in complex outdoor, in-vehicle, or industrial noisy environments, where environmental noise can severely mask the target voice signal.

[0003] However, in existing technologies, traditional acoustic echo cancellation (AEC) algorithms (such as adaptive filters based on minimum mean square error (LMS) or normalized minimum mean square error (NLMS) primarily rely on linear system assumptions. In small hardware acoustic cavities, miniature speakers are prone to entering nonlinear operating regions at high volumes, resulting in severe nonlinear distortion and causing traditional linear adaptive filters to fail. Furthermore, existing deep learning noise reduction and echo cancellation algorithms typically have a large number of model parameters and high computational complexity, making them difficult to run in real-time on small hardware chips (such as MCUs and low-power DSPs) with limited power consumption and computing power. Simply pruning or quantizing the deep learning model directly can lead to a sharp drop in speech recognition (ASR) accuracy in noisy environments with extremely low signal-to-noise ratios (SNR) and strong echo coupling, failing to meet practical commercial needs.

[0004] Therefore, designing a deep learning echo suppression and noise reduction method that can effectively suppress nonlinear strong echoes and high-intensity additive noise, while having extremely low computational complexity, being deployable on limited hardware platforms, and significantly improving speech recognition rates in noisy environments, based on the physical characteristics of small hardware acoustic cavities, is a highly necessary and urgent technical problem that needs to be solved in the current field of intelligent voice hardware. Summary of the Invention

[0005] To address the technical problems of severe nonlinear echo coupling caused by the limited space of small hardware acoustic cavities in existing technologies, the inability of traditional algorithms to effectively eliminate nonlinear echoes, the high complexity of existing deep learning models making them difficult to deploy on constrained hardware, and the low speech recognition rate in noisy environments, this invention provides a deep learning-based method for echo suppression and noise reduction in small hardware acoustic cavities. This method, through the collaborative design of hardware circuit state awareness and lightweight deep neural networks, achieves complete elimination of nonlinear echoes and deep noise reduction with extremely low hardware resource consumption, significantly improving speech recognition accuracy in noisy environments.

[0006] The objective of this invention can be achieved through the following technical solutions: A method for echo suppression and noise reduction in a small hardware acoustic cavity based on deep learning includes the following steps: Step 1: Hardware-level physical state perception and multi-channel signal synchronous acquisition: The Smart PA in the hardware circuit is used to acquire the feedback current signal and feedback voltage signal at both ends of the speaker in real time, as hardware characteristic signals characterizing the nonlinear physical state of the speaker; at the same time, the mixed input signal is acquired through the microphone, and the original reference signal of the speaker is obtained through the audio codec; the feedback current signal, feedback voltage signal, mixed input signal and original reference signal are clocked and aligned at the hardware level. Step 2: Nonlinear pre-modeling and linear echo initial filtering based on physical acoustic cavity state: The synchronized feedback current signal, feedback voltage signal, and original reference signal are input into a lightweight physical acoustic cavity nonlinear simulator built in hardware DSP to reconstruct a nonlinear reference signal containing speaker thermal compression and displacement constraint effects; using this nonlinear reference signal, a sub-band adaptive filtering algorithm is used to initially eliminate linear and low-order nonlinear echoes in the mixed input signal collected by the microphone, and the initial filtered residual signal is output. Step 3: Time-frequency domain feature extraction and multimodal feature fusion: Perform short-time Fourier transform on the initial filtered residual signal, the original reference signal, and the nonlinear reference signal to obtain their respective time-frequency domain complex spectra; extract the logarithmic power spectrum of the initial filtered residual signal, the coherence features of the original reference signal and the initial filtered residual signal, and perform multimodal feature splicing with the time-frequency representation of the hardware feature signal to construct a multidimensional fused feature tensor; Step 4: Hardware-aware lightweight deep neural network processing: The multidimensional fusion feature tensor is input into a pre-trained and quantized lightweight deep neural network. The lightweight deep neural network adopts an architecture that combines a deep separable temporal convolutional network with a self-attention mechanism. The network output predicts complex ratio masking. Step 5: Signal Reconstruction and Post-processing: Multiply the predicted complex ratio mask by the complex spectrum of the initial filtered residual signal to obtain the estimated complex spectrum of the near-end speech; perform an inverse short-time Fourier transform on it to reconstruct the time-domain near-end speech signal; perform amplitude smoothing on the reconstructed time-domain near-end speech signal using a dynamic range control algorithm to output a clean near-end speech signal.

[0007] Furthermore, in step 1, the specific implementation of the clock synchronization alignment is as follows: a single master clock source is used to simultaneously drive the analog-to-digital converter (ADC) of the Smart PA and the analog-to-digital converter (ADC) of the audio codec, and the feedback current signal, feedback voltage signal, mixed input signal and original reference signal are written into the shared static random access memory (SRAM) through the dual buffer mechanism of hardware direct memory access (DMA), so as to control the hardware synchronization error to within 1 microsecond.

[0008] Furthermore, in step 2, the process of reconstructing the nonlinear reference signal using the physical acoustic cavity nonlinear simulator includes: Based on the feedback current signal and feedback voltage signal, calculate the real-time temperature of the speaker voice coil and the DC resistance of the voice coil as a function of temperature. A real-time physical displacement state observer for a loudspeaker diaphragm is constructed using electromagnetic and mechanical equations. The state observer is then solved in real time using the fourth-order Runge-Kutta method to obtain the real-time physical displacement of the loudspeaker diaphragm. The state observer considers nonlinear spring stiffness and nonlinear magnetic force factor that vary with displacement.

[0009] Furthermore, in step 2, the nonlinear reference signal The reconstruction formula is: ; in, The original reference signal, For real-time temperature, The ambient reference temperature The thermal decay coefficient, For real-time physical displacement, These are higher-order nonlinear distortion coefficients. It is a symbolic function.

[0010] Furthermore, in step 2, the specific process of the sub-band adaptive filtering algorithm is as follows: the mixed input signal and the nonlinear reference signal are decomposed into multiple sub-band signals through a uniform discrete Fourier transform analysis filter bank; the linear echo is estimated using a normalized minimum mean square error (NLMS) adaptive filter on each sub-band; the adaptive step size is dynamically adjusted by a coherence-based two-end call factor; and finally, the time-domain initial filter residual signal is reconstructed through a synthetic filter bank.

[0011] Furthermore, in step 3, the window function of the short-time Fourier transform is a Hanning window, with a window length of 32 milliseconds, a frame shift of 8 milliseconds, and an overlap rate of 75%. The coherence characteristics are obtained by estimating the cross power spectrum of the original reference signal and the initial filtered residual signal, the self-power spectrum of the original reference signal, and the self-power spectrum of the initial filtered residual signal using a first-order infinite impulse response (IIR) smoothing filter, and by calculating the ratio of the square of the cross power spectrum to the product of the two self-power spectra.

[0012] Furthermore, in step 3, the construction process of the multidimensional fusion feature tensor includes: The real-time temperature and the voice coil DC resistance are normalized by maximum and minimum to obtain normalized temperature and normalized resistance. The normalized temperature and normalized resistance are copied along the frequency axis to form a hardware state feature vector consistent with the number of frequency points. In the channel dimension, the logarithmic power spectrum of the initial filtered residual signal, the coherence feature, the hardware state feature vector, and the real and imaginary parts of the complex spectrum of the initial filtered residual signal are concatenated to construct a multidimensional fusion feature tensor with 6 channels.

[0013] Furthermore, in step 4, the architecture of the lightweight deep neural network sequentially includes: A two-dimensional depthwise separable convolutional layer is used to extract local time-frequency features; Three cascaded temporal convolutional blocks, each employing a one-dimensional causal dilated depthwise separable convolution with an exponentially increasing dilation factor per layer, are used to capture long temporal context. A lightweight self-attention layer for global correlation modeling of frequency domain features; A linear output layer is used to predict the real and imaginary parts of the complex ratio mask, and the output range is limited by a hyperbolic tangent activation function.

[0014] Furthermore, the lightweight deep neural network is symmetrically quantized to 8-bit integer INT8 before being deployed on the hardware. Its forward propagation multiply-accumulate operations are performed entirely at INT8 precision and are optimized at the operator level using the CMSIS-NN library.

[0015] Furthermore, in step 5, the specific process of the dynamic range control algorithm is as follows: The transient envelope of the near-end speech signal in the time domain is calculated using a single-pole IIR filter; Calculate the static gain curve based on the transient envelope, the set amplitude limiting threshold, and the compression ratio; The static gain curve is smoothed using the start-up time constant and the release time constant to obtain the dynamic gain; The dynamic gain is multiplied by the time-domain near-end speech signal to output a clean near-end speech signal.

[0016] Compared with the prior art, the present invention has the following advantages: (1) Deep collaboration between hardware and algorithm to break through the bottleneck of nonlinear modeling: This invention creatively introduces current and voltage feedback signals from hardware circuit (Smart PA). Since current and voltage directly reflect the nonlinear physical states such as the physical displacement of the micro loudspeaker diaphragm and the coil temperature, the algorithm does not need to rely solely on acoustic signals to blindly estimate complex nonlinear distortion, thereby improving the nonlinear echo suppression capability by more than 15dB.

[0017] (2) Extremely low computational complexity and memory usage, enabling micro-hardware deployment: The physical acoustic cavity nonlinear simulator offloads most of the nonlinear modeling work, so that the subsequent deep learning network does not need to use a large number of parameters to fit the nonlinearity. The number of parameters of the deep learning model can be compressed to less than 150K, and the RAM usage is less than 50KB, which can be fully implemented in real time on low-power MCU or DSP chips.

[0018] (3) Significant improvement in speech recognition rate in noisy environments: Since the present invention uses complex ratio masking (CRM) to repair both phase and amplitude, and multimodal feature fusion provides strong noise discrimination capability, the word error rate (WER) of downstream automatic speech recognition (ASR) is reduced by more than 30% in extremely noisy environments with signal-to-noise ratio (SNR) as low as -10dB and accompanied by strong echo, which greatly expands the application scenarios of small smart hardware. Detailed Implementation

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] This invention provides a method for echo suppression and noise reduction in a small hardware acoustic cavity based on deep learning. This method achieves complete elimination of nonlinear echoes and deep noise reduction with extremely low hardware resource consumption through the collaborative design of hardware circuit state awareness and a lightweight deep neural network. The following provides a detailed, complete, and rigorous disclosure of the specific functional implementation process of each module of this invention.

[0021] Example 1 Implementation of hardware-level physical state sensing and simultaneous acquisition of multiple signals: In small hardware acoustic cavities (such as smart Bluetooth headsets, smart glasses, and miniature smart speakers), due to extremely limited space, the distance between the speaker and microphone is very close (usually between 5mm and 30mm), and they share the same miniature physical acoustic cavity. When the speaker operates under large signal drive, its diaphragm displacement is very likely to approach the physical limit, and the voice coil temperature will rise sharply due to the Joule heating effect, resulting in severe electromagnetic, mechanical, and acoustic nonlinear distortion.

[0022] To accurately capture these nonlinear physical states, this embodiment constructs a hardware-level physical state sensing system, and its hardware connection and signal acquisition process are as follows: 1.1 Hardware Connection Architecture: The hardware system in this embodiment mainly includes: a miniature speaker, a smart power amplifier (Smart PA), a bidirectional current / voltage sampling circuit, a miniature microphone, an audio codec, and a main control chip (this embodiment uses a low-power double-precision floating-point DSP based on the ARM Cortex-M7 core with a main frequency of 400MHz).

[0023] The Smart PA connects the main control chip and the miniature speaker to drive the speaker to play audio. The Smart PA integrates a high-precision current sensing resistor (10mΩ, ±0.5% accuracy) and a differential amplifier.

[0024] A bidirectional current / voltage sampling circuit is deployed at the output of the Smart PA. The differential amplifier acquires the feedback current flowing through the speaker voice coil in real time and converts it into a digital feedback current signal via an analog-to-digital converter (ADC). Simultaneously, the feedback voltage across the speaker is acquired in real time through a high-impedance voltage divider network and converted into a digital feedback voltage signal via an ADC. .

[0025] A miniature microphone is deployed near the acoustic cavity to collect mixed input signals. . Includes near-end target speech signal Environmental additive noise signal And strong echo signals generated by the combination of solid conduction and air coupling within the acoustic cavity. ,Right now: .

[0026] An audio codec is used to convert analog signals picked up by a microphone into mixed digital input signals. At the same time, the digital raw reference signal output by the main control chip The audio source signal to be played is transmitted to the Smart PA.

[0027] 1.2 Hardware-level clock synchronization and alignment mechanism: In adaptive filtering and deep learning processing, clock drift (Sampling Rate Offset, SRO) and latency jitter between multiple signals can directly cause adaptive filters to diverge and render deep learning feature alignment ineffective. Therefore, this embodiment designs a hardware-level clock synchronization and alignment mechanism: (1) Single master clock source drive: The system uses a high-precision active crystal oscillator (nominal frequency of 24.576MHz, temperature drift less than ±10ppm) as the master clock source (MCLK). The MCLK is connected to the ADC clock input pin of the Smart PA and the ADC clock input pin of the audio codec, eliminating the frequency deviation between the two sampling chips from the physical source.

[0028] (2) DMA dual-buffer (Ping-Pong Buffer) mechanism: Two sets of direct memory access (DMA) channels are configured in the main control DSP. Channel A is bound to the I2S / TDM interface of the Smart PA for receiving... and Channel B is bound to the I2S interface of the audio codec for receiving audio signals. And send .

[0029] (3) Hardware interrupt alignment: The two DMA channels share the same hardware timer trigger source. When the timer count reaches the set frame length sampling point number (in this embodiment, the sampling rate...) When the frequency is 16 kHz, the frame length is 256 sampling points, and the corresponding time is 16 ms, a DMA transfer completion interrupt is triggered simultaneously. In the interrupt service routine (ISR), the data pointers of the two sets of buffers are aligned and written into the shared static random access memory (SRAM).

[0030] Through the above-described physical clock common-source and DMA interrupt alignment design, this embodiment will feed back the current signal. Feedback voltage signal Mixed input signals and the original reference signal Hardware synchronization error between Strictly controlled at: This enables absolute time synchronization and alignment of multiple signals at the microsecond level.

[0031] Example 2 Implementation of nonlinear pre-modeling and linear echo pre-filtering based on physical acoustic cavity conditions: This module runs in the main control DSP. Its core idea is to use the acquired current and voltage signals that directly reflect the physical state of the loudspeaker to reconstruct the nonlinear distortion behavior of the loudspeaker in the digital domain, generate a nonlinear reference signal, and then eliminate most of the linear echo and low-order nonlinear echo through a sub-band adaptive filter, thereby reducing the modeling burden of the subsequent deep learning network.

[0032] 2.1 Nonlinear physical state modeling of loudspeakers: When a miniature loudspeaker is working, the voice coil generates an electromagnetic force through an electric current, which drives the diaphragm to move. The resistance of the voice coil increases with temperature (thermal compression effect), while excessive diaphragm displacement causes nonlinear decay of the spring stiffness and magnetic force factor (displacement limiting effect).

[0033] In this embodiment, the real-time temperature of the speaker voice coil is first calculated. Based on the thermodynamic law of conservation of energy, the first-order differential equation of state for the voice coil temperature is constructed as follows: in: (1) The ambient reference temperature is set to 25℃.

[0034] (2) The thermal resistance from the voice coil to the loudspeaker magnet and the environment is 35 K / W.

[0035] (3) The thermal capacitance of the voice coil is 0.02 J / K.

[0036] (4) The instantaneous electrical power of the input voice coil is calculated using the following formula: ,in, The real-time DC resistance of the voice coil varies with temperature.

[0037] Since resistance has a linear relationship with temperature, real-time DC resistance The calculation formula is: ,in: (1) The nominal DC resistance at room temperature (20°C) is 8.0 Ω in this embodiment.

[0038] (2) The temperature coefficient of the voice coil copper conductor is 0.00393 K. -1 .

[0039] The above continuous-time differential equation is discretized using a first-order forward Euler method in the DSP to obtain the iterative calculation formula for real-time temperature (assuming a sampling interval). ): , Next, the real-time physical displacement of the speaker diaphragm is constructed. State observer. According to the mechanical equilibrium equations (Newton's second law) and the law of electromagnetic induction, the motion of the diaphragm satisfies the following second-order nonlinear differential equation: , in: (1) The equivalent vibrating mass of the diaphragm and voice coil is taken as 0.15 g (i.e., 1.5 × 10⁻⁶ g). -4 kg).

[0040] (2) The mechanical equivalent damping is taken as 0.5 N·s / m.

[0041] (3) For the nonlinear spring stiffness that varies with displacement, a third-order Taylor expansion approximation is used: ,in, =1200N / m is the static stiffness. =-150 N / m 2 It is a first-order nonlinear stiffness coefficient. =-500 N / m 3 It is a second-order nonlinear stiffness coefficient.

[0042] (4) The nonlinear magnetic force factor (the product of magnetic flux density and voice coil wire length) varies with displacement: ,in, =1.5 T·m is the static magnetic force factor. =-0.2 T is the first-order nonlinear magnetic coefficient. =-0.8 T / m is the second-order nonlinear magnetic coefficient.

[0043] To solve this second-order nonlinear differential equation with high accuracy and real-time performance in a DSP, this embodiment transforms it into a first-order state-space equation system. Let the state variables... (i.e., the diaphragm velocity), then we have: ; Discretized numerical solutions are obtained using the Runge-Kutta 4th Order (RK4) method. Let the current sampling time be n, and the iteration step size be... .

[0044] Define state vector The state transition function is ,in: ; ; The specific iterative calculation steps for RK4 are as follows: ; ; ; ; ; Through the RK4 iterations described above, the state vector is updated in each sampling period, thereby obtaining high-precision physical displacement of the diaphragm in real time. .

[0045] 2.2 Nonlinear reference signal reconstruction: After obtaining the real-time temperature of the voice coil and the real-time physical displacement of the diaphragm Subsequently, this embodiment reconstructs a nonlinear reference signal in the digital domain that includes thermal compression effects and higher-order displacement saturation nonlinearities. Its reconstruction formula is: ; in: (1) The raw reference signal obtained by the audio codec.

[0046] (2) The preset speaker thermal attenuation coefficient is used to characterize the decrease in sound pressure level caused by the weakening of magnetic force and the increase of resistance due to temperature rise. In this embodiment, the value is taken as 0.002K. -1 .

[0047] (3) is a high-order nonlinear distortion coefficient used to characterize the odd harmonic distortion generated when the diaphragm displacement approaches the physical limit. In this embodiment, the value is taken as 0.05.

[0048] (4) For symbolic functions, it is defined as: .

[0049] 2.3 Implementation of Subband Adaptive Filtering (SAF) Algorithm: To significantly reduce the computational complexity of the DSP while ensuring filtering performance, this embodiment adopts a sub-band adaptive filtering architecture.

[0050] 1. Analysis Filter Bank: A uniform discrete Fourier transform (DFT) modulation analysis filter bank with M=64 channels is used. The time-domain microphone mixed input signal is... and reconstructed nonlinear reference signal The sub-band microphone signal was obtained by analyzing the filter bank, performing bandpass filtering, and decimation (downsampling) by a factor of D=32. and sub-band nonlinear reference signal ,in For sub-band index, This is the index of the downsampled time frame.

[0051] 2. Subband Adaptive Filtering and Update: In each subband Above, deploy a complex-domain normalized minimum mean square error (NLMS) adaptive filter. Let the coefficient vector of the adaptive filter be: The filter order (length) is set to L=8.

[0052] Subband residual signal The calculation formula is: ,in, The state vector of the sub-band nonlinear reference signal. This indicates the conjugate transpose.

[0053] The coefficient update formula for the adaptive filter is: , in: (1) To prevent regularization constants with denominators of zero.

[0054] (2) Subband residual signal The conjugate of complex numbers.

[0055] (3) To achieve a dynamically adaptive step size, it employs a coherence-based two-end call detection (DTD) algorithm for real-time control, preventing the adaptive filter from diverging when the near-end speaker is speaking.

[0056] Dynamic adaptive step size The control formula is: ,in, = 0.2 is the maximum step size set. The coherence-based two-way communication factor is calculated as follows: ,in: =0.9 is the smoothing factor. =10 -5 . This represents the cross-power spectrum of the subband microphone signal and the subband residual signal. and The power spectra of each signal are calculated online using a first-order IIR filter. When there is speech at the near end, the residual signal... With microphone signal The coherence is extremely high. Approaching 1, step size Automatically decrease to near 0, then stop updating the filter coefficients; conversely, during the single-ended echo stage, Approaching 0, step size Increasing the value accelerates the convergence speed.

[0057] 3. Synthesis Filter Bank: This bank synthesizes the processed residual signals from each subband. The input is fed into the corresponding synthesis filter bank, where interpolation and bandpass filtering are performed to synthesize and reconstruct the time-domain initial-filter residual signal. .at this time, The system has eliminated most of the linear echoes and low-order nonlinear echoes, mainly including residual high-order nonlinear echoes, environmental additive noise, and near-end target speech.

[0058] Example 3 Practical implementation of time-frequency domain feature extraction and multimodal feature fusion: This module converts time-domain signals to the time-frequency domain and deeply integrates acoustic features with physical state features obtained from hardware perception, providing multimodal input representations for subsequent neural networks.

[0059] 3.1 Short-Time Fourier Transform (STFT): Time-domain initial filtering residual signal Original reference signal and nonlinear reference signal Perform a short-time Fourier transform (STFT).

[0060] (1) Window function: The Hanning window is used, and its time-domain expression is: , ; Among them, window length =512 sampling points (corresponding to 32 ms at a sampling rate of 16kHz).

[0061] (2) Frame shift: Set the frame shift =128 sampling points (corresponding to 8ms, with an overlap rate of 75% between adjacent frames).

[0062] (3) FFT points: Set the number of points for the Fast Fourier Transform. =512. Due to the symmetry of the spectrum of real-number signals, only the positive frequency band is retained. Each frequency point. After transformation, the time-frequency domain complex spectrum is expressed as follows: , and ,in For frequency point index, For time frame index.

[0063] 3.2 Feature Extraction and Calculation: 1. Logarithmic power spectrum (LPS) of the initial filtered residual signal: ;in, A very small constant to prevent overflow in logarithmic operations.

[0064] 2. Acoustic Coherence Characteristics (COH): Coherence characteristics are used to quantitatively characterize the correlation between the reference signal and the residual signal in the frequency domain, and are a key indicator for distinguishing echoes from near-end speech. The calculation formula is as follows: ;in, Cross-power spectrum Self-power spectrum and The estimation is performed using a first-order infinite impulse response (IIR) smoothing filter: ; ; ; Among them, smoothing factor =0.7.

[0065] 3. Hardware Physical State Feature Normalization and Broadcasting: To eliminate the influence of different physical dimensions on neural network training, the hardware physical state features calculated in step 2 are normalized and broadcast. Real-time temperature of the voice coil corresponding to the frame and real-time DC resistance Perform max-min normalization: , Among them, a temperature mapping range is set based on hardware limit testing. Resistance mapping range Due to the normalization and It is a scalar, while the acoustic feature has a dimension of in the frequency domain. =257. To perform feature fusion, these two scalars are copied along the frequency axis. Next, the dimension is formed as Hardware state feature vector and : , ; in, Represents the tensor product. It is a column vector consisting entirely of 1s.

[0066] 3.3 Multimodal feature stitching: At the channel level, the four features extracted above, along with the real part of the complex spectrum of the initial filtered residual signal, are considered. and the virtual part By splicing the data, a multi-dimensional fused feature tensor is constructed. : ; in, This represents the concatenation of channel dimensions; the input is the number of channels. This represents the number of data frames currently being processed. This fused feature tensor simultaneously contains acoustic energy, acoustic correlation, hardware physical states (temperature, resistance), and phase and amplitude information of the original complex spectrum.

[0067] Example 4 A practical implementation of hardware-aware lightweight deep neural network processing: This module employs a lightweight neural network architecture. Since the front-end physical acoustic cavity nonlinear simulator has eliminated most of the deterministic nonlinear echoes, the neural network only needs to fit the residual minute nonlinear distortion and environmental noise, which allows the network to be designed to be extremely lightweight.

[0068] 4.1 Network Architecture Design: The lightweight deep neural network designed in this embodiment consists of a two-dimensional depth-separable convolutional layer (DS-Conv2D), three cascaded temporal convolutional blocks (TCN Blocks), and a lightweight self-attention layer.

[0069] 1. Two-dimensional depthwise separable convolutional layer (DS-Conv2D): The input tensor is First, depthwise convolution is used... The size is convolution kernel ( Local time-frequency feature extraction is performed within the channel: Then, pointwise convolution is performed using a size of [size missing]. convolution kernel Perform a linear combination of cross-channel features, with the number of output channels set to... : ;in, , To parameterize and modify the activation function of the linear unit, This is the bias term. Compared to ordinary 2D convolution, depthwise separable convolution reduces the number of parameters and computational cost by approximately 80%.

[0070] 2. Temporal Convolutional Blocks (TCN Blocks): The network cascades three temporal convolutional blocks (TCN Blocks 1, 2, and 3) to capture long temporal context. Each TCN block employs a one-dimensional causal dilation depthwise separable convolution. To increase the receptive field without increasing the number of parameters, the first TCN block... layer( The dilation factor of ) grows exponentially by 2, that is .

[0071] The formula for calculating one-dimensional causal dilated convolution is: ; Among them, the kernel size Due to the use of causal convolution design (convolution only on the current frame). and historical frames The network has no look-ahead latency on the time axis, ensuring real-time operation of the hardware. After convolution, nonlinear mapping is performed sequentially through layer normalization and gated linear units (GLUs).

[0072] 3. Lightweight Self-Attention Layer: In order to model the global correlation in the frequency domain without significantly increasing the computational cost, a lightweight self-attention mechanism is introduced after the TCN output.

[0073] feature map Average pooling is performed along the time axis to obtain the frequency domain feature vector. Define the query matrix. Key matrix Sum matrix : , , ; in, , , For learnable projection matrices, attention dimension Calculate the frequency domain attention weight matrix. : .

[0074] The frequency domain features are reconstructed using attention weights to obtain the enhanced frequency domain features. And broadcast it back to the timeline, along with the original features. Add the residuals together.

[0075] 4. Complex Ratio Masking (CRM) Prediction: The last layer of the network outputs a complex ratio mask through a linear mapping layer. .

[0076] To limit the output range and ensure numerical stability, the hyperbolic tangent (tanh) activation function is used: ; .

[0077] in, and These are the real and imaginary parts of the output from the last layer of the network, respectively. This represents the maximum gain limit for masking.

[0078] 4.2 Quantitative Deployment of Hardware: To deploy the trained model on the resource-constrained Cortex-M7 DSP chip, this embodiment employs symmetric 8-bit integer (INT8) post-training quantization (PTQ) technology: For any floating-point activation value tensor The formula for quantizing it to INT8 is: , where scaling factor The calculation is as follows: .

[0079] During hardware forward propagation, operator-level optimizations are performed using the CMSIS-NN library provided by ARM. CMSIS-NN fully utilizes the SIMD (Single Instruction Multiple Data) instruction set of the Cortex-M7, enabling two 16-bit or four 8-bit multiply-accumulate (MAC) operations to be performed simultaneously within one clock cycle. After quantization, the model parameters are only 118KB, and the RAM usage is controlled within 38KB, fully realizing real-time deployment on a micro-hardware platform.

[0080] Example 5 Practical implementation of signal reconstruction and dynamic range control (DRC) post-processing: This module applies the frequency domain complex masking of the neural network output to the initial filtered residual signal to reconstruct clean speech in the time domain, and eliminates transient distortion through dynamic range control.

[0081] 5.1 Complex multiplication in the frequency domain: Mask the predicted complex ratio Complex spectrum of the initial filtered residual signal Complex multiplication is performed to reconstruct the complex spectrum estimate of the near-end clean speech. : ; .

[0082] By multiplying complex numbers, not only was the amplitude spectrum of the speech reconstructed, but the imaginary part information was also used to effectively repair the phase distortion caused by acoustic cavity reflection and noise interference.

[0083] 5.2 Inverse Short-Time Fourier Transform (ISTFT): right Perform ISTFT to recover the temporal near-end speech signal. The overlap-add (OLA) method is used: .

[0084] 5.3 Dynamic Range Control (DRC) Post-processing: In extremely noisy environments, abrupt changes in environmental noise or residual transient nonlinear echoes can lead to reconstructed time-domain signals. This can lead to localized amplitude overshoot, resulting in distortion and nonlinear harmonic distortion during hardware playback or when sent to the downstream ASR engine. This embodiment employs a DRC algorithm for post-processing. 1. Transient envelope detection: Real-time tracking of the transient envelope of a signal using a single-pole IIR filter. : ; Among them, the envelope smoothing coefficient .

[0085] 2. Static gain curve calculation: According to the set limit threshold and compression ratio Calculate the static gain : In this embodiment, the limiting threshold... (correspond Compression ratio .

[0086] 3. Dynamic gain smoothing: To prevent sudden changes in the gain coefficient from introducing new high-frequency harmonic distortion, a start-up time constant is used. and release time constant Smooth the gain: Among them, the startup time constant = 0.05 (corresponding to a fast compression response of approximately 10ms to prevent distortion); release time constant =0.995 (corresponds to a slow release of approximately 300ms to prevent gain sucking effect).

[0087] 4. Final signal output: The smoothed dynamic gain It acts on the reconstructed signal to output a clean near-end speech signal. : .

[0088] Example 6 Experimental testing and effect verification: To verify the practical effectiveness of the method of this invention, a small hardware acoustic cavity test platform was built in a standard acoustic laboratory. The microphone and speaker of the test hardware were spaced 12mm apart and shared a volume of 1.5cm². 3 The sound chamber is made of plastic. The main control chip is a Cortex-M7 MCU (operating frequency 400MHz).

[0089] 6.1 Test conditions: Echo source: Playing high-volume music signals causes the speakers to enter a deep non-linear region (total harmonic distortion (THD) > 10%).

[0090] Ambient noise: Play 85dB of noisy vehicle noise and subway station noise in the test space.

[0091] Near-end speech signal-to-noise ratio (SNR): Set to -10dB to 0dB.

[0092] 6.2 Performance Indicator Comparison: The method of this invention is compared with the traditional Linear Adaptive Filtering (NLMS) algorithm, the traditional Subband Adaptive Filtering (SAF) algorithm, and the pure deep learning algorithm without hardware awareness (Vanilla-DL). The comparison metrics include: (1) ERLE (Echo Loss Enhancement, in dB): The larger the value, the more thorough the echo suppression.

[0093] (2) PESQ (Perceptual Speech Quality Evaluation, score range [-0.5, 4.5]): The higher the value, the higher the speech fidelity.

[0094] (3) WER (Downstream ASR engine word error rate, unit %): Tested in an 85dB noisy environment, the lower the value, the better.

[0095] (4) CPU utilization (%): The computing power consumed on the Cortex-M7 chip.

[0096] The test results are shown in Table 1: Table 1 Test Results As can be seen from the data in Table 1: The method of this invention improves the nonlinear echo suppression capability (ERLE) by more than 25dB compared with traditional algorithms and by about 15dB compared with pure deep learning algorithms through the deep collaboration of hardware-level physical state perception (Smart PA current / voltage feedback) and lightweight neural network.

[0097] Meanwhile, since the physical acoustic cavity nonlinear simulator undertakes most of the nonlinear modeling work, the number of parameters and computational complexity of the deep learning model are extremely low. The CPU utilization rate on the Cortex-M7 chip is only 24.5%, and the running memory is only 38KB, which fully meets the commercial real-time processing requirements of small hardware.

[0098] Most importantly, in an extremely noisy environment with dual interference of -10dB strong noise and strong nonlinear echo, the word error rate (WER) of the speech processed by the method of this invention is sent to the downstream ASR engine, which is drastically reduced from 94.5% before processing to 12.8%, achieving excellent speech recognition results.

[0099] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0100] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for echo suppression and noise reduction in a small hardware acoustic cavity based on deep learning, characterized in that, Includes the following steps: Step 1: Hardware-level physical state perception and multi-channel signal synchronous acquisition: The Smart PA in the hardware circuit is used to acquire the feedback current signal and feedback voltage signal at both ends of the speaker in real time, as hardware characteristic signals characterizing the nonlinear physical state of the speaker; at the same time, the mixed input signal is acquired through the microphone, and the original reference signal of the speaker is obtained through the audio codec. Hardware-level clock synchronization and alignment are performed on the feedback current signal, feedback voltage signal, mixed input signal, and original reference signal; Step 2: Nonlinear pre-modeling and linear echo initial filtering based on physical acoustic cavity state: The synchronized feedback current signal, feedback voltage signal, and original reference signal are input into a lightweight physical acoustic cavity nonlinear simulator built in hardware DSP to reconstruct a nonlinear reference signal containing speaker thermal compression and displacement constraint effects; using this nonlinear reference signal, a sub-band adaptive filtering algorithm is used to initially eliminate linear and low-order nonlinear echoes in the mixed input signal collected by the microphone, and the initial filtered residual signal is output. Step 3: Time-frequency domain feature extraction and multimodal feature fusion: Perform short-time Fourier transform on the initial filtered residual signal, the original reference signal, and the nonlinear reference signal to obtain their respective time-frequency domain complex spectra; extract the logarithmic power spectrum of the initial filtered residual signal, the coherence features of the original reference signal and the initial filtered residual signal, and perform multimodal feature splicing with the time-frequency representation of the hardware feature signal to construct a multidimensional fused feature tensor; Step 4: Hardware-aware lightweight deep neural network processing: The multidimensional fusion feature tensor is input into a pre-trained and quantized lightweight deep neural network. The lightweight deep neural network adopts an architecture that combines a deep separable temporal convolutional network with a self-attention mechanism. The network output predicts complex ratio masking. Step 5: Signal Reconstruction and Post-processing: Multiply the predicted complex ratio mask by the complex spectrum of the initial filtered residual signal to obtain the estimated complex spectrum of the near-end speech; perform an inverse short-time Fourier transform on it to reconstruct the time-domain near-end speech signal; perform amplitude smoothing on the reconstructed time-domain near-end speech signal using a dynamic range control algorithm to output a clean near-end speech signal.

2. The method for echo suppression and noise reduction of a small hardware acoustic cavity based on deep learning according to claim 1, characterized in that, In step 1, the specific implementation of the clock synchronization alignment is as follows: a single master clock source is used to simultaneously drive the analog-to-digital converter (ADC) of the Smart PA and the analog-to-digital converter (ADC) of the audio codec, and the feedback current signal, feedback voltage signal, mixed input signal and original reference signal are written into the shared static random access memory (SRAM) through the dual buffer mechanism of hardware direct memory access (DMA) to control the hardware synchronization error to within 1 microsecond.

3. The method for echo suppression and noise reduction of a small hardware acoustic cavity based on deep learning according to claim 1, characterized in that, In step 2, the process of reconstructing the nonlinear reference signal using the physical acoustic cavity nonlinear simulator includes: Based on the feedback current signal and feedback voltage signal, calculate the real-time temperature of the speaker voice coil and the DC resistance of the voice coil as a function of temperature. A real-time physical displacement state observer for a loudspeaker diaphragm is constructed using electromagnetic and mechanical equations. The state observer is then solved in real time using the fourth-order Runge-Kutta method to obtain the real-time physical displacement of the loudspeaker diaphragm. The state observer considers nonlinear spring stiffness and nonlinear magnetic force factor that vary with displacement.

4. The method for echo suppression and noise reduction of a small hardware acoustic cavity based on deep learning according to claim 1, characterized in that, In step 2, the nonlinear reference signal The reconstruction formula is: ； in, The original reference signal, For real-time temperature, The ambient reference temperature The thermal decay coefficient, For real-time physical displacement, These are higher-order nonlinear distortion coefficients. It is a symbolic function.

5. The method for echo suppression and noise reduction of a small hardware acoustic cavity based on deep learning according to claim 1, characterized in that, In step 2, the specific process of the sub-band adaptive filtering algorithm is as follows: the mixed input signal and the nonlinear reference signal are decomposed into multiple sub-band signals through a uniform discrete Fourier transform analysis filter bank; the linear echo is estimated using a normalized minimum mean square error (NLMS) adaptive filter on each sub-band; the adaptive step size is dynamically adjusted by a coherence-based two-end call factor; and finally, the time-domain initial filter residual signal is reconstructed through a synthetic filter bank.

6. The method for echo suppression and noise reduction of a small hardware acoustic cavity based on deep learning according to claim 1, characterized in that, In step 3, the window function of the short-time Fourier transform is the Hanning window, with a window length of 32 milliseconds, a frame shift of 8 milliseconds, and an overlap rate of 75%. The coherence characteristics are obtained by estimating the cross power spectrum of the original reference signal and the initial filtered residual signal, the auto-power spectrum of the original reference signal, and the auto-power spectrum of the initial filtered residual signal using a first-order infinite impulse response (IIR) smoothing filter, and by calculating the ratio of the square of the cross power spectrum to the product of the two auto-power spectra.

7. The method for echo suppression and noise reduction of a small hardware acoustic cavity based on deep learning according to claim 3, characterized in that, Step 3, the construction process of the multidimensional fusion feature tensor includes: The real-time temperature and the voice coil DC resistance are normalized by maximum and minimum to obtain normalized temperature and normalized resistance. The normalized temperature and normalized resistance are copied along the frequency axis to form a hardware state feature vector consistent with the number of frequency points. In the channel dimension, the logarithmic power spectrum of the initial filtered residual signal, the coherence feature, the hardware state feature vector, and the real and imaginary parts of the complex spectrum of the initial filtered residual signal are concatenated to construct a multidimensional fusion feature tensor with 6 channels.

8. The method for echo suppression and noise reduction of a small hardware acoustic cavity based on deep learning according to claim 1, characterized in that, In step 4, the architecture of the lightweight deep neural network includes, in sequence: A two-dimensional depthwise separable convolutional layer is used to extract local time-frequency features; Three cascaded temporal convolutional blocks, each employing a one-dimensional causal dilated depthwise separable convolution with an exponentially increasing dilation factor per layer, are used to capture long temporal context. A lightweight self-attention layer for global correlation modeling of frequency domain features; A linear output layer is used to predict the real and imaginary parts of the complex ratio mask, and the output range is limited by a hyperbolic tangent activation function.

9. The method for echo suppression and noise reduction of a small hardware acoustic cavity based on deep learning according to claim 1, characterized in that, The lightweight deep neural network is symmetrically quantized using 8-bit integer INT8 before deployment on hardware. Its forward propagation multiply-accumulate operations are performed entirely at INT8 precision and are optimized at the operator level using the CMSIS-NN library.

10. The method for echo suppression and noise reduction of a small hardware acoustic cavity based on deep learning according to claim 1, characterized in that, In step 5, the specific process of the dynamic range control algorithm is as follows: The transient envelope of the near-end speech signal in the time domain is calculated using a single-pole IIR filter; Calculate the static gain curve based on the transient envelope, the set amplitude limiting threshold, and the compression ratio; The static gain curve is smoothed using the start-up time constant and the release time constant to obtain the dynamic gain; The dynamic gain is multiplied by the time-domain near-end speech signal to output a clean near-end speech signal.

Images