A hybrid deep learning-based speech noise suppression method and system based on joint optimization of ARM Cortex-A55 architecture and NEON instruction set.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-03
- Publication Date
- 2026-08-11
AI Technical Summary
[0004]本发明的主要目的在于克服现有技术偏见，解决高质量深度学习降噪模型在中端嵌入式CPU平台上因GRU时序依赖和访存瓶颈导致无法实时运行的问题
1、打破算力瓶颈，实现严苛实时处理（RTF大幅下降）：由于采用了GRU权重矩阵合并与NEON MAC指令的并行化处理，使得原本在Cortex-A55平台上单核运行RTF高达1.52的算法，成功降至0.87（实时率提升42.8%）。使得在无NPU的中端芯片上运行高质量深度学习降噪成为现实。
Smart Images
Figure CN122551812A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of speech signal processing and embedded artificial intelligence technology, specifically to a hardware and software co-optimization method and system for real-time speech noise suppression using a hybrid DSP and deep learning model on a resource-constrained ARM Cortex-A55 embedded platform without a dedicated NPU. Background Technology
[0002] In aviation communication scenarios, background noise (such as engine noise and airflow disturbance) severely affects the clarity and reliability of voice communication. Traditional denoising algorithms, such as spectral subtraction and Wiener filtering, have low computational cost, but their denoising effect is poor in non-stationary noise environments, leading to speech distortion. Pure deep learning models, such as SEGAN and Conv-TasNet, have good denoising effects, but they have a huge number of parameters and extremely high computational complexity, making it difficult to implement in real time on embedded edge devices with limited computing power and power consumption.
[0003] The RNNoise algorithm employs a hybrid architecture combining DSP feature extraction and GRU inference, achieving a balance between efficiency and quality. However, a common industry bias holds that the GRU layer, due to its multiple gating units (update gates, reset gates, candidate hidden states) and strong temporal cyclic data dependencies, is difficult to perform effective parallel computation. Therefore, deploying it on a mid-range embedded CPU like the Cortex-A55, which lacks a dedicated NPU, for real-time execution is considered a significant challenge, or even impossible. In practical engineering deployments, simple compiler-level optimizations (such as configuring the O3 optimization level) cannot effectively solve the parallelization problem of GRU. Its computational bottleneck lies in the frequent multiplication and addition operations of small matrices and the serial execution of nonlinear activation functions, leading to frequent CPU pipeline stalls and low cache hit rates. Directly deploying standard RNNoise on an RK3568 (Cortex-A55) typically results in a single-core real-time rate (RTF) greater than 1.5, completely failing to meet the stringent low-latency requirements of voice communication. Therefore, meticulous instruction-level optimization of the algorithm's core computational graph and memory layout is necessary. Summary of the Invention
[0004] The main objective of this invention is to overcome the biases of existing technologies and solve the problem that high-quality deep learning noise reduction models cannot run in real time on mid-range embedded CPU platforms due to GRU timing dependencies and memory access bottlenecks. By introducing a custom data memory layout, cache blocking strategy, and deep instruction-level optimization of ARM NEON SIMD, this invention breaks through the traditional serial computing bottleneck and provides a speech noise suppression solution that can maintain high audio quality at the deep learning level while meeting stringent real-time requirements (RTF < 1.0) and low power consumption.
[0005] This invention provides a hybrid deep learning-based speech noise suppression method based on ARM Cortex-A55 SoC, comprising the following steps: S1. Audio stream acquisition and framing: The PCM audio stream is acquired in real time through the audio interface of ARM RK3568, high-pass filtered to remove DC components, and overlapped and framed according to preset duration. S2. Frequency Domain Transformation and Feature Extraction: Window the framed signal, transform it to the frequency domain using FFT, and extract the acoustic feature vector; S3. Joint Optimization of Core Computation with NEON Instruction Set and Memory: Features are input into an RNN network containing multiple GRU layers and one-dimensional convolutional layers, and deep reconstruction is performed using the NEON instruction set of the ARMv8 architecture, including: S3.1 Custom vectorized reconstruction of GRU circular dependencies: The weight matrix in GRU computation is interleaved and merged in memory to ensure 16-byte data alignment. The input vector and the merged weight data are continuously loaded using NEON's vld1q_f32 instruction. The matrix-vector multiplication and accumulation operation is performed in parallel using the vfmaq_f32 instruction. S3.2 Complex FFT optimization for Cortex-A55 architecture: The real and imaginary parts of the FFT input data are interleaved and assembled in the register, and the butterfly operation of multiple pairs of floating-point numbers is processed in parallel in a single cycle using the NEON instruction. S3.3 Memory access mode optimization matching the cache level: For the L1 data cache size of Cortex-A55, a cache blocking mechanism is introduced, the GRU weight matrix is divided into sub-blocks that are adapted to the L1 cache, so that multiply-accumulate operations are completed in the L1 cache, and the next block of data is loaded into the cache in advance using data prefetch instructions. S4. Gain Calculation, Waveform Synthesis and Precision Protection: The network output bandwidth gain is interpolated, multiplied by the original spectrum, and synthesized using IFFT. A single-precision floating-point arithmetic pipeline is used throughout the NEON acceleration process to handle potential overflow during the accumulation process.
[0006] Preferably, in step S1, the audio sampling rate is 48kHz and the frame duration is 10ms / frame.
[0007] Preferably, in step S2, the signal is windowed using the Vorbis window function, and the extracted acoustic feature vector is 42-dimensional, including the logarithmic energy of 22 Bark frequency bands, pitch period correlation features, and cepstral coefficients.
[0008] Preferably, in step S3.1, the staggered merging arrangement of the weight matrix includes merging the weight matrices of the reset gate and the update gate, and in step S3.3, the weight matrix is divided into 32×32 sub-blocks.
[0009] This invention also discloses a hybrid deep learning speech noise suppression system based on an ARM Cortex-A55 SoC, comprising: The signal acquisition module is used to acquire PCM audio streams; The memory is used to store the interleaved and merged, 16-byte aligned GRU weight matrix; A processor, wherein the processor is a Cortex-A55 architecture, is configured to execute NEON instructions to implement the method according to any one of claims 1 to 4; The audio playback module is used to output the noise-reduced audio.
[0010] The beneficial effects of this invention are as follows: by establishing a rigorous underlying optimized causal chain, it has the following significant advantages compared with the prior art: 1. Breaking through computing power bottlenecks and achieving rigorous real-time processing (significantly reduced RTF): Due to the adoption of GRU weight matrix merging and parallel processing with NEON MAC instructions, the algorithm, which originally had an RTF as high as 1.52 on a single core of the Cortex-A55 platform, was successfully reduced to 0.87 (a 42.8% improvement in real-time rate). This makes it possible to run high-quality deep learning noise reduction on mid-range chips without an NPU.
[0011] 2. Significantly reduced computing power consumption and memory access latency: Thanks to the introduction of cache blocking strategies and memory prefetching technology, the number of off-chip DDR memory accesses is reduced by approximately 60%, and the CPU clock cycles waiting for data are significantly reduced. Single-core CPU utilization is reduced from 68.5% to 41.2% (a reduction of 39.9%), reserving valuable resources for the embedded system to run other core business operations.
[0012] 3. Pareto optimality of sound quality and efficiency: By employing FP32-precision NEON instructions for equivalent mathematical reconstruction rather than brute-force low-bit quantization, this method perfectly maintains the high noise reduction of deep learning methods while improving speed. In the objective test set, the PESQ (Perceptual Speech Quality) and STOI (Short-Time Objective Intelligibility) metrics are completely consistent with the unoptimized baseline floating-point model, overcoming the sound quality degradation problem caused by traditional fixed-point or lightweight models.
[0013] 4. Reduce overall system power consumption: Due to the shortened computing time and the sharp drop in DDR read / write frequency, the average power consumption of the SoC when performing noise reduction tasks is significantly reduced, making it extremely suitable for battery-powered portable communication terminals and IoT edge devices. Attached Figure Description
[0014] Figure 1 Here is a diagram of the RNNoise algorithm framework; Figure 2 Here is a flowchart of the noise suppression system; Figure 3 This is a hardware processing flowchart; Figure 4 The graph shows the RTF of four noise reduction methods on the RK3568 as a function of the number of CPU cores. Figure 5 The bar chart shows the computational complexity and memory resource requirements of four noise suppression algorithms. Detailed Implementation
[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in further detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.
[0016] First Embodiment This embodiment provides a hybrid deep learning speech noise suppression method based on an ARM Cortex-A55 SoC. This method is implemented on a Rockchip RK3568 development board, which integrates a quad-core ARM Cortex-A55 processor with a clock speed of 1.4GHz, 32KB L1 data cache, and an external USB microphone for sound acquisition. The processed audio is output through a 3.5mm audio interface. Figure 3 As shown, the hardware processing flow is based on the RK3568 development board. The sound collected by the USB microphone is noise-reduced by the processor and then output through the audio interface, clearly demonstrating the physical hardware connection method.
[0017] The method in this embodiment includes the following steps: S1. Audio Stream Acquisition and Framing. A 48kHz PCM audio stream is acquired in real time via the ARM RK3568's audio interface. High-pass filtering removes the DC component, and overlapping frames are performed at 10ms / frame. This step ensures the continuity and stability of the input signal, laying the foundation for subsequent frequency domain processing.
[0018] S2. Frequency Domain Transformation and Feature Extraction. The signal is windowed using the Vorbis window function and transformed to the frequency domain via FFT. A 42-dimensional acoustic feature vector is extracted, including the logarithmic energy, pitch period correlation features, and cepstral coefficients of 22 Bark bands. The division of the Bark bands conforms to the characteristics of human hearing, which helps improve the subjective quality of noise reduction. Figure 2As shown, the flowchart of this noise suppression system fully presents the entire process from framing, windowing, FFT (time-frequency conversion), and feature extraction of the input signal, to RNN model inference, and finally to synthesizing the noise-reduced audio output through IFFT, demonstrating how traditional digital signal processing and deep learning can work together.
[0019] S3. Joint optimization of the NEON instruction set and memory for core computation. Features are input into an improved RNN network containing multiple GRU layers and one-dimensional convolutional layers. For example... Figure 1 As shown, the RNNoise algorithm framework adopts a hybrid architecture, comprising three main branches: Voice Activity Detection (VAD), noise spectrum estimation, and spectral subtraction gain calculation. Its core utilizes a gated recurrent unit layer to process time-series features, achieving complex noise judgment with a minimal number of parameters. In this process, the core tasks are deeply reconstructed using the NEON instruction set unique to the ARMv8 architecture. Specifically, this includes: S3.1 Custom Vectorized Reconstruction of GRU Circular Dependencies. Addressing Reset Gates in GRU Computation. and the update gate To address the serialization problem, this invention redesigns the computation graph and data layout. The equal-weight matrices are interleaved and merged in memory to ensure 16-byte data alignment. During computation, the input vector and merged weight data are continuously loaded using NEON's vld1q_f32 instruction through loop expansion, and then matrix-vector multiplication and accumulation operations are performed in parallel using the vfmaq_f32 instruction. This allows for the simultaneous completion of partial linear transformations of multiple gates within a single instruction cycle, completely overcoming the inherent serial computation bottleneck of the GRU.
[0020] S3.2 Optimization for Complex FFT in Cortex-A55 Architecture. This invention abandons the traditional scalar complex number operations that require multiple instructions to process the real and imaginary parts separately. For time-frequency conversion, it interleaves and assembles the real and imaginary parts of the input data in a register, utilizing NEON's vadd_f32 and vmul_f32 instruction combinations to process two / four pairs of floating-point numbers in parallel within a single cycle, significantly reducing the number of instruction cycles.
[0021] S3.3 Optimization of Memory Access Mode Matching Cache Level. The bottleneck of embedded systems lies in memory access. For the typical 32KB L1 data cache of the Cortex-A55, this invention introduces a cache blocking mechanism. The large GRU weight matrix is divided into 32×32 sub-blocks or sub-blocks of suitable L1 cache size. The NEON coprocessor only needs to load the current sub-block from off-chip DDR; all subsequent multiply-accumulate operations are completed within the L1 cache. Simultaneously, using data prefetch instructions (__builtin_prefetch or NEONpld), the next block of data is pre-loaded into the cache while the CPU is calculating the current block, effectively hiding memory access latency. In this step, dividing the weight matrix into 32×32 sub-blocks is the preferred solution, as it matches well with the L1 cache size of the Cortex-A55.
[0022] S4. Gain Calculation, Waveform Synthesis, and Accuracy Protection. The network output bandwidth gain is interpolated, multiplied by the original spectrum, and synthesized using IFFT. Throughout the NEON acceleration process, a single-precision floating-point (FP32) arithmetic pipeline is strictly employed, and potential overflow during accumulation is handled to ensure that the optimized numerical results are completely consistent with the original model without any loss of accuracy.
[0023] This embodiment also relates to a system for implementing the above method, comprising: a signal acquisition module for acquiring a 48kHz PCM audio stream; a memory for storing a GRU weight matrix that is interleaved and merged and aligned to 16 bytes; a processor, wherein the processor is a quad-core ARM Cortex-A55 architecture with a main frequency of 1.4GHz, for executing NEON instructions to implement the above method; and an audio playback module for outputting the noise-reduced audio.
[0024] To verify the technical effectiveness of this embodiment, the following comparative experiments were conducted. For example... Figure 4 As shown, the real-time performance (RTF) of the four algorithms was compared under different numbers of CPU cores. The red line (RNNoise baseline) was always greater than 1.0 (unable to perform in real time), while the blue line (RNNoise-NEON) successfully dropped below 1.0, which intuitively demonstrates the real-time performance advantage of the optimization of this invention. Figure 5 The algorithm demonstrates the computational complexity and memory usage of different algorithms. The left bar chart shows the number of floating-point operations (FLOPs), and the right bar shows the memory usage. This algorithm (cyan bar) reduces the computational cost per frame from 15.2M FLOPs to 8.7M FLOPs, quantifying the specific computational power savings brought by SIMD technology.
[0025] Table 1: Performance Comparison under Different Optimization Strategies
[0026] Table 1 shows that the problem cannot be solved by the compiler alone, and the hardware and software co-optimization of this invention is necessary.
[0027] Table 2: Performance Analysis of Each Module
[0028] The significant decrease in FFT / IFFT latency in Table 2 indicates that the NEON optimization for complex numbers is effective. The decrease in GRU inference latency indicates that the custom vectorized reconstruction of GRU circular dependencies and the memory access mode optimization matching the cache level can overcome the computation and memory access bottlenecks.
[0029] Table 3: Objective Indicators of Sound Quality and Power Consumption Data
[0030] Table 3 shows the results of tests conducted on the DNS Challenge noise dataset (0dB SNR), demonstrating that the method employed in this invention achieves lossless sound quality and reduced power consumption.
[0031] Second Embodiment
[0032] This embodiment provides another hybrid deep learning speech noise suppression method based on ARM Cortex-A55 SoC. The difference between this method and the first embodiment lies in the hardware platform, the selection of some parameters, and the adaptive adjustment of the cache blocking strategy, so as to prove that the technical solution of the present invention has universality and scalability.
[0033] This embodiment uses another system-on-a-chip (SoC) that integrates an ARM Cortex-A55 processor. This chip has a quad-core configuration, a clock speed of 1.8GHz, a 64KB L1 cache, and LPDDR4X system memory, with a more stringent power consumption budget. The application scenario for this embodiment is a portable walkie-talkie device, which requires extremely low processing latency (end-to-end latency <20ms) and lower power consumption.
[0034] The method in this embodiment includes the following steps: S1. Audio Stream Acquisition and Framing. A 16kHz PCM audio stream is acquired in real-time via the SoC's I2S digital audio interface (the sampling rate is lower than in the first embodiment to reduce overall data bandwidth). High-pass filtering removes the DC component, and overlapping frames are performed at 5ms / frame. The shorter frame length helps further reduce the latency introduced by the algorithm, meeting the stringent latency requirements of intercom devices.
[0035] S2. Frequency Domain Transformation and Feature Extraction. The signal is windowed using a sinusoidal window function and transformed to the frequency domain via FFT. A 36-dimensional acoustic feature vector is extracted, including the logarithmic energy of 20 Bark bands, pitch period correlation features, and cepstral coefficients. The reduction in feature dimensions is based on a balance between latency and accuracy, further reducing computational load while ensuring noise reduction quality.
[0036] S3. Joint optimization of the NEON instruction set and memory for core computation. Features are input into an RNN network containing multiple GRU layers and one-dimensional convolutional layers. This embodiment further optimizes the data layout and memory access strategy based on the first embodiment: S3.1 Custom Vectorized Reconstruction of GRU Cyclic Dependencies. Similar to the first embodiment, the weight matrices in the GRU computation are interleaved and merged in memory to ensure 16-byte data alignment. However, this embodiment employs a more aggressive cycle unrolling strategy for a larger L1 cache (64KB), merging multiple consecutive vector operations into a longer SIMD pipeline, further reducing the number of instruction issues.
[0037] S3.2 Optimization of complex FFT for Cortex-A55 architecture. For inputs with a 16kHz sampling rate, the number of FFT points is reduced accordingly. This embodiment uses a hybrid-radix FFT algorithm, utilizing NEON instructions to process butterfly arithmetic units of different radixes in parallel, and completing all transformations within a single core through register reuse, avoiding the additional overhead caused by cross-core data exchange.
[0038] S3.3 Memory access pattern optimization matching the cache level. For the 64KB L1 data cache in this embodiment, the block size of the GRU weight matrix is adjusted to 64×64 to fully utilize the larger cache space and reduce the number of off-chip memory accesses. At the same time, NEON's built-in prefetch instructions (such as vld1q_f32 in conjunction with prefetch offset) are used to accurately prefetch the data block required for the next operation while loading the current data block, further hiding the memory access latency. For the weight matrix part that cannot be completely loaded into the L1 cache, a double-layer blocking strategy is adopted, and the blocks are re-divided at the L2 cache level to maximize the utilization of the chip's multi-level cache structure.
[0039] S4. Gain Calculation, Waveform Synthesis, and Precision Protection. The network output bandwidth gain is interpolated, multiplied by the original spectrum, and then synthesized using IFFT. In this embodiment, NEON instructions are also used to accelerate complex number operations during the IFFT synthesis stage, forming a symmetrical acceleration with the optimization in the FFT stage. Throughout the entire NEON acceleration process, a single-precision floating-point (FP32) arithmetic pipeline is strictly used, and accumulator saturation detection is introduced to ensure numerical stability.
[0040] This embodiment also provides a corresponding system, including: a signal acquisition module for acquiring a 16kHz PCM audio stream via an I2S interface; a memory for storing a GRU weight matrix that is interleaved and merged, aligned to 16 bytes, and optimized for block size indexing; a processor, which is a quad-core ARM Cortex-A55 architecture with a main frequency of 1.8GHz, for executing NEON instructions to implement the above method; and an audio playback module for outputting noise-reduced audio, which is also connected to an audio codec via an I2S interface.
[0041] The technical effectiveness of this embodiment is verified by the following experimental data: Under a 16kHz sampling rate and a 5ms frame length, the method of this embodiment achieves a single-core RTF of 0.65 and an end-to-end processing latency of 12ms on the target platform, with the average system power consumption reduced by 28% compared to the unoptimized baseline version. Simultaneously, in terms of objective audio quality indicators, the PESQ score improved from 1.72 for the original noisy speech to 2.85, and the STOI improved from 0.70 to 0.91, comparable to the audio quality improvement achieved by the first embodiment at 48kHz. This demonstrates that the technical solution of this invention can achieve excellent results under different sampling rates, different frame lengths, and different hardware parameters.
Claims
1. A hybrid deep learning speech noise suppression method based on ARM Cortex-A55 SoC, characterized in that, Includes the following steps: S1. Audio Stream Acquisition and Framing: The PCM audio stream is acquired in real time through the audio interface of the ARM RK3568, high-pass filtered to remove DC components, and overlapped into frames according to a preset duration. S2. Frequency Domain Transformation and Feature Extraction: Window the framed signal, transform it to the frequency domain using FFT, and extract the acoustic feature vector; S3. Joint Optimization of Core Computation with NEON Instruction Set and Memory: Features are input into an RNN network containing multiple GRU layers and one-dimensional convolutional layers, and deep reconstruction is performed using the NEON instruction set of the ARMv8 architecture, including: S3.1 Custom vectorized reconstruction of GRU circular dependencies: The weight matrix in GRU computation is interleaved and merged in memory to ensure 16-byte data alignment. The input vector and the merged weight data are continuously loaded using NEON's vld1q_f32 instruction. The matrix-vector multiplication and accumulation operation is performed in parallel using the vfmaq_f32 instruction. S3.2 Complex FFT optimization for Cortex-A55 architecture: The real and imaginary parts of the FFT input data are interleaved and assembled in the register, and the butterfly operation of multiple pairs of floating-point numbers is processed in parallel in a single cycle using the NEON instruction. S3.3 Memory access mode optimization matching the cache level: For the L1 data cache size of Cortex-A55, a cache blocking mechanism is introduced, the GRU weight matrix is divided into sub-blocks that are adapted to the L1 cache, so that multiply-accumulate operations are completed in the L1 cache, and the next block of data is loaded into the cache in advance using data prefetch instructions. S4. Gain Calculation, Waveform Synthesis and Precision Protection: The network output bandwidth gain is interpolated, multiplied by the original spectrum, and synthesized using IFFT. A single-precision floating-point arithmetic pipeline is used throughout the NEON acceleration process to handle potential overflow during the accumulation process.
2. The hybrid deep learning speech noise suppression method based on ARM Cortex-A55 SoC according to claim 1, characterized in that, In step S1, the audio sampling rate is 48kHz and the frame duration is 10ms / frame.
3. The hybrid deep learning speech noise suppression method based on ARM Cortex-A55 SoC according to claim 1, characterized in that, In step S2, the signal is windowed using the Vorbis window function, and the extracted acoustic feature vector is 42-dimensional, including the logarithmic energy of 22 Bark frequency bands, pitch period correlation features, and cepstral coefficients.
4. The hybrid deep learning speech noise suppression method based on ARM Cortex-A55 SoC according to claim 1, characterized in that, In step S3.1, the staggered merging arrangement of the weight matrix includes merging the weight matrices of the reset gate and the update gate. In step S3.3, the weight matrix is divided into 32×32 sub-blocks.
5. An ARM Cortex-A55 SoC based hybrid deep learning speech noise suppression system, characterized in that, include: The signal acquisition module is used to acquire PCM audio streams; The memory is used to store the interleaved and merged, 16-byte aligned GRU weight matrix; A processor, wherein the processor is a Cortex-A55 architecture, is configured to execute NEON instructions to implement the method according to any one of claims 1 to 4; The audio playback module is used to output the noise-reduced audio.