An audio noise reduction method and system based on time series signal processing

Abstract

The present application relates to a kind of audio noise reduction method and system based on time series signal processing, it belongs to audio signal processing field, method includes the continuous audio data frame is divided into even frame and odd frame according to time series, even frame executes full quantity frequency domain transformation and pitch analysis and feature is written into delay buffer register;Odd frame bypasses frequency domain transformation, reconstructs frequency domain feature based on time domain sampling signal and historical complex spectrum and executes cross-frame correlation mathematical compensation, by multistage energy gate control determination, when signal energy is lower than threshold value, using silence maintenance countdown counter to force to maintain recurrent neural network hot engine state to protect hidden state context memory, finally, the high-speed batch conversion of floating point data format to fixed point data format is realized in combination with SIMD vectorization instruction.The present application reduces nearly half of the front-end algorithm while maintaining high-fidelity noise reduction, completely eliminates the "soft start" artifact of speech wake-up initial stage, and significantly improves system-level processing throughput.

Description

Technical Field

[0001] This invention relates to the field of audio signal processing, and in particular to an audio noise reduction method and system based on time series signal processing. Background Technology

[0002] Currently, the closest existing technology widely used in the industry is a hybrid audio noise reduction architecture based on recurrent neural networks (RNNs) combined with classical digital signal processing (DSP) feature extraction (typically represented by the RNNoise open-source framework and its derivative systems).

[0003] In its implementation, existing systems typically segment continuous time-domain audio signals into frames within a fixed time window (e.g., 10 milliseconds). For each audio frame, the system first performs preprocessing and a high-point (e.g., 960-point) forward Fast Fourier Transform (FFT) to convert the signal from the time domain to the frequency domain. Next, the system performs computationally intensive "pitch analysis": calculating correlations globally to find pitch periods, thereby constructing a comb filter to filter noise present between the harmonics of the speech pitch. Finally, the extracted frequency band energy and pitch features are fed into a GRU network, which calculates an ideal frequency domain gain mask through forward inference. After the mask is applied to the original spectrum, a clean time-domain speech is reconstructed using an inverse Fast Fourier Transform (IFFT).

[0004] However, existing technologies have the following serious drawbacks:

[0005] 1. Feature extraction suffers from extremely high temporal computational redundancy: The fundamental frequency of human speech changes very slowly over extremely short periods (such as tens of milliseconds). Current techniques indiscriminately force a massive forward FFT and global pitch period search for every 10-millisecond audio frame. This single pre-feature extraction step consumes 50% to 60% of the overall computation time for a single frame, resulting in a serious waste of computational power.

[0006] 2. RNN hidden states suffer catastrophic decay in pure silence: In real-world communication, the duty cycle of silence or pure noise is extremely high. Current technologies, if maintaining full-power inference without voice input, will unnecessarily drain the battery; while using a traditional hard silence threshold to cut off inference, the hidden state matrix within the GRU network, which maintains contextual memory, will quickly reach zero. When the user resumes speaking, the network needs tens of milliseconds to re-accumulate speech feature context, causing the first syllable to be abruptly swallowed, producing a destructive "soft start" machine artifact.

[0007] 3. Existing frame skipping strategies disrupt the continuity of time series: Existing network inference layer frame skipping techniques, once they abruptly cut off the input, completely tear apart the continuity of time series data, causing gaps in the feature sequence. This usually requires complex secondary retraining of the neural network with minimal benefits.

[0008] 4. Scalar format conversion causes congestion in the underlying instruction pipeline: At the end of the audio processing pipeline, the 32-bit floating-point (F32) data output by the neural network must be converted into 16-bit signed integers (S16) required by the hardware sound card. Existing technologies typically use scalar traversal computation sample by sample, which seriously wastes the vectorized computation potential of modern CPUs when processing tens of thousands of samples per second.

[0009] Therefore, there is an urgent need for an audio noise reduction method based on time series signal processing to solve the above-mentioned shortcomings. Summary of the Invention

[0010] To address the technical problems of high redundancy in feature extraction computation, easy decay of hidden states in recurrent neural networks during silent segments, disruption of time series continuity by frame skipping strategies, and low efficiency of data format conversion in existing technologies, this invention provides an audio noise reduction method and system based on time series signal processing.

[0011] In a first aspect of the present invention, an audio noise reduction method based on time-series signal processing is provided. The method includes:

[0012] The continuously input audio data frames are divided into even frames and odd frames according to the time sequence. The first valid audio data frame after the system cold start is forcibly marked as an even frame.

[0013] When the current frame is an even frame, the input signal is windowed and then subjected to full frequency domain transformation and pitch analysis to extract the complex spectrum, band energy, pitch index, mean square energy and time domain sampled signal, and the complex spectrum, band energy, pitch index and mean square energy are written into the delay buffer register.

[0014] When the current frame is an odd frame, bypass full frequency domain transformation and pitch analysis, directly read the historical complex spectrum, historical pitch index, historical mean square energy, and historical time-domain sampled signal of the previous valid even frame from the delay buffer register, and obtain the mean square energy and time-domain sampled signal of the odd frame. Based on the time-domain sampled signal of the odd frame and the historical complex spectrum, perform cross-frame correlation mathematical compensation, fuse and generate the complex spectrum of the odd frame, map the complex spectrum of the odd frame onto the Buck band, calculate the frequency band energy of the odd frame, and reuse the historical pitch index as the pitch index of the current odd frame.

[0015] A feature vector is constructed based on the frequency band energy, mean square energy, and pitch index of the current frame. The mean square energy of the current frame is then processed by a multi-level energy gating system and fed into a recurrent neural network for gain inference, generating a frequency domain gain mask.

[0016] Apply the frequency domain gain mask to the complex spectrum of the current frame, and reconstruct the time domain signal waveform through inverse fast Fourier transform;

[0017] During the output phase, based on the hardware architecture capabilities of the host CPU, single instruction multiple data stream vectorization instructions are invoked to convert time-domain signal waveforms from floating-point data format to fixed-point data format in batches.

[0018] Optionally, the method for calculating the mean square energy of the even-numbered frames includes:

[0019] Obtain the frame length, time-domain sampled signal, and index number of even-numbered frames;

[0020] The mean square energy of even-numbered frames is determined based on the frame length, time-domain sampled signal, and index number.

[0021] The method for calculating the mean square energy of the odd-numbered frames includes:

[0022] Obtain the frame length, time-domain sampled signal, and index number of odd-numbered frames;

[0023] The mean square energy of odd-numbered frames is determined based on the frame length, time-domain sampled signal, and index number.

[0024] The formula for calculating the mean square energy of even-numbered frames is the same as that for odd-numbered frames, both being:

[0025]

[0026] in, Mean square energy, For frame length, For index number, This is a time-domain sampled signal.

[0027] Optionally, the cross-frame correlation mathematical compensation includes:

[0028] The mean square energy of odd-numbered frames is extracted as the transient reference for amplitude compensation;

[0029] Perform an inverse fast Fourier transform on the historical complex spectrum to reconstruct the historical time-domain sampled signal, calculate the cross-frame normalized cross-correlation function between the time-domain sampled signal of odd frames and the historical time-domain sampled signal, and determine the optimal time offset that maximizes the cross-frame normalized cross-correlation function through local search.

[0030] Based on the time-shifting characteristics of Fourier transform, a phase rotation factor with the corresponding optimal time offset is applied to the historical complex spectrum, and amplitude correction is performed using the ratio of the mean square energy of odd frames to the historical mean square energy to synthesize the complex spectrum of odd frames.

[0031] The synthesized complex spectrum is mapped onto the Barker band, and the energy of each band is calculated.

[0032] Optionally, the formula for calculating the cross-frame normalized cross-correlation function is as follows:

[0033]

[0034] in, For cross-frame normalized cross-correlation function, This is the time offset. This is a historical time-domain sampled signal;

[0035] The formula for calculating the complex spectrum of the odd-numbered frames is as follows:

[0036]

[0037]

[0038] in, This is the ratio of the mean square energy of odd-numbered frames to the historical mean square energy. The mean square energy of odd-numbered frames. For historical mean square energy, For the complex spectrum of odd-numbered frames, For the historical complex spectrum, For discrete frequency points in the frequency domain, The imaginary unit, , This is the maximum time offset;

[0039] The formula for calculating the frequency band energy of the odd-numbered frames is as follows:

[0040]

[0041] in, For the frequency band energy of odd-numbered frames, For frequency band indexing, This refers to the ending frequency index, which represents the position of the ending boundary frequency point corresponding to the b-th coarse-grained frequency band on the discrete frequency spectrum axis. This is the starting frequency index, which is the position of the starting boundary frequency point corresponding to the b-th coarse-grained frequency band on the discrete spectrum axis.

[0042] Optionally, the multi-level energy gating includes L1-level gating, L2-level gating, and L3-level gating, wherein:

[0043] The criteria for L1 gating include:

[0044] Obtain the mean square energy of the current frame;

[0045] Determine whether the mean square energy of the current frame is lower than a preset extreme background noise threshold;

[0046] If so, proceed to the L3 level gated mute maintenance judgment;

[0047] If not, it is determined that there is a valid sound input, the mute countdown counter is reset to its full-scale value, and the L2 gating judgment is entered; wherein, the full-scale value of the mute countdown counter is configured as the number of audio frames corresponding to the preset duration;

[0048] The criteria for L2 gating include:

[0049] Determine whether the frequency band energy of the current frame has speech harmonic characteristics;

[0050] If so, the feature vector of the current frame is fed into the recurrent neural network;

[0051] If not, the feature vector of the current frame is fed into the recurrent neural network, but the frequency domain gain mask output is suppressed, and the context memory weight update of the hidden state of the network is maintained.

[0052] Optionally, the mute retention determination includes:

[0053] When the L1-level gating determines that the mean square energy of the current frame is lower than the preset extreme background noise threshold, it blocks the transmission of the feature vector of the current frame to the recurrent neural network.

[0054] Check the current value of the mute hold countdown counter;

[0055] If the current value is greater than zero, the current value of the mute countdown counter will be decremented by one, and the feature vector of the current frame will be forcibly fed into the recurrent neural network for gain inference in order to maintain the context memory weights of the hidden state matrix inside the recurrent neural network.

[0056] If the current value is zero, the transmission of the feature vector of the current frame to the recurrent neural network is blocked and the system enters a silent sleep state.

[0057] Optionally, the recurrent neural network employs a gated recurrent unit architecture, and the gain inference includes:

[0058] The input feature vector is multiplied and added to the hidden state matrix retained from the previous time step.

[0059] The temporal coherence of speech is analyzed by utilizing the update gate and reset gate within the gated loop unit architecture;

[0060] The Sigmoid activation function outputs a frequency domain gain mask with values ​​between 0.0 and 1.0, which defines the proportion of speech components preserved in each specific frequency band of the current frame.

[0061] Optionally, the reconstruction of the time-domain signal waveform via inverse fast Fourier transform includes:

[0062] Frequency domain filtering is achieved by dot-multiplying the frequency domain gain mask onto the complex spectrum of the current frame.

[0063] The filtered complex spectrum is restored to a time-domain sampling point sequence using the inverse fast Fourier transform;

[0064] The temporal sampling point sequence of the current frame is overlapped and added with the temporal sampling point sequence of the adjacent frame according to a preset frame overlap rate to generate a continuous denoised digital audio waveform.

[0065] Optionally, the step of invoking single-instruction multiple-data vectorized instructions based on the hardware architecture capabilities of the host central processing unit specifically includes:

[0066] It can sniff out the hardware architecture of the host central processing unit and identify the supported vectorized instruction set types, including the AVX2 instruction set or SSE2 instruction set for the x86_64 architecture and the NEON instruction set for the ARM architecture.

[0067] Based on the recognition results, the corresponding wide hardware register is called, and multiple floating-point samples are truncated, limited, and converted into fixed-point data format in a single central processing unit instruction.

[0068] A double-loop unrolling strategy is adopted to achieve high-speed data flushing of the memory bus.

[0069] The second aspect of the present invention provides an audio noise reduction system based on time-series signal processing. The system includes:

[0070] The frame division module is used to divide continuously input audio data frames into even frames and odd frames according to the time sequence. The first valid audio data frame after the system cold start is forcibly marked as an even frame.

[0071] The even-numbered frame processing module is used to perform full frequency domain transformation and pitch analysis on the input signal after windowing when the current frame is an even-numbered frame. It extracts the complex spectrum, frequency band energy, pitch index, mean square energy and time domain sampled signal, and writes the complex spectrum, frequency band energy, pitch index and mean square energy into the delay buffer register.

[0072] The odd-numbered frame processing module is used to bypass full frequency domain transformation and pitch analysis when the current frame is an odd-numbered frame. It directly reads the historical complex spectrum, historical pitch index, historical mean square energy, and historical time-domain sampled signal of the previous valid even-numbered frame from the delay buffer register, and obtains the mean square energy and time-domain sampled signal of the odd-numbered frame. Based on the time-domain sampled signal of the odd-numbered frame and the historical complex spectrum, it performs cross-frame correlation mathematical compensation, fuses and generates the complex spectrum of the odd-numbered frame, maps the complex spectrum of the odd-numbered frame to the Buck band, calculates the frequency band energy of the odd-numbered frame, and reuses the historical pitch index as the pitch index of the current odd-numbered frame.

[0073] The gated inference module is used to construct a feature vector based on the frequency band energy, mean square energy and pitch index of the current frame. The mean square energy of the current frame is then fed into a recurrent neural network for gain inference after being determined by multi-level energy gating, and a frequency domain gain mask is generated.

[0074] The inverse transform reconstruction module is used to apply the frequency domain gain mask to the complex spectrum of the current frame and reconstruct the time domain signal waveform through inverse fast Fourier transform.

[0075] The vectorized output module is used to call single instruction multiple data stream vectorization instructions according to the hardware architecture capabilities of the host central processing unit during the output stage, and to convert time domain signal waveforms from floating-point data format to fixed-point data format in batches.

[0076] In summary, the present invention has at least one of the following beneficial technical effects:

[0077] 1. Breaking through the lower limit of computing power for single-frame processing latency: By introducing staggered frame temporal decoupling and cross-frame feature stitching mechanism, the system directly eliminates 50% of the huge forward FFT and pitch search calculations from a mathematical perspective without changing the physical audio sampling length, thus freeing up valuable CPU interrupt response time.

[0078] 2. Completely eliminate the "soft start" sound quality artifacts in the initial wake-up stage and achieve extreme energy saving: By introducing a high-order protection gating system with a physical countdown timer, the short pauses between words in speech are perfectly filled within a forced warm-up delay of 200 milliseconds. This ensures that when the user resumes speaking, the recurrent neural network can immediately output an accurate gain mask with the highest precision without having to re-accumulate features from several audio frames. This achieves zero inference computing power consumption during long periods of silence while maintaining zero damage to the first syllable of sudden speech in subjective hearing.

[0079] 3. Maintaining high fidelity of noise reduction network accuracy under extreme computing power reduction: Through creative fusion of current time domain features and historical frequency domain base, the feature distribution of odd-numbered frames fed into the neural network does not show rigid stagnation or phase breakage. Under the condition of reducing the front-end computing power by nearly half, it only causes a loss of noise reduction depth below the threshold of human hearing perception (about 0.3 dB).

[0080] 4. Limiting throughput leap based on instruction set-level parallelism: By combining SIMD instruction set optimization in the outbound phase, the system completely vectorizes the most time-consuming floating-point to fixed-point operation, and eliminates memory processing blocking by combining double loop unrolling technology, which greatly improves the single-core computing power throughput of the system, while significantly reducing the global processing latency of a single frame. Attached Figure Description

[0081] Figure 1 This is the overall program logic flowchart of the embodiment of this application.

[0082] Figure 2 This is a flowchart of the multi-level energy gating logic in an embodiment of this application. Detailed Implementation

[0083] The present invention will be further described in detail below with reference to the accompanying drawings.

[0084] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they fall within the scope of the claims of the present invention.

[0085] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0086] The overall approach of this invention abandons the fine-tuning of deep learning model weights, and instead focuses on reconstructing the time-series feature scheduling layer, energy state machine control layer, and hardware instruction execution layer. The overall technical solution comprises three core steps:

[0087] The staggered-frame fundamental tone spectrum multiplexing steps are as follows: The continuous time-series input frames are divided into even-numbered frames and odd-numbered frames. In even-numbered frames, a full frequency domain transformation and fundamental tone analysis are performed, and their core features are written into a delay buffer register. In odd-numbered frames, the massive frequency domain transformation is actively bypassed. Cross-frame correlation mathematical compensation is performed using the extremely low-cost time-domain features extracted from the current frame and the historical frequency-domain features in the delay buffer register, fusing them to generate a smooth feature vector.

[0088] A multi-level energy gating process with hidden state protection: Before the feature vector is fed into the recurrent neural network, a multi-level energy detector is deployed. When the mean square energy is detected to fall below a preset extreme background noise threshold, the network inference is not immediately cut off, but a "silence hold" countdown counter of a set duration is triggered. Before this counter reaches zero, the RNN is forcibly kept running hot to protect the context weights; only after the countdown reaches zero is the data flow completely blocked to achieve zero load.

[0089] The CPU instruction set-based vectorized adaptation output steps are as follows: After completing noise reduction inference and inverse frequency domain transformation, the host CPU's hardware architecture capabilities are detected, specific SIMD vectorized instructions are called, and a large number of audio format conversions are completed through register-level parallelism.

[0090] The following is an explanation of the technical terms involved in this invention:

[0091] RNN (Recurrent Neural Network): A neural network architecture for processing time series data, possessing short-term memory capabilities.

[0092] GRU (Gated Recurrent Unit): A variant of RNN, it alleviates the gradient vanishing problem by introducing update gates and reset gates, and is the core network structure for performing noise reduction inference in this case.

[0093] SIMD (Single Instruction, Multiple Data): A low-level hardware instruction set architecture (such as AVX2, NEON) that allows the same operation to be performed on multiple data in parallel within a single clock cycle.

[0094] Based on the following Figure 1 and Figure 2 The present invention will be described in detail below.

[0095] Step S1: Divide the continuously input audio data frames into even frames and odd frames according to the time sequence. The first valid audio data frame after the system cold start is forcibly marked as an even frame.

[0096] Specifically, when consecutive audio data frames (e.g., each frame is 10 milliseconds long and the frame length N is 480 sampling points) enter the system, the scheduler assigns an even / odd frame status identifier.

[0097] For system cold starts, the scheduler has an internal state lock. The system forces the first valid audio data frame acquired to be marked as an even frame (i.e., index number n=0), thus ensuring that a full calculation is performed during the cold start phase, thereby ensuring that a valid initial historical frequency domain base is written into the delay buffer register. If the first valid audio data frame acquired after a system cold start is an odd frame (e.g., due to sequence fragmentation caused by packet loss and subsequent resynchronization), the system will trigger a safety mechanism. By default, the delay buffer register is initialized with extremely low-energy white noise parameters, and the frame is forced to undergo a full frequency domain transformation and pitch analysis according to the even frame processing procedure. This ensures that a valid initial historical frequency domain base exists in the delay buffer register for subsequent odd-numbered frames to use for cross-frame correlation mathematical compensation.

[0098] If the current frame is an even frame, the system performs a full forward fast Fourier transform on the input signal after windowing, calculates the complex spectrum including phase and amplitude, extracts the energy of each frequency band, and performs a global search to obtain the pitch index, and extracts the time-domain sampled signal of the even frame. Simultaneously, the system calculates the mean square energy of the current even frame. Before the end of the current even frame, the system continuously writes the complex spectrum, frequency band energy, pitch index, and mean square energy into a persistent delay buffer register.

[0099] The calculation process for the mean square energy of the current even-numbered frames is as follows:

[0100] Obtain the frame length, index number, and signal of the current even-numbered frame. Determine the mean square energy of the current even-numbered frame based on its frame length, index number, and time-domain sampled signal. The calculation formula is as follows:

[0101]

[0102] in, The mean square energy of the current even-numbered frame. The frame length of the current even-numbered frame. The current even-numbered frame index number. This is the time-domain sampled signal for the current even-numbered frame.

[0103] When the next odd-numbered frame arrives (i.e., when the current frame is an odd-numbered frame), the system instructs, in order to reconstruct high-fidelity frequency domain features without performing a high-overhead forward fast Fourier transform, to directly read the historical complex spectrum, historical pitch index, historical mean square energy, and historical time-domain sampled signal of the previous valid even-numbered frame from the delay buffer register, and obtain the mean square energy and time-domain sampled signal of the odd-numbered frame. Based on the time-domain sampled signal of the odd-numbered frame and the historical complex spectrum, cross-frame correlation mathematical compensation is performed, and the complex spectrum of the odd-numbered frame is generated by fusing them. The complex spectrum of the odd-numbered frame is mapped onto the Buck band, the frequency band energy of the odd-numbered frame is calculated, and the historical pitch index is reused as the pitch index of the current odd-numbered frame.

[0104] Among them, cross-frame correlation mathematical compensation includes:

[0105] The mean square energy of odd-numbered frames is extracted as the transient reference for amplitude compensation;

[0106] Perform an inverse fast Fourier transform on the historical complex spectrum to reconstruct the historical time-domain sampled signal, calculate the cross-frame normalized cross-correlation function between the time-domain sampled signal of odd frames and the historical time-domain sampled signal, and determine the optimal time offset that maximizes the cross-frame normalized cross-correlation function through local search.

[0107] Based on the time-shifting characteristics of Fourier transform, a phase rotation factor with the corresponding optimal time offset is applied to the historical complex spectrum, and amplitude correction is performed using the ratio of the mean square energy of odd frames to the historical mean square energy to synthesize the complex spectrum of odd frames.

[0108] The synthesized complex spectrum is mapped onto the Barker band, and the energy of each band is calculated.

[0109] The detailed explanation of cross-frame correlation mathematical compensation is as follows:

[0110] 1. Extracting the transient energy of the current odd-numbered frame: The system does not need to convert to the frequency domain. It directly extracts the frame length, index number, and time-domain sampled signal of the current odd-numbered frame in the time domain. Based on the frame length, index number, and time-domain sampled signal of the current odd-numbered frame, the mean square energy of the current odd-numbered frame is determined as the transient reference for amplitude compensation. The calculation formula is as follows:

[0111]

[0112] in, The mean square energy of the current odd-numbered frames. The frame length of the current odd-numbered frame. The current odd-numbered frame index number. This is the time-domain sampled signal of the current odd-numbered frame.

[0113] 2. Cross-frame correlation offset calculation (phase alignment): Extract the cross-frame normalized cross-correlation function between the current odd-numbered frame's temporal sampled signal and the historical temporal sampled signal to find the continuity offset. The formula for calculating the cross-frame normalized cross-correlation function is as follows:

[0114]

[0115] in, For cross-frame normalized cross-correlation function, This is the time offset. This is a historical time-domain sampled signal.

[0116] By performing a local search within a very small time window, the method can be found to make... Optimal time offset for taking the maximum value This optimal time offset represents the acoustic phase slip of the current odd-numbered frame relative to historical frames.

[0117] 3. Mathematical synthesis of frequency domain base: Based on the time-shifting characteristics of Fourier transform (time-domain delay is equivalent to frequency-domain phase rotation), the system directly applies a rotation factor with the corresponding optimal time offset to the historical complex spectrum in the complex domain, and at the same time uses the time-domain energy ratio for amplitude correction to synthesize the complex spectrum of the current odd frame.

[0118] The time-domain ratio is:

[0119] The formula for calculating the complex spectrum of the current odd-numbered frames is:

[0120]

[0121] in, This is the ratio of the mean square energy of odd-numbered frames to the historical mean square energy. The mean square energy of odd-numbered frames. For historical mean square energy, For the complex spectrum of odd-numbered frames, For the historical complex spectrum, For discrete frequency points in the frequency domain, The imaginary unit, , This is the optimal time offset.

[0122] 4. Reconstructing the frequency band energy mapping: Mapping the synthesized complex spectrum of the current odd-numbered frames onto coarse-grained Barker bands, and calculating the energy of each band. The calculation formula is as follows:

[0123]

[0124] in, For the frequency band energy of odd-numbered frames, For frequency band indexing, This refers to the ending frequency index, which represents the position of the ending boundary frequency point corresponding to the b-th coarse-grained frequency band on the discrete frequency spectrum axis. This is the starting frequency index, which is the position of the starting boundary frequency point corresponding to the b-th coarse-grained frequency band on the discrete spectrum axis.

[0125] Through the above four-step mathematical synthesis, the system reconstructs a high-fidelity frequency domain dynamic input vector with only a very small amount of time-domain correlation and complex rotation operations, which not only prevents phase breakage but also fundamentally reduces the amount of pre-calculation per frame by nearly half.

[0126] Step S2: Construct a feature vector based on the frequency band energy, mean square energy, and pitch index of the current frame. Then, after the mean square energy of the current frame is determined by multi-level energy gating, it is fed into a recurrent neural network for gain inference to generate a frequency domain gain mask.

[0127] Specifically, a feature vector is constructed based on the frequency band energy, mean square energy, and pitch index of the current frame. After the feature vector of the current frame is generated, the system enters the multi-level energy gating judgment stage, which includes L1-level gating, L2-level gating, and L3-level gating.

[0128] The system first obtains the mean square energy of the current frame (odd or even frame) through L1 gating (as described above), and determines whether the mean square energy of the current frame is lower than the preset extreme background noise threshold (such as -70dBFS).

[0129] If the mean square energy of the current frame is not lower than the preset extreme background noise threshold, it is determined that there is a valid sound input, and the mute hold countdown counter is immediately reset to the full scale value (e.g., 20 frames, corresponding to 200 milliseconds of material physical time), and then enters L2 level gating;

[0130] If the mean square energy of the current frame is lower than the preset extreme background noise threshold, the instantaneous sleep command is absolutely blocked, and the value of the mute hold countdown counter is checked. If the value is greater than zero, the value of the mute hold countdown counter is decremented by one, and the L3 level gating mute hold determination is activated. If the value is not greater than zero, the data stream is cut off.

[0131] The system then performs L2-level gating to check whether the frequency band energy of the current frame has speech harmonic characteristics;

[0132] If so, the feature vector of the current frame is fed into the recurrent neural network;

[0133] If not, the feature vector of the current frame is fed into the recurrent neural network, but the frequency domain gain mask output is suppressed, and the context memory weight update of the hidden state of the network is maintained.

[0134] The detailed technical implications of L3 gating are as follows: Under normal circumstances, pure silent frames are discarded. However, after activating L3 gating, the system forcibly establishes a signal transmission channel through a bypass mechanism, forcing the downstream recurrent neural network to treat the current weak background trailing feature as normal speech for forward inference. This operation physically simulates an engine's "idling warm-up" state, continuously using extremely weak ambient sounds to stimulate the cellular state units within the GRU network, thereby completely preventing its context memory matrix from collapsing and losing memory due to sudden interruption of flow. Only when the silence countdown counter decrements to 0 does L1 gating truly cut off the data flow and enter silent sleep mode.

[0135] Step S3: Apply the frequency domain gain mask to the complex spectrum of the current frame and reconstruct the time domain signal waveform through inverse fast Fourier transform.

[0136] Feature vectors that are gated for entry or forcibly activated by L3-level gating formally enter the core reconstruction flow:

[0137] First, a recurrent network (using a gated recurrent unit architecture) gain inference operation is performed: the system calls the underlying GRU model architecture to multiply and add the input feature vector with the hidden state matrix retained from the previous time step. The temporal coherence of the speech is analyzed using the update and reset gates within the recurrent network, and finally, a set of frequency domain gain masks with values ​​between 0.0 and 1.0 is output through the sigmoid activation function. This frequency domain gain mask defines the proportion of speech components preserved in each specific frequency band of the current frame.

[0138] Then, the inverse fast Fourier transform reconstruction operation is performed: after obtaining the frequency domain gain mask, the system multiplies it by the complex spectrum of the current frame to achieve frequency domain filtering. Subsequently, the inverse fast Fourier transform is used to deconstruct and restore the filtered frequency domain signal back to the time domain sampling point sequence. Then, the time domain sampling point sequence of the current frame is overlapped and added with the time domain sampling point sequence of the adjacent frames according to the preset frame overlap rate to generate a continuous denoised digital audio waveform.

[0139] Step S4: In the output stage, based on the hardware architecture capabilities of the host CPU, call the Single Instruction Multiple Data Stream Vectorization instruction to convert the time-domain signal waveform from floating-point data format to fixed-point data format in batches.

[0140] Specifically, perform single instruction stream multiple data stream optimization operations:

[0141] The system sniffs the host CPU's hardware architecture capabilities (such as the AVX2 instruction set for x86_64 architecture or the NEON instruction set for ARM architecture), identifies the supported vectorized instruction set types, obtains the identification results, and calls the corresponding wide hardware registers (such as the 256-bit YMM register) based on the identification results. The system can directly truncate, limit, and convert 8 or more F32 samples into S16 format at once within a single CPU instruction. Combined with a double loop unrolling strategy, it achieves high-speed data flushing of the memory bus and completes end-to-end ultra-low latency output.

[0142] It's important to know that the reason for implementing single instruction stream multiple data stream optimization is as follows:

[0143] Interleaved frames (i.e., alternating even and odd frames) and multi-level energy gating (i.e., L1-level gating, L2-level gating, and L3-level gating) form the algorithmic foundation for the noise reduction efficiency and sound quality of this invention. However, the time-domain waveform obtained from deep learning calculations is necessarily a 32-bit floating-point number (F32). In real computer operating systems and physical sound cards, if it cannot be quickly converted to the device standard 16-bit signed integer (S16), this conversion bottleneck will consume all the time saved by the front-end algorithm, resulting in still high system-level latency.

[0144] The present invention also provides an audio noise reduction system based on time series signal processing, the system comprising:

[0145] The frame division module is used to divide continuously input audio data frames into even frames and odd frames according to the time sequence. The first valid audio data frame after the system cold start is forcibly marked as an even frame.

[0146] The even-numbered frame processing module is used to perform full frequency domain transformation and pitch analysis on the input signal after windowing when the current frame is an even-numbered frame, extract the complex spectrum, frequency band energy, pitch index and mean square energy, and write the complex spectrum, frequency band energy, pitch index and mean square energy into the delay buffer register.

[0147] The odd-numbered frame processing module is used to bypass full frequency domain transformation and pitch analysis when the current frame is an odd-numbered frame. It directly reads the historical complex spectrum, historical pitch index, historical mean square energy, and historical time-domain sampled signal of the previous valid even-numbered frame from the delay buffer register, and obtains the mean square energy and time-domain sampled signal of the odd-numbered frame. Based on the time-domain sampled signal of the odd-numbered frame and the historical complex spectrum, it performs cross-frame correlation mathematical compensation, fuses and generates the complex spectrum of the odd-numbered frame, maps the complex spectrum of the odd-numbered frame to the Buck band, calculates the frequency band energy of the odd-numbered frame, and reuses the historical pitch index as the pitch index of the current odd-numbered frame.

[0148] The gated inference module is used to input the mean square energy of the current frame into a recurrent neural network for gain inference after passing through multi-level energy gating judgment, and generate a frequency domain gain mask.

[0149] The inverse transform reconstruction module is used to apply the frequency domain gain mask to the complex spectrum of the current frame and reconstruct the time domain signal waveform through inverse fast Fourier transform.

[0150] The vectorized output module is used to call single instruction multiple data stream vectorization instructions according to the hardware architecture capabilities of the host central processing unit during the output stage, and to convert time domain signal waveforms from floating-point data format to fixed-point data format in batches.

[0151] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. An audio noise reduction method based on time series signal processing, characterized in that, include: The continuously input audio data frames are divided into even frames and odd frames according to the time sequence. The first valid audio data frame after the system cold start is forcibly marked as an even frame. When the current frame is an even frame, the input signal is windowed and then subjected to full frequency domain transformation and pitch analysis to extract the complex spectrum, band energy, pitch index, mean square energy and time domain sampled signal, and the complex spectrum, band energy, pitch index and mean square energy are written into the delay buffer register. When the current frame is an odd frame, bypass the full frequency domain transformation and pitch analysis, directly read the historical complex spectrum, historical pitch index, historical mean square energy, and historical time-domain sampled signal of the previous valid even frame from the delay buffer register, and obtain the mean square energy and time-domain sampled signal of the odd frame. Based on the time-domain sampled signal of the odd frame and the historical complex spectrum, perform cross-frame correlation mathematical compensation, fuse and generate the complex spectrum of the odd frame, map the complex spectrum of the odd frame onto the Buck band, calculate the frequency band energy of the odd frame, and reuse the historical pitch index as the pitch index of the current odd frame. A feature vector is constructed based on the frequency band energy, mean square energy, and pitch index of the current frame. The mean square energy of the current frame is then processed by a multi-level energy gating system and fed into a recurrent neural network for gain inference, generating a frequency domain gain mask. Apply the frequency domain gain mask to the complex spectrum of the current frame, and reconstruct the time domain signal waveform through inverse fast Fourier transform; During the output phase, based on the hardware architecture capabilities of the host CPU, single instruction multiple data stream vectorization instructions are invoked to convert time-domain signal waveforms from floating-point data format to fixed-point data format in batches.

2. The audio noise reduction method based on time series signal processing according to claim 1, characterized in that, The method for calculating the mean square energy of the even-numbered frames includes: Obtain the frame length, time-domain sampled signal, and index number of even-numbered frames; The mean square energy of even-numbered frames is determined based on the frame length, time-domain sampled signal, and index number. The method for calculating the mean square energy of the odd-numbered frames includes: Obtain the frame length, time-domain sampled signal, and index number of odd-numbered frames; The mean square energy of odd-numbered frames is determined based on the frame length, time-domain sampled signal, and index number. The formula for calculating the mean square energy of even-numbered frames is the same as that for odd-numbered frames, both being: ， in, Mean square energy, For frame length, For index number, This is a time-domain sampled signal.

3. The audio noise reduction method based on time series signal processing according to claim 2, characterized in that, The cross-frame correlation mathematical compensation includes: The mean square energy of odd-numbered frames is extracted as the transient reference for amplitude compensation; Perform an inverse fast Fourier transform on the historical complex spectrum to reconstruct the historical time-domain sampled signal, calculate the cross-frame normalized cross-correlation function between the time-domain sampled signal of odd frames and the historical time-domain sampled signal, and determine the optimal time offset that maximizes the cross-frame normalized cross-correlation function through local search. Based on the time-shifting characteristics of Fourier transform, a phase rotation factor with the corresponding optimal time offset is applied to the historical complex spectrum, and amplitude correction is performed using the ratio of the mean square energy of odd frames to the historical mean square energy to synthesize the complex spectrum of odd frames. The synthesized complex spectrum is mapped onto the Barker band, and the energy of each band is calculated.

4. The audio noise reduction method based on time series signal processing according to claim 3, characterized in that, The formula for calculating the cross-frame normalized cross-correlation function is as follows: ， in, For cross-frame normalized cross-correlation function, This is the time offset. This is a historical time-domain sampled signal; The formula for calculating the complex spectrum of the odd-numbered frames is as follows: ， ， in, This is the ratio of the mean square energy of odd-numbered frames to the historical mean square energy. The mean square energy of odd-numbered frames. For historical mean square energy, For the complex spectrum of odd-numbered frames, For the historical complex spectrum, For discrete frequency points in the frequency domain, The imaginary unit, , This is the maximum time offset; The formula for calculating the frequency band energy of the odd-numbered frames is as follows: ， in, For the frequency band energy of odd-numbered frames, For frequency band indexing, This refers to the ending frequency index, which represents the position of the ending boundary frequency point of the b-th coarse-grained frequency band on the discrete frequency spectrum axis. This is the starting frequency index, which is the position of the starting boundary frequency point corresponding to the b-th coarse-grained frequency band on the discrete spectrum axis.

5. The audio noise reduction method based on time series signal processing according to claim 1, characterized in that, The multi-level energy gating includes L1-level gating, L2-level gating, and L3-level gating, wherein: The criteria for L1 gating include: Obtain the mean square energy of the current frame; Determine whether the mean square energy of the current frame is lower than a preset extreme background noise threshold; If so, proceed to the L3 level gated mute maintenance judgment; If not, it is determined that there is a valid sound input, the mute countdown counter is reset to its full-scale value, and the L2 gating judgment is entered; wherein, the full-scale value of the mute countdown counter is configured as the number of audio frames corresponding to the preset duration; The criteria for L2 gating include: Determine whether the frequency band energy of the current frame has speech harmonic characteristics; If so, the feature vector of the current frame is fed into the recurrent neural network; If not, the feature vector of the current frame is fed into the recurrent neural network, but the frequency domain gain mask output is suppressed, and the context memory weight update of the hidden state of the network is maintained.

6. The audio noise reduction method based on time series signal processing according to claim 5, characterized in that, The determination of maintaining silence includes: When the L1-level gating determines that the mean square energy of the current frame is lower than the preset extreme background noise threshold, it blocks the transmission of the feature vector of the current frame to the recurrent neural network. Check the current value of the mute hold countdown counter; If the current value is greater than zero, the current value of the mute countdown counter will be decremented by one, and the feature vector of the current frame will be forcibly fed into the recurrent neural network for gain inference in order to maintain the context memory weights of the hidden state matrix inside the recurrent neural network. If the current value is zero, the transmission of the feature vector of the current frame to the recurrent neural network is blocked and the system enters a silent sleep state.

7. The audio noise reduction method based on time series signal processing according to claim 1, characterized in that, The recurrent neural network employs a gated recurrent unit architecture, and the gain inference includes: The input feature vector is multiplied and added to the hidden state matrix retained from the previous time step. The temporal coherence of speech is analyzed by utilizing the update gate and reset gate within the gated loop unit architecture; The Sigmoid activation function outputs a frequency domain gain mask with values ​​between 0.0 and 1.0, which defines the proportion of speech components preserved in each specific frequency band of the current frame.

8. The audio noise reduction method based on time series signal processing according to claim 1, characterized in that, The reconstruction of the time-domain signal waveform via inverse fast Fourier transform includes: Frequency domain filtering is achieved by dot-multiplying the frequency domain gain mask onto the complex spectrum of the current frame. The filtered complex spectrum is restored to a time-domain sampling point sequence using the inverse fast Fourier transform; The temporal sampling point sequence of the current frame is overlapped and added with the temporal sampling point sequence of the adjacent frame according to a preset frame overlap rate to generate a continuous denoised digital audio waveform.

9. The audio noise reduction method based on time series signal processing according to claim 1, characterized in that, The invocation of single-instruction multiple-data vectorized instructions based on the hardware architecture capabilities of the host central processing unit specifically includes: It can sniff out the hardware architecture of the host central processing unit and identify the supported vectorized instruction set types, including the AVX2 instruction set or SSE2 instruction set for the x86_64 architecture and the NEON instruction set for the ARM architecture. Based on the recognition results, the corresponding wide hardware register is called, and multiple floating-point samples are truncated, limited, and converted into fixed-point data format in a single central processing unit instruction. A double-loop unrolling strategy is adopted to achieve high-speed data flushing of the memory bus.

10. An audio noise reduction system based on time series signal processing, characterized in that, include: The frame division module is used to divide continuously input audio data frames into even frames and odd frames according to the time sequence. The first valid audio data frame after the system cold start is forcibly marked as an even frame. The even-numbered frame processing module is used to perform full frequency domain transformation and pitch analysis on the input signal after windowing when the current frame is an even-numbered frame. It extracts the complex spectrum, frequency band energy, pitch index, mean square energy and time domain sampled signal, and writes the complex spectrum, frequency band energy, pitch index and mean square energy into the delay buffer register. The odd-numbered frame processing module is used to bypass full frequency domain transformation and pitch analysis when the current frame is an odd-numbered frame. It directly reads the historical complex spectrum, historical pitch index, historical mean square energy, and historical time-domain sampled signal of the previous valid even-numbered frame from the delay buffer register, and obtains the mean square energy and time-domain sampled signal of the odd-numbered frame. Based on the time-domain sampled signal of the odd-numbered frame and the historical complex spectrum, it performs cross-frame correlation mathematical compensation, fuses and generates the complex spectrum of the odd-numbered frame, maps the complex spectrum of the odd-numbered frame to the Buck band, calculates the frequency band energy of the odd-numbered frame, and reuses the historical pitch index as the pitch index of the current odd-numbered frame. The gated inference module constructs a feature vector based on the frequency band energy, mean square energy, and pitch index of the current frame. This vector is used to input the mean square energy of the current frame into a recurrent neural network for gain inference after passing through multi-level energy gating, and to generate a frequency domain gain mask. The inverse transform reconstruction module is used to apply the frequency domain gain mask to the complex spectrum of the current frame and reconstruct the time domain signal waveform through inverse fast Fourier transform. The vectorized output module is used to call single instruction multiple data stream vectorization instructions according to the hardware architecture capabilities of the host central processing unit during the output stage, and to convert time domain signal waveforms from floating-point data format to fixed-point data format in batches.

Images