Front-end audio processing method
By constructing a multi-stage audio processing pipeline based on WebAssembly in the browser environment, the insufficient suppression capability and latency of the browser's audio processing module in complex acoustic environments are solved, achieving high-quality real-time audio processing and improving speech clarity and communication stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-29
- Publication Date
- 2026-04-03
AI Technical Summary
The browser's default audio processing module has limited ability to suppress noise in complex acoustic environments, and cannot effectively handle environmental noise and echo, resulting in decreased speech clarity and increased processing latency, making it difficult to meet the high-quality requirements of real-time audio and video communication.
It employs a multi-stage audio processing pipeline built on WebAssembly, including high-pass filtering, deep learning noise reduction, echo cancellation, and post-processing enhancement, combined with real-time parameter adjustment, to achieve real-time audio signal processing in a browser environment through WebAudioAPI and WebRTC.
It significantly improves voice clarity and intelligibility, reduces interference from environmental noise and echo, ensures the stability and low latency of audio and video communication, is suitable for various terminals and scenarios, and is easy to deploy and highly compatible.
Smart Images
Figure CN121789705A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a front-end audio processing method, belonging to the field of audio data processing technology. Background Technology
[0002] With the widespread adoption of remote work, online education, live video streaming, online customer service, and real-time audio and video interaction on web browsers, the browser environment has gradually become the primary platform for audio and video communication. Standardized communication frameworks such as WebRTC have driven the rapid growth of browser-based video conferencing. However, at the same time, the inadequacy of front-end audio processing capabilities has gradually become a key factor limiting user experience and industry development.
[0003] In typical real-time call scenarios, the raw microphone audio captured by the browser and the system audio during desktop / window sharing are often interfered with by complex acoustic environments. Because the browser's default audio processing module is a general-purpose implementation, it cannot perform in-depth optimization for dynamic sound fields, echo path changes, and high-noise environments in actual use. This directly results in the front-end audio quality failing to meet the requirements of high-fidelity communication. The currently prevalent technical problems mainly fall into the following two categories:
[0004] The browser's default audio processing module performs poorly in complex acoustic environments. It has weak suppression capabilities against common environmental interferences such as keyboard typing, mouse clicks, air conditioning noise, office background noise, and traffic noise. In scenarios involving speaker playback, near-field reflections, or "screen sharing with system sound," the acoustic coupling between the speaker and microphone often creates strong echoes or even feedback, resulting in severely noisy and echoing sound received by distant users, and a significant decrease in speech clarity and naturalness.
[0005] Existing audio processing methods relying on pure JavaScript, whether employing traditional digital signal processing algorithms or loading deep learning-based noise reduction models, are constrained by script execution efficiency, browser scheduling mechanisms, and single-threaded limitations. The long loading times of deep models and the significant time required for real-time inference per frame significantly increase end-to-end processing latency, easily leading to issues such as intermittent audio, stuttering, and audio-visual desynchronization, making it difficult to meet the extremely low latency and stability requirements of real-time conferencing. Summary of the Invention
[0006] The purpose of this invention is to overcome the shortcomings of the existing technology. The existing audio acquisition and processing mechanisms have limited ability to suppress environmental noise, background noise and non-speech interference in complex acoustic environments. At the same time, due to the limitations of script execution performance, it is difficult to stably control the processing delay during real-time processing, which can easily lead to problems such as audio stuttering, voice interruption and audio-visual asynchrony.
[0007] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution:
[0008] A front-end audio processing method is provided, including the following steps:
[0009] Acquire microphone audio signals;
[0010] Obtain the pre-defined audio processing pipeline built on WebAssembly;
[0011] The microphone audio signal is input into the audio processing pipeline with a fixed frame length, and the audio processing pipeline performs audio processing on the microphone audio signal through a preset module.
[0012] The system acquires environmental and processing status parameters in the audio processing pipeline in real time, and adjusts the microphone audio signal in the audio processing pipeline based on these acquired parameters.
[0013] Furthermore, the preset module includes: a pre-processing module, a deep learning noise reduction module, an echo cancellation module, and a post-processing enhancement module;
[0014] The audio processing pipeline performs audio processing on the microphone audio signal through preset modules, including: preprocessing the audio signal, deep learning noise reduction processing, echo cancellation processing, and post-processing enhancement.
[0015] Furthermore, the preprocessing includes high-pass filtering;
[0016] The high-pass filtering process is implemented using a differential method:
[0017] y[n] = x[n] - x[n - 1]+αy[m - 1] ;
[0018] Where n is the index of the time sampling point, m is the index of the time sampling point, m = n − d, and d is the fixed delay. x[n] For the input signal at time point The sampled values, y[n] For filtered output, x[n - 1] , y[m - 1] The output value is the historical time position. These are the filter coefficients.
[0019] Furthermore, the deep learning noise reduction process includes feature extraction;
[0020] The feature extraction includes: performing a Fast Fourier Transform on the audio signal and calculating the logarithmic amplitude spectrum, the calculation method being as follows:
[0021] );
[0022] in, It is a frequency index. This indicates the first frequency domain of the input audio. Complex spectral values at each frequency point To prevent tiny constants with values of zero, It is a logarithmic amplitude spectrum.
[0023] Furthermore, the deep learning noise reduction process also includes: a neural network model and frequency domain reconstruction, wherein the neural network model includes gated recurrent units and fully connected layers; the extracted features are processed through fully connected layers and gated recurrent unit layers to obtain the gain coefficient for each frequency point;
[0024] The neural network inference includes outputting the gain coefficient for each frequency point based on the neural network model. And perform suppression treatment:
[0025] ;
[0026] in, It is a frequency index. The output of the neural network is the first Gain coefficient at each frequency point This indicates the first frequency domain of the input audio. Complex spectral values at each frequency point Spectral amplitude after noise suppression;
[0027] The frequency domain reconstruction includes: reconstructing the time domain waveform from the original phase of the frequency points after neural network inference using inverse fast Fourier transform.
[0028] Furthermore, the echo cancellation process includes: acquiring a speaker reference signal, constructing an adaptive filtering model based on the speaker reference signal, and subtracting the speaker reference signal from the microphone audio signal based on the acquired speaker reference signal; the acquired speaker reference signal satisfies:
[0029] ;
[0030] in, For speaker signals, This is an index for the echo path coefficients. For time sampling point index, The length of the adaptive filter, The first signal output by the speaker One sampling point, For the true echo signal at the 1st The value of each sampling point, The first in the acoustic echo path One coefficient;
[0031] The subtraction from the microphone audio signal based on the acquired speaker reference signal is achieved by the following formula:
[0032] ;
[0033] in, For time sampling point index, The nth sampling point of the microphone input signal. The echo is predicted by the adaptive filter. This is the output after echo cancellation.
[0034] Furthermore, the step of constructing the adaptive filtering model based on the loudspeaker reference signal includes updating the model using a normalized least mean square algorithm.
[0035] ;
[0036] in, For time sampling point index, For the number of iterations, δ is the step size factor, and δ is the stability factor. The first signal output by the speaker One sampling point, For speaker signal energy, , For the adaptive filter in the th The coefficient vector before and after the iteration update.
[0037] Furthermore, the post-processing enhancements include residual noise suppression, automatic gain control, and limiting processing.
[0038] Furthermore, the audio processing pipeline also includes a real-time parameter adjustment module.
[0039] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:
[0040] 1. This invention constructs a unified front-end audio processing flow, enabling the acquired audio signals to complete noise suppression, echo cancellation, and audio enhancement processing in a pure front-end environment, effectively reducing the interference of environmental noise and echo on the speech signal, thereby significantly improving the speech clarity and intelligibility in real-time audio and video communication; at the same time, it organizes the audio processing flow in stages and uniformly schedules the processing process.
[0041] 2. This invention dynamically adjusts the audio processing status and environmental changes to maintain a stable and consistent audio processing effect, avoiding significant fluctuations in processing quality due to environmental changes. In addition, this invention is applicable to browser operating environments and can achieve high-quality audio processing without relying on native plugins or local clients. It has the advantages of convenient deployment and strong compatibility, making it easy to promote and use in various terminals and application scenarios. Attached Figure Description
[0042] Figure 1 The diagram shown is a flowchart of a front-end audio processing method provided in an embodiment of the present invention. Detailed Implementation
[0043] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations thereof. In the absence of conflict, the embodiments and technical features in the embodiments can be combined with each other.
[0044] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.
[0045] The front-end audio processing method described in this invention runs in a browser environment, specifically in modern browsers that support WebAudioAPI, WebAssembly (WASM), and WebRTC.
[0046] like Figure 1 As shown, this embodiment provides a front-end audio processing method, including the following steps:
[0047] Acquire microphone audio signals;
[0048] Obtain the pre-defined audio processing pipeline built on WebAssembly;
[0049] The microphone audio signal is input into the audio processing pipeline with a fixed frame length, and the audio processing pipeline performs audio processing on the microphone audio signal through a preset module.
[0050] The system acquires environmental and processing status parameters in the audio processing pipeline in real time, and adjusts the microphone audio signal in the audio processing pipeline based on these acquired parameters.
[0051] In this embodiment, the audio signal is acquired from the microphone device via the WebAudio API provided by the browser. The acquired audio is typically in Linear Pulse Code Modulation (PCM) format. To meet real-time processing requirements, the acquired continuous audio stream is divided into fixed-length audio frames; the fixed frame length refers to maintaining a consistent number of audio samples sent to subsequent processing pipelines each time. Before entering the processing pipeline, each audio frame undergoes timestamp alignment and format unification processing to ensure data consistency between subsequent processing modules.
[0052] In this embodiment, audio frames are input into a multi-stage audio processing pipeline built on WebAssembly. The multi-stage audio processing pipeline includes at least the following processing stages: a pre-processing stage, a model loading and acceleration module, a deep learning noise reduction stage, an echo cancellation stage, a post-processing enhancement stage, and a real-time parameter adjustment module.
[0053] The audio processing pipeline performs audio processing on the microphone audio signal through preset modules, including: preprocessing the audio signal, deep learning noise reduction processing, echo cancellation processing, and post-processing enhancement.
[0054] Pre-processing is used to perform basic cleaning and normalization on the raw audio signal, including high-pass filtering.
[0055] The high-pass filtering process is implemented using a differential method:
[0056] y[n] = x[n] - x[n - 1]+αy[m - 1] ;
[0057] Where n is the index of the time sampling point, m is the index of the time sampling point, m = n − d, and d is a fixed delay. x[n] For the input signal at time point The sampled values, y[n] For filtered output, x[n - 1] , y[m - 1] The output value is the historical time position. These are the filter coefficients.
[0058] In real-world environments, audio signals captured by microphones often contain low-frequency noise. Therefore, in this embodiment, a high-pass filter is used in the preprocessing stage to suppress low-frequency components. The high-pass filter employs a first- or second-order digital filter structure, the core idea of which is to attenuate low-frequency signals through a linear combination of the current sample value, historical sample value, and historical output value.
[0059] In this embodiment, the preprocessing further includes speech activity detection, which can be based on at least one of the following features: short-time energy of the audio frame, zero-crossing rate, frequency domain spectral features, or spectral entropy. It should be noted that the speech activity detection result does not directly change the audio content, but rather serves as a control signal to adjust the processing intensity or parameter update speed of subsequent deep learning noise reduction and echo cancellation stages.
[0060] In the preprocessing stage, since the audio amplitude values collected by different users and different devices vary greatly, this embodiment performs energy normalization processing on the audio frames to ensure that the audio signal amplitude falls within a preset range.
[0061] In this embodiment, the audio processing pipeline further includes deep learning noise reduction processing, which is used to suppress non-speech components such as background noise and environmental noise.
[0062] The audio frame is first converted to the frequency domain, specifically by obtaining the complex spectrum through Fast Fourier Transform (FFT); then the logarithm of the spectrum amplitude is taken to obtain the logarithmic amplitude spectrum feature.
[0063] Specifically, the deep learning noise reduction process includes feature extraction;
[0064] The feature extraction includes: performing a Fast Fourier Transform on the audio signal and calculating the logarithmic amplitude spectrum, the calculation method being as follows:
[0065] );
[0066] in, It is a frequency index. This indicates the first frequency domain of the input audio. Complex spectral values at each frequency point To prevent tiny constants with values of zero, Logarithmic amplitude spectrum.
[0067] The deep learning noise reduction process employs a lightweight neural network model, which includes a fully connected layer and a gated recurrent unit (GRU) structure. The extracted features are processed through the fully connected layer and the GRU layer to obtain the gain coefficient for each frequency point. The model has been quantized and optimized before deployment and compiled into the WebAssembly executable format.
[0068] The neural network model then outputs the gain coefficient corresponding to each frequency point, which is used to suppress the input spectrum point by point; and retains the original phase information, and reconstructs the time-domain audio signal through inverse fast Fourier transform (IFFT).
[0069] Specifically, the neural network inference includes outputting the gain coefficient for each frequency point based on the neural network model. And perform suppression treatment:
[0070] ;
[0071] in, It is a frequency index. The output of the neural network is the first Gain coefficient at each frequency point This indicates the first frequency domain of the input audio. Complex spectral values at each frequency point Spectral amplitude after noise suppression;
[0072] The frequency domain reconstruction includes: reconstructing the time domain waveform from the original phase of the frequency points after neural network inference using inverse fast Fourier transform.
[0073] In scenarios where audio is played through speakers or screen-sharing systems, the speaker sound may be picked up again by the microphone, creating an echo.
[0074] To address this issue, this embodiment introduces an echo cancellation process.
[0075] An adaptive filter is constructed to model the acoustic echo path, and the echo components are predicted using a loudspeaker reference signal.
[0076] The filter parameters are updated using the Normalized Least Mean Square (NLMS) algorithm. The algorithm normalizes the error signal to ensure that the filter can still converge stably under different signal energy conditions. The predicted echo signal is subtracted from the microphone input signal to obtain the echo-cancelled audio output.
[0077] Specifically, the echo cancellation process includes: acquiring a speaker reference signal, constructing an adaptive filtering model based on the speaker reference signal, and subtracting it from the microphone audio signal based on the acquired speaker reference signal; the acquired speaker reference signal satisfies:
[0078] ;
[0079] in, For speaker signals, This is an index for the echo path coefficients. For time sampling point index, The length of the adaptive filter, The first signal output by the speaker One sampling point, For the true echo signal at the 1st The value of each sampling point, The first in the acoustic echo path One coefficient;
[0080] The subtraction from the microphone audio signal based on the acquired speaker reference signal is achieved by the following formula:
[0081] ;
[0082] in, For time sampling point index, The nth sampling point of the microphone input signal. The echo is predicted by the adaptive filter. This is the output after echo cancellation.
[0083] The step of constructing the adaptive filtering model based on the loudspeaker reference signal includes updating it using a normalized minimum mean square algorithm.
[0084] ;
[0085] in, For time sampling point index, For the number of iterations, δ is the step size factor, and δ is the stability factor. The first signal output by the speaker One sampling point, For speaker signal energy, , For the adaptive filter in the th The coefficient vector before and after the iteration update.
[0086] To further improve the output audio quality, this embodiment introduces a post-processing enhancement stage after echo cancellation; specifically, it includes residual noise suppression, automatic gain control, and limiting processing.
[0087] The residual noise suppression addresses low-energy noise that has not been completely suppressed by supplementing it through noise thresholding or secondary attenuation. The automatic gain control automatically adjusts the overall gain based on the short-term energy of the output audio to ensure consistent voice volume for different users and at different distances. The amplitude limiting process limits the amplitude of the audio signal to prevent signal overflow or distortion caused by gain superposition or splicing.
[0088] In this embodiment, the system continuously monitors the voice activity status, ambient noise intensity, echo path changes, and processing delay, and dynamically adjusts the parameters of each processing stage based on the monitoring results.
[0089] The audio processing pipeline also includes a real-time parameter adjustment module; based on real-time acquired environmental parameters and processing status parameters, the processing parameters in the multi-stage audio processing pipeline are adaptively adjusted.
[0090] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A front-end audio processing method, characterized in that: Includes the following steps: Acquire microphone audio signals; Obtain the pre-defined audio processing pipeline built on WebAssembly; The microphone audio signal is input into the audio processing pipeline with a fixed frame length, and the audio processing pipeline performs audio processing on the microphone audio signal through a preset module. The system acquires environmental and processing status parameters in the audio processing pipeline in real time, and adjusts the microphone audio signal in the audio processing pipeline based on these acquired parameters.
2. The front-end audio processing method according to claim 1, characterized in that, The preset modules include: a pre-processing module, a deep learning noise reduction module, an echo cancellation module, and a post-processing enhancement module; The audio processing pipeline performs audio processing on the microphone audio signal through preset modules, including: preprocessing the audio signal, deep learning noise reduction processing, echo cancellation processing, and post-processing enhancement.
3. The front-end audio processing method according to claim 2, characterized in that, The preprocessing includes high-pass filtering; The high-pass filtering process is implemented using a differential method: ； Where n is the index of the time sampling point, m is the index of the time sampling point, m = n − d, and d is a fixed delay. For the input signal at time point The sampled values, For filtered output, , The output value is the historical time position. These are the filter coefficients.
4. The front-end audio processing method according to claim 2, characterized in that, The deep learning noise reduction process includes feature extraction; The feature extraction includes: performing a Fast Fourier Transform on the audio signal and calculating the logarithmic amplitude spectrum, the calculation method being as follows: )； in, It is a frequency index. This indicates the first frequency domain of the input audio. Complex spectral values at each frequency point To prevent tiny constants with values of zero, Logarithmic amplitude spectrum.
5. The front-end audio processing method according to claim 4, characterized in that, The deep learning noise reduction process also includes: a neural network model and frequency domain reconstruction, wherein the neural network model includes gated recurrent units and fully connected layers; the extracted features are processed through fully connected layers and gated recurrent unit layers to obtain the gain coefficient of each frequency point; The neural network inference includes outputting the gain coefficient for each frequency point based on the neural network model. And perform suppression treatment: ； in, It is a frequency index. The output of the neural network is the first Gain coefficient at each frequency point This indicates the first frequency domain of the input audio. Complex spectral values at each frequency point Spectral amplitude after noise suppression; The frequency domain reconstruction includes: reconstructing the time domain waveform from the original phase of the frequency points after neural network inference using inverse fast Fourier transform.
6. The front-end audio processing method according to claim 2, characterized in that, The echo cancellation process includes: acquiring a speaker reference signal, constructing an adaptive filtering model based on the speaker reference signal, and subtracting the acquired speaker reference signal from the microphone audio signal; the acquired speaker reference signal satisfies: ； in, For speaker signals, This is an index for the echo path coefficients. For time sampling point index, The length of the adaptive filter, The first signal output by the speaker One sampling point, For the true echo signal at the 1st The value of each sampling point, The first in the acoustic echo path One coefficient; The subtraction from the microphone audio signal based on the acquired speaker reference signal is achieved by the following formula: ； in, For time sampling point index, The nth sampling point of the microphone input signal. The echo is predicted by the adaptive filter. This is the output after echo cancellation.
7. The front-end audio processing method according to claim 6, characterized in that, The step of constructing the adaptive filtering model based on the loudspeaker reference signal includes updating it using a normalized minimum mean square algorithm. ； in, For time sampling point index, For the number of iterations, δ is the step size factor, and δ is the stability factor. The first signal output by the speaker One sampling point, For speaker signal energy, , For the adaptive filter in the th The coefficient vector before and after the next iteration update.
8. The front-end audio processing method according to claim 2, characterized in that, The post-processing enhancements include residual noise suppression, automatic gain control, and amplitude limiting.
9. A front-end audio processing method according to claim 2, characterized in that, The audio processing pipeline also includes a real-time parameter adjustment module.