Voice processing collaboration method based on dual-cuda stream

Abstract

This invention relates to a collaborative speech processing method based on dual CUDA streams, belonging to the field of speech processing technology. The method includes: creating a first CUDA computation stream and a second CUDA computation stream on a GPU; the first CUDA computation stream is used for acoustic echo cancellation and speech activity detection, and the second CUDA computation stream is used for speaker separation; receiving near-end microphone audio data packets and far-end reference audio data packets through a WebSocket access layer; a signal alignment module performing timestamp verification, queue buffering, and frame-level pairing output on the near-end microphone audio data packets and far-end reference audio data packets to form aligned frame data; performing acoustic echo cancellation frame-by-frame on the aligned frame data in the first CUDA computation stream and generating echo-cancelled audio, while simultaneously generating speech activity detection results; this invention achieves hardware-level parallel execution of the dual computation streams and on-demand dynamic allocation of computing resources, significantly reducing end-to-end processing latency and improving the efficiency of graphics processor resource utilization.

Description

Technical Field

[0001] This disclosure relates to, but is not limited to, the field of speech processing technology, and in particular to a collaborative speech processing method based on dual CUDA streams. Background Technology

[0002] Acoustic echo cancellation technology aims to remove echo components formed by the playback of a far-end reference signal through a loudspeaker and re-acquisition by the microphone from near-end microphone signals. Traditional acoustic echo cancellation methods are based on adaptive filtering theory, estimating the impulse response of the echo path using the least mean square algorithm or recursive least squares algorithm, and then predicting and eliminating the echo. However, adaptive filtering methods can only handle linear echo components and are difficult to effectively eliminate harmonic components generated by nonlinear distortion of loudspeakers and the long-tail effect of room reverberation. In recent years, acoustic echo cancellation methods based on deep neural networks have gradually become a research hotspot. These methods learn the complex mapping relationship between the near-end signal and the far-end reference signal, and can simultaneously suppress linear echoes and nonlinear residues, achieving a significant improvement in echo cancellation performance.

[0003] Speech activity detection technology is used to determine whether the current audio frame contains valid human voice components. Accurate speech activity detection results are crucial for subsequent speech recognition, speech synthesis interruption, and bandwidth-adaptive transmission. Traditional speech activity detection methods rely on manually designed features such as short-time energy, zero-crossing rate, and spectral entropy. While they achieve good detection results in high signal-to-noise ratio (SNR) environments, their performance degrades sharply in low SNR or complex acoustic environments with non-stationary noise. Deep learning-based speech activity detection methods learn robust acoustic feature representations through end-to-end training, significantly enhancing their adaptability to noisy environments and have been widely deployed in industry. Speaker separation technology addresses scenarios where multiple speakers are speaking simultaneously by decomposing the mixed speech signal into independent speech signals for each speaker. This technology plays a key role in applications such as conference transcription, cocktail party problem solving, and multi-person dialogue speech recognition. Early speaker separation methods were based on computational auditory scene analysis theory, utilizing features such as pitch trajectories and spatial cues to achieve sound source separation. However, these methods are sensitive to acoustic conditions and have high computational complexity. The era of deep learning has given rise to a variety of high-performance speaker separation architectures, including deep clustering, permutation-invariant training, and temporal convolutional separation networks. These methods have achieved performance levels close to human auditory separation capabilities on standard evaluation sets. Summary of the Invention

[0004] This disclosure provides a collaborative speech processing method based on dual CUDA streams, which realizes hardware-level parallel execution of dual computational streams and on-demand dynamic allocation of computing resources, significantly reducing end-to-end processing latency, improving the efficiency of graphics processor resource utilization, and enhancing the robustness and response speed of real-time voice interaction systems under complex acoustic and network conditions.

[0005] To solve the above problems, the technical solution of the present invention is implemented as follows: A collaborative speech processing method based on dual CUDA streams includes: creating a first CUDA computation stream and a second CUDA computation stream on the GPU; the first CUDA computation stream is used for acoustic echo cancellation and speech activity detection, and the second CUDA computation stream is used for speaker separation; receiving near-end microphone audio data packets and far-end reference audio data packets through a WebSocket access layer; a signal alignment module performs timestamp verification, queue buffering, and frame-level pairing output on the near-end microphone audio data packets and far-end reference audio data packets to form aligned frame data; performing acoustic echo cancellation on the aligned frame data frame by frame in the first CUDA computation stream and generating echo-cancelled audio, while simultaneously generating speech activity detection results; receiving the speech activity detection results in the second CUDA computation stream and triggering conditions, performing speaker separation inference on the buffered audio and generating separated audio only when the speech activity detection results meet the trigger conditions; outputting a two-stage response through a Future asynchronous object; the first stage outputs a first-stage result containing the speech activity state, echo-cancelled audio, and interruption flag after the first CUDA computation stream completes speech activity detection; the second stage outputs the separated audio as the second-stage result after the second CUDA computation stream completes speaker separation.

[0006] Furthermore, the first CUDA computation stream and the second CUDA computation stream respectively perform CUDA graph capture to generate graph execution instances on a fixed operator sequence. The graph execution instances include time-frequency feature extraction, mask inference, spectral mask processing, speech activity detection smoothing, and speaker separation inference. A graph execution instance cache pool is maintained in the session dimension, and the matching graph execution instance is selected for replay based on the input frame length, sliding window length, and GPU load to maintain end-to-end latency stability when the audio frame length changes.

[0007] Furthermore, a circular buffer partitioned by session is established in the GPU memory. The circular buffer stores aligned frame data, speech activity detection results output by the first CUDA compute stream, and timestamp records. After writing to the circular buffer, the first CUDA compute stream writes a monotonically increasing sequence number and records CUDA events. After the condition is triggered, the second CUDA compute stream waits for CUDA events and reads the corresponding segments according to the sequence number. After reading, it writes back the consumption cursor to complete lock-free synchronization across CUDA streams. When a discontinuous sequence number is detected, silence padding or window reconstruction is performed to avoid speaker separation inference input breaks.

[0008] Furthermore, a trigger gating controller is constructed. Within a sliding time window, the trigger gating controller collects false trigger samples of the voice activity detection results and performs distribution-independent confidence calibration to generate an adaptive trigger threshold that meets the target false trigger rate constraint. The network arrival interval, the latency accumulation index output by the signal alignment module, and the queuing latency index of the first and second CUDA computation streams are used as context features and input into the context router. The context router selects a trigger strategy from the candidate trigger strategy set. The candidate trigger strategy includes at least the trigger threshold, the speaker separation sliding window length, and the duration of continuous execution after triggering, which is used to adaptively schedule the second CUDA computation stream between silent periods, single-speaking periods, and double-speaking periods.

[0009] Furthermore, a directed acyclic graph architecture is adopted to manage the processing pipeline. The signal alignment module acts as a broadcast node to simultaneously distribute the alignment frame data to the first processing pipeline and the second processing pipeline. The first processing pipeline is used to perform acoustic echo cancellation and speaker separation when the far-end reference audio data packet exists. The second processing pipeline is used to skip acoustic echo cancellation and perform speaker separation when the far-end reference audio data packet does not exist.

[0010] Furthermore, the signal alignment module includes: performing timestamp extraction and sampling rate conversion on the near-end microphone audio data packets and the far-end reference audio data packets respectively, and writing them into the near-end FIFO queue and the far-end FIFO queue; performing timestamp difference verification on the header data of the near-end FIFO queue and the far-end FIFO queue and performing pairing output; determining network packet loss when the timestamp difference is greater than the data duration, performing mute padding on the missing segments, and then pairing output; determining processing delay accumulation when the actual processing time exceeds the data push time and discarding expired frames to restore real-time performance; and resetting the alignment state when the timestamp rolls back or the timestamp returns to zero.

[0011] Furthermore, the first CUDA computation flow performs acoustic echo cancellation and speech activity detection by: performing bandpass filtering preprocessing on the aligned frame data; generating in parallel near-end Fbank features, linear acoustic echo cancellation Fbank features, and echo estimation Fbank features; performing three-channel feature concatenation on the near-end Fbank features, linear acoustic echo cancellation Fbank features, and echo estimation Fbank features, and then inputting them into the deep feedforward sequence memory network (DFSMN) mask model after mean and variance normalization. The deep feedforward sequence memory network (DFSMN) mask model simultaneously outputs the spectral mask and speech activity detection probability in a single forward inference.

[0012] Furthermore, the first CUDA computation stream performs a Sigmoid mapping on the spectral mask to obtain the mask result, and performs a Sigmoid mapping on the speech activity detection probability to obtain the speech activity detection probability result; the near-end microphone audio in the aligned frame data is transformed by a short-time Fourier transform to obtain a time-frequency representation, the time-frequency representation and the mask result are multiplied element-wise and then transformed by an inverse short-time Fourier transform to generate echo-cancelled audio; the speech activity detection probability result is smoothed by an exponential moving average to form a speech activity score, and an interruption flag is generated when the speech activity score and the endpoint logic satisfy the real-time interruption condition.

[0013] Furthermore, the speaker separation in the second CUDA computation stream includes: writing the near-end microphone audio output by the signal alignment module into a history queue to form a sliding window buffer; when the speech activity score exceeds the trigger threshold, extracting the window audio from the sliding window buffer and submitting it to the thread pool for execution, the thread pool is used to execute speaker separation inference tasks and CPU-intensive preprocessing tasks, the thread pool tasks perform end-to-end speaker separation inference in the second CUDA computation stream and output multi-channel separated audio; calculating the sum of squares of samples as an energy index for the multi-channel separated audio in the same analysis window and selecting the channel with the largest energy index as the target output channel, writing the target output channel into a Future asynchronous object to generate the second-stage result; skipping speaker separation inference to release GPU resources when the speech activity score does not exceed the trigger threshold, and maintaining continuous triggering in the dual-talk state where the speech activity score is continuously higher than the trigger threshold to improve the separation update frequency.

[0014] Furthermore, an independent session state copy is maintained for each WebSocket connection. The session state copy stores the historical queue, voice activity score smoothing status, near-first-out queue and far-first-out queue used by the signal alignment module, and timestamp records for latency calculation. The session state copy is managed using a weak reference keyed dictionary with the WebSocket connection object as the key, so that the session state copy is automatically reclaimed by the garbage collection mechanism after the WebSocket connection is closed. A dual event loop thread model is adopted, with the main thread event loop executing WebSocket sending and receiving, business logic processing, and GPU inference task scheduling, and the background thread event loop executing cross-module data transfer and queue forwarding. The main thread event loop and the background thread event loop communicate through thread-safe queues.

[0015] The mobile phone file compression system of the present invention has the following beneficial effects: The present invention constructs a dual CUDA computing stream parallel architecture on the graphics processor, assigns the acoustic echo cancellation and voice activity detection tasks to the first CUDA computing stream for execution, and assigns the speaker separation task to the second CUDA computing stream for execution. The two computing streams achieve true parallel scheduling at the hardware level, breaking through the latency accumulation bottleneck of the traditional serial processing architecture, significantly reducing the end-to-end processing latency in real-time voice interaction scenarios, enabling users to obtain near-instantaneous system response when initiating voice interruption, and greatly improving the fluency and naturalness of human-computer dialogue.

[0016] The condition-triggered execution strategy adopted in this invention enables the second CUDA computation stream to start speaker separation inference only when the speech activity detection result meets the triggering condition. During silent periods and pure noise periods, execution is automatically skipped to release graphics processor resources. This realizes on-demand dynamic allocation of computing resources, effectively avoiding the waste of resources caused by performing high-computation inference on invalid audio segments. While ensuring the quality of speech separation, it significantly reduces the average power consumption and heat output of the graphics processor and extends the service life of hardware devices.

[0017] The two-stage asynchronous response mechanism designed in this invention achieves phased output of results through Future asynchronous objects. In the first stage, after acoustic echo cancellation and speech activity detection are completed, the speech activity status, echo cancellation audio, and interruption flag are returned immediately for downstream applications to perform real-time interruption detection and interface response. In the second stage, after speaker separation is completed, high-quality separated audio is returned asynchronously for speech recognition. This decoupled design allows the time-latency-sensitive interruption detection stage to not wait for the time-consuming speaker separation task to complete, achieving a balance between response speed and processing quality.

[0018] The signal alignment mechanism of this invention effectively solves the problem of time offset and jitter caused by differences in network transmission paths between near-end microphone audio and far-end reference audio through timestamp verification, queue buffering and frame-level pairing output. Combined with packet loss detection and silence filling strategies, it ensures that the acoustic echo cancellation algorithm can always obtain time-aligned input data, and can maintain stable echo cancellation performance even in complex deployment environments with fluctuating network conditions. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the response curve for packet loss detection and processing delay compensation provided in an embodiment of the present invention; Figure 2 A schematic diagram illustrating the time-dimensional alignment mechanism between the near-end microphone audio signal and the far-end reference audio signal provided in an embodiment of the present invention. Figure 3 The diagram shows the multi-channel output waveform of the second CUDA computation stream when performing a speaker separation task, as well as the target channel selection logic based on energy maximization, provided in the embodiments of the present invention. Detailed Implementation

[0020] To make the technical problems, technical solutions, and beneficial effects to be solved by this disclosure clearer and more understandable, the disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this disclosure and are not intended to limit this disclosure.

[0021] A collaborative speech processing method based on dual CUDA streams includes: creating a first CUDA computation stream and a second CUDA computation stream on the GPU; the first CUDA computation stream is used for acoustic echo cancellation and speech activity detection, and the second CUDA computation stream is used for speaker separation; receiving near-end microphone audio data packets and far-end reference audio data packets through a WebSocket access layer; a signal alignment module performs timestamp verification, queue buffering, and frame-level pairing output on the near-end microphone audio data packets and far-end reference audio data packets to form aligned frame data; performing acoustic echo cancellation on the aligned frame data frame by frame in the first CUDA computation stream and generating echo-cancelled audio, while simultaneously generating speech activity detection results; receiving the speech activity detection results in the second CUDA computation stream and triggering conditions, performing speaker separation inference on the buffered audio and generating separated audio only when the speech activity detection results meet the trigger conditions; outputting a two-stage response through a Future asynchronous object; the first stage outputs a first-stage result containing the speech activity state, echo-cancelled audio, and interruption flag after the first CUDA computation stream completes speech activity detection; the second stage outputs the separated audio as the second-stage result after the second CUDA computation stream completes speaker separation.

[0022] Real-time voice interaction scenarios place extremely stringent requirements on processing latency. The end-to-end latency perceived by users during a conversation typically needs to be kept below 150 milliseconds; otherwise, noticeable stuttering will occur. Under this constraint, if the three core processing tasks—acoustic echo cancellation, speech activity detection, and speaker separation—are executed sequentially using traditional methods, latency accumulation will inevitably occur, making it difficult to meet real-time requirements. To address this issue, this invention constructs a dual-CUDA compute stream parallel architecture on a graphics processor, significantly reducing processing latency through task-level parallelism and pipelined overlapping execution.

[0023] When creating a CUDA compute stream on a graphics processor, the underlying runtime interface needs to be called to instantiate the stream object. The first CUDA compute stream handles acoustic echo cancellation and speech activity detection tasks. These two tasks are strongly coupled; the output of acoustic echo cancellation directly serves as the input for speech activity detection. Placing them in the same compute stream avoids the overhead of cross-stream synchronization. The second CUDA compute stream is dedicated to speaker separation, a computationally complex task that typically requires processing hundreds of milliseconds of audio segments in a single inference iteration, resulting in a relatively long execution time. Separating the speaker separation task into the second CUDA compute stream allows it to form a pipelined parallel relationship with the first CUDA compute stream. While the first CUDA compute stream processes the current frame data, the second CUDA compute stream can simultaneously perform speaker separation inference on the accumulated data of previous frames, thus fully utilizing computational resources.

[0024] In the specific implementation, the system initialization phase first completes the selection of the graphics processing unit (GPU) device and the creation of the context. For servers equipped with multiple GPUs, the optimal device can be selected based on the device's computing power, memory capacity, and current load status. In one optional implementation, the device selection strategy prioritizes GPUs with a computing power level of at least 7.0, as these devices support Tensor Core acceleration, which can significantly improve the performance of deep neural network inference. After completing the device selection, the stream creation interface is called to instantiate the first and second CUDA computing streams respectively. To ensure that the two computing streams can truly achieve hardware-level parallel execution, a non-blocking flag needs to be specified during creation, allowing kernel function calls in different computing streams to be independently allocated to different stream multiprocessors by the GPU scheduler for execution.

[0025] The first CUDA compute stream is set to high priority, and the second CUDA compute stream is set to normal priority. This differentiated priority configuration is based on the following considerations: Acoustic echo cancellation and speech activity detection tasks are extremely sensitive to latency, and their processing results directly affect the response speed of interruption detection. After a user initiates a speech interruption, it needs to be recognized by the system within a very short time. Therefore, the first CUDA compute stream must obtain priority in hardware resource scheduling. Although the speaker separation task is crucial to the final audio quality, its results are mainly used in the subsequent speech recognition stage, and the requirement for immediate response is relatively relaxed. Setting its priority to normal level will not have a significant impact on the user experience. In an optional implementation, the priority values ​​adopt a relative encoding method, with the priority value of the first CUDA compute stream set to -1 and the priority value of the second CUDA compute stream set to 0. The smaller the value, the higher the priority.

[0026] Audio data access is achieved through full-duplex communication via the WebSocket protocol. The WebSocket access layer maintains a persistent connection with the client, continuously receiving near-end microphone audio data packets and far-end reference audio data packets. The near-end microphone audio data packets originate from the microphone of the user's terminal device and include the user's voice, ambient noise, and echo components generated by the content played from the speaker in the indoor space. The far-end reference audio data packets originate from the synthesized speech played by the system to the user, typically generated by a text-to-speech engine. This signal serves as the reference input for echo cancellation processing, used to estimate and eliminate echo components in the near-end signal.

[0027] Each audio data packet uses a uniform encapsulation format, containing audio sample data, timestamp information, and a sequence number. The audio sample data is a raw waveform sample in pulse code modulation format, with a sampling precision of a 16-bit signed integer. The sampling rate can be configured to 16000 Hz or 8000 Hz depending on the application scenario. The timestamp information records the acquisition or generation time of the data packet, represented by a 64-bit unsigned integer in milliseconds, with the time base being the cumulative number of milliseconds since 00:00 UTC on January 1, 1970. The sequence number is a 32-bit unsigned integer, monotonically increasing within each session, used to detect data packet loss and out-of-order delivery.

[0028] In a typical configuration, each audio data packet carries 20 milliseconds of audio sample data. At a sampling rate of 16000 Hz, 20 milliseconds corresponds to 320 sample points, each occupying 2 bytes, resulting in a single data packet's audio payload of 640 bytes. The choice of 20 milliseconds as the data packet length is based on a trade-off between network transmission efficiency and processing latency: shorter packet lengths reduce single-frame processing latency but increase network protocol overhead and exacerbate small packet transmission problems; longer packet lengths improve transmission efficiency but introduce additional buffering latency. A 20-millisecond frame length has become a de facto standard in the field of speech codecs, aligning with the frame lengths of mainstream speech encoders and facilitating system integration.

[0029] Upon receiving an audio data packet, the WebSocket access layer first performs an integrity check at the protocol layer to verify that the packet length and format meet expectations. If the check passes, it distinguishes between near-end microphone audio packets and far-end reference audio packets based on the type identifier in the packet header, and forwards them to their respective processing queues. In high-concurrency scenarios, the WebSocket access layer employs an asynchronous, non-blocking, event-driven model, allowing a single access layer instance to maintain thousands of concurrent connections, each corresponding to an independent session context.

[0030] Signal alignment is a prerequisite for collaborative processing of dual CUDA compute streams. Near-end microphone audio packets and far-end reference audio packets arrive at the server via different network paths, resulting in uncertain time offsets and jitter. If the unaligned near-end signal and far-end reference signal are directly fed into acoustic echo cancellation processing, the echo cancellation algorithm will be unable to correctly estimate the echo path delay parameters, leading to severe degradation or even complete failure of the echo cancellation effect. Therefore, time alignment of the near-end signal and far-end reference signal must be completed before acoustic echo cancellation processing.

[0031] The first step in signal alignment is timestamp extraction and normalization. The timestamps carried by the near-end microphone audio data packets and the far-end reference audio data packets may be based on different time bases; for example, the near-end timestamp might be based on the terminal device's local system clock, while the far-end timestamp might be based on the server's system clock. To eliminate the impact of time base differences, both timestamps need to be uniformly converted to the server's time base. The conversion process utilizes the clock offset obtained from the clock synchronization protocol executed during the WebSocket connection establishment phase. Let the original timestamp of the near-end data packet be... The clock offset is The normalized near-end timestamp Calculate using the following formula: ;in This represents the original timestamp value carried in the audio data packet from the near-end microphone, in milliseconds. This represents the offset between the terminal device clock and the server clock. This value is calculated during the connection establishment phase by measuring round-trip time delay. The unit is milliseconds. A positive value indicates that the terminal device clock is ahead of the server clock, and a negative value indicates that it is behind. This represents the near-end timestamp converted to the server's time base, in milliseconds. The far-end reference audio data packet is generated by the server, and its timestamp... It is based directly on the server clock, without the need for time base conversion.

[0032] refer to Figure 2 The graph is divided into two parts. The upper curve reflects the change in the difference between the near-end and far-end timestamps, while the lower curve reflects the corresponding change in the alignment buffer depth. The horizontal axis uniformly represents the sequence number of the data frame, covering the entire process from normal transmission to network anomaly. In real-time voice interaction systems, the audio data packets collected by the near-end microphone and the reference audio data packets generated by the far-end text-to-speech engine arrive at the server via different network paths. Due to the uncertainty of network transmission, there is often a non-linear deviation in their arrival times. The solid black line in the upper curve depicts the actual timestamp difference. The fluctuations. The calculation formula is ,in The timestamp representing the header frame of the near-first-in-first-out queue. This represents the timestamp of the header frame of the remote FIFO queue. To assess alignment quality, two red dashed lines are used in the diagram as tolerance threshold boundaries, corresponding to the positive tolerance threshold. and negative tolerance threshold Typical value Milliseconds. When the actual timestamp difference When the signal is within the band formed by the two red dashed lines, the system determines that the near-end and far-end signals are substantially aligned in time. At this point, the signal alignment module directly pops the header frames of the near-end and far-end queues, combines them into aligned frame data, and outputs it to the first CUDA compute stream. This processing method can absorb minor network jitter and avoid interruptions in the audio stream due to over-adjustment. The black solid lines in most areas of the diagram fluctuate within the tolerance range, indicating that the system can maintain stable frame-level pairing under good network conditions.

[0033] However, sudden changes in the network environment can cause drastic fluctuations in timestamp differences. In the figure, at sequence numbers approximately... The location illustrates a typical network packet loss or congestion event. At this point, the actual timestamp difference... A sudden and significant increase occurred, exceeding the positive tolerance threshold. This abnormal period is marked by a red-filled area in the graph, corresponding to the logic branch that triggers either silent filling or a wait. When detected... A significant increase indicates a lag or absence of the distant reference signal. According to the formula... The system calculates the increment of the sequence number. If a data packet is found to be discontinuous, then... The number of silent frames that need to be filled is calculated, and silent frames with all-zero data are inserted during the missing time periods to maintain the continuity of the timeline and ensure that the input buffer of the subsequent acoustic echo cancellation algorithm does not underflow. The lower curve shows the dynamic change of the alignment buffer depth, which is highly correlated with the timestamp difference curve in the upper part. The blue solid line represents the number of pending data frames currently backed up in the buffer. In the normal pairing region, the buffer depth is maintained at a low level, for example... Around 100 frames, this corresponds to approximately Millisecond-level jitter buffer capacity. When the aforementioned packet loss or congestion events occur, some data frames cannot be paired immediately and must wait in the queue for subsequent data to arrive, causing the buffer depth to increase rapidly. The maximum latency threshold is marked by an orange dashed line in the figure. Once the buffer depth curve exceeds this threshold, it indicates that the accumulated latency has exceeded the acceptable range for real-time interaction. At this point, the system triggers a discarding policy, actively discarding expired frames at the head of the queue, forcibly lowering the buffer level, and enabling the system to quickly return to a real-time state.

[0034] Furthermore, the right side of the diagram illustrates the phenomenon of accumulated clock drift. Over time, if there is a slight frequency deviation between the terminal device's clock and the server's clock, the actual timestamp difference will increase. It will exhibit a slow, linear drift trend. As the drift gradually accumulates and approaches the tolerance threshold, the system corrects this deviation through fine-tuning the resampling rate or periodic alignment state resets. Specifically, the system monitors... If the long-term mean of the clock is found to deviate significantly from zero, the clock offset is recalculated. And update the timestamp normalization formula. The parameters in the algorithm are adjusted to pull the black solid line back to near the zero axis. Through this multi-dimensional detection and compensation mechanism, this scheme can ensure the timing synchronization of dual CUDA stream inputs in complex network environments.

[0035] In one alternative implementation, the clock offset is measured using a simplified variant of the Network Time Protocol. During the connection establishment phase, the server sends a time request message to the terminal device and records the sending time. Upon receiving the request, the terminal device immediately returns a response message along with its local time. The server records the time of receipt after receiving the response. Assuming symmetrical uplink and downlink delays on the network, the clock offset is estimated using the following formula: ;in Indicates the time when the server sent the time request message. This indicates the moment when the terminal device receives the request and sends a response. This indicates the time when the server received the response message, with all three times referenced to the local clock of their respective devices. To improve estimation accuracy, multiple round-trip measurements can be performed, and the median can be used as the final offset estimate.

[0036] After timestamp normalization, sampling rate conversion needs to be performed on the near-end signal and the far-end reference signal to ensure they have a consistent sampling rate. In some applications, the sampling rate of the near-end microphone may be 48000 Hz, while the output sampling rate of the far-end text-to-speech engine is 16000 Hz, a difference of 3 times. Without unification, subsequent frame-level pairing will not align correctly. Sampling rate conversion is implemented using a polyphase filter, a method that maintains anti-aliasing performance while having low computational complexity. Let the original sampling rate be... The target sampling rate is The resampling ratio for: ;in The original sampling rate of the input signal is expressed in Hertz. This represents the target sampling rate, expressed in Hertz. For the dimensionless resampling ratio, when Downsampling is performed at that time. Upsampling is performed during the process. This invention unifies the sampling rate of the near-end signal and the far-end reference signal to 16000 Hz, which can cover the effective frequency band of the speech signal, while controlling the computational load of subsequent processing.

[0037] After sampling rate conversion, the near-end signal and the far-end reference signal are written into separate first-in-first-out (FIFO) queues for buffering. The near-end FIFO queue buffers audio frames from the microphone, while the far-end FIFO queue buffers reference frames from the text-to-speech engine. The queue capacity needs to be designed to take into account network jitter and memory usage. In one optional implementation, each FIFO queue is set to hold 500 milliseconds of audio data, corresponding to 25 standard frames. This capacity can absorb transmission jitter under common network conditions while keeping buffer memory usage within a reasonable range. When the queue reaches its capacity limit, the oldest data is discarded to make room for newly arrived data, ensuring the system's real-time performance.

[0038] Frame-level pairing output is the core step in signal alignment. The pairing process periodically reads data from the heads of the near-end FIFO queue and the far-end FIFO queue, determines their time correspondence based on timestamp information, and generates time-aligned frame data pairs. The pairing logic follows these rules: First, calculate the timestamp of the head frame of the near-end queue. Timestamp of the remote queue header frame The difference between : ;in This represents the timestamp of the header frame of the near-first-in-first-out queue. This represents the timestamp of the header frame of the remote first-in-first-out queue. This represents the time difference between two header frames, in milliseconds. A positive value indicates that the near-end frame was acquired later than the far-end frame was generated, while a negative value indicates that the near-end frame was acquired earlier than the far-end frame was generated.

[0039] when Less than or equal to the frame duration tolerance threshold If two header frames are considered to be time-aligned, the header frames from the near-end queue and the far-end queue are popped simultaneously, combined to form a pair of aligned frame data, and output to subsequent processing stages. Frame duration tolerance threshold. A typical value is 10 milliseconds, which is half the duration of a standard frame. This tolerance can absorb small time offsets introduced by network transmission. ;in Represents the absolute value of the time difference. This represents the frame duration tolerance threshold, in milliseconds.

[0040] when When this occurs, it indicates that the timestamp of the near-end frame is significantly later than that of the far-end frame. This means that there are earlier frames in the far-end queue that have not yet been matched by the near-end frames. In this case, only the head frame of the far-end queue is popped and discarded, while the near-end queue remains unchanged, and then the matching process is repeated. This situation typically occurs when the far-end signal arrives at the server before the near-end signal, such as when the synthesis speed of a text-to-speech engine is faster than the actual playback speed, resulting in the premature accumulation of far-end signals.

[0041] when When the timestamp of the near-end frame is significantly earlier than that of the far-end frame, it indicates that an earlier frame in the near-end queue cannot find a corresponding far-end reference frame. At this point, it's necessary to determine whether the far-end signal is lost or has not yet arrived. If the far-end FIFO queue is empty, it may indicate a significant transmission delay in the far-end signal; in this case, the pairing operation should be suspended until the far-end signal arrives. If the far-end FIFO queue is not empty but the timestamp of the header frame is still significantly later than that of the near-end frame, it is determined that the far-end signal is missing within the corresponding time period. The header frame of the near-end queue should be popped, and a frame of all-zero silent data should be generated as the far-end reference frame for pairing and output. Using a silent frame to replace the missing far-end reference signal ensures the integrity of the input format for the acoustic echo cancellation algorithm. Although echo cancellation is not possible at this time, at least the processing flow will not be interrupted due to missing input.

[0042] During network transmission, data packets may be lost due to congestion, routing changes, or link failures. Packet loss detection is achieved by determining the continuity of sequence numbers. Let the sequence number of the currently received data packet be... The sequence number of the last successfully received data packet is The difference between the two is Reflects a continuous state: ;in Indicates the sequence number of the current data packet. This indicates the sequence number of the previous data packet. Indicates the sequence number increment. When When this occurs, it means that data packets arrive continuously without any packet loss. When, it indicates existence. If a data packet is lost, silence padding needs to be performed on the lost time period to maintain the continuity of the timeline. The number of frames for silence padding. Calculate using the following formula: ;in This indicates the number of silent frames that need to be filled. All sample values ​​for each silent frame are set to zero, and the frame length is consistent with that of a normal data frame. Timestamps are interpolated and filled according to the expected timestamps of the lost frames to ensure the continuity of the timeline.

[0043] refer to Figure 1This graph, with time on the horizontal axis and processing delay on the vertical axis, comprehensively demonstrates the dynamic response characteristics and compensation mechanism of the signal alignment module under network transmission anomalies. The horizontal axis represents system runtime in milliseconds, ranging from 0 to 500 milliseconds, with increments of 50 milliseconds. The vertical axis represents the cumulative processing delay of the audio frame from acquisition to alignment output, also in milliseconds, ranging from 0 to 60 milliseconds, with increments of 10 milliseconds. The graph includes three key reference lines: the actual processing delay curve is plotted as a solid line, showing the dynamic change in delay throughout the observation period; the delay tolerance threshold is marked with a red dashed line, corresponding to a value of [value missing]. Milliseconds represent the maximum allowable cumulative latency of the system; the baseline latency is marked with a green dotted line, corresponding to a value of approximately 20 milliseconds, representing the steady-state processing latency under ideal network conditions.

[0044] Within the 500-millisecond observation period shown in the figure, the system experienced three network packet loss events, occurring at approximately 60 milliseconds, 160 milliseconds, and 300 milliseconds, respectively. These are marked with vertical orange-red lines in the figure and labeled "Packet Loss 1," "Packet Loss 2," and "Packet Loss 3." After each packet loss event, a gray-filled area with a width of 20 milliseconds appears at the corresponding time position, representing the time period during which the system performs a silent filling operation. Silent filling is the core compensation strategy for network packet loss in this invention. When a discontinuous data packet sequence is detected, the system automatically generates an all-zero silent frame to fill the missing time period, ensuring that the continuity of the timeline is not disrupted. The packet loss detection mechanism is based on the continuity judgment of sequence numbers. Let the sequence number of the currently received data packet be... The sequence number of the last successfully received data packet is Then the sequence number increment According to the formula Calculation. When When the data packets arrive continuously, no compensation operation is required; when Time indicates existence If a data packet is lost, the number of silence frames that need to be filled in will be [number missing]. equal The formula box marked in the figure clearly illustrates this detection and compensation logic.

[0045] The actual latency curve shows that the system maintains a stable latency of approximately 20 milliseconds under normal operating conditions, with slight random fluctuations of about ±3 milliseconds, reflecting the natural variations in network jitter and processing time. After the first packet loss event, the processing latency experiences a step increase of about 5 milliseconds. This is because the silent padding operation itself consumes processing time, and the data discontinuity caused by the packet loss requires additional state recovery overhead. In the following 50 to 100 milliseconds, the latency curve shows a slow downward trend, eventually returning to the baseline latency level, demonstrating the system's self-recovery capability. The second and third packet loss events triggered similar latency response patterns, but due to the cumulative effect of multiple packet losses, the latency peak gradually increased. After all three packet loss events occurred, the peak latency curve approached 35 milliseconds, but remained within the latency tolerance threshold. The following demonstrates that the compensation mechanism of the present invention can effectively control the accumulation of delays and prevent the system from entering an unrecoverable overload state.

[0046] When cumulative delay Exceeding the delay tolerance threshold When this happens, the system will activate the expired frame discarding mechanism to restore real-time performance. Let the current system time be... The timestamp of the frame to be processed is Then the cumulative delay According to the formula Calculate. If The system continuously discards expired frames at the head of the queue until the latency drops to an acceptable range. Although the latency in the graph does not reach the 200-millisecond threshold, this mechanism remains in a ready state as a safety guarantee. The response curve visually verifies the robustness of the signal alignment module of this invention: when faced with multiple network packet loss interferences, the system maintains time axis continuity through silent padding, controls latency accumulation through adaptive recovery, and prevents system crashes through threshold protection, thereby providing stable and reliable aligned frame data input for subsequent parallel processing of dual CUDA computing streams.

[0047] Besides network packet loss, the accumulation of processing latency is also a significant factor affecting real-time performance. When the server's processing speed temporarily falls below the data arrival rate, more and more unprocessed data accumulates in the queue, causing end-to-end latency to continuously increase. To prevent latency from accumulating indefinitely, a latency tolerance limit needs to be set, and expired data should be proactively discarded when this limit is exceeded. Let the current system time be... The timestamp of the head frame of the near-end queue is The current processing delay for: ;in Indicates the current system time. Indicates the timestamp of the frame at the head of the queue. This represents the cumulative delay from the time the frame was captured to the current moment, in milliseconds. When the cumulative delay... Exceeding the preset latency tolerance threshold At that time, continuously discard expired frames at the head of the queue until the latency drops to an acceptable level: ;in This represents the latency tolerance threshold, typically set to 200 milliseconds. Data frames exceeding this threshold lose their real-time value even after processing; discarding these expired frames allows the system to quickly return to a real-time state. In one optional implementation, the number of expired frames discarded is recorded and reported to the monitoring system, facilitating system performance bottleneck analysis by maintenance personnel.

[0048] Timestamp anomalies are another situation requiring special handling. Normally, the timestamps of the same data stream should monotonically increase, with each data packet's timestamp increasing by one frame duration compared to the previous packet. When timestamp rollback or a sudden timestamp returning to zero is detected, it indicates a possible abnormal event such as client restart, clock reset, or data stream switching. Let the current frame timestamp be... The timestamp of the previous frame is The frame duration is The criteria for determining an anomaly in the timestamp are: or ;in Indicates the timestamp of the current frame. Indicates the timestamp of the previous frame. This indicates the standard frame duration, with a value of 20 milliseconds. This is the anomaly detection coefficient, typically set to 10. This means that an anomaly is detected when the timestamp jump exceeds 10 frames in length.

[0049] Upon detecting a timestamp anomaly, an alignment state reset operation is performed. The reset process includes clearing all buffered data in both the near-end FIFO and far-end FIFO queues, updating the expected sequence number to the currently received sequence number, and updating the timestamp reference to the currently received timestamp. After the reset, the signal alignment process restarts from the current data packet to establish alignment. This fault-tolerance mechanism ensures that the system can automatically recover from various abnormal states without manual intervention.

[0050] The alignment frame data generated by frame-level pairing is encapsulated in a structured format, including near-end microphone audio sampling data, far-end reference audio sampling data, alignment timestamp, and alignment quality identifier. The alignment quality identifier distinguishes between different states such as perfect alignment, near-end silence fill, far-end silence fill, and abnormal recovery, allowing subsequent processing stages to adopt appropriate processing strategies based on the alignment quality. For example, when the alignment quality identifier is far-end silence fill, acoustic echo cancellation processing can skip the echo cancellation algorithm and directly output the near-end signal, avoiding meaningless calculations in the absence of a valid reference signal.

[0051] In an alternative implementation, the signal alignment process can further incorporate correlation analysis to optimize alignment accuracy. Since the far-end reference signal is played through a speaker and then captured by a near-end microphone, there is a high correlation between the echo component in the near-end signal and the far-end reference signal. Utilizing this characteristic, the cross-correlation function between the near-end signal and the far-end reference signal with different delays can be calculated; the delay corresponding to the peak cross-correlation value is the optimal alignment offset. Let the near-end signal be... The remote reference signal is Then the cross-correlation function Defined as: ;in The first one represents the near-end signal. One sampling point, Indicates the delay of the remote reference signal The th sampling point after the th sampling point One sampling point, This represents the total number of sampling points involved in the calculation. Indicates a delay of The cross-correlation value at time t. The delay that causes the cross-correlation function to reach its maximum value. This is the estimated alignment offset: ;in This represents the optimal delay estimate, expressed in units of sampling points. This value can be converted to milliseconds to correct for timestamp alignment errors. It's important to note that cross-correlation-based alignment optimization is only effective in the presence of significant echoes. When the far end is silent or the echo path is severely attenuated, the cross-correlation peak may be insignificant; in such cases, the timestamp alignment result should be used.

[0052] After the aligned frame data is generated, the parallel processing stage of the dual CUDA computation streams begins. The first CUDA computation stream undertakes the core computational tasks of acoustic echo cancellation and speech activity detection. These two tasks have a natural dependency in the computational flow. The output signal of acoustic echo cancellation also serves as the input of speech activity detection. Therefore, placing them in the same computation stream can eliminate unnecessary cross-stream synchronization overhead and achieve optimal processing efficiency.

[0053] After receiving the aligned frame data, the first CUDA compute stream performs bandpass filtering preprocessing on the near-end microphone audio and the far-end reference audio. The purpose of bandpass filtering is to remove ultra-low frequency and ultra-high frequency components that are not beneficial to speech processing. Ultra-low frequency components mainly originate from mechanical vibrations such as air conditioner operation and elevator operation. These components typically have frequencies below 80 Hz. Although they have high energy, they do not carry speech information and can interfere with the accuracy of subsequent feature extraction. Ultra-high frequency components may include high-frequency noise introduced during the sampling process and interference signals generated by some electronic devices. The lower cutoff frequency of the bandpass filter is set to 80 Hz, and the upper cutoff frequency is set to 7500 Hz. This frequency band fully covers the fundamental frequency of human speech and its main harmonic components, while effectively suppressing out-of-band interference.

[0054] The filter is implemented using an infinite impulse response structure, specifically a second-order Butterworth bandpass filter. The Butterworth filter has the flattest amplitude-frequency response within the passband, thus avoiding distortion of the spectral envelope of the speech signal. The second-order structure achieves a good balance between filtering performance and computational complexity; its difference equation is as follows: ;in Indicates the input signal at the th The sampled values ​​at each sampling time. Indicates the filter output signal at the th The sampled values ​​at each sampling time. , , These are the feedforward coefficients of the filter. , These are the feedback coefficients of the filter. , This represents the historical value of the input signal at the first two sampling times. , This represents the historical values ​​of the output signal at the first two sampling times. The filter coefficients are pre-calculated and stored using a bilinear transform method based on the sampling rate and cutoff frequency.

[0055] After bandpass filtering, the first CUDA computation stream performs three-way feature extraction in parallel, generating features for the near-end filter bank, the linear acoustic echo cancellation filter bank, and the echo estimation filter bank, respectively. These three features describe the spectral characteristics of the near-end signal, the echo-cancelled signal, and the estimated echo signal from different perspectives, providing rich input information for subsequent deep neural network mask inference.

[0056] Near-end filter bank features are extracted directly from the bandpass-filtered near-end microphone audio. The feature extraction process first involves framing and windowing the audio signal. The frame length is set to 25 milliseconds, corresponding to 400 sampling points, and the frame shift is set to 10 milliseconds, corresponding to 160 sampling points. The 25-millisecond frame length is chosen because it is sufficient to cover a complete pitch cycle of the speech signal, ensuring frequency resolution for spectral analysis; while the 10-millisecond frame shift ensures sufficient temporal resolution to capture rapid changes in the speech signal. The windowing operation uses a Hamming window, which effectively suppresses spectral leakage. Its window function is defined as: ;in The window function is in the first position. The values ​​of each sampling point This represents the number of sampling points corresponding to the window length, with a value of 400. Pi is the mathematical constant of a circle.

[0057] The windowed signal frame is transformed to the frequency domain using a Fast Fourier Transform (FFT). The number of transform points is set to 512, which is greater than the frame length, to achieve frequency interpolation and improve the smoothness of the spectrum. The output of the FFT is a complex spectrum. The power spectrum is obtained by taking the square of its amplitude. : ;in Indicates the first Complex spectral values ​​at each frequency point The magnitude of the complex spectrum. Represents the real part of the spectrum of a complex number. The imaginary part of the spectrum of a complex number. Indicates the first Power spectrum values ​​at each frequency point.

[0058] The power spectrum is then frequency-compressed using a Mel filter bank. Human frequency perception is not linear; the ability to resolve low frequencies is stronger than high frequencies. The Mel frequency scale simulates this auditory characteristic. A Mel filter bank consists of a series of triangular bandpass filters, with the center frequencies of the filters uniformly distributed along the Mel frequency axis. (Linear frequency) With Mel frequency The conversion relationship between them is as follows: ;in This represents the linear frequency, measured in Hertz (Hz). This indicates the corresponding Mel frequency, in Mel units. This represents a logarithmic function with base 10, where constants 2595 and 700 are empirical fitting parameters. This invention employs 80 Mel filters, covering a frequency range from 80 Hz to 7500 Hz, to generate an 80-dimensional filter bank feature vector.

[0059] The logarithmic filter bank characteristics are obtained by taking the logarithm of the weighted summation of the power spectrum after passing it through the filter bank. ;in Indicates the first The output characteristic values ​​of each filter Indicates the first The triangular filter in the first... The weighting coefficients for each frequency point and They represent the first The filter covers a frequency index with lower and upper bounds. Indicates the first Power spectrum values ​​at each frequency point To prevent small positive numbers from underflowing in logarithmic operations, the value is set to... , This represents the natural logarithm function. Taking the logarithm transforms multiplicative noise into additive noise, while also compressing the dynamic range of the features, which is beneficial for the stable training and inference of the subsequent neural network.

[0060] Extracting features from a linear acoustic echo cancellation filter bank requires first performing linear adaptive filtering to obtain a preliminary echo cancellation signal. The linear acoustic echo cancellation algorithm is based on adaptive filter theory. It passes the far-end reference signal through an adjustable filter to obtain an echo estimation signal, which is then subtracted from the near-end signal to eliminate the echo component. The adaptive filter uses a normalized least mean square algorithm for coefficient updates, and its update equation is: ;in Indicates the first The filter coefficient vector at time n, with dimension equal to the filter order. A typical value is 512, corresponding to an echo path modeling capability of 32 milliseconds; This represents the historical sample vector of the far-end reference signal, containing data from the first... Starting from that moment, working backwards... The signal values ​​at each sampling point; The error signal is equal to the near-end signal minus the echo estimation signal; This represents the step size parameter, which controls the update speed of the filter coefficients; a typical value is 0.5. This is a regularization constant to prevent numerical instability in division operations; its value is [value missing]. ; This represents the transpose of the far-end reference signal vector.

[0061] Echo estimation signal Obtained by convolving the filter coefficients with the far-end reference signal: ;in Indicates the first The echo estimate at time 1. The first element of the filter coefficient vector represents the... One portion, Indicates the far-end reference signal at the 1st The sampled values ​​at each time step. Linear acoustic echo cancellation output signal. Subtract the echo estimation signal from the near-end signal: ;in Indicates the near-end microphone signal at the 1st The sampled value at each moment. This represents the output signal after linear acoustic echo cancellation. This output signal is then used to extract 80-dimensional logarithmic filter bank features using the same process as the near-end filter bank features, forming the linear acoustic echo cancellation filter bank features.

[0062] The characteristics of the echo estimation filter bank are directly derived from the echo estimation signal. Extracted from [the source]. This feature vector is also 80-dimensional, and its function is to provide explicit information about the current echo intensity to the deep neural network. When the echo path changes, causing a decrease in the performance of linear acoustic echo cancellation, a deviation will appear between the echo estimation signal and the actual echo. The deep neural network can learn the echo remnant pattern by comparing these three features, thereby achieving more accurate nonlinear echo cancellation.

[0063] The features of the three 80-dimensional filter banks are concatenated along the channel dimension to form a 240-dimensional joint feature vector. The concatenation order is fixed as follows: near-end filter bank features, linear acoustic echo cancellation filter bank features, and echo estimation filter bank features; this order must be maintained during both training and inference phases. The joint feature vector is then subjected to mean-variance normalization, standardizing the numerical distribution of each feature dimension to zero mean and unit variance. ;in Represents the first digit before normalization. 3D eigenvalues Indicates the first The global mean of the dimensional features. Indicates the first Global standard deviation of 3D features Represents the normalized i-th 3D feature values. The global mean and standard deviation are statistically obtained from a large-scale speech dataset during model training and directly loaded and used during inference. Normalization eliminates the dimensional differences between different feature dimensions, ensuring that the inputs of each layer of the neural network maintain a stable numerical range, which helps to accelerate convergence and improve generalization ability.

[0064] The normalized feature sequence is input into a deep feedforward sequence memory network for masked inference. The deep feedforward sequence memory network is a neural network architecture specifically designed for sequence modeling. Compared to recurrent neural networks, it has higher computational parallelism, and compared to standard feedforward networks, it can effectively model long-term dependencies. The network structure employs a 9-layer stacked architecture, with each layer containing a memory block and a feedforward transform.

[0065] The core design of the memory block lies in its ability to compress and encode historical frame information using a learnable projection matrix. Let the input features of the current frame be... A memory block maintains a sequence of memory vectors. ,in Indicates the length of the memory window. Output of the memory block. Obtained by weighted summation of historical memory vectors: ;in Indicates the first The memory output vector of the frame. Indicates the first The memory vector of a frame. Indicates the first The weighting coefficients of each historical frame This indicates the number of historical frames covered by the memory window, typically 10. Weighting coefficient. These are learnable parameters that are automatically optimized during training to adapt to dependencies across different time spans. (Memory vector) Generate by fusing the current frame input with the memory output: ;in This represents the weight matrix generated by memory. This represents the bias vector. This represents the concatenated vector of the current frame input and the memory output. This design allows the memory vector to simultaneously contain information from the current frame and historical context, achieving efficient information transfer.

[0066] Deep feedforward sequence memory networks employ a strictly causal structure design, meaning the output of the current frame depends only on the input of the current frame and its historical frames, without using any information from future frames. This causal structure is achieved through left-padded convolutions, where the convolution kernel slides only to the left of the time axis (in the historical direction), ensuring zero look-ahead latency. This characteristic is crucial for real-time streaming processing, allowing each frame of data to be processed and output immediately upon arrival, without waiting for subsequent frames.

[0067] To expand the network's receptive field and capture long-term dependencies, each convolutional layer employs an exponentially increasing dilation rate. Let the... The expansion rate of the layer is ,but: ;in The layer index is represented, starting from 1. Indicates the first Layer dilation rate. Dilated convolution expands the receptive field without increasing the number of parameters. The dilation rate of layer 1 is 1, layer 2 is 2, layer 3 is 4, and so on. The total receptive field of the 9-layer network covers approximately 500 milliseconds of historical information, sufficient to capture syllable-level and word-level contextual information in speech signals. The network also employs depthwise separable convolution to reduce the number of parameters and improve inference speed. Depthwise separable convolution decomposes standard convolution into two steps: channel-wise convolution and pointwise convolution, with a computational complexity approximately one-third that of standard convolution. times, of which The kernel size is [size]. This refers to the number of channels. In the configuration of this invention, this decomposition reduces the computational cost of a single-layer convolution by approximately 8 times.

[0068] Each layer of the network is configured with residual connections, which directly add the input of that layer to its output: ;in Indicates the first Layer input, Indicates the first The nonlinear transformation function of the layer, Indicates the first The output of the layer. Residual connections effectively alleviate the vanishing gradient problem in deep networks, making the training process more stable. They also enable the network to learn identity mappings, preventing the introduction of additional perturbations when the input is already close to its optimal representation. The top layer of the deep feedforward sequence memory network is configured with a multi-task output head, simultaneously outputting the spectral mask and speech activity detection probability in a single forward inference. The two tasks share the backbone feature extraction part of the network, branching into two independent output branches only in the last layer. The design of the shared backbone network is based on the observation that both the spectral mask prediction task and the speech activity detection task need to understand the speech content and noise environment of the current frame. The low-level feature representations required by the two are highly similar, and shared feature extraction can fully utilize this similarity and reduce redundant computation.

[0069] The spectral mask output branch contains a fully connected layer that maps the hidden representation of the backbone network to a 257-dimensional spectral mask space, where 257 corresponds to the number of non-negative frequency points in the 512-point Fast Fourier Transform. The spectral mask is then mapped to the range of 0 to 1 using the Sigmoid activation function. ;in The output of the fully connected layer represents the first... Original score, This represents the Sigmoid activation function. It is a natural constant. Indicates the first The mask value for each frequency point. A mask value close to 1 indicates that the signal component at that frequency point is mainly near-end speech and should be preserved, while a mask value close to 0 indicates that the signal component at that frequency point is mainly echo or noise and should be suppressed.

[0070] The speech activity detection output branch also contains a fully connected layer that maps the hidden layer representation to a 1-dimensional activity probability space. The original score is then activated by a sigmoid function to obtain the speech activity detection probability. ;in This represents the raw score of the speech activity detection branch. This represents the probability of speech activity in the current frame, ranging from 0 to 1. A probability value close to 1 indicates that the current frame is highly likely to contain valid speech, while a probability value close to 0 indicates that the current frame is highly likely to be silent or pure noise.

[0071] Echo cancellation is achieved by performing element-wise multiplication between the spectral mask and the short-time Fourier transform result of the near-end signal. Let the short-time Fourier transform result of the near-end signal be... The spectrum after echo cancellation for: ;in Indicates the near-end signal at the 1st Complex spectral values ​​at each frequency point This represents the mask value for the corresponding frequency point. This represents the complex spectral value after echo cancellation. The multiplication operation is performed in the complex domain, meaning that both the amplitude and phase of the spectrum are modulated simultaneously. The echo-cancelled spectrum is reconstructed into a time-domain waveform using an inverse short-time Fourier transform. ;in This indicates the inverse short-time Fourier transform operation. The reconstructed echo-cancelled audio is represented in the first... The values ​​of each sampling point. The inverse transform uses the overlapping addition method corresponding to the forward transform. The overlapping areas between frames are smoothly transitioned through weighted averaging, avoiding discontinuous jumps at frame boundaries.

[0072] Speech activity detection probabilities are smoothed using an exponential moving average to eliminate random fluctuations in frame-level prediction. The raw frame-by-frame probability output may exhibit rapid jumps, and directly using it for triggering decisions can lead to frequent false triggers and missed triggers. The smoothing process obtains a stable speech activity score by weighting and accumulating historical probability values. ;in Indicates the first Speech activity score after frame smoothing, This indicates the speech activity score of the previous frame. Indicates the first The probability of detecting raw speech activity in a frame. This represents the smoothing coefficient, which controls the weighting ratio between historical and current information. A typical smoothing coefficient value is 0.9, meaning that 90% of the current frame score comes from historical accumulation and 10% comes from the current observation, corresponding to a time constant of approximately 100 milliseconds.

[0073] The speech activity score and endpoint logic jointly determine the generation of the interruption flag. In human-computer dialogue scenarios, users may interrupt the system during playback. In this case, the system needs to quickly identify the interruption intent and terminate the current playback. The core conditions for interruption detection include: the far-end reference signal is active, meaning the system is playing audio to the user; the near-end speech activity score exceeds the interruption threshold; and the duration of the speech activity exceeds the minimum interruption duration. An interruption flag is generated when all three conditions are met simultaneously. A typical value for the interruption threshold is 0.7, and a typical value for the minimum interruption duration is 150 milliseconds. The purpose of setting the minimum interruption duration is to filter out occasional short noises and avoid misinterpreting non-speech behaviors such as coughing or throat clearing as interruption intent.

[0074] After the first CUDA computation flow completes the above processing, it encapsulates the speech activity state, echo-cancelled audio, and interruption flag into the first-stage result. This first-stage result is immediately published through the partially completed interface of the Future asynchronous object. The client can obtain this result and execute the corresponding interface response without waiting for the speaker separation task to complete. The motivation behind this two-stage response mechanism is that the speech activity state and interruption flag are extremely sensitive to response latency. After a user initiates an interruption, they expect to see immediate system feedback. Waiting for the speaker separation task to complete before responding would introduce unacceptable interaction delays. While the speaker separation task is time-consuming, its output is mainly used in subsequent speech recognition stages, and the requirement for immediate response is relatively relaxed.

[0075] The second CUDA compute stream employs a condition-triggered strategy, meaning it doesn't perform speaker separation inference on every frame of data. Instead, it determines whether inference needs to be initiated based on the speech activity detection results output by the first CUDA compute stream. This design significantly reduces the average load on the graphics processor, concentrating computational resources on truly valuable speech segments.

[0076] The conditional triggering logic is based on a comparison between the speech activity score and a preset trigger threshold. Let the trigger threshold be... When the voice activity score Speaker separation reasoning is triggered when the following conditions are met: ;in This represents the speech activity score for the current frame. This represents the trigger threshold, typically set to 0.5. The threshold setting needs to strike a balance between sensitivity and false trigger rate: a threshold that is too low will lead to frequent triggering during silent or noisy periods, wasting computational resources; a threshold that is too high will cause the initial speech segment to be missed, affecting the separation effect. In an optional implementation, the trigger threshold can be dynamically adjusted based on historical statistical information to adapt to different noise environments.

[0077] The second CUDA computation stream maintains a sliding window buffer that continuously receives signal-aligned output audio from the near-end microphone and writes it to a history queue. The sliding window buffer has a capacity of 2 seconds, corresponding to 32,000 samples, which can cover a complete short sentence or speech segment. When a trigger condition is met, all buffered audio data is extracted from the sliding window buffer and used as input for speaker separation inference.

[0078] The speaker separation inference task is computationally intensive. To avoid blocking the real-time processing of the first CUDA computation stream, the inference task is submitted to a thread pool for asynchronous execution. The thread pool pre-creates a fixed number of worker threads, typically configured with 4 to 8 threads, matching the number of physical cores on the server. After the inference task enters the task queue of the thread pool, it is retrieved and executed by an idle worker thread. The tasks executed by the thread pool first perform preprocessing operations on the central processing unit, including format conversion and normalization of the audio data, and then submit the preprocessed data to the second CUDA computation stream for neural network inference.

[0079] The speaker separation model employs an end-to-end deep learning architecture, directly predicting the separated speech waveforms of each speaker from the mixed speech waveform, without the need for explicit feature extraction or sound source localization. The model input is a single-channel mixed speech, and the output is multi-channel separated speech, with each output channel corresponding to an estimated speaker source. The number of output channels is determined during model training, typically configured with 2 to 4 channels to cover most multi-person dialogue scenarios.

[0080] The network structure of the separation model is based on a temporal convolutional architecture. First, the input waveform is encoded into a high-dimensional feature representation through a one-dimensional convolutional layer. Then, speaker-specific embedding representations are obtained through multiple layers of temporal modeling. Finally, the embedding representations are decoded into temporal waveforms through deconvolutional layers. The encoder's convolutional kernel length is set to 20 sampling points, and the stride is set to 10 sampling points. This configuration is comparable to the frame shift of the short-time Fourier transform, enabling it to capture the time-varying characteristics of speech signals with similar temporal resolution.

[0081] After the separation inference is completed, multi-channel separated audio is output, with each channel containing the speech signal of an estimated speaker. Since the speaker separation model cannot know in advance which speaker is the target speaker, a selection strategy needs to be designed to determine the target output channel from multiple output channels. This invention adopts an energy maximization selection strategy based on the following assumption: In a near-end microphone acquisition scenario, the user, as the near-end speaker, is usually closest to the microphone, and their speech has the highest energy proportion in the mixed signal. Therefore, the channel with the highest energy after separation is most likely to correspond to the user's speech.

[0082] refer to Figure 3 The diagram is divided into two sub-diagrams, illustrating how the system accurately extracts the target user's speech signal in complex acoustic scenarios with multiple speakers. The horizontal axis represents time in milliseconds, covering a typical... Millisecond analysis window. The upper subplot depicts the raw separated audio waveforms output by the deep neural network model. This model employs an end-to-end temporal convolutional architecture, directly mapping the input single-channel mixed speech to multi-channel separated speech. The figure shows two waveform curves, representing two independent output channels. The blue solid line corresponds to separated channel A, whose waveform amplitude is large and its envelope is clear, exhibiting obvious speech characteristics; the red solid line corresponds to separated channel B, whose waveform amplitude is small and cluttered, representing background interference or speech remnants from non-target speakers. In actual near-end microphone acquisition scenarios, because the target user is close to the microphone, their direct sound energy is usually significantly higher than environmental noise or interference from other speakers. This physical characteristic is directly reflected in the waveform amplitude differences of each channel after separation. The lower subplot details the mathematical principles of the target channel selection decision. To determine which of the multiple unlabeled output channels contains the target user's speech, the system calculates an energy index for each channel. The energy index is calculated based on the sum of squares of samples within a sliding window. For the Each output channel has an energy curve. From the formula Received, among which Is this channel in the 1st The amplitude value of each sampling point To calculate the window size. The thick blue solid line in the graph represents the energy index of channel A. The red dashed line represents the energy index of channel B. .

[0083] The energy curve shows that channel A exhibits a significant peak in the active speech range, and is consistently higher than channel B for most of the time. The graph uses blue shading to indicate this peak. This region visually represents the decision criteria for the optimization logic. At the end of each analysis frame, the system compares the cumulative energy values ​​of each channel and performs a selection operation: In this embodiment, because Persistently greater than The algorithm will resolutely select channel A as the target output channel. The "maximum energy selection" arrow in the diagram points to the energy peak moment, indicating that at this moment the system confirms the ownership of the target channel and writes the data of that channel into the Future asynchronous object. This energy-based blind source separation post-processing strategy effectively solves the problem of uncertain output channel arrangement in end-to-end separation models (PermutationInvariantTraining problem). In the dual CUDA stream architecture, stream 2 is dedicated to this computationally intensive inference task. When the first CUDA stream detects valid speech activity and triggers stream 2, stream 2 utilizes the parallel computing power of the GPU to quickly complete the waveform separation of the entire window and immediately executes the aforementioned energy calculation and channel selection. The diagram shows... The millisecond time window is only a snapshot of the processing; in actual operation, the logic is executed continuously on a sliding buffer.

[0084] If there are two speakers with similar volumes in the scene (dual-speaker mode), the energy curve may alternately lead. To avoid auditory discomfort caused by frequent switching of target channels, the system also introduces a smoothing mechanism, namely... Based on this, a delayed comparison logic or authentication module has been added. If combined with voiceprint recognition technology, the system will further calculate the cosine similarity between each channel and the target user's voiceprint embedding vector, in addition to energy filtering. However, in most near-field interaction scenarios... Figure 3 The pure energy-based decision logic shown provides sufficiently high selection accuracy while maintaining extremely low computational overhead, meeting the performance requirements of real-time systems. In this way, the second-stage response can provide the application layer with high-quality target speech free from background interference, offering clean acoustic input for subsequent speech recognition or semantic understanding. The calculation of the energy index is performed within the same analysis window. Let the first... The separate audio for each output channel is The analysis window length is If there are 1 sampling point, then the energy index of that channel is... Defined as the sample sum of squares: ;in Indicates the first The output channel is in the first Separated audio values ​​from each sampling point This indicates the number of sampling points covered by the analysis window, typically 4000, corresponding to an analysis duration of 250 milliseconds. Indicates the first The energy index of each channel is selected. The channel with the highest energy index is chosen as the target output channel. ;in Indicates the index of the target output channel. The separated audio from this channel is extracted as the final speaker separation result.

[0085] In an alternative implementation, the selection of the target speaker can also be combined with speaker embedding vectors for authentication. The system pre-stores the voiceprint feature vector of the target user. During separation inference, voiceprint features are extracted from each output channel and compared with the stored templates for similarity. The channel with the highest similarity is selected as the target output. This method can handle complex scenarios in multi-person dialogues where the target user is not the largest energy source, such as when a user speaks softly while others are talking loudly.

[0086] When the speech activity score does not exceed the trigger threshold, the second CUDA computation stream skips speaker separation inference to free up GPU resources. Performing speaker separation inference during periods of silence or pure noise not only wastes computational resources but may also produce abnormal separation results due to low input signal quality. Skipping execution allows the GPU to allocate computational power to other concurrent tasks or enter a power-saving state.

[0087] When the speech activity score consistently exceeds the trigger threshold, it indicates continuous speech input in the near end, potentially corresponding to a two-way speech state, where the user and system speak simultaneously. In a two-way speech state, echo interference is most severe, and speaker separation is most valuable. At this time, the second CUDA computation stream continues to execute continuously, initiating an inference whenever the sliding window buffer accumulates enough new data, increasing the update frequency of the separation results to track rapidly changing speech content.

[0088] The separated audio from the target output channel is written to a Future asynchronous object to generate the second-stage result. A Future asynchronous object is an asynchronous programming primitive that represents a computation result that is not yet complete but will be completed at some point in the future. The caller can register a callback function on the Future object or execute a blocking wait, automatically receiving notification or returning from the blocking state when the result is ready.

[0089] The two-phase response mechanism is implemented through phased completion of a single asynchronous Future object. Internally, the Future object maintains two completion flags and two result slots, corresponding to the first and second phases respectively. After the first CUDA computation stream completes processing, the first-phase completion flag is set and the first-phase result is written. The caller waiting for the first-phase result is then notified and can read the voice activity status, echo-cancelled audio, and interruption flag. After the second CUDA computation stream completes processing, the second-phase completion flag is set and the separated audio is written. The caller waiting for the second-phase result is then notified and can read the high-quality separated result.

[0090] In one optional implementation, the two-stage results also include timestamps and processing latency statistics. The timestamps record the audio acquisition time corresponding to the result, and the processing latency statistics include a breakdown of the time consumed in each stage, facilitating time synchronization and performance monitoring by downstream applications. The typical processing latency for the first-stage result is 15 to 25 milliseconds, and the typical processing latency for the second-stage result is 80 to 150 milliseconds, with the specific values ​​depending on the computing power of the graphics processor and the current load.

[0091] The preferred embodiments of this disclosure have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of this disclosure. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of this disclosure shall be within the scope of the claims of this disclosure.

Claims

1. A collaborative speech processing method based on dual CUDA streams, characterized in that, include: A first CUDA compute stream and a second CUDA compute stream are created on the GPU. The first CUDA compute stream is used for acoustic echo cancellation and speech activity detection, while the second CUDA compute stream is used for speaker separation. Near-end microphone audio data packets and far-end reference audio data packets are received via the WebSocket access layer. The signal alignment module performs timestamp verification, queue buffering, and frame-level pairing output on the near-end microphone audio data packets and far-end reference audio data packets to form aligned frame data. In the first CUDA compute stream, acoustic echo cancellation is performed frame-by-frame on the aligned frame data to generate echo-cancelled audio, while simultaneously generating speech activity detection results. In the second CUDA compute stream, the speech activity detection results are received and conditional triggering is applied. Speaker separation inference is performed on the buffered audio and separated audio is generated only when the speech activity detection results meet the triggering conditions. A two-stage response is output via a Future asynchronous object. The first stage outputs a first-stage result containing the speech activity state, echo-cancelled audio, and interruption flag after the first CUDA compute stream completes speech activity detection. The second stage outputs the separated audio as the second-stage result after the second CUDA compute stream completes speaker separation.

2. The method according to claim 1, characterized in that, The first and second CUDA computation flows respectively execute CUDA graph capture to generate graph execution instances on a fixed operator sequence. The graph execution instances include time-frequency feature extraction, mask inference, spectral mask processing, speech activity detection smoothing, and speaker separation inference. A graph execution instance cache pool is maintained in the session dimension. The graph execution instance is selected for replay based on the input frame length, sliding window length, and GPU load to maintain end-to-end latency stability when the audio frame length changes.

3. The method according to claim 2, characterized in that, A circular buffer is established in the GPU memory, partitioned by session. The circular buffer stores the aligned frame data, the speech activity detection results and timestamp records output by the first CUDA compute stream. After writing to the circular buffer, the first CUDA compute stream writes a monotonically increasing sequence number and records CUDA events. After the condition is triggered, the second CUDA compute stream waits for the CUDA event and reads the corresponding segment according to the sequence number. After reading, it writes back the consumption cursor to complete the lock-free synchronization across CUDA streams. When a discontinuous sequence number is detected, silence filling or window reconstruction is performed to avoid speaker separation inference input breaks.

4. The method according to claim 3, characterized in that, Construct a trigger gating controller. The trigger gating controller counts false trigger samples of voice activity detection results within a sliding time window and performs distribution-independent confidence calibration to generate an adaptive trigger threshold that meets the target false trigger rate constraint. The network arrival interval, the latency accumulation index output by the signal alignment module, and the queuing latency index of the first CUDA computing stream and the second CUDA computing stream are used as context features and input into the context router. The context router selects a triggering strategy from the candidate triggering strategy set. The candidate triggering strategy includes at least the triggering threshold, the speaker separation sliding window length, and the continuous execution duration after triggering, which is used to adaptively schedule the second CUDA computing stream between the silent period, the single-speaking period, and the double-speaking period.

5. The method according to claim 4, characterized in that, A directed acyclic graph architecture is used to manage the processing pipeline. The signal alignment module acts as a broadcast node to distribute the alignment frame data to both the first and second processing pipelines. The first processing pipeline is used to perform acoustic echo cancellation and speaker separation when the far-end reference audio data packet is present. The second processing pipeline is used to skip acoustic echo cancellation and perform speaker separation when the far-end reference audio data packet is absent.

6. The method according to claim 5, characterized in that, The signal alignment module includes: performing timestamp extraction and sampling rate conversion on the near-end microphone audio data packets and the far-end reference audio data packets, and writing them into the near-end FIFO queue and the far-end FIFO queue, respectively; performing timestamp difference verification on the header data of the near-end FIFO queue and the far-end FIFO queue and performing pairing output; determining network packet loss when the timestamp difference is greater than the data duration, performing mute padding on the missing segments, and then pairing output; determining processing delay accumulation when the actual processing time exceeds the data push time and discarding expired frames to restore real-time performance; and resetting the alignment state when the timestamp rolls back or the timestamp returns to zero.

7. The method according to claim 6, characterized in that, The first CUDA computation flow for acoustic echo cancellation and speech activity detection includes: performing bandpass filtering preprocessing on aligned frame data; generating in parallel near-end Fbank features, linear acoustic echo cancellation Fbank features, and echo estimation Fbank features; performing three-channel feature concatenation on the near-end Fbank features, linear acoustic echo cancellation Fbank features, and echo estimation Fbank features, and then inputting the result into a deep feedforward sequence memory network (DFSMN) mask model after mean-variance normalization. The DFSMN mask model outputs a spectral mask and speech activity detection probability simultaneously in a single forward inference.

8. The method according to claim 7, characterized in that, The first CUDA computation stream performs a Sigmoid mapping on the spectral mask to obtain the mask result, and performs a Sigmoid mapping on the speech activity detection probability to obtain the speech activity detection probability result; the near-end microphone audio in the aligned frame data is transformed by a short-time Fourier transform to obtain a time-frequency representation, and the time-frequency representation is multiplied element-wise with the mask result and then transformed by an inverse short-time Fourier transform to generate echo-cancelled audio; The probability result of voice activity detection is smoothed by exponential moving average to form a voice activity score, and an interruption flag is generated when the voice activity score and the endpoint logic meet the real-time interruption condition.

9. The method according to claim 8, characterized in that, Speaker separation in the second CUDA computation flow includes: writing the near-end microphone audio output from the signal alignment module into a history queue to form a sliding window buffer; when the speech activity score exceeds the trigger threshold, extracting the window audio from the sliding window buffer and submitting it to the thread pool for execution; the thread pool is used to execute speaker separation inference tasks and CPU-intensive preprocessing tasks; the thread pool tasks perform end-to-end speaker separation inference in the second CUDA computation flow and output multi-channel separated audio; calculating the sum of squares of samples as an energy index for the multi-channel separated audio within the same analysis window and selecting the channel with the largest energy index as the target output channel; writing the target output channel into a Future asynchronous object to generate the second-stage result; skipping speaker separation inference to release GPU resources when the speech activity score does not exceed the trigger threshold, and maintaining continuous triggering in the dual-speaker state where the speech activity score is consistently higher than the trigger threshold to improve the separation update frequency.

10. The method according to claim 9, characterized in that, An independent session state copy is maintained for each WebSocket connection. The session state copy stores the history queue, voice activity score smoothing status, near-first-out queue and far-first-out queue used by the signal alignment module, and timestamp records for delay calculation. The session state copy is managed by a weak reference keyed dictionary with the WebSocket connection object as the key, so that the session state copy is automatically reclaimed by the garbage collection mechanism after the WebSocket connection is closed. It adopts a dual-event-loop thread model. The main thread event loop executes WebSocket sending and receiving, business logic processing and GPU inference task scheduling, while the background thread event loop executes cross-module data transfer and queue forwarding. The main thread event loop and the background thread event loop communicate through a thread-safe queue.

Images