Data processing method and system for streaming voice-driven digital human video

Abstract

The application discloses a data processing method and system for streaming voice-driven digital human video, and the data processing method comprises the following steps: transmitting an audio data block generated by a voice synthesis module in a streaming mode to a video rendering module; according to the number of sampling points of the audio data block and the number of sampling points required by a single frame of image, the audio data block is split to obtain a plurality of video frames and residual sampling points; the residual sampling points are stored; the next audio data block is obtained, and the residual sampling points of the previous audio data block are spliced before the sampling points of the next audio data block. The application can effectively eliminate the accumulation of audio-visual synchronization errors of long-time continuous broadcasting, avoid picture jump caused by short pauses between sentences, and completely remove the remaining old data after interruption to prevent content mixing. Overall, the synchronization accuracy and visual continuity of streaming interactive digital human video playing are significantly improved on the premise of ensuring real-time.

Description

Technical Field

[0001] This invention relates to a data processing method and system for streaming voice-driven digital human video. Background Technology

[0002] In interactive digital human systems, large language models typically generate response text sentence by sentence and sub-sentence in a streaming manner. The text-to-speech (TTS) module then generates corresponding audio segments sentence by sentence in a streaming manner. The digital human video rendering module needs to generate matching lip-sync / facial animation video frames in real time based on the continuously arriving audio segments, and send them to the client for continuous playback through the real-time communication (RTC) channel to form a visual and auditory effect of "digital human speaking" that is perceived by the user.

[0003] The following problems exist in existing streaming voice-driven digital human video scenarios: audio-visual synchronization errors accumulate linearly with the broadcast duration; brief silence intervals between sentences cause visual jumps when the screen switches between speaking and standby states; and new and old audio and video content are mixed in scenarios where voice interruption occurs. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the defects of existing technologies in streaming voice-driven digital human video, such as the linear accumulation of audio-visual synchronization error with the broadcast duration, visual jumps caused by short silence intervals between sentences when switching between speaking and standby states, and mixed playback of old and new audio-visual content in scenarios where voice is interrupted. The present invention provides a data processing method and system for streaming voice-driven digital human video that can effectively eliminate the accumulation of audio-visual synchronization error during long continuous broadcasts, avoid visual jumps caused by short pauses between sentences, and thoroughly clear residual old data after interruption to prevent content mixing.

[0005] This application solves the aforementioned technical problem through the following technical solution:

[0006] A data processing method for streaming voice-driven digital human video, characterized in that the data processing method includes:

[0007] The audio data blocks generated by the speech synthesis module in a streaming manner are transmitted to the video rendering module;

[0008] Based on the number of sampling points in the audio data block and the number of sampling points required for a single frame of video, the audio data block is segmented to obtain several video frames and the remaining sampling points.

[0009] Store the remaining sampling points;

[0010] Obtain the next audio data block and append the remaining sampling points of the previous audio data block to the sampling points of the next audio data block.

[0011] Preferably, the data processing method includes:

[0012] Determine whether the audio data block is the last reply content. If not, store the remaining sampling points. If so, generate the final video frame together with the final silence.

[0013] Preferably, the data processing method includes:

[0014] The actual number of final frames is determined by rounding the audio data block at the end.

[0015] When an interruption signal or a change in the generation identifier is detected, the carryover status quantity is cleared synchronously.

[0016] The speech synthesis module segments the audio data output in multiples of the video frame length.

[0017] Preferably, the video rendering module includes a dedicated timing thread, which is independent of the threads for audio reception and model inference speed, and the data processing method includes:

[0018] The dedicated timing thread calculates the target output time for each frame of image based on a high-precision monotonic clock, with the target frame rate configured by the system as the period.

[0019] The dedicated timing thread will output the corresponding video frame at the target output time.

[0020] Preferably, the dedicated timing thread outputs the corresponding video frame image at the target output time, including:

[0021] For a frame of image, determine whether the time elapsed between the frame of image and the target output time is less than a preset value. If so, the dedicated timing thread reads the high-precision monotonic clock at high frequency until the target output time and outputs the frame of image. Otherwise, the dedicated timing thread calls the operating system sleep function to reduce power consumption.

[0022] Preferably, the dedicated timing thread outputs the corresponding video frame image at the target output time, including:

[0023] For each output moment, a dedicated timing thread determines whether there is a corresponding frame of image to be output at that output moment. If so, the image to be output is output. Otherwise, it determines whether the duration of the empty frame is less than the preset duration. If it is less than the preset duration, the last frame of the output image is played repeatedly. If it is greater than the preset duration, the preset standby loop image is output.

[0024] Preferably, the video rendering module includes a multi-level buffer queue, and the data processing method includes:

[0025] Determine if the system has received an interruption signal. If so, clear the multi-level buffer queue and add an identifier to the global video rendering.

[0026] For each processing queue in the video rendering module, determine whether the identifier of the output data of the processing queue is the latest identifier. If it is, the output data enters the next processing queue; otherwise, the output data is discarded.

[0027] Preferably, the data processing method is obtained through a generation method, the generation method comprising:

[0028] A dedicated diagnostic log module is constructed to record the original byte length, the number of frames that participate in the generation of video frames after segmentation, the number of remaining sampling points that are carried over and retained after segmentation, and whether it is a sentence end flag bit each time the video rendering module receives an audio data block. On the speech synthesis module side, the number of audio bytes actually sent each time is recorded.

[0029] The dedicated diagnostic log module is used to obtain the first cause of the gradual increase in the synchronization error between lip movements and speech over the duration of broadcasting;

[0030] The step of carrying over and retaining the remaining sampling points to the next audio data block in the data processing method that utilizes the first cause;

[0031] The data processing method is executed again, and the second cause of the slight audio-visual desynchronization at the end of the segment is obtained using the dedicated diagnostic log module.

[0032] The step of determining the actual number of final frames by rounding the last audio data block in the data processing method based on the second cause;

[0033] The data processing method is executed again, and the dedicated diagnostic log module is used to obtain the third cause of audio-visual asynchrony from the beginning of the next round of broadcast after the interruption.

[0034] The step of synchronously clearing the carryover state quantity in the data processing method obtained by the third cause;

[0035] The data processing method is executed again, and the fourth cause of the irregular fluctuation of synchronization error in the long-term continuous broadcast scenario is obtained by using the dedicated diagnostic log module.

[0036] The step in the fourth cause-based data processing method where the speech synthesis module segments and outputs audio data in an integer multiple of the video frame length during the audio data output stage;

[0037] The data processing method is executed again, and the dedicated diagnostic log module is used to obtain the fifth cause, which is that the lip movements on the screen stop before the speech stops and the phenomenon becomes more and more obvious as the number of sentences increases;

[0038] The step of determining whether an audio data block is the last sentence of the reply content in the data processing method obtained by the fifth cause.

[0039] Preferably, the generation method includes:

[0040] The dedicated diagnostic log module is used to obtain the reasons why the video frame to be output cannot be obtained within the current output cycle;

[0041] Analyze the reasons why the video frames in the speech to be output cannot be obtained within the current output cycle, and obtain the steps of establishing a dedicated timing thread in the data processing method;

[0042] The dedicated diagnostic log module was used to obtain the cause of the long-term freezing in the open mouth state when the hold frame mechanism was triggered on the frame of the image where the jaw was open.

[0043] The analysis method obtains the numerical value of the preset duration of the gap frame used for comparison in the data processing method to obtain the cause of the long-term frozen mouth state.

[0044] Preferably, the generation method includes:

[0045] The dedicated diagnostic log module is used to obtain the reason why a new round of responses cannot be played normally after the interruption;

[0046] Get the steps to perform a clear operation on a multi-level buffer queue and add an identifier globally for video rendering.

[0047] Based on common knowledge in the field, the preferred conditions described can be combined arbitrarily to obtain various preferred embodiments of the present invention.

[0048] The positive and progressive effects of this invention are as follows:

[0049] It can effectively eliminate the accumulation of audio-visual synchronization errors during long-term continuous broadcasting, avoid screen jumps caused by short pauses between sentences, and completely clear residual old data after interruption to prevent content mixing. Overall, it significantly improves the synchronization accuracy and visual coherence of streaming interactive digital human video playback while ensuring real-time performance.

[0050] Specifically, the integer frame alignment and remainder carryover mechanism fundamentally eliminates the problem of audio-visual synchronization error accumulation caused by zero-padding and discarding in block by block, so that the lip-sync accuracy in continuous multi-sentence and long-duration broadcast scenarios does not deteriorate with the increase of broadcast duration.

[0051] An independent frame rate metronome decouples the output rhythm of video frames from the instantaneous fluctuations in the inference speed of the underlying model, thereby improving the stability of the output rhythm.

[0052] The short-gap frame smoothing mechanism eliminates the screen transition problem caused by normal pauses between sentences without introducing additional recognition delay, while retaining the ability to switch standby screens in truly long silent scenes.

[0053] The multi-level queue linkage clearing mechanism based on generational identifiers ensures that old data remaining in the rendering pipeline during interruption is completely and consistently discarded, avoiding the abnormal playback of mixed old and new content after interruption. It can be used in conjunction with the applicant's other invention, "Interactive Digital Human Voice Interruption Defense Method and System without Hardware Echo Cancellation," to form a complete interruption response closed loop.

[0054] The data processing method of this application has been tested and verified in a long-term, multi-round practical interactive digital human system. It can effectively solve the three typical problems mentioned in the background technology, namely, the accumulation of audio-visual synchronization error, the intermittent image jump, and the content mixing after interruption, while ensuring real-time performance. Attached Figure Description

[0055] Figure 1 This is a flowchart of the data processing method in Embodiment 1 of the present invention.

[0056] Figure 2 This is another flowchart of the data processing method in Embodiment 1 of the present invention. Detailed Implementation

[0057] The present invention will be further illustrated by way of embodiments below, but the present invention is not limited to the scope of the embodiments described herein.

[0058] Example 1

[0059] This embodiment provides a data processing system for streaming voice-driven digital human video. The data processing system can be implemented using a server in conjunction with input and output devices.

[0060] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0061] The data processing system is used for:

[0062] The audio data blocks generated by the speech synthesis module in a streaming manner are transmitted to the video rendering module;

[0063] Based on the number of sampling points in the audio data block and the number of sampling points required for a single frame of video, the audio data block is segmented to obtain several video frames and the remaining sampling points.

[0064] Store the remaining sampling points;

[0065] Obtain the next audio data block and append the remaining sampling points of the previous audio data block to the sampling points of the next audio data block.

[0066] Furthermore, the data processing system is used for:

[0067] Determine whether the audio data block is the last reply content. If not, store the remaining sampling points. If so, generate the final video frame together with the final silence.

[0068] The data processing system is also used for:

[0069] The actual number of final frames is determined by rounding the audio data block at the end.

[0070] When an interruption signal or a change in the generation identifier is detected, the carryover status quantity is cleared synchronously.

[0071] The speech synthesis module segments the audio data output in multiples of the video frame length.

[0072] Because audio sampling rate and video frame rate are usually not integer multiples of each other, or the length of the arriving audio segments in bytes is random (for example, a 24000Hz sampling rate and 25 frames per second correspond to 960 sampling points per video frame, but the actual length of the arriving audio data block is often not an integer multiple of 960), the common approach in existing implementations is to divide each audio data block according to the video frame length, and then pad the remaining part (less than one frame) with zeros to make it a whole number to participate in the generation of the video frame for this block. The remaining part is then discarded and not passed to subsequent data blocks. When a complete response consists of multiple consecutive speech sentences, and there are no long pauses between the sentences but they are played continuously, the aforementioned "padding and discarding blocks" processing method will produce a time alignment error ranging from a few tenths of a millisecond to tens of milliseconds at each block. This error accumulates block by block as the number of sentences increases, eventually manifesting as the technical problem that "the synchronization between digital lip movements and speech gradually deteriorates over time," which is the so-called audio-visual synchronization error accumulation problem.

[0073] For continuously arriving streaming audio data blocks, this embodiment does not directly pad the remaining sampling points at the end of each data block (which are less than the length of a video frame, such as 960 sampling points, corresponding to a sampling rate of 24000Hz and a frame rate of 25 frames per second) with zeros to participate in the generation of the current video frame. Instead, the remaining sampling points are retained, and when a new audio data block is received next time, the retained remaining sampling points are spliced ​​(prepositioned) before the new data block, and then uniformly divided according to the video frame length: the divisible part participates in the generation of this video frame, and the newly generated remaining part after the division continues to be retained and carried over to the next data block processing.

[0074] Since the audio data involved in the generation of video frames at any given moment is the result of splicing together multiple integer frames of "actually arrived audio sampling points" without manual zero padding, after multiple consecutive audio segments arrive in sequence and are processed by the mechanism, the audio-visual timing alignment error within each segment and between segments will not be generated and accumulated due to zero padding and discarding of remainders. This achieves the technical effect that the audio-visual synchronization error of the entire continuous broadcast content (regardless of how many segments it contains) does not accumulate over time.

[0075] Furthermore, when it is detected that the current sentence audio is indeed the last sentence of the entire response content (end_of_speech is marked as true), a different ending processing rule is adopted than that for non-last sentences: the remaining sampling points of less than one frame at the end of the last sentence are no longer retained and carried over (because there are no subsequent sentences to splice). At this time, the remaining part is processed as needed to generate a closing video frame together with the necessary closing silence, ensuring that the entire response ends naturally.

[0076] This embodiment provides a method for generating a data processing system and method, the method comprising:

[0077] A dedicated diagnostic log module is constructed to record the original byte length, the number of frames that participate in the generation of video frames after segmentation, the number of remaining sampling points that are carried over and retained after segmentation, and whether it is a sentence end flag bit each time the video rendering module receives an audio data block. On the speech synthesis module side, the number of audio bytes actually sent each time is recorded.

[0078] The dedicated diagnostic log module is used to obtain the first cause of the gradual increase in the synchronization error between lip movements and speech over the duration of broadcasting;

[0079] The step of carrying over and retaining the remaining sampling points to the next audio data block in the data processing method that utilizes the first cause;

[0080] The data processing method is executed again, and the second cause of the slight audio-visual desynchronization at the end of the segment is obtained using the dedicated diagnostic log module.

[0081] The step of determining the actual number of final frames by rounding the last audio data block in the data processing method based on the second cause;

[0082] The data processing method is executed again, and the dedicated diagnostic log module is used to obtain the third cause of audio-visual asynchrony from the beginning of the next round of broadcast after the interruption.

[0083] The step of synchronously clearing the carryover state quantity in the data processing method obtained by the third cause;

[0084] The data processing method is executed again, and the fourth cause of the irregular fluctuation of synchronization error in the long-term continuous broadcast scenario is obtained by using the dedicated diagnostic log module.

[0085] The step in the fourth cause-based data processing method where the speech synthesis module segments and outputs audio data in an integer multiple of the video frame length during the audio data output stage;

[0086] The data processing method is executed again, and the dedicated diagnostic log module is used to obtain the fifth cause, which is that the lip movements on the screen stop before the speech stops and the phenomenon becomes more and more obvious as the number of sentences increases;

[0087] The step of determining whether an audio data block is the last sentence of the reply content in the data processing method obtained by the fifth cause.

[0088] Specifically, the complete investigation of the audio-visual asynchrony problem went through multiple iterations. Each iteration first constructed reproducible observation conditions, and then, based on the specific regularity of the observed phenomena (whether it is linearly cumulative, whether it only appears in a specific location, and whether it has regularity), proposed and verified diagnostic hypotheses. The specific investigation records are as follows:

[0089] Diagnostic method description: During the investigation, a dedicated diagnostic log and offline analysis tool were built. Each time the rendering module received an audio data block, it recorded the original byte length of the data block, the number of frames that participated in the generation of the current video frame after segmentation, the number of remaining sampling points carried over after segmentation, and whether it was a sentence end flag. It also recorded the number of audio bytes actually output downstream each time in conjunction with the speech synthesis module.

[0090] Based on the records, the difference between "cumulative audio sampling points" and "cumulative video frames × standard sampling points per frame (960)" (i.e., cumulative drift) is calculated offline. The drift is plotted against the broadcast time to observe the growth pattern (linear increase, step jump, or irregular fluctuation). Based on this, the nature and location of the problem in different rounds are distinguished, avoiding judgment based solely on subjective listening.

[0091] In the first round, actual testing revealed that in scenarios involving continuous long text broadcasts, the synchronization error between lip movements and speech gradually increased with the duration of the broadcast. The aforementioned cumulative drift plotted over time showed a stable linear increasing trend, and the increasing slope basically matched the arrival frequency of the audio data blocks. Based on this, it was determined that the error originated from the processing stage of each data block rather than from random factors. The diagnosis was that when audio data blocks were segmented according to video frame length, the remaining sampling points that were less than one video frame length (e.g., 960 sampling points, corresponding to a 24000Hz sampling rate and 25 frames per second) were directly padded with zeros to participate in the generation of the current video frame. This zero-padding operation introduced a fixed, small time error at the boundary of each data block (the upper limit of the time introduced by each zero-padding was the duration corresponding to one video frame, with a typical measured value on the order of tens of milliseconds), and this error accumulated linearly with the increase in the number of data blocks, consistent with the linear trend observed in the aforementioned drift plotting. After improving the method to use rounding down and carrying over the remaining sampling points to the next data block for unified processing, and then retesting with the same diagnostic method, the linear increasing trend of the cumulative drift disappeared, and the source of this error was confirmed and eliminated.

[0092] Second Round: After the first round of repairs, further testing revealed a slight audio-visual desynchronization at the end of each complete playback (usually within the duration of the last one or two video frames). This phenomenon only occurred at the end of each complete playback and did not amplify with the total playback duration, unlike the "block-by-block linear accumulation" pattern observed in the first round. Therefore, it was determined to be a localized issue in the end-processing stage rather than a problem with the carry-over mechanism itself. The diagnosis was that the end data block used the same upward rounding and zero-padding logic as the middle data blocks. After the carry-over mechanism took effect, the remaining sampling points in the end data block, even if insufficient for a full frame, were unconditionally padded to a complete frame of silence, resulting in unnecessary full-frame silence padding. The improvement was to use a rounding method to determine the actual number of ending frames for the end data block (i.e., padding to a full frame when the remaining sampling points are more than half a frame, and discarding those less than half a frame without generating additional silence frames). This source of error was confirmed and eliminated.

[0093] The third round: Further testing revealed that the next round of broadcasting immediately following an interruption exhibited audio-visual desynchronization from the very beginning. The magnitude of this desynchronization corresponded to the number of remaining sample points carried over from the previous round at the time of the interruption (the larger the carry-over remainder, the more pronounced the misalignment at the start of the next round). This was identified as a cross-round state contamination issue. The diagnosis was that the state variable used to store the remaining sample points carried over across data blocks was only reset once upon session establishment, and not synchronously reset each time an interruption signal occurred or a new round began. This resulted in the carry-over remainder from the previous round contaminating the alignment reference of the next round. The improvement was to synchronously clear this carry-over state variable when an interruption signal or generational identifier change was detected. This confirmed and eliminated the source of the error.

[0094] The fourth round (final location of the root cause of the signal): After completing the aforementioned three rounds of downstream repairs, irregular fluctuations in synchronization errors were still observed in long-term continuous broadcast scenarios. The cumulative drift plotted over time no longer showed the previous regular linear increase, but instead exhibited irregular jumps with no obvious period and sometimes positive and sometimes negative. Based on this, it was determined that the objects of the previous three rounds of repairs (the segmentation and transfer logic on the receiving side of the rendering module) had been confirmed to be effective, and the problem should be located in a more upstream link outside the receiving side of the rendering module. To this end, the scope of the diagnostic log recording was moved forward to the audio output interface of the speech synthesis module, recording the number of audio bytes actually sent each time. Statistics showed that these byte numbers were randomly distributed, had no fixed multiple relationship with each other, and were not integer multiples of the number of bytes corresponding to a single video frame (1920 bytes, corresponding to 960 sampling points and 16-bit quantization). Further investigation into the internal implementation of the speech synthesis module revealed the root cause: the module outputs the entire contents of the current buffer when the accumulated number of bytes in the internal buffer exceeds a preset threshold. The timing of this output depends on the random rhythm of the synthesis process itself, resulting in a random number of bytes sent each time. This means the data is not an integer multiple of the frame length from the signal source. Even with ideal integer frame alignment and carry-over processing downstream, only cumulative errors can be prevented; the irregular error components introduced by the incomplete overlap between the random byte count and the actual speech content segmentation points within a single output cannot be eliminated. The final solution was to segment the audio data output of the speech synthesis module into integer multiples of the video frame length (accumulated data in the buffer is first output in multiples of 1920 bytes; any remaining data less than a frame is stored in the module's internal buffer and merged with newly generated synthesized data before calculating the next integer multiple output). This ensures that each data block received downstream is an integer multiple of the frame length from the signal source. After the repair was implemented, the same drift amount was plotted and tested again. The drift amount curve in the long-term continuous broadcast scenario was stable and close to zero, and no longer accumulated or irregular jumps occurred. In multiple rounds of repeated tests with different text lengths, the synchronization problem was not reproduced again, which verified that this item was the key root cause repair to solve the problem of synchronization error accumulation. The other three rounds of downstream repair were retained as necessary supporting measures to eliminate low-order and local residual errors.

[0095] The fifth round (localizing special cases at sentence boundaries): After completing the aforementioned source repair, it was further discovered that when a complete response is streamed from a large language model into a dozen or more independent speech sentences, and each sentence is processed independently into an independent audio stream unit, the phenomenon of "the lip movements on screen stopping before the speech" still occurs. This phenomenon becomes more pronounced as the number of sentences increases (it is basically imperceptible in short responses with fewer sentences, but becomes noticeable after accumulating in long responses with a dozen or more sentences). After re-statistically analyzing the diagnostic logs by sentence dimension, it was found that the internal drift of each independent sentence has indeed approached zero (verifying that the fourth round of repair is effective within a single sentence). However, a fixed-order (tens of milliseconds) jump drift increment was observed at the junction of every two adjacent sentences, and this increment value is directly proportional to the total number of sentences. Based on this, it was determined that the problem does not occur in the signal processing within a single sentence, but rather in the logic of connecting sentences. The diagnosis revealed that the original implementation treated each sentence as an independent audio stream unit, executing a "zero-padding and clearing of carry-over remainders" logic at the end of each sentence (i.e., reusing the ending rules of a single complete response but incorrectly applying them to the end of each intermediate sentence). When a complete response contained more than a dozen sentences, this resulted in more than a dozen zero-padding operations, accumulating to cause significant audio-visual misalignment. The solution was to uniformly adjust the ending logic between sentences (excluding the final sentence of a complete response) to the same "integer frame alignment, remainder carry-over across sentences" method consistent with the intermediate processing logic of data blocks. The carry-over state was no longer cleared at the end of each non-final sentence; only when it was confirmed as the final sentence of a complete response was the actual zero-padding and remainder clearing performed. This ensured that continuous playback of multiple sentences was treated as a continuous audio stream at the signal processing level, and sentence boundaries were no longer treated specially, rather than being spliced ​​together from multiple independent audio segments. After the repair was implemented, the drift increment at the junction of adjacent sentences was re-counted at the sentence level, the jump phenomenon disappeared, and the audio-visual synchronization problem in long reply (a dozen sentences) scenarios was no longer reproduced in multiple rounds of actual tests, marking the final establishment of the complete closed loop of the integer frame alignment and remainder carry-over mechanism of the present invention.

[0096] The video rendering module includes a dedicated timing thread, which is independent of the threads for audio reception and model inference speed. The data processing system is also used for:

[0097] The dedicated timing thread calculates the target output time for each frame of image based on a high-precision monotonic clock, with the target frame rate configured by the system as the period.

[0098] The dedicated timing thread will output the corresponding video frame at the target output time;

[0099] For a frame of image, determine whether the time elapsed between the frame of image and the target output time is less than a preset value (e.g., 2ms). If so, the dedicated timing thread reads the high-precision monotonic clock at high frequency until the target output time and outputs the frame of image. Otherwise, the dedicated timing thread calls the operating system sleep function to reduce power consumption.

[0100] For each output moment, a dedicated timing thread determines whether there is a corresponding frame of image to be output at that output moment. If so, the image to be output is output. Otherwise, it determines whether the duration of the empty frame is less than the preset duration. If it is less than the preset duration, the last frame of the output image is played repeatedly. If it is greater than the preset duration, the preset standby loop image is output.

[0101] The output rhythm of the digital human video frames is controlled by a dedicated timing thread (frame rate metronome) independent of the audio reception and model inference speed. This thread calculates the target output time of each frame based on a high-precision monotonic clock, using the system's configured target frame rate (e.g., 25 frames per second) as the period. It then uses a hybrid scheduling strategy of "coarse waiting (sleep) combined with busy waiting (busy-spin)" to approximate the target time before outputting the frame, thus achieving frame output time stability that is superior to the operating system's default scheduling accuracy. This mechanism ensures that the actual output rhythm of the video frames is consistent with the actual duration of the audio, and is not directly affected by instantaneous fluctuations in the inference speed of the underlying image generation model (such as batch processing queuing, GPU usage fluctuations, etc.).

[0102] In each output cycle of the frame rate metronome, if there is a completed and ready-to-output "speaking" video frame (and its corresponding audio segment), the frame is output normally. If there is no ready-to-output "speaking" video frame in the current cycle due to short pauses between sentences, the present invention further determines: if the duration of the gap does not exceed a preset short gap threshold (e.g., 0.35 seconds), and there was a previous "speaking" state, the last "speaking" frame (i.e., hold-frame) is repeatedly output instead of switching to the preset standby screen, thereby avoiding screen jumps in normal short pause scenarios between sentences; if the duration of the gap exceeds the threshold, it is determined to be a true long period of silence, and the preset standby loop screen is switched.

[0103] The generation method includes:

[0104] The dedicated diagnostic log module is used to obtain the reasons why the video frame to be output cannot be obtained within the current output cycle;

[0105] Analyze the reasons why the video frames in the speech to be output cannot be obtained within the current output cycle, and obtain the steps of establishing a dedicated timing thread in the data processing method;

[0106] The dedicated diagnostic log module was used to obtain the cause of the long-term freezing in the open mouth state when the hold frame mechanism was triggered on the frame of the image where the jaw was open.

[0107] The analysis of the causes of prolonged frozen mouth state is based on the numerical value of the preset duration of the gap frame used for comparison in the data processing method.

[0108] In real-world testing of long-text continuous playback scenarios, it was observed that the screen periodically switched briefly to a standby screen and then returned to normal. Statistical analysis confirmed that the frequency of this phenomenon was positively correlated with the count of events where "no video frame could be retrieved during the current output cycle." Further investigation revealed that the speech feature extraction processing stage and the image generation model (denoising / decoding network) inference stage within the rendering pipeline shared the same underlying computational resource mutex. Under the condition of competition for this mutex, the measured single-frame processing time of the speech feature extraction stage (approximately 43 milliseconds) exceeded the single-frame processing time budget of the image generation consumption side (approximately 40 milliseconds), causing the feature supply rate to consistently fall below the consumption rate, resulting in a persistent underload of the rendering pipeline. The solution was to separate the mutexes for the speech feature extraction stage and the image generation model inference stage into two independent mutexes, allowing them to execute in parallel. After this fix, the feature supply rate recovered to exceed the consumption rate, and the count of "no video frame could be retrieved during the current output cycle" returned to zero during long-term playback tests.

[0109] The initial threshold (preset duration) for the gap duration in the short-gap hold-frame mechanism was set to 0.6 seconds. Actual testing revealed that triggering the hold-frame mechanism on a frame where the jaw is open resulted in an unnatural visual effect of "being frozen in an open mouth state for an extended period." Lowering this threshold to 0.35 seconds avoided the prolonged freeze-frame issue caused by the hold-frame mechanism, while ensuring that normal sentence breaks (based on statistical testing, the typical duration of a normal pause is comparable to this threshold) do not trigger unnecessary standby screen transitions. This 0.35-second value is the parameter for the embodiment determined by combining the aforementioned two sets of experimental data.

[0110] The video rendering module includes a multi-level buffer queue, and the data processing system is further used for:

[0111] Determine if the system has received an interruption signal. If so, clear the multi-level buffer queue and add an identifier to the global video rendering.

[0112] For each processing queue in the video rendering module, determine whether the identifier of the output data of the processing queue is the latest identifier. If it is, the output data enters the next processing queue; otherwise, the output data is discarded.

[0113] The digital human video rendering pipeline internally incorporates multi-level buffer queues, including an audio feature extraction queue, a model inference queue (such as a denoising / generating network) queue, and a result synthesis queue. These queues decouple the arrival rhythm of streaming audio from the processing rhythm of each internal processing stage. When the system receives an interruption signal, this invention performs a uniform clearing operation on all queues and increments a globally incrementing generation ID by one. Any intermediate data still in the processing chain carrying the old generation ID is identified as expired data and actively discarded when it flows to the next processing stage. Only new data matching the latest generation ID is accepted to continue participating in subsequent processing, thus ensuring that no lingering semi-finished audio and video data from before the interruption is mixed into the new round of broadcasting.

[0114] The generation method further includes:

[0115] The dedicated diagnostic log module is used to obtain the reason why a new round of responses cannot be played normally after the interruption;

[0116] Get the steps to perform a clear operation on a multi-level buffer queue and add an identifier globally for video rendering.

[0117] During the implementation of the queue clearing mechanism, two types of boundary situations requiring additional avoidance were discovered and addressed: First, if the interruption response logic clears all buffer queues within the rendering pipeline (including the final output routing queue for downstream real-time communication modules), it might mistakenly clear the new round of response data generated immediately after the interruption, causing the problem that "the new round of responses after the interruption cannot be played normally." The fix was to limit the clearing operation to intermediate buffer queues for internal processing stages (including feature extraction queues, model inference queues, and result synthesis queues), without affecting the queues responsible for the final output routing. Second, after testing, it was confirmed that performing a one-time queue clearing operation only at the moment of receiving the interruption signal is insufficient to cope with the objectively existing processing delay window between the interruption signal and the arrival of the new round of data—several processing stages within the rendering pipeline may still generate new intermediate data within this delay window, but which actually belong to the old round before the interruption. If not handled, this data will be mixed back into the queue after the clearing action. To address this, in addition to the one-time queue clearing, the aforementioned generational identifier linkage mechanism is further introduced. This enables each processing stage of the rendering pipeline to proactively identify and discard data carrying old generational identifiers at any stage of data flow, rather than relying solely on a one-time queue clearing operation at a specific moment. The complete mechanism has been verified through multiple rounds of continuous interruption tests on an actual deployed system, and the problem of mixed playback of new and old content after interruption has not recurred.

[0118] See Figure 1 Using the aforementioned data processing system for streaming voice-driven digital human video, this embodiment also provides a data processing method, including:

[0119] Step 100: Transmit the audio data blocks generated by the speech synthesis module in a streaming manner to the video rendering module;

[0120] Step 101: Based on the number of sampling points in the audio data block and the number of sampling points required for a single frame of video, the audio data block is segmented to obtain several video frames and the remaining sampling points.

[0121] Step 102: Store the remaining sampling points;

[0122] Step 103: Obtain the next audio data block and concatenate the remaining sampling points of the previous audio data block before the sampling points of the next audio data block.

[0123] Step 104: Determine if the audio data block is the last reply. If not, proceed to step 102; if so, proceed to step 105.

[0124] Step 105: Generate the final video frame by combining the remaining sampling points with the final silence.

[0125] The data processing method further includes:

[0126] The actual number of final frames is determined by rounding the audio data block at the end.

[0127] When an interruption signal or a change in the generation identifier is detected, the carryover status quantity is cleared synchronously.

[0128] The speech synthesis module segments the audio data output in multiples of the video frame length.

[0129] See Figure 2 The video rendering module includes a dedicated timing thread, which is independent of the threads for audio reception and model inference speed. The data processing method includes:

[0130] Step 200: The dedicated timing thread calculates the target output time for each frame of image based on a high-precision monotonic clock, with the target frame rate configured by the system as the period.

[0131] Step 201 includes:

[0132] Step 2011: For a frame of image, determine whether the time elapsed between the frame of image and the target output time is less than a preset value. If so, the dedicated timing thread reads the high-precision monotonic clock at high frequency until the target output time and executes step 2013. Otherwise, the dedicated timing thread calls the operating system sleep function to reduce power consumption and then executes step 2011 again.

[0133] Step 2012: For an output moment, a dedicated timing thread determines whether there is a corresponding frame of image to be output at the output moment. If yes, then 2013. Otherwise, determine whether the duration of the empty frame is less than the preset duration. If it is less than the preset duration, then repeat the playback of the last frame of the output image. If it is greater than the preset duration, then output the preset standby loop image.

[0134] Step 2013: The dedicated timing thread outputs the corresponding video frame image at the target output time.

[0135] The video rendering module includes a multi-level buffer queue, and the data processing method further includes:

[0136] Determine if the system has received an interruption signal. If so, clear the multi-level buffer queue and add an identifier to the global video rendering.

[0137] For each processing queue in the video rendering module, determine whether the identifier of the output data of the processing queue is the latest identifier. If it is, the output data enters the next processing queue; otherwise, the output data is discarded.

[0138] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. A data processing method for streaming voice-driven digital human video, characterized in that, The data processing method includes: The audio data blocks generated by the speech synthesis module in a streaming manner are transmitted to the video rendering module; Based on the number of sampling points in the audio data block and the number of sampling points required for a single frame of video, the audio data block is segmented to obtain several video frames and the remaining sampling points. Store the remaining sampling points; Obtain the next audio data block and append the remaining sampling points of the previous audio data block to the sampling points of the next audio data block.

2. The data processing method for streaming voice-driven digital human video as described in claim 1, characterized in that, The data processing method includes: Determine whether the audio data block is the last reply content. If not, store the remaining sampling points. If so, generate the final video frame together with the final silence.

3. The data processing method for streaming voice-driven digital human video as described in claim 1, characterized in that, The data processing method includes: For the last audio data block, the actual number of the last frame is determined by rounding. When an interruption signal or a change in the generation identifier is detected, the carryover status quantity is cleared synchronously. The speech synthesis module segments the audio data output in multiples of the video frame length.

4. The data processing method for streaming voice-driven digital human video as described in claim 1, characterized in that, The video rendering module includes a dedicated timing thread, which is independent of the threads for audio reception and model inference speed. The data processing method includes: The dedicated timing thread calculates the target output time for each frame of image based on a high-precision monotonic clock, with the target frame rate configured by the system as the period. The dedicated timing thread will output the corresponding video frame at the target output time.

5. The data processing method for streaming voice-driven digital human video as described in claim 4, characterized in that, The dedicated timing thread outputs the corresponding video frame image at the target output time, including: For a frame of image, determine whether the time elapsed between the frame of image and the target output time is less than a preset value. If so, the dedicated timing thread reads the high-precision monotonic clock at high frequency until the target output time and outputs the frame of image. Otherwise, the dedicated timing thread calls the operating system sleep function to reduce power consumption.

6. The data processing method for streaming voice-driven digital human video as described in claim 4, characterized in that, The dedicated timing thread outputs the corresponding video frame image at the target output time, including: For each output moment, a dedicated timing thread determines whether there is a corresponding frame of image to be output at that output moment. If so, the image to be output is output. Otherwise, it determines whether the duration of the empty frame is less than the preset duration. If it is less than the preset duration, the last frame of the output image is played repeatedly. If it is greater than the preset duration, the preset standby loop image is output.

7. The data processing method for streaming voice-driven digital human video as described in claim 1, characterized in that, The video rendering module includes a multi-level buffer queue, and the data processing method includes: Determine if the system has received an interruption signal. If so, clear the multi-level buffer queue and add an identifier to the global video rendering. For each processing queue in the video rendering module, determine whether the identifier of the output data of the processing queue is the latest identifier. If it is, the output data enters the next processing queue; otherwise, the output data is discarded.

8. The data processing method for streaming voice-driven digital human video as described in claim 1, characterized in that, The data processing method obtains the data through a generation method, the generation method including: A dedicated diagnostic log module is constructed to record the original byte length, the number of frames that participate in the generation of video frames after segmentation, the number of remaining sampling points that are carried over and retained after segmentation, and whether it is a sentence end flag bit each time the video rendering module receives an audio data block. On the speech synthesis module side, the number of audio bytes actually sent each time is recorded. The dedicated diagnostic log module is used to identify the first cause of the gradual increase in the synchronization error between lip movements and speech over the duration of broadcasting. The step in the data processing method that utilizes the first cause to carry over and retain the remaining sampling points to the next audio data block; The data processing method is executed again, and the second cause of the slight audio-visual desynchronization at the end of the segment is obtained using the dedicated diagnostic log module. The step in the data processing method using the second cause to determine the actual number of the last audio data block by rounding; The data processing method is executed again, and the dedicated diagnostic log module is used to obtain the third cause of audio-visual asynchrony from the beginning of the next round of broadcast after the interruption. The step of synchronously clearing the carryover state quantity in the data processing method using the third cause; The data processing method is executed again, and the fourth cause of the irregular fluctuation of synchronization error in the long-term continuous broadcast scenario is obtained by using the dedicated diagnostic log module; The step in the speech synthesis module of the fourth cause acquisition data processing method to segment and output audio data in an integer multiple of the video frame length during the audio data output stage; The data processing method is executed again, and the dedicated diagnostic log module is used to obtain the fifth cause, which is that the lip movements on the screen stop before the speech stops and the phenomenon becomes more and more obvious as the number of sentences increases; The step of determining whether an audio data block is the last sentence of the reply content in the data processing method using the fifth cause.

9. The data processing method for streaming voice-driven digital human video as described in claim 8, characterized in that, The generation method includes: The dedicated diagnostic log module is used to obtain the reasons why the video frame to be output cannot be obtained within the current output cycle; Analyze the reasons why the video frames in the speech to be output cannot be obtained within the current output cycle, and obtain the steps of establishing a dedicated timing thread in the data processing method; The dedicated diagnostic log module was used to obtain the cause of the long-term freezing in the open mouth state when the hold frame mechanism was triggered on the frame of the image where the jaw was open. The analysis of the causes of prolonged frozen mouth state is based on the numerical value of the preset duration of the gap frame used for comparison in the data processing method. The generation method further includes: The dedicated diagnostic log module is used to obtain the reason why a new round of responses cannot be played normally after the interruption; Get the steps to perform a clear operation on a multi-level buffer queue and add an identifier globally for video rendering.

10. A data processing system for streaming voice-driven digital human video, characterized in that, The data processing system is used to implement the data processing method for streaming voice-driven digital human video as described in any one of claims 1 to 9.

Images