A method and system for generating virtual humans of unlimited length driven by real-time streaming audio

Abstract

This invention belongs to the field of audio-driven virtual human generation technology, and discloses a real-time streaming audio-driven method and system for generating infinitely long virtual humans. It constructs a causal autoregressive generation mechanism based on frame latent variable blocks, ensuring that the generation of the current frame block depends only on historical frame blocks, a rolling reference frame, audio conditional information, and text conditional information. It also constructs a causal video coding mechanism to perform causal constraint coding on the video data during the training phase, ensuring that the latent variable at any given time is determined only by the current frame and its preceding frames. Furthermore, it constructs a long-term consistency coordination mechanism to suppress generation distribution drift through dynamically updated generation reference frames and to suppress inference pattern drift through rolling relative time position control. Finally, it constructs an autoregressive student model and uses a bidirectional teacher model and a pseudo-score model for distillation constraints, enabling the student model to approximate the generation distribution of the bidirectional diffusion model while maintaining the causal generation structure.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision, digital media technology and real-time communication technology, specifically involving the research and application of core technologies for real-time generation and streaming of virtual digital humans. Background Technology

[0002] Existing virtual human generation technologies based on diffusion models suffer from the following key problems:

[0003] ① Real-time fidelity dilemma: While large diffusion models (such as those with 14-bit parameters) can provide high visual detail, they rely on sequential denoising computation, resulting in extremely high inference latency. Existing technologies often sacrifice model capacity (such as using a 1.3-bit lightweight model) or perform aggressive quantization to meet real-time requirements (≥20 FPS), leading to a severe decrease in visual fidelity; while solutions that maintain model capacity (such as WanS2V and OmniAvatar) only achieve frame rates of 0.16-0.25 FPS, which cannot meet the needs of interactive scenarios.

[0004] ② Lack of identity consistency over long periods: Existing methods are prone to problems such as identity drift, color shift, and temporal instability when generating data over long periods (e.g., minutes or more). This is because the conditional patterns in the inference phase deviate from those in the training phase (inference pattern drift), and the distribution of generated frames gradually deviates from the distribution of real videos (distribution drift), resulting in a cumulative degradation of visual quality over time.

[0005] ③ Insufficient streaming adaptation: Traditional diffusion models are non-causal structures and cannot directly support streaming generation; existing streaming adaptation methods (such as CausVid) achieve streaming generation through distillation and KV caching, but there are overexposure artifacts and it is difficult to balance large model capacity, high fidelity and real-time performance.

[0006] ④ High computational and bandwidth costs, and poor adaptability: Traditional virtual digital human generation technology adopts a full-frame rendering generation mode, which means that a complete rendering calculation is performed on each frame of the digital human image, resulting in an average inference time of 50-100ms per frame. Taking a 1080p resolution, 30fps video stream as an example, the bandwidth requirement for raw image data transmission is about 1.5Gbps. Such high computing power requirements are difficult for ordinary hardware devices to handle, and the high bandwidth consumption far exceeds the carrying capacity of ordinary network environments, making it difficult for small and medium-sized enterprises and individual users to conveniently apply this type of technology, thus limiting the scope of technology's popularization.

[0007] ⑤ The contradiction between real-time performance and latency is prominent, failing to meet interactive needs: In real-time interactive scenarios, low latency is a core indicator for ensuring user experience, typically requiring end-to-end latency of ≤200ms. However, existing encoding schemes often use a large number of B-frames for encoding to improve compression rates. The bidirectional prediction characteristics of B-frames introduce an additional decoding latency of ≥300ms, resulting in delayed audio and video interactive feedback. This severely impacts the user experience in scenarios such as real-time interaction in virtual anchor live streaming and instant response in digital human customer service, becoming a key bottleneck restricting the implementation of real-time interactive digital human applications.

[0008] Existing related technologies include Hallo3 (5B parameters, 0.26 FPS), StableAvatar (1.3B parameters, 0.64 FPS), OmniAvatar (14B parameters, 0.16 FPS), and WanS2V (14B parameters, 0.25 FPS), but none of them can simultaneously solve the three core problems of real-time performance, long-term consistency, and high fidelity.

[0009] The latest publicly available technology that comes closest to solving the three core problems, Live Avatar, proposes a system framework for real-time audio-driven virtual human generation. This method, based on a large-scale diffusion model, achieves real-time generation and long-sequence consistency to some extent through distributed parallel inference and a reference frame mechanism. However, it still relies on an autoregressive sampling process, leading to high inference latency and insufficient pipeline resource utilization. Furthermore, it suffers from identity drift and insufficient long-term consistency maintenance when generating extremely long sequences. These problems severely restrict the performance and practical deployment capabilities of high-fidelity, low-latency, and infinitely long real-time audio-driven virtual human generation. Therefore, a solution that can overcome the sequential denoising bottleneck, improve inference efficiency, and simultaneously enhance long-term consistency is urgently needed. Summary of the Invention

[0010] To address the problems existing in the prior art, this invention provides a method and system for generating virtual humans of unlimited length driven by real-time streaming audio.

[0011] The technical solution of this invention focuses on solving key technical problems such as dynamic generation, real-time encoding and decoding transmission, and long-term identity consistency assurance of virtual digital humans in audio-driven scenarios. It can be widely applied to various scenarios that require low latency, high identity consistency, and unlimited-length streaming output, such as virtual anchor live streaming, digital human interactive customer service, real-time remote digital human interaction, online education virtual lecturers, virtual conference digital human participation, and metaverse real-time digital human interaction. It can achieve end-to-end real-time mapping and stable transmission from audio signals to digital human video streams, providing core technical support for various real-time interactive scenarios of digital humans.

[0012] The specific technical implementation of this invention, a method for generating an infinitely long virtual human driven by real-time streaming audio, includes:

[0013] Step 1: Construct a causal autoregressive generation mechanism based on frame latent variable blocks, so that the generation of the current frame block depends only on historical frame blocks, rolling reference frames, audio conditional information and text conditional information, thereby avoiding future information from participating in the generation of the current frame.

[0014] Step 2: Construct a causal video coding mechanism to encode the video data during the training phase with causal constraints, so that the latent variable at any time is determined only by the current frame and the frames before it, in order to ensure that the training distribution is consistent with the streaming inference distribution.

[0015] Step 3: Construct a long-term consistency coordination mechanism. Suppress generation distribution drift by dynamically updating the generation reference frame and suppress inference mode drift by rolling relative time position control, so that identity consistency and temporal continuity remain stable simultaneously during the long-term generation process.

[0016] Step 4: Construct an autoregressive student model and apply distillation constraints through a bidirectional teacher model and a pseudo-score model. This allows the student model to approximate the generation distribution of the bidirectional diffusion model while maintaining the causal generation structure, thus balancing generation quality and real-time reasoning ability.

[0017] Furthermore, the causal autoregressive generation mechanism adopts a block-level structure, dividing the video latent variables into multiple consecutive frame blocks. Each frame block is generated using only the frame block information from the previous moment and even earlier moments as conditional inputs.

[0018] Furthermore, the long-term consistency coordination mechanism includes an adaptive attention benchmark mechanism and a rolling relative position control mechanism. The adaptive attention benchmark mechanism uses the first frame in the generated video as the new generation benchmark frame to suppress distribution drift, while the rolling relative position control mechanism maintains the relative temporal consistency between the training phase and the inference phase by continuously shifting the time position.

[0019] Another object of the present invention is to provide a parallel reasoning method for real-time streaming audio-driven virtual human generation, including the following cooperative mechanism:

[0020] The multiple time steps in the denoising process are fixedly allocated to multiple parallel computing units, so that each computing unit executes the same denoising step for a long time, thereby transforming the sequential denoising process into a spatial pipeline process.

[0021] In the initial stage, a sequential filling method is used to allow the latent variables to pass through all computational units in turn to complete the first denoising. In the stable stage, each computational unit only processes the corresponding step and passes the intermediate latent variables to the next computational unit.

[0022] The conditional buffer of historical frame blocks is kept consistent with the noise scale of the current denoising state, thereby ensuring that the conditional distribution during the inference phase is consistent with the conditional distribution during the training phase.

[0023] Furthermore, historical frame block conditional caching is reused locally, and the video decoding process is offloaded from the main denoising pipeline to an independent execution unit to improve overall throughput.

[0024] Furthermore, the initial stage of the pipeline is executed only once, and its time consumption is less than the single-step execution time of the stable pipeline stage.

[0025] Another object of the present invention is to provide a real-time streaming audio-driven virtual human generation system of unlimited length, comprising:

[0026] Audio analysis and control module: As the system input layer, it is responsible for receiving real-time streaming audio signals, extracting core control parameters through multi-algorithm fusion, and providing precise instructions for digital human generation.

[0027] Digital Human Generation Module: Based on the control parameters output by the audio parsing module, it adopts a hybrid generation mode of "keyframe + P-frame" to achieve efficient generation of keyframes and lightweight P-frames for complete digital humans.

[0028] Encoding / decoding module: Employs dedicated encoding / decoding configuration and layered encoding strategy to efficiently encode and compress the generated digital human image data, while ensuring a fast and stable decoding process.

[0029] Real-time transmission module: Based on optimized transmission protocols and synchronization mechanisms, it achieves low-latency and high-reliability transmission of encoded data, adapting to different network environments.

[0030] Decoding and Reconstruction Module: As the system output layer, it is responsible for receiving transmitted data, completing decoding and image reconstruction, and realizing precise synchronous playback of digital human video streams and audio signals.

[0031] Another object of the present invention is to provide a computer device including a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the real-time streaming audio-driven infinite-length virtual human generation method.

[0032] Another object of the present invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the real-time streaming audio-driven infinite-length virtual human generation method.

[0033] Another objective of this invention is to provide an information data processing terminal for implementing the real-time streaming audio-driven infinite-length virtual human generation system.

[0034] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0035] Core problems of existing technologies and the targeted solutions of this invention:

[0036] Existing technologies suffer from five major pain points in real-time streaming, unlimited-length virtual human generation scenarios. This invention, based on the research and development concept of "identity consistency as the core, real-time performance as the premise, and efficient compression as the goal," forms a precise and quantifiable solution through end-to-end collaborative optimization of "generation-encoding-transmission-reconstruction," as detailed below:

[0037] Technical solutions to address the issues of "excessive computing and bandwidth costs and poor adaptability".

[0038] The fundamental pain point of existing technologies is that traditional full-frame rendering mode results in an average frame inference time of 50-100ms, and the original bandwidth requirement for 1080p@30fps video reaches 1.5Gbps, which cannot be supported by ordinary hardware (such as home computers) and networks (such as 5Mbps broadband), thus limiting the popularization of the technology.

[0039] Solution of this invention:

[0040] Innovative Lightweight Generation Architecture: Breaking away from the full-frame rendering paradigm, this architecture employs a hybrid mode of "full keyframe generation + lightweight P-frame prediction." Keyframes are triggered only at GOP initiation, identity changes, and packet loss recovery (once every 30 frames), rendering all features (facial texture, facial contours, etc.) based on the Diffusion / GAN model, with an average frame time of ≤50ms. P-frames do not require full rendering; they only generate three layers of lightweight data: "motion vectors + feature residuals + control parameters," with an average frame time of ≤10ms. This architecture directly reduces computational costs by 70-90% (in R&D testing, the computational cost of traditional technology was 100%, while this invention's was only 20%, with keyframes accounting for 15% and P-frames for 5%).

[0041] Layered encoding and structured compression: The base layer encodes core data (identity features, key motion vectors) at a bitrate of 2-4 Mbps; the enhancement layer encodes detail residuals at a bitrate of 1-2 Mbps (for transmission when bandwidth is sufficient). Combined with P-frame structured design (adapting to the motion patterns of digital humans, such as smooth head rotation and fixed blink cycles), compression efficiency is further improved. R&D testing shows that in a 1080p@30fps scenario, the base layer bandwidth required by this invention is only 2-3 Mbps, a 30-50% reduction compared to the 20 Mbps bandwidth requirement of traditional compression techniques.

[0042] Breakthrough in Adaptability:

[0043] A typical mid-range server (Intel Core i7-12700K + RTX 3080) can support real-time generation, and a bandwidth of 5Mbps or more can meet the basic layer transmission requirements, making it convenient for small and medium-sized enterprises and individual users to use. This solves the core problems of traditional technologies such as "high hardware threshold and poor network compatibility".

[0044] A technical approach to address the prominent contradiction between real-time performance and latency, which fails to meet interactive needs.

[0045] The fundamental pain points of existing technologies are: bidirectional B-frame prediction introduces a decoding delay of ≥300ms, long GOP (≥2 seconds) buffering leads to latency accumulation, and end-to-end latency is generally ≥300ms, which cannot meet the requirements for real-time interaction (requirement ≤200ms).

[0046] This invention's solution (technical solution + R&D data support):

[0047] Encoding and decoding configuration optimization: Design a "short GOP + I-frame / P-frame combination" architecture, disable B-frames to completely eliminate bidirectional prediction latency; the default GOP length is "frame rate × 1 second" (30 frames in a 30fps scene) to avoid long GOP latency accumulation.

[0048] Frame-level pipelined transmission: It adopts a synchronous execution mode of "generating 1 frame → encoding 1 frame → transmitting 1 frame", without GOP-level buffering. The transmission prioritizes the WebRTC protocol (end-to-end transmission latency ≤100ms) and is compatible with RTMP / HTTP-FLV protocols to adapt to multiple scenarios.

[0049] Latency Quantification Control: Through R&D testing and verification, the latency of each part of the system is strictly controllable: audio parsing ≤30ms, keyframe generation and encoding ≤50ms, P-frame generation and encoding ≤10ms, transmission latency ≤100ms, decoding and reconstruction ≤20ms, and the final end-to-end latency is controlled within 200ms (185ms in Example 1 and 135ms in Example 2), which is ≥35% higher than the existing technology and fully meets the real-time interactive needs of virtual anchor live broadcasts, digital human customer service, etc.

[0050] A technical approach to address the issue of "lack of consistency in identity over long periods and insufficient stability".

[0051] The fundamental pain point of existing technologies is that the accumulation of inter-frame prediction errors leads to the gradual deformation of digital human facial features, textures, and contours, resulting in long-term (≥10 hours) identity consistency of ≤85%, which cannot support scenarios such as 7×24-hour live streaming.

[0052] This invention's solution (technical solution + R&D data support):

[0053] Encoding layer identity anchoring mechanism: A 128-dimensional identity feature vector is embedded in the I-frame as the identity benchmark for all frames in the GOP; the P-frame carries an identity verification sub-vector, and the similarity with the identity features of the key frame is verified in real time (threshold ≥ 95%).

[0054] Global identity verification and reset: A global verification is triggered every 10 GOPs (approximately 10 seconds). If the similarity is less than 95%, an IDR frame is forcibly inserted, the reference buffer is cleared, and the keyframe is re-rendered.

[0055] Stability verification: In the R&D test, the identity consistency retention rate of the present invention reached 98.8% in the 12-hour live broadcast scenario of virtual anchor (Example 1), and 98.2% in the 8-hour digital human interactive customer service scenario (Example 2). Compared with the existing technology's 82% consistency in 12 hours, it is improved by ≥15%, which fundamentally eliminates feature drift and deformation in long-term generation.

[0056] Technical approaches to address "insufficient error propagation and fault tolerance, and weak anti-interference capabilities".

[0057] The fundamental pain point of existing technologies is that the long GOP (≥2 seconds) design causes errors to propagate along the inter-frame prediction chain after packet loss, resulting in an error recovery time of ≥2 seconds and extremely poor user experience in network fluctuation scenarios.

[0058] This invention's solution (technical solution + R&D data support):

[0059] Short GOP dynamic adjustment: The GOP length can be adaptively switched between "frame rate × 0.5 seconds" and "frame rate × 1 second", limiting the error propagation range to 1 GOP (maximum 1 second, minimum 0.5 seconds).

[0060] Dynamic I-frame density and frame-level retransmission: When network bandwidth is <5Mbps or packet loss rate is ≥3%, the GOP is shortened to 0.5 seconds, and one I-frame is inserted every 5 frames; when packets are lost, only the last 3 critical P-frames are retransmitted to avoid increased latency caused by a large number of retransmissions.

[0061] Fault tolerance data verification: R&D tests show that the error recovery time of this invention is ≤0.5 seconds (only 0.25 seconds in Example 2), which is ≥87% better than the recovery cycle of ≥2 seconds in the prior art. It can still maintain stable playback even in unstable environments such as mobile networks.

[0062] Technical solutions to address the issues of "low audio-visual synchronization accuracy and lack of interactive realism".

[0063] The fundamental pain point of existing technologies is the lack of a frame-level synchronization mechanism, resulting in misalignment between lip movements, facial expressions, and speech, with a synchronization error of ≥50ms, leading to poor user immersion.

[0064] This invention's solution (technical solution + R&D data support):

[0065] Triple synchronization design: ① Each frame of data carries a timestamp with an accuracy of ≤1ms to establish a timing mapping between audio and video frames; ② The SEI frame carries facial expression parameters and timestamp calibration information to assist the decoding end in synchronization; ③ The P-frame control parameter layer corresponds one-to-one with the control parameters of the audio parsing module to ensure that lip movements, facial expressions and speech are matched frame by frame.

[0066] Synchronization accuracy verification: In the R&D test, the synchronization error of the virtual anchor scene is ≤8ms (Example 1) and the digital human customer service scene is ≤7ms (Example 2). Compared with the existing technology with an error of ≥50ms, the optimization is ≥85%. Users cannot perceive the misalignment at all, which significantly improves the realism of the interaction.

[0067] The inventive technical effects of this invention:

[0068] The technical effect of this invention is not a simple combination of existing technologies, but a deep cross-innovation based on the motion laws of digital humans, encoding and decoding principles, and real-time transmission characteristics. Its creativity is reflected in the following five dimensions, all of which have been verified by actual test data:

[0069] Breakthrough in efficiency: Significantly reducing the barriers to technology application;

[0070] Computing costs are reduced by 70-90%, enabling ordinary hardware such as terminal servers and home computers to carry out real-time generation tasks without the need for professional high-performance computing clusters.

[0071] Bandwidth consumption is reduced by 30-50%, with the basic layer requiring only 2-4Mbps bandwidth. It is compatible with ordinary broadband and mobile networks of 5Mbps and above, promoting the virtual digital human technology from "professional fields" to "mass applications" and providing low-cost solutions for SMEs and individual users.

[0072] Real-time performance assurance: Overcoming core bottlenecks in interactive scenarios;

[0073] End-to-end latency is strictly controlled within 200ms, with a minimum of 135ms, meeting the low latency requirements of scenarios such as virtual anchor live streaming, digital human customer service, and real-time remote interaction, and solving the core pain point of traditional technology that "can generate but cannot interact".

[0074] Frame-level pipelined transmission and short GOP design enable a buffer-free, uninterrupted end-to-end "generation-encoding-transmission-playback" process, ensuring real-time interactive feedback and enhancing user experience.

[0075] Stability breakthrough: Expanding the application boundaries to an infinite length;

[0076] The identity consistency retention rate is ≥98% over long periods (≥10 hours), with no issues such as facial deformation or feature jumps. It is the first time that virtual human generation with unlimited length for 7×24 hours has been achieved, expanding the application boundaries of virtual digital humans in long-term scenarios such as live streaming, online education, and virtual meetings.

[0077] The coding layer identity anchoring and global verification mechanism solves the problem of inter-frame error accumulation from the root, and its technical stability far exceeds that of existing technologies (≤85% consistency), providing reliable support for large-scale commercial applications.

[0078] Enhanced fault tolerance: adaptable to complex network environments;

[0079] Error propagation range ≤ 0.5 seconds, error recovery time ≤ 0.5 seconds, significantly improving fault tolerance compared to the recovery cycle of several seconds in traditional technologies;

[0080] Dynamic GOP adjustment and frame-level retransmission mechanism adapt to complex scenarios such as mobile networks and bandwidth fluctuations, ensuring stable playback even in environments with a packet loss rate of ≤3%, thus improving system robustness.

[0081] Optimized synchronization accuracy: Enhances interactive immersion;

[0082] The audio and video synchronization error is ≤10ms, and the lip movements, facial expressions and voice are accurately matched to achieve the perception standard of natural interaction between "human-machine-environment";

[0083] Layered coding and SEI frame synchronization mechanism take into account both basic quality and detail optimization, ensuring synchronization under low bandwidth and improving image detail under high bandwidth, achieving a dual balance of "precision and quality".

[0084] Creative summary: The non-obviousness and irreplaceability of the technical solution.

[0085] Breakthrough in technical approach: Breaking away from the traditional fixed paradigm of "full-frame generation + long GOP + B-frame encoding", a hybrid generation mode of "key frame anchoring + non-key frame lightweight prediction" is proposed. Combined with collaborative strategies such as short GOP, disabling B-frames, and layered encoding, it is a cross-innovation of multiple technical fields (computer vision, encoding and decoding, real-time transmission) and cannot be achieved by conventional adjustments by those skilled in the art.

[0086] Synergy of the solution: The innovations are closely related and deeply collaborative (e.g., the lightweight generation architecture relies on the P-frame structured design, and the dynamic GOP adjustment relies on data feedback from the encoding / decoding and transmission modules), forming a closed loop through extensive experimental iterations and optimizations, rather than a simple superposition of isolated technologies;

[0087] Irreplaceable effects: Core indicators such as 70-90% reduction in computing costs, latency ≤200ms, and long-term identity consistency ≥98% cannot be achieved through partial improvements to existing technologies (e.g., traditional technologies struggle to control the transmission bandwidth of 1080p@30fps to within 10Mbps, while the base layer of this invention only requires 2-4Mbps), demonstrating significant technological advantages;

[0088] With wide-ranging applications, it is compatible with multiple fields such as virtual anchors, interactive customer service, and online education, solving common pain points in different scenarios, promoting the industrial application of virtual digital human technology, and has significant economic and social value.

[0089] In summary, this invention systematically solves the five core problems of existing technologies through end-to-end technological innovation. Its creative technological effect is not only reflected in the quantitative improvement of key indicators, but also in breaking through the technical bottlenecks of real-time, long-term, and low-cost application of virtual digital humans, providing core support for the technological upgrade of the digital media and real-time communication fields.

[0090] The technical solution of this invention fills a technological gap in the industry both domestically and internationally:

[0091] Existing virtual digital human generation technologies, both domestically and internationally, suffer from the shortcoming of "single-index optimization and insufficient end-to-end coordination." No end-to-end solution has yet emerged that can simultaneously satisfy "low latency (≤200ms), long-term identity consistency (≥98%), low computation / bandwidth costs (computation reduced by 70-90%, bandwidth reduced by 30-50%), high fault tolerance (error recovery ≤0.5 seconds), and high-precision audio and video synchronization (error ≤10ms)." This invention precisely fills this technological gap, specifically manifested in:

[0092] ① Limitations of existing technologies at home and abroad:

[0093] At the international level: Virtual human technologies from companies such as Meta and Google focus on image quality and generation efficiency, but suffer from problems such as long-term identity drift (consistency ≤85%) and excessively high latency (≥300ms); professional encoding and decoding technologies (such as H.265 / H.266) are not optimized for the movement patterns of digital humans, and bandwidth consumption is still high (≥10Mbps), making them unsuitable for real-time streaming scenarios.

[0094] At the domestic level: mainstream platforms such as Unity+Live2D and NetEase Yaotai adopt the "full-frame rendering + long GOP + B-frame encoding" mode, which has high computation and bandwidth costs and has not established an effective identity anchoring mechanism, resulting in insufficient stability for long-term generation; some lightweight solutions have reduced costs, but sacrificed real-time performance and consistency, and cannot meet the needs of interactive scenarios.

[0095] ② The core basis for this invention to fill the gap:

[0096] This invention is the first to construct a collaborative optimization system for the entire "generation-encoding-transmission-reconstruction" chain, filling a gap through the following groundbreaking designs:

[0097] For the first time, it achieves a deep integration of "keyframe anchoring + P-frame lightweight prediction" and "short GOP + layered coding", simultaneously solving the three core pain points of computational cost, bandwidth consumption, and real-time performance. There are no similar technical solutions disclosed at home and abroad.

[0098] For the first time, a triple identity guarantee mechanism was proposed, consisting of 128-dimensional identity feature vector embedding in the coding layer, P-frame identity verification sub-vector, and IDR frame forced refresh, which enables identity consistency of ≥98% over long periods (≥10 hours), breaking through the technical bottleneck of "inter-frame error accumulation" in existing technologies.

[0099] For the first time, a triple synchronization mechanism of "frame-level timestamp (accuracy ≤1ms) + SEI frame synchronization + control parameter timing mapping" has been implemented, with audio and video synchronization error ≤10ms, far exceeding the synchronization accuracy of existing technologies ≥50ms;

[0100] No technical solution has been found in domestic and international patents, technical documents and commercial products that simultaneously meet the requirements of "latency ≤200ms, consistency ≥98%, bandwidth ≤4Mbps, and error recovery ≤0.5 seconds". The end-to-end optimization design of this invention fills the technical gap in this field. Attached Figure Description

[0101] Figure 1 This is a flowchart of a method for generating an infinitely long virtual human driven by real-time streaming audio, provided in an embodiment of the present invention.

[0102] Figure 2 This is a block diagram of a real-time streaming audio-driven infinite-length virtual human generation system provided in an embodiment of the present invention.

[0103] Figure 3 This is an end-to-end delay comparison chart provided in an embodiment of the present invention.

[0104] Figure 4 This is a long-term identity consistency change diagram provided in an embodiment of the present invention.

[0105] Figure 5 This is a comparison chart of bandwidth consumption provided in an embodiment of the present invention.

[0106] Figure 6 This is a diagram illustrating the composition and optimization of computational costs provided in an embodiment of the present invention. Detailed Implementation

[0107] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0108] like Figure 1 As shown, the real-time streaming audio-driven method for generating an infinitely long virtual human provided in this embodiment of the invention includes:

[0109] S101, Construct a causal autoregressive generation mechanism based on frame latent variable blocks, so that the generation of the current frame block depends only on historical frame blocks, rolling reference frames, audio conditional information and text conditional information, thereby avoiding future information from participating in the generation of the current frame;

[0110] S102, Construct a causal video coding mechanism to perform causal constraint coding on the video data during the training phase, so that the latent variable at any time is determined only by the current frame and the frames before it, so as to ensure that the training distribution is consistent with the streaming inference distribution.

[0111] S103, construct a long-term consistency coordination mechanism, suppress generation distribution drift by dynamically updating the generation reference frame, and suppress inference mode drift by rolling relative time position control, so that identity consistency and temporal continuity remain stable simultaneously during long-term generation.

[0112] S104 constructs an autoregressive student model and uses a bidirectional teacher model and a pseudo-score model for distillation constraints, so that the student model can approximate the generation distribution of the bidirectional diffusion model while maintaining the causal generation structure, thus balancing generation quality and real-time reasoning ability.

[0113] The causal autoregressive generation mechanism provided in this embodiment of the invention adopts a block-level structure, which divides the video latent variables into multiple consecutive frame blocks. Each frame block is generated by using only the frame block information from the previous time step and earlier time steps as conditional input.

[0114] The long-term consistency coordination mechanism provided in this embodiment of the invention includes an adaptive attention benchmark mechanism and a rolling relative position control mechanism. The adaptive attention benchmark mechanism uses the first frame in the generated video as the new generation benchmark frame to suppress distribution drift, while the rolling relative position control mechanism maintains the relative temporal consistency between the training phase and the inference phase by continuously shifting the time position.

[0115] This invention provides a parallel inference method for generating virtual humans driven by real-time streaming audio, comprising the following collaborative mechanism:

[0116] The multiple time steps in the denoising process are fixedly allocated to multiple parallel computing units, so that each computing unit executes the same denoising step for a long time, thereby transforming the sequential denoising process into a spatial pipeline process.

[0117] In the initial stage, a sequential filling method is used to allow the latent variables to pass through all computational units in turn to complete the first denoising. In the stable stage, each computational unit only processes the corresponding step and passes the intermediate latent variables to the next computational unit.

[0118] The conditional buffer of historical frame blocks is kept consistent with the noise scale of the current denoising state, thereby ensuring that the conditional distribution during the inference phase is consistent with the conditional distribution during the training phase.

[0119] The historical frame block condition cache provided in this embodiment of the invention is reused locally, and the video decoding process is offloaded from the main denoising pipeline to an independent execution unit to improve the overall throughput.

[0120] The initial stage of the pipeline provided in this embodiment of the invention is executed only once, and its time consumption is less than the single-step execution time of the stable pipeline stage.

[0121] like Figure 2 As shown, an embodiment of the present invention provides a real-time streaming audio-driven infinite-length virtual human generation system, comprising:

[0122] Audio analysis and control module: As the system input layer, it is responsible for receiving real-time streaming audio signals, extracting core control parameters through multi-algorithm fusion, and providing precise instructions for digital human generation.

[0123] Digital Human Generation Module: Based on the control parameters output by the audio parsing module, it adopts a hybrid generation mode of "keyframe + P-frame" to achieve efficient generation of keyframes and lightweight P-frames for complete digital humans.

[0124] Encoding / decoding module: Employs dedicated encoding / decoding configuration and layered encoding strategy to efficiently encode and compress the generated digital human image data, while ensuring a fast and stable decoding process.

[0125] Real-time transmission module: Based on optimized transmission protocols and synchronization mechanisms, it achieves low-latency and high-reliability transmission of encoded data, adapting to different network environments.

[0126] Decoding and Reconstruction Module: As the system output layer, it is responsible for receiving transmitted data, completing decoding and image reconstruction, and realizing precise synchronous playback of digital human video streams and audio signals.

[0127] The real-time streaming audio-driven infinite-length virtual human generation system provided in this embodiment of the invention operates by executing audio input, control parameter extraction, virtual human image generation, encoding transmission, and decoding reconstruction in sequence, forming a complete data processing closed loop, and realizing real-time driving and synchronous output of audio signals to the virtual human video stream.

[0128] During system operation, the audio parsing and control module first receives real-time streaming audio signals from external input. This module preprocesses and analyzes the received audio signals and extracts control parameters for driving the virtual human from the audio through multi-algorithm fusion. These control parameters characterize information such as speech content, pronunciation rhythm, pitch changes, and emotional state, and serve as input to the subsequent digital human generation module, thereby establishing a mapping relationship between the audio signals and the virtual human's visual performance.

[0129] Subsequently, the digital human generation module performs the generation of virtual human image sequences based on the control parameters. This module uses a hybrid generation mechanism of "keyframe + P-frame," periodically generating keyframes containing complete virtual human state information along the timeline, and inserting P-frames predicted based on temporal continuity between the keyframes to maintain continuous changes in the virtual human's movements and expressions. This mechanism ensures a natural and coherent virtual human generation effect while effectively reducing overall computational load and data redundancy, enabling the system to support continuous output of infinitely long video streams.

[0130] The generated virtual human image data is transmitted to the encoding / decoding module for encoding processing. This module employs a layered encoding strategy and dedicated encoding / decoding parameters to perform differentiated compression on keyframes and P-frames, thereby reducing the bandwidth resources required for data transmission while ensuring video quality, and providing stable data format support for subsequent network transmission and decoding playback.

[0131] The encoded data stream is sent to the terminal by the real-time transmission module. Based on an optimized transmission protocol and synchronization control mechanism, the real-time transmission module adaptively adjusts the transmission process according to changes in network bandwidth and latency to ensure the continuity and reliability of the data stream under different network environments, and maintains the time synchronization between the audio and video streams to avoid audio-visual asynchrony.

[0132] The decoding and reconstruction module on the terminal side receives the encoded data from the real-time transmission module, decodes and reconstructs it into a playable video image sequence, and outputs it synchronously with the corresponding audio signal, thereby presenting a virtual human video stream that matches the audio content in real time on the information data processing terminal.

[0133] Through the collaborative work of the above modules, this invention achieves continuous, low-latency, and highly stable output of real-time audio-driven virtual human generation, solving the problems of limited virtual human generation length, high computational burden, and difficulty in ensuring audio-visual synchronization in the prior art. This improves the usability and engineering practical value of the virtual human system in real-time interaction and long-term operation scenarios.

[0134] To address the shortcomings of existing virtual human generation technologies in terms of real-time performance, continuity, temporal synchronization, and computational efficiency, this invention proposes a real-time streaming audio-driven method and system for generating virtual humans of unlimited length as an optimization solution. This system is designed for real-time interactive and long-term stable output scenarios. Through multi-module collaboration, it forms an end-to-end closed-loop processing link, realizing an automated processing flow from real-time audio input to virtual human video stream output. Its overall system architecture is as follows: Figure 2 As shown.

[0135] This system comprises five core functional units: an audio analysis and control module, a digital human generation module, an encoding and decoding module, a real-time transmission module, and a decoding and reconstruction module. Each module works collaboratively according to a predetermined data flow and control flow relationship.

[0136] The audio parsing and control module, serving as the system input layer, receives real-time streaming audio signals from external inputs and performs preprocessing, framing, and feature extraction operations on the audio signals to parse out multi-dimensional control parameters related to the virtual human's driving mechanism. These control parameters include at least lip-sync parameters corresponding to the speech content, timing control parameters corresponding to the speech rhythm, and facial expression control parameters related to emotion or intonation. These parameters are then sent to the subsequent digital human generation module through a unified control interface, thereby achieving audio-to-visual behavior mapping control.

[0137] The digital human generation module generates virtual human image sequences according to a preset generation strategy based on the control parameters output by the audio parsing and control module. This module employs a hybrid generation mechanism of "keyframes + prediction frames (P-frames)," where keyframes express complete virtual human state information, and prediction frames are used for lightweight compensation generation based on temporal continuity between keyframes. This ensures the continuity and naturalness of virtual human movements while reducing the overall computational load and data size, enabling stable generation of infinitely long video streams.

[0138] The encoding / decoding module encodes and compresses the image data output by the digital human generation module. This module employs layered encoding and dedicated parameter configuration to perform differentiated encoding on keyframes and prediction frames, thereby reducing transmission bandwidth requirements while maintaining visual quality and improving the stability and real-time performance of subsequent decoding processes.

[0139] The real-time transmission module is used to transmit the encoded data stream in the network environment. This module achieves reliable transmission of encoded data under different network bandwidth and latency conditions through an adaptive transmission strategy and synchronization control mechanism, and maintains time synchronization between the audio and video streams, avoiding audio-visual asynchrony or stuttering during playback.

[0140] The decoding and reconstruction module, as the system output layer, receives the data stream sent by the real-time transmission module, decodes and reconstructs it into a playable video image sequence, and outputs it synchronously with the original audio signal, thereby forming a continuous, stable virtual human video stream that matches the audio content on the terminal device, achieving real-time driving and display effects.

[0141] Through the collaborative work of the above modules, this invention achieves continuous generation and low-latency output of virtual humans driven by real-time audio, effectively solving the problems of limited virtual human generation length, high computational load, and difficulty in audio-visual synchronization in the prior art, and has strong engineering practicality and promotional value.

[0142] In an embodiment of the present invention, the audio parsing and control module: the core function of this module is to convert real-time audio signals into precise control parameters required for digital human generation, and the specific implementation process is as follows:

[0143] (1) Receive real-time streaming audio signals, support multiple input methods such as real-time microphone input and audio file streaming, and uniformly set the sampling rate to 16kHz to ensure the consistency of signal processing.

[0144] (2) Speech feature extraction: The speech rate and tone features are analyzed by speech recognition algorithm, and the emotional intensity parameters (such as the quantitative values ​​of emotions such as joy, anger, and calmness) are extracted by emotion analysis model to provide emotional basis for digital human expression generation.

[0145] (3) Lip-sync parameter generation: Based on audio temporal features, lip-sync algorithm generates lip movement coordinates that match speech frame by frame, covering the dynamic position information of 68 / 106 facial key points, ensuring accurate correspondence between lip shape and speech.

[0146] (4) Generation of facial expression and motion parameters: Based on the FACS (Facial Action Coding System) standard, the intensity values ​​of facial expression units (such as AU12 smile intensity, AU4 sadness intensity, etc.) are generated, and regular motion instructions such as head movement frequency and blink cycle are extracted as motion auxiliary parameters.

[0147] (5) Timing mapping: Establish a one-to-one correspondence between control parameters and audio frames to provide a foundation for subsequent audio and video synchronization.

[0148] In an embodiment of the present invention, the digital human generation module is the core generation unit of the system, employing a hybrid generation mode to balance generation efficiency and image quality.

[0149] 1) Keyframe generation logic:

[0150] Triggering conditions: Keyframe generation is automatically triggered when the GOP starts, the digital human's identity changes (such as changing clothes or switching facial expressions), or when network packet loss is recovered.

[0151] Generation process: Based on the Diffusion / GAN generation model, render a complete digital human image, including full features such as facial texture, facial contours, basic expressions, and basic body postures. The image resolution supports 1080p and above, and the frame rate is 30fps by default.

[0152] Identity Anchoring: A 128-dimensional identity feature vector is attached to each keyframe. This vector is extracted based on the initial image of the digital human and contains core identity information such as facial features and contour features, serving as the identity reference benchmark for all frames within the GOP.

[0153] 2) P-frame generation logic:

[0154] Input data: Control parameters output by the audio parsing module + data from the previous reference frame (keyframe or generated P-frame).

[0155] Generated content:

[0156] Motion vector layer: contains 3D coordinates of 68 / 106 facial key points, head / limb rotation and translation vectors, and pixel-level optical flow information, accurately describing the motion trend of the digital human.

[0157] Residual layer: Records residual data between the current P-frame and the reference frame in terms of subtle differences in facial texture, compensation data for occluded areas, and supplementary information on micro-expressions, ensuring the integrity of details.

[0158] Control parameter layer: carries FACS unit intensity, lip-sync parameters, and identity verification sub-vector (a simplified vector extracted from keyframe identity feature vectors) to ensure facial expression consistency and identity stability.

[0159] Optimization strategy: Utilize the regularity of digital human movement (such as smooth head rotation, fixed blink cycle, and correlation of facial muscle movements) to optimize the motion prediction algorithm, reduce redundant data, and reduce the amount of residual data.

[0160] In an embodiment of the present invention, the encoding / decoding module employs an encoding / decoding strategy specifically designed for digital human scenarios to achieve efficient compression and low-latency decoding.

[0161] 1) Core encoding configuration:

[0162] GOP structure: Default length = frame rate × 1 second (30 frames in a 30fps scene), dynamically adjustable range is frame rate × 0.5 seconds to frame rate × 1 second, adapting to different scene requirements.

[0163] Frame type combination: The base layer uses an "I-frame + P-frame" combination and disables B-frames to eliminate bidirectional prediction latency; the enhancement layer for non-real-time scenes can add 1-2 B-frames to improve the compression rate by 15-20%.

[0164] Quantization parameters (QP): QP for I-frames is set to 24-26 to ensure the integrity of identity features and core image information; QP for P-frames is set to 28-30 to further reduce the amount of data while balancing compression rate and image quality.

[0165] Special frame applications: When the identity of a digital human changes, an IDR frame is forcibly inserted to clear the reference buffer and avoid identity confusion; the SEI frame carries facial expression parameters and timestamp information to provide auxiliary data for synchronization and decoding.

[0166] 2) Dynamic GOP adjustment mechanism:

[0167] Expression complexity driven: The expression difference between consecutive frames is calculated in real time through an inter-frame expression change rate detection algorithm. When the expression change rate of 3 consecutive frames is ≥50% (such as fast speech, singing, etc.), the GOP is automatically shortened to frame rate × 0.5 seconds; when the expression change rate is ≤20% (such as static expression, slow speech, etc.), the GOP is restored to the default length.

[0168] Network status adaptation: Real-time monitoring of bandwidth and packet loss rate data of the transmission link. When the bandwidth is ≥10Mbps and the packet loss rate is <1%, the standard GOP configuration is maintained; when the bandwidth is <5Mbps or the packet loss rate is ≥3%, the GOP is shortened to frame rate × 0.5 seconds, and the I-frame density is increased (one I-frame is inserted every 5 frames) to improve fault tolerance.

[0169] 3) Layered encoding implementation:

[0170] Base layer: Encodes the core data of I-frames and P-frames, including identity feature vectors, key motion vectors, core facial expression parameters, etc., with the bitrate controlled at 2-4Mbps to ensure real-time transmission and basic quality playback even in low-bandwidth environments.

[0171] Enhancement layer: Encodes detailed residual data of P-frames and additional B-frame data (in non-real-time scenarios), with the bitrate controlled at 1-2Mbps, and transmitted only when bandwidth is sufficient (≥6Mbps), further improving image sharpness and texture details.

[0172] In an embodiment of the present invention, a real-time transmission module is included. This module is responsible for low-latency, high-reliability transmission of encoded data, and its core technical details are as follows:

[0173] 1) Transmission protocol: WebRTC protocol is preferred, with end-to-end latency ≤100ms to meet the needs of real-time interaction; at the same time, it is compatible with RTMP / HTTP-FLV protocol to adapt to browser playback scenarios without plugins and improve system compatibility.

[0174] 2) Transmission mode: The frame-level pipeline transmission mechanism is adopted, which generates 1 frame → encodes 1 frame → encapsulates RTP packets → transmits immediately, without GOP-level buffering, to avoid latency accumulation.

[0175] 3) Synchronization mechanism: Each frame of data carries a precise timestamp with an accuracy of ≤1ms. The timing alignment of audio and video with the digital human's movements is achieved through timestamp calibration.

[0176] 4) Fault tolerance: A frame-level retransmission mechanism is adopted. When the network loses packets, only the critical P-frame data (the last 3 frames) is retransmitted. Combined with the short GOP design, the error propagation range is limited to 1 GOP, and the error recovery time is ≤0.5 seconds.

[0177] Decoding and Reconstruction Module: This module is responsible for receiving transmitted data, completing decoding and image reconstruction, and achieving high-quality synchronized playback.

[0178] 1) Data reception: Parse RTP packets, separate I-frame, P-frame, and SEI-frame data and corresponding timestamp information, and perform data integrity verification.

[0179] 2) Keyframe Decoding: Decode the complete image data and 128-dimensional identity feature vector of the I-frame, initialize the reference frame buffer, and establish the identity baseline.

[0180] 3) P-frame reconstruction process:

[0181] Step 1: Analyze the motion vector layer data, perform image distortion transformation on the reference frame (keyframe or previous P-frame), and initially generate the motion contour of the current frame.

[0182] Step 2: Overlay feature residual layer data to supplement texture details and occlusion compensation information, thereby improving image integrity.

[0183] Step 3: Based on the FACS unit intensity and lip-sync parameters of the control parameter layer, correct facial expressions and lip movements; perform similarity matching (threshold ≥ 95%) between the identity verification sub-vector and the keyframe identity feature vector to ensure identity consistency.

[0184] 4) Output optimization: Integrate the decoded and reconstructed digital human image with the original audio signal, and calibrate based on the timestamp information of the SEI frame to achieve accurate synchronization of lip movements, facial expressions and speech, and output a stable digital human video playback stream.

[0185] The implementation process of this invention follows a pipeline logic of "audio input → encoding generation → real-time transmission → decoding and playback". The specific steps are as follows, and the timing connection and parameter configuration of each step ensure the efficient and stable operation of the system:

[0186] Step 1: Audio signal analysis and control parameter generation (time ≤ 30ms);

[0187] After the system starts up, the audio parsing and control module is initialized, the sampling rate is set to 16kHz, and it supports real-time microphone input or audio file streaming.

[0188] It receives real-time audio streams, extracts speech rate (unit: words / second) and tone (quantized as a value of 0-100) features through speech recognition algorithms, and outputs emotion intensity parameters (such as joy: 85, calm: 50, anger: 90, etc.) through emotion analysis models.

[0189] Based on audio temporal features, a lip-sync algorithm is used to generate frame-by-frame lip movement coordinates, covering 3D coordinate information of 106 facial key points, and establishing a one-to-one temporal relationship with the audio frames.

[0190] Based on the FACS standard, expression unit intensity values ​​are generated (e.g., AU12 smile intensity: 70, AU4 sadness intensity: 30, etc.), while extracting motion auxiliary parameters such as head movement frequency (unit: times / second) and blink cycle (unit: seconds).

[0191] All control parameters are packaged, timestamps are added (accuracy ≤ 1ms), and then transmitted to the digital human generation module.

[0192] Step 2: Keyframe generation and encoding (keyframe time ≤ 50ms, triggered once every 30 frames);

[0193] The digital human generation module receives control parameters and starts keyframe generation when trigger conditions such as GOP start frame, identity change, or network packet loss recovery are detected.

[0194] The Diffusion model is invoked to render a complete digital human image (1080p resolution, 30fps) based on control parameters, including full features such as facial texture, facial contours, basic expressions, and basic body postures.

[0195] The 128-dimensional identity feature vector of the digital human is extracted and embedded into the keyframe data to complete the identity anchoring.

[0196] The encoding and decoding module uses QP=25 to perform intra-frame encoding on keyframes, generating I-frame data and SEI frames, which carry expression parameters and timestamp information.

[0197] The real-time transmission module encapsulates I-frames and SEI frames into RTP packets, adds a precise timestamp, and immediately sends them to the transmission link via the WebRTC protocol without buffering delay.

[0198] Step 3: Lightweight P-frame generation and encoding (single frame time ≤ 10ms);

[0199] During non-keyframe moments, the digital human generation module receives control parameters and data from the previous reference frame (keyframe or already generated P-frame) and initiates P-frame generation.

[0200] Based on the motion regularity of digital humans, the 3D coordinates of 68 facial key points, head rotation and translation vectors, and pixel-level optical flow fields are predicted to generate a motion vector layer.

[0201] Calculate the facial texture difference between the current frame and the reference frame, the occlusion area compensation data, and the micro-expression supplementary information to generate a feature residual layer.

[0202] Extract FACS unit intensity, lip-sync parameters, and identity verification sub-vector (64-dimensional, simplified based on keyframe identity feature vector) to generate control parameter layer.

[0203] The encoding / decoding module uses QP=29 to perform inter-frame encoding on the three-layer structure data of the P-frame to generate P-frame data.

[0204] The real-time transmission module encapsulates P-frames into RTP packets, associates them with the identity verification information of the corresponding key frames, and transmits them in real time in chronological order.

[0205] Step 4: Dynamic GOP adjustment (executed in real time);

[0206] The encoding / decoding module monitors the inter-frame facial expression change rate in real time and calculates the facial expression change rate by comparing the intensity differences of FACS units in consecutive frames. When the facial expression change rate is ≥50% for 3 consecutive frames, the GOP is shortened to 15 frames (0.5 seconds at 30fps); when the facial expression change rate is ≤20%, the GOP is restored to 30 frames.

[0207] The real-time transmission module monitors the network status in real time and dynamically adjusts the GOP length and I-frame density based on bandwidth and packet loss rate data fed back from the transmission link.

[0208] When the bandwidth is ≥10Mbps and the packet loss rate is <1%, the standard GOP configuration (30 frames) is maintained, and I-frames are inserted once every 30 frames.

[0209] When the bandwidth is less than 5 Mbps or the packet loss rate is greater than or equal to 3%, the GOP is shortened to 15 frames, and I-frames are inserted once every 5 frames.

[0210] Step 5: Real-time streaming (end-to-end latency ≤100ms);

[0211] The real-time transmission module prioritizes the WebRTC protocol, establishing point-to-point connections or forwarding through edge nodes to ensure low-latency transmission.

[0212] Transmission priority settings: I-frames have the highest priority and are transmitted first; P-frames are transmitted in chronological order; SEI-frames are transmitted synchronously with their corresponding video frames.

[0213] Fault tolerance: Packet loss is detected through the RTP packet verification mechanism. When a packet is lost, only the last 3 P-frames of data are retransmitted to avoid increased latency caused by a large number of retransmissions.

[0214] Synchronization Guarantee: Each RTP packet carries a precise timestamp to ensure that the decoding end can achieve audio and video synchronization based on the timestamp.

[0215] Step 6: Decoding, reconstruction, and playback (single frame time ≤ 20ms);

[0216] The decoding and reconstruction module receives RTP packets, sorts them by timestamp, separates I-frames, P-frames, and SEI-frames, and performs data integrity verification.

[0217] Decode I-frame data, extract complete digital human image and 128-dimensional identity feature vector, initialize reference frame buffer, and establish identity benchmark.

[0218] Decode the P-frame data and reconstruct the image using the following procedure:

[0219] The motion vector layer is analyzed, and the reference frame is subjected to image distortion transformation to generate motion contours.

[0220] Superimposed feature residual layer data supplements texture details and occlusion compensation information.

[0221] Based on the FACS unit intensity and lip-sync parameters of the control parameter layer, facial expressions and lip movements are corrected; similarity matching is performed between the identity verification sub-vector and the keyframe identity feature vector. If the similarity is ≥95%, the verification is passed; otherwise, local adjustment is triggered.

[0222] By combining the timestamp calibration information of the SEI frame with the decoded and reconstructed digital human image and the original audio signal, accurate synchronization of lip movements, facial expressions and speech is achieved, and a stable playback stream (1080p@30fps) is output.

[0223] Step 7: Long-term optimization (continuous execution);

[0224] A global identity verification is triggered every 10 GOPs (approximately 10 seconds), comparing the similarity of the 128-dimensional identity feature vectors of the current frame with those of the initial keyframe.

[0225] If the similarity is less than 95%, force the insertion of an IDR frame, clear the reference frame buffer, re-render the complete keyframe and embed the identity feature vector, reset the identity baseline, and avoid feature drift.

[0226] It receives playback status (such as stuttering or screen tearing) and network status data from the decoding end in real time, and dynamically adjusts encoding parameters (such as QP value and layered encoding strategy) and transmission strategies (such as retransmission mechanism) to ensure the stability of the entire link.

[0227] Implementation Case:

[0228] Example 1: Virtual Anchor Live Streaming Scenario:

[0229] Scenario requirements:

[0230] Virtual anchor live streaming scenario requirements: end-to-end latency ≤200ms, support for 24 / 7 unlimited length live streaming, high consistency of digital human identity (no obvious distortion), bandwidth consumption adapted to the uplink bandwidth of ordinary anchors (≥5Mbps), accurate synchronization of lip movements and voice, and natural facial expressions.

[0231] System Configuration:

[0232] Hardware configuration: The broadcasting end uses a mid-range server (CPU: Intel Core i7-12700K, GPU: NVIDIA RTX3080), and the decoding end uses a regular computer for viewers (CPU: Intel Core i5-10400, GPU: NVIDIA GTX 1650).

[0233] Network configuration: Uplink bandwidth for the broadcaster is ≥5Mbps, and downlink bandwidth for the viewer is ≥2Mbps.

[0234] Software configuration: The audio parsing module adopts a Transformer-based speech recognition and emotion analysis model, the digital human generation module adopts the Diffusion generation model, the encoding and decoding module is optimized based on the H.265 standard, and the transmission protocol adopts WebRTC.

[0235] Implementation steps:

[0236] The host inputs real-time voice through a microphone. The audio parsing module receives the signal at a sampling rate of 16kHz, extracts parameters such as speech rate (approximately 200 words / second), tone (quantization value 60), and emotional intensity (joy: 80), and generates frame-by-frame lip shape coordinates and control parameters such as AU12 smile intensity (75) and head movement frequency (0.5 times / second), taking ≤25ms.

[0237] The digital human generation module generates one keyframe every 30 frames (1 second), renders a complete 1080p digital human image based on control parameters, and embeds a 128-dimensional identity feature vector, taking ≤45ms; non-keyframes generate P-frames, taking ≤8ms.

[0238] The encoding / decoding module uses the default GOP configuration (30 frames), with I-frame QP=25 and P-frame QP=29. The base layer bitrate is controlled at 3Mbps, and the enhancement layer bitrate is controlled at 1.5Mbps (enabled when the uplink bandwidth of the broadcaster is ≥6Mbps).

[0239] The real-time transmission module transmits encoded data via the WebRTC protocol. The end-to-end transmission delay is ≤90ms. There is no retransmission when the packet loss rate is <1%, and the last 3 P frames are retransmitted when the packet loss rate is ≥1%.

[0240] The audience-side decoding and reconstruction module receives data, with decoding I-frames taking ≤15ms and decoding P-frames taking ≤10ms. Combined with SEI frame timestamp calibration, it achieves lip-sync with speech, with a synchronization error of ≤8ms.

[0241] Long-term optimization: Identity verification is performed every 10 seconds, and the similarity is maintained above 98.5%. There is no feature drift or deformation even after 12 hours of continuous live streaming.

[0242] Implementation results:

[0243] With an end-to-end latency of 185ms, bandwidth consumption of 3Mbps (base layer) / 4.5Mbps (base layer + enhancement layer), an identity consistency rate of 98.8%, and an audio-visual synchronization error of 8ms, viewers experience no stuttering or distortion, meeting the core requirements of virtual anchor live streaming.

[0244] Example 2: Digital Human Interactive Customer Service Scenario:

[0245] Scenario requirements:

[0246] Requirements for digital human interactive customer service scenarios: response latency ≤150ms, support for multiple users to access simultaneously (≥100 people), network environment adaptation (some users use mobile networks, bandwidth fluctuates greatly), accurate synchronization of digital human lip reading and customer service voice, and stable identity for long-term service (≥8 hours).

[0247] System Configuration:

[0248] Hardware configuration: The server adopts a cluster deployment (4 servers, CPU: Intel Core i9-13900K, GPU: NVIDIA RTX 4090), and the user is a mobile phone (Android / iOS) or computer.

[0249] Network configuration: Server-side uplink bandwidth ≥ 100Mbps, user-side downlink bandwidth ≥ 2Mbps (mobile network / broadband).

[0250] Software configuration: The audio parsing module uses a lightweight speech recognition model (adapted to real-time response), the digital human generation module uses a GAN generation model (to improve generation speed), the encoding and decoding module optimizes decoding efficiency on mobile devices, and the transmission protocol is compatible with WebRTC and HTTP-FLV (adapted to mobile browsers).

[0251] Implementation steps:

[0252] Users input voice inquiries through the customer service system. The audio signal is transmitted to the server in real time. The audio parsing module receives the audio at a sampling rate of 16kHz and extracts parameters such as speech rate (approximately 180 words / second), tone (quantization value 55), and emotional intensity (calm: 60) to generate control parameters such as mouth shape coordinates and AU01 frown intensity (30). The time taken is ≤20ms.

[0253] The server-side digital human generation module generates data at 30fps, with keyframes triggered once every 30 frames, taking ≤40ms; P-frames take ≤7ms, supporting concurrent generation of 100 people.

[0254] The encoding / decoding module enables dynamic GOP adjustment: when the user-end network bandwidth is ≥5Mbps and the packet loss rate is <1%, GOP=30 frames; when the bandwidth is <3Mbps or the packet loss rate is ≥3%, GOP=15 frames, and one I-frame is inserted every 5 frames. The QP of the I-frame is 26, the QP of the P-frame is 30, and the base layer bitrate is controlled at 2Mbps.

[0255] The real-time transmission module selects the transmission protocol according to the user's network environment: WebRTC is used on the computer (latency ≤80ms) and HTTP-FLV is used on the mobile phone (latency ≤100ms). When a packet is lost, the key P frame is retransmitted and the error recovery time is ≤0.3 seconds.

[0256] The client-side decoding and reconstruction module is adapted to mobile hardware, with decoding time of ≤12ms for I-frames and ≤8ms for P-frames. Combined with timestamp calibration, the audio and video synchronization error is ≤7ms.

[0257] Long-term optimization: Identity verification is performed every 10 seconds, with a similarity of ≥98%, and continuous service for 8 hours without distortion or jump.

[0258] Implementation results:

[0259] With an end-to-end latency of 135ms, it can support 100 concurrent users without lag, has an error recovery time of 0.25 seconds in mobile network environments, maintains an identity consistency rate of 98.2%, and has an audio and video synchronization error of 7ms, meeting the low latency, high concurrency, and strong adaptability requirements of digital human interactive customer service.

[0260] Evidence related to the technical effects obtained by the embodiments of the present invention.

[0261] Comparison Table of Core Performance Indicators

[0262]

[0263] Figure 3 End-to-end latency comparison.

[0264] Figure 4 Changes in identity consistency over a long period of time.

[0265] Figure 5 Bandwidth consumption comparison.

[0266] Figure 6 :Calculation cost composition and optimization.

[0267] All the above data were obtained through actual testing, and the test environment is as follows:

[0268] 1. Hardware environment: Servers and terminal devices configured according to Embodiment 1 and Embodiment 2.

[0269] 2. Software environment: Existing technologies use mainstream virtual digital human generation platforms (such as Unity + Live2D), and this invention uses the prototype system implemented by the above technical solution.

[0270] 3. Test scenarios: Virtual anchor live streaming (7×24 hours), digital human interactive customer service (100 concurrent users).

[0271] 4. Testing tools: latency testing tool (Wireshark + custom timing analysis tool), bandwidth testing tool (iPerf3), identity consistency testing tool (feature vector similarity calculation tool), synchronization error testing tool (audio and video timing comparison tool).

[0272] Test results show that the present invention is significantly superior to existing technologies in core indicators such as computing cost, latency, bandwidth consumption, identity consistency, synchronization accuracy, and fault tolerance, and has outstanding technical advantages and application value.

[0273] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0274] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for generating an infinitely long virtual human driven by real-time streaming audio, characterized in that, The method includes the following collaborative mechanism steps: Step 1: Construct a causal autoregressive generation mechanism based on frame latent variable blocks, so that the generation of the current frame block depends only on historical frame blocks, rolling reference frames, audio conditional information and text conditional information, thereby avoiding future information from participating in the generation of the current frame. Step 2: Construct a causal video coding mechanism to encode the video data during the training phase with causal constraints, so that the latent variable at any time is determined only by the current frame and the frames before it, in order to ensure that the training distribution is consistent with the streaming inference distribution. Step 3: Construct a long-term consistency coordination mechanism. Suppress generation distribution drift by dynamically updating the generation reference frame and suppress inference mode drift by rolling relative time position control, so that identity consistency and temporal continuity remain stable simultaneously during the long-term generation process. Step 4: Construct an autoregressive student model and apply distillation constraints through a bidirectional teacher model and a pseudo-score model. This allows the student model to approximate the generation distribution of the bidirectional diffusion model while maintaining the causal generation structure, thus balancing generation quality and real-time reasoning ability.

2. The method as described in claim 1, characterized in that, The causal autoregressive generation mechanism adopts a block-level structure, dividing the video latent variables into multiple consecutive frame blocks. Each frame block is generated using only the frame block information from the previous moment and even earlier moments as conditional inputs.

3. The method as described in claim 1, characterized in that, The long-term consistency coordination mechanism includes an adaptive attention benchmark mechanism and a rolling relative position control mechanism. The adaptive attention benchmark mechanism uses the first frame in the generated video as the new generation benchmark frame to suppress distribution drift, while the rolling relative position control mechanism maintains the relative temporal consistency between the training and inference phases by continuously shifting the time position.

4. A parallel inference method for generating virtual humans driven by real-time streaming audio, implementing the real-time streaming audio-driven infinite-length virtual human generation method as described in any one of claims 1-3, characterized in that, This includes the following collaborative mechanisms: The multiple time steps in the denoising process are fixedly allocated to multiple parallel computing units, so that each computing unit executes the same denoising step for a long time, thereby transforming the sequential denoising process into a spatial pipeline process. In the initial stage, a sequential filling method is used to allow the latent variables to pass through all computational units in turn to complete the first denoising. In the stable stage, each computational unit only processes the corresponding step and passes the intermediate latent variables to the next computational unit. The conditional buffer of historical frame blocks is kept consistent with the noise scale of the current denoising state, thereby ensuring that the conditional distribution during the inference phase is consistent with the conditional distribution during the training phase.

5. The method as described in claim 4, characterized in that, Historical frame blocks are conditionally cached locally, and the video decoding process is offloaded from the main denoising pipeline to an independent execution unit to improve overall throughput.

6. The method as described in claim 4, characterized in that, The initial stage of the pipeline is executed only once, and its time consumption is less than the single-step execution time of the stable pipeline stage.

7. A real-time streaming audio-driven infinite-length virtual human generation system implementing the real-time streaming audio-driven infinite-length virtual human generation method as described in any one of claims 1-3, characterized in that, include: Audio analysis and control module: As the system input layer, it is responsible for receiving real-time streaming audio signals, extracting core control parameters through multi-algorithm fusion, and providing precise instructions for digital human generation; Digital Human Generation Module: Based on the control parameters output by the audio parsing module, it adopts a "keyframe + P-frame" hybrid generation mode to achieve efficient generation of complete digital human keyframes and lightweight P-frames; Encoding / decoding module: Employs dedicated encoding / decoding configuration and layered encoding strategy to efficiently encode and compress the generated digital human image data, while ensuring a fast and stable decoding process; Real-time transmission module: Based on optimized transmission protocols and synchronization mechanisms, it achieves low-latency and high-reliability transmission of encoded data, adapting to different network environments; Decoding and Reconstruction Module: As the system output layer, it is responsible for receiving transmitted data, completing decoding and image reconstruction, and realizing precise synchronous playback of digital human video streams and audio signals.

8. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the real-time streaming audio-driven infinite-length virtual human generation method as described in any one of claims 1-3.

9. A computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the real-time streaming audio-driven infinite-length virtual human generation method as described in any one of claims 1-3.

10. An information data processing terminal, characterized in that, The information data processing terminal is used to implement the real-time streaming audio-driven infinite-length virtual human generation system as described in claim 7.

Images