A digital human driving system and method supporting multi-platform matrix live broadcast

Abstract

The application belongs to the technical field of Internet live broadcast platforms, and in particular to a digital human driving system and method supporting multi-platform matrix live broadcast. The system is responsible for training and distributing digital human models through a digital human training machine. Each node in the computing power machine cluster independently handles one live broadcast. Speech synthesis and image rendering are completed locally through a LoRA fine-tuned TTS model and an emotion-driven digital human model. The collector converts digital audio and video signals into standard virtual microphone and camera signals through a kernel virtual driver. The live broadcast machine pushes virtual device signals to each live broadcast platform through official platform software. The system effectively improves the realism and appeal of live broadcasts and enhances the live broadcast interactive experience. The collector converts digital signals into standard virtual camera and microphone signals, allowing any live broadcast software to seamlessly connect. The cloud only transmits lightweight model files, significantly reducing bandwidth costs and ensuring efficient and stable local video streaming.

Description

Technical Field

[0001] This invention belongs to the field of Internet live streaming platform technology, specifically relating to a digital human driving system and method that supports multi-platform matrix live streaming. Background Technology

[0002] With the rapid development of live-streaming e-commerce and live-streaming content, digital human live-streaming has been increasingly widely used due to its advantages such as uninterrupted operation, low cost, and scalability.

[0003] Most current digital human live streaming systems adopt centralized cloud rendering or stand-alone integrated solutions. Although centralized cloud rendering solutions have powerful computing power, the network bandwidth cost is extremely high when pushing the rendered video stream to multiple platforms, and it is easy to cause live streaming lag and audio-visual desynchronization due to network fluctuations. Stand-alone integrated solutions are limited by the computing power of a single device, making it difficult to drive multiple high-fidelity digital humans to live stream at the same time, and thus unable to achieve matrix operation.

[0004] In addition, existing solutions generally rely on third-party TTS (text-to-speech) and lip-sync APIs, which not only incurs continuous call costs but also introduces additional latency due to network requests.

[0005] More importantly, many current digital human technologies heavily rely on original video images, and their driving methods have obvious flaws. Most can only change lip movements according to the voice content, while body postures, gestures, and other actions are simply reused or looped from the original recorded video. This "mouth moves but body doesn't" mode results in digital humans appearing stiff and unnatural, unable to fine-tune their body movements and facial expressions according to the emotional changes (such as excitement, sadness, joy) of the live broadcast script or interactive content, greatly weakening the appeal and realism of the live broadcast.

[0006] Meanwhile, existing systems lack effective real-time interactive mechanisms. Digital humans mostly provide one-way content output and cannot respond to real-time behaviors such as audience entry, likes, and bullet comments, resulting in a rigid live streaming experience, poor user stickiness, and a serious impact on conversion rates. In addition, integrating digital human images and sounds into live streaming platforms usually requires complex virtual cameras and audio routing software, which are cumbersome to configure and have poor stability, making it difficult to meet the needs of large-scale, standardized matrix live streaming. Summary of the Invention

[0007] This invention aims to provide a digital human driving system and method that supports multi-platform matrix live streaming, in order to solve the technical problems existing in digital human live streaming technology, such as high cloud rendering bandwidth costs, limited single-machine computing power preventing matrixing, high costs and high latency due to reliance on third-party APIs, stiff performance due to disconnect between actions and emotions, lack of real-time interactive capabilities, and complex and unstable audio and video signal access to live streaming platforms.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A digital human-driven system supporting multi-platform matrix live streaming is provided, including: The digital human training machine serves as the asset production layer of the system. It is responsible for collecting real-person video and audio data, using digital human driving models and voice training models to generate high-fidelity digital human image models and personalized voice models, and pushing the trained model files to the cloud model repository for unified storage and distribution. The computing power cluster serves as the inference computing layer of the system, comprising multiple independent computing power nodes configured with GPUs. Each computing power node corresponds to a live streaming room and is responsible for pulling a specified digital human model from the cloud model repository, completing the live streaming script generation, speech synthesis, digital human image driving, and real-time interactive processing locally, generating a complete digital human audio and video stream. The computing power nodes have built-in local speech synthesis models and digital human driving models that have been fine-tuned with low-rank adaptation LoRA. The data acquisition unit serves as the signal conversion layer of the system. Each data acquisition unit corresponds to a computing node. Through virtual device driver technology, it converts the digital audio and video streams output by the computing node into physical device signals that the operating system can recognize, including converting AI voice data streams into virtual microphone signals and digital human video streams into virtual camera signals. The live streaming machine is used as the system's push output layer. Each live streaming machine corresponds to a collector, which installs the official live streaming software of the corresponding live streaming platform. It uses the virtual camera and virtual microphone generated by the corresponding collector as audio and video sources and pushes the digital human audio and video streams to the corresponding live streaming platform.

[0009] Preferably, the local speech synthesis model of the computing node is obtained using a low-rank adaptation LoRA fine-tuning scheme based on the Index-TTS unified speech model.

[0010] Preferably, the acquisition of the Index-TTS unified speech model using a low-rank adaptation LoRA fine-tuning scheme includes: During the training phase, the pre-trained pedestal weights are not updated; only the LoRA low-rank matrix on the GPT module and the learnable condition vector mean_condition_{speaker_id} for each scene are trained. The training process is as follows: First, audio and corresponding text for specific scenes are collected, and abnormal samples with a duration of 1-20 seconds are filtered out. Then, each audio is resampled to 24kHz, and a 100-dimensional logarithmic Mel spectrum is extracted. This spectrum is quantized into discrete Mel codes by a pre-trained DVAE and then processed by a Conformer+Perceiver encoder to obtain a condition latent with a shape of (32, 1024). Next, the pairwise distance matrix is ​​calculated for all condition latents, and the medoid is selected as the scene center condition vector and registered as a trainable parameter. The data is then divided into a training set and a validation set in a 9:1 ratio.

[0011] Preferably, the acquisition of the Index-TTS unified speech model using the low-rank adaptation LoRA fine-tuning scheme further includes: During the model loading phase, all base parameters of Unified Voice are frozen, and LoRA is injected only into four linear layers (attn.c_attn, attn.c_proj, mlp.c_fc, and mlp.c_proj) out of the 20 layers of GPT. During training, the input sequence is constructed and forward propagated through GPT, using a joint loss function of text loss and Mel-spectral loss. L = 0.1·L_text + 0.9·L_mel Where L_text is the text prediction loss and L_mel is the mel spectrum prediction loss; during inference, the fine-tuned conditional vector is loaded according to the anchor identifier, and after the autoregression generates mel codes, it is decoded into a 24kHz audio waveform by the BigVGAN vocoder.

[0012] Preferably, the digital human driving model of the computing power node is obtained based on the DH_live architecture using a parameter-efficient fine-tuning scheme. In this scheme, a LoRA module is injected into the attention mechanism layer of the pre-trained Transformer backbone network. The low-rank adaptation matrix of the LoRA module satisfies:

[0013] Where ΔW is the low-rank weight update matrix; B is the upgraded low-rank matrix with dimension 1. ； A is a dimension-reduced low-rank matrix with dimensions of . ； During training, the reconstruction loss function is minimized as shown in the following formula:

[0014] Where Lrec is the reconstruction loss; Xtarget is the target real audio feature; Xaudio is the input raw audio feature; fDH(·) is a pre-trained audio feature mapping network used to map the input audio features to the output predicted audio features; Θfrozen is the frozen parameter of the pre-trained model; ΔΘ is the fine-tunable incremental parameter.

[0015] Preferably, the digital human-driven model introduces an action style embedding vector E_motion, and injects emotional features into the intermediate feature layer of the decoder through an adaptive instance normalization AdaIN layer. The feature transformation process is as follows:

[0016] Where AdaIN(h, Emotion) is the final output of the adaptive instance normalization operation; h is the intermediate layer feature map of the decoder; μ(·) and σ(·) represent the mean and standard deviation calculation operations of the features, respectively; is the channel-level standard deviation of the intermediate layer feature map h; Emotion represents the motion style feature; σ(Emotion) is the channel-level standard deviation of the motion style feature Emotion; μ(Emotion) is the channel-level mean of the motion style feature Emotion; The features after emotion modulation are ultimately mapped to the facial vertex deformation parameter ΔV, and its driving function is formalized as follows:

[0017] Where ΔV is the real-time speech feature increment of the final output; DecoderDHlive is a deep latent space decoder for real-time scenarios; Faudio is the basic audio feature input; Emotion is the motion / emotion feature input; Θfrozen is the frozen weight of the pre-trained decoder; and ΔΘLoRA is the incremental parameter of the LoRA low-rank adaptation.

[0018] Preferably, during training, a time smoothing regularization term is also added to the loss function as follows:

[0019] Where Ltemp is the temporal smoothing loss; ∑t is the summation operation over all time steps; ΔVt is the output increment at time step t; and ΔVt-1 is the output increment at time step t-1.

[0020] Preferably, the virtual audio driver conversion process of the acquisition device is as follows: S_mic = VirtualAudioDriver(y(t)) Where y(t) is the AI ​​voice data stream output by the computing node; S_mic is the virtual microphone input signal recognized by the operating system; The virtual camera driver conversion process is as follows: S_cam = VirtualCameraDriver(I_frame(n)) Where I_frame(n) is the nth frame of the digital human video output by the computing node; S_cam is the signal captured by the virtual camera recognized by the operating system.

[0021] This invention also provides a digital human driving method supporting multi-platform matrix live streaming, implemented based on the digital human driving system supporting multi-platform matrix live streaming described in any of the preceding claims, the method comprising: Model training and distribution: Collect real-person audio and video data on the digital human training machine, train and generate digital human image models and personalized voice models, and push the trained models to the cloud model repository. Local inference and content generation: Each computing node pulls the required digital human model from the cloud, completes the live broadcast script generation locally, converts the text into AI speech through the local speech synthesis model, and then drives the digital human image based on speech features and emotional embedding vectors through the digital human driving model to generate digital human video frames, forming a complete digital human audio and video stream. Real-time interactive processing: The computing power nodes monitor the live streaming platform API in real time, capture audience interaction events, generate interactive text in real time according to the event type, and send it to the speech synthesis and image-driven process first to generate interactive audio and video content with corresponding facial expressions and actions and insert it into the live stream. Virtual signal conversion: The acquisition device converts AI voice into a virtual microphone signal that the system can recognize through a virtual audio driver, and converts digital human video stream into a virtual camera signal that the system can recognize through a virtual camera driver; Multi-platform streaming: Each live streaming device selects the corresponding virtual device as the audio and video source, and pushes the digital human content to the corresponding live streaming platform through the platform's official live streaming software to achieve multi-platform matrix live streaming.

[0022] Preferably, the real-time interactive processing specifically includes: the computing power node listens to the open APIs of each live streaming platform in real time through a long connection; when it detects that a viewer enters the live room, likes, follows, sends gifts, or sends bullet comments, it inputs the event content and live streaming context into the large language model to generate interactive response text that conforms to the live streaming context in real time; the interactive text is marked as high priority and immediately sent to the speech synthesis and image-driven queue, and is processed before ordinary live streaming script content to generate interactive audio and video with welcome and thank you corresponding expressions and actions, and is immediately inserted into the live stream for playback.

[0023] Compared with the prior art, the beneficial effects of the present invention are: This multi-platform matrix live streaming system employs a four-layer distributed architecture, completely decoupling the training, inference, acquisition, and live streaming stages. This allows the system to flexibly increase or decrease computing power and live streaming terminals according to the scale of the live stream, easily achieving large-scale matrix live streaming with hundreds or even thousands of concurrent streams. Simultaneously, the system fully integrates the speech synthesis and image-driven engine into the local computing power, completely eliminating reliance on third-party cloud APIs. This achieves zero call costs while reducing interactive response latency to milliseconds, significantly improving the smoothness and stability of the live stream. In terms of expressiveness, an emotion-driven motion fine-tuning mechanism is introduced, enabling the digital human's body movements and facial micro-expressions to dynamically change with the emotions conveyed in the voice, effectively solving the stiffness of traditional digital humans that are "mouth-moving but body-still," greatly enhancing the realism and appeal of the live stream. Furthermore, the system constructs an efficient real-time interactive closed loop, allowing the digital human to provide instant voice and motion feedback based on audience entry, likes, and bullet comments, effectively enhancing the interactive experience and user conversion rate of the live stream. In terms of compatibility and cost control, the acquisition device converts digital signals into standard virtual camera and microphone signals, allowing any live streaming software to be seamlessly connected without complex software configuration. At the same time, the cloud is only responsible for transmitting lightweight model files, avoiding the network transmission of heavy video streams, significantly reducing bandwidth costs, and ensuring the efficiency and stability of local video streaming. Attached Figure Description

[0024] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall architecture diagram of an embodiment of the digital human driving system supporting multi-platform matrix live streaming according to the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] In one embodiment, a digital human-driven system supporting multi-platform matrix live streaming is provided, such as... Figure 1 As shown, the digital human-driven system that supports multi-platform matrix live streaming includes four levels: digital human training machine, computing power cluster, collector, and live streaming machine.

[0027] The first layer, the digital human training machine, serves as the system's asset production layer. It is responsible for collecting real-person video and audio data, generating high-fidelity digital human image models and personalized voice models using digital human-driven models and voice training models, and pushing the trained model files to the cloud model repository for unified storage and distribution.

[0028] Therefore, the digital human training machine can be seen as the system's "central factory," responsible for the production and distribution of digital human assets. After training, these model files are pushed to the cloud model repository via the OBS streamer or file synchronization service for downstream computing power to use. For example, the digital human training machine can use a workstation equipped with an RTX 3090 / 4090 level GPU to be responsible for the production of digital human assets: capturing 20 minutes of high-definition video and audio recorded by a real person in front of a green screen, extracting pure human images through portrait segmentation, performing facial key point detection and 3DMM parameter regression to build the basic image model of the person; at the same time, denoising and aligning the audio data, training the person's exclusive LoRA voice model and driver model; after training, uploading the model file of about 200MB to the Alibaba Cloud OSS cloud model repository for version management and distribution.

[0029] The second layer of computing power clusters is used as the inference computing layer of the system, including multiple independent computing power nodes configured with GPUs. Each computing power node corresponds to a live broadcast room and is responsible for pulling the specified digital human model from the cloud model repository, completing the live broadcast script generation, speech synthesis, digital human image driving and real-time interactive processing locally, and generating a complete digital human audio and video stream. The computing power nodes have built-in local speech synthesis model and digital human driving model that have been fine-tuned with low-rank adaptation LoRA. The computing power cluster can be seen as the system's "distributed brain," with each GPU-equipped computing power machine independently responsible for the complete content generation and real-time interaction of a single live streaming room. During the model loading phase, the computing power machines retrieve specified digital human avatar and voice models from the cloud. In the script generation phase, the system automatically generates a live streaming script based on preset brand, product, and other live streaming information using built-in AI tools. In the localized speech synthesis phase, the system utilizes a self-developed speech training model to convert the live streaming script or interactive text into AI speech in real time; this process is entirely completed locally without the need for third-party APIs.

[0030] Specifically, the local speech synthesis model built into the computing node, which has been fine-tuned by low-rank adaptation (LoRA), is a fine-tuned model based on the indexTTS model. This model is obtained by a low-rank adaptation (LoRA) fine-tuning scheme based on a specific scenario dataset for the IndexTTS unified speech model. When training the model, this scheme does not update the pre-trained base weights, but only trains the LoRA low-rank matrix on the GPT module and the learnable condition vector mean_condition_{speaker_id} for each scenario. This achieves the improvement of speech prosody, naturalness and timbre consistency in the target scenario with a lower computing cost. The training process is as follows: First, collect audio and corresponding text for a specific scene, and filter out abnormal samples with a duration of 1 to 20 seconds; then, resample each audio to 24kHz, extract the 100-dimensional logarithmic Mel spectrum, quantize it into discrete Mel codes using a pre-trained DVAE, and obtain a condition latent with a shape of (32, 1024) using a Conformer+Perceiver encoder; then, calculate the pairwise distance matrix for all condition latents, select the medoid as the scene center condition vector and register it as a trainable parameter; divide the data into training and validation sets in a 9:1 ratio.

[0031] During the model loading phase, all base parameters of UnifiedVoice are frozen, and LoRA (r=16, α=32, dropout=0.1) is injected only into the four linear layers of GPT: attn.c_attn, attn.c_proj, mlp.c_fc, and mlp.c_proj. During training, the input sequence [condition(32)|text(+start / stop)|mel codes(+start / stop)] is constructed based on the speaker_id and routed to the corresponding mean_condition. After forward propagation through GPT2, the next token is predicted by text_head (vocabulary 12000) and mel_head (codebook 8194), respectively. Calculate the masked cross-entropy loss at the valid locations; the joint loss is: L = 0.1 * L_text + 0.9 * L_mel Where L_text is the text prediction loss and L_mel is the Mel spectrum prediction loss; AdamW (lr=5×10) was used -5Alternatively, use LoRA+ optimizer, Cosine Warmup scheduling (warmup ratio 0.1), and gradient pruning (threshold 1.0) to update parameters; monitor mel loss and Top-k accuracy on the validation set every epoch, and save the best model based on the lowest mel loss. If there is no improvement after 30 consecutive epochs, stop early. When saving, merge the LoRA weights into the GPT weights.

[0032] During the inference phase, the text is segmented into sentences after being processed by the same TextNormalizer and BPE as during training. GPT loads the finely tuned mean_condition autoregressive code (top_p=0.8, num_beams=3, max_mel_tokens=600) according to speaker_id, and then the BigVGAN vocoder combines the reference audio Mel conditional decoding to output a 24kHz waveform.

[0033] In the localization-driven phase, the system's built-in digital human driving model is deeply fine-tuned and optimized based on the open-source digital human DH_live architecture. Addressing the issues of stiff action performance and lack of personalized features in long-term inference using the original DH_live model, this invention proposes a self-developed driving scheme based on Parameter Efficient Fine-Tuning (PEFT). Specifically, a low-rank adaptation (LoRA) module is injected into the pre-trained Transformer backbone network of DH_live. The query matrix is ​​only applied to the attention mechanism layer. The low-rank adaptation matrix of the LoRA module satisfies ΔW = BA, where... , And rank r ≤ d, ΔW is the low-rank weight update matrix; B is an upgraded low-rank matrix with dimension . A is a dimension-reduced low-rank matrix with dimensions of . During training, the following reconstruction loss function is used to minimize the following:

[0034] Where Lrec is the reconstruction loss; Xtarget is the target real audio feature; Xaudio is the input raw audio feature; fDH(·) is a pre-trained audio feature mapping network used to map the input audio features to the output predicted audio features; Θfrozen is the frozen parameter of the pre-trained model; ΔΘ is the fine-tunable incremental parameter.

[0035] By minimizing the reconstruction loss function, the model can learn the distribution of micro-expressions and body movements of a specific digital human while retaining the strong generative prior of DH_live.

[0036] During the inference phase, the system first uses Wav2Vec 2.0 to extract deep temporal features of the audio. Faudio is then input into the DH_live decoder, which has been fine-tuned by LoRA.

[0037] To achieve fine-grained control of actions, an action style embedding vector (Emotion) is introduced and injected into the intermediate feature layer of the decoder through an adaptive instance normalization (AdaIN) layer. The feature transformation process can be represented as follows:

[0038] Where AdaIN(h, Emotion) is the final output of the adaptive instance normalization operation; h is the intermediate layer feature map of the decoder; μ(·) and σ(·) represent the mean and standard deviation calculation operations of the features, respectively. is the channel-level standard deviation of the intermediate layer feature map h; Emotion represents the motion style feature; σ(Emotion) is the channel-level standard deviation of the motion style feature Emotion; μ(Emotion) is the channel-level mean of the motion style feature Emotion.

[0039] The features processed by the fine-tuned network are mapped to the deformation parameters ΔV of the facial vertices, and their driving function is formalized as:

[0040] Where ΔV is the real-time speech feature increment of the final output; DecoderDHlive is a deep latent space decoder for real-time scenarios; Faudio is the basic audio feature input; Emotion is the motion / emotion feature input; Θfrozen is the frozen weight of the pre-trained decoder; and ΔΘLoRA is the incremental parameter of the LoRA low-rank adaptation.

[0041] In addition, to ensure strict synchronization between lip movements and speech and to eliminate jitter, a time smoothing regularization term was added to the loss function: .

[0042] Where Ltemp is the temporal smoothing loss; ∑t is the summation operation over all time steps; ΔVt is the output increment at time step t; and ΔVt-1 is the output increment at time step t-1.

[0043] Ultimately, the system uses the △V generated by this self-developed model to drive a high-fidelity digital human model, achieving a high degree of naturalness in real-time driving that combines the generation quality of DH_live with specific action styles.

[0044] In terms of real-time interactive closed-loop, the computing power machine monitors the API interface of the live streaming platform in real time, capturing events such as audience entry, likes, and bullet comments. When an interactive event occurs, the integrated AI tools, such as the large language model, will generate relevant interactive text in real time, such as "Welcome XX to the live stream" or "Thank you XX for your like," based on the event type and content. The generated interactive text will immediately enter the localized speech synthesis and localized image-driven process to generate corresponding interactive voice and digital human video frames with corresponding expressions and actions, which will be preferentially inserted into the live stream to achieve human-like real-time feedback.

[0045] For example, a computing power cluster can consist of multiple hosts configured with RTX 3080 / 4070 level GPUs. Each host can run 2 to 4 independent computing power nodes simultaneously (each node corresponds to a live broadcast room). Each computing power node is allocated independent GPU memory and CPU resources and runs in Docker containers for isolation. One minute before the live broadcast starts, the computing power node pulls the model file of the specified digital human from OSS via a REST interface and loads it into the GPU memory. During the live broadcast, the node's built-in large language model automatically generates live broadcast scripts every minute based on a preset live broadcast script framework, combined with real-time product information and marketing activities, and sends them to the local speech synthesis module. The speech synthesis module runs a fine-tuned Index-TTS model to quickly generate highly natural audio. The audio is sent to an audio queue for transmission and to a Wav2Vec 2.0 model to extract temporal audio features.

[0046] After receiving the audio features, the digital human driving module first selects the corresponding emotion embedding vector "Emotion" based on the emotional tags of the current live-stream speech (generated simultaneously by the large language model, such as "introducing a product - excited," "answering a question - calm," and "thanking for a gift - joyful"). The audio features and emotion embedding vector are then input into the DH_live decoder, which is finely tuned using LoRA. The decoder performs feature modulation at each layer using AdaIN, injecting emotional information into the generation process. The decoder ultimately outputs a 512-dimensional 3DMM vertex deformation parameter ΔV, including coefficients for lip movements, facial expressions, head posture, and eye movements. ΔV is then input into the Unity / Unreal rendering engine to drive the high-fidelity digital human model to render 1080P 25fps video frames. Temporal smoothing filtering is incorporated during rendering to ensure smooth changes in motion parameters between adjacent frames, without jitter or abrupt transitions.

[0047] The real-time interactive thread of the computing cluster runs in parallel with the main thread, establishing long-term connections with the open platforms of various live streaming platforms via WebSocket to subscribe to live streaming event messages in real time. Upon receiving a "user enters" event, a welcome message is immediately generated; upon receiving a "gift" event, different levels of gratitude messages and corresponding emoticons are generated based on the gift's value; upon receiving a question in the chat, the question and product knowledge base are fed into a large language model to generate a professional answer. All interactive content is marked as high priority, sent to a priority queue, interrupting the playback of the current standard speech, and prioritizing the synthesis of interactive voice and actions to achieve rapid end-to-end interactive response, making viewers feel as if the digital human is truly interacting in real time.

[0048] The third layer of data acquisition is used as the system's signal conversion layer. Each data acquisition corresponds to a computing node. Through virtual device driver technology, it converts the digital audio and video streams output by the computing node into physical device signals that the operating system can recognize, including converting AI voice data streams into virtual microphone signals and digital human video streams into virtual camera signals.

[0049] The data collector acts as the system's "signal converter," responsible for converting the digital audio and video streams output by the computing power unit into physical device signals that the operating system can directly recognize, without requiring any special configuration from the live streaming software. Each data collector corresponds to a computing power node, and its core is kernel-level virtual device driver technology. In terms of audio signal conversion, the system writes the AI ​​voice data stream y(t) received via the network into the kernel circular buffer of the virtual audio device. The operating system and any live streaming software will recognize it as a real physical microphone input signal S_mic. The conversion process can be abstracted as follows: S_mic = VirtualAudioDriver(y(t)) Where y(t) is the AI ​​voice data stream output by the computing node; S_mic is the virtual microphone input signal recognized by the operating system; In terms of video signal conversion, the system simulates the received digital human video stream I_frame(n) (where n is the frame number) into a standard UVC camera acquisition signal S_cam through the kernel-mode virtual camera driver. The conversion process can be abstracted as follows: S_cam = VirtualCameraDriver(I_frame(n)) Where I_frame(n) is the nth frame of the digital human video output by the computing node; S_cam is the signal captured by the virtual camera recognized by the operating system.

[0050] The virtual device driver fully complies with the standard audio and video device specifications of the operating system, requiring no modification to the live streaming software and exhibiting extremely high compatibility.

[0051] The data acquisition unit can be a mini-PC based on an ARM architecture or run on the same machine as the computing power unit, installing a kernel-mode virtual audio driver (implemented based on Virtual Audio Cable) and a virtual camera driver (implemented based on v4l2loopback). Audio generated by the computing node is transmitted to the data acquisition unit in low-latency mode via the RTP protocol and written to the kernel buffer of the virtual audio device. The operating system immediately recognizes it as a microphone device named "Digital Mic". Video frames are transmitted to the data acquisition unit via shared memory and written to the frame buffer of the virtual camera device. The operating system recognizes it as a UVC camera device named "DigitalCamera". The entire conversion process is completed in kernel mode, resulting in low latency.

[0052] The fourth layer of live streaming machines is used as the system's push output layer. Each live streaming machine corresponds to a collector, which installs the official live streaming software of the corresponding live streaming platform. The virtual camera and virtual microphone generated by the corresponding collector are used as audio and video sources to push the digital human audio and video streams to the corresponding live streaming platform.

[0053] The live streaming device, acting as a "live streaming terminal," is responsible for the final streaming operation. The official live streaming tools of various platforms (such as Live Streaming Companion, Kuaishou Companion, etc.) are installed on the live streaming device, and the camera and microphone sources are selected as virtual devices S_cam and S_mic created by the collector. The live streaming device reads audio and video signals from these virtual devices and pushes them to the corresponding self-media platforms, thereby completing the live streaming.

[0054] Specifically, the live streaming device can be a regular mini PC without a dedicated GPU, connected to a monitor via HDMI, and equipped with the official live streaming companion software for the corresponding platform. In the live streaming companion settings, select "Digital Camera" for the camera and "Digital Mic" for the microphone, set the resolution to 1080P and the bitrate to 4Mbps, and then you can start live streaming. The live streaming software is completely unaware that this is a virtual device, and it captures audio and video and encodes and streams them to the platform server according to the normal process. When it is necessary to simultaneously live stream on four platforms—Douyin, Kuaishou, WeChat Channels, and Taobao—only one live streaming device, corresponding to one computing node and one acquisition device, is needed for each platform. This allows for four digital avatars to live stream simultaneously, and each digital avatar can have a different appearance, different content, and independent interaction.

[0055] The system supports elastic scaling. When it is necessary to temporarily increase the number of live streaming channels during a major promotion, simply add GPU nodes to the computing power cluster and connect the corresponding number of collectors and live streaming machines to quickly launch the new live streaming room; the cloud model warehouse and training machine do not require any modifications.

[0056] In one embodiment, a digital human driving method supporting multi-platform matrix live streaming is provided, implemented based on the digital human driving system supporting multi-platform matrix live streaming in the previous embodiment. The method includes: (1) Model training and distribution: Collect real-person audio and video data on the digital human training machine, train and generate digital human image model and personalized voice model, and push the trained model to the cloud model repository.

[0057] (2) Local reasoning and content generation: Each computing node pulls the required digital human model from the cloud, completes the live script generation locally, converts the text into AI speech through the local speech synthesis model, and then drives the digital human image according to the speech features and emotional embedding vector through the digital human driving model to generate digital human video frames, forming a complete digital human audio and video stream.

[0058] (3) Real-time interactive processing: The computing power node listens to the live streaming platform API in real time, captures audience interaction events, generates interactive text in real time according to the event type, prioritizes sending it into the speech synthesis and image-driven process, generates interactive audio and video content with corresponding facial expressions and actions, and inserts it into the live stream.

[0059] Real-time interactive processing specifically includes: computing nodes monitor the open APIs of various live streaming platforms in real time through long connections. When they detect events such as viewers entering the live room, liking, following, sending gifts, or sending bullet comments, they input the event content and live streaming context into the large language model to generate interactive response text that matches the live streaming context in real time. The interactive text is marked as high priority and immediately sent to the speech synthesis and image-driven queue, where it is processed before ordinary live streaming script content. This generates interactive audio and video with welcome and thank-you emoticons and actions, which are then inserted into the live stream for playback.

[0060] (4) Virtual signal conversion: The collector converts AI voice into a virtual microphone signal that the system recognizes through virtual audio driver, and converts digital human video stream into a virtual camera signal that the system recognizes through virtual camera driver.

[0061] (5) Multi-platform streaming: Each live streaming machine selects the corresponding virtual device as the audio and video source, and pushes the digital human content to the corresponding live streaming platform through the platform's official live streaming software to achieve multi-platform matrix live streaming.

[0062] This multi-platform matrix live streaming-supporting digital human-driven method is based on the previously implemented multi-platform matrix live streaming-supporting digital human-driven system. By deploying a high-fidelity digital human-driven model on local computing nodes and combining LoRA fine-tuning and AdaIN emotion injection mechanisms, it accurately reproduces the micro-expressions and body movement styles of specific digital humans while retaining the powerful generation capabilities of DH_live, effectively solving the problem of stiff performance caused by the disconnect between movement and emotion. By constructing an end-to-end localized speech synthesis and image-driven pipeline, it avoids dependence on cloud rendering and third-party APIs, significantly reducing bandwidth costs and system performance. Latency; leveraging virtual device driver technology, AI-generated audio and video streams are seamlessly converted into operating system standard microphone and camera signals, enabling plug-and-play stable access without modifying live streaming software; through a three-layer decoupled architecture of computing nodes, collectors, and live streaming machines, it supports on-demand elastic expansion, with a single GPU host capable of concurrently supporting multiple independent live streaming rooms, breaking through the computing power bottleneck of traditional single-machine single-cast; at the same time, based on an event-driven real-time interactive closed-loop mechanism, the digital human can provide anthropomorphic, high-priority, and instantaneous feedback to audience behavior, truly realizing intelligent live streaming interaction with perception, understanding, and response capabilities.

[0063] It should be noted that, in this document, terms such as “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0064] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A digital human driving system supporting multi-platform matrix live streaming, characterized in that, include: The digital human training machine serves as the asset production layer of the system. It is responsible for collecting real-person video and audio data, using digital human driving models and voice training models to generate high-fidelity digital human image models and personalized voice models, and pushing the trained model files to the cloud model repository for unified storage and distribution. The computing power cluster serves as the inference computing layer of the system, comprising multiple independent computing power nodes configured with GPUs. Each computing power node corresponds to a live streaming room and is responsible for pulling a specified digital human model from the cloud model repository, completing the live streaming script generation, speech synthesis, digital human image driving, and real-time interactive processing locally, generating a complete digital human audio and video stream. The computing power nodes have built-in local speech synthesis models and digital human driving models that have been fine-tuned with low-rank adaptation LoRA. The data acquisition unit serves as the signal conversion layer of the system. Each data acquisition unit corresponds to a computing node. Through virtual device driver technology, it converts the digital audio and video streams output by the computing node into physical device signals that the operating system can recognize, including converting AI voice data streams into virtual microphone signals and digital human video streams into virtual camera signals. The live streaming machine is used as the system's push output layer. Each live streaming machine corresponds to a collector, which installs the official live streaming software of the corresponding live streaming platform. It uses the virtual camera and virtual microphone generated by the corresponding collector as audio and video sources to push the digital human's audio and video streams to the corresponding live streaming platform.

2. The digital human driving system supporting multi-platform matrix live streaming according to claim 1, characterized in that, The local speech synthesis model of the computing node is obtained using a low-rank adaptation LoRA fine-tuning scheme based on the Index-TTS unified speech model.

3. The digital human driving system supporting multi-platform matrix live streaming according to claim 2, characterized in that, The low-rank adaptation LoRA fine-tuning scheme used to obtain the Index-TTS unified speech model includes: During the training phase, the pre-trained pedestal weights are not updated; only the LoRA low-rank matrix on the GPT module and the learnable condition vector mean_condition_{speaker_id} for each scene are trained. The training process is as follows: First, audio and corresponding text for specific scenes are collected, and abnormal samples with a duration of 1-20 seconds are filtered out. Then, each audio is resampled to 24kHz, and a 100-dimensional logarithmic Mel spectrum is extracted. This spectrum is quantized into discrete Mel codes by a pre-trained DVAE and then processed by a Conformer+Perceiver encoder to obtain a condition latent with a shape of (32, 1024). Next, the pairwise distance matrix is ​​calculated for all condition latents, and the medoid is selected as the scene center condition vector and registered as a trainable parameter. The data is then divided into a training set and a validation set in a 9:1 ratio.

4. The digital human driving system supporting multi-platform matrix live streaming according to claim 3, characterized in that, The acquisition of the Index-TTS unified speech model using the low-rank adaptation LoRA fine-tuning scheme also includes: During the model loading phase, all base parameters of Unified Voice are frozen, and LoRA is injected only into four linear layers (attn.c_attn, attn.c_proj, mlp.c_fc, and mlp.c_proj) out of the 20 layers of GPT. During training, the input sequence is constructed and forward propagated through GPT, using a joint loss function of text loss and Mel-spectral loss. L = 0.1·L_text + 0.9·L_mel Where L_text is the text prediction loss and L_mel is the mel spectrum prediction loss; during inference, the fine-tuned conditional vector is loaded according to the anchor identifier, and after the autoregression generates mel codes, it is decoded into a 24kHz audio waveform by the BigVGAN vocoder.

5. The digital human driving system supporting multi-platform matrix live streaming according to claim 1, characterized in that, The digital human driving model of the computing power node is obtained based on the DH_live architecture using a parameter-efficient fine-tuning scheme. In this scheme, a LoRA module is injected into the attention mechanism layer of the pre-trained Transformer backbone network. The low-rank adaptation matrix of the LoRA module satisfies: ； Where ΔW is the low-rank weight update matrix; B is the upgraded low-rank matrix with dimension 1. ； A is a dimension-reduced low-rank matrix with dimensions of . ； During training, the reconstruction loss function is minimized as shown in the following formula: ； Where Lrec is the reconstruction loss; Xtarget is the target real audio feature; Xaudio is the input raw audio feature; fDH(·) is a pre-trained audio feature mapping network used to map the input audio features to the output predicted audio features; Θfrozen is the frozen parameter of the pre-trained model; ΔΘ is the fine-tunable incremental parameter.

6. The digital human driving system supporting multi-platform matrix live streaming according to claim 5, characterized in that, The digital human-driven model introduces an action style embedding vector E_motion, and injects emotional features into the intermediate feature layer of the decoder through an adaptive instance normalization AdaIN layer. The feature transformation process is as follows: ； Where AdaIN(h, Emotion) is the final output of the adaptive instance normalization operation; h is the intermediate layer feature map of the decoder; μ(·) and σ(·) represent the mean and standard deviation calculation operations of the features, respectively; is the channel-level standard deviation of the intermediate layer feature map h; Emotion represents the motion style feature; σ(Emotion) is the channel-level standard deviation of the motion style feature Emotion; μ(Emotion) is the channel-level mean of the motion style feature Emotion; The features after emotion modulation are ultimately mapped to the facial vertex deformation parameter ΔV, and its driving function is formalized as follows: ； Where ΔV is the real-time speech feature increment of the final output; DecoderDHlive is a deep latent space decoder for real-time scenarios; Faudio is the basic audio feature input; Emotion is the motion / emotion feature input; Θfrozen is the frozen weight of the pre-trained decoder; and ΔΘLoRA is the incremental parameter of the LoRA low-rank adaptation.

7. The digital human driving system supporting multi-platform matrix live streaming according to claim 6, characterized in that, During training, a time smoothing regularization term is also added to the loss function as follows: ； Where Ltemp is the temporal smoothing loss; ∑t is the summation operation over all time steps; ΔVt is the output increment at time step t; and ΔVt-1 is the output increment at time step t-1.

8. The digital human driving system supporting multi-platform matrix live streaming according to claim 1, characterized in that, The virtual audio driver conversion process of the acquisition device is as follows: S_mic = VirtualAudioDriver(y(t)) Where y(t) is the AI ​​voice data stream output by the computing node; S_mic is the virtual microphone input signal recognized by the operating system; The virtual camera driver conversion process is as follows: S_cam = VirtualCameraDriver(I_frame(n)) Where I_frame(n) is the nth frame of the digital human video output by the computing node; S_cam is the signal captured by the virtual camera recognized by the operating system.

9. A digital human-driven method supporting multi-platform matrix live streaming, characterized in that, Based on the digital human driving system supporting multi-platform matrix live streaming as described in any one of claims 1-8, the method includes: Model training and distribution: Collect real-person audio and video data on the digital human training machine, train and generate digital human image models and personalized voice models, and push the trained models to the cloud model repository. Local inference and content generation: Each computing node pulls the required digital human model from the cloud, completes the live broadcast script generation locally, converts the text into AI speech through the local speech synthesis model, and then drives the digital human image based on speech features and emotional embedding vectors through the digital human driving model to generate digital human video frames, forming a complete digital human audio and video stream. Real-time interactive processing: The computing power nodes monitor the live streaming platform API in real time, capture audience interaction events, generate interactive text in real time according to the event type, and send it to the speech synthesis and image-driven process first to generate interactive audio and video content with corresponding facial expressions and actions and insert it into the live stream. Virtual signal conversion: The acquisition device converts AI voice into a virtual microphone signal that the system can recognize through a virtual audio driver, and converts digital human video stream into a virtual camera signal that the system can recognize through a virtual camera driver; Multi-platform streaming: Each live streaming device selects the corresponding virtual device as the audio and video source, and pushes the digital human content to the corresponding live streaming platform through the platform's official live streaming software to achieve multi-platform matrix live streaming.

10. The digital human driving method supporting multi-platform matrix live streaming according to claim 9, characterized in that, The real-time interactive processing specifically includes: the computing nodes monitor the open APIs of various live streaming platforms in real time through long connections. When they detect that a viewer has entered the live stream, liked, followed, sent a gift, or sent a bullet screen, they input the event content and the live stream context into the large language model to generate interactive response text that matches the live stream context in real time. The interactive text is marked as high priority and immediately sent to the speech synthesis and image-driven queue, where it is processed before ordinary live stream script content. Interactive audio and video with welcome and thank you expressions and actions are generated and immediately inserted into the live stream for playback.

Images