A scheduling method and system for multi-digital split screen synchronization driving

Abstract

The application provides a scheduling method and system for multi-digital avatar synchronous screen driving, wherein the method comprises: collecting real-time facial key point data and emotion feature vectors of a driving source, the facial key point data comprising key point coordinates, eye opening degree, mouth opening degree, FACS activation intensity, and the yaw angle, pitch angle and roll angle of the head posture; inputting the emotion feature vectors into an emotion-action mapping model to generate corresponding action control parameter sequences; distributing the action control parameter sequences to multiple digital avatars on the same screen through a timestamp alignment mechanism, so that each digital avatar completes synchronous update of expressions and actions within the same frame; and dynamically allocating scheduling priorities of each digital avatar to control end-to-end driving delay within a preset threshold. The application can greatly reduce the inter-frame synchronization error of multiple digital avatars, significantly improve the naturalness of digital avatar expression transition, and improve the visual continuity of multi-digital avatar synchronous screen driving.

Description

Technical Field

[0001] This invention relates to the field of digital human image generation technology, and in particular to a scheduling method and system for synchronously driving multiple digital clones on the same screen. Background Technology

[0002] With the widespread application of digital human technology in scenarios such as virtual anchors, online education, and metaverse social networking, existing single-channel digital avatar driving solutions typically employ a serial pipeline of "acquisition-inference-rendering," resulting in end-to-end latency generally exceeding 100ms. This makes it difficult to meet the stringent synchronization requirements of real-time multi-user interaction. When multiple digital avatars need to be driven simultaneously on the same screen, the lack of a unified time reference between the avatars leads to noticeable inter-frame misalignment and facial expression asynchrony, severely impacting the user experience.

[0003] In terms of emotion and action mapping, existing solutions mostly use rule-based mapping tables or simple linear interpolation, which can only handle discrete basic expression categories (such as joy, anger, sorrow, and happiness). They cannot perform fine-grained modeling of continuous emotional changes, resulting in stiff facial expression transitions and a disconnect between actions and emotional semantics in digital avatars. In addition, existing multi-path rendering solutions typically allocate fixed computing resources to all digital avatar instances, failing to differentiate scheduling based on the salience of each avatar on the screen and the user's interaction status, resulting in wasted GPU resources and difficulty in balancing overall frame rate and latency performance.

[0004] In summary, the existing technology has the following main drawbacks: First, the multi-channel digital clones lack a unified timestamp alignment mechanism, resulting in insufficient synchronization accuracy on the same screen; second, the emotion-action mapping granularity is coarse, and the ability to model continuous emotion space is weak; third, the rendering resource scheduling strategy is statically fixed and cannot be dynamically optimized according to the runtime state, resulting in high end-to-end latency and poor visual continuity. Summary of the Invention

[0005] Therefore, it is necessary to provide a scheduling method and system for simultaneous driving of multiple digital clones on the same screen to address the above-mentioned technical problems.

[0006] A scheduling method for simultaneous driving of multiple digital clones on the same screen includes the following steps: collecting real-time facial key point data and emotion feature vectors from the driving source, wherein the facial key point data includes key point coordinates, eye opening and closing degree, mouth opening and closing degree, FACS activation intensity, and head posture yaw angle, pitch angle, and roll angle; inputting the emotion feature vectors into an emotion-action mapping model to generate a corresponding action control parameter sequence; distributing the action control parameter sequence to multiple digital clones on the same screen through a timestamp alignment mechanism, so that each digital clone can complete the synchronous update of expression and action within the same frame; dynamically allocating the scheduling priority of each digital clone to control the end-to-end driving delay within a preset threshold.

[0007] In one embodiment, the acquisition of real-time facial key point data and emotional feature vectors from the driving source includes: acquiring the driving source video stream and extracting the three-dimensional coordinates of multiple key points in each frame of the driving source video stream; smoothing the inter-frame trajectories of the key points using Kalman filtering technology; and regressing a three-dimensional emotional feature vector based on the smoothed key point geometric features. The three-dimensional emotional feature vector includes valence, arousal, and dominance, where valence is used to characterize the positive or negative nature of the emotion, arousal is used to characterize the degree of emotional activation, and dominance is used to characterize the sense of emotional control.

[0008] In one embodiment, the step of inputting the emotion feature vector into the emotion-action mapping model to generate a corresponding action control parameter sequence includes: receiving the three-dimensional emotion feature vector, performing forward inference through a multilayer perceptron, and outputting a multi-dimensional action control parameter sequence, wherein the action control parameter sequence includes facial motion unit intensity coefficients and joint angle offsets.

[0009] In one embodiment, the step of distributing the motion control parameter sequence to multiple digital clones on the same screen through a timestamp alignment mechanism, so that each digital clone can synchronize the update of expressions and actions within the same frame, includes: attaching a globally monotonically increasing timestamp to the motion control parameter sequence of the current frame, and pushing the parameter packet with the timestamp into the circular buffer queue of each digital clone; when each digital clone retrieves a frame, if the timestamp of the data at the head of the queue is earlier than the reference timestamp of the current rendering frame, the expired frame is discarded until the latest frame that satisfies the requirement that the timestamp of the data at the head of the queue is greater than or equal to the reference timestamp is retrieved, so that all clones use time-consistent driving parameters within the same rendering frame, thus completing frame synchronization.

[0010] In one embodiment, the scheduling priority of each digital clone is dynamically allocated, including: calculating a priority score based on the display area ratio of each digital clone on the screen and the frequency of user interaction; performing priority scheduling allocation based on the priority score; and adopting a frame interpolation compensation strategy for the corresponding digital clone when the priority score is lower than the score threshold.

[0011] In one embodiment, the facial key point data is extracted in real time using a lightweight face mesh model, and the number of facial key points is no less than 468.

[0012] A scheduling system for simultaneous driving of multiple digital clones on the same screen, used to implement the scheduling method for simultaneous driving of multiple digital clones on the same screen as described above, includes: a face capture module, an emotion-action mapping module, a synchronization scheduler, and a multi-path rendering engine; the face capture module is used to collect facial key point data of the driving source in real time and extract emotion feature vectors; the emotion-action mapping module, connected to the face capture module, is used to convert the emotion feature vectors into a sequence of action control parameters; the synchronization scheduler, connected to the emotion-action mapping module, is used to distribute the sequence of action control parameters to multiple digital clones based on a timestamp alignment mechanism and dynamically allocate the scheduling priority of each instance; the multi-path rendering engine, connected to the synchronization scheduler, is used to drive each digital clone in parallel to complete the synchronous rendering output of expressions and actions in the same frame.

[0013] In one embodiment, the face capture module includes a lightweight face mesh inference chip and a Kalman filter processor. The lightweight face mesh inference chip is used to extract facial key points from the driving source in real time, and the Kalman filter processor is used to smooth the inter-frame trajectory of the facial key points.

[0014] In one embodiment, the synchronization scheduler includes a frame buffer queue and a priority arbiter. The frame buffer queue maintains an independent timestamped parameter buffer for each digital clone, and the priority arbiter dynamically adjusts the GPU time slice allocation ratio of each digital clone based on the display area ratio and interaction frequency.

[0015] In one embodiment, the multi-path rendering engine adopts a layered rendering architecture with a shared skeletal animation base layer and an independent differential expression layer. The skeletal animation base layer is used to share the same set of skeletal skinning calculation results for all digital clones, and broadcasts them to each digital clone after calculation. The differential expression layer is used to independently calculate the blended shape weight driven by the AU intensity coefficient of each digital clone.

[0016] Compared with existing technologies, the advantages and beneficial effects of this invention are as follows: By acquiring the video stream of the driving source, real-time facial key point data and emotional feature vectors are obtained. The facial key point data includes key point coordinates, eye opening and closing, mouth opening and closing, FACS activation intensity, and head posture yaw angle, pitch angle, and roll angle, thus achieving full acquisition of facial and emotional data. The emotional feature vectors are input into the emotion-action mapping model to generate corresponding action control parameter sequences to control the subsequent actions of the digital clones. A timestamp alignment mechanism is used to distribute the action control parameter sequences to multiple digital clones on the same screen, enabling each digital clone to complete the synchronous update of the wristwatch and actions within the same frame, achieving synchronous driving of multiple digital clones on the same screen. The scheduling priority of each digital clone is dynamically allocated to keep the end-to-end driving delay within a preset threshold. The rendering output of each digital clone is completed according to the scheduling priority, which greatly reduces the inter-frame synchronization error of multiple digital clones, significantly improves the naturalness of the digital clone's expression transition and response rate, and improves the visual continuity of synchronous driving of multiple digital clones on the same screen. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a scheduling method for simultaneous driving of multiple digital clones on the same screen in one embodiment.

[0018] Figure 2 This is a schematic diagram of the structure of a scheduling system for simultaneous driving of multiple digital clones on the same screen in one embodiment. Detailed Implementation

[0019] Before describing the specific embodiments of the present invention, the overall concept of the present invention will be explained as follows: This invention is mainly based on the digital clone generation process. Currently, the progress of simultaneous synchronization of multiple digital clones on the same screen is insufficient, the ability to model continuous emotion space is weak, and the end-to-end latency is relatively high.

[0020] Therefore, this invention proposes a scheduling method for simultaneous driving of multiple digital clones on the same screen. By acquiring the video stream of the driving source, real-time facial key point data and emotional feature vectors are obtained. The facial key point data includes key point coordinates, eye opening and closing degree, mouth opening and closing degree, FACS activation intensity, and head posture yaw angle, pitch angle, and roll angle, thus achieving full acquisition of facial and emotional data. The emotional feature vector is input into the emotion-action mapping model to generate a corresponding action control parameter sequence to control the subsequent actions of the digital clones. A timestamp alignment mechanism is used to distribute the action control parameter sequence to multiple digital clones on the same screen, enabling each digital clone to complete the synchronous update of the wristwatch and actions within the same frame, achieving simultaneous driving of multiple digital clones on the same screen. The scheduling priority of each digital clone is dynamically allocated to keep the end-to-end driving delay within a preset threshold. The rendering output of each digital clone is completed according to the scheduling priority, which significantly reduces the inter-frame synchronization error of multiple digital clones, significantly improves the naturalness of the digital clones' expression transitions and response rate, and improves the visual continuity of simultaneous driving of multiple digital clones on the same screen.

[0021] Having introduced the overall concept of the present invention, to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0022] In one embodiment, such as Figure 1 As shown, a scheduling method for simultaneous driving of multiple digital clones on the same screen is provided, including the following steps: Step S110: Collect real-time facial key point data and emotion feature vectors from the driving source. The facial key point data includes key point coordinates, eye opening and closing, mouth opening and closing, FACS activation intensity, and head posture yaw angle, pitch angle, and roll angle.

[0023] Specifically, the driving source can be a camera or audio. When the camera or audio is detected to be turned on, the video stream from the driving source is continuously acquired at a set frame rate, such as 60fps, to obtain real-time facial key point data and emotional feature vectors. The facial key point data can include key point coordinates, eye opening and closing, mouth opening and closing, FACS (fluorescence activated cell sorter) activation intensity, and head posture yaw angle, pitch angle, and roll angle. Based on the facial key point data, such as mouth opening and closing and eye opening and closing, an emotional feature vector can be obtained for subsequent digital clone rendering.

[0024] If no valid face is detected in the current frame, an empty vector flag is output, the current frame is replaced by the previous valid frame, and a degradation notification is sent to the upper layer. If more than the set frame threshold is lost consecutively, the system switches to the pre-recorded driving sequence or silent posture.

[0025] Among them, the facial key point data is extracted in real time using a lightweight face mesh model, and the number of facial key points is no less than 468.

[0026] Specifically, when collecting area key point data, a lightweight face mesh model can be used for real-time extraction to achieve rapid extraction of key point data with a small footprint; no less than 468 facial key point data points are extracted to ensure sufficient acquisition of facial data and ensure the accuracy of subsequent digital clone rendering.

[0027] Step S110 includes: acquiring the driving source video stream and extracting the three-dimensional coordinates of multiple key points in each frame image; using Kalman filtering technology to smooth the inter-frame trajectory of the key points; and regressing a three-dimensional emotion feature vector based on the geometric features of the smoothed key points. The three-dimensional emotion feature vector includes valence, arousal, and dominance. Valence is used to characterize the positive or negative nature of emotion, arousal is used to characterize the degree of emotional activation, and dominance is used to characterize the sense of emotional control.

[0028] Specifically, the video stream of the driving source is acquired, and the three-dimensional coordinates of multiple key points in each frame of the image are extracted based on the video stream of the driving source. Kalman filtering technology is used to smooth the inter-frame trajectory of the key points to suppress jitter noise. The state equation is set as a uniform motion model, the process noise covariance Q=0.01I, and the observation noise covariance R=0.1I, which effectively suppresses jitter noise caused by changes in lighting or rapid movement and significantly reduces the jitter amplitude of key points. Based on the geometric features of the smoothed key points, a three-dimensional emotion feature vector is regressed. The three-dimensional emotion feature vector (V, A, D) is defined as follows: valence V∈[-1,1] represents the positive or negative nature of emotion, arousal A∈[0,1] represents the degree of emotional activation, and dominance D∈[0,1] represents the sense of emotional control. This allows for the construction of a continuous emotion space based on valence, arousal, and dominance to render the emotions of the digital clone.

[0029] Step S120: Input the emotion feature vector into the emotion-action mapping model to generate the corresponding action control parameter sequence.

[0030] Specifically, after obtaining the emotion feature vector, it is input into the emotion-action mapping model. The emotion-action mapping model is equipped with a fine-grained emotion-action mapping table based on probability distribution, so as to obtain the corresponding action according to the emotion mapping and generate the corresponding action control parameter sequence, which facilitates the control of the digital clone to perform subsequent actions and ensures the continuity of the digital clone's subsequent emotions.

[0031] Step S120 includes: receiving a three-dimensional emotion feature vector, performing forward reasoning through a multilayer perceptron, and outputting a multi-dimensional motion control parameter sequence, which includes facial motion unit intensity coefficients and joint angle offsets.

[0032] Specifically, the received emotion feature vector is input into the emotion-action mapping model, which employs a multilayer perceptron structure. The input layer receives the three-dimensional emotion feature vector, which passes through two hidden layers, each with 256 neurons and ReLU activation function. Finally, the output layer outputs 52 facial motion unit intensity coefficients and 24 joint angle offsets for the whole body, totaling a 78-dimensional sequence of motion control parameters. The model has approximately 200K parameters, and the inference time for a single iteration is approximately 0.8ms. Taking the "happy" emotion (V=0.8, A=0.7, D=0.5) as an example, the output intensity of AU6 (cheek lift) is 0.72, and the output intensity of AU12 (corner of the mouth turn up) is 0.85, with a mean absolute error (MAE) of 0.031 compared to the manually labeled baseline.

[0033] Through emotion-action mapping, the mapping table between emotions and actions can be updated in real time without stopping the system or interrupting rendering. In addition, copy-on-write technology can be used to seamlessly switch between the old and new mapping tables.

[0034] Step S130: The motion control parameter sequence is distributed to multiple digital clones on the same screen through a timestamp alignment mechanism, so that each digital clone can complete the synchronous update of expressions and actions within the same frame.

[0035] Specifically, a timestamp alignment mechanism is adopted to distribute the sequence of motion control parameters to multiple digital clones on the same screen, so that each digital clone can complete the synchronous update of expressions and actions within the same frame, achieve global timestamp alignment, ensure that all digital clones use time-consistent driving parameters within the same rendering frame, and significantly reduce synchronization errors.

[0036] Step S130 includes: attaching a globally monotonically increasing timestamp to the current frame action control parameter sequence, and pushing the timestamped parameter packet into the circular buffer queue of each digital clone; when each digital clone retrieves a frame, if the timestamp of the queue head data is earlier than the reference timestamp of the current rendering frame, the expired frame is discarded until the latest frame that satisfies the requirement that the timestamp of the queue head data is greater than or equal to the reference timestamp is retrieved, so that all clones use time-consistent driving parameters in the same rendering frame, thus completing frame synchronization.

[0037] Specifically, during global timestamp alignment, a globally monotonically increasing timestamp is appended to the current frame's motion control parameter sequence. This timestamp can be generated by a high-precision timer of the main control CPU with an accuracy of 1μs. The timestamped parameter packet is then pushed into the circular buffer queue of each digital clone. When each digital clone retrieves a frame, if the timestamp T of the data at the head of the queue is... head Earlier than the base timestamp T of the current rendered frame render If the expired frame is not found, it will be discarded until a frame that satisfies T is found. head ≥T render The latest frame ensures that all clones use the same driving parameters within the same rendering frame, achieving frame synchronization and reducing synchronization errors.

[0038] Step S140: Dynamically allocate the scheduling priority of each digital clone to control the end-to-end drive delay within a preset threshold.

[0039] Specifically, since there are multiple digital clones, in order to render multiple digital clones simultaneously, the scheduling priority of each digital clone is dynamically allocated, thereby controlling the end-to-end driving latency to always be within a preset threshold. The preset threshold can be set according to the actual situation, and the setting range can be 50ms~200ms, so as to realize the synchronous driving of multiple digital clones on the same screen.

[0040] The specific steps for dynamically allocating the scheduling priority of each digital clone include: calculating a priority score based on the display area ratio of each digital clone on the screen and the frequency of user interaction; allocating priority scheduling based on the priority score; and using a frame interpolation compensation strategy for the corresponding digital clone when the priority score is lower than the score threshold.

[0041] Specifically, priority scores are calculated based on the screen area ratio of each digital clone and the frequency of user interaction, using the following formula: P i =0.6 × display area ratio + 0.4 × interaction frequency; In the formula, P i Score based on priority.

[0042] Priority scheduling and allocation are performed based on priority scores. Digital clones with higher priority scores receive more GPU computing time slices. When a priority score is lower than the score threshold, a frame interpolation compensation strategy is adopted for the corresponding digital clone to improve visual continuity and dynamic clarity. Ultimately, this enables simultaneous driving of multiple digital clones on the same screen, ensuring the continuity of simultaneous driving of multiple digital clones on the same screen.

[0043] In this embodiment, real-time facial key point data and emotion feature vectors are obtained by acquiring the driving source video stream. The facial key point data includes key point coordinates, eye opening and closing, mouth opening and closing, FACS activation intensity, and head posture yaw angle, pitch angle, and roll angle, thus achieving full acquisition of facial and emotion data. The emotion feature vector is input into the emotion-action mapping model to generate a corresponding action control parameter sequence to control the subsequent actions of the digital clone. A timestamp alignment mechanism is used to distribute the action control parameter sequence to multiple digital clones on the same screen, enabling each digital clone to complete the synchronous update of the wristwatch and actions within the same frame, achieving synchronous driving of multiple digital clones on the same screen. The scheduling priority of each digital clone is dynamically allocated to keep the end-to-end driving delay within a preset threshold. The rendering output of each digital clone is completed according to the scheduling priority, which greatly reduces the inter-frame synchronization error of multiple digital clones, significantly improves the naturalness of the digital clone's expression transition and response rate, and improves the visual continuity of synchronous driving of multiple digital clones on the same screen.

[0044] like Figure 2 As shown, a scheduling system 100 for simultaneous driving of multiple digital clones on the same screen is provided to implement the scheduling method for simultaneous driving of multiple digital clones on the same screen as described above. The system includes: a face capture module, an emotion-action mapping module, a synchronization scheduler, and a multi-path rendering engine. Specifically: the face capture module 10 is used to collect facial keypoint data from the driving source in real time and extract emotion feature vectors; the emotion-action mapping module 20 is connected to the face capture module 10 and is used to convert the emotion feature vectors into a sequence of action control parameters; the synchronization scheduler 30 is connected to the emotion-action mapping module 20 and is used to distribute the action control parameter sequence to multiple digital clone rendering instances based on a timestamp alignment mechanism and dynamically allocate the scheduling priority of each instance; the multi-path rendering engine 40 is connected to the synchronization scheduler 30 and is used to drive each digital clone in parallel to complete the synchronous rendering output of expressions and actions within the same frame.

[0045] The face capture module 10 includes a lightweight face mesh inference chip and a Kalman filter processor. The lightweight face mesh inference chip is used to extract facial key points from the driving source in real time; the Kalman filter processor is used to smooth the inter-frame trajectory of the facial key points.

[0046] Specifically, the face capture module 10 is deployed on the driver-end device (such as a camera acquisition host), and incorporates a lightweight face mesh inference chip and a Kalman filter processor. The lightweight face mesh inference chip 11 uses the MobileNetV3 backbone network, with a single-frame inference time of no more than 3ms, and can extract the three-dimensional coordinates (x, y, z) of no less than 468 key points on the driver source face in real time. The Kalman filter processor smooths the inter-frame trajectory of each key point, sets the state equation to a uniform motion model, and sets the process noise covariance Q=0.01I and the observation noise covariance R=0.1I, effectively suppressing jitter noise caused by changes in illumination or rapid movement, reducing the jitter amplitude of key points by approximately 73% compared to the unfiltered version.

[0047] Specifically, the emotion-action mapping unit 20 is carried by an independent inference accelerator card and connected to the main control CPU via a PCIe interface. The emotion-action mapping unit 20 adopts a multilayer perceptron structure. The input layer receives a three-dimensional emotion feature vector (valence V, arousal A, dominance D), which passes through two hidden layers (each with 256 neurons and the activation function ReLU). The output layer outputs the intensity coefficients of 52 facial action units (AU) and the angle offsets of 24 joints throughout the body, totaling a 76-dimensional sequence of action control parameters. The model has approximately 200K parameters, and a single inference operation takes approximately 0.8ms.

[0048] The synchronization scheduler includes a frame buffer queue and a priority arbiter. The frame buffer queue maintains an independent timestamped parameter buffer for each digital clone rendering instance. The priority arbiter dynamically adjusts the GPU time slice allocation ratio of each digital clone based on the display area ratio and interaction frequency.

[0049] Specifically, the synchronization scheduler 30 is integrated into the main control CPU and includes two sub-modules: a frame buffer queue manager and a priority arbiter. The frame buffer queue manager maintains an independent circular buffer queue for each digital clone rendering instance i (i=1…N), with a queue depth of 4 frames. Each frame of data carries a globally monotonically increasing timestamp T. The priority arbiter periodically (every frame) reads the display area percentage S of each digital clone. i Frequency of user interaction F i According to formula P i = α·S i + β·F i (α=0.6, β=0.4) Calculate the priority score P i Based on this, it requests time slices from the GPU scheduler, and the proportion of GPU time slices obtained by high-priority instances is no less than its P. i The percentage of the total score.

[0050] The priority arbiter performs a priority recalculation before the start of each rendering frame. Taking four digital clones on screen simultaneously as an example: Clone A's display area occupies S... A =0.40, interaction frequency F A =0.8, P A =0.6×0.40+0.4×0.8=0.56; S of clone B B =0.30, F B =0.3, P B =0.30; S of clone C C =0.20, F C =0.1, P C =0.16; S of clone D D =0.10, F D =0.0, P D =0.06. GPU time slices are calculated by P. i With normalized allocation, clone A receives approximately 49% of the time slice, while clone D receives only approximately 5%. For P... i For rendering instances below the threshold θ=0.10 (such as clone D), the frame interpolation compensation strategy is enabled: based on the output of the previous frame, the current frame is generated by linear interpolation to maintain visual continuity, while freeing up GPU computing power for high-priority instances.

[0051] The multi-path rendering engine adopts a layered rendering architecture with a shared skeletal animation base layer and an independent differential expression layer. The skeletal animation base layer is used to share the same set of skeletal skinning calculation results for all digital clones, and broadcasts them to each digital clone after calculation. The differential expression layer is used to independently calculate the blend shape weight driven by the AU intensity coefficient of each digital clone.

[0052] Specifically, the multi-path rendering engine 40 is carried by a GPU cluster and interconnected via NVLink, supporting parallel driving of N-path (N≥2) digital clone rendering instances 1 to N.

[0053] The multi-path rendering engine 40 employs a layered rendering architecture with a shared skeletal animation base layer and independent differentiated expression layers. The base layer shares the same set of skeletal skinning calculation results across all digital avatar rendering instances, calculating only once (differentiated expression parameters) and then broadcasting it to each instance, saving approximately 60% of skeletal transformation calculations. The differentiated expression layer independently calculates blend shape weights driven by AU intensity coefficients for each instance, ensuring personalized differences in the expressions of each avatar.

[0054] In one embodiment, to verify the technical effect of the present invention, a quantitative comparison was made with two comparative schemes on an experimental platform configured with an Intel Core i9-13900K CPU, an NVIDIA RTX 4090 GPU, and 64GB of memory, using simultaneous driving of four digital clones on the same screen as the test scenario.

[0055] Comparison with Option 1 (Baseline Option): This option uses polling scheduling, has no timestamp alignment, no priority allocation, and each clone renders its skeletal animation independently.

[0056] Comparison Scheme 2 (partially optimized): Introduces timestamp alignment, but lacks priority scheduling and layered rendering. The present invention's solution: Fully implements all the above-mentioned technical features.

[0057] The experimental results are shown in Table 1 below: Table 1 Experimental Results

[0058] As shown in Table 1 above, the end-to-end driving latency of the present invention is 23.6ms, which is about 73% lower than the baseline solution and about 55% lower than some optimized solutions, meeting the requirements of the preset threshold (≤30ms); the synchronization rate of 4-way same frame reaches 99.7%, which is 38.5 percentage points higher than the baseline solution; the GPU utilization rate is reduced from 94.8% to 72.5%, effectively releasing computing power margin and ensuring system stability; the frame rate is stably maintained at 60fps, achieving the smooth driving standard.

[0059] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0060] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a computer storage medium (ROM / RAM, magnetic disk, optical disk) for execution by the computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Therefore, the present invention is not limited to any particular hardware and software combination.

[0061] The above description, in conjunction with specific embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such deductions or substitutions should be considered within the scope of protection of the present invention.

Claims

1. A scheduling method for simultaneous driving of multiple digital clones on the same screen, characterized in that, Includes the following steps: Real-time facial key point data and emotion feature vectors of the driving source are collected. The facial key point data includes key point coordinates, eye opening and closing, mouth opening and closing, FACS activation intensity, and head posture yaw angle, pitch angle and roll angle. The emotion feature vector is input into the emotion-action mapping model to generate a corresponding sequence of action control parameters. The motion control parameter sequence is distributed to multiple digital clones on the same screen through a timestamp alignment mechanism, so that each digital clone can complete the synchronous update of expressions and actions within the same frame. The scheduling priority of each digital clone is dynamically allocated to keep the end-to-end drive latency within a preset threshold.

2. The scheduling method for simultaneous driving of multiple digital clones on the same screen according to claim 1, characterized in that, The real-time facial key point data and emotion feature vectors acquired by the acquisition driver include: Acquire the video stream from the driving source and extract the 3D coordinates of multiple key points in each frame of the video stream. Kalman filtering is used to smooth the inter-frame trajectories of the key points; Based on the smoothed key point geometric features, a three-dimensional emotion feature vector is regressed. The three-dimensional emotion feature vector includes valence, arousal and dominance. Valence is used to characterize the positive or negative nature of emotion, arousal is used to characterize the degree of emotional activation, and dominance is used to characterize the sense of emotional control.

3. The scheduling method for simultaneous driving of multiple digital clones on the same screen according to claim 2, characterized in that, The step of inputting the emotion feature vector into the emotion-action mapping model to generate a corresponding sequence of action control parameters includes: The three-dimensional emotion feature vector is received and forward reasoning is performed through a multilayer perceptron to output a multi-dimensional motion control parameter sequence, which includes facial motion unit intensity coefficients and joint angle offsets.

4. The scheduling method for simultaneous driving of multiple digital clones on the same screen according to claim 1, characterized in that, The process of distributing the motion control parameter sequence to multiple digital clones on the same screen via a timestamp alignment mechanism, enabling each digital clone to synchronize its facial expressions and movements within the same frame, includes: Append a globally monotonically increasing timestamp to the current frame motion control parameter sequence, and push the timestamped parameter packet into the circular buffer queue of each digital clone; When each digital clone retrieves a frame, if the timestamp of the queue head data is earlier than the base timestamp of the current rendering frame, the expired frame will be discarded until the latest frame that satisfies the condition that the timestamp of the queue head data is greater than or equal to the base timestamp is retrieved. This ensures that all clones use the same time driving parameters within the same rendering frame, thus completing frame synchronization.

5. The scheduling method for simultaneous driving of multiple digital clones on the same screen as described in claim 1, characterized in that, The dynamic allocation of scheduling priorities for each digital clone includes: Priority scores are calculated based on the screen area ratio of each digital clone and the frequency of user interaction. Priority scheduling and allocation are performed based on the priority scores. When the priority score is lower than the score threshold, a frame interpolation compensation strategy is adopted for the corresponding digital clone.

6. The scheduling method for simultaneous driving of multiple digital clones on the same screen as described in claim 1, characterized in that, The facial key point data is extracted in real time using a lightweight face mesh model, and the number of facial key points is no less than 468.

7. A scheduling system for simultaneous driving of multiple digital clones on the same screen, characterized in that, A scheduling method for implementing the multi-digital clone simultaneous screen synchronous driving as described in any one of claims 1-6 includes: Facial capture module, emotion-motion mapping module, synchronization scheduler, and multi-path rendering engine; The face capture module is used to collect facial key point data of the driving source in real time and extract emotional feature vectors. The emotion-action mapping module and the face capture module are used to convert the emotion feature vector into a sequence of action control parameters; The synchronization scheduler is connected to the emotion-action mapping module and is used to distribute the action control parameter sequence to multiple digital clones based on the timestamp alignment mechanism, and dynamically allocate the scheduling priority of each instance. The multi-path rendering engine is connected to the synchronization scheduler and is used to drive each digital clone in parallel to complete the synchronous rendering output of expressions and actions in the same frame.

8. A scheduling system for simultaneous driving of multiple digital clones on the same screen according to claim 7, characterized in that, The face capture module includes a lightweight face mesh inference chip and a Kalman filter processor. The lightweight face mesh inference chip is used to extract facial key points from the driving source in real time. The Kalman filter processor is used to smooth the inter-frame trajectory of the facial key points.

9. A scheduling system for simultaneous driving of multiple digital clones on the same screen according to claim 7, characterized in that, The synchronization scheduler includes a frame buffer queue and a priority arbiter. The frame buffer queue maintains an independent timestamped parameter buffer for each digital clone. The priority arbiter dynamically adjusts the GPU time slice allocation ratio of each digital clone based on the display area ratio and interaction frequency.

10. A scheduling system for simultaneous driving of multiple digital clones on the same screen according to claim 7, characterized in that, The multi-path rendering engine adopts a layered rendering architecture with a shared skeletal animation base layer and an independent differentiated expression layer. The skeletal animation base layer is used to share the same set of skeletal skinning calculation results for all digital clones, and broadcasts them to each digital clone after calculation. The differentiated expression layer is used to independently calculate the blended shape weight driven by the AU intensity coefficient of each digital clone.

Images