Virtual human driving method and device based on video frame extraction and frame supplementation, equipment and medium

Abstract

The invention discloses a virtual human driving method and device based on video frame extraction and frame supplementation, equipment and a medium, and relates to the technical field of artificial intelligence. Video frame extraction is performed on a virtual human action video to obtain each frame of picture, action and expression recognition and confidence labeling are performed on the frame of picture, and a video frame library is generated based on the labeled frame of picture; determining the state of a to-be-driven virtual human to generate a virtual human driving command; selecting a target key frame from the video frame library, performing trajectory fitting on the target key frame to obtain a motion curve, and if the variance between the motion curve and the target key frame is smaller than a preset threshold, generating and optimizing a frame compensation picture; and a driving video is generated based on the state of the to-be-driven virtual human, the target key frame and the optimized complementary frame picture, and the to-be-driven virtual human is driven, so that the problems of action advanced voice or voice lagging action and conflict between virtual human expressions and voice intentions are solved, the driving flexibility and controllability of the virtual human are improved, and the scene adaptability and long-time-sequence consistency are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a virtual human driving method, apparatus, device, and medium based on video frame extraction and interpolation. Background Technology

[0002] Currently, methods for driving audio-visual synchronization in 2D (Two-dimensional) virtual humans include motion capture and audio binding-based technologies, video template matching-based technologies, and end-to-end generative audio-visual synchronization technologies. For motion capture and audio binding-based technologies, audio-visual alignment accuracy depends on hardware, easily leading to situations where actions precede speech or speech lags behind actions. Furthermore, these technologies require specialized equipment and fixed environments, making them unsuitable for mobile devices or lightweight scenarios, resulting in poor scene adaptability. For video template matching-based technologies, reliance on preset templates limits flexibility, and obvious stitching artifacts can lead to conflicts between virtual human expressions and speech intentions. End-to-end generative audio-visual synchronization technologies suffer from poor long-term consistency, low controllability, and difficulty in manually intervening in generated motion details (such as gesture amplitude and head rotation angle). They also have complex structures, containing numerous parameters and multi-layered network structures, requiring significant computational resources during training and inference, resulting in high hardware investment and energy consumption costs.

[0003] As can be seen from the above, how to solve the problems of actions leading speech or speech lagging behind actions, and the conflict between virtual human expressions and speech intentions, improve the flexibility and controllability of virtual human driving, and enhance scene adaptability and long-term temporal consistency are problems that need to be solved in this field. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a virtual human driving method, apparatus, device, and medium based on video frame extraction and interpolation, which can solve the problems of actions leading speech or speech lagging behind actions, and conflicts between virtual human expressions and speech intentions, thereby improving the flexibility and controllability of virtual human driving, and enhancing scene adaptability and long-term temporal consistency. The specific solution is as follows:

[0005] In a first aspect, this application discloses a virtual human driving method based on video frame extraction and interpolation, including:

[0006] The video frame is extracted from the preset virtual human action video to obtain each frame. The frame is then subjected to action and expression recognition and confidence labeling to obtain the labeled frame. A video frame library is generated based on the labeled frame.

[0007] Determine the state of the virtual human to be driven, and generate driving commands for the virtual human based on the state; the state includes a listening state and an expression state.

[0008] Based on the virtual human driving command, a target key frame is selected from the video frame library. The target key frame is fitted with a trajectory to obtain a motion curve. It is determined whether the variance between the motion curve and the target key frame is less than a preset threshold. If the variance between the motion curve and the target key frame is less than the preset threshold, a supplementary frame for video transition is generated. The supplementary frame is optimized based on the target key frame to obtain the optimized supplementary frame.

[0009] Based on the state of the virtual human to be driven, the target keyframe, and the optimized interpolated frame, a corresponding driving video is generated, and the driving video is used to drive the virtual human to be driven.

[0010] Optionally, the step of extracting frames from a preset virtual human action video to obtain individual frames, and performing action and expression recognition and confidence labeling on the frames to obtain labeled frames includes:

[0011] According to a preset fixed frequency, the preset virtual human motion video is subjected to frame extraction to obtain each frame.

[0012] The frame image is used to perform action and expression recognition using a deep learning network architecture library. The recognized people, actions and expressions are then assigned coordinate values ​​and confidence scores to obtain the labeled frame image.

[0013] Optionally, determining the state of the virtual human to be driven and generating virtual human driving commands based on the state includes:

[0014] Utilizing a multimodal large model to identify the state of the virtual human to be driven;

[0015] If the virtual human to be driven is in a listening state, then the first virtual human driving command is generated, including the action, expression, listening time, and confidence level.

[0016] If the virtual human to be driven is in an expressive state, then a second virtual human driving command is generated, including the action, expression, start time, transition time, end time, and confidence level.

[0017] Optionally, the step of selecting target keyframes from the video frame library based on the virtual human driving command and performing trajectory fitting on the target keyframes includes:

[0018] The virtual human driving command is input into the preset frame sequence model so that the preset frame sequence model selects the target key frame from the video frame library based on the confidence level in the virtual human driving command.

[0019] The target keyframes are fitted with a trajectory using a cubic spline curve interpolation algorithm.

[0020] Optionally, the virtual human driving method based on video frame extraction and interpolation further includes:

[0021] Data training packages are generated based on virtual human-driven commands, target keyframes, and optimized interpolated frames.

[0022] The preset frame sequence model is trained using the data training package.

[0023] Optionally, if the variance between the motion curve and the target keyframe is less than a preset threshold, then a supplementary frame for video transition is generated, and the supplementary frame is optimized based on the target keyframe, including:

[0024] If the variance between the motion curve and the target keyframe is less than a preset threshold, then keypoint interpolation and a generative adversarial network model are used to generate interpolated frames for video transitions.

[0025] A hash algorithm is used to evaluate the similarity between the target keyframe and the supplementary frame to obtain the similarity score.

[0026] Determine whether the similarity is greater than a preset similarity threshold;

[0027] If the similarity is not greater than a preset similarity threshold, the supplementary frame is optimized based on the target keyframe.

[0028] Optionally, if the variance between the motion curve and the target keyframe is less than a preset threshold, then a supplementary frame for video transition is generated, and the supplementary frame is optimized based on the target keyframe, including:

[0029] If the variance between the motion curve and the target keyframe is less than a preset threshold, then keypoint interpolation and a generative adversarial network model are used to generate interpolated frames for video transitions.

[0030] A hash algorithm is used to evaluate the similarity between the target keyframe and the supplementary frame to obtain the similarity score.

[0031] Determine whether the similarity is greater than a preset similarity threshold;

[0032] If the similarity is not greater than a preset similarity threshold, the supplementary frame is optimized based on the target keyframe.

[0033] Secondly, this application discloses a virtual human driving device based on video frame extraction and interpolation, comprising:

[0034] The video frame extraction and recognition annotation module is used to extract frames from a preset virtual human action video to obtain each frame, perform action and expression recognition and confidence annotation on the frame to obtain the annotated frame, and generate a video frame library based on the annotated frame.

[0035] A driver command generation module is used to determine the state of the virtual human to be driven and generate driver commands for the virtual human based on the state; the state includes a listening state and an expression state.

[0036] The trajectory fitting and frame interpolation optimization module is used to select target keyframes from the video frame library based on the virtual human driving command, perform trajectory fitting on the target keyframes to obtain motion curves, determine whether the variance between the motion curve and the target keyframes is less than a preset threshold, and if the variance between the motion curve and the target keyframes is less than the preset threshold, generate frame interpolation frames for video transition, and optimize the frame interpolation frames based on the target keyframes to obtain the optimized frame interpolation frames.

[0037] The virtual human driving module is used to generate a corresponding driving video based on the state of the virtual human to be driven, the target key frame, and the optimized interpolated frame, and to drive the virtual human to be driven using the driving video.

[0038] Thirdly, this application discloses an electronic device, including:

[0039] Memory, used to store computer programs;

[0040] A processor is used to execute the computer program to implement the aforementioned virtual human driving method based on video frame extraction and interpolation.

[0041] Fourthly, this application discloses a computer storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed virtual human driving method based on video frame extraction and interpolation.

[0042] As can be seen, this application provides a virtual human driving method based on video frame extraction and interpolation, including: extracting frames from a preset virtual human action video to obtain individual frames; performing action and expression recognition and confidence labeling on the frames to obtain labeled frames; generating a video frame library based on the labeled frames; determining the state of the virtual human to be driven; generating a virtual human driving command based on the state; the state includes a listening state and an expression state; selecting a target keyframe from the video frame library based on the virtual human driving command; performing trajectory fitting on the target keyframe to obtain a motion curve; determining whether the variance between the motion curve and the target keyframe is less than a preset threshold; if the variance between the motion curve and the target keyframe is less than the preset threshold, generating interpolated frames for video transition; optimizing the interpolated frames based on the target keyframe to obtain optimized interpolated frames; generating a corresponding driving video based on the state of the virtual human to be driven, the target keyframe, and the optimized interpolated frames; and using the driving video to drive the virtual human to be driven. This application performs frame extraction on virtual human motion videos to obtain individual frames, enabling precise extraction of motion details. It then performs motion and facial expression recognition and confidence labeling on each frame to improve accuracy, generating a video frame library. This determines the virtual human's state and generates virtual human driving commands. Based on these commands, target keyframes are selected from the video frame library to improve the correlation between language and action, addressing the issue of poor synchronization between virtual human action and speech. Trajectory fitting is performed on the target keyframes to obtain motion curves. If the variance between the motion curve and the target keyframe is less than a preset threshold, supplementary frames for video transitions are generated. This allows for continuous optimization of the continuity and rationality of virtual human actions, avoiding flickering and achieving motion continuity. Based on the state of the virtual human to be driven, the target keyframes, and the optimized supplementary frames, corresponding driving videos are generated without requiring real-time model generation, reducing the computational cost of virtual human actions. This also resolves the conflict between virtual human facial expressions and speech intent, improving the flexibility and controllability of virtual human driving, enhancing scene adaptability, and long-term temporal consistency. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0044] Figure 1 This is a flowchart of a virtual human driving method based on video frame extraction and interpolation disclosed in this application;

[0045] Figure 2This is a flowchart illustrating a specific process for generating a video frame library as disclosed in this application;

[0046] Figure 3 This is an output diagram of a deep learning network architecture disclosed in this application;

[0047] Figure 4 This application discloses a specific flowchart for generating virtual human-driven commands that include keyframes for listening and speaking.

[0048] Figure 5 This is a result diagram of one type of Llama4 output disclosed in this application;

[0049] Figure 6 This is a flowchart of a method for generating a frame sequence as disclosed in this application;

[0050] Figure 7 This is a diagram illustrating the real-time training process of a pre-defined frame sequence model disclosed in this application.

[0051] Figure 8 This application discloses a video frame extraction flowchart;

[0052] Figure 9 This application discloses a flowchart of action and expression recognition and confidence labeling.

[0053] Figure 10 This application discloses a dialogue analysis diagram;

[0054] Figure 11 This is a diagram illustrating frame matching and target keyframe selection as disclosed in this application;

[0055] Figure 12 This application discloses a flowchart for generating interpolated frames.

[0056] Figure 13 This application discloses a real-time frame-laying flowchart for the listening phase;

[0057] Figure 14 This is a schematic diagram of a virtual human driving device based on video frame extraction and interpolation disclosed in this application;

[0058] Figure 15 This application provides a structural diagram of an electronic device. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] Currently, 2D virtual human audio-visual synchronization driving methods include motion capture and audio binding-based driving technology, video template matching-based audio-visual synchronization technology, and end-to-end generative audio-visual synchronization driving technology. For motion capture and audio binding-based driving technology, audio-visual alignment accuracy depends on hardware, easily leading to situations where actions precede speech or speech lags behind actions. Furthermore, it requires specialized equipment and a fixed environment, making it unsuitable for mobile devices or lightweight scenarios, resulting in poor scenario adaptability. For video template matching-based audio-visual synchronization technology, it relies on preset templates, limiting flexibility and exhibiting obvious splicing artifacts, leading to conflicts between virtual human facial expressions and speech intentions. For end-to-end generative audio-visual synchronization driving technology, it suffers from poor long-term temporal consistency, low controllability, and difficulty in manually intervening in generated motion details (such as gesture amplitude and head rotation angle). Moreover, its complex structure, containing numerous parameters and multi-layered network structures, requires significant computational resources during training and inference, resulting in high hardware investment and energy consumption costs. As can be seen from the above, how to solve the problems of actions leading speech or speech lagging behind actions, and the conflict between virtual human expressions and speech intentions, improve the flexibility and controllability of virtual human driving, and enhance scene adaptability and long-term temporal consistency are problems that need to be solved in this field.

[0061] See Figure 1 As shown, this embodiment of the invention discloses a virtual human driving method based on video frame extraction and interpolation, which specifically includes:

[0062] Step S11: Extract frames from the preset virtual human action video to obtain each frame. Perform action and expression recognition and confidence labeling on the frame to obtain the labeled frame. Generate a video frame library based on the labeled frame.

[0063] In this embodiment, video frames are extracted from a preset virtual human action video at a preset fixed frequency to obtain each frame; the frame frames are then used to perform action and expression recognition using a deep learning network architecture library, and the recognized characters, actions, and expressions are assigned coordinate values ​​and confidence labels to obtain the labeled frame frames; a video frame library is then generated based on the labeled frame frames.

[0064] In this embodiment, the specific process for generating the video frame library is as follows: Figure 2As shown, the frame video library can be used to generate virtual humans and also for model training, improving the accuracy of HRNet (High Resolution Network, a high-resolution deep learning network architecture) in annotation. First, the pre-prepared virtual human motion video is decomposed into several frames at a fixed frame rate, providing basic units for subsequent motion analysis, annotation, and matching. For example, a 10-second, MP4 (MPEG-4 Part 14) video at 24 FPS (Frames Per Second) can be extracted into 240 frames using OpenCV (Open Source Computer Vision Library). Frame extraction discretizes continuous motion into recognizable static frames, facilitating accurate extraction of motion details. The more frames, the less cost is required for subsequent model generation of interpolation frames. Therefore, the virtual human motion video must ensure the diversity and duration of motion.

[0065] This application utilizes a deep learning network architecture to perform action and expression recognition on frames obtained through image extraction, labeling action types and confidence levels. Specifically, using the HRNet library, keypoint coordinates are assigned to the figures within the frames (the number of keypoints is adjusted according to the accuracy requirements of the virtual human driver), digitizing the actions and expressions in the frames. The output of the deep learning network architecture is as follows: Figure 3 As shown, it includes:

[0066] (1) XY coordinates and confidence level of key points;

[0067] (2) Actions and confidence levels;

[0068] (3) Facial expression and confidence level.

[0069] Data with a confidence level less than 0.5 is not used. Actions, expressions, and confidence levels can be labeled multiple times so that a single frame can be used for multiple action matching. For example: (Action: 0.93 waving, 0.78 raising hand, 0.32 saying goodbye) (Emotional state: 0.91 happy, 0.36 joyful, 0.18 affirmative). Labeling frames and building a video frame library provides a data foundation for subsequent matching of speech and coherent actions.

[0070] Step S12: Determine the state of the virtual human to be driven, and generate virtual human driving commands based on the state; the state includes listening state and expression state.

[0071] In this embodiment, a multimodal large model is used to identify the state of the virtual human to be driven. If the state of the virtual human to be driven is listening, a first virtual human driving command including action, expression, listening time, and confidence level is generated. If the state of the virtual human to be driven is expressing, a second virtual human driving command including action, expression, start time, transition time, end time, and confidence level is generated.

[0072] The specific process for generating virtual human driving commands, including keyframes for listening and expression, in this application is as follows: Figure 4 As shown, a multimodal large-scale dialogue model for virtual humans is defined. This model includes prompts from Llama4 (Large Language Model Meta AI 4), specifying that the large-scale model outputs virtual human-driven commands, including actions and facial expressions, based on the current listening or speaking dialogue state.

[0073] (1) If the virtual human to be driven is in a listening state, the first virtual human driving command is generated every second, including the action, expression, listening time, and confidence level.

[0074] (2) If the virtual human to be driven is in an expressive state, then the time point plus the action and expression data is given according to the speech duration, and the start, transition and end time points of the action are given (i.e., the second virtual human driving command including action, expression, start time, transition time, end time and confidence level).

[0075] For example, the Prompt is as follows:

[0076] core:

[0077] You are a virtual avatar, and you need to define your (the virtual avatar's) actions and emotions. That is, the actions and emotions a virtual avatar should exhibit in conversational scenarios, needing to project an image of enthusiasm, cheerfulness, and a friendly, approachable demeanor. You have two states: listening and speaking.

[0078] Role:

[0079] The virtual account opening guide for financial institutions is primarily responsible for assisting users in completing the entire account opening process (including information filling, document uploading, risk warnings, etc.), and providing real-time answers to various questions encountered by users during the account opening process. It serves as a dedicated service assistant for users during the account opening process.

[0080] character:

[0081] Enthusiastic and proactive, they will pay attention to the user's operation progress and potential needs; cheerful and approachable, they have a relaxed and friendly aura that can quickly close the distance with users; gentle and patient when providing service, they will not get impatient when faced with repeated inquiries or operation errors; approachable and down-to-earth, they communicate without any sense of distance, and are as reliable as a familiar friend.

[0082] Speech rate:

[0083] Maintain a moderate to slow pace overall, approximately 120-140 words per minute. This pace conveys enthusiasm without overwhelming the user, aligning with a gentle and approachable personality. When users articulate their needs or questions, the pace naturally slows to around 110 words per minute, accompanied by pauses to convey attentive listening. When proactively introducing operational steps or promotional activities, the pace can be slightly increased (not exceeding 150 words per minute) to demonstrate enthusiasm through a slightly faster rhythm, while always maintaining clarity and avoiding ambiguity.

[0084] Output language style tendencies:

[0085] Use conversational language and avoid stiff technical jargon. If jargon must be used, immediately provide a clear explanation, such as, "The 'KYC (Know Your Customer) information' you need to fill in is basically the basic information that proves your identity." Use friendly, polite language and a consultative tone, such as, "Thank you for your help," "Is this the correct way to do it?" "Would you like me to demonstrate again?" Proactively offer support, such as, "If you get stuck at any step during the account opening process, just let me know," or "This step might be a little tricky, let me help you." When a user makes a mistake, reassure them before resolving the issue, such as, "It's okay, everyone makes mistakes sometimes, let's try again," to reduce the user's anxiety.

[0086] Action and facial expression tendencies:

[0087] Listening posture: Tilt your head slightly forward about 10 degrees, look directly at the user (or at the user's perspective as simulated on the screen), maintain a natural smile, and nod gently every 3-5 seconds, accompanied by responses such as "Hmm" or "I understand" to make the user feel valued; when the user mentions key information (such as "I don't know how to fill this in"), furrow your eyebrows slightly for 0.5 seconds and then relax them to convey the signal "I understand your confusion".

[0088] Expression: Maintain a toothy smile throughout (corners of the mouth raised about 15°), raise both hands naturally while speaking, and make slight auxiliary gestures (such as raising the fingers together when pointing to the screen operation area); when introducing beneficial information (such as "new user gift pack for successful account opening"), blink lightly and clap your hands slightly (palms lightly touch twice) to enhance enthusiasm; when guiding complex steps, turn your body slightly to simulate the posture of "looking at the screen with the user", with gentle and non-exaggerated movements to avoid distracting the user.

[0089] Output definition:

[0090] Listening state: Output the virtual human's listening actions and expressions per second (accurate to 0.04s, matching 24FPS). The actions need to refer to the action confidence of the frame images in the video frame library (e.g., "0.93 waving"), and the expressions need to refer to the emotion confidence (e.g., "0.91 happy"). The output format includes the core actions (e.g., "head slightly tilted forward") and the corresponding expressions (e.g., "focused") within that second.

[0091] Note: Actions should be fluid and natural, avoiding excessive movements or overly complex expressions, such as performing the same action and expression consecutively within one second. Generally, when listening, people will only change their action every 2-3 seconds.

[0092] Expression status: When the user pauses for ≥1.5 seconds, it is determined that the speech has ended, and a virtual human response text (i.e., response_content) is generated. The estimated speech duration is calculated based on the text length at 2-3 words / second.

[0093] Based on the semantics and context of the response content, output an action and expression timeline: each action must include a start time (accurate to 0.001s), a transition time (the dynamic change stage from keyframe to transition frame, accurate to 0.001s), and an end time (accurate to 0.001s). The action must correspond to the keyframe type in the frame set (such as "raise hand start frame", "raise hand transition frame", "raise hand end frame").

[0094] Facial expressions need to be labeled simultaneously, including the duration ([start seconds, end seconds], accurate to 0.001s) and the emotion confidence of the reference frame set (e.g., "0.73 mild").

[0095] For example, given a user's speech lasting 5 seconds and their response lasting 3 seconds, Llama4 would produce the following result: Figure 5 As shown.

[0096] The final output of the virtual human driving commands in JSON (JavaScript Object Notation, Lightweight Data Interchange) format should include:

[0097] listening_phase: An array containing the actions and expressions displayed per second (0.04s interval) during listening;

[0098] response_content: String, the text of the virtual human's response;

[0099] response_phase: An array containing the name, start time, transition time, end time, and corresponding facial expression and duration of each action during the expression (which must match the frame annotation logic of the frame sequence model to facilitate subsequent keyframe selection and transition frame completion).

[0100] Step S13: Select a target keyframe from the video frame library based on the virtual human driving command, perform trajectory fitting on the target keyframe to obtain a motion curve, determine whether the variance between the motion curve and the target keyframe is less than a preset threshold, if the variance between the motion curve and the target keyframe is less than the preset threshold, generate a supplementary frame for video transition, optimize the supplementary frame based on the target keyframe to obtain the optimized supplementary frame.

[0101] In this embodiment, a virtual human driving command is input into a preset frame sequence model, so that the preset frame sequence model selects a target keyframe from a video frame library based on the confidence level in the virtual human driving command. A cubic spline curve interpolation algorithm is used to fit the trajectory of the target keyframe to obtain a motion curve. It is then determined whether the variance between the motion curve and the target keyframe is less than a preset threshold. If the variance is less than the preset threshold, keypoint interpolation and a generative adversarial network model are used to generate interpolated frames for video transition. A hash algorithm is used to evaluate the similarity between the target keyframe and the interpolated frames to obtain a similarity score. It is then determined whether the similarity score is greater than a preset similarity threshold. If the similarity score is not greater than the preset similarity threshold, the interpolated frames are optimized based on the target keyframe.

[0102] In this embodiment, a frame sequence is generated based on the time frame of actions and expressions, and interpolated frames are created to improve the video frame library. The process of generating the frame sequence is as follows: Figure 6 As shown.

[0103] First, target keyframes are selected. The virtual human driving commands output by Llama4 are imported as context into a preset frame sequence model. The preset frame sequence model searches for the most relevant frame in the frame set and selects it as the target keyframe. Relevance is determined by a combination of semantic relevance and confidence level of the action and expression; for example, greeting is matched with raising a hand. Frames are selected based on the highest confidence values ​​for the action and expression. Accuracy is ensured through manual labeling, and the target keyframe is inserted at the keyframe time point.

[0104] For example, regarding the voice message "Hello! It's nice to meet you, I'm your account opening assistant, Xiaohua," as specified above, frames corresponding to / close to the "raising hand" action are inserted at 0.50s, 1.00s, and 1.50s, respectively, representing the start, transition, and end of the action. Then, frames corresponding to / close to the "retracting the hand to the abdomen" action are inserted at 2.00s, 2.50s, and 3.00s, respectively. This establishes a preliminary association between the action and the voice.

[0105] Then, a frame sequence is generated, using the keypoint coordinates of the selected target keyframe, with the preceding and following frames serving as the start and end points of the coordinates. Considering human motion patterns, such as joint rotation angle limitations, and to avoid physiologically inappropriate postures like "backward elbow bending," a cubic spline curve (Catmull-Rom interpolation algorithm) is used to fit the keypoint trajectory, generating a more natural motion curve. Similar numerical keyframes are matched in the video frame library, and the initial frame sequence is arranged according to "minimum difference (using a hash algorithm to evaluate the similarity between two frames) + motion pattern."

[0106] Then, interpolated frames for video transition are generated. It is determined whether the variance between the motion curve and the target keyframe is less than a preset threshold. If it is less than the preset threshold, it indicates that the gap between intermediate values ​​is too large. At this time, keypoint interpolation and GAN (Generative Adversarial Network) image generation techniques are used to generate interpolated frames for intermediate frame transition.

[0107] Principle: Train StyleGAN to generate human images with a consistent virtual human style.

[0108] Intermediate frames are generated by interpolating in the latent space of a GAN using HRNet keypoints as conditions (initial and final keypoint values). Pivotal Tuning can be used to fine-tune the frames with a small number of samples.

[0109] The similarity between the target keyframe and the interpolated frame is evaluated using a hash algorithm, and the optical flow consistency between adjacent frames is calculated to avoid flickering. For areas with poor interpolation results, keypoints or optical flow parameters can be manually adjusted. Virtual human movements are generated multiple times, and the coherence and rationality of the virtual human movements are continuously optimized based on the above steps until an ideal level is achieved in the virtual human application scenario.

[0110] In addition, the real-time training process of the preset frame sequence model includes: generating a data training package based on virtual human driving commands, target keyframes, and optimized interpolated frames; and using the data training package to train the preset frame sequence model.

[0111] In this embodiment, the real-time training process of the preset frame sequence model is as follows: Figure 7 As shown, the virtual human driving commands generated by the Llama4 model and the matching target keyframes are combined to form a data training package for model training. This ensures that the frame sequence model can produce appropriate frame sequences based on the content generated by the Llama4 model, and that the frame sequences conform to the rationality and continuity of human actions. The frame sequence model is then used in production to replace complex workflows, thereby improving system performance.

[0112] Furthermore, this application allows for multiple verifications after generating coherent movements to complete the video frame library. Based on the frame sequences produced by the trained frame sequence model, all matching virtual human movement frames are arranged and rendered, and the animation frames of the virtual human's movements are loaded in real time to achieve a coherent, unlimited-duration sequence of movement frames. Ultimately, the virtual human's movement is driven, much like temporary train tracks: the train moves while the tracks are laid out frame by frame.

[0113] Step S14: Generate a corresponding driving video based on the state of the virtual human to be driven, the target keyframe, and the optimized interpolated frame, and use the driving video to drive the virtual human to be driven.

[0114] In this embodiment, if the virtual human to be driven is in a listening state, the target keyframe and the optimized interpolated frame are stitched together to generate a first driving video; if the virtual human to be driven is in an expressing state, the frame vector values ​​of the target keyframe and the optimized interpolated frame are calculated using a motion vector algorithm, and a target frame is selected from the target keyframe and the optimized interpolated frame based on the frame vector values, and a second driving video is generated using the target frame.

[0115] In this embodiment, if the virtual human to be driven is in a listening state, it does not need to be driven by lip movements; if the virtual human to be driven is in a speaking state, it needs to be driven by the virtual human.

[0116] Taking the virtual account opening project as an example, the application process of this invention is as follows:

[0117] 1. Pre-capture virtual human motion videos, such as "introducing the product" or "confirming the operation." After frame extraction, thousands of frames are obtained. The video frame extraction process is as follows: Figure 8 As shown;

[0118] 2. Use the HRNet library to perform action and facial expression recognition and confidence labeling on the frames; the process for action and facial expression recognition and confidence labeling is as follows: Figure 9 As shown;

[0119] 3. The virtual human listens or responds based on the dialogue. For example, when the virtual human needs to output the voice "Hello, I am the account opening assistant Xiaohua," semantic analysis is used to match virtual human driving commands containing the actions of "greeting" and the emotion of "happy." The virtual human driving commands include a subset of frames; dialogue analysis is as follows... Figure 10 As shown;

[0120] 4. Select "one hand raised" (starting frame), "both hands swinging" (turning frame), and "hands behind back" (ending frame) as target keyframes; frame matching and target keyframe selection are as follows: Figure 11 As shown;

[0121] 5. The model generates interpolated frames based on target keyframes such as "hand raising process" and "arm movement." The frame rate is set to 10 frames / second. Due to the simplicity of the action, the frequency is reduced to improve efficiency. The process of generating interpolated frames is as follows: Figure 12 As shown;

[0122] 6. In the expression stage, the target keyframes and supplementary frames are stitched together at a frame rate of 24 frames per second to form a coherent "raise hand + explanation" action, generating the corresponding driving video to achieve synchronous audio and video driving during the virtual human account opening process.

[0123] During the listening phase, based on the target keyframes generated by the model per second, frames are selected and laid out in real time according to the motion vector algorithm to form a coherent motion; the real-time frame laying process during the listening phase is as follows: Figure 13 As shown.

[0124] The core principle of this application is: by extracting frames, existing virtual human motion videos are converted into frame images, key frames are selected by combining the semantics and scene of the speech to be output, and then transitional interpolation frames are selected / generated by the model, and finally spliced ​​into a coherent action, so as to realize the synchronous driving of virtual human and speech.

[0125] The innovation of this application lies in: breaking down virtual human motion videos into frames and labeling them with scenes, actions, and expressions; using a large model to analyze and output the actions and expressions the virtual human should exhibit in a dialogue scene; training the large model to interpolate key motion points; real-time frame-by-frame rendering of the virtual human image during listening; and driving the virtual human to engage in free dialogue with the user based on a multimodal model and a lip-syncing model. This improves the correlation between virtual human actions and speech, enhances the coherence and rationality of virtual human actions, solves the problem of asynchronous virtual human actions and speech, and reduces the computational cost of virtual humans.

[0126] The entire process requires the collaboration of three major models: Llama4, the underlying multimodal chat model for the virtual human, which receives text, voice, and video input from the user; then analyzes whether the virtual human is in a listening or speaking state, and produces the response content, timing, and facial expressions; the frame sequence model, which annotates frames and produces frame sequences based on the content returned by the multimodal model; and the audio lip-sync model, which merges video and audio, lip-syncs the audio, and returns the results to the front end.

[0127] Furthermore, this application also has certain business value, enabling users to engage in real-time dialogues with virtual humans based on a large dialogue model. The virtual humans can synchronize their voice and actions and possess a certain degree of plausibility.

[0128] 1. In the account opening scenario, virtual human dialogue can be used to guide users to open an account, thereby improving the user experience and service experience, and thus improving account opening data;

[0129] 2. Live streaming platforms can use virtual avatars for real-time broadcasts, answering user questions and even interacting with users via voice chat. This reduces the cost of live streaming with real people while maintaining a certain level of authenticity.

[0130] 3. On the stock platform, provide a personal virtual assistant to help users analyze stock transactions and improve their stock trading experience.

[0131] In this embodiment, a preset virtual human action video is subjected to frame extraction to obtain individual frames. Action and expression recognition and confidence labeling are performed on these frames to obtain labeled frames. A video frame library is generated based on these labeled frames. The state of the virtual human to be driven is determined, and a virtual human driving command is generated based on the state. The state includes a listening state and an expression state. Target keyframes are selected from the video frame library based on the virtual human driving command. Trajectory fitting is performed on the target keyframes to obtain motion curves. It is determined whether the variance between the motion curve and the target keyframe is less than a preset threshold. If the variance is less than the preset threshold, a supplementary frame for video transition is generated. The supplementary frame is optimized based on the target keyframe to obtain optimized supplementary frames. A corresponding driving video is generated based on the state of the virtual human to be driven, the target keyframe, and the optimized supplementary frames. The driving video is then used to drive the virtual human to be driven. This application performs frame extraction on virtual human motion videos to obtain individual frames, enabling precise extraction of motion details. It then performs motion and facial expression recognition and confidence labeling on each frame to improve accuracy, generating a video frame library. This determines the virtual human's state and generates virtual human driving commands. Based on these commands, target keyframes are selected from the video frame library to improve the correlation between language and action, addressing the issue of poor synchronization between virtual human action and speech. Trajectory fitting is performed on the target keyframes to obtain motion curves. If the variance between the motion curve and the target keyframe is less than a preset threshold, supplementary frames for video transitions are generated. This allows for continuous optimization of the continuity and rationality of virtual human actions, avoiding flickering and achieving motion continuity. Based on the state of the virtual human to be driven, the target keyframes, and the optimized supplementary frames, corresponding driving videos are generated without requiring real-time model generation, reducing the computational cost of virtual human actions. This also resolves the conflict between virtual human facial expressions and speech intent, improving the flexibility and controllability of virtual human driving, enhancing scene adaptability, and long-term temporal consistency.

[0132] See Figure 14 As shown, this embodiment of the invention discloses a virtual human driving device based on video frame extraction and interpolation, which may specifically include:

[0133] The video frame extraction and recognition annotation module 11 is used to extract video frames from a preset virtual human action video to obtain each frame, perform action and expression recognition and confidence annotation on the frame to obtain the annotated frame, and generate a video frame library based on the annotated frame.

[0134] The driving command generation module 12 is used to determine the state of the virtual human to be driven and generate driving commands for the virtual human based on the state; the state includes a listening state and an expression state.

[0135] The trajectory fitting and frame interpolation optimization module 13 is used to select a target key frame from the video frame library based on the virtual human driving command, perform trajectory fitting on the target key frame to obtain a motion curve, determine whether the variance between the motion curve and the target key frame is less than a preset threshold, and if the variance between the motion curve and the target key frame is less than the preset threshold, generate a frame interpolation image for video transition, and optimize the frame interpolation image based on the target key frame to obtain the optimized frame interpolation image.

[0136] The virtual human driving module 14 is used to generate a corresponding driving video based on the state of the virtual human to be driven, the target key frame, and the optimized interpolated frame, and to drive the virtual human to be driven using the driving video.

[0137] In this embodiment, a preset virtual human action video is subjected to frame extraction to obtain individual frames. Action and expression recognition and confidence labeling are performed on these frames to obtain labeled frames. A video frame library is generated based on these labeled frames. The state of the virtual human to be driven is determined, and a virtual human driving command is generated based on the state. The state includes a listening state and an expression state. Target keyframes are selected from the video frame library based on the virtual human driving command. Trajectory fitting is performed on the target keyframes to obtain motion curves. It is determined whether the variance between the motion curve and the target keyframe is less than a preset threshold. If the variance is less than the preset threshold, a supplementary frame for video transition is generated. The supplementary frame is optimized based on the target keyframe to obtain optimized supplementary frames. A corresponding driving video is generated based on the state of the virtual human to be driven, the target keyframe, and the optimized supplementary frames. The driving video is then used to drive the virtual human to be driven. This application performs frame extraction on virtual human motion videos to obtain individual frames, enabling precise extraction of motion details. It then performs motion and facial expression recognition and confidence labeling on each frame to improve accuracy, generating a video frame library. This determines the virtual human's state and generates virtual human driving commands. Based on these commands, target keyframes are selected from the video frame library to improve the correlation between language and action, addressing the issue of poor synchronization between virtual human action and speech. Trajectory fitting is performed on the target keyframes to obtain motion curves. If the variance between the motion curve and the target keyframe is less than a preset threshold, supplementary frames for video transitions are generated. This allows for continuous optimization of the continuity and rationality of virtual human actions, avoiding flickering and achieving motion continuity. Based on the state of the virtual human to be driven, the target keyframes, and the optimized supplementary frames, corresponding driving videos are generated without requiring real-time model generation, reducing the computational cost of virtual human actions. This also resolves the conflict between virtual human facial expressions and speech intent, improving the flexibility and controllability of virtual human driving, enhancing scene adaptability, and long-term temporal consistency.

[0138] In some specific embodiments, the video frame extraction and identification annotation module 11 may specifically include:

[0139] The video frame extraction module is used to extract frames from a preset virtual human action video at a preset fixed frequency to obtain each frame.

[0140] The identification and annotation module is used to perform action and expression recognition on the frame using a deep learning network architecture library, and to assign coordinate values ​​and confidence scores to the identified people, actions, and expressions to obtain the annotated frame.

[0141] In some specific embodiments, the driver command generation module 12 may specifically include:

[0142] The state recognition module is used to identify the state of the virtual human to be driven using a multimodal large model;

[0143] The first virtual human driving command generation module is used to generate a first virtual human driving command including actions, expressions, listening time, and confidence level if the state of the virtual human to be driven is listening.

[0144] The second virtual human driving command generation module is used to generate a second virtual human driving command including action, expression, start time, transition time, end time, and confidence level if the state of the virtual human to be driven is the expression state.

[0145] In some specific embodiments, the trajectory fitting and frame interpolation optimization module 13 may specifically include:

[0146] The target keyframe selection module is used to input virtual human driving commands into a preset frame sequence model, so that the preset frame sequence model selects target keyframes from the video frame library based on the confidence level in the virtual human driving commands.

[0147] The trajectory fitting module is used to fit the trajectory of the target keyframe using a cubic spline curve interpolation algorithm.

[0148] In some specific embodiments, the virtual human driving device based on video frame extraction and interpolation may further include:

[0149] The data training package generation module is used to generate data training packages based on virtual human driving commands, target keyframes, and optimized interpolated frames.

[0150] The model training module is used to train the preset frame sequence model using the data training package.

[0151] In some specific embodiments, the trajectory fitting and frame interpolation optimization module 13 may specifically include:

[0152] The frame interpolation generation module is used to generate frame interpolation images for video transitions if the variance between the motion curve and the target keyframe is less than a preset threshold, using keypoint interpolation and a generative adversarial network model.

[0153] The similarity assessment module is used to evaluate the similarity between the target keyframe and the supplementary frame using a hash algorithm to obtain the similarity score.

[0154] The judgment module is used to determine whether the similarity is greater than a preset similarity threshold;

[0155] The frame interpolation optimization module is used to optimize the frame interpolation based on the target keyframe if the similarity is not greater than a preset similarity threshold.

[0156] In some specific embodiments, the virtual human driving module 14 may specifically include:

[0157] The first driving video generation module is used to stitch the target key frame with the optimized interpolated frame if the virtual human to be driven is in a listening state, so as to generate the first driving video.

[0158] The second driving video generation module is used to calculate the frame vector values ​​of the target keyframe and the optimized interpolated frame image using a motion vector algorithm if the state of the virtual human to be driven is the expression state, and to select the target frame image from the target keyframe and the optimized interpolated frame image based on the frame vector values, and to generate the second driving video using the target frame image.

[0159] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the virtual human driving method based on video frame extraction and interpolation disclosed in any of the foregoing embodiments.

[0160] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0161] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.

[0162] The operating system 221 manages and controls the various hardware devices on the electronic device 20 and the computer program 222 to enable the processor 21 to perform operations and processing on the data 223 in the memory 22. The operating system 221 can be Windows, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the virtual human driving method based on video frame extraction and interpolation as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the virtual human driving device based on video frame extraction and interpolation from external devices, and may also include data collected by its own input / output interface 25.

[0163] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0164] Furthermore, embodiments of this application also disclose a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, it implements the virtual human driving method steps based on video frame extraction and interpolation disclosed in any of the foregoing embodiments.

[0165] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0166] The foregoing has provided a detailed description of a virtual human driving method, apparatus, device, and storage medium based on video frame extraction and interpolation provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A virtual human driving method based on video frame extraction and interpolation, characterized in that, include: The video frame is extracted from the preset virtual human action video to obtain each frame. The frame is then subjected to action and expression recognition and confidence labeling to obtain the labeled frame. A video frame library is generated based on the labeled frame. Determine the state of the virtual human to be driven, and generate driving commands for the virtual human based on the state; the state includes a listening state and an expression state. Based on the virtual human driving command, a target key frame is selected from the video frame library. The target key frame is fitted with a trajectory to obtain a motion curve. It is determined whether the variance between the motion curve and the target key frame is less than a preset threshold. If the variance between the motion curve and the target key frame is less than the preset threshold, a supplementary frame for video transition is generated. The supplementary frame is optimized based on the target key frame to obtain the optimized supplementary frame. Based on the state of the virtual human to be driven, the target keyframe, and the optimized interpolated frame, a corresponding driving video is generated, and the driving video is used to drive the virtual human to be driven.

2. The virtual human driving method based on video frame extraction and interpolation according to claim 1, characterized in that, The process of extracting frames from a preset virtual human action video to obtain individual frames, and then performing action and expression recognition and confidence labeling on these frames to obtain labeled frames includes: According to a preset fixed frequency, the preset virtual human motion video is subjected to frame extraction to obtain each frame. The frame image is used to perform action and expression recognition using a deep learning network architecture library. The recognized people, actions and expressions are then assigned coordinate values ​​and confidence scores to obtain the labeled frame image.

3. The virtual human driving method based on video frame extraction and interpolation according to claim 1, characterized in that, The process of determining the state of the virtual human to be driven and generating virtual human driving commands based on the state includes: Utilizing a multimodal large model to identify the state of the virtual human to be driven; If the virtual human to be driven is in a listening state, then the first virtual human driving command is generated, including the action, expression, listening time, and confidence level. If the virtual human to be driven is in an expressive state, then a second virtual human driving command is generated, including the action, expression, start time, transition time, end time, and confidence level.

4. The virtual human driving method based on video frame extraction and interpolation according to claim 3, characterized in that, The step of selecting target keyframes from the video frame library based on the virtual human-driven command and performing trajectory fitting on the target keyframes includes: The virtual human driving command is input into the preset frame sequence model so that the preset frame sequence model selects the target key frame from the video frame library based on the confidence level in the virtual human driving command. The target keyframes are fitted with a trajectory using a cubic spline curve interpolation algorithm.

5. The virtual human driving method based on video frame extraction and interpolation according to claim 4, characterized in that, Also includes: Data training packages are generated based on virtual human-driven commands, target keyframes, and optimized interpolated frames. The preset frame sequence model is trained using the data training package.

6. The virtual human driving method based on video frame extraction and interpolation according to claim 1, characterized in that, If the variance between the motion curve and the target keyframe is less than a preset threshold, then a supplementary frame for video transition is generated, and the supplementary frame is optimized based on the target keyframe, including: If the variance between the motion curve and the target keyframe is less than a preset threshold, then keypoint interpolation and a generative adversarial network model are used to generate interpolated frames for video transitions. A hash algorithm is used to evaluate the similarity between the target keyframe and the supplementary frame to obtain the similarity score. Determine whether the similarity is greater than a preset similarity threshold; If the similarity is not greater than a preset similarity threshold, the supplementary frame is optimized based on the target keyframe.

7. The virtual human driving method based on video frame extraction and interpolation according to any one of claims 1 to 6, characterized in that, The process of generating a corresponding driving video based on the state of the virtual human to be driven, the target keyframe, and the optimized interpolated frame, and using the driving video to drive the virtual human to be driven, includes: If the virtual human to be driven is in a listening state, the target keyframe is stitched together with the optimized interpolated frame to generate the first driving video. If the virtual human to be driven is in the expression state, the motion vector algorithm is used to calculate the frame vector values ​​of the target keyframe and the optimized interpolated frame. Based on the frame vector values, the target frame is selected from the target keyframe and the optimized interpolated frame, and the second driving video is generated using the target frame.

8. A virtual human driving device based on video frame extraction and interpolation, characterized in that, include: The video frame extraction and recognition annotation module is used to extract frames from a preset virtual human action video to obtain each frame, perform action and expression recognition and confidence annotation on the frame to obtain the annotated frame, and generate a video frame library based on the annotated frame. A driver command generation module is used to determine the state of the virtual human to be driven and generate driver commands for the virtual human based on the state; the state includes a listening state and an expression state. The trajectory fitting and frame interpolation optimization module is used to select target keyframes from the video frame library based on the virtual human driving command, perform trajectory fitting on the target keyframes to obtain motion curves, determine whether the variance between the motion curve and the target keyframes is less than a preset threshold, and if the variance between the motion curve and the target keyframes is less than the preset threshold, generate frame interpolation frames for video transition, and optimize the frame interpolation frames based on the target keyframes to obtain the optimized frame interpolation frames. The virtual human driving module is used to generate a corresponding driving video based on the state of the virtual human to be driven, the target key frame, and the optimized interpolated frame, and to drive the virtual human to be driven using the driving video.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the virtual human driving method based on video frame extraction and interpolation as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the virtual human driving method based on video frame extraction and interpolation as described in any one of claims 1 to 7.

Images