Real-time generation method of 2D digital human driven by semantic and matching with bone motion vector

Abstract

The application discloses a kind of 2D digital human real-time generation methods of semantic driving and bone movement vector matching, it is related to the cross technical field of generative artificial intelligence and virtual person, the generation method uses large language model to generate language and non-language behavior description;Behavior description is mapped into bone movement vector in real time by text-action mapping encoder, and cosine similarity retrieval is carried out in motion primitive library, and a kind of semantic corresponds to the zero-delay corresponding of one gesture;Smooth transition frame is generated between two action primitive image sequences using the action splicing of video generation model, forming smooth action video;The lip shape of the smooth action video is synchronized processing.The real-time generation method of the application constructs the joint representation space of language-action, realizes the end-to-end low-delay generation, significantly improves the naturalness and realism of 2D digital human interaction, and can be widely used in virtual customer service, live avatar and metaverse real-time interaction scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of generative artificial intelligence and virtual human cross, in particular to a 2D digital human real-time generation method based on semantic driving and bone motion vector matching. BACKGROUND

[0002] With the prevalence of the metaverse, applications related to virtual avatars and digital humans are becoming more and more widespread. Current digital human technology mainly falls into two technical routes:

[0003] (1) 3D digital human technology

[0004] Although it supports multi-view rendering and complex scene interaction, it is limited by the "uncanny valley" effect of three-dimensional modeling, and it is difficult to achieve a high degree of realism in micro-expression and material details. According to the Meta Human Interaction White Paper data in 2024, the average score of user interaction naturalness for 3D digital humans is only 2.3 out of 5.

[0005] (2) 2D digital human technology

[0006] Although based on video shooting or high-precision image synthesis has advantages in facial expression and texture realism, existing technologies generally use a separate architecture for voice generation modules and pre-recorded video segments, resulting in the following technical defects:

[0007] (a) Insufficient modality coordination

[0008] Existing 2D digital human solutions generally use a separate training architecture for language models and action models, and the correspondence between text semantics and action semantics relies on artificial preset rules, resulting in the same text being bound to the same action sequence in different contexts; real-time speech flow and pre-recorded actions lack a coupling mechanism in the time dimension, requiring dynamic time warping (DTW) to force alignment, and the linear upward trend increases with the length of the speech, and the audience can clearly perceive the "lip-motion" misalignment.

[0009] (b) Action synthesis quality defects

[0010] The average duration of pre-recorded action units is only 2.3 seconds (Samsung AI Research, 2023), and when the speech duration exceeds this length, multiple actions need to be spliced; existing methods use optical flow to generate transition frames, which cannot maintain joint acceleration and angular velocity continuity at the level of kinematics, resulting in a loss of joint motion continuity of up to 37%. The audience can clearly perceive the stiffness of the action.

[0011] (c) Emotion-lip misalignment

[0012] Existing lip-sync technology only replaces the lip area pixel by pixel based on the audio waveform, without recognizing the facial emotional semantics already carried in the pre-recorded video, causing cross-modal conflict between spoken and facial emotions; viewers can intuitively perceive that "the spoken words are one emotion, but the face shows another emotion", significantly reducing the naturalness of the interaction.

[0013] Overall, digital humans lack a match between their verbal content and non-verbal behavior (including facial expressions and gestures), and the non-verbal behavior is too repetitive, which seriously affects the realism of the interaction. Summary of the Invention

[0014] This invention provides a real-time generation method for 2D digital humans based on semantic driving and skeletal motion vector matching, which solves the problem of language and action modality separation in the prior art, leading to lip-semantics-limb mismatch and movement jumps.

[0015] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows:

[0016] A real-time generation method for 2D digital humans based on semantic driving and skeletal motion vector matching is disclosed. The real-time generation method uses a large language model to generate linguistic and non-linguistic behavioral descriptions; a text-action mapping encoder maps the behavioral descriptions to skeletal motion vectors in real time, and performs cosine similarity retrieval in a motion primitive library to achieve zero-latency correspondence between one semantic and one posture; a video generation model is used to stitch actions together to generate smooth transition frames between two primitive segments, forming a smooth motion video; and lip-sync processing is applied to the smooth motion video.

[0017] A further improvement to the above technical solution is as follows:

[0018] Preferably, in the process of generating language and non-language behavior descriptions using a large language model, the user's voice command is first received, the voice signal is converted into a text string, and the large language model parses the command to generate language response text and non-language behavior descriptions that are semantically consistent with the language response text.

[0019] Preferably, the text-motion mapping encoder includes a CLIP text encoder and a mapping network. The CLIP text encoder is used to convert text into a 512-dimensional text feature vector, and the mapping network is used to project the 512-dimensional text feature vector onto a 256-dimensional skeletal motion vector space.

[0020] Preferably, the real-time mapping of behavioral description text to skeletal motion vectors includes the following steps:

[0021] S3-1, Construction and Mapping of 256-Dimensional Skeletal Motion Vector Space: Extracting continuous image frames from motion primitive image sequences; using DwPose technology to extract human skeletal key point sequences from image frames to form skeletal motion vectors Va; multiple skeletal motion vectors Va and their corresponding motion primitive image sequences together constitute a motion primitive library.

[0022] In S3-2, the skeletal motion vector Va is used as the real vector, and the predicted skeletal motion vector Vp is used as the predicted vector. The loss function is L=1-cos(Vp,Va), which is the cosine similarity loss. The average cosine similarity between the predicted vector Vp and the real vector Va is calculated. After training, the MLP_map parameters are fixed and it is deployed as a "mapping network" in the online inference process to realize the real-time output of text mapping to 256-dimensional skeletal motion vectors.

[0023] S3-3, the CLIP text encoder first converts the input text into a 512-dimensional text feature vector; then, the mapping network projects this 512-dimensional text feature vector onto a 256-dimensional skeletal motion vector space.

[0024] Preferably, cosine similarity retrieval is performed within the motion primitive library, including: using the similarity between the 256-dimensional skeletal motion vector Vq and each skeletal motion vector Va in the motion primitive library to retrieve matching motion primitive image sequences; the retrieval algorithm determines the optimal match by calculating the cosine similarity between Vq and all Va.

[0025] Preferably, generating a smooth transition frame between two motion primitive image sequences includes the following steps:

[0026] S4-1, All pre-recorded motion primitive image sequences require the model to return to the same static pose, and the displacement difference of the skeletal key points between the first and last frames is ≤10 pixels;

[0027] S4-2 samples the displacement difference of the skeletal key points in the first and last frames multiple times and automatically selects the candidates with the lowest LPIPS and skeletal point MPJPE ≤ 10 pixels. Under the constraint of consistency between the first and last frames, the video generation model achieves LPIPS ≤ 0.035 and skeletal point MPJPE ≤ 7.2 pixels on the 1080p / 30fps test set, meeting the requirements of live-streaming visual continuity.

[0028] Preferably, the time generation method employs a real-time generation system, comprising the following units:

[0029] Large language model unit, used to receive user input and generate language response text and non-language behavior text description;

[0030] The semantic mapping and retrieval unit is used to map the text description of the non-verbal behavior into skeletal motion vectors, and retrieve the corresponding motion primitive image sequence from the motion primitive library accordingly.

[0031] The multimodal synthesis unit is used to generate a smooth motion video from multiple retrieved motion primitive image sequences, generate synchronized lip movements based on speech, and synthesize and output the generated smooth motion video and the corresponding speech.

[0032] Large language model units include ASR, LLM, and system prompt words;

[0033] The semantic mapping and retrieval unit includes a pre-trained mapping network and a cosine similarity-based retrieval module;

[0034] The multimodal synthesis unit includes first and last frame images, video stitching, and lip-sync.

[0035] The semantically driven and skeletal motion vector matching method for real-time generation of 2D digital humans provided by this invention has the following advantages compared with existing technologies:

[0036] This invention presents a real-time 2D digital human generation method based on semantic-driven and skeletal motion vector matching. It employs a large language model to uniformly generate linguistic and non-linguistic behavioral descriptions, ensuring semantic consistency. A text-action mapping encoder maps behavioral descriptions to skeletal motion vectors in real time, and performs cosine similarity retrieval within a motion primitive library, achieving zero-latency correspondence between one semantic and one pose. Furthermore, it utilizes the Wan2.2 video generation model's action stitching technology to generate smooth transition frames between two action primitive image sequences, eliminating jumps, and the Wav2Lip model to ensure lip-sync. This invention constructs a joint language-action representation space, achieving end-to-end low-latency generation, significantly improving the naturalness and realism of 2D digital human interaction, and can be widely used in virtual customer service, live-streaming avatars, and real-time interactive scenarios in the metaverse. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the training process of the text-action mapping encoder in this invention.

[0038] Figure 2 This is a schematic diagram illustrating the working principle of the motion splicing technology in this invention.

[0039] Figure 3 This is a diagram showing the connection of key points throughout the body in this invention.

[0040] Figure 4 (a) is the last frame of the preceding action of the video generation model in this invention.

[0041] Figure 4 (b) is the transition image frame of the video generation model in this invention.

[0042] Figure 4 (c) is the starting frame of the subsequent action of the video generation model in this invention.

[0043] Figure 5 (a) is an image frame before audio and video stream synthesis synchronization in this invention (lip movements are in a natural state).

[0044] Figure 5 (b) is the same image frame after audio and video stream synthesis and synchronization in this invention (lip movements match specific pronunciations). Detailed Implementation

[0045] The following provides a detailed description of specific embodiments of the present invention. It should be understood that the specific embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit the scope of the invention.

[0046] The semantic-driven and skeletal motion vector matching method for real-time generation of 2D digital humans of the present invention specifically includes the following steps:

[0047] Step S1: Generate instruction parsing and content; the user obtains and processes the instruction.

[0048] The system captures user voice input commands and converts the voice signal into a text string using an Automatic Speech Recognition (ASR) module. For example, "Hello, please introduce yourself."

[0049] Step S2: Generate the language response text and the non-language behavior description.

[0050] Taking a text string as input, the locally deployed Qwen3-14B large language model is invoked. Using preset system prompts, it generates structured JSON output containing two fields: `speech_text` (language response) and `action_description` (non-language behavior description). This output includes the language response text and a non-language behavior description semantically consistent with the language response text. After the model inference is complete, it returns the following data:

[0051] {"speech_text": "Hello, I am an AI digital human dedicated to providing you with a natural and smooth interactive experience.","action_description": "With a friendly and confident smile, lean slightly forward and make a natural welcoming gesture."}

[0052] Step S3: Semantic mapping and retrieval of skeletal motion vectors.

[0053] like Figure 1As shown, a nonverbal behavior description is input into a text-action mapping encoder to generate skeletal motion vectors. The text-action mapping encoder includes a CLIP text encoder and a mapping network, used to project the nonverbal behavior description onto a 256-dimensional skeletal motion vector space. The CLIP text encoder is used to convert the text into a 512-dimensional text feature vector, and the mapping network is used to project the 512-dimensional text feature vector onto the 256-dimensional skeletal motion vector space.

[0054] Specifically, it includes the following:

[0055] S3-1, Construction and mapping of a 256-dimensional skeletal motion vector space.

[0056] (1) Extract continuous image frames from the action primitive image sequence to obtain a continuous RGB image frame sequence.

[0057] (2) The DwPose algorithm (a real-time single-person pose estimation algorithm based on a lightweight convolutional pose machine) is used to extract the human skeleton key point sequence for each frame, and 133 key points with two-dimensional coordinates (x, y) and corresponding confidence scores e are obtained to form a temporal key point sequence.

[0058] (3) Perform a time-series sliding window on the keypoint sequence and execute the following encoding:

[0059] The window length is 16 frames (approximately 0.53 seconds, covering a complete action primitive unit, such as "waving" or "nodding"), and the step size is 8 frames (50% overlap, enhancing temporal continuity and avoiding boundary truncation).

[0060] Using the hip center as the origin, translation normalization is performed on all keypoints to eliminate interference from global translation (such as camera movement or character movement); the joint displacements Δx and Δy between adjacent frames are calculated and spliced ​​to obtain 133×2×15=3990-dimensional motion features (i.e., 133 joints × 2 coordinates × 15 frame differences = 3990 dimensions, with 15 displacement segments due to the 16-frame window); the dimensions are reduced to 256 dimensions using PCA (principal component analysis) (retaining more than 95% of the variance to eliminate redundancy and overfitting risks), and L2 normalization is performed to ensure that all motion vectors lie on the unit hypersphere. This is then adapted to cosine similarity retrieval to obtain the skeletal motion vector Va of the action segment.

[0061] (4) Multiple skeletal motion vectors Va and their corresponding motion primitive image sequences together constitute a motion primitive library, which is used for subsequent cosine similarity retrieval.

[0062] Figure 3 This paper presents a full-body skeletal keypoint diagram that accurately depicts the pose, extracted from the image using the DwPose algorithm.

[0063] The generated 256-dimensional vector is not a static encoding of a single frame's pose, but rather obtained by performing PCA dimensionality reduction on the displacement sequence of 133 key points within a multi-frame temporal window, containing motion direction, velocity, and phase information. For example, the differences in displacement sign and principal and secondary components between "smiling + waving" and "smiling + backing away" can be distinguished by the vector space.

[0064] S3-2, Supervised mapping network training from text to skeletal motion vectors.

[0065] The training process is as follows:

[0066] (1) Training data collection:

[0067] A large number of 2-4 second real-person motion videos were selected, and a 256-dimensional skeletal motion vector Va was generated as the "real vector" label. For each real-person motion video, an annotator wrote a 20-40 word motion description text, forming <text, Va> pairs of samples, which were divided into training / validation / test sets in an 8:1:1 ratio.

[0068] (2) Mapping network structure:

[0069] The CLIP ViT-L / 14 text encoder is frozen, and the output is a 512-dimensional text feature ftxt. The weights are fixed during training to avoid catastrophic forgetting in large text models.

[0070] Trainable layers:

[0071] ① Linear: A fully connected dimensionality reduction layer that linearly maps 512-dimensional text features to a 256-dimensional latent space;

[0072] ② ReLU: Linear rectified activation function, zero-truncation for negative values, introducing nonlinearity;

[0073] ③ Dropout: Randomly deactivates 10% of neurons to suppress overfitting;

[0074] ④ Linear: Further linear transformation to enhance expressive power;

[0075] ⑤ L2 normalization: Reduces the length of the output vector to 1, making it fall on the unit hypersphere, which facilitates subsequent cosine retrieval.

[0076] The four trainable layers mentioned above are collectively called "MLP_map (Multilayer Perceptron Mapping Network)". The final output is normalized by L2 to obtain the predicted skeletal motion vector Vp.

[0077] (3) Monitoring objectives:

[0078] The loss function is L=1-cos(Vp,Va), which is the "cosine similarity loss". The goal is to minimize the angle between the predicted vector Vp and the ground truth vector Va. The parameters of the mapping network are optimized through backpropagation of the loss function.

[0079] (4) Training results:

[0080] On the validation set, the average cosine similarity between the predicted vector Vp and the real vector Va reached 0.934. After training, the MLP_map parameters were fixed and deployed as a "mapping network" in the online inference process to achieve real-time output of text mapping to 256-dimensional skeletal motion vectors.

[0081] S3-3, Text Feature Vector Extraction and Projection.

[0082] The action description text is fed into a text-action mapping encoder. The CLIP ViT-L / 14 text encoder within this encoder first converts the input text into a 512-dimensional text feature vector. Subsequently, the mapping network projects this 512-dimensional text feature vector onto a 256-dimensional skeletal motion vector space, and after L2 normalization, obtains the skeletal motion vector Vq. An example of the normalized Vq is shown below (rounded to four decimal places):

[0083] Vq=[-0.1034,0.5821,-0.0449,0.8912,0.1577,...,-0.2356], ‖Vq‖2=1.

[0084] S3-4, Action Element Image Sequence Matching.

[0085] The similarity between the queried skeleton motion vector and each skeleton motion vector in the motion primitive library is calculated, and the matching motion primitive image sequence is retrieved.

[0086] An Approximate Nearest Neighbor (ANN) search algorithm was employed, with cosine similarity used for similarity calculation. The retrieval strategy was approximate nearest neighbor search, with a confidence threshold ≥0.85. The search was conducted within an action primitive library. Each action primitive in the library corresponds to a 256-dimensional skeletal motion vector Va. The optimal match was determined by calculating the cosine similarity between Vq and all Va values. An example of the search results is shown below:

[0087] (1) Action ID: A0851 (Smiling Welcome), Cosine Similarity: 0.9618;

[0088] (2) Action ID: A1123 (nodding in greeting), cosine similarity: 0.8734;

[0089] (3) Action ID: B0240 (reaching out to lead), cosine similarity: 0.8599.

[0090] Based on a preset confidence threshold of 0.85, all three action primitives are selected, and their corresponding action primitive image sequences can be obtained. Publicly available experimental data shows that when the cosine similarity threshold is set to 0.85, the false positive rate can be controlled within 1% in all three scenarios: mass spectrometry retrieval, text classification, and model update detection, and the overall precision-recall ratio is optimal. Therefore, this method adopts this value as the confidence criterion for action retrieval.

[0091] The action primitive library of this method is not designed for "everyday human behavior," but rather for constructing a closed set of actions performed by a single pre-recorded digital human in its specific business scenarios (customer service, live streaming, tour guiding, etc.). The actual collection scope is only 30-50 sets of high-frequency actions, and after repeated collection at different speeds, amplitudes, and emotional intensities, the total number is approximately 3,500.

[0092] Step S4: Action splicing based on video generation model.

[0093] like Figure 2 As shown, when the retrieval result of step S3 contains two temporally consecutive motion primitive image sequences, a transitional image frame is generated between the two through motion stitching to form a smooth motion video. Specifically, if there is only one motion primitive image sequence, it is repeated once and reversed to obtain the second motion primitive image sequence.

[0094] If the digital human's current state is "standing and waiting", for example, action ID: A0001, in order to smoothly transition to "smiling and welcoming", action ID: A0851, the action splicing takes the last frame of the previous action and the starting frame of the subsequent action as input conditions, and infers to generate multiple kinematically continuous transition frames to achieve seamless splicing.

[0095] In this embodiment, the splicing process can be referred to Figure 4 . Figure 4 (a) in the image represents the last frame of the preceding action. Figure 4 (c) in the image represents the starting frame of the subsequent action. Figure 4 (b) in the image is a transitional image frame generated by the engine between the two.

[0096] The method of this invention significantly reduces splicing errors through a "pre-generation + static alignment" strategy, and includes the following steps:

[0097] S4-1, All pre-recorded motion primitive image sequences require the model to return to the same static pose, and the displacement difference of the skeletal key points between the first and last frames is ≤10 pixels;

[0098] S4-2 samples the displacement difference of skeletal key points in the first and last frames multiple times and automatically selects candidates with the lowest LPIPS (Learning Perceptual Image Patch Similarity) and skeletal point MPJPE (Mean Average Joint Position Error) ≤ 10 pixels. Under the constraint of consistency between the first and last frames, the video generation model achieves LPIPS ≤ 0.035 and skeletal point MPJPE ≤ 7.2 pixels on the 1080p / 30fps test set, meeting the requirements of live-streaming visual continuity.

[0099] Step S5: Audio and video stream synthesis and output.

[0100] The smooth motion video from step S4 is subjected to lip-sync processing to ensure that its lip movements match the synthesized speech corresponding to the spoken response text, and the audio and video stream is output in real time, including the following steps:

[0101] S5-1, the speech_text generated in step S2 is converted into WAV audio using the indexTTS2 engine. The video sequence generated in step S4 is received using Wav2Lip technology to generate a lip-synced video sequence.

[0102] Synchronization effect such as Figure 5 As shown, Figure 5 (a) is the image frame before synchronization (lip movements are in a natural state). Figure 5 (b) in the image is the same image frame after synchronization (lip movements match specific pronunciations).

[0103] The semantically driven and skeletal motion vector matching method for real-time generation of 2D digital humans of the present invention employs a real-time generation system, comprising the following units:

[0104] Large language model unit, used to receive user input and generate language response text and non-language behavior text description;

[0105] The semantic mapping and retrieval unit is used to map the text description of the non-verbal behavior into skeletal motion vectors, and retrieve the corresponding motion primitive image sequence from the motion primitive library accordingly.

[0106] The multimodal synthesis unit enables motion splicing, lip-sync, and audio-video synthesis. It is used to generate a smooth motion video from multiple retrieved motion primitive image sequences, generate synchronized lip movements based on speech, and then synthesize and output the generated smooth motion video and the corresponding speech.

[0107] The large language model unit includes ASR (speech recognition), LLM (large language model), and system prompts.

[0108] The semantic mapping and retrieval unit includes a pre-trained mapping network and a cosine similarity-based retrieval module.

[0109] The multimodal synthesis unit includes first and last frame images, video stitching, and lip-sync.

[0110] The above embodiments are merely preferred examples of the present invention and are not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A method for real-time generation of 2D digital humans based on semantic-driven and skeletal motion vector matching, characterized in that, The real-time generation method employs a large-scale language model to generate verbal and non-verbal behavior descriptions. A text-action mapping encoder maps the behavior descriptions to skeletal motion vectors in real time, and performs cosine similarity retrieval within a motion primitive library to achieve zero-latency correspondence between one semantic and one posture. A video generation model is used to stitch actions together to generate smooth transition frames between two primitive segments, forming a smooth motion video. Lip-sync processing is applied to the smooth motion video. In the generation of verbal and non-verbal behavior descriptions using a large-scale language model, user voice commands are first received, and the voice signal is converted into a text string. The large-scale language model parses the command to generate verbal response text and non-verbal behavior descriptions semantically consistent with the verbal response text. The text-action mapping encoder includes a CLIP text encoder and a mapping network. The CLIP text encoder converts the text into a 512-dimensional text feature vector, and the mapping network projects the 512-dimensional text feature vector onto a 256-dimensional skeletal motion vector space. Mapping behavior descriptions to skeletal motion vectors in real time includes the following steps: S3-1, Construction and Mapping of 256-Dimensional Skeletal Motion Vector Space: Extracting continuous image frames from motion primitive image sequences; using DwPose technology to extract human skeletal key point sequences from image frames to form skeletal motion vectors Va; multiple skeletal motion vectors Va and their corresponding motion primitive image sequences together constitute a motion primitive library. In S3-2, the skeletal motion vector Va is used as the real vector, and the predicted skeletal motion vector Vp is used as the predicted vector. The loss function is L=1-cos(Vp,Va), which is the cosine similarity loss. The average cosine similarity between the predicted vector Vp and the real vector Va is calculated. After training, the MLP_map parameters are fixed and it is deployed as a "mapping network" in the online inference process to realize the real-time output of text mapping to 256-dimensional skeletal motion vectors. S3-3, the CLIP text encoder first converts the input text into a 512-dimensional text feature vector; then, the mapping network projects this 512-dimensional text feature vector onto a 256-dimensional skeletal motion vector space.

2. The method for real-time generation of 2D digital humans based on semantic-driven and skeletal motion vector matching according to claim 1, characterized in that, The cosine similarity retrieval within the motion primitive library includes: using the similarity between the 256-dimensional skeletal motion vector Vq and each skeletal motion vector Va in the motion primitive library to retrieve matching motion primitive image sequences; and the retrieval algorithm determines the optimal match by calculating the cosine similarity between Vq and all Va.

3. The method for real-time generation of 2D digital humans based on semantic-driven and skeletal motion vector matching according to claim 1, characterized in that, The process of generating a smooth transition frame between two primitive segments includes the following steps: S4-1, All pre-recorded motion primitive image sequences require the model to return to the same static pose, and the displacement difference of the skeletal key points between the first and last frames is ≤10 pixels; S4-2 samples the displacement difference of the skeletal key points in the first and last frames multiple times and automatically selects the candidates with the lowest LPIPS and skeletal point MPJPE ≤ 10 pixels. Under the constraint of consistency between the first and last frames, the video generation model achieves LPIPS ≤ 0.035 and skeletal point MPJPE ≤ 7.2 pixels on the 1080p / 30fps test set, meeting the requirements of live-streaming visual continuity.

4. The method for real-time generation of 2D digital humans based on semantic-driven and skeletal motion vector matching according to claim 1, characterized in that, The real-time generation method employs a real-time generation system, which includes the following units: Large language model unit, used to receive user input and generate language response text and non-language behavior text description; The semantic mapping and retrieval unit is used to map the text description of the non-verbal behavior into skeletal motion vectors, and retrieve the corresponding motion primitive image sequence from the motion primitive library accordingly. The multimodal synthesis unit is used to generate a smooth motion video from multiple retrieved motion primitive image sequences, generate synchronized lip movements based on speech, and synthesize and output the generated smooth motion video and the corresponding speech. Large language model units include ASR, LLM, and system prompt words; The semantic mapping and retrieval unit includes a pre-trained mapping network and a cosine similarity-based retrieval module; The multimodal synthesis unit includes first and last frame images, video stitching, and lip-sync.

Images