Digital human generation and interaction method, equipment and medium

Abstract

The invention provides a digital human generation and interaction method and device and a medium, and relates to the technical field of digital humans, and the method comprises the steps: constructing a universal digital human image and a reference action video library matched with the universal digital human image; generating a personalized action video library corresponding to the target digital human image based on the target digital human image provided by the user and the reference action video segment; responding to voice wake-up information sent by a user, and generating corresponding reply text and reply voice; generating a plurality of reply action video segments based on the reply voice and the personalized action video library; the reply voice, the reply action video segment and the reply text are subjected to time sequence alignment, then the reply text is bound to serve as subtitles, and a synchronous digital human interaction video with real-time synchronization of the voice, the action and the subtitles is obtained; the problems that traditional personalized digital human customization is large in difficulty and high in cost are solved by means of universal action resources, and in the digital human interaction stage, the sense of reality, fluency and real-time response capacity of digital human interaction are remarkably enhanced.

Description

Technical Field

[0001] This application generally relates to the field of digital human technology, and specifically to a method, device and medium for digital human generation and interaction. Background Technology

[0002] With the rapid development of metaverse, virtual reality, and human-computer interaction technologies, digital humans, as a key carrier connecting the real world and digital space, have shown broad application prospects in various fields such as education, finance, customer service, entertainment, and media. An ideal digital human system should be able to quickly generate realistic virtual avatars and understand user intentions, enabling natural, smooth, and expressive real-time interactions.

[0003] However, existing digital human technology systems still face bottlenecks. Firstly, regarding digital human appearance, current solutions largely rely on professional artists manually creating 3D models and rigging skeletons for specific characters—a time-consuming, labor-intensive, and costly process; or they rely on recording with live models, which requires specific recording environments and is also time-consuming and labor-intensive. These limitations severely restrict the large-scale production and rapid deployment of digital humans. Secondly, in terms of motion generation and actuation, existing technologies either rely on predefined, fixed animation libraries for random playback, resulting in a lack of semantic relevance in motion performance; or they use algorithms to generate motion based on speech, but it is difficult to achieve a balance between the naturalness, diversity, and real-time nature of the motion. Summary of the Invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a highly versatile method, device and medium for digital human generation and interaction.

[0005] In a first aspect, this application provides a method for digital human generation and interaction, comprising the following steps: Construct a general digital human avatar and a reference action video library that matches it; the reference action video library includes at least a number of reference action video segments, which are video segments of multiple actions displayed by the general digital human avatar; Based on the image of the target digital human provided by the user and the reference action video segment, a personalized action video library corresponding to the target digital human is generated; the personalized action video library includes at least a number of target action video segments, which are video segments in which the target digital human displays the multiple actions. In response to a user's voice wake-up message, a corresponding reply text and a corresponding reply voice are generated; Based on the response voice and the personalized action video library, multiple response action video segments are generated; The response audio, the response action video segment, and the response text are time-aligned, and then the response text is bound as a subtitle to obtain a synchronized digital human interactive video with real-time synchronization of audio, action, and subtitle.

[0006] Based on the technical solution provided in this application, a general digital human image and a matching reference motion video library are constructed, specifically including the following steps: Obtain digital human template prompts and input the digital human template prompts into the text-based image model to obtain digital human template images; Obtain action behavior description prompts, and input the digital human template image and the action behavior description prompts into the video generation model to obtain a general digital human image and a reference action video library that matches it.

[0007] According to the technical solution provided in this application, a personalized motion video library corresponding to the target digital human image is generated based on the image of the target digital human provided by the user and the reference motion video segment, specifically including the following steps: The image of the target digital human is input into the image-generated image-large model for processing to generate a full-body frontal image of the target digital human; the full-body frontal image of the target digital human includes the complete frontal body shape of the target digital human. Based on motion transfer technology, the motions displayed by the general digital human image in the reference motion video library are transferred to the target digital human image, generating multiple target motion video segments, and obtaining a personalized motion video library corresponding to the target digital human image.

[0008] According to the technical solution provided in this application, based on motion transfer technology, the motion displayed by the general digital human image in the reference motion video library is transferred to the target digital human image to generate multiple target motion video segments, specifically including the following steps: Replace the actions displayed by the general digital human image in the reference action video library with the target digital human image to generate an initial personalized action video segment; Based on the image of the target digital human, the initial personalized action video segment is subjected to face replacement and frame processing to obtain the repaired personalized action video segment. The repaired personalized action video segment is subjected to frame interpolation processing to generate a frame-connected personalized action video segment with consistent first and last frames. Super-resolution reconstruction is performed on all the inter-frame stitched personalized motion video segments to obtain the target motion video segment.

[0009] According to the technical solution provided in this application, in response to a user's voice wake-up message, a corresponding reply text and a corresponding reply voice are generated, specifically including the following steps: Receive voice wake-up information from the user, and activate the voice recognition model after recognizing the voice wake-up word carried in the voice wake-up information; The speech recognition model converts the user's voice interaction information into text. The text is input into the large language model, and the corresponding reply text is output. At the same time, the reply voice corresponding to each reply text is output.

[0010] According to the technical solution provided in this application, multiple response action video segments are generated based on the response voice and the personalized action video library, specifically including the following steps: Obtain the target action video segment that matches the reply voice from the personalized action video library as the interactive action video segment; The response voice is cut into multiple audio segments, and the audio segments are matched and bound to the video frames of the interactive action video segment to obtain the bound audio segments and video frames; The bound audio clips and video frames are input into the lip-sync model to generate corresponding lip motion frames; The lip movement frame is fused with the interactive action video segment to obtain the response action video segment.

[0011] According to the technical solution provided in this application, the method further includes the following steps: The response action video segment is played at a preset video frame rate, and the response action video segment to be played is cached in a buffer queue according to a preset rule.

[0012] According to the technical solution provided in this application, the video segment of the response action to be played is cached into a buffer queue according to preset rules, specifically including the following steps: S1. Determine whether the number of frames of the response action video segment to be played temporarily stored in the buffer queue exceeds the preset buffer threshold; if the number of frames of the response action video segment to be played temporarily stored in the buffer queue does not exceed the preset buffer threshold, then perform a write operation to store the response action video segment to be played into the buffer queue. S2. If the number of frames of the response action video segment to be played temporarily stored in the buffer queue exceeds the preset buffer threshold, the writing operation and the response action video segment synthesis process are paused and the interval waiting stage is entered. S3. After the interval waiting phase ends, S1-S2 are executed again until the user exits.

[0013] Secondly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the digital human generation and interaction method described above.

[0014] Thirdly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the digital human generation and interaction method described above.

[0015] As can be seen from the above technical solution, this application has at least the following beneficial effects: This application provides a method for digital human generation and interaction, including constructing a general digital human image and a reference action video library matching it; the reference action video library includes at least multiple reference action video segments, which are video segments of multiple actions displayed by the general digital human image; based on the image of the target digital human image provided by the user and the reference action video segments, a personalized action video library corresponding to the target digital human image is generated; the personalized action video library includes at least multiple target action video segments, which are video segments of multiple actions displayed by the target digital human image; in response to the user's voice wake-up information, a corresponding reply text and a reply voice corresponding to the reply text are generated; based on the reply voice and the personalized action video library, multiple reply action video segments are generated; the reply voice, reply action video segments and reply text are time-aligned, and then the reply text is bound as a subtitle to obtain a synchronized digital human interactive video with real-time synchronization of voice, action and subtitle.

[0016] This application lays a reusable foundation for personalized digital human generation by constructing a general digital human image and a matching reference action video library. Combined with the user-provided target digital human image and reference action video segments, actions from the reference action video library are transferred to the target digital human image to generate a personalized action video library. This leverages general action resources to solve the problems of high difficulty and cost in traditional personalized digital human customization, achieving rapid and high-quality generation of personalized digital human action videos. Furthermore, during the digital human interaction stage, voice-driven and semantic understanding, combined with action scheduling and multimodal temporal alignment technology, ensures that the most matching action video segment is selected from the pre-made high-quality reference action video library as the target action video segment. Voice, lip movements, body movements, and subtitles are then synchronously and smoothly integrated in real time, significantly enhancing the realism, fluency, and real-time responsiveness of digital human interaction. Attached Figure Description

[0017] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0018] Figure 1 A flowchart for digital human generation and interaction methods.

[0019] Figure 2Example diagrams generated for response text and action tags.

[0020] Figure 3 Example diagram of speech preprocessing for responses.

[0021] Figure 4 This is an example diagram showing the synchronization of audio clips with video frames and lip movements.

[0022] Figure 5 This is an example diagram of video frame synthesis.

[0023] Figure 6 Example diagram for audio and video playback.

[0024] Figure 7 Example image of a general digital human.

[0025] Figure 8 An example image of a waving gesture for a generic digital human avatar.

[0026] Figure 9 Example image of a human face in an image of a target digital human figure.

[0027] Figure 10 Example image of a frontal full-body image of a target digital human.

[0028] Figure 11 An example image of a waving gesture for a target digital human figure.

[0029] Figure 12 Example image of the facial region extracted before face replacement.

[0030] Figure 13 This is an example image of the facial area after face replacement.

[0031] Figure 14 This is an example of the first frame of a video segment where the first and last frames are consistent for inter-frame seamless motion.

[0032] Figure 15 This is an example diagram of a waving gesture corresponding to a video segment with consistent first and last frames.

[0033] Figure 16 This is an example diagram of the last frame of an inter-frame motion video segment where the first and last frames are consistent.

[0034] Figure 17 This is a schematic diagram of the electronic device.

[0035] Figure 18 Example diagrams for digital human generation and interaction.

[0036] Figure 19 Example images created for digital human templates and reference motion videos.

[0037] Figure 20 Example image created for a specific digital human character.

[0038] Figure 21 Example diagram for a specific digital human application.

[0039] Figure 22 Example diagrams generated for digital human responses to text and action tags.

[0040] Figure 23 Example diagram of speech preprocessing for responses.

[0041] Figure 24 This is an example diagram showing the synchronization of audio clips with video frames and lip movements.

[0042] Figure 25 This is an example diagram of video frame synthesis.

[0043] Figure 26 Example diagram for audio and video playback.

[0044] The following numbers are used in the diagram: 500, Electronic device; 501, CPU; 502, ROM; 503, RAM; 504, Bus; 505, I / O interface; 506, Input section; 507, Output section; 508, Storage section; 509, Communication section; 510, Driver; 511, Removable media. Detailed Implementation

[0045] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0046] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0047] like Figure 1 As shown, this application provides a method for digital human generation and interaction, including the following steps: S100. Construct a general digital human image and a reference action video library that matches it; the reference action video library includes at least multiple reference action video segments, which are video segments of multiple actions displayed by the general digital human image.

[0048] Here, the generic digital human avatar is a baseline template generated by a cue word-driven textual graph model.

[0049] Furthermore, a general digital human avatar and a matching reference motion video library are constructed, specifically including the following steps: Obtain digital human template prompts and input them into the text-based image model to obtain digital human template images; Obtain action behavior description prompts, and input the digital human template image and action behavior description prompts into the video generation model to obtain a general digital human image and a reference action video library that matches it.

[0050] It should be noted that the digital human template prompts include basic identity features, facial detail features, clothing style features, background and image quality features, and posture features. Basic identity features refer to the digital human's age range, gender, regional style, and other fundamental attributes. Facial detail features refer to facial expressions, skin texture, hairstyle, and other details. Clothing style features refer to the type, color, and style of clothing, while also considering both general applicability and scene suitability. Background and image quality features refer to the background style, avoiding complex backgrounds that might interfere with the subject; at the same time, clearly defining image quality requirements to provide a high-quality image foundation for subsequent action video generation. Posture features refer to the basic standing posture. Examples of text-based image models include Stable Diffusion, MidJourney, Qianwen Image, Flux2, Nano Banana, and Hunyuan Image.

[0051] For example, the prompt for the digital human template is an Asian female digital human aged 5-30, with a friendly and gentle face, a natural smile, clear and flawless skin texture, slightly wavy brown medium-length hair, and smooth hair texture. She is wearing a simple black business suit, standing naturally with her arms hanging down. The background is pure white, with an ultra-realistic style, 8K resolution, photo-quality, clear and sharp image, and bright eyes.

[0052] like Figure 7 As shown, the designed digital human template prompts are input into the text-based image model. The model generates corresponding digital human template images based on the text descriptions of the prompts using a deep learning algorithm. After generation, initial verification is performed. First, it is confirmed whether the digital human template image fully matches the prompt description and whether the image has no obvious flaws. If deviations exist, the prompt details are adjusted, and the digital human template image is regenerated until a digital human template image that meets the requirements of generalization and high quality is obtained. This image serves as the fixed subject for all subsequent action video generation, ensuring that all reference actions are generated based on the same image, achieving a strong binding between action and image.

[0053] The action description prompts include default actions and specific actions. Default actions are basic dynamic actions when there are no specific interaction commands, used to avoid the digital human image being static and stiff. For example, the prompt for natural standing and breathing is: while standing naturally, the chest rises and falls slightly with breathing at a frequency of 18-20 times per minute, the arms hang naturally without any extra movements, and the head does not shake significantly; the prompt for slight nodding is: the head nods slowly downward with the cervical spine as the axis, the amplitude is about 10-15 degrees, it lasts for 0.5 seconds and then slowly returns to the starting position, the movement is smooth and without any pauses.

[0054] Specific actions are functional actions that meet the needs of daily interaction scenarios. For example, the prompt for waving with the right hand is to start from a natural standing posture, slowly raise the right hand with the shoulder joint as the axis, bend the elbow at an angle of about 120 degrees, open the palm naturally with the five fingers together, and swing the wrist and palm horizontally outward from the body with an amplitude of 15-20 degrees, swinging 2-3 times in total, with each swing lasting 0.8-1 seconds. The left arm keeps hanging naturally, and the head turns slightly in the direction of the wave (with an amplitude of no more than 10 degrees). After the action is completed, the right hand slowly returns to the initial position.

[0055] like Figure 8 As shown in the diagram, the arrows indicate the movement trajectory of the hand. The corresponding action description is a right hand wave "hello". Starting from the default standing posture, the right hand is slowly raised around the shoulder joint, with the elbow slightly bent (the bending angle is about 120 degrees). The palm is naturally open (five fingers together, fingertips pointing forward). The wrist leads the palm to swing lightly outward (away from the body) in a horizontal direction. The swing amplitude is controlled at 15-20 degrees, and the number of swings is 2-3 times (each swing lasts 0.8-1 seconds). During the movement, the left arm remains in the default hanging position, and the head can be slightly turned in the direction of the wave (the rotation amplitude does not exceed 10 degrees). After completion, the right hand slowly returns to the default position.

[0056] Here, the video generation model is, for example, Veo3, Wan, Hunyuan Video, etc.

[0057] The digital human template image and a single action description prompt are used as joint input data to the video generation model. The video generation model first identifies the main human figure in the digital human template image, including its outline, skeletal structure, and initial pose. Then, based on the description content of the action description prompt, a temporal generation algorithm adds a dynamic frame sequence that matches the action details to the main human figure, ultimately synthesizing it into a continuous video, with a video length of, for example, 3-5 seconds. This process requires generating a corresponding reference video for each action to ensure that each action has an independent video carrier.

[0058] Following the above generation process, after generating reference videos for all single actions, firstly, the completeness and smoothness of the actions in each reference video are checked manually or using automated tools. If problems are found, the corresponding action prompts need to be adjusted, and the videos need to be re-entered into the video generation model until all videos meet the standards. For example, if a waving action does not complete the fall back, the corresponding action prompts need to be adjusted to include the requirement that the right hand fully falls back to a natural hanging position after the action. Secondly, all approved reference videos are standardized to the same format, resolution, and frame rate to avoid compatibility issues during subsequent personalized action migration due to format differences. For example, the format is MP4, the resolution is 1920×1080 pixels, and the frame rate is 30fps. Finally, the videos are categorized by action type, and a clear directory structure is established to form the final reference action video library that matches the general digital human image, as shown in Table 1. Each video in the library uses a unified digital human template image as its main body, and the actions are standard and directly reusable.

[0059] Table 1 Reference Action Video Library

[0060] S101. Based on the image of the target digital human image and reference action video segments provided by the user, generate a personalized action video library corresponding to the target digital human image; the personalized action video library includes at least multiple target action video segments, which are video segments in which the target digital human image displays multiple actions.

[0061] The process of generating a personalized motion video library corresponding to the target digital human image, based on images and reference motion video clips provided by the user, includes the following steps: The image of the target digital human is input into the image-generated large model for processing to generate a full-body frontal image of the target digital human; the full-body frontal image of the target digital human includes the complete frontal body shape of the target digital human. Based on motion transfer technology, the motions displayed by a general digital human image in a reference motion video library are transferred to the target digital human image, generating multiple target motion video segments and obtaining a personalized motion video library corresponding to the target digital human image.

[0062] It should be noted that the image of the target digital human is, for example, Figure 9 As shown, the image of the target digital human (with a frontal face as the core) is input into a raw image model. Examples of raw image models include Qwen-Image-Edit, NanoBanana raw image mode, Flux Context, and Keling raw image mode. Raw image models are commonly used in this field, and their operation process is existing technology, so it will not be described in detail here. The generated full-body frontal image of the target digital human is shown below. Figure 10 As shown. Specifically, a frontal facial photograph of the target digital human is input into the image-based image model. Based on facial features, skin tone, hairstyle, and other information displayed in the frontal facial photograph, the image-based image model completes the full-body details. Furthermore, the body shape is derived from a complete body shape template generated by the image-based image model, ultimately resulting in a full-body frontal image of the target digital human, containing the target digital human's complete frontal body shape. Here, body details include torso proportions, limb posture, and basic clothing style adaptation.

[0063] Furthermore, based on motion transfer technology, the motions displayed by a general digital human avatar in the reference motion video library are transferred to the target digital human avatar, generating multiple target motion video segments. This specifically includes the following steps: Replace the actions displayed by the generic digital human in the reference action video library with the target digital human to generate an initial personalized action video clip; Based on the image of the target digital human, the initial personalized action video segment is subjected to face replacement and frame processing to obtain the repaired personalized action video segment. Frame interpolation is performed on the repaired personalized motion video segments to generate inter-frame stitched personalized motion video segments with consistent first and last frames. Super-resolution reconstruction is performed on all inter-frame stitched personalized motion video segments to obtain the target motion video segment.

[0064] Action transfer techniques can employ deep learning-based action cloning / transfer models or graph-based video models using ControlNet technology, such as Mimicmotion, Wan2.1 Vace, and Wan2.2-Animate. Specifically, the deep learning-based action cloning / transfer model first analyzes the reference action video frame by frame, extracting key skeletal points of the general digital human image in each frame, such as key nodes like the shoulder, elbow, hip, and knee joints, using pose estimation algorithms (e.g., OpenPose). Second, it annotates the key skeletal points of the target digital human and establishes a matching skeletal framework based on the target digital human's body proportions. Then, it maps the skeletal motion trajectory of the reference action proportionally onto the target digital human's skeletal framework. For example, if the right hand is raised 30 degrees in the reference action, the model calculates the corresponding actual raising angle and movement distance based on the target digital human's shoulder width and arm length, ensuring the movement amplitude matches the target digital human's body size and preventing small target digital humans from making excessively large movements or large target digital humans from making excessively small movements. Finally, based on the mapped skeletal trajectory, it generates frame-by-frame dynamic images of the target digital human's full-body image. While maintaining the target person's head features (facial features, hairstyle), body shape, and clothing, it moves the torso and limbs according to the mapped trajectory, thus creating the initial personalized action video segment, such as... Figure 11As shown, taking waving hands as an example, the action logic in this video library is completely consistent with the reference action video library, but the executing subject changes from a general digital human image to a target digital human image, realizing the initial combination of action reuse and image customization.

[0065] Here, face-swapping models can be used to replace the face in the initial personalized motion video segment, such as Inswapper, Blendswap, SimSwaap, Uniface, etc. Specifically, for example... Figure 12 As shown, a complete facial region is extracted from the full-body frontal image of the target digital human provided by the user as a facial reference template. This facial region includes the forehead, eyebrows, eyes, nose, mouth, and chin, with no extraneous background, ensuring that the facial features, skin tone, and lighting effects of this template are completely consistent with the target digital human. Then, the initial personalized motion video segment is analyzed frame by frame, and the facial region in each frame is located using a facial detection algorithm, such as... Figure 13 As shown, a facial detection algorithm is used to align the facial reference template with the facial regions in the video frames of the initial personalized motion video segment. Then, two faces are input into the face-swapping model, which outputs a face image with facial features identical to the facial reference template, and facial expressions and shadows identical to the face in the video frame. Finally, the face output by the face-swapping model is pasted back into the original video frame, and a mask fusion algorithm is used to ensure natural edge transitions. Here, facial detection algorithms such as YOLO-Face and RetinaFace are used. After face-swapping, a high-definition facial restoration model is used to repair blurred skin textures, optimize eye details, and correct hairstyle edges, ensuring that the facial details in each frame are clear, realistic, and consistent with the target digital human image, thus obtaining the restored personalized motion video segment. Here, high-definition facial restoration models such as GFPGAN and CodeFormer are used.

[0066] Furthermore, frame interpolation processing is performed on the repaired personalized motion video segments, such as... Figure 14 and Figure 16 As shown, by ensuring that the last frame of each repaired personalized action video segment matches the first frame of its next matching repaired personalized action video segment, a seamlessly connected personalized action video segment with consistent first and last frames is obtained. This smooth transition of actions lays a solid foundation for the continuous playback of subsequent actions. Here, frame interpolation processing is performed using methods such as RIFE or M2M. In a video sequence with consistent first and last frames, Figure 15 This shows the state of a certain intermediate frame of the waving action.

[0067] For the personalized motion video segments with inter-frame stitching, a super-resolution reconstruction model (such as ESRGAN or ClearReality) is used to eliminate noise and artifacts (such as blurry jagged edges of motion) in the image, optimize clothing texture (such as restoring the fabric texture of the suit and the stripe details of the shirt), and improve background clarity to ensure that the overall video quality meets the high-definition interactive standard (no blur and no distortion). The target motion video segments are then obtained. Finally, the target motion video segments obtained through the above processing can be used to form a personalized motion video library corresponding to the target digital human image.

[0068] S102. In response to the user's voice wake-up message, generate the corresponding reply text and the corresponding reply voice.

[0069] Here, in response to the user's voice wake-up message, a corresponding reply text and a corresponding reply voice are generated, specifically including the following steps: It receives voice wake-up information from the user and activates the voice recognition model after recognizing the voice wake-up word carried in the voice wake-up information; The speech recognition model converts the user's voice interaction information into text. Input the text into the large language model and output the corresponding reply text. At the same time, output the reply voice corresponding to each reply text.

[0070] It's important to note that speech recognition models such as SensVoice, Whisper, and SeACoParaformer, when they recognize a wake-up word in the wake-up message, are activated and switch to working mode. They then recognize the user's voice interaction and convert it into text. This text is then input into a Large Language Model (LLM). The LLM first analyzes the core intent of the text, while also recognizing its emotional tone and interaction scenario, laying the semantic foundation for subsequent responses and action label selection. Based on this semantic understanding, it generates a natural language response that matches the scenario and the user's intent, ensuring the response fully addresses the need and guides subsequent interactions. According to the semantics, tone, and interaction purpose of the response text, it selects the most matching label from a predefined list of action labels as the action instruction. For example, if the response text is a greeting, such as "Hello!", it matches a wave; if it's confirmation / acknowledgment, it matches a slight nod; if it emphasizes key information, it matches emphasizing with both hands; if the response text has no clear emotional or action tendency, it matches natural standing and breathing.

[0071] For example, such as Figure 2As shown, the voice wake-up word is "hello." After being recognized by the speech recognition model, the model is activated. The speech recognition model then converts the voice interaction information into text and inputs it into the large language model, generating a matching digital human response and action type. The response text is then preprocessed, including word segmentation and sentence breaking, while matching corresponding predefined action tags. Each text segment, along with its corresponding predefined action tag, is stored in the large model's response queue for subsequent text-to-speech synthesis, until the entire response is complete.

[0072] Here, if the large language model is streaming, the response text preprocessing process does not need to wait for the large model to finish responding. Thus, while the large language model is responding, the already generated response text can be simultaneously converted into speech to reduce response latency.

[0073] While the large language model outputs the response text, a text-to-speech (TTS) synthesis task is initiated in parallel to convert the response text into corresponding response speech. The speech synthesis process involves obtaining text content from the large model's response queue, inputting it into the TTS model to generate response speech with natural intonation and rhythm.

[0074] S103. Based on the response voice and personalized action video library, generate multiple response action video segments, specifically including the following steps: Retrieve target action video segments that match the reply voice from a personalized action video library as interactive action video segments; The response voice is cut into multiple audio segments, and the audio segments are matched and bound to the video frames of the interactive action video segments to obtain the bound audio segments and video frames; The bound audio clips and video frames are input into the lip-sync model to generate corresponding lip motion frames; By fusing lip movement frames with interactive action video segments, a response action video segment is obtained.

[0075] It should be noted that, as Figure 3 As shown, the process involves obtaining the action command corresponding to the response speech, searching for a target action video segment matching the action command in a personalized action video library, and using this segment as the interactive action video segment to match the response speech. Then, the response speech is segmented into multiple audio segments according to its duration. Each audio segment is then matched with a corresponding video frame in the interactive action video segment, ensuring that the audio and video are synchronized at each moment. Examples of lip-sync models include Wav2Lip, LipGAN, and MuseTalk. Figure 4As shown, the bound audio clips and video frames are processed using a lip-sync model to obtain corresponding lip motion frames. These lip motion frames are then overlaid onto the transparent facial area of ​​the body frame according to the facial bounding box, ensuring that the size and angle of the lip motion frames perfectly match the body frame, avoiding any disproportionate lips and body or misaligned angles. The merged video frames are then image quality adjusted to ensure consistent resolution, brightness, and contrast between the facial and body areas, preventing visual disjointedness due to quality differences.

[0076] Furthermore, after inputting the bound audio segments and video frames into the lip-sync model to generate corresponding lip motion frames, if the current audio segment is a silent segment, the lip-sync processing is skipped, and the corresponding original facial frame in the target action video segment is directly used as the lip motion frame to reduce the computational consumption during the lip-sync process. Here, a silent segment refers to generating a corresponding silent audio segment of equal length when the audio segment information queue is determined to be empty.

[0077] S104. Align the response voice, response action video segment, and response text in sequence, and then bind the response text as a subtitle to obtain a synchronized digital human interactive video with real-time synchronization of voice, action, and subtitle.

[0078] The process involves aligning the timestamps of the response voice, response action video clips, and response text, using the response text as subtitles to obtain a synchronized digital human interactive video, and storing the synchronized digital human interactive video in a buffer queue for playback.

[0079] like Figure 5 As shown, the buffer queue is a temporary storage container based on memory or disk. It is designed with corresponding storage units for three types of resources: response voice, response action video segments, and response text, ensuring that resources can be stored in chronological order and retrieved in chronological order during playback. The synchronized digital human interactive video has synchronized video frames, each corresponding to a response voice, response action video segment, and response text. The buffer queue is shown in Table 2, where the queue index increases sequentially according to the resource generation order.

[0080] Table 2 Buffer Queue

[0081] like Figure 6 As shown, the generated response video segments are batch-written into a buffer queue in ascending order of timestamp. When a response voice read from the buffer queue is marked as the first frame, its corresponding response text is immediately displayed as a subtitle on the playback interface. When a response voice is marked as the last frame, the current subtitle is stopped after the segment is played. This mechanism ensures that the subtitle content is strictly synchronized with the start and end times of the response voice, and is aligned with the lip movements and actions in time, thereby enhancing the immersiveness and comprehensibility of the interaction.

[0082] This method manages asynchronous tasks and data streams during the interaction process through a multi-queue collaboration mechanism. Specifically, it receives user voice streams in real time, performs wake-word detection and speech recognition; stores response text and corresponding action commands generated by a large language model; receives response text and asynchronously generates corresponding response speech; schedules and preloads interactive action video segments from a personalized action video library based on action commands; and receives and temporarily stores response speech, response action video segments, and response text that have undergone lip-syncing and frame fusion, sorted by timestamps, for sequential reading by the playback module. The queues collaborate through event-driven mechanisms and state machines to ensure a smooth, low-latency end-to-end connection from voice input to multimodal video output.

[0083] Furthermore, the method also includes the following steps: The response video segment is played at a preset video frame rate, and the response video segment to be played is cached in the buffer queue according to preset rules.

[0084] Here, the preset video frame rate is, for example, 25 frames per second.

[0085] The process of caching the video segment of the response action to be played into a buffer queue according to preset rules includes the following steps: S1. Determine whether the number of frames of the response action video segment to be played temporarily stored in the buffer queue exceeds the preset buffer threshold; if the number of frames of the response action video segment to be played temporarily stored in the buffer queue does not exceed the preset buffer threshold, then perform the write operation to store the response action video segment to be played into the buffer queue. S2. If the number of frames of the response action video segment temporarily stored in the buffer queue exceeds the preset buffer threshold, the writing operation and the response action video segment synthesis process are paused and the interval waiting stage is entered. S3. After the interval waiting phase ends, S1-S2 are executed again until the user exits.

[0086] Here, the preset buffer threshold can be set according to actual needs.

[0087] The above playback and write logic mainly involves dynamically adjusting the write operation based on the load status of the buffer queue. Writes proceed normally when not overloaded, and pause when overloaded. This interval waiting process avoids queue congestion and ensures smooth interaction. Specifically, if the number of frames of the pending response video segment in the buffer queue does not exceed the preset buffer threshold, it indicates that the pending response video segment can be added to the end of the buffer queue. If the number of pending response video segments in the buffer queue exceeds the preset buffer threshold, it indicates that the current buffer queue is overloaded, and the pending response video segment cannot be directly added to the buffer queue. In this case, an interval waiting phase begins, where the write operation waits for the specified interval before execution.

[0088] The formula for calculating the waiting interval is as follows: ; In the formula, For the waiting interval, This represents the remaining number of frames in the buffer queue of the response video segment awaiting playback. To set a preset buffer threshold, This represents the number of frames per second played in the currently playing response video segment.

[0089] To facilitate understanding, the digital human generation and interaction methods in this application will be briefly introduced below.

[0090] like Figure 18 As shown, steps S11-S13 demonstrate how, by using prompt words to drive text-to-image and video generation models, digital human templates and reference action video segments are created, forming a general digital human image and reference action video library, laying a reusable foundation for personalized customization. Then, using the image of the target digital human image provided by the user as input, through steps such as image-to-image generation, motion transfer, and facial optimization, a specific digital human image and a unique personalized action video library are generated. Finally, this specific digital human is applied in the corresponding business scenario, that is, through voice wake-up to trigger interaction, through text generation, speech synthesis, motion scheduling, lip-sync, and multimodal synthesis, a digital human interactive video with real-time synchronization of voice, action, and subtitles is output, realizing an end-to-end closed loop from image generation to natural interaction.

[0091] like Figure 19As shown, steps S14-S17 demonstrate the design of digital human template prompts that include basic identity, appearance, clothing, background, and posture features. These prompts are input into a text-based image model to generate a digital human template image that meets generalization requirements. After generation, it is necessary to verify whether the prompt descriptions are completely matched. If there are discrepancies, the prompts are adjusted and regenerated. Action description prompts for default actions (such as natural standing and breathing) and specific actions (such as waving with the right hand) are designed, with each action having its own dedicated prompt. The digital human template image and individual action description prompts are input into a video generation model to generate an independent reference video for each action, with the video length controlled between 3 and 5 seconds. The generated reference videos are manually or automatically reviewed to determine the completeness and smoothness of the actions. Unqualified videos are regenerated after adjusting the prompts. All approved reference videos are standardized in format, resolution, and frame rate, and categorized by action type to form a standardized reference action video library.

[0092] like Figure 20 As shown, steps S18-S23 demonstrate obtaining a frontal facial image of the target digital human (requiring complete facial features, no occlusion, and a frontal angle) provided by the user, inputting it into a large image model, and the model completes the body details such as the torso and limbs based on the facial features to generate a full-body frontal image of the target digital human containing a complete frontal body shape; inputting the full-body frontal image of the target digital human and a reference motion video library into a motion transfer model, the model extracts the skeletal motion trajectory and temporal parameters of the reference motion, maps the trajectory according to the body proportion of the target digital human, and generates an initial personalized motion video; the facial region in the target person's face image is extracted as a baseline template, and face swapping is performed frame by frame on the initial personalized motion video, and then the skin texture, eye details, etc. are optimized through a high-definition facial restoration model to obtain a restored personalized motion video; the restored video is input into a frame interpolation model to generate a video with consistent posture at the beginning and end of the frame to ensure smooth motion transitions; the video noise and artifacts are eliminated through a super-resolution reconstruction model to optimize the image quality, and finally a high-quality specific digital human motion video library is formed.

[0093] like Figure 21 As shown, steps S24-S32 demonstrate the process of initiating the digital human's response text and action tag generation, converting the user's speech into text and generating corresponding action commands, which are then pushed to the large model's response queue. Next, the response speech preprocessing process is entered, where the response text is synthesized into speech and segmented into attributed audio segments, which are stored in the audio segment information queue. Subsequently, through the audio segment corresponding to video frame and lip-sync process, the audio segment is bound to the action video frame to generate a lip-aligned facial frame, which is pushed to the audio-facial video frame queue. Then, through the video frame synthesis process, the facial frame is merged with the body action video to generate a response action video segment, which is stored in the buffer queue. Finally, through the playback process, resources are retrieved from the buffer queue to synchronously play the audio and video and display subtitles, completing the human-computer interaction.

[0094] like Figure 22 As shown, steps S33-S37 demonstrate that the user issues a voice wake-up message, the voice wake-up model recognizes the wake-up word and returns a response script, and at the same time activates the speech recognition model; the speech recognition model receives the user's subsequent voice interaction information and converts it into text; the converted text is input into the large language model, the model parses the text intent and context, generates natural language response text, and matches predefined action tags corresponding to the response semantics and tone (such as matching the "wave" tag for greeting responses); during the generation process, the end punctuation of sentences is detected in real time, and the content fragments before the punctuation are extracted, while the remaining responses continue to be generated until the entire response is completed; the extracted response text fragments are associated with the corresponding action tags one by one and pushed to the large model response queue in sequence, waiting for further processing.

[0095] like Figure 23 As shown, steps S38-S43 demonstrate the system's cyclical detection of whether the large model response queue is empty. If it is empty, it waits for a specified time before re-detecting; if it is not empty, it proceeds to the next step. A response content and its corresponding action tag are retrieved from the large model response queue. The response content is input into the speech synthesis model, which performs preprocessing on the text, such as word segmentation, sentence breaking, and prosodic marking, to generate a response speech with natural intonation and rhythm. The response speech is cut into continuous fixed-length audio segments according to the video frame rate. Finally, attribute information is added to each audio segment, including whether the segment is the first or last frame of the audio, the corresponding response content, and the associated action tag, and then pushed to the audio segment information queue in sequence.

[0096] like Figure 24 As shown, steps S44-S58 demonstrate checking whether the remaining duration of the audio and video in the audio and video buffer queue is greater than the buffer playback threshold. If so, the waiting interval is dynamically calculated based on the queue, and the check is repeated after waiting. Otherwise, proceed to the next step. Check whether the audio segment information queue is empty. If so, generate a silent audio segment with the same duration as the audio segment as the processing object. Otherwise, retrieve an audio segment and attribute information from the queue. Determine whether the audio segment is the first frame of the audio. If so, further check whether the number of unprocessed frames of the current action video is less than the action switching threshold. If so, set the action tag in the attribute information as the action to be played. Check whether the current action video has been processed. If so, determine whether there is an action to be played. If so, set the corresponding action video as the current processing video. Otherwise, enable the default action video. Extract the first unprocessed face portion from the currently processed action video and bind it to the audio segment being processed as an audio segment-video frame pair. Input the binding pair into the lip-sync model to generate a facial video frame with lip movements aligned with the audio. Push the audio segment, attribute information, and generated facial video frame together to the audio-facial video frame queue.

[0097] like Figure 25 As shown, steps S59-S63 demonstrate that the system continuously checks whether the audio-facial video frame queue is empty. If it is empty, it waits for a specified time and then checks again; if it is not empty, it proceeds to the next step. The system retrieves the audio segment, the corresponding attribute information, and the facial video frame aligned with the lip movements from the queue. The facial video frame is then overlaid and merged with the body part of the high-quality specific digital human specified action video according to the facial bounding box position, ensuring that the size, angle, and image quality of the face and body are consistent to avoid visual disjointedness. A high-quality specific digital human specified action video frame with synchronized speech and lip movements is generated. Finally, the synthesized synchronized video frame, the corresponding audio segment, and the attribute information are pushed together to the audio-video buffer queue to prepare for subsequent playback.

[0098] like Figure 26 As shown, steps S64-S71 demonstrate the system's detection of whether the audio / video buffer queue is empty. If it is empty, it waits for a specified time and then checks again; if it is not empty, it proceeds to the next step. The system retrieves synchronized video frames, audio segments, and corresponding attribute information from the audio / video buffer queue. It determines whether the audio segment is the first frame of the audio; if so, the response content in the attribute information is used as subtitles and immediately displayed on the playback interface; otherwise, it maintains the current subtitle display state. It then determines whether the audio segment is the last frame of the audio; if so, it stops displaying the current subtitles after the segment finishes playing; otherwise, it continues displaying. The system synchronously plays the retrieved audio segment and the high-quality, specific digital human's designated action video frames on the playback interface, ensuring that the timing of the voice, video, and subtitles is consistent, thus completing the interactive output.

[0099] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the digital human generation and interaction method as described in the above embodiments.

[0100] Among them, such as Figure 17 As shown, the electronic device 500 includes a CPU 501, which can perform various appropriate actions and processes according to a program stored in ROM 502 or a program loaded from storage section 508 into RAM 503.

[0101] RAM 503 also stores various programs and data required for system operation. CPU 501, ROM 502, and RAM 503 are interconnected via bus 504. I / O interface 505 is also connected to bus 504.

[0102] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.

[0103] Specifically, according to embodiments of this application, the above reference flow Figure 1 The described process can be implemented as a computer software program.

[0104] For example, this application includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by CPU 501, it performs the functions defined in the system of this application.

[0105] It should be noted that the computer-readable medium shown in this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, RAM (random access memory), ROM (read-only memory), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0106] In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0107] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment—or portion of code—containing one or more executable instructions for implementing a specified logical function.

[0108] It should also be noted that in some alternative implementations, the functions marked in the boxes may occur in a different order than those shown in the accompanying drawings. For example, two consecutively indicated boxes may actually be executed substantially in parallel, or sometimes in reverse order, depending on the functions involved. It should also be noted that each box in a block diagram or flowchart, and combinations of boxes in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0109] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself. The described units or modules can also be located in a processor.

[0110] This application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to implement the digital human generation and interaction method provided in the above embodiments.

[0111] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A digital human generation and interaction method, characterized in that, The method comprises the following steps: constructing a general digital human image and a reference action video library matched therewith; The reference action video library at least comprises a plurality of reference action video segments, which are video segments of a plurality of actions demonstrated by the general digital human image; Based on the image of the target digital human image provided by the user and the reference action video segment, an individualized action video library corresponding to the target digital human image is generated; the individualized action video library at least comprises a plurality of target action video segments, which are video segments of the plurality of actions demonstrated by the target digital human image; In response to voice wake-up information issued by the user, corresponding reply text and reply voice corresponding to the reply text are generated; Based on the reply voice and the individualized action video library, a plurality of reply action video segments are generated; The reply voice, the reply action video segment and the reply text are time-aligned, and the reply text is bound as subtitles to obtain a synchronous digital human interaction video with real-time synchronization of voice, action and subtitles.

2. The method of claim 1, wherein, The method of constructing a general digital human image and a reference action video library matched therewith comprises the following steps: obtain a digital human template prompt word, and input the digital human template prompt word into a text-to-image model to obtain a digital human template image; obtain an action behavior description prompt word, and input the digital human template image and the action behavior description prompt word into a video generation model to obtain a general digital human image and a reference action video library matched therewith.

3. The method of claim 1, wherein, Based on the image of the target digital human image provided by the user and the reference action video segment, an individualized action video library corresponding to the target digital human image is generated, which comprises the following steps: input the image of the target digital human image into a picture-to-picture large model for processing to generate a front full-body image of the target digital human; the front full-body image of the target digital human includes the complete front body shape of the target digital human; Based on the action migration technology, the actions demonstrated by the general digital human image in the reference action video library are migrated to the target digital human image to generate a plurality of target action video segments, and an individualized action video library corresponding to the target digital human image is obtained.

4. The method of claim 3, wherein, Based on the action migration technology, the actions demonstrated by the general digital human image in the reference action video library are migrated to the target digital human image to generate a plurality of target action video segments, which comprises the following steps: replace the actions demonstrated by the general digital human image in the reference action video library with the target digital human image to generate an initial individualized action video segment; based on the image of the target digital human image, perform face replacement and frame processing on the initial individualized action video segment to obtain a repaired individualized action video segment; perform frame interpolation processing on the repaired individualized action video segment to generate a frame-interconnected individualized action video segment with consistent first and last frames; perform super-resolution reconstruction on all the frame-interconnected individualized action video segments to obtain target action video segments.

5. The method of claim 1, wherein, In response to voice wake-up information issued by the user, corresponding reply text and reply voice corresponding to the reply text are generated, which comprises the following steps: Receiving voice wake-up information issued by a user, and activating a voice recognition model after recognizing a voice wake-up word carried by the voice wake-up information; Converting voice interaction information issued by the user into text through the voice recognition model; Inputting the text into a large language model to output corresponding reply text, and simultaneously outputting reply voice corresponding to each sentence of the reply text.

6. The method of claim 1, wherein, Based on the reply voice and the personalized action video library, a plurality of reply action video segments are generated, specifically including the following steps: Obtaining a target action video segment matching the reply voice from the personalized action video library as an interactive action video segment; Cutting the reply voice into a plurality of audio segments, and matching and binding the audio segments with video frames of the interactive action video segment to obtain bound audio segments and video frames; Inputting the bound audio segments and video frames into a lip synchronization model to generate corresponding lip action frames; Fusing the lip action frames with the interactive action video segment to obtain a reply action video segment.

7. The method of claim 6, wherein, The method further includes the following steps: Playing the reply action video segment at a preset video frame rate, and simultaneously caching the reply action video segment to be played to a buffer queue according to a preset rule.

8. The method of claim 7, wherein, Caching the reply action video segment to be played to the buffer queue according to the preset rule, specifically including the following steps: S1, determining whether the number of frames of the reply action video segment to be played temporarily stored in the buffer queue exceeds a preset buffer threshold; if the number of frames of the reply action video segment to be played temporarily stored in the buffer queue does not exceed the preset buffer threshold, performing a write operation of storing the reply action video segment to be played into the buffer queue; S2, if the number of frames of the reply action video segment to be played temporarily stored in the buffer queue exceeds the preset buffer threshold, pausing the write operation and the reply action video segment synthesis process and entering an interval waiting stage; S3, after the interval waiting stage ends, re-executing S1-S2 until the user exits.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the digital person generation and interaction method of any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the steps of the digital person generation and interaction method of any one of claims 1-8.

Images