Large model driven digital human real-time interaction method

Abstract

The invention discloses a large-model-driven digital human real-time interaction method, which relates to the technical field of digital humans and comprises the following steps: S1, inputting a question text through a man-machine conversation front end and sending the question text to a large-model rear end; s2, the large model rear end performs large model reasoning according to the question text and the knowledge base, generates an answer abstract and a complete answer, and sends the answer abstract and the complete answer to the digital human rear end and the man-machine conversation front end for display; s3, the classifier generates a plurality of classification labels according to the answer abstract; the digital human rear end retrieves and matches a preset video library according to the classification label to obtain a target preset video, and sends the target preset video to the man-machine conversation front end; s4, the digital human processing engine generates a digital human abstract video according to the answer abstract, and pushes the video to the player; and the player sequentially plays the target preset video and the digital human abstract video. And finally, smoother digital human real-time question and answer interaction experience is realized, and distortion experience is reduced.

Description

Technical Field

[0001] The present invention relates to the field of digital human technology, and in particular to a real-time digital human interaction method driven by a large model. Background Art

[0002] Digital humans, also known as virtual digital humans, are digital images close to human images created through computer graphics (CG) and artificial intelligence technologies. Digital virtual humans rely on display devices to exist. They not only visually imitate human appearance, but also use technology to give specific character identity settings to achieve more realistic emotional interaction. In terms of nature, digital humans have the following characteristics: (1) They have human appearance, with specific features such as appearance, gender, and personality; (2) They have human behavior, with the ability to express themselves through language, facial expressions, and body movements; (3) They have human thoughts, with the ability to recognize the external environment and communicate and interact with people.

[0003] In recent years, the booming development of large models has injected new vitality into the development of digital humans. Large model technology provides digital humans with nearly unlimited training parameters and autonomous generation capabilities, enabling them to achieve richer and more complex movements, including facial expressions and body movements. Combined with a rich knowledge base model, digital humans can become experts in any field at any time, providing users with 24-hour, uninterrupted, and extremely realistic professional consulting and services.

[0004] For example, the patent document with publication number CN118377865A provides a real-time question-and-answer method and system for digital humans based on a large model and deep learning. The method includes the following steps: generating silent audio; obtaining user questions; when obtaining the user questions, generating corresponding question-and-answer text from the user questions through the large model, and converting the text into several standard-length question-and-answer audios; when the user questions are not obtained, generating silent audio and recycling it; using a deep model to calculate and render corresponding facial image frames based on the question-and-answer audio, the silent audio, and the corresponding facial image; processing the question-and-answer audio, the silent audio, and the facial image frames, inputting them into corresponding channels, and obtaining a real-time rendered lip-shaped facial video; and using real-time drive technology to push the lip-shaped facial video to the user end. Ultimately, the user can experience the virtual reality product in real time, and with the help of the characteristics of the large model, a more reasonable interactive template can be generated to increase the flexibility of the product.

[0005] However, this approach still has flaws. Specifically, it introduces multiple delays in the various steps leading up to the final digital human output, including the generation of the question-and-answer text, the conversion of the question-and-answer text to audio, the rendering of the facial image based on the audio, and the production of the digital human face video with animated lip movements. The combined delays from these multiple steps prevent the generated digital human from truly achieving "real-time response," making it difficult for users to maintain a high level of immersion and fidelity, and thus easily losing the smoothness and realism of interacting with the digital human. Summary of the Invention

[0006] In response to the problems existing in the existing technology, the present invention provides a large-model driven digital human real-time interaction method to achieve a smoother digital human real-time question-and-answer interaction experience and reduce the distortion experience.

[0007] The technical solution of the present invention is achieved as follows:

[0008] A large-model driven digital human real-time interaction method, comprising a human-computer dialogue front-end, a large-model back-end, a digital human back-end, and a digital human front-end;

[0009] The large model backend is provided with a knowledge base; the digital human backend is provided with a digital human processing engine, a preset video library and a classifier; the preset video library stores a plurality of preset videos, which are pre-recorded digital human videos; the preset videos are associated with at least one classification tag; the classification tags have multiple types; the digital human frontend is provided with a player;

[0010] The method comprises the following steps:

[0011] S1. The user inputs a question text through the human-computer dialogue front end, and the question text is sent to the large model back end;

[0012] S2. The large model backend performs large model reasoning based on the question text and the knowledge base to generate an answer summary and a complete answer; the complete answer is sent to the human-computer dialogue frontend for display; the answer summary is sent to the digital human backend;

[0013] S3, the classifier performs classification label division processing based on the answer summary to generate at least one classification label; the digital human backend searches and matches the preset video library based on the generated classification label to obtain a target preset video; and sends the target preset video to the human-computer dialogue front end;

[0014] The preset videos mainly provide general basic explanations and answers to related questions. These materials cover various common problem categories and scenarios. This category is not fixed and can be associated with different vertical fields.

[0015] S4. The player plays the target preset video; the digital human processing engine generates a digital human summary video according to the answer summary, and streams the digital human summary video to the player; after the target preset video finishes playing, the digital human summary video continues to play.

[0016] The human-computer interaction front-end and the digital human front-end described above do not specifically refer to two independent devices. Instead, they should be understood primarily as interfaces, or different parts of an interactive interface on the same terminal, or other settings with similar logical interactions. Of course, they can also be understood as two independent devices. Similarly, the large model back-end and the digital human back-end can be understood as two independent modules of the same device / cloud platform, or two different devices / cloud platforms.

[0017] Playing pre-set videos can fill the gap between generating the digital human video and generating a complete answer. Users can use the pre-set videos to understand general explanations of related questions or some pre-stored answers related to specific fields. Then, by viewing the digital human summary video, they can gain a comprehensive understanding of the final, complete answer. This ultimately creates a smoother, interactive experience with real-time digital human Q&A, minimizing distortion.

[0018] As a further optimization of the above solution, the preset videos include general videos and vertical field videos; in step S3, the target preset video is one of the general videos, or one of the vertical field videos, or any number of the general videos and any number of the vertical field videos spliced ​​together.

[0019] As a further optimization of the above solution, in step S3, the classifier is a semantic classifier; the digital human processing engine predicts the required generation time of the digital human summary video based on the answer summary; the digital human backend generates the target preset video in the preset video library based on the classification label matching, and the total time of the target preset video is the same as the generation time.

[0020] The generated duration is the same as the total duration of the target preset video. The purpose is to make the playback of the final target preset video and the digital human summary video as seamless as possible, further achieve a smoother digital human real-time question-and-answer interactive experience, and reduce the distortion experience.

[0021] As a further optimization of the above solution, in step S2, the generation of the answer summary and the complete answer are processed in parallel; or the large model backend preferentially generates the answer summary and sends it to the digital human backend, and then generates the complete answer and sends it to the human-computer dialogue frontend.

[0022] The ultimate goal is to prioritize the display of the digital human summary video, so that users can pay attention to and understand the answer summary first, and then browse the complete answer.

[0023] As a further optimization of the above solution, the digital human backend performs text segmentation on the answer summary to obtain multiple script sequence texts; the digital human processing engine includes multiple digital human processing threads; the digital human processing threads correspond to the script sequence texts one by one;

[0024] The digital human processing thread generates a behavior expression video sequence and an audio sequence according to the script sequence text, and performs lip alignment on the obtained behavior expression video sequence and the audio sequence to generate a digital human summary video clip;

[0025] Multiple digital human summary video clips are played in sequence; and multiple digital human processing threads are processed in parallel.

[0026] As a further optimization of the above solution, the digital human backend also stores an emotion dataset and a behavior dataset; the digital human processing engine analyzes the answer summary based on the emotion dataset and the behavior dataset to obtain emotion tags and behavior tags; the digital human processing engine generates the behavior expression video sequence based on the text corresponding to the answer summary, the emotion tags and the behavior tags, and generates the audio sequence based on the text corresponding to the answer summary and the emotion tags.

[0027] As a further optimization of the above solution, the player plays the carousel video when it does not receive the preset video or the digital human summary video.

[0028] The carousel video can be any video. Furthermore, the carousel video displays a digital human video without any movement or sound.

[0029] As a further optimization of the above solution, the human-computer dialogue front end is also provided with a voice input module and a preset question bank;

[0030] The user inputs the question text by typing, or inputting a voice, or selecting a question from the preset question bank as the question text;

[0031] The voice input module converts the input voice into text as the question text.

[0032] As a further optimization of the above solution, in step S4, the digital human backend directly streams the digital human summary video in the memory to the player.

[0033] Pushing the stream from memory can reduce read and write losses and shorten the total time required for users to input questions and see the digital human summary video.

[0034] Compared with the prior art, the present invention achieves the following beneficial effects:

[0035] This invention provides a large-scale model-driven real-time digital human interaction method. By playing pre-set videos, the delay between generating the digital human video and generating a complete answer can be mitigated. Users can use the pre-set videos to understand general explanations of related questions or partially pre-stored answers related to specific vertical fields. Then, by browsing the digital human summary video, they can gain a comprehensive understanding of the final complete answer. This ultimately achieves a smoother real-time digital human question-and-answer interaction experience with less distorted experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a large model-driven digital human real-time interaction method provided by an embodiment of the present invention;

[0037] Figure 2 1 is a flow chart of matching a target preset video according to an embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram of the process of generating a digital human summary video provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0040] like Figures 1 to 3 As shown, this embodiment provides a large model-driven digital human real-time interaction method, including a human-computer dialogue front end, a large model back end, a digital human back end and a digital human front end; in this embodiment, the human-computer dialogue front end and the digital human front end are different display parts of the interactive interface on the same terminal.

[0041] The backend of the large model has a knowledge base;

[0042] The digital human backend is equipped with a digital human processing engine, a preset video library and a classifier, and also stores emotion data sets and behavior data sets; in this embodiment, the classifier is a semantic classifier;

[0043] The preset video library stores multiple preset videos, which are pre-recorded digital human video materials; the preset videos are associated with at least one classification tag; there are multiple types of classification tags;

[0044] The preset videos mainly provide general basic explanations and answers to related questions. These materials cover various common problem categories and scenarios. This category is not fixed and can be associated with different vertical fields.

[0045] In this embodiment, the preset videos include general videos and vertical field videos;

[0046] The digital human front end is equipped with a player. In this embodiment, if the player does not receive a preset video or a digital human summary video, it plays a carousel video. The carousel video can be any video. In this embodiment, the carousel video displays a video of the digital human without any movement or sound.

[0047] The method comprises the following steps:

[0048] S1. The user inputs a question text through the human-computer dialogue front end and sends the question text to the large model back end; in this embodiment, the human-computer dialogue front end is also provided with a voice input module and a preset question bank; the user has multiple input methods.

[0049] The user can input the question text by typing on the keyboard;

[0050] The user can also input voice, and the voice input module will convert the input voice into text as the question text;

[0051] Users can also quickly select questions from the preset question bank as question text.

[0052] S2. The large model backend performs large model inference based on the question text and knowledge base to generate an answer summary and a complete answer; the complete answer is sent to the human-computer dialogue frontend for display; the answer summary is sent to the digital human backend; in this embodiment, the generation process of the answer summary and the complete answer is processed in parallel.

[0053] S3: The classifier performs classification and labeling processing based on the answer summary, generating at least one classification label. In this embodiment, the Digital Human processing engine predicts the required generation time of the Digital Human summary video based on the answer summary. The Digital Human backend then obtains the target preset video from the preset video library based on the classification label provided by the classifier. The total duration of the target preset video is the same as the generation time. In this embodiment, the target preset video is a general video or a vertical video. The target preset video is sent to the human-computer interaction front end.

[0054] The generated duration is the same as the total duration of the target preset video. The purpose is to make the playback of the final target preset video and the digital human summary video as seamless as possible, and to ensure the playback of the digital human summary video as soon as possible.

[0055] S4. The player plays the target preset video; the Digital Human processing engine generates a Digital Human summary video based on the answer summary;

[0056] In this embodiment, the digital human backend performs text segmentation on the answer summary to obtain multiple script sequence texts; the digital human processing engine includes multiple digital human processing threads; the digital human processing threads correspond to the script sequence texts one by one;

[0057] Each digital human processing thread analyzes the script sequence text based on the emotion dataset and behavior dataset to obtain emotion labels and behavior labels; the digital human processing thread generates a behavior expression video sequence based on the script sequence text, emotion labels and behavior labels, and generates an audio sequence based on the script sequence text and emotion labels, and lip-aligns the obtained behavior expression video sequence and audio sequence to generate a digital human summary video clip.

[0058] Multiple digital human processing threads are processed in parallel to obtain multiple digital human summary video clips.

[0059] The Digital Human processing engine streams the Digital Human summary video clip to the player. In this embodiment, the Digital Human backend directly streams the Digital Human summary video clip from memory to the player. Streaming from memory reduces read and write overhead, shortening the total time it takes for the user to enter a question and see the Digital Human summary video.

[0060] After the target preset video is played, the digital human summary video clips are played in sequence, that is, the digital human summary video is played.

[0061] Playing pre-set videos can fill the gap between generating the digital human video and generating a complete answer. Users can use the pre-set videos to understand general explanations of related questions or some pre-stored answers related to specific fields. Then, by viewing the digital human summary video, they can gain a comprehensive understanding of the final, complete answer. This ultimately creates a smoother, interactive experience with real-time digital human Q&A, minimizing distortion.

[0062] Based on the disclosure and teachings of the above description, those skilled in the art may also make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and modifications and variations of the present invention should also fall within the scope of protection of the claims of the present invention. In addition, although certain specific terms are used in this description, these terms are only for convenience of description and do not constitute any limitation to the present invention.

Claims

1. A real-time digital human interaction method driven by a large model, characterized in that: Including human-computer dialogue front end, large model back end, digital human back end and digital human front end; The large model backend is provided with a knowledge base; the digital human backend is provided with a digital human processing engine, a preset video library and a classifier; the preset video library stores a plurality of preset videos, and the preset videos are pre-recorded digital human videos; the preset videos are associated with at least one classification tag; the classification tags have multiple types; the digital human frontend is provided with a player; The method comprises the following steps: S1, the user sends a question text to the large model back end through the human-computer dialogue front end; S2, the large model backend performs large model reasoning based on the question text and the knowledge base to generate an answer summary and a complete answer; the complete answer is sent to the human-computer dialogue frontend for display; the answer summary is sent to the digital human backend; S3, the classifier performs classification label division processing according to the answer summary to generate at least one classification label; the digital human backend searches and matches the preset video library according to the generated classification label to obtain a target preset video; and sends the target preset video to the human-computer dialogue front end; S4, the digital human processing engine generates a digital human summary video according to the answer summary, and streams the digital human summary video to the player; The player plays the target preset video and the digital human summary video in sequence.

2. A large model driven digital human real-time interaction method according to claim 1, characterized in that: The preset videos include general videos and vertical field videos; in step S3, the target preset video is one of the general videos, or one of the vertical field videos, or any number of the general videos and any number of the vertical field videos spliced ​​together.

3. The method for real-time digital human interaction driven by a large model according to claim 1, characterized in that: In step S3, the classifier is a semantic classifier; the digital human processing engine predicts the required generation time of the digital human summary video according to the answer summary; the digital human backend generates the target preset video in the preset video library according to the classification label matching, and the total time of the target preset video is the same as the generation time.

4. The method for real-time digital human interaction driven by a large model according to claim 1, characterized in that: In step S2, the generation of the answer summary and the complete answer is processed in parallel; or the large model backend generates the answer summary first and sends it to the digital human backend, and then generates the complete answer and sends it to the human-computer dialogue frontend.

5. The method for real-time digital human interaction driven by a large model according to claim 1, characterized in that: The digital human backend performs text segmentation on the answer summary to obtain multiple script sequence texts; the digital human processing engine includes multiple digital human processing threads; the digital human processing threads correspond to the script sequence texts one by one; The digital human processing thread generates a behavior expression video sequence and an audio sequence according to the script sequence text, and performs lip alignment on the obtained behavior expression video sequence and the audio sequence to generate a digital human summary video clip; The multiple digital human summary video clips are played in sequence; and the multiple digital human processing threads are processed in parallel.

Images