Multi-language 3D digital human interaction method adopting GPT

Abstract

The invention discloses a multi-language 3D digital human interaction method adopting GPT. The method comprises the following steps: 1) creating a digital human unreal engine; 2) analyzing multi-modal problem contents of voice, texts, pictures and documents input by the user; 3) constructing a multi-modal content review model to review the multi-modal problem; 4) calling GPT streaming response to obtain question answers according to the user input with normal content review; 5) performing intelligent sentence segmentation on question answers (texts) to obtain sentence queues; 6) calling TTS nodes simultaneously for each sentence in the text queue in an asynchronous mode to generate voice, and establishing a voice queue; 7) asynchronously executing mouth shape animation generation on the voice queue in the previous step to obtain an animation queue; and 8) synchronously playing the voice and the animation in sequence, and presenting the interaction effect in real time. Compared with the prior art, the method supports multi-language and multi-model selection, is compatible with multi-modal input of pictures, voices, documents and the like, and realizes a multi-modal content review mechanism.

Description

Technical Field

[0001] The present invention belongs to the field of generative artificial intelligence, and in particular relates to a multi-language 3D digital human interaction method using GPT. Background Art

[0002] With the rapid development of generative artificial intelligence technology, digital humans, as a kind of virtual character image, are widely used in many fields such as virtual assistants, online education, customer service, virtual anchors, etc. Traditional digital human interaction systems usually rely on pre-set dialogue scripts and simple speech synthesis technology, resulting in a lack of flexibility and intelligence in the interaction process, and it is difficult to meet the needs of users for personalized and real-time dialogue. In addition, such systems usually only support a single language or a small number of languages, which limits their applicability in landing application scenarios. At the same time, with the deepening of China's cybersecurity strategy, especially under the framework of the "Cybersecurity Law" and the "Interim Measures for the Management of Generative Artificial Intelligence Services", digital human systems must strengthen content review and data protection to ensure that the system can meet national security standards, prevent the spread of harmful content and protect user privacy and data security.

[0003] After searching, patent CN117539425A discloses a method and system for enhancing the dialogue effect of MetaHuman digital human by using ChatGPT and Unreal Engine, which mainly includes the following steps: 1. Create a MetaHuman digital human model and import it into Unreal Engine, and adjust expressions and actions through animation blueprints; 2. Receive multimodal input through the user interface, generate text answers using ChatGPT, and convert them into voice files; 3. Play the voice using the URL address parsed by the VaRest plug-in, and synchronize the lip movements of the digital human through the lip animation blueprint. The invention has limited language support and cannot be used in a wide range of landing scenarios; the invention only performs intelligent sentence segmentation at the GPT response text, and the subsequent text-to-speech and animation generation steps do not mention the use of asynchronous mechanisms; the invention has deficiencies in the legitimacy check of user input content, and cannot promptly identify and handle harmful operations of malicious users. This not only increases the risk of the system facing the spread of harmful content, but also reduces the security and credibility of the user experience. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-language 3D digital human interaction method using GPT, which combines the natural language processing capabilities of GPT with the MetaHuman technology of Unreal Engine, and can achieve more flexible and personalized real-time human-computer dialogue. The present invention integrates an advanced content review mechanism to ensure the legitimacy and security of multi-modal content such as voice, text, and pictures input by users, and prevent the spread of bad information. The system supports multiple languages ​​and multiple dialects, and can synchronize speech synthesis and emotional expression through the cloud to meet the needs of global applications. By adopting an asynchronous mechanism, the response speed of the system is significantly improved, ensuring a good user experience, so as to solve the technical problems mentioned in the background technology.

[0005] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows:

[0006] A multi-language 3D digital human interaction method using GPT, characterized by comprising the following steps:

[0007] Step 1: Create a MetaHuman 3D digital human Unreal Engine project, set default actions and expressions, build interactive interface controls, and provide multi-language and multi-model options;

[0008] Step 2: The user inputs questions in the form of text, pictures, voice and documents through the interactive interface. The multi-language 3D digital human interactive system parses the multi-modal question content including pictures, voice and documents input by the user, uploads the input pictures to the cloud, generates picture URLs, converts the input voice into text through the speech-to-text node of the Microsoft cloud computing service, and parses the input document into text through the file parsing service in the cloud;

[0009] Step 3: Build a multimodal content review model for text and images to review the input question content. If the review result shows that the content is harmful, return the violation category and warning information, then end the topic and require the user to use it in a standardized manner.

[0010] Step 4: If the content review is normal, the above-mentioned pictures and text contents marked as harmless are called the OpenAI interface plug-in chat node, and a RESTful API based on the hypertext transfer protocol HTTP is used to obtain the streaming output of GPT as the answer to the question;

[0011] Step 5: GPT intelligently divides the output answer text into sentences and adds the divided sentences to the text queue;

[0012] Step 6: Asynchronously call the TTS node to generate voice for each sentence in the text queue, obtain the voice URL corresponding to each sentence, and establish a voice queue;

[0013] Step 7: Input the voice queue into a lip sync animation node to asynchronously generate a lip animation sequence;

[0014] Step 8: Play the digital human's lip animation and voice simultaneously in the queue order to present the digital human's interactive effect in real time.

[0015] Furthermore, the interactive interface controls include gender, language, and voice switching functions, support various Chinese dialects (Suzhou dialect, Northeastern dialect, Minnan dialect, etc.), English (British, American), can freely switch large models, and can also conduct multiple rounds of conversations, historical conversations are retained, and support voice, document, and text input.

[0016] Furthermore, the multi-language 3D digital human interaction system analyzes the user input including the following steps:

[0017] A1. When the user provides voice input, voice-to-text conversion is performed in the level blueprint after the input is completed;

[0018] A2. When the user provides document input, the open source file parsing service deployed in the cloud is used to parse the file and obtain the document content;

[0019] A3. When the user provides image input, the image is uploaded to Amazon S3 and the image URL is generated.

[0020] Furthermore, the constructed multimodal content review model classifies the input text or image through cross-language and multimodal semantic analysis, and predicts the probability that the user input content belongs to various harmful categories. If the probability is higher than the set threshold, the content is determined to belong to the corresponding category.

[0021] Furthermore, the chat node in the OpenAI interface plug-in is called using user input with normal content review to obtain the streaming output of GPT in real time, and perform intelligent sentence segmentation based on semantics; each sentence will be added to the asynchronous task queue to ensure the order and coherence of the conversation content; the system calls the TTS node for each sentence in the text queue at the same time to generate a natural speech URL for each sentence, and adds the generated voice resources to the voice playback queue in an orderly manner to achieve smooth voice output.

[0022] The multi-language 3D digital human interaction method using GPT of the present invention has the following advantages:

[0023] 1. This invention integrates a multimodal content review mechanism to ensure that the system can effectively filter and prevent the spread of harmful information such as hatred, violence, pornography, etc., greatly improving the security and compliance of the digital human interaction system. This not only meets the requirements of current network security regulations, but also further enhances the user's trust and usage experience.

[0024] 2. The system supports multi-language and multi-dialect interaction, including Chinese dialects, English and other commonly used languages, which greatly expands its application scenarios. The system can not only be applied to the domestic market, but also meet global needs, especially in multinational companies or education, entertainment and other fields that require multi-language support.

[0025] 3. By combining GPT's natural language processing capabilities with MetaHuman technology, the system can achieve efficient and natural real-time conversations. In addition, the use of streaming output and intelligent sentence segmentation technology ensures the order and coherence of the conversation content, provides a highly immersive user experience, and further enhances the interactive expressiveness and entertainment of digital humans. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the overall process of the patent of this invention;

[0027] Figure 2 It is a schematic diagram of the multi-modal input processing flow of the patent of the present invention;

[0028] Figure 3 It is a schematic diagram of the content review process of the patent of this invention;

[0029] Figure 4 This is a flowchart of the asynchronous calling method of the present invention. DETAILED DESCRIPTION

[0030] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a multi-language 3D digital human interaction method using GPT of the present invention in conjunction with the accompanying drawings.

[0031] Embodiment 1:

[0032] like Figure 1 As shown, a multi-language 3D digital human interaction method using GPT includes the following steps:

[0033] Step 1. Create a 3D digital human model through MetaHuman Creator, import the Unreal Engine project MetahumanGPT, adjust the facial expression through the control device (Control Rig), use the animation blueprint (AnimationBlueprint) to create actions, and loop this set of actions and expressions as the default idle animation;

[0034] Step 2. Add a widget blueprint to MetahumanGPT. This blueprint is an interactive interface that supports voice and large language model switching, accepts user multimodal question input and voice and model option settings, and returns the results to the level blueprint (Level Blueprint) as the main program. The interactive interface includes multiple languages ​​(English and various Chinese dialects) and GPT model option boxes, text input boxes, file upload buttons, voice input buttons, historical conversation lists, message boxes, and send buttons. The language option box provides gender, language, and voice option boxes, and you can quickly filter out the required voices by gender and voice options. The GPT model option box provides a variety of large model options. Images and files are uploaded through the file upload button. The voice input button is a microphone icon. Clicking it automatically turns on the microphone on the user's device for recording, and stops recording when the user pauses for more than 5 seconds. The backend will automatically parse according to the format of the user input content. The historical conversation list will display historical conversation records, and each conversation title is automatically summarized by GPT based on the topic content;

[0035] Step 3: Build a multimodal content review model to review the questions sent by users and check whether there is harmful content. If no harmful content is detected, proceed to the next step, otherwise the front end returns a warning message;

[0036] Step 4: If the content review is normal, the above-mentioned pictures and text content marked as harmless are called to obtain the GPT streaming response by the chat node of the OpenAI interface plug-in;

[0037] Step 5: Segment the output text into sentences and add the segmented sentences into the text queue;

[0038] Step 6: Asynchronously call the TTS node to generate voice for each sentence in the text queue, obtain the voice URL corresponding to each sentence, and establish a voice queue;

[0039] Step 7: Input the voice queue into the Metahuman SDK lip sync animation node to asynchronously generate a lip animation sequence.

[0040] Step 8: The front end plays the digital human animation and voice simultaneously in the queue order, presenting the digital human interaction effect in real time.

[0041] Embodiment 2:

[0042] Based on the multi-language 3D digital human interaction method using GPT in Example 1, the present invention is further described and elaborated as follows:

[0043] like Figure 2 As shown, the multimodal input in step 2 includes: text, voice, picture, document:

[0044] A1. When the user provides voice input, voice-to-text conversion is performed in the level blueprint after the input is completed;

[0045] A2. When the user provides document input, the open source file parsing service deployed in the cloud is used to parse the file and obtain the document content;

[0046] A3. When the user provides image input, the image is uploaded to Amazon S3 and the image URL is generated.

[0047] The above method processes each input in a specific manner, which expands the application scope of the present invention and improves its practicality and diversity.

[0048] like Figure 3 As shown, the content review steps in step 3 are as follows:

[0049] Receive the user multimodal input content processed in step 2;

[0050] By using natural language processing technology based on deep learning, text and images are converted into feature vectors that can capture the semantic information of text and images. The system uses multimodal audit model reasoning to evaluate whether it belongs to a certain category of harmful content based on these feature vectors. The model ultimately outputs a probability distribution for each harmful category, indicating the possibility that the input content belongs to a potentially harmful category. If the probability is higher than the set threshold, the content will be marked.

[0051] If harmful information is contained, the harmful content category and warning message will be displayed.

[0052] If the question is not marked, proceed to step 4.

[0053] Get the GPT streaming response by calling the OpenAI interface plug-in chat node and intelligently segment the input text;

[0054] Through asynchronous speech synthesis, save the voice file URL queue;

[0055] Generate lip-sync animation asynchronously and save the animation queue;

[0056] The front end plays animation and voice.

[0057] Finally, if Figure 4 As shown, in step 7, the voice queue is asynchronously input into a lip sync animation node, and the steps for generating the animation sequence are as follows:

[0058] Split the text content into sentences and obtain the text queue;

[0059] Asynchronously call the TTS node to convert each sentence in the text queue into speech;

[0060] Wait for the response result and determine whether all text queues have been converted;

[0061] If the conversion is complete, proceed to the next step to generate lip sync animation;

[0062] The generated speech and lip animation are matched, and the animation control node of Metahuman SDK is used to play the speech and lip animation simultaneously in the queue order to achieve a smooth digital human conversation effect.

[0063] In summary, the present invention has the following advantages: 1. Multi-language support: The system supports multiple languages ​​and multiple dialects, can be used globally, and is particularly suitable for real-time interaction in multi-cultural and multi-language scenarios. 2. Multimodal input: Supports multimodal input such as voice, text, and pictures, adapts to a variety of user interaction methods, and enhances the flexibility and practicality of the system. 3. Real-time intelligent dialogue: By combining the natural language processing capabilities of GPT, the system can achieve highly intelligent real-time dialogue, improving the user's personalization and interactive experience. 4. Content security: It integrates a multimodal content review mechanism, which can effectively filter and prevent the spread of bad information, meets the requirements of China's network security strategy, and protects user data privacy and system security. 5. Natural and smooth lip synchronization: By asynchronously calling TTS nodes and Metahuman SDK animation nodes, the system can generate lip animations that are highly matched with voice, providing a realistic digital human dialogue experience.

[0064] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A multi-language 3D digital human interaction method using GPT, characterized in that: The following steps are involved: Step 1: Create a MetaHuman 3D digital human Unreal Engine project, set default actions and expressions, build interactive interface controls, and provide multi-language and multi-model options; Step 2: The user inputs questions in the form of text, pictures, voice and documents through the interactive interface. The multi-language 3D digital human interactive system parses the multi-modal question content including pictures, voice and documents input by the user, uploads the input pictures to the cloud, generates picture URLs, converts the input voice into text through the speech-to-text node of the Microsoft cloud computing service, and parses the input document into text through the file parsing service in the cloud; Step 3: Build a multimodal content review model for text and images to review the input question content. If the review result shows that the content is harmful, return the violation category and warning information, then end the topic and require the user to use it in a standardized manner. Step 4: If the content review is normal, the above-mentioned pictures and text contents marked as harmless are used to call the OpenAI interface plug-in chat node to obtain the streaming output of GPT through a RestfulAPI interface based on the Hypertext Transfer Protocol HTTP as the answer to the question; Step 5: GPT intelligently divides the output answer text into sentences and adds the divided sentences to the text queue; Step 6: Asynchronously call the TTS node to generate voice for each sentence in the text queue, obtain the voice URL corresponding to each sentence, and establish a voice queue; Step 7: Input the voice queue into a lip sync animation node to asynchronously generate a lip animation sequence; Step 8: Play the digital human's lip animation and voice simultaneously in the queue order to present the digital human's interactive effect in real time.

2. The multi-language 3D digital human interaction method using GPT according to claim 1, characterized in that: The interactive interface controls in step 1 include gender, language, voice, and model switching functions, support session record saving, and support voice, document, and text input.

3. The multi-language 3D digital human interaction method using GPT according to claim 1, characterized in that: In step 2, the multi-language 3D digital human interaction system analyzes the user's multi-modal input, including the following steps: A1. When the user provides voice input, voice-to-text conversion is performed in the level blueprint after the input is completed; A2. When the user provides document input, the open source file parsing service deployed in the cloud is used to parse the file and obtain the document content; A3. When the user provides image input, the image is uploaded to Amazon S3 and the image URL is generated.

4. The multi-language 3D digital human interaction method using GPT according to claim 1, characterized in that: The multimodal content review model constructed in step 3 classifies the input text or image through cross-language and multimodal semantic analysis, and predicts the probability that the user input content belongs to various harmful categories. If the probability is higher than the set threshold, the content is determined to belong to the corresponding category.

Images