A multilingual 3D digital human interaction method using GPT

Abstract

The application discloses a kind of multilingual 3D digital human interaction methods using GPT, comprising: 1) creating digital human unreal engine;2) parse the multimodal problem content of voice, text, picture, document input by user;3) build multimodal content review model and review the above multimodal problem;4) the user input of content review normal calls GPT streaming response to obtain question answer;5) question answer (text) is intelligently divided into sentences, and sentence queue is obtained;6) each sentence in text queue is called TTS node to generate voice in asynchronous mode, and voice queue is established;7) the voice queue of last step is executed asynchronously to generate lip sync animation, and animation queue is obtained;8) play voice and animation in sequence synchronously, and real-time presentation interaction effect.Compared with prior art, the application supports multilingual and multi-model selection, compatible with multiple modal inputs such as picture, voice and document, and realizes multimodal content review mechanism.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of generative artificial intelligence, and particularly relates to a multilingual 3D digital human interaction method using GPT. BACKGROUND

[0002] With the rapid development of generative artificial intelligence technology, digital humans, as a kind of virtual character, are widely used in virtual assistants, online education, customer service, virtual anchors and other fields. Traditional digital human interaction systems usually rely on pre-set dialogue scripts and simple speech synthesis technology, resulting in a lack of flexibility and intelligence in the interaction process, making it difficult to meet the needs of personalized and real-time dialogue of users. In addition, such systems usually only support a single language or a small number of languages, limiting their applicability in landing application scenarios. At the same time, digital human systems must strengthen content review and data protection to ensure that the system meets national security standards, prevent the spread of harmful content and protect user privacy and data security.

[0003] After searching, patent CN117539425A discloses a method and system for enhancing the dialogue effect of MetaHuman digital humans using ChatGPT and Unreal Engine, mainly including the following steps: 1, creating a MetaHuman digital human model and importing it into Unreal Engine, adjusting expressions and actions through animation blueprints; 2, receiving multi-modal input through the user interface, generating text answers using ChatGPT, and converting them into speech files; 3, playing the speech using the URL address parsed by the VaRest plugin, and synchronizing the digital human's lip movements through the lip animation blueprint. The invention has limited language support and cannot be used in a wide range of landing scenarios; the invention only intelligently divides sentences at the GPT response text, and the subsequent text-to-speech and animation generation steps do not mention the use of an asynchronous mechanism; the invention lacks checks on the legality of user input content, and cannot timely identify and handle harmful operations by malicious users. This not only increases the risk of harmful content spreading in the system, but also reduces the safety and credibility of user experience. SUMMARY

[0004] The present application aims to provide a multilingual 3D digital human interaction method using GPT, which combines the natural language processing capabilities of GPT and the MetaHuman technology of Unreal Engine, enabling more flexible and personalized real-time human-computer dialogue. The present application integrates advanced content review mechanisms to ensure the legality and safety of user input multi-modal content such as speech, text and pictures, preventing the spread of harmful information. The system supports multiple languages and dialects, and can perform synchronous speech synthesis and emotional expression through the cloud, adapting to global application needs. By using an asynchronous mechanism, the response speed of the system is significantly improved, ensuring a good user experience, to solve the technical problems mentioned in the background technology.

[0005] To solve the above technical problems, the specific technical solutions of the present application are as follows:

[0006] A multilingual 3D digital human interaction method using GPT, characterized by the following steps:

[0007] Step 1, create a MetaHuman 3D digital human Unreal Engine project, set default actions and expressions, build interaction interface controls, and provide multilingual and multi-model options;

[0008] Step 2, the user inputs questions in the form of text, pictures, voice and documents through the interaction interface, the multilingual 3D digital human interaction system analyzes the user's input multi-modal question content including pictures, voice and documents, uploads the input pictures to the cloud, generates picture URLs, converts the input voice into text through the voice-to-text node of Microsoft cloud computing service, and parses the input documents into text through the file parsing service of the cloud;

[0009] Step 3, build a multi-modal content review model for text and pictures, review the above input question content, if the review result shows that the content is harmful, return the violation category and warning information, then end the topic and require the user to use it properly;

[0010] Step 4, if the content review is normal, the above picture and text content marked as harmless are called OpenAI interface plug-in chat nodes, through one of the Hypertext Transfer Protocol (HTTP) interfaces Restful API, the GPT's streaming output is obtained as the answer to the question;

[0011] Step 5, GPT intelligently divides the output answer text into sentences, and adds the divided sentences to the text queue;

[0012] Step 6, asynchronously call the TTS node to generate voice for each sentence in the text queue, get the voice URL corresponding to each sentence, and build a voice queue;

[0013] Step 7, input the voice queue into a lip synchronization animation node to asynchronously generate lip animation sequences;

[0014] Step 8, play the digital human lip animation and voice simultaneously in queue order, and present the digital human interaction effect in real time.

[0015] Further, the interaction interface controls include gender, language, and voice switching functions, support various Chinese dialects (Suzhou dialect, Northeastern dialect, Minnan dialect, etc.), English (British, American), can freely switch between large models, can also have multi-round conversations, historical conversations are preserved, and supports voice, document, and text input.

[0016] Further, the multilingual 3D digital human interaction system analyzes user input, including the following steps:

[0017] A1. When the user provides voice input, perform voice-to-text conversion in the level blueprint after the input is completed.

[0018] A2. When the user provides document input, use a cloud-deployed open-source file parsing service to parse the file and obtain the document content.

[0019] A3. When the user provides picture input, upload the picture to Amazon S3 and generate a picture URL.

[0020] Further, the constructed multi-modal content review model classifies and predicts the probability of user input content belonging to various harmful categories through cross-language and multi-modal semantic analysis. If the probability is higher than the set threshold, the content is determined to belong to the corresponding category.

[0021] Further, the normal user input using the content review calls the chat node in the OpenAI interface plugin to obtain the streaming output of GPT in real time, and performs intelligent sentence processing based on semantics; each sentence will be added to an asynchronous task queue to ensure the order and coherence of the conversation content; the system simultaneously calls the TTS node for each sentence in the text queue to generate a natural speech URL for each sentence, and adds the generated speech resources to a speech playback queue in order to achieve smooth speech output.

[0022] The multilingual 3D digital human interaction method using GPT has the following advantages:

[0023] 1. The present application integrates a multi-modal content review mechanism to ensure that the system can effectively filter and prevent the spread of harmful information, greatly improving the security and compliance of the digital human interaction system. This not only meets the current network security regulations, but also further enhances user trust and use experience.

[0024] 2. The system supports multilingual and multi-dialect interaction, covering Chinese dialects, English, and other commonly used languages, greatly expanding its application scenarios. The system can not only be applied to the domestic market, but also meet global demand, especially in multinational enterprises or education, entertainment, and other fields that require multilingual support.

[0025] 3. Through the combination of natural language processing capabilities of GPT and MetaHuman technology, the system can achieve efficient and natural real-time dialogue. Moreover, by using streaming output and intelligent sentence splitting technology, the system ensures the sequentiality and coherence of the dialogue content, providing a highly immersive user experience and further enhancing the interactive performance and entertainment of digital humans. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is the overall flowchart of the present invention patent;

[0027] Figure 2 is the multi-modal input processing flowchart of the present invention patent;

[0028] Figure 3 is the content review flowchart of the present invention patent;

[0029] Figure 4 is the asynchronous calling method flowchart of the present invention patent. DETAILED DESCRIPTION

[0030] In order to better understand the purpose, structure and function of the present invention, the following will combine the drawings to further describe the multi-language 3D digital human interaction method using GPT in detail. EMBODIMENT

[0031] As shown in Figure 1 , a multi-language 3D digital human interaction method using GPT includes the following steps:

[0032] Step 1: Create a 3D digital human model through MetaHuman Creator, import the MetahumanGPT into the Unreal Engine project, adjust the facial expressions through the Control Rig, create actions using Animation Blueprint, and loop play this set of actions and expressions as the default idle animation;

[0033] Step 2, add a widget blueprint in MetahumanGPT, which is an interactive interface supporting voice, GPT model switching, accepting user multi-modal question input and voice, model option setting, and returning the result to the level blueprint as the main program. The interactive interface includes multi-language (English and Chinese local dialects) and GPT model option box, text input box, file upload button, voice input button, history session list, message box, and send button. The language option box provides gender, language, and voice options, which can quickly filter the required voice through gender and voice options. The GPT model option box provides multiple large model options. Pictures and files are uploaded through the file upload button. The voice input button is a microphone icon that automatically opens the microphone on the user's device for recording when clicked. When the user pauses for more than 5 seconds, the recording stops. The backend automatically parses the user input content format. The history session list displays historical conversation records, and each session title is automatically summarized by GPT according to the topic content;

[0034] Step 3, build a multi-modal content review model to review the user's sent question, check if there is harmful content, if no harmful content is detected, proceed to the next step, otherwise the front end returns a warning message;

[0035] Step 4, if the content review is normal, mark the above picture and text content as harmless, call the chat node of the OpenAI interface plugin to get the GPT streaming response;

[0036] Step 5, split the output text by sentence and add the split sentences to the text queue;

[0037] Step 6, asynchronously call the TTS node for each sentence in the text queue to generate voice, get the voice URL corresponding to each sentence, and establish a voice queue;

[0038] Step 7, input the voice queue into the lip sync animation node of Metahuman SDK to asynchronously generate lip animation sequences;

[0039] Step 8, the front end plays the digital lip animation and voice simultaneously according to the queue order, and presents the digital human interaction effect in real time.

[0040] Based on the multi-language 3D digital human interaction method using GPT in embodiment 1, the following further explains and describes the present application:

[0041] Step 1, build a multi-modal content review model to review the user's sent question, check if there is harmful content, if no harmful content is detected, proceed to the next step, otherwise the front end returns a warning message;

[0042] For example, Figure 2As shown, the multi-modal input in step 2 includes: text, voice, picture, document:

[0043] A1. When the user provides voice input, perform voice-to-text conversion in the level blueprint after the input is completed;

[0044] A2. When the user provides document input, use a cloud-deployed open-source file parsing service to parse the file and obtain the document content;

[0045] A3. When the user provides picture input, upload the picture to Amazon S3 and generate a picture URL.

[0046] The above method processes each type of input in a specific way, expanding the scope of application of the invention and improving its practicality and diversity.

[0047] As shown in Figure 3 , the content review step in step 3 is as follows:

[0048] Receive the user's multi-modal input content processed in step 2;

[0049] By using deep learning-based natural language processing techniques, text and images are converted into feature vectors that capture the semantic information of the text and images. The system evaluates whether the input content belongs to a certain harmful content category based on these feature vectors through a multi-modal review model. The model finally outputs the probability distribution of each harmful category, indicating the likelihood of the input content belonging to a potential harmful category. If the probability is higher than the set threshold, the content will be marked.

[0050] If it contains harmful information, display the harmful content category and warning information.

[0051] If the problem is not marked, proceed to step 4.

[0052] Intelligently split the input text by calling the OpenAI interface plugin chat node to obtain GPT streaming responses;

[0053] Save the voice file URL queue through asynchronous speech synthesis;

[0054] Asynchronously generate lip-sync animation and save the animation queue;

[0055] Play the animation and voice on the front end.

[0056] Finally, as shown in Figure 4 , in step 7, the voice queue is asynchronously input into a lip-sync animation node to generate an animation sequence, as follows:

[0057] Split the text content by sentence to obtain the text queue;

[0058] Asynchronous call TTS node, convert each sentence in the text queue into speech;

[0059] Wait for the response result and determine whether the text queue has been completely converted;

[0060] If the conversion is complete, proceed to the next step of generating lip-sync animation;

[0061] Match the generated speech and lip animation, play the speech and lip animation simultaneously according to the queue order through the animation control node of Metahuman SDK, and achieve smooth digital human conversation effect.

[0062] In summary, the present application has the following advantages: 1. Multi-language support: The system supports multiple languages and dialects, and can be applied globally, especially suitable for real-time interaction in multi-cultural and multi-language scenarios. 2. Multi-modal input: Support voice, text, picture and other multi-modal input, adapt to various user interaction modes, enhance the flexibility and practicality of the system. 3. Real-time intelligent conversation: By combining the natural language processing ability of GPT, the system can realize high-intelligent real-time conversation, improving the user's individualization and interactive experience. 4. Content security: Integrated multi-modal content review mechanism, can effectively filter and prevent the spread of harmful information, protect user data privacy and system security. 5. Natural and smooth lip synchronization: By asynchronously calling TTS nodes and Metahuman SDK animation nodes, the system can generate highly matched lip animation with speech, providing realistic digital human conversation experience.

[0063] It can be understood that the present application is described through some embodiments, and those skilled in the art know that various changes or equivalent replacements can be made to these features and embodiments without departing from the spirit and scope of the present application. In addition, under the guidance of the present application, these features and embodiments can be modified to adapt to specific conditions and materials without departing from the spirit and scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application are within the scope of the present application.

Claims

1. A multilingual 3D digital human interaction method using GPT, characterized in that, Comprising the following steps: Step 1, create a MetaHuman 3D digital human Unreal Engine project, set default actions and expressions, build interactive interface controls, provide multi-language and multi-model options; Step 2, users input questions in the form of text, pictures, voice and documents through the interactive interface, the multi-language 3D digital human interaction system analyzes the user's input multi-modal question content including pictures, voice and documents, uploads the input pictures to the cloud, generates picture URLs, converts the input voice into text through the Microsoft cloud computing service's voice-to-text node, and parses the input documents into text through the cloud file parsing service; Step 3, build a multi-modal content review model for text and pictures, review the above input question content, if the review result shows that the content is harmful, return the violation category and warning information, then end the topic and require the user to use it properly; Step 4, if the content review is normal, the above picture and text content marked as harmless are called OpenAI interface plugin chat nodes, the GPT's streaming output is obtained through a Hypertext Transfer Protocol (HTTP) based interface Restful API, and the output is used as the answer to the question; Step 5, GPT intelligently divides the output answer text into sentences, and adds the divided sentences to the text queue; Step 6, asynchronously call the TTS node to generate voice for each sentence in the text queue, get the voice URL corresponding to each sentence, and build a voice queue; Step 7, input the voice queue into a lip sync animation node to asynchronously generate lip sync animation sequences; Step 8, play the digital human lip sync animation and voice simultaneously in queue order to present the digital human interaction effect in real time.

2. The multi-language 3D digital human interaction method with GPT of claim 1, wherein, The interactive interface controls in step 1 include gender, language, voice, and model switching functions, support conversation recording and saving, and support voice, document, and text input.

3. The multi-language 3D digital human interaction method with GPT of claim 1, wherein, The multi-language 3D digital human interaction system in step 2 analyzes user multi-modal input as follows: A1, when the user provides voice input, perform voice-to-text conversion in the input end; A2, when the user provides document input, use the open source file parsing service deployed on the cloud to parse the file and obtain the document content; A3, when the user provides picture input, upload the picture to Amazon S3 and generate a picture URL.

4. The multi-language 3D digital human interaction method with GPT of claim 1, wherein, The multi-modal content review model built in step 3 performs cross-language and multi-modal semantic analysis on the input text or picture, classifies the input text or picture, and predicts the probability that the user input content belongs to various harmful categories. If the probability is higher than the set threshold, the content is determined to belong to the corresponding category.

Images