Virtual digital human interaction method and device based on large language model

Abstract

The application relates to an interactive device, system and method of a virtual digital person, wherein the method comprises the following steps: acquiring environment information and user information based on a multi-modal sensor of the virtual digital person; the virtual digital person interacts with the user based on a large language prompt word engineering; and the virtual digital person is dressed based on a preset character setting of the large language model prompt word engineering. The application can realize real-time natural language communication between the virtual person and the user in a specific environment, and can realize the technical effect of improving the fusion of the virtual digital person and the real scene by cooperating with the facial expression, action and professional dress of the virtual digital person.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence applications, and specifically relates to a virtual digital human interaction method and device based on a large language model. Background Technology

[0002] 3D virtual human IPs created through motion capture are becoming increasingly attractive and lifelike. However, these 3D virtual human IPs are built upon the individual abilities of the person behind them. Current technologies only allow large language models to input and output text, not to engage in dialogue-style communication between people, which limits their application in on-site navigation environments. Summary of the Invention

[0003] This invention provides a virtual digital human interaction method and apparatus based on a large language model, aiming to solve at least one of the technical problems existing in the prior art.

[0004] The technical solution of this invention relates to a virtual digital human interaction method based on a large language model, the method comprising:

[0005] S200, a multimodal sensor based on virtual digital human, acquires environmental and user information;

[0006] S300, based on the large language prompt word project, enables virtual digital humans to interact with users;

[0007] Step S300 includes the following steps;

[0008] S310. After entering the working state, the system receives the user's image and voice through the multimodal sensor, and transmits the voice and image to the voice and image extraction network to extract the voice and image.

[0009] In step S310, the speech-image extraction network includes: a speech separation network and a visual separation network.

[0010] The speech separation network includes a hybrid speech receiving module, an STFT speech time-frequency transformation module, a speech feature extraction module, a speech upsampling module, and an ISTFT frequency-time transformation module connected in sequence.

[0011] The visual separation network includes a hybrid visual receiving module, a VGGFace feature extraction module, and a visual feature network module connected in sequence.

[0012] It also includes a fusion module, the input of which is connected to the speech feature extraction module and the VGG feature extraction module respectively, and the output of which is connected to the speech upsampling module;

[0013] S320, based on speech-to-text service, converts speech into text;

[0014] S330, based on a large language model service, outputs language text;

[0015] S340. Based on the text-to-speech service, convert the language text output by the large language model service into speech;

[0016] S350, based on a virtual human animation control program, outputs sound, facial expressions, and movements;

[0017] S360. If a response is received from the user, repeat steps S310 to S350 to provide feedback on the received content, outputting voice, facial expressions, and actions.

[0018] S400, based on the large language model, prompt word engineering, adjusts the character settings of virtual digital humans.

[0019] Furthermore, in step S200,

[0020] The multimodal sensor of the virtual digital human includes at least a depth sensor, an RGB camera, and a microphone. The multimodal sensor is used for at least facial recognition, skeletal recognition, posture, and gesture recognition of the human.

[0021] Furthermore, in step S330, the large language model includes a corpus system module, a pre-trained model, and a fine-tuning module connected in sequence.

[0022] The corpus system module includes pre-training corpus and fine-tuning corpus. The pre-training corpus includes text data collected from books, magazines and encyclopedias. The fine-tuning corpus includes annotated text data processed from open-source code libraries, expert annotations, and user dialogue.

[0023] The pre-trained model is first trained on a large number of general-purpose training data using unsupervised learning methods to obtain a general-purpose language model with strong generalization ability. The training methods of the pre-trained model include at least word vector embedding, contrastive pre-training, and context learning.

[0024] The fine-tuning module includes freezing the bottom layer of the pre-trained model and adjusting the weights of the upper layers. The bottom layer includes word vectors, and the upper layers include a classifier.

[0025] The training steps of the pre-trained model include: collecting datasets and performing supervised fine-tuning; collecting comparative data and training a scoring model; and using reinforcement learning to optimize the cloud-trained model for the reward model.

[0026] The process of collecting the dataset and performing supervised fine-tuning includes sampling a prompt from the dataset; labeling the output answer with preferences; and performing supervised fine-tuning of the pre-trained model based on the collected data.

[0027] The process of collecting and comparing data and training the scoring model includes sampling a prompt and several corresponding model outputs; scoring and sorting the outputs; and training the scoring model on the scored and sorted data.

[0028] The use of reinforcement learning to optimize the cloud-trained model for the reward model includes resampling a cue; the reward model scores the output and optimizes the model parameters.

[0029] Furthermore, step S300 also includes receiving a task instruction, breaking down the task instruction into simple steps, and executing them sequentially.

[0030] Furthermore, in step S400, the virtual digital human is controlled to change clothes based on the character settings preset in the 3D clothing asset library. The 3D clothing asset library includes at least the clothing of the virtual digital human for intelligent hospital guidance, the clothing of the virtual digital human for museum interpretation, and the clothing of the virtual digital human for restaurant greeting.

[0031] Furthermore, in step S350, the virtual human animation control program includes a voice-to-expression service Audio2FaceService and a voice-to-action service Audio2ActionService. The voice-to-expression service Audio2FaceService is used to convert the virtual human's voice information into the virtual human's facial expressions, and the voice-to-action service Audio2ActionService is used to convert the virtual human's voice information into the virtual human's body movements.

[0032] The audio2FaceService, a speech-to-expression service, is a speech-animation synthesis model based on the BlendShapes method. This model includes a 3D face control parameter prediction module based on different speech emotions, an expression base construction module based on sample expressions, and a 3D face animation synthesis module.

[0033] The 3D face control parameter prediction module based on different voice emotions includes a voice preprocessing module, a voice feature extraction module, an audio-visual mapping module, and a 3D face control parameter module connected in sequence.

[0034] The expression base construction module based on sample expressions includes a sample expression model library, an expression feature extraction module, an expression transfer module, and an expression base construction module connected in sequence.

[0035] The 3D face animation synthesis module includes a 3D face control parameter smoothing module and a 3D face animation synthesis module connected in sequence. The input of the 3D face control parameter smoothing module is connected to the output of the 3D face control parameter module, and the input of the 3D face animation synthesis module is connected to the output of the expression base construction module.

[0036] Furthermore, the present invention also proposes an interaction method for a virtual digital human, including the aforementioned method for waking up the virtual digital human, and the interaction method for the virtual digital human further includes:

[0037] S100. Confirm that a wake-up signal has been received, so that the virtual digital human enters the wake-up process and switches from the standby state to the working state. The wake-up process includes a voice wake-up process and a visual wake-up process.

[0038] The voice wake-up process includes the following steps:

[0039] S110. In standby mode, collect ambient sound and detect whether there is a human voice wake-up signal. The human voice wake-up signal includes one or more combinations of human voice, wake-up keywords and voiceprint features.

[0040] S120. If the detected human voice wake-up signal reaches the preset threshold, wake up the virtual digital human and switch the virtual digital human from standby state to working state.

[0041] The visual wake-up process includes the following steps:

[0042] S130. In standby mode, control the visual sensor of the virtual digital human to acquire images;

[0043] S140. Detect human skeleton recognition and / or face recognition to detect if someone is approaching;

[0044] S150. If someone is detected approaching, wake up the virtual digital human and switch the virtual digital human from standby mode to working mode.

[0045] Furthermore, the present invention also proposes an interactive device for a virtual digital human, comprising:

[0046] The display device is a vertical large screen;

[0047] A multimodal input device, comprising a 4K high-definition RGB camera, a ToF depth camera, and a microphone array, is disposed above the display device;

[0048] A real-time rendering device is used to render and output the movements and expressions of virtual digital humans in real time.

[0049] A heat dissipation device, which is used to dissipate heat for the real-time rendering device, and is installed on the real-time rendering device.

[0050] The storage device includes a long-term storage device and a short-term storage device, wherein the long-term storage device is a database, and the database includes a relational database and a vector database, and the short-term storage device is memory;

[0051] A processing device is used to process interactive data between virtual digital humans and users. The processing device is installed on the back of the display device, and the display device, the multimodal input device, the real-time rendering device, and the storage device are respectively connected to the processing device.

[0052] Furthermore, the database includes an action library and an emoticon library. Each action record in the action library includes an action tag, and each emoticon record in the emoticon library includes an emoticon tag. The action tags and the emoticon tags are respectively invoked by the virtual digital human when replying to the user.

[0053] It also includes tool library interfaces, which at least include printer interfaces, search interfaces, encyclopedia interfaces, date and time interfaces, and code execution interfaces.

[0054] Furthermore, the present invention also proposes a computer-readable storage medium storing program instructions thereon, which, when executed by a processor, implement the virtual digital human interaction method described above.

[0055] According to some embodiments of the present invention, the beneficial effects of the present invention are as follows:

[0056] The present invention provides an interactive method for virtual digital humans that enables real-time natural language communication between virtual humans and users in specific environments. By combining the virtual digital human's facial expressions, movements, and professional attire, it achieves a technical effect that enhances the integration of virtual digital humans with real-world scenarios. Attached Figure Description

[0057] Figure 1 This is a flowchart of the interaction method for virtual digital humans in an embodiment of the present invention.

[0058] Figure 2 This is a flowchart of the voice wake-up process and the visual wake-up process in the virtual digital human interaction method of this invention.

[0059] Figure 3 This is a flowchart illustrating the interaction process between a virtual digital human and a user in the virtual digital human interaction method based on large language prompts in an embodiment of the present invention.

[0060] Figure 4 This is a schematic diagram of the virtual human AI brain (AI agent) architecture in an embodiment of the present invention.

[0061] Figure 5 This is a schematic diagram of the overall software architecture in an embodiment of the present invention.

[0062] Figure 6 This is a schematic diagram of the visual wake-up process in an embodiment of the present invention.

[0063] Figure 7 This is a schematic diagram of the voice wake-up process in an embodiment of the present invention.

[0064] Figure 8 This is a schematic diagram of an AI multimodal interactive 3D virtual human large-screen all-in-one machine in an embodiment of the present invention.

[0065] Figure 9 This is a schematic diagram of an AI multimodal interactive 3D virtual human transparent screen all-in-one machine in an embodiment of the present invention.

[0066] Figure 10 This is a flowchart of voiceprint recognition in an embodiment of the present invention.

[0067] Figure 11 This is a flowchart illustrating the data modeling of sound in an embodiment of the present invention.

[0068] Figure 12 This is a basic schematic diagram of the speech image extraction network in an embodiment of the present invention.

[0069] Figure 13 This is a basic schematic diagram of the large language model network structure in an embodiment of the present invention.

[0070] Figure 14 This is a flowchart illustrating the training process of the pre-trained model in an embodiment of the present invention.

[0071] Figure 15 This is a schematic diagram of the basic structure of the speech animation synthesis model based on the BlendShapes method in an embodiment of the present invention.

[0072] Figure 16 This is a schematic diagram of the lip movement detection module in an embodiment of the present invention. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] The following will provide a clear and complete description of the concept, specific structure, and technical effects of the present invention in conjunction with the embodiments and accompanying drawings, so as to fully understand the purpose, solution, and effects of the present invention.

[0075] It should be noted that, unless otherwise specified, when a feature is referred to as "fixed" or "connected" to another feature, it can be directly fixed or connected to the other feature, or indirectly fixed or connected to the other feature. The singular forms "a," "described," and "the" used herein are also intended to include the plural forms, unless the context clearly indicates otherwise. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing particular embodiments only and not for limiting the invention. The term "and / or" as used herein includes any combination of one or more of the associated listed items.

[0076] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various elements, these elements should not be limited to these terms. These terms are only used to distinguish elements of the same type from each other. For example, a first element may also be referred to as a second element without departing from the scope of this disclosure, and similarly, a second element may also be referred to as a first element. Any and all instances or exemplary language (“e.g.,” “such as,” etc.) used herein are intended only to better illustrate embodiments of the invention and, unless otherwise required, do not impose a limitation on the scope of the invention. Furthermore, the industry term “pose” as used herein refers to the position and orientation of an element relative to a spatial coordinate system.

[0077] Reference Figures 1 to 16 This invention provides an interactive method, apparatus, and storage medium for a virtual digital human, characterized in that, with reference to Figure 1 The method includes:

[0078] S100. Confirm that a wake-up signal has been received, so that the virtual digital human enters the wake-up process and switches from the standby state to the working state. The wake-up process includes a voice wake-up process and a visual wake-up process.

[0079] S200, a multimodal sensor based on virtual digital human, acquires environmental and user information;

[0080] S300, based on the large language prompt word project, enables virtual digital humans to interact with users;

[0081] S400, based on Large Language Model (LLM) prompt word engineering, adjusts the character settings of virtual digital humans.

[0082] According to some embodiments of the present invention, the beneficial effects of the present invention are as follows:

[0083] The present invention provides an interactive method for virtual digital humans that enables real-time natural language communication between virtual humans and users in specific environments. By combining the virtual digital human's facial expressions, movements, and professional attire, it achieves a technical effect that enhances the integration of virtual digital humans with real-world scenarios.

[0084] In current technology, 3D virtual humans created through motion capture are becoming increasingly visually appealing and lifelike. However, these 3D virtual human IPs are built upon the personal abilities of the person behind them. Bilibili's A-soul has experienced incidents such as the departure and exposure of its persona. Therefore, the over-reliance on the person's abilities hinders the industrialization and productization of 3D virtual human IPs. If AI could replace the person behind the 3D virtual human, it would undoubtedly solve the industrialization and productization problems. Large language models (such as Wenxin Yiyan, ChatGPT, and Bard) as question-and-answer robots can only output content in text and voice. Users cannot experience the warmth of AI-driven service. We believe that the next generation of interaction will inevitably be based on natural language. Therefore, giving large language models a virtual avatar is an inevitable trend.

[0085] To realize the above ideas, we still need to solve the following three problems. The issues are categorized into several levels: Product-level issues, including the dependence of 3D virtual humans on a real-life human, how to make the interaction between humans and large language models feel more personal, which real-world scenarios should products combining 3D virtual humans and large language models serve, what problems can these products actually solve, and how to enhance the expressiveness of 3D virtual humans to make them feel more human. Software-level issues include: the driving force of virtual human movements and expressions without a real-life human, the character design and personality of 3D virtual humans, how 3D virtual humans acquire external information (i.e., "ears," "eyes," "spatial awareness"), the need for 3D virtual humans to have memory capabilities, the ability to autonomously use tools (generally software tools), and the ability to break down complex tasks into sub-tasks and execute them step-by-step. Hardware-level issues include: the medium on which the virtual human's image is displayed, and how to make this image appear life-sized; the need for one or more sensors to allow the virtual human to acquire external information; and the real-time rendering of 3D virtual humans, as sufficiently detailed 3D virtual humans have a high geometric facet count and large texture size. High-performance 3D rendering capabilities require high-performance hardware, which inevitably leads to high heat generation and heat dissipation issues. The addition of fans to solve heat dissipation problems can cause acoustic degradation problems, sound input problems in noisy environments, and the problem of distinguishing speech from speakers in multi-person environments.

[0086] Therefore, we propose the following solutions:

[0087] (1) Solutions to product-level problems

[0088] In the past, 3D virtual humans relied on a human as their "soul"—the human being was the soul, and without the human, a virtual human could do nothing. Our 3D virtual humans use a large language model to replace the human as the soul. This also solves the problem of virtual humans serving multiple scenarios. The All-in-One AI Multimodal Interactive 3D Virtual Human Large Screen breaks away from the traditional interactive solution where large language models can only output voice and text, making the large language model the soul of the virtual human, and the virtual human the beautiful shell of the large language model. The All-in-One AI Multimodal Interactive 3D Virtual Human Large Screen adds a visual element to the output of large language models, making services more personal. Most importantly, the All-in-One AI Multimodal Interactive 3D Virtual Human Large Screen can independently complete some tasks, with an elegant appearance, bringing tangible foot traffic to businesses.

[0089] (2) Solutions to software technology issues

[0090] Reference Figure 4 and Figure 5 To enable the large language model to act as the soul of the virtual human, the driving problem of the virtual human needs to be solved. The Audio2Animation algorithm was developed in-house. This algorithm consists of two parts: Audio2Face (speech-to-facial expression conversion) and Audio2Action (speech-to-action conversion). The virtual reality all-in-one machine uses the large language model's PromptEngineering feature to pre-set several character designs and provides matching 3D clothing assets to give the virtual human professional attributes and a more professional appearance. Multimodal sensors allow the virtual human to acquire various information by understanding the external environment. Depth sensors reconstruct the 3D environment, enabling the virtual human to understand it. RGB cameras perform visual segmentation of the environment and object recognition, face recognition, human skeleton recognition, posture and gesture recognition on the segmented images. A localized vector database and relational database give the virtual human long-term memory. A sufficient tool library is provided for the virtual human (including at least printer interfaces, search interfaces, encyclopedia interfaces, date and time interfaces, and code execution interfaces), allowing the virtual human to select and use appropriate tools through the large language model. The Phantom All-in-One Machine, through large-scale language model prompting engineering, enables virtual humans to solve complex problems. This means the AI ​​virtual human can break down complex task commands into simpler steps and execute them sequentially. During execution, it can choose tools or plugins, or make its own selections.

[0091] (3) Hardware-level solutions

[0092] Using an 86-inch vertical screen and an 86-inch transparent screen, the rendered virtual human is rendered to be life-size, and on the transparent screen, the virtual human has a naked-eye 3D effect. The virtual human possesses "glasses," "ears," and "spatial awareness" to acquire external information. A circular 7-microphone array and a 4K high-definition RGB camera with HDR are selected, along with a ToF depth camera sensor. To integrate these three sensors, the Azure Kinect DK depth vision sensor, used in industrial robots, is chosen. For high-quality real-time rendering of the 3D virtual human, NVIDIA's high-performance RTX 3080 graphics card with ray tracing is selected. Due to the heat dissipation issues caused by the high-performance graphics card, and the noise and vibration issues caused by the added fans, we opted for a modular, modular hardware design for the all-in-one machine. This solved the heat dissipation and sound recording problems, while also reducing maintenance costs and transportation difficulties. The issue of fan-induced chassis vibration affecting the built-in microphone array was resolved by integrating the microphones and various sensors into a single box and placing it externally on the top of the chassis. We have developed our own acoustic noise reduction and voice enhancement algorithms to solve the problem of voice input in noisy environments. We also utilize a narrow beam algorithm combined with facial recognition for directional sound pickup, resulting in better sound reception. In multi-person conversations, we combine facial recognition with a lip movement recognition module to accurately and directionally pick up voices from multiple speakers, and then use a stored voiceprint database to separate the voiceprint of the individual speaker from the multiple voices.

[0093] In some specific embodiments, the lip movement recognition module includes a target detection module, a lip movement detection module, and an interaction module connected in sequence.

[0094] The object detection module needs to accurately locate the lips of a face and segment the image using predicted coordinates, ensuring the lips are centered within the image. The project implements the Yolov5 algorithm, trained using a minimal pre-trained model. In creating the object detection dataset, data integrity must be ensured, and the labeled lips should be located in the center of the image to achieve accurate lip localization. This significantly accelerates the fitting speed of the subsequent classification network.

[0095] The lip movement detection module employs a composite network of 3DResNet and GRU. Data processed using the Yolov5 algorithm is fed into this network. The residual structure extracts features, while GRU ensures the transmission and preservation of temporal information. Softmax is then used to obtain the prediction result. In this lip movement detection module, the quantity of training data is crucial. Therefore, data augmentation is used in preprocessing to ensure the network has sufficient data for training. Secondly, the network structure is also very important. The residual network ResNet is composed of multiple stacked networks, solving the gradient vanishing problem caused by network depth. A deeper network extracts more image information features, making it more effective. The GRU, a variant of the recurrent neural network (RNN), uses a gate control mechanism to effectively preserve temporal information, allowing the neural network to focus more on the dynamic changes in lip movements over time.

[0096] Specifically, in some specific embodiments, reference is made to Figure 16 The lip movement detection module is a GRU with one layer embedded in ResNet, forming a composite network of CNN and RNN. However, the network does not simply use 3D convolution and pooling structures. In the feature extraction process, it tends to use CNN to extract pixel features and let GRU extract features that change over time.

[0097] The interaction module receives the prediction results and uses HTML and JavaScript to facilitate front-end and back-end interaction, enabling the system's recognition functionality. The interaction module utilizes the Flask framework, known for its lightweight and flexible design. In the login function, HTML directly submits the form to the back-end, which simply validates the form against the database to complete the login process. In the recognition function, JavaScript, in conjunction with HTML, reads local video and submits it to the back-end for processing. The processing involves first segmenting the video into frames, then using a Yolov5 model to predict lip coordinates in each frame, followed by image segmentation and saving. Finally, a classification network is used to obtain the predicted words, which are then displayed on the front-end page, completing the entire workflow.

[0098] Specifically, in step S400, the prompt word project provides the character role setting for the virtual human, not just for changing clothes, but to let the virtual human play a professional role, such as a doctor, lawyer, or primary school English teacher.

[0099] Furthermore, refer to Figure 2 and Figure 7 In step S100, the voice wake-up process includes the following steps:

[0100] S110. In standby mode, collect ambient sound and detect whether there is a human voice wake-up signal. The human voice wake-up signal includes one or more combinations of human voice, wake-up keywords and voiceprint features.

[0101] S120. If the detected human voice wake-up signal reaches the preset threshold, wake up the virtual digital human and switch the virtual digital human from standby state to working state.

[0102] Furthermore, in step S110, detecting the presence of a human voice wake-up signal includes the following steps:

[0103] S111. Perform noise reduction preprocessing on the received ambient sound, the noise reduction preprocessing including framing the sound signal, windowing, pre-emphasis and adaptive endpoint detection (VAD);

[0104] S112. Extract speech features based on MDCC and input the speech features into the voiceprint recognition model;

[0105] S113. Detect whether there is a human voice wake-up signal, and perform voiceprint matching between the speech features and the existing voiceprints in the voiceprint matching library.

[0106] S114. If the voiceprint matching is successful, retrieve the user information that matches the voiceprint; if the voiceprint matching is unsuccessful, create a new user storage space.

[0107] In some specific embodiments, in step S114, the user storage space is set within Amazon's private repository.

[0108] In some specific embodiments, the voiceprint recognition process first extracts speech features, then feeds these features into a model for training, and finally finds the result with the highest or closest score. (See reference...) Figure 10 Voiceprint recognition includes a training phase and a testing phase. The training phase includes inputting training speech, feature extraction, model training, and searching the voiceprint database or generating new target user registration speech. The testing phase includes inputting test speech, feature extraction, voiceprint matching and scoring, and distinguishing between target users and non-target users. For virtual digital humans, target users can be defined as users who have already interacted with the virtual digital human and whose information is stored in the memory. Non-target users can be defined as users who are interacting with the virtual digital human for the first time. The virtual digital human needs to create new storage space to store user information.

[0109] In some specific embodiments, refer to Figure 11 The flowchart shows how a virtual digital human models sound data. The virtual digital human first receives sound, performs preprocessing, MFCC, and voiceprint recognition on the sound, and then communicates with the user or creates a new user based on the recognition results.

[0110] Furthermore, in step S112,

[0111] The speech features include linear prediction coefficients (LPCC) and Mel cepstral coefficients (MFCC).

[0112] Specifically, speech contains many characteristics. Feature extraction involves converting the continuous time-domain speech signal into a discrete frequency-domain signal to analyze the energy of the speech signal and extract the desired speech features from the continuous time-domain speech signal to represent the speech. There are many types of speech features, including linear prediction coefficients (LPCC) and Mel-frequency cepstral coefficients (MFCC) (Girin-Laurent).

[0113] The cochlea, a human organ, helps humans receive and distinguish sounds in noisy environments, thus playing a role in filtering noise during speech signal reception. Therefore, the idea is to construct a filter that utilizes the human method of sound discrimination to replace the cochlea's function and achieve the same noise-filtering effect. The human cochlea is similar to a combination of multiple filters. It is highly sensitive to low-frequency signal energy. It exhibits a linear relationship with frequencies below 1000Hz and a logarithmic relationship with frequencies above 1000Hz. This irregular frequency division is a key characteristic distinguishing speech signals from Mel-frequency cepstrum and ordinary cepstrum. Mel-frequency cepstrum is one of the most common speech features in speech signal processing. Research on the biological function of the human ear has revealed that the human ear has different auditory sensitivities to speech signals of different amplitudes and frequencies. The range of amplitude and frequency that the human ear can receive is limited. For speech signals with amplitudes and frequencies exceeding the range that the human ear can receive, such as ultrasound frequencies around 20,000Hz, the human ear cannot hear them. Speech signals with frequencies below 20Hz are also inaudible. The frequency range of speech signals that the human ear can hear is between 20Hz and 20000Hz. Noise in the 200Hz to 5000Hz frequency range has the greatest impact on speech signals, making the text content unclear.

[0114] Furthermore, in step S113, the voice wake-up signal includes a wake-up word, which is stored in the public area of ​​the voiceprint recognition model.

[0115] In some embodiments, the virtual digital human voice monitoring software collects ambient sound and detects the presence of a human voice wake-up signal. If the detected human voice wake-up signal reaches a threshold, the virtual digital human is awakened and enters working mode. After entering working mode, the virtual digital human outputs a response signal to the user and collects the user's voice signal through a microphone. The virtual digital human converts the collected user voice signal into text through a speech-to-text service, inputs it into a large language model service, outputs text through the large language model service, and outputs speech through a text-to-speech service. The virtual familiar human outputs facial expressions and actions based on an animation control program.

[0116] Furthermore, refer to Figure 2 and Figure 6 In step S100, the visual wake-up process includes the following steps:

[0117] S130. In standby mode, control the visual sensor of the virtual digital human to acquire images;

[0118] S140. Detect human skeleton recognition and / or face recognition to detect if someone is approaching;

[0119] S150. If someone is detected approaching, wake up the virtual digital human and switch the virtual digital human from standby mode to working mode.

[0120] In some specific embodiments, the virtual digital human's visual sensors acquire images, and detect whether someone is approaching through human skeleton recognition and facial recognition. If someone is detected, the virtual digital human requests the AI ​​brain to output spoken text through a large language model service. Based on a text-to-speech service, the spoken text output by the large language model service is converted into speech. Based on the virtual human animation control program, sound, facial expressions, and actions are output. If the microphone receives a user's response, the received content is output as feedback. The microphone receives the user's response and the voice information of the questions asked. Based on a speech-to-text service, the voice information is converted into text. Based on the large language model service, text information is input and a response text information is output. Based on a text-to-speech service, the response text information is converted into virtual human voice information. Based on the virtual human animation control program, facial expressions and actions are output.

[0121] In some specific embodiments, as an alternative to wake-up algorithms, the state of the speaker's face and lips is captured visually, specifically including the following steps:

[0122] Identify the currently interacting person using facial or skeletal features;

[0123] Capture the lip state of the person currently interacting with the device using computer vision algorithms;

[0124] When the lips are open or closed, the microphone is activated to pick up directional sound; when the lips are closed, the microphone is deactivated to pick up directional sound. This solves the problem of mis-picking sound in noisy environments or when multiple people are talking.

[0125] The lip-reading algorithm converts continuously recorded lip-movement video footage into lip-reading text, thereby enhancing the microphone's speech recognition capabilities.

[0126] Furthermore, in step S100, after starting the virtual digital human, the standby state can be skipped and the working state can be entered directly.

[0127] Specifically, once the virtual digital human is powered on, it can attract the user's attention by playing animations and voices directly on the screen without detecting whether the user is approaching.

[0128] Furthermore, in step S200,

[0129] The multimodal sensor of the virtual digital human includes at least a depth sensor, an RGB camera, and a microphone. The multimodal sensor is used for at least facial recognition, skeletal recognition, posture, and gesture recognition of the human.

[0130] Furthermore, refer to Figure 3 , Figure 6 and Figure 7 Step S300 includes the following steps;

[0131] S310. After entering the working state, the virtual digital human receives the user's images and voice through a multimodal sensor, and transmits the voice and images to the voice and image extraction network to extract the voice and images.

[0132] S320, based on speech-to-text service, converts speech into text;

[0133] S330, based on a large language model service, outputs language text;

[0134] S340. Based on the text-to-speech service, convert the language text output by the large language model service into speech;

[0135] S350, based on a virtual human animation control program, outputs sound, facial expressions, and movements;

[0136] S360. If the multimodal sensor receives a response from the user, repeat steps S310 to S350 to provide feedback on the received content, including voice, facial expressions, and gestures.

[0137] Furthermore, refer to Figure 12 In step S310, the speech-image extraction network includes: a speech separation network and a visual separation network.

[0138] The speech separation network includes a hybrid speech receiving module, an STFT speech time-frequency transformation module, a speech feature extraction module, a speech upsampling module, and an ISTFT frequency-time transformation module connected in sequence.

[0139] The visual separation network includes a hybrid visual receiving module, a VGGFace feature extraction module, and a visual feature network module connected in sequence.

[0140] It also includes a fusion module, the input of which is connected to the speech feature extraction module and the VGG feature extraction module respectively, and the output of which is connected to the speech upsampling module.

[0141] Specifically, refer to Figure 12 The speech-image extraction network structure mainly consists of two parts: a speech separation network and a visual separation network. The speech separation network receives the mixed speech spectrogram and outputs a spectral mask. Its network model structure adopts an encoding / decoding structure similar to U-Net, referred to as the feature extraction module and the upsampling module, respectively. There are skip connections between the feature extraction module and the upsampling module. Their main function is to use the shallow features from the feature extraction module to help the upsampling module predict a high-resolution time-frequency mask. However, the performance of feature extraction modules at different depths varies greatly on different tasks. To enable the network to adaptively learn network features at different depths, residual connections are added to the feature extraction module. The visual separation network receives a set of faces that completely correspond to the speech spectrogram and outputs visual features that are distributed in the same way as those of the speech separation network. It adds residual connections and includes an audio-video feature fusion module to the U-Net model.

[0142] Furthermore, refer to Figure 13 In step S330, the large language model includes a corpus system module, a pre-trained model, and a fine-tuning module connected in sequence.

[0143] The corpus system module includes pre-training corpus and fine-tuning corpus. The pre-training corpus includes text data collected from books, magazines and encyclopedias. The fine-tuning corpus includes annotated text data processed from open-source code libraries, expert annotations, and user dialogue.

[0144] The pre-trained model is first trained on a large number of general-purpose training data using unsupervised learning methods to obtain a general-purpose language model with strong generalization ability. The training methods of the pre-trained model include at least word vector embedding, contrastive pre-training, and context learning.

[0145] The fine-tuning module includes freezing the bottom layer of the pre-trained model and adjusting the weights of the upper layers. The bottom layer includes word vectors, and the upper layers include classifiers.

[0146] Specifically, the corpus system includes pre-training corpus and fine-tuning corpus, the latter including code and dialogue fine-tuning corpus. The pre-training corpus includes massive amounts of text data collected from books, magazines, encyclopedias, and other sources, enabling the model to learn how to express logical relationships in language; the fine-tuning corpus includes high-quality annotated text data processed from open-source code repositories, expert annotations, user dialogues, and other methods, further enhancing its dialogue capabilities.

[0147] In some embodiments, pre-training is fundamental to building large-scale language models. It refers to performing extensive general training on large-scale training data using unsupervised learning methods to obtain a general and highly generalizable language model. Based on this large-scale data, pre-training enables the model to initially possess the ability to understand human language and learn context, capturing semantic similarity features of text and code fragments, thereby generating more accurate text and code vectors and supporting subsequent fine-tuning tasks.

[0148] In some embodiments, fine-tuning is essential for the practical application of the model. It refers to further training the pre-trained model on a task-specific dataset, typically involving freezing the lower layers (e.g., word vectors) and adjusting the weights of higher layers (e.g., classifiers). Fine-tuning the pre-trained model significantly reduces training time, saves computational resources, and accelerates training convergence. Building upon a pre-trained model with strong generalization capabilities, integrating code-based training and instruction-based fine-tuning using a specific dataset enhances its question-and-answer dialogue text generation capabilities.

[0149] Furthermore, refer to Figure 14 The training steps of the pre-trained model include: collecting datasets and performing supervised fine-tuning; collecting comparative data and training a scoring model; and using reinforcement learning to optimize the cloud-trained model for the reward model.

[0150] Furthermore, refer to Figure 14 The process of collecting the dataset and performing supervised fine-tuning includes sampling a prompt from the dataset; labeling the output answer with preferences; and performing supervised fine-tuning of the pre-trained model based on the collected data.

[0151] Furthermore, refer to Figure 14 The process of collecting and comparing data and training the scoring model includes sampling a prompt and several corresponding model outputs; scoring and sorting the outputs; and training the scoring model on the scored and sorted data.

[0152] Furthermore, refer to Figure 14 The use of reinforcement learning to optimize the cloud-trained model for the reward model includes resampling a cue; the reward model scores the output and optimizes the model parameters.

[0153] Furthermore, step S300 also includes receiving a task instruction, breaking down the task instruction into simple steps, and executing them sequentially.

[0154] Furthermore, in step S400, the virtual digital human is controlled to change clothes based on the character settings preset in the 3D clothing asset library. The 3D clothing asset library includes at least the clothing of the virtual digital human for intelligent hospital guidance, the clothing of the virtual digital human for museum interpretation, and the clothing of the virtual digital human for restaurant greeting.

[0155] Furthermore, in step S350, the virtual human animation control program includes a voice-to-expression service Audio2FaceService and a voice-to-action service Audio2ActionService. The voice-to-expression service Audio2FaceService is used to convert the virtual human's voice information into the virtual human's facial expressions, and the voice-to-action service Audio2ActionService is used to convert the virtual human's voice information into the virtual human's body movements.

[0156] Furthermore, refer to Figure 15 The audio2FaceService, a speech-to-expression service, is a speech-animation synthesis model based on the BlendShapes method. This model includes a 3D face control parameter prediction module based on different speech emotions, an expression base construction module based on sample expressions, and a 3D face animation synthesis module.

[0157] The 3D face control parameter prediction module based on different voice emotions includes a voice preprocessing module, a voice feature extraction module, an audio-visual mapping module, and a 3D face control parameter module connected in sequence.

[0158] The expression base construction module based on sample expressions includes a sample expression model library, an expression feature extraction module, an expression transfer module, and an expression base construction module connected in sequence.

[0159] The 3D face animation synthesis module includes a 3D face control parameter smoothing module and a 3D face animation synthesis module connected in sequence. The input of the 3D face control parameter smoothing module is connected to the output of the 3D face control parameter module, and the input of the 3D face animation synthesis module is connected to the output of the expression base construction module.

[0160] Furthermore, refer to Figure 8 and Figure 9 The present invention also proposes an interactive device for a virtual digital human, characterized in that it comprises:

[0161] The display device is a vertical large screen;

[0162] A multimodal input device, comprising a 4K high-definition RGB camera, a ToF depth camera, and a microphone array, is disposed above the display device;

[0163] A real-time rendering device is used to render and output the movements and expressions of virtual digital humans in real time.

[0164] A heat dissipation device, the heat dissipation device being used to dissipate heat for the real-time rendering device, the heat dissipation device being installed on the real-time rendering device;

[0165] The storage device includes a long-term storage device and a short-term storage device, wherein the long-term storage device is a database, and the database includes a relational database and a vector database, and the short-term storage device is memory;

[0166] A processing device is used to process interactive data between virtual digital humans and users. The processing device is installed on the back of the display device, and the display device, the multimodal input device, the real-time rendering device, and the storage device are respectively connected to the processing device.

[0167] Furthermore, the database includes an action library and an emoticon library. Each action record in the action library includes an action tag, and each emoticon record in the emoticon library includes an emoticon tag. The action tags and the emoticon tags are respectively invoked by the virtual digital human when replying to users.

[0168] Furthermore, it also includes a tool library interface, which includes at least a printer interface, a search interface, an encyclopedia interface, a date and time interface, and an executable code interface.

[0169] Furthermore, the processing device includes an AI algorithm service program, an interactive backend program, a backend web program, and a virtual human rendering program.

[0170] Furthermore, the AI ​​algorithm service program is a Python program, which includes Audio2Face algorithm service, Text2Action algorithm service, large language model distribution service, and AI Agent service.

[0171] Furthermore, the interactive backend program is a Java program, which includes an account management system, a memory configuration system, a 3D asset management system, and a large model interactive management system. The interactive backend program is connected to the database.

[0172] Furthermore, the backend web application is an H5 application, which includes an account management system page, a content configuration system page, a 3D asset management system page, and a large model interaction management page.

[0173] Furthermore, the virtual human rendering program includes a UE5 program, which includes virtual human rendering, motion animation logic, and external interaction programs.

[0174] Furthermore, the display device is an LED screen, a naked-eye 3D screen, or a holographic screen.

[0175] Furthermore, the present invention also proposes a computer-readable storage medium storing program instructions thereon, which, when executed by a processor, implement the virtual digital human interaction method described above.

[0176] It should be understood that the method steps in the embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium. The method can use standard programming techniques. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if necessary, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit (ASIC).

[0177] Furthermore, the procedures described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by the context. The procedures described herein (or variations and / or combinations thereof) may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The computer program comprises a plurality of instructions executable by one or more processors.

[0178] Furthermore, the method can be implemented in any suitable type of computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices, etc. Aspects of the invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein. Furthermore, the machine-readable code, or portions thereof, can be transmitted via wired or wireless networks. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media comprises instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. When programmed according to the methods and techniques described in the invention, the invention may also include the computer itself.

[0179] A computer program can be applied to input data to perform the functions described herein, thereby transforming the input data to generate output data stored in non-volatile memory. The output information can also be applied to one or more output devices, such as a display. In a preferred embodiment of the invention, the transformed data represents physical and tangible objects, including specific visual depictions of physical and tangible objects generated on the display.

[0180] The above description is merely a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention, as long as they achieve the technical effects of the present invention by the same means, should be included within the scope of protection of the present invention. Within the scope of protection of the present invention, the technical solutions and / or implementation methods can have various modifications and variations.

Claims

1. A virtual digital human interaction method based on a large language model, characterized in that, The method includes: S200, a multimodal sensor based on virtual digital human, acquires environmental and user information; S300, based on the large language prompt word project, enables virtual digital humans to interact with users; Step S300 includes the following steps; S310. After entering the working state, the system receives the user's image and voice through the multimodal sensor, and transmits the voice and image to the voice and image extraction network to extract the voice and image. In step S310, the speech-image extraction network includes: a speech separation network and a visual separation network. The speech separation network includes a hybrid speech receiving module, an STFT speech time-frequency transformation module, a speech feature extraction module, a speech upsampling module, and an ISTFT frequency-time transformation module connected in sequence. The visual separation network includes a hybrid visual receiving module, a VGGFace feature extraction module, and a visual feature network module connected in sequence. It also includes a fusion module, the input of which is connected to the speech feature extraction module and the VGG feature extraction module respectively, and the output of which is connected to the speech upsampling module; S320, based on speech-to-text service, converts speech into text; S330. Based on the large language model service, output the language text; in step S330, the large language model includes a corpus system module, a pre-trained model, and a fine-tuning module connected in sequence. The corpus system module includes pre-training corpus and fine-tuning corpus. The pre-training corpus includes text data collected from books, magazines and encyclopedias. The fine-tuning corpus includes annotated text data processed from open-source code libraries, expert annotations, and user dialogue. The pre-trained model is first trained on a large number of general-purpose training data using unsupervised learning methods to obtain a general-purpose language model with strong generalization ability. The training methods of the pre-trained model include at least word vector embedding, contrastive pre-training, and context learning. The fine-tuning module includes freezing the bottom layer of the pre-trained model and adjusting the weights of the upper layers. The bottom layer includes word vectors, and the upper layers include a classifier. The training steps of the pre-trained model include: collecting datasets and performing supervised fine-tuning; collecting comparative data and training a scoring model; and using reinforcement learning to optimize the cloud-trained model for the reward model. The process of collecting the dataset and performing supervised fine-tuning includes sampling a prompt from the dataset; labeling the output answer with preferences; and performing supervised fine-tuning of the pre-trained model based on the collected data. The process of collecting and comparing data and training the scoring model includes sampling a prompt and several corresponding model outputs; scoring and sorting the outputs; and training the scoring model on the scored and sorted data. The use of reinforcement learning to optimize the cloud-trained model for the reward model includes resampling a cue; the reward model scores the output and optimizes the model parameters; S340. Based on the text-to-speech service, convert the language text output by the large language model service into speech; S350, based on a virtual human animation control program, outputs sound, facial expressions, and movements; S360. If a response is received from the user, repeat steps S310 to S350 to provide feedback on the received content, outputting voice, facial expressions, and actions. S400, based on the large language model, prompt word engineering, adjusts the character settings of virtual digital humans.

2. The virtual digital human interaction method based on a large language model according to claim 1, characterized in that, In step S200 The multimodal sensor of the virtual digital human includes at least a depth sensor, an RGB camera, and a microphone. The multimodal sensor is used for at least facial recognition, skeletal recognition, posture, and gesture recognition of the human.

3. The virtual digital human interaction method based on a large language model according to claim 1, characterized in that, Step S300 further includes receiving a task instruction, breaking down the task instruction into simple steps, and executing them sequentially.

4. The virtual digital human interaction method based on a large language model according to claim 1, characterized in that, In step S400, the virtual digital human is controlled to change clothes. Based on the character settings preset in the 3D clothing asset library, the 3D clothing asset library includes at least the clothing of the hospital intelligent guide virtual digital human, the clothing of the museum guide virtual digital human, and the clothing of the restaurant greeter virtual digital human.

5. The virtual digital human interaction method based on a large language model according to claim 1, characterized in that, In step S350, the virtual human animation control program includes a voice-to-expression service Audio2FaceService and a voice-to-action service Audio2ActionService. The voice-to-expression service Audio2FaceService is used to convert virtual human voice information into virtual human facial expressions, and the voice-to-action service Audio2ActionService is used to convert virtual human voice information into virtual human body movements. The audio2FaceService, a speech-to-expression service, is a speech-animation synthesis model based on the BlendShapes method. This model includes a 3D face control parameter prediction module based on different speech emotions, an expression base construction module based on sample expressions, and a 3D face animation synthesis module. The 3D face control parameter prediction module based on different voice emotions includes a voice preprocessing module, a voice feature extraction module, an audio-visual mapping module, and a 3D face control parameter module connected in sequence. The expression base construction module based on sample expressions includes a sample expression model library, an expression feature extraction module, an expression transfer module, and an expression base construction module connected in sequence. The 3D face animation synthesis module includes a 3D face control parameter smoothing module and a 3D face animation synthesis module connected in sequence. The input of the 3D face control parameter smoothing module is connected to the output of the 3D face control parameter module, and the input of the 3D face animation synthesis module is connected to the output of the expression base construction module.

6. A method for interacting with a virtual digital human, characterized in that, Including the virtual digital human interaction method based on a large language model as described in any one of claims 1 to 5, the virtual digital human interaction method further includes: S100. Confirm that a wake-up signal has been received, so that the virtual digital human enters the wake-up process and switches from the standby state to the working state. The wake-up process includes a voice wake-up process. The voice wake-up process includes the following steps: S110. In standby mode, collect ambient sound and detect whether there is a human voice wake-up signal, wherein the human voice wake-up signal includes human voice, wake-up keywords and voiceprint features; S120. If the detected human voice wake-up signal reaches the preset threshold, wake up the virtual digital human and switch the virtual digital human from standby state to working state.

7. An interactive device for a virtual digital human, used to implement the virtual digital human interaction method based on a large language model as described in any one of claims 1 to 5, characterized in that, include: The display device is a vertical large screen; A multimodal input device, comprising a 4K high-definition RGB camera, a ToF depth camera, and a microphone array, is disposed above the display device; A real-time rendering device is used to render and output the movements and expressions of virtual digital humans in real time. A heat dissipation device, which is used to dissipate heat for the real-time rendering device, and is installed on the real-time rendering device. The storage device includes a long-term storage device and a short-term storage device, wherein the long-term storage device is a database, and the database includes a relational database and a vector database, and the short-term storage device is memory; A processing device is used to process interactive data between virtual digital humans and users. The processing device is installed on the back of the display device, and the display device, the multimodal input device, the real-time rendering device, and the storage device are respectively connected to the processing device.

8. The interactive device for a virtual digital human according to claim 7, characterized in that, The database includes an action library and an emoticon library. Each action record in the action library includes an action tag, and each emoticon record in the emoticon library includes an emoticon tag. The action tags and the emoticon tags are respectively invoked by the virtual digital person when replying to the user. It also includes tool library interfaces, which at least include printer interfaces, search interfaces, encyclopedia interfaces, date and time interfaces, and code execution interfaces.

9. A computer-readable storage medium having stored thereon program instructions that, when executed by a processor, perform the method as described in any one of claims 1 to 5.

Images