AI digital human construction method integrating knowledge graph and emotional rendering

Abstract

The invention discloses an AI digital human construction method integrating a knowledge graph and emotional rendering, and belongs to the technical field of virtual digital humans. Comprising the following steps: acquiring and preprocessing multi-modal data, transmitting the processed data to a server in real time, converting the multi-modal data into a text sequence, performing parallel processing on the text sequence, performing retrieval in a pre-constructed knowledge graph based on the text sequence to obtain structured knowledge, converting the structured knowledge into a natural language, and storing the natural language in a database. Generating a reply text based on the structured knowledge converted into the natural language and the emotional state vector, generating a synthetic voice and a visual element sequence aligned with a synthetic voice timestamp based on the reply text and the emotional state vector, and synchronously issuing the emotional state vector, the synthetic voice and the visual element sequence aligned with the synthetic voice timestamp to a client. And driving the digital human model to carry out real-time rendering of expressions and mouth shapes, and synchronously playing the synthesized voice. According to the method, the AI digital human construction integrating the knowledge graph and emotional rendering is realized.

Description

Technical Field

[0001] This invention relates to the fields of virtual reality and artificial intelligence, and in particular to a method for constructing AI digital humans that integrates knowledge graphs and emotional rendering. Background Technology

[0002] Current virtual digital human systems in the industry typically consist of five major modules: character generation, voice generation, animation generation, audio-visual synthesis and display, and interaction. Their core technologies can be summarized into three major processes: modeling, driving, and rendering. Key modules generally include Natural Language Processing (NLP), Automatic Speech Recognition (ASR), Text-to-Speech (TTS), Computer Vision (CV), and motion capture.

[0003] Overall, these solutions rely on advancements in Chinese voice and image technologies, achieving character modeling, voice interaction, and animation through multimodal fusion. However, their applications are primarily focused on marketing, navigation, and intelligent customer service. Despite the rapid development of digital human technology in these scenarios, existing solutions still have significant shortcomings in psychological counseling and educational applications. Specifically, these include: 1. Functional fragmentation: For example, some companies focus on academic tutoring but lack a mental health module; others rely on static resource libraries and cannot dynamically connect with academic data.

[0004] 2. Lack of Emotion: The facial expressions and movements of existing digital humans are mostly based on preset, rigid triggers (such as nodding triggered by keywords), lacking continuous emotional changes based on psychological models. This leads to the "uncanny valley" effect in the interaction process, making it difficult to establish deep emotional connections.

[0005] 3. Multimodal latency: In the audio and video synthesis and network transmission stages, there are often problems such as lip movements and speech not being synchronized and high response latency, which seriously affect the user experience.

[0006] To address the aforementioned issues, there is an urgent need for an AI digital human construction method that integrates knowledge graphs and emotional rendering to solve the problems associated with traditional methods. Summary of the Invention

[0007] The purpose of this invention is to provide an AI digital human construction method that integrates knowledge graphs and emotional rendering. It proposes a dual-driven architecture of "cognition-emotion", which uses knowledge graphs (KG) to enhance the cognitive accuracy of digital humans, uses multi-dimensional emotion computing to endow digital humans with "personalized" characteristics, and uses WebRTC streaming technology to ensure the real-time nature of interaction. This significantly improves the accuracy of digital humans in answering questions in professional fields, achieves deep emotional interaction through psychological models, and ensures low-latency multimodal synchronous output, greatly enhancing the realism and practicality of interaction.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for constructing AI digital humans that integrates knowledge graphs and emotional rendering includes: Step 1: Acquire and preprocess the multimodal data input by the user, and transmit the processed data to the server in real time; Step 2: Convert the multimodal data into a text sequence and process the text sequence in parallel: on the one hand, predict the user's psychological state and update the sentiment state vector through sentiment analysis; on the other hand, trigger the knowledge retrieval process. Step 3: Based on the text sequence, retrieve structured knowledge from the pre-built knowledge graph and convert it into natural language; Step 4: Generate response text based on structured knowledge converted into natural language and sentiment state vectors; generate synthesized speech and a sequence of visual pixels aligned with the timestamp of the synthesized speech based on the response text and sentiment state vectors. Step 5: Simultaneously send the emotion state vector, synthesized speech, and a sequence of visual pixels aligned with the timestamp of the synthesized speech to the client to drive the digital human model to render facial expressions and lip movements in real time, and play the synthesized speech synchronously.

[0009] Furthermore, in step 1, the multimodal data input by the user is acquired and preprocessed, and the processed data is transmitted to the server in real time, specifically as follows: Voice data is collected via microphone, and noise reduction and automatic gain control are performed using AudioContext based on front-end JavaScript. DataChannel, based on WebRTC, establishes a low-latency transmission channel to transmit processed data to the server.

[0010] Further, in step 2, the multimodal data is converted into a text sequence, specifically as follows: The server-side gRPC interface receives the processed data, pushes it to the ASR model, and outputs a text sequence based on the ASR model.

[0011] Furthermore, in step 2, the text sequence is processed in parallel, specifically as follows: The text sequence is input into the sentiment analysis module and the retrieval module in parallel. The sentiment analysis module predicts the current user's SCL-90 dimension score based on the text sequence using the BERT model and updates the sentiment state vector.

[0012] Furthermore, in step 3, based on the text sequence, structured knowledge is retrieved from the pre-built knowledge graph and converted into natural language, specifically as follows: Identify at least one entity from a text sequence; Using the identified entities as anchor points, multi-hop subgraph retrieval is performed in the pre-built knowledge graph to obtain structured knowledge associated with the entities; The retrieved structured knowledge is converted into natural language.

[0013] Furthermore, in step 4, the response text is generated based on the structured knowledge converted into natural language and the sentiment state vector, specifically as follows: The structured knowledge and sentiment state vectors converted into natural language are injected into the LLM's prompt; LLM generates a response text based on the Prompt.

[0014] Further, in step 4, synthesized speech and a sequence of visual pixels aligned with the timestamp of the synthesized speech are generated based on the response text and the emotion state vector, specifically as follows: The TTS service obtains the response text and sentiment vector, and generates synthesized speech and a sequence of visual pixels aligned with the timestamp of the synthesized speech based on the SoVITS model.

[0015] Furthermore, in step 5, the emotional state vector, synthesized speech, and a sequence of visual pixels aligned with the timestamp of the synthesized speech are simultaneously sent to the client to drive the digital human model to perform real-time rendering of facial expressions and lip movements, and to play the synthesized speech synchronously. Specifically: The emotional state vector, synthesized speech, and a sequence of visual pixels aligned with the timestamp of the synthesized speech are simultaneously delivered via WebRTC. The client-side web application receives the data sent to it and updates the fixed points of the digital human model using the Live2D engine. Specifically, it updates facial expression parameters based on emotion state vectors, updates lip shape parameters based on synthesized speech, and simultaneously plays the synthesized speech. In summary, the present invention has at least one of the following beneficial technical effects: 1. The psychological depth and sophistication of emotional interaction have been significantly improved, enabling the simulation of continuous and dynamic emotional changes that conform to the logic of clinical psychology, and the emotional expression has personalized characteristics.

[0016] 2. The accuracy of domain knowledge question answering and the ability of global reasoning have been greatly enhanced, significantly reducing factual errors in question answering in specific domains (such as campus rules and psychological counseling knowledge), and possessing the ability to handle complex logical relationships and macro-level issues.

[0017] 3. The audio and video synchronization is highly real-time and the interaction is smooth, achieving millisecond-level interaction latency from end to end, and maintaining a high degree of synchronization between voice and lip movements even in weak network environments. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the interactive process of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0020] like Figure 1 and Figure 2 As shown, this invention provides a method for constructing AI digital humans that integrates knowledge graphs and emotional rendering, including: Step 1: Acquire and preprocess the multimodal data input by the user, and transmit the processed data to the server in real time; Step 2: Convert the multimodal data into a text sequence and process the text sequence in parallel: on the one hand, predict the user's psychological state and update the sentiment state vector through sentiment analysis; on the other hand, trigger the knowledge retrieval process. Step 3: Based on the text sequence, retrieve structured knowledge from the pre-built knowledge graph and convert it into natural language; Step 4: Generate response text based on structured knowledge converted into natural language and sentiment state vectors; generate synthesized speech and a sequence of visual pixels aligned with the timestamp of the synthesized speech based on the response text and sentiment state vectors. Step 5: Simultaneously send the emotion state vector, synthesized speech, and a sequence of visual pixels aligned with the timestamp of the synthesized speech to the client to drive the digital human model to render facial expressions and lip movements in real time, and play the synthesized speech synchronously.

[0021] In step 1, the multimodal data input by the user is acquired and preprocessed, and the processed data is transmitted to the server in real time. Specifically: Voice data is collected through a microphone, and noise reduction and automatic gain control (AGC) are performed using AudioContext based on front-end JavaScript. DataChannel, based on WebRTC, establishes a low-latency transmission channel to transmit processed data to the server. Compared to WebSocket, DataChannel is based on the SCTP protocol and supports out-of-order transmission, which greatly reduces the head-of-line blocking effect of real-time voice streams. In addition, the front end monitors the volume threshold in real time. Once a valid voice action (VAD) is detected, it is sliced ​​and encapsulated into a binary frame and sent to the server.

[0022] In step 2, the multimodal data is converted into a text sequence, specifically as follows: The server-side gRPC interface receives the processed data and pushes it to the ASR model. The ASR model outputs a text sequence based on the ASR model, which adopts the Conformer architecture, combining the local feature extraction capability of CNN and the global context modeling capability of Transformer to output a text sequence.

[0023] In step 2, the text sequence is processed in parallel, specifically as follows: The text sequence is input into the sentiment analysis module and the retrieval module in parallel. The sentiment analysis module predicts the current user's SCL-90 dimension score based on the text sequence using the BERT model and updates the sentiment state vector (E_state).

[0024] In step 3, based on the text sequence, structured knowledge is retrieved from the pre-built knowledge graph and converted into natural language, specifically as follows: Identify at least one entity from a text sequence; Using the identified entities as anchor points, multi-hop subgraph retrieval is performed in the pre-constructed knowledge graph (this invention uses the Neo4j graph database) to obtain structured knowledge associated with the entities; the algorithm of this invention starts from the anchor entity, walks along the edge for 2 steps (2-hop), and collects all adjacent nodes and relation descriptions; If a macro-level issue is involved, a community summary retrieval based on the Leiden algorithm will be triggered. The retrieved structured knowledge is transformed into natural language context.

[0025] In step 4, the response text is generated based on the structured knowledge converted into natural language and the sentiment state vector, specifically as follows: The structured knowledge and sentiment state vectors converted into natural language are injected into the LLM's prompt; LLM generates a response text based on the Prompt.

[0026] In step 4, synthesized speech and a sequence of visual pixels aligned with the timestamp of the synthesized speech are generated based on the response text and the sentiment state vector. Specifically: The TTS service obtains the response text and sentiment vector, and generates synthesized speech and a sequence of visions aligned with the timestamp of the synthesized speech based on the SoVITS model.

[0027] In step 5, the emotional state vector, synthesized speech, and a sequence of visual pixels aligned with the timestamp of the synthesized speech are simultaneously sent to the client to drive the digital human model to perform real-time rendering of facial expressions and lip movements, and to play the synthesized speech synchronously. Specifically: The emotional state vector, synthesized speech, and a sequence of visual pixels aligned with the timestamp of the synthesized speech are simultaneously delivered via WebRTC. The client web application receives the data sent down and updates the fixed points of the digital human model through the Live2D engine. Specifically, it updates the facial expression parameters based on the emotional state vector, updates the lip shape parameters based on the synthesized speech, and plays the synthesized speech synchronously. Ultimately, the user's screen will be presented with a digital human figure that can accurately answer professional questions and express appropriate emotions according to the context.

[0028] Next, we will provide a detailed introduction to the equipment or services involved in the method, step by step: 1. Users input voice messages or directly enter text information via microphone. The front end receives audio and text data in real time.

[0029] 2. During data transmission, the front-end uses the WebSocket protocol to forward received audio and text data to the back-end server. Specifically, it employs WebRTC, an open-source protocol that supports peer-to-peer (P2P) real-time audio and video streaming and data transmission. Unlike the traditional client-server model, WebRTC establishes a direct connection between the browser or device, achieving low-latency, high-real-time communication, making it suitable for scenarios such as audio and video calls and screen sharing.

[0030] 3. Using ASR conversion, the ASR module receives audio data transmitted from the backend, performs speech recognition processing, and sends the recognized text back to the backend for further processing. This invention provides an embodiment to illustrate the implementation process in the ASR module: (1) Acquisition of speech signals: capture raw speech signals through devices such as microphones.

[0031] (2) Audio preprocessing: The audio data format is one-dimensional discrete sampling points. The audio data is read and preprocessed based on torchaudio. The model input is processed by a feature extractor to extract features, and then further extracted after encoding.

[0032] (3) Text preprocessing: The text data format is characters. The text in character form is converted into discrete numbers. The input text is converted into a vector form that the model can process to obtain a continuous tensor.

[0033] (4) Acoustic model construction: Based on deep learning frameworks such as TensorFlow and PyTorch, an acoustic model based on LSTM and CNN is constructed.

[0034] (5) Language model application: Use an n-gram-based language model to capture the statistical relationships between words and guide the decoding of the ASR system.

[0035] (6) Decoding and output: Data is passed from the input layer to the output layer, and the corresponding text is decoded from the audio.

[0036] 4. RAG processing is employed. The RAG module processes the user input received from the backend, uses the LLM module for information retrieval, and sends the generated response text back to the backend for speech synthesis. This invention provides an embodiment illustrating the implementation process of the RAG section as follows: (1) Intent understanding: Establish an effective connection between user questions and knowledge in the knowledge base, and use the embedding model to automatically convert the data in the knowledge base and the keywords entered by the user into comparable high-dimensional vectors.

[0037] (2) Information retrieval: Effectively match user queries with relevant content in the knowledge base.

[0038] (3) Generation: Effectively integrate retrieved knowledge and user input questions, and let LLMs generate the corresponding output. In terms of model selection, support OpenAI's API interface and utilize various LLMs, including Ollama, Claude_llm, OpenAI, etc. Integrate pre-trained LLMs into the system. 5. TTS Generation: Receive the response text from the backend and generate highly natural audio data, which is sent to the frontend via the WebSocket protocol. The TTS generation implementation scheme adopts Generative Pre-trained Transformer (GPT) and SoVITS (So-Visit Transformer) technology, achieving high-quality speech cloning and text-to-speech conversion with a small amount of sample data. Among them: (1) Few samples: With 1 minute of training data, users can fine-tune the model to improve the similarity and realism of the voices.

[0039] (2) Voice cloning: Through training, it can learn and replicate the voice features of a specific speaker, thereby generating synthesized speech that is extremely similar to the voice of the target speaker. 6. Live2D Rendering: The Live2D renderer receives data from the front end and controls the digital human's expressions and movements to achieve dynamic visual feedback. Simultaneously, the front end plays audio, synchronized with the digital human's animation effects.

[0040] Furthermore, the present invention provides an embodiment, specifically as follows: 1. The emotion modeling and recognition module uses the PAD model. An introduction to the PAD model follows: The PAD model (also known as the VAD model: Valence-Arousal-Dominance) describes emotion as a coordinate vector in a continuous space, rather than a discrete label. In this alternative, the emotional core of the digital human consists of three orthogonal continuous dimensions: (1) Pleasure / Valence: measures the positive or negative aspect of emotions (e.g., pain vs. ecstasy).

[0041] (2) Arousal / Activation: measures the energy level of emotions (e.g., drowsiness vs. extreme excitement).

[0042] Dominance / Control: Measures an individual's sense of control over a situation (e.g., helplessness vs. dominance).

[0043] 2. The knowledge retrieval and reasoning architecture adopts Fine-Tuning (model fine-tuning internalization). This approach internalizes knowledge into the model parameters by performing full fine-tuning or efficient parameter fine-tuning (PEFT / LoRA) on large-scale domain-specific data.

[0044] 3. Visual rendering and animation are driven by Wav2Lip / SadTalker (GAN / Diffusion2D driven) based on a single static photo, using generative adversarial networks (GAN) to generate mouth and facial movements.

[0045] The input consists of an ID photo and a voice recording. The processing involves the model injecting voice features into the latent space of the face image and generating lip-synced video frames through a decoder.

[0046] Embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0047] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0048] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0049] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0050] Contents not described in detail in this specification are prior art known to those skilled in the art. It is hereby indicated that the above description is intended to help those skilled in the art understand this invention, but does not limit the scope of protection of this invention. Any equivalent substitutions, modifications, improvements, or simplifications of the above descriptions that do not depart from the essential content of this invention fall within the scope of protection of this invention.

Claims

1. A method for constructing AI digital humans that integrates knowledge graphs and emotional rendering, characterized in that, include: Step 1: Acquire and preprocess the multimodal data input by the user, and transmit the processed data to the server in real time; Step 2: Convert the multimodal data into a text sequence and process the text sequence in parallel: on the one hand, predict the user's psychological state and update the sentiment state vector through sentiment analysis; on the other hand, trigger the knowledge retrieval process. Step 3: Based on the text sequence, retrieve structured knowledge from the pre-built knowledge graph and convert it into natural language; Step 4: Generate response text based on structured knowledge converted into natural language and sentiment state vectors; generate synthesized speech and a sequence of visual pixels aligned with the timestamp of the synthesized speech based on the response text and sentiment state vectors. Step 5: Simultaneously send the emotion state vector, synthesized speech, and a sequence of visual pixels aligned with the timestamp of the synthesized speech to the client to drive the digital human model to render facial expressions and lip movements in real time, and play the synthesized speech synchronously.

2. The method for constructing an AI digital human integrating knowledge graph and emotional rendering according to claim 1, characterized in that, In step 1, the multimodal data input by the user is acquired and preprocessed, and the processed data is transmitted to the server in real time. Specifically: Voice data is collected via microphone, and noise reduction and automatic gain control are performed using AudioContext based on front-end JavaScript. DataChannel, based on WebRTC, establishes a low-latency transmission channel to transmit processed data to the server.

3. The method for constructing an AI digital human integrating knowledge graph and emotional rendering according to claim 2, characterized in that, In step 2, the multimodal data is converted into a text sequence, specifically as follows: The server-side gRPC interface receives the processed data, pushes it to the ASR model, and outputs a text sequence based on the ASR model.

4. The method for constructing an AI digital human integrating knowledge graph and emotional rendering according to claim 3, characterized in that, In step 2, the text sequence is processed in parallel, specifically as follows: The text sequence is input into the sentiment analysis module and the retrieval module in parallel. The sentiment analysis module predicts the current user's SCL-90 dimension score based on the text sequence using the BERT model and updates the sentiment state vector.

5. The method for constructing an AI digital human integrating knowledge graph and emotional rendering according to claim 4, characterized in that, In step 3, based on the text sequence, structured knowledge is retrieved from the pre-built knowledge graph and converted into natural language, specifically as follows: Identify at least one entity from a text sequence; Using the identified entities as anchor points, multi-hop subgraph retrieval is performed in the pre-built knowledge graph to obtain structured knowledge associated with the entities; The retrieved structured knowledge is converted into natural language.

6. The method for constructing an AI digital human integrating knowledge graph and emotional rendering according to claim 5, characterized in that, In step 4, the response text is generated based on the structured knowledge converted into natural language and the sentiment state vector, specifically as follows: The structured knowledge and sentiment state vectors converted into natural language are injected into the LLM's prompt; LLM generates a response text based on the Prompt.

7. The method for constructing an AI digital human integrating knowledge graph and emotional rendering according to claim 6, characterized in that, In step 4, synthesized speech and a sequence of visual pixels aligned with the timestamp of the synthesized speech are generated based on the response text and the sentiment state vector. Specifically: The TTS service obtains the response text and sentiment vector, and generates synthesized speech and a sequence of visual pixels aligned with the timestamp of the synthesized speech based on the SoVITS model.

8. The method for constructing an AI digital human integrating knowledge graph and emotional rendering according to claim 7, characterized in that, In step 5, the emotional state vector, synthesized speech, and a sequence of visual pixels aligned with the timestamp of the synthesized speech are simultaneously sent to the client to drive the digital human model to perform real-time rendering of facial expressions and lip movements, and to play the synthesized speech synchronously. Specifically: The emotional state vector, synthesized speech, and a sequence of visual pixels aligned with the timestamp of the synthesized speech are simultaneously delivered via WebRTC. The client-side web application receives the data sent down and updates the fixed points of the digital human model through the Live2D engine. Specifically, it updates the facial expression parameters based on the emotional state vector, updates the lip shape parameters based on the synthesized speech, and plays the synthesized speech synchronously.

Images