Digital human generation method based on multi-modal large model

Abstract

The disclosure provides a digital human generation method based on a multi-modal large model, comprising: constructing a digital human basic model; generating a structured training set; generating a question and answer model supporting multi-channel interaction; outputting a semantic answer to a user question, extracting a text emotional tendency of the semantic answer, and outputting an emotional intensity parameter; generating facial muscle movement trajectory data, and performing real-time rendering on the digital human basic model according to the facial muscle movement trajectory data to output a digital human three-dimensional image with emotional expression. Through cross-modal alignment of text, image and audio data, and by using a joint training method to optimize a multi-modal large model containing a visual, speech and knowledge model, a more natural and smooth multi-channel interaction experience is realized. In addition, by introducing an emotional recognition model and a facial interaction model, the emotional tendency implied in the semantic answer can be more accurately captured and reflected, thereby outputting a digital human three-dimensional image with real emotional expression.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, storage medium, and electronic device for generating digital humans based on a multimodal large model. Background Technology

[0002] With the development of artificial intelligence technology, digital humans have been widely used in entertainment, education, customer service, and many other fields. However, existing digital human generation methods often face several key problems: First, the lack of effective cross-modal data processing capabilities results in insufficient integration between visual, speech, and knowledge models, making it difficult to achieve a high-quality multi-channel interactive experience; second, the realism and subtlety of emotional expression are insufficient, especially in the generation of facial expressions, making it difficult to accurately capture and reflect the emotional changes in user questions, thus affecting the user experience. Summary of the Invention

[0003] The purpose of this disclosure is to provide a method, apparatus, storage medium, and electronic device for generating digital humans based on a multimodal large model, in order to solve the problems existing in the prior art.

[0004] The embodiments of this disclosure adopt the following technical solution: a digital human generation method based on a multimodal large model, comprising: constructing a digital human basic model based on a 3D human body model and a text-to-speech synthesis model; acquiring multimodal data in a target domain, unifying text, image, and audio data in the multimodal data using a cross-modal alignment model, and generating a structured training set; jointly training the multimodal large model containing visual, speech, and knowledge models based on the structured training set to generate a question-answering model supporting multi-channel interaction; acquiring user questions, and outputting semantic answers to the user questions through the question-answering model, extracting the textual sentiment tendency of the semantic answers through an emotion recognition model, and outputting an emotion intensity parameter; inputting the semantic answers and the emotion intensity parameter into a facial interaction model to generate facial muscle movement trajectory data, and rendering the digital human basic model in real time based on the facial muscle movement trajectory data to output a 3D digital human image with emotional expression.

[0005] In some embodiments, the construction of a digital human basic model based on a three-dimensional human body model and a text-to-speech synthesis model includes: generating an anonymized three-dimensional human body mesh model through a generative adversarial network, and randomizing the height, shoulder width, and facial contour parameters using differential privacy technology; constructing a standardized timbre feature library, generating speech samples containing five basic timbres for both men and women using a text-to-speech synthesis model, and achieving scene-specific adaptation of pitch and speech rate parameters through a timbre transfer model; and spatiotemporally aligning the three-dimensional human body mesh model with the standardized timbre feature library to generate a digital human basic model that supports automatic synchronization of speech and lip movements.

[0006] In some embodiments, the step of acquiring multimodal data in the target domain and unifying the text, image, and audio data in the multimodal data using a cross-modal alignment model to generate a structured training set includes: extracting text knowledge data, image knowledge data, and audio knowledge data from the knowledge base of the target domain, and removing privacy data from the knowledge data; using a contrastive learning model to perform feature space mapping on the text knowledge data and the image knowledge data, and performing Mel-spectrum transformation on the audio knowledge data to achieve synchronous timestamp alignment of text-image-audio, thereby generating a structured training set.

[0007] In some embodiments, the method further includes: applying a random region mask to the image knowledge data, performing synonym replacement on the text knowledge data, and generating a structured training set that includes noise resistance.

[0008] In some embodiments, the joint training of a multimodal large model containing visual, speech, and knowledge models based on the structured training set to generate a question-answering model supporting multi-channel interaction includes: a visual generation model training step: associating the text knowledge data and image knowledge data through a cross-attention mechanism; a speech processing model training step: using an end-to-end training method, loading a pronunciation rule library, and generating tone patterns and professional terminology stress marks that conform to the target domain scenario; and a knowledge reasoning model training step: encoding the text knowledge data into graph neural network nodes, generating a diagnostic logic chain through multi-hop reasoning, and performing gradient joint updates with the visual generation model and the speech processing model.

[0009] In some embodiments, the steps of obtaining user questions and outputting semantic answers to user questions through the question-answering model, and extracting the textual sentiment tendency of the semantic answers through a sentiment recognition model to output sentiment intensity parameters include: using a BERT model to decompose the semantic answers into sentiment dimensions and outputting probability distribution vectors containing different emotion categories; using the OpenSmile toolkit to extract fundamental frequency jitter, speech rate change rate, and formant offset indices to generate temporal variation curves of sentiment intensity parameters; and using a temporal convolutional network to dynamically weight the text sentiment probability and the speech sentiment curve to generate an 8-dimensional sentiment encoding vector that drives facial expressions.

[0010] In some embodiments, the step of inputting the semantic response and the emotion intensity parameter into a facial interaction model to generate facial muscle movement trajectory data, and then rendering the digital human base model in real time based on the facial muscle movement trajectory data to output a digital human 3D image with emotional expression, includes: inputting the emotion encoding vector into a 3D deformation model to generate a muscle contraction intensity matrix containing 52 facial action units; calculating the skin vertex displacement based on the muscle contraction intensity matrix, and achieving physical consistency between eye movement and head posture through rigid body dynamics simulation; generating high-fidelity hair and pupil details using a neural radiation field engine, and compensating for facial texture distortion caused by rapid lip movements using optical flow.

[0011] This disclosure also provides a digital human generation device based on a multimodal large model, comprising: a basic model construction module for constructing a basic digital human model based on a 3D human body model and a text-to-speech synthesis model; a training set generation module for acquiring multimodal data in a target domain, unifying text, image, and audio data in the multimodal data using a cross-modal alignment model, and generating a structured training set; a question-answering model training module for jointly training the multimodal large model containing visual, speech, and knowledge models based on the structured training set, generating a question-answering model supporting multi-channel interaction; an interaction module for acquiring user questions, outputting semantic answers to the user questions through the question-answering model, extracting the textual sentiment tendency of the semantic answers through an emotion recognition model, and outputting an emotion intensity parameter; and a real-time rendering module for inputting the semantic answers and the emotion intensity parameter into a facial interaction model to generate facial muscle movement trajectory data, and rendering the basic digital human model in real time based on the facial muscle movement trajectory data to output a 3D digital human image with emotional expression.

[0012] This disclosure also provides a storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described digital human generation method based on a multimodal large model.

[0013] This disclosure also provides an electronic device, including at least a memory and a processor. The memory stores a computer program, and the processor executes the computer program in the memory to implement the steps of the above-described digital human generation method based on a multimodal large model.

[0014] The beneficial effects of this disclosure are as follows: The digital human generation method provided in this embodiment significantly improves the interactivity and emotional realism of digital humans. By performing cross-modal alignment of text, image, and audio data, and optimizing a multimodal large model including visual, speech, and knowledge models using a joint training method, a more natural and fluent multi-channel interactive experience is achieved. In addition, by introducing an emotion recognition model and a facial interaction model, the emotional tendencies contained in semantic responses can be captured and reflected more accurately, thereby outputting a three-dimensional digital human image with realistic emotional expression, greatly enhancing the immersion and satisfaction of the user experience. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating a method for generating digital humans based on a multimodal large model, provided for one or more embodiments of this specification;

[0017] Figure 2 This is a schematic diagram of the structure of a digital human generation device based on a multimodal large model, provided for one or more embodiments of this specification. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0019] With the development of artificial intelligence technology, digital humans have been widely used in entertainment, education, customer service, and many other fields. However, existing digital human generation methods often face several key problems: First, the lack of effective cross-modal data processing capabilities results in insufficient integration between visual, speech, and knowledge models, making it difficult to achieve a high-quality multi-channel interactive experience; second, the realism and subtlety of emotional expression are insufficient, especially in the generation of facial expressions, making it difficult to accurately capture and reflect the emotional changes in user questions, thus affecting the user experience.

[0020] To address the aforementioned issues, the first embodiment of this disclosure provides a digital human generation method based on a multimodal large model, the flowchart of which is shown below. Figure 1 As shown, it mainly includes steps S10 to S50:

[0021] S10 constructs a basic model of a digital human based on a 3D human body model and a text-to-speech synthesis model.

[0022] First, based on the needs of digital human generation, a basic model is constructed to facilitate the use of different fields and purposes. The general basic model can be used to learn specific domain knowledge to meet personalized usage needs.

[0023] In the actual construction of digital human basic models, Generative Adversarial Networks (GANs) can be used to generate anonymized 3D human body mesh models. GAN is a deep learning technique consisting of a generator and a discriminator. The generator is responsible for generating realistic-looking samples, while the discriminator attempts to distinguish between real and generated samples. During this process, differential privacy techniques are used to randomize parameters such as height, shoulder width, and facial contours to protect user privacy. In some embodiments, the randomization of parameters such as height and shoulder width can involve adding Laplacian noise within ±5cm of the original values; for facial contours, a k-anonymization algorithm can be used to blur the distance between cheekbones and the angle of the jaw, ensuring that it is impossible to reverse-identify an individual.

[0024] Subsequently, a standardized timbre feature library is constructed as the speech foundation for generating digital humans. To achieve this goal, this embodiment records speech samples with various timbres from different speakers and stores them in a database. Audio analysis tools are used to extract the acoustic features of each speech sample, such as fundamental frequency, formants, and speech rate. Then, a text-to-speech (TTS) model is used to generate speech samples containing five basic timbres for both male and female speakers. Through a timbre transfer model, parameters such as pitch and speech rate are adjusted according to different application scenarios to better meet actual needs.

[0025] Finally, the 3D human body mesh model is spatiotemporally aligned with a standardized timbre feature library to generate a basic digital human model that supports automatic lip-syncing. In actual execution, a corresponding timestamp can be assigned to each speech sample and matched with the lip movements in the 3D human body mesh model. The Dynamic Time Warping (DTW) algorithm is then used to synchronize the 3D model with the speech.

[0026] S20: Acquire multimodal data in the target domain, and use a cross-modal alignment model to unify text, image, and audio data in the multimodal data to generate a structured training set.

[0027] To meet the practical needs of digital humans, they require relevant knowledge data in different target domains to satisfy user requirements. Therefore, it is necessary to acquire and organize corresponding knowledge data for different target domains as the data foundation for digital human learning. In this embodiment, multimodal data, including textual, image, and audio knowledge data, is first extracted from the target domain's knowledge base. Privacy data, such as patient personal information in the medical field and user information in the service industry, is then removed to ensure user privacy. During knowledge data extraction, the data sources can be public sources such as literature, websites, and databases, or content from a private database held by the party constructing the digital human. For privacy removal, Natural Language Processing (NLP) and Computer Vision (CV) technologies can be used to identify and remove sensitive information from the data.

[0028] For text, image, and audio knowledge data with different characteristics, a cross-modal approach is used to unify their models and generate a structured training set. Specifically, a contrastive learning model is used to map the feature spaces of text and image knowledge data, mapping them to the same feature space. The contrastive loss function of the CLIP model minimizes the cosine distance between the text and image feature vectors to complete the alignment operation between text and images. For audio knowledge data, Mel-spectrum transformation is performed, and the text, image, and audio data are synchronized and timestamped according to the chronological order of events to generate a structured training set.

[0029] In some embodiments, noise can be added when generating a structured training set to improve the robustness of subsequent model training and avoid overfitting. Specifically, noise can be added by applying random region masks to image knowledge data, performing synonym replacement on text knowledge data, or using back translation to perturb the text between Chinese and English, thereby generating a structured training set with noise resistance.

[0030] S30 uses a structured training set to jointly train a large multimodal model that includes visual, speech, and knowledge models, generating a question-answering model that supports multi-channel interaction.

[0031] The structured training set can be used to train a multimodal large model to generate a question-answering model that supports multi-channel interaction, serving as the "brain" of the digital human and providing users with intelligent interactive question-answering functions. The question-answering model in this embodiment mainly includes the following parts: a visual generation model, a speech processing model, and a knowledge reasoning model. The visual generation model generates a dynamic image of the digital human through 3D modeling and motion capture technology, including real-time rendering of facial expressions and body movements, such as precise synchronization of lip movements and speech waveforms. It also combines text semantic understanding to generate visual content, achieving cross-modal association between text and images. The speech processing model's main task is to convert text into speech, providing auditory feedback to the user. This model also needs to have speech recognition capabilities, understanding the user's speech input and converting it into a processable text format. The knowledge reasoning model encodes a structured knowledge base into graph neural network nodes, supporting multi-hop logical reasoning. It uses natural language processing technology to parse user questions, generating semantic answers that conform to domain knowledge. It is also responsible for understanding and remembering contextual information in the dialogue to provide coherent answers in subsequent interactions.

[0032] During the training of the visual generative model, a cross-attention mechanism is used to associate text and image data. This means that when generating images, the model simultaneously considers related text information to ensure that the generated content is closely related to the user's question. The speech processing model is trained end-to-end, directly from input data to output results. It uses a standard pronunciation rule library and the OpenSmile toolkit to extract metrics such as fundamental frequency jitter, speech rate change rate, and formant shift, generating temporal variation curves of emotional intensity parameters to obtain tone patterns and professional terminology stress markers that conform to the target domain scenario. For the knowledge reasoning model, the training process first encodes the textual knowledge data into graph neural network nodes to form a knowledge graph containing various knowledge points and their relationships. Then, multi-hop reasoning generates diagnostic logic chains to help answer complex questions. Finally, gradient joint updates are performed with the visual generative model and the speech processing model to ensure consistency and synergy among the modules.

[0033] S40: Obtain user questions, output semantic answers to user questions through question-answering models, extract the textual sentiment tendency of semantic answers through sentiment recognition models, and output sentiment intensity parameters.

[0034] When users actually use digital humans to solve their problems, they ask questions via voice or text. After the question is acquired, it is analyzed and answered by a question-answering model, generating semantic responses that meet the user's expectations within the target domain. However, before the digital human displays these responses, emotional processing is required to provide a more human-like experience for the user. First, the BERT model is used to decompose the semantic responses into emotional dimensions, outputting probability distribution vectors containing different emotion categories. In this embodiment, the emotion categories include at least calm, joy, anger, sadness, and fear. Then, the OpenSmile toolkit is used to extract fundamental frequency jitter, speech rate change rate, and formant shift indices to generate a temporal variation curve of emotional intensity parameters. The fundamental frequency jitter reflects the speaker's emotional fluctuations, the speech rate change rate analyzes the trend of speech rate changes and captures rapid changes in emotion, and the formant shift index detects the movement of formant positions and assesses changes in emotional state. Finally, a temporal convolutional network is used to dynamically weight the text emotional probability and the speech emotional curve, dynamically adjusting the weights according to different emotional intensities to generate an 8-dimensional emotion encoding vector that drives facial expressions.

[0035] S50 inputs semantic responses and emotional intensity parameters into the facial interaction model to generate facial muscle movement trajectory data. Based on the facial muscle movement trajectory data, it performs real-time rendering of the digital human basic model to output a digital human 3D image with emotional expression.

[0036] After the above steps, the facial movements required for the digital human to actually respond can be generated using semantic responses and emotional intensity parameters. This ensures that the digital human's facial expressions and mouth shapes correspond to the text and are not delayed when responding to the user. In this embodiment, the semantic responses and emotional intensity parameters are first input into the facial interaction model. The facial interaction model is built based on a 3D deformation model and includes a muscle contraction intensity matrix of 52 facial action units. An 8-dimensional emotion encoding vector driving facial expressions is used to calculate the contraction intensity of each facial action unit. Then, based on the muscle contraction intensity, the displacement of the skin surface is calculated to simulate real facial expression changes. Rigid body dynamics simulation is used to achieve physical consistency between eye movements and head posture, ensuring the naturalness and smoothness of the overall movement. Finally, a neural radiation field engine is used to generate high-fidelity hair and pupil details. An optical flow algorithm is used to compensate for the distortion of facial textures, maintaining the realism and clarity of facial expressions, and outputting a three-dimensional digital human image with emotional expression.

[0037] The digital human generation method provided in this embodiment significantly improves the interactivity and emotional realism of digital humans. By performing cross-modal alignment of text, image, and audio data, and optimizing a multimodal large model including visual, speech, and knowledge models using a joint training method, a more natural and fluid multi-channel interactive experience is achieved. Furthermore, by introducing emotion recognition and facial interaction models, the emotional tendencies inherent in semantic responses can be captured and reflected more accurately, thereby outputting a 3D digital human image with realistic emotional expression, greatly enhancing the immersion and satisfaction of the user experience. In some embodiments, corresponding clothing and movement designs can be applied to the basic digital human model for different target domains to further enhance the interactivity and immersion of the user experience.

[0038] Based on the same inventive concept, the second embodiment of this disclosure provides a digital human generation device based on a multimodal large model, the structural schematic diagram of which is shown below. Figure 2 As shown, it mainly includes: a basic model construction module 10, used to construct a basic digital human model based on a 3D human body model and a text-to-speech synthesis model; a training set generation module 20, used to acquire multimodal data in the target domain, and use a cross-modal alignment model to unify text, image, and audio data in the multimodal data to generate a structured training set; a question-answering model training module 30, used to jointly train a large multimodal model containing visual, speech, and knowledge models based on the structured training set to generate a question-answering model that supports multi-channel interaction; an interaction module 40, used to acquire user questions, output semantic answers to user questions through a question-answering model, extract the text sentiment tendency of the semantic answers through an emotion recognition model, and output emotion intensity parameters; and a real-time rendering module 50, used to input the semantic answers and emotion intensity parameters into a facial interaction model to generate facial muscle movement trajectory data, and render the basic digital human model in real time based on the facial muscle movement trajectory data to output a 3D digital human image with emotional expression.

[0039] In some embodiments, the basic model building module 10 is specifically used to: generate an anonymized three-dimensional human body mesh model through a generative adversarial network, and randomize the height, shoulder width, and facial contour parameters using differential privacy technology; construct a standardized timbre feature library, generate speech samples containing five basic timbres for both men and women using a text-to-speech synthesis model, and achieve scene-specific adaptation of pitch and speech rate parameters through a timbre transfer model; and spatiotemporally align the three-dimensional human body mesh model with the standardized timbre feature library to generate a digital human basic model that supports automatic synchronization of speech and lip movements.

[0040] In some embodiments, the training set generation module 20 is specifically used to: extract text knowledge data, image knowledge data, and audio knowledge data from a knowledge base in the target domain, and remove privacy data from the knowledge data; use a contrastive learning model to perform feature space mapping on the text knowledge data and image knowledge data, and perform Mel-spectrum transformation on the audio knowledge data to achieve synchronized timestamp alignment of text-image-audio, generating a structured training set. Further, it is also used to apply random region masks to the image knowledge data and perform synonym replacement on the text knowledge data to generate a structured training set with noise resistance capabilities.

[0041] In some embodiments, the question-answering model training module 30 is specifically used to perform the following training steps, including: a visual generation model training step: associating text knowledge data and image knowledge data through a cross-attention mechanism; a speech processing model training step: using an end-to-end training method, loading a pronunciation rule library, and generating tone patterns and professional terminology accent marks that conform to the target domain scenario; and a knowledge reasoning model training step: encoding text knowledge data into graph neural network nodes, generating diagnostic logic chains through multi-hop reasoning, and performing gradient joint updates with the visual generation model and the speech processing model.

[0042] In some embodiments, the interaction module 40 is specifically used to: decompose the semantic response into sentiment dimension using the BERT model and output a probability distribution vector containing different emotion categories; extract the fundamental frequency jitter, speech rate change rate, and formant offset indicators using the OpenSmile toolkit to generate a temporal variation curve of the sentiment intensity parameter; and dynamically weight the text sentiment probability and the speech sentiment curve using a temporal convolutional network to generate an 8-dimensional sentiment encoding vector that drives facial expressions.

[0043] In some embodiments, the real-time rendering module 50 is specifically used to: input the emotion encoding vector into the 3D deformation model to generate a muscle contraction intensity matrix containing 52 facial motion units; calculate the skin vertex displacement based on the muscle contraction intensity matrix, and achieve physical consistency between eye movement and head posture through rigid body dynamics simulation; generate high-fidelity hair and pupil details using a neural radiation field engine, and compensate for facial texture distortion caused by rapid lip movements using optical flow.

[0044] The digital human generation method provided in this embodiment significantly improves the interactivity and emotional realism of digital humans. By performing cross-modal alignment of text, image, and audio data, and optimizing a multimodal large model including visual, speech, and knowledge models using a joint training method, a more natural and fluid multi-channel interactive experience is achieved. In addition, by introducing an emotion recognition model and a facial interaction model, the emotional tendencies implied in semantic responses can be captured and reflected more accurately, thereby outputting a 3D digital human image with realistic emotional expression, greatly enhancing the immersion and satisfaction of the user experience.

[0045] Based on the same inventive concept, the third embodiment of this disclosure provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the digital human generation method based on a multimodal large model provided in the first embodiment of this disclosure.

[0046] Based on the same inventive concept, the fourth embodiment of this disclosure provides at least a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program in the memory, implements the steps of the digital human generation method based on a multimodal large model provided in the first embodiment of this disclosure.

[0047] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this disclosure.

Claims

1. A method for generating digital humans based on a multimodal large model, characterized in that, include: Based on the 3D human body model and the text-to-speech synthesis model, a basic model of digital human is constructed. Acquire multimodal data in the target domain, and use a cross-modal alignment model to unify the text, image, and audio data in the multimodal data to generate a structured training set; Based on the structured training set, a multimodal large model including visual, speech and knowledge models is jointly trained to generate a question-answering model that supports multi-channel interaction. The system acquires user questions and outputs semantic answers to them through the question-answering model. It also extracts the textual sentiment tendency of the semantic answers through a sentiment recognition model and outputs sentiment intensity parameters. The semantic response and the emotional intensity parameter are input into the facial interaction model to generate facial muscle movement trajectory data. Based on the facial muscle movement trajectory data, the digital human basic model is rendered in real time to output a digital human 3D image with emotional expression. The process of acquiring multimodal data in the target domain, unifying text, image, and audio data within the multimodal data using a cross-modal alignment model, and generating a structured training set includes: Text knowledge data, image knowledge data, and audio knowledge data are extracted from the knowledge base of the target domain, and privacy data in the knowledge data is removed. A contrastive learning model is used to perform feature space mapping on the text knowledge data and the image knowledge data, and Mel spectrum transformation is performed on the audio knowledge data to achieve synchronized timestamp alignment of text, image and audio, generating a structured training set.

2. The digital human generation method according to claim 1, characterized in that, The construction of a basic digital human model based on a 3D human body model and a text-to-speech synthesis model includes: Anonymized 3D human body mesh models are generated using generative adversarial networks, and differential privacy technology is used to randomize the parameters of height, shoulder width, and facial contour. A standardized timbre feature library is constructed, and a text-to-speech synthesis model is used to generate speech samples containing five basic timbres for both men and women. A timbre transfer model is used to achieve contextual adaptation of pitch and speech rate parameters. The three-dimensional human body mesh model is spatiotemporally aligned with a standardized timbre feature library to generate a basic digital human model that supports automatic synchronization of speech and lip movements.

3. The digital human generation method according to claim 1, characterized in that, Also includes: A random region mask is applied to the image knowledge data, and synonym replacement is performed on the text knowledge data to generate a structured training set with noise resistance capabilities.

4. The digital human generation method according to claim 1, characterized in that, The joint training of a multimodal large model containing visual, speech, and knowledge models based on the structured training set to generate a question-answering model supporting multi-channel interaction includes: Visual generative model training steps: The text knowledge data and image knowledge data are associated through a cross-attention mechanism; Speech processing model training steps: adopt an end-to-end training method, load the pronunciation rule library, and generate tone patterns and professional terminology stress marks that conform to the target domain scenario; Knowledge reasoning model training steps: Encode the text knowledge data into graph neural network nodes, generate diagnostic logic chains through multi-hop reasoning, and perform gradient joint updates with the visual generation model and the speech processing model.

5. The digital human generation method according to any one of claims 1 to 4, characterized in that, The process involves acquiring user questions, outputting semantic answers to the user questions through the question-answering model, extracting the textual sentiment tendency of the semantic answers through a sentiment recognition model, and outputting sentiment intensity parameters, including: The BERT model is used to decompose the semantic response into sentiment dimension, and the output is a probability distribution vector containing different emotion categories; The OpenSmile toolkit is used to extract indicators such as fundamental frequency jitter, speech rate change rate, and formant offset to generate time-series variation curves of emotion intensity parameters. A temporal convolutional network is used to dynamically weight the text sentiment probability and the speech sentiment curve to generate an 8-dimensional sentiment encoding vector that drives facial expressions.

6. The digital human generation method according to claim 5, characterized in that, The process of inputting the semantic response and the emotional intensity parameters into the facial interaction model to generate facial muscle movement trajectory data, and then rendering and outputting a digital human 3D image with emotional expression based on the facial muscle movement trajectory data in real time, includes: The emotion encoding vector is input into a 3D deformation model to generate a muscle contraction intensity matrix containing 52 facial motion units; The displacement of the skin vertex is calculated based on the muscle contraction intensity matrix, and the physical consistency between eye movement and head posture is achieved through rigid body dynamics simulation. The neural radiation field engine generates high-fidelity hair and pupil details, and optical flow compensates for facial texture distortion caused by rapid lip movements.

7. A digital human generation device based on a multimodal large model, characterized in that, include: The basic model building module is used to build a basic model of a digital human based on a 3D human body model and a text-to-speech synthesis model. The training set generation module is used to acquire multimodal data in the target domain, and use a cross-modal alignment model to unify the text, image and audio data in the multimodal data to generate a structured training set. The question-answering model training module is used to jointly train a multimodal large model containing visual, speech and knowledge models based on the structured training set, and generate a question-answering model that supports multi-channel interaction. The interaction module is used to obtain user questions, output semantic answers to user questions through the question-answering model, extract the text sentiment tendency of the semantic answers through the sentiment recognition model, and output sentiment intensity parameters. The real-time rendering module is used to input the semantic response and the emotional intensity parameters into the facial interaction model, generate facial muscle movement trajectory data, and perform real-time rendering of the digital human basic model based on the facial muscle movement trajectory data to output a digital human 3D image with emotional expression. The training set generation module is specifically used to: extract text knowledge data, image knowledge data, and audio knowledge data from the knowledge base of the target domain, and remove privacy data from the knowledge data; A contrastive learning model is used to perform feature space mapping on the text knowledge data and the image knowledge data, and Mel spectrum transformation is performed on the audio knowledge data to achieve synchronized timestamp alignment of text, image and audio, generating a structured training set.

8. A storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the digital human generation method based on a multimodal large model as described in any one of claims 1 to 6.

9. An electronic device, comprising at least a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program on the memory, it implements the steps of the digital human generation method based on a multimodal large model as described in any one of claims 1 to 6.

Images