A virtual digital human intelligent generation method based on multi-modal fusion

Abstract

The application discloses a kind of based on multi-modal fusion virtual digital human intelligent generation method and system thereof, including extracting corresponding data features to generate first single modal data, second single modal data and third single modal data;Through multi-modal feature fusion network, carry out feature level multi-modal fusion, obtain multi-modal data features;The multi-modal data features are traversed matching with the pre-stored data interaction knowledge base, obtain corresponding data question and answer article and the first interactive question and answer information of associated derivative article, and respond to the first interactive question and answer information, trigger and obtain the second interactive question and answer information of real-time reflection article information adaptability;The interactive voice information of initial virtual digital person is constructed;Using the first single modal data, second single modal data, third single modal data and interactive voice information, using DiffGAN fusion model generates initial virtual digital person, generates virtual digital person with high fidelity and natural interaction ability.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and machine learning technology, and specifically relates to a method for intelligent generation of virtual digital humans based on multimodal fusion. Background Technology

[0002] With the rapid development of artificial intelligence and machine learning technologies, the generation and interaction capabilities of virtual digital humans have been significantly improved, achieving a high degree of intelligent optimization in multiple fields. This enables virtual digital humans to more realistically simulate human appearance, behavior, and language interaction, leading to their widespread application in various industries such as entertainment, education, healthcare, and customer service.

[0003] Despite significant progress in virtual human generation technology, most existing methods rely on single-modal data, such as using only text, images, or audio data. In some cases, single-modal methods may fail to fully leverage the complementarity of multimodal data, thus limiting the quality and interactivity of the generated virtual humans. For example, relying solely on text data may fail to accurately capture physical features and voice intonation; while relying solely on image data may prevent the generation of virtual humans with natural language interaction capabilities. Furthermore, single-modal methods also have limitations when handling complex interactive scenarios. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for intelligent generation of virtual digital humans based on multimodal fusion. By integrating multiple modal data such as text, audio, and images, a virtual digital human with high realism and natural interaction capabilities can be generated.

[0005] To achieve the above objectives, the present invention is implemented using the following technical solution: In a first aspect, the present invention provides a method for intelligent generation of virtual digital humans based on multimodal fusion, characterized by comprising the following steps: Based on the collected text, audio, and image multimodal data, the corresponding data features are extracted to generate first monomodal data, second monomodal data, and third monomodal data; By using a multimodal feature fusion network, feature-level multimodal fusion is performed on the first unimodal data, the second unimodal data, and the third unimodal data to obtain multimodal data features; The multimodal data features are traversed and matched with a pre-stored data interaction knowledge base to obtain the first interactive question and answer information of the corresponding data question and answer entries and their associated derivative entries. In response to the first interactive question and answer information, a second interactive question and answer information reflecting the adaptability of the entry information in real time is triggered and obtained. Based on the first interactive question-and-answer information and the second interactive question-and-answer information, the interactive voice information of the initial virtual digital human is constructed; Using the first monomodal data, the second monomodal data, the third monomodal data, and interactive voice information, an initial virtual digital human is generated using the DiffGAN fusion model.

[0006] Furthermore, by utilizing end-user input, preset scripts, or natural language interaction methods, text data associated with the rendering of the virtual digital human is collected, including character descriptions, dialogue content, and scene background; this data is used to extract scene features of the virtual digital human. Audio data associated with the rendering of the virtual digital human, including voice commands and background sound effects, is collected using a microphone or other audio devices. This audio data can be used to extract the voice and emotional characteristics of the virtual digital human. Using cameras or other image acquisition devices, image data associated with the rendering of the virtual digital human is captured, including photos of people and scene images. This image data can be used to extract the physical features of the virtual digital human. Based on the scene features, a preset application scenario is matched and associated, and a corresponding multimodal fusion strategy is selected based on the preset application scenario to obtain multimodal data features.

[0007] Furthermore, multimodal feature fusion based on weighted fusion strategies includes: Align the different modal features of the first unimodal data, the second unimodal data, and the third unimodal data into the same semantic space; Weights are assigned based on the importance of the multimodal data using an empirical method. Each modal feature is multiplied by its corresponding weight to obtain weighted data features, and then the weighted sum is used to generate multimodal data features.

[0008] The fused multimodal data features are normalized, and then the multimodal data features are traversed and matched with a pre-stored data interaction knowledge base.

[0009] Furthermore, using the DiffGAN fusion model, the initial virtual digital human is generated, including: Generative adversarial networks are used to generate the physical features and facial expressions of virtual digital humans. The diffusion model was used to generate lip movements and sounds for virtual digital humans.

[0010] Furthermore, it triggers and retrieves second interactive question-and-answer information that reflects the real-time adaptability of term information, including: Generate preliminary response information tags for the virtual digital human based on the first interactive question-and-answer information; Feedback information in response to the initial answer information tags is collected in real time using speech recognition and natural language processing technologies; Analyze and determine the suitability tendency of the feedback information: if it is negative, then based on the feedback information, obtain the question and answer entries that match the feedback information from the data interaction knowledge base as the second interactive question and answer information.

[0011] Furthermore, the interactive voice information of the initial virtual digital human is constructed, including: Integrate the first and second interactive question-and-answer information to extract keywords, semantic content, and sentiment. The extracted keywords and semantic content are used to generate natural and fluent response text using natural language processing models; The response text is generated into a speech signal using speech synthesis technology, and the tone, speed, and volume of the speech are adjusted in conjunction with the emotional tone.

[0012] Secondly, the present invention provides a virtual digital human intelligent generation system based on multimodal fusion, comprising: The multimodal data feature extraction module is used to extract corresponding data features from collected text, audio, images and other multimodal data to generate first unimodal data, second unimodal data and third unimodal data; The virtual digital human generation module is used to generate an initial virtual digital human based on the first monomodal data, the second monomodal data, and the third monomodal data using the DiffGAN fusion model; The multimodal feature fusion module is used to perform feature-level multimodal fusion on the first unimodal data, the second unimodal data, and the third unimodal data through a multimodal feature fusion network to obtain multimodal data features; The interactive question-and-answer information acquisition module is used to traverse and match the multimodal data features with a pre-stored data interaction knowledge base to obtain the first interactive question-and-answer information of the corresponding data question-and-answer terms and their associated derivative terms; at the same time, in response to the first interactive question-and-answer information, it triggers and acquires the second interactive question-and-answer information that reflects the adaptability of term information in real time. An interactive voice information construction module is used to construct the interactive voice information of an initial virtual digital human based on the first interactive question-and-answer information and the second interactive question-and-answer information.

[0013] Thirdly, the present invention provides an electronic device, comprising: Memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the virtual digital human intelligent generation method based on multimodal fusion as described in any of the first aspects.

[0014] Fourthly, the present invention provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the virtual digital human intelligent generation method based on multimodal fusion as described in any of the first aspects.

[0015] Fifthly, the present invention provides a computer program product comprising a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps in the multimodal fusion-based intelligent generation method for virtual digital humans as described in any of the first aspects.

[0016] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: The intelligent generation method and system for virtual digital humans based on multimodal fusion provided by this invention significantly improves the generation quality and realism of virtual digital humans by combining the complementarity of different modal data such as images, audio, and text through multimodal data fusion. Simultaneously, by utilizing speech recognition, natural language processing, and image recognition technologies, the performance of virtual digital humans in complex interactive scenarios is enhanced, providing a more natural and fluid interactive experience. Furthermore, through dynamic weight adjustment and real-time feedback mechanisms, virtual digital humans can flexibly adapt to different application scenarios and user needs, further optimizing interactive effects and improving user satisfaction. Attached Figure Description

[0017] Figure 1 A flowchart of a method for intelligent generation of virtual digital humans based on multimodal fusion, provided for an embodiment of the present invention. Detailed Implementation

[0018] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0019] Example

[0020] like Figure 1 As shown in the figure, this embodiment of the invention provides a method for intelligent generation of virtual digital humans based on multimodal fusion, including the following steps: Based on the collected text, audio, and image multimodal data, the corresponding data features are extracted to generate first monomodal data, second monomodal data, and third monomodal data; By using a multimodal feature fusion network, feature-level multimodal fusion is performed on the first unimodal data, the second unimodal data, and the third unimodal data to obtain multimodal data features; The multimodal data features are traversed and matched with a pre-stored data interaction knowledge base to obtain the first interactive question and answer information of the corresponding data question and answer entries and their associated derivative entries. In response to the first interactive question and answer information, a second interactive question and answer information reflecting the adaptability of the entry information in real time is triggered and obtained. Based on the first interactive question-and-answer information and the second interactive question-and-answer information, the interactive voice information of the initial virtual digital human is constructed; Using the first monomodal data, the second monomodal data, the third monomodal data, and interactive voice information, an initial virtual digital human is generated using the DiffGAN fusion model.

[0021] In some embodiments, text features can be extracted using NLP techniques, audio features can be extracted using acoustic models, and image features can be extracted using models such as CNNs.

[0022] By utilizing end-user input, preset scripts, or natural language interaction, text data associated with the rendering of virtual digital humans is collected, including character descriptions, dialogue content, and scene backgrounds; this data is used to extract scene features and semantic features of the virtual digital humans. Audio data associated with the rendering of the virtual digital human, including voice commands and background sound effects, is collected using a microphone or other audio devices. This audio data can be used to extract the voice and emotional characteristics of the virtual digital human. Using cameras or other image acquisition devices, image data associated with the rendering of the virtual digital human is captured, including photos of people and scene images. This image data can be used to extract the physical features of the virtual digital human. Furthermore, based on the scene features, a preset application scenario is matched and associated, and a corresponding multimodal fusion strategy is selected based on the preset application scenario to obtain multimodal data features.

[0023] In this embodiment, appropriate fusion strategies are selected based on different application scenarios, including weighted fusion, concatenation fusion, and deep learning fusion. Weighted fusion assigns weights based on the importance of different modalities; for example, in some scenarios, image features may be more important than text features, and the fusion result can be optimized by adjusting the weights. Concatenation fusion directly concatenates feature vectors from different modalities to form a high-dimensional feature vector, which is then further processed using a deep learning model. Deep learning fusion utilizes deep learning models (such as Transformer and GAN) for end-to-end fusion, automatically learning the complex relationships between different modalities.

[0024] In some embodiments, for the collected multimodal data of text, audio, and images, it is necessary to remove noise and invalid information from the multimodal data, convert multimodal data of different formats into a unified format, and extract feature vectors from the multimodal data.

[0025] For text data, cleaning and standardization processes are performed to remove irrelevant characters and stop words. Word embedding techniques (such as Word2Vec, GloVe, or BERT) are used to convert the text data into a high-dimensional vector representation, capturing the semantic information of the text. Natural Language Processing (NLP) techniques are then used to extract semantic, sentiment, and topic features of the text. For example, sentiment analysis models are used to extract the sentiment tendency of the text, and topic models are used to extract the topic distribution of the text. The extracted text features are then integrated into the first unimodal data.

[0026] For audio data, noise reduction and dereverberation are performed to improve audio quality. For example, short-time Fourier transform or Mel-spectrum analysis is used to extract spectral features of the audio, capturing its frequency and temporal information; acoustic models are used to extract speech features, including pitch, intensity, and duration; and emotion recognition models are used to extract emotional features, such as happiness, sadness, and anger. The extracted audio features are then integrated into a second monomodal dataset.

[0027] For image data, processing such as cropping, scaling, and normalization is performed. A pre-trained convolutional neural network is used to extract spatial features of the image, capturing the texture, shape, and color information of the image. Through feature alignment technology, image features are aligned with text and audio features into the same semantic space. Then, the extracted image features are integrated into third monomodal data.

[0028] In some embodiments, feature-level fusion using deep learning networks (such as the Transformer architecture) can effectively integrate data from different modalities.

[0029] In some embodiments, multimodal feature fusion based on a weighted fusion strategy includes: Align the different modal features of the first unimodal data, the second unimodal data, and the third unimodal data into the same semantic space; Weights are assigned based on the importance of the multimodal data using an empirical method. Each modal feature is multiplied by its corresponding weight to obtain weighted data features, and then the weighted sum is used to generate multimodal data features.

[0030] The fused multimodal data features are normalized, and then the multimodal data features are traversed and matched with a pre-stored data interaction knowledge base.

[0031] In some embodiments, an initial virtual digital human is generated using a DiffGAN fusion model, specifically a Generative Adversarial Network (GAN) and a diffusion model. The DiffGAN fusion model includes a generator and a discriminator. The generator is used to generate the appearance and behavioral characteristics of the virtual digital human, and the discriminator is used to evaluate the realism and naturalness of the generated result.

[0032] In this embodiment, the specific steps for generating the initial virtual digital human include: Generative adversarial networks are used to generate the physical features and facial expressions of virtual digital humans. The diffusion model was used to generate lip movements and sounds for virtual digital humans.

[0033] In some embodiments, triggering and obtaining second interactive question-and-answer information that reflects the real-time adaptability of term information includes: Generate preliminary response information tags for the virtual digital human based on the first interactive question-and-answer information; Feedback information in response to the initial answer information tags is collected in real time using speech recognition and natural language processing technologies; Analyze and determine the suitability tendency of the feedback information: if it is negative, then based on the feedback information, obtain the question and answer entries that match the feedback information from the data interaction knowledge base as the second interactive question and answer information.

[0034] In some embodiments, constructing the interactive voice information of the initial virtual digital human includes: Integrate the first and second interactive question-and-answer information to extract keywords, semantic content, and sentiment. The extracted keywords and semantic content are used to generate natural and fluent response text using natural language processing models; The response text is generated into a speech signal using speech synthesis technology, and the tone, speed, and volume of the speech are adjusted in conjunction with the emotional tone.

[0035] This invention also provides a virtual digital human intelligent generation system based on multimodal fusion, comprising: The multimodal data feature extraction module is used to extract corresponding data features from collected text, audio, images and other multimodal data to generate first unimodal data, second unimodal data and third unimodal data; The virtual digital human generation module is used to generate an initial virtual digital human based on the first monomodal data, the second monomodal data, and the third monomodal data using the DiffGAN fusion model; The multimodal feature fusion module is used to perform feature-level multimodal fusion on the first unimodal data, the second unimodal data, and the third unimodal data through a multimodal feature fusion network to obtain multimodal data features; The interactive question-and-answer information acquisition module is used to traverse and match the multimodal data features with a pre-stored data interaction knowledge base to obtain the first interactive question-and-answer information of the corresponding data question-and-answer terms and their associated derivative terms; at the same time, in response to the first interactive question-and-answer information, it triggers and acquires the second interactive question-and-answer information that reflects the adaptability of term information in real time. An interactive voice information construction module is used to construct the interactive voice information of an initial virtual digital human based on the first interactive question-and-answer information and the second interactive question-and-answer information.

[0036] In some embodiments, the multimodal data feature extraction module includes multiple units, such as: The text acquisition unit is used to collect the text description input by the user; The audio acquisition unit is used to acquire audio data input by the user. The image acquisition unit is used to acquire image data input by the user.

[0037] In some embodiments, the multimodal data feature extraction module is further provided with a video acquisition unit for acquiring video data input by the user.

[0038] The data cleaning unit is used to remove noise and invalid information from multimodal data; The format conversion unit is used to convert multimodal data of different formats into a unified format; The feature extraction unit is used to extract feature vectors from multimodal data, including text features, audio features, image features, etc., which are used as first unimodal data, second unimodal data, and third unimodal data.

[0039] In some embodiments, the multimodal feature fusion module includes a feature fusion unit and a fusion strategy unit.

[0040] The feature fusion unit is used to fuse the first single-modal data, the second single-modal data, and the third single-modal data to generate fused multimodal data features; The fusion strategy unit is used to select the appropriate fusion strategy according to different application scenarios, including weighted fusion, splicing fusion and deep learning fusion.

[0041] In some embodiments, the virtual digital human generation module includes a GAN generation unit, a diffusion model generation unit, and a generation optimization unit.

[0042] GAN generation units are used to generate the appearance features and facial expressions of virtual digital humans using generative adversarial networks; The diffusion model generation unit is used to generate the lip movements and sounds of a virtual digital human using a diffusion model. An optimization unit is used to optimize the generated virtual digital human.

[0043] In some embodiments, the interactive question-and-answer information acquisition module includes a real-time input receiving unit, an interaction driving unit, and an interaction optimization unit.

[0044] The real-time input receiving unit is used to receive real-time input from the user, including voice commands, gestures, etc. The interactive driving unit is used to drive the virtual digital human to make corresponding responses based on real-time input, including actions, facial expressions and voice feedback; The interaction optimization unit is used to optimize the interaction-driven unit based on real-time user feedback, thereby improving the naturalness and smoothness of the interaction.

[0045] In some embodiments, the interactive voice information construction module includes an information construction unit for constructing the interactive voice information of the initial virtual digital human based on the first interactive question-and-answer information and the second interactive question-and-answer information.

[0046] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for intelligent generation of virtual digital humans based on multimodal fusion, characterized in that, Includes the following steps: Based on the collected text, audio, and image multimodal data, the corresponding data features are extracted to generate first monomodal data, second monomodal data, and third monomodal data; By using a multimodal feature fusion network, feature-level multimodal fusion is performed on the first unimodal data, the second unimodal data, and the third unimodal data to obtain multimodal data features; The multimodal data features are traversed and matched with a pre-stored data interaction knowledge base to obtain the first interactive question and answer information of the corresponding data question and answer entries and their associated derivative entries. In response to the first interactive question and answer information, a second interactive question and answer information reflecting the adaptability of the entry information in real time is triggered and obtained. Based on the first interactive question-and-answer information and the second interactive question-and-answer information, the interactive voice information of the initial virtual digital human is constructed; Using the first monomodal data, the second monomodal data, the third monomodal data, and interactive voice information, an initial virtual digital human is generated using the DiffGAN fusion model.

2. The method for intelligent generation of virtual digital humans based on multimodal fusion according to claim 1, characterized in that, By utilizing end-user input, preset scripts, or natural language interaction methods, text data associated with the rendering of virtual digital humans is collected to extract scene features of the virtual digital humans. Audio data associated with the rendering of virtual digital humans is collected using microphones to extract the voice and emotional features of the virtual digital humans; Image data captured by a camera and associated with the rendering of a virtual digital human is used to extract the appearance features of the virtual digital human. Based on the scene features, a preset application scenario is matched and associated, and a corresponding multimodal fusion strategy is selected based on the preset application scenario to obtain multimodal data features.

3. The method for intelligent generation of virtual digital humans based on multimodal fusion according to claim 2, characterized in that, Multimodal feature fusion based on weighted fusion strategies includes: Align the different modal features of the first unimodal data, the second unimodal data, and the third unimodal data into the same semantic space; Based on the importance of multimodal data, weights are assigned using an empirical method. Each modal feature is multiplied by its corresponding weight to obtain weighted data features, and then the weighted sum is calculated to generate multimodal data features. The fused multimodal data features are normalized, and then the multimodal data features are traversed and matched with a pre-stored data interaction knowledge base.

4. The method for intelligent generation of virtual digital humans based on multimodal fusion according to claim 1, characterized in that, Using the DiffGAN fusion model, the initial virtual digital human was generated, including: Generative adversarial networks are used to generate the physical features and facial expressions of virtual digital humans. The diffusion model was used to generate the lip movements and voices of a virtual digital human.

5. The method for intelligent generation of virtual digital humans based on multimodal fusion according to claim 1, characterized in that, Trigger and retrieve second interactive question-and-answer information that reflects the real-time adaptability of term information, including: Generate preliminary response information tags for the virtual digital human based on the first interactive question-and-answer information; Real-time collection of feedback information in response to the initial answer information tags; Analyze and determine the suitability tendency of the feedback information: if it is negative, then based on the feedback information, obtain the question and answer entries that match the feedback information from the data interaction knowledge base as the second interactive question and answer information.

6. The method for intelligent generation of virtual digital humans based on multimodal fusion according to claim 5, characterized in that, Constructing the interactive voice information of the initial virtual digital human, including: Integrate the first and second interactive question-and-answer information to extract keywords, semantic content, and sentiment. The extracted keywords and semantic content are used to generate natural and fluent response text using natural language processing models; The response text is generated into a speech signal using speech synthesis technology, and the tone, speed, and volume of the speech are adjusted in conjunction with the emotional tone.

7. A virtual digital human intelligent generation system based on multimodal fusion, characterized in that, include: The multimodal data feature extraction module is used to extract corresponding data features from collected text, audio, images and other multimodal data to generate first unimodal data, second unimodal data and third unimodal data; The virtual digital human generation module is used to generate an initial virtual digital human based on the first monomodal data, the second monomodal data, and the third monomodal data using the DiffGAN fusion model; The multimodal feature fusion module is used to perform feature-level multimodal fusion on the first unimodal data, the second unimodal data, and the third unimodal data through a multimodal feature fusion network to obtain multimodal data features; The interactive question-and-answer information acquisition module is used to traverse and match the multimodal data features with a pre-stored data interaction knowledge base to obtain the first interactive question-and-answer information of the corresponding data question-and-answer terms and their associated derivative terms; at the same time, in response to the first interactive question-and-answer information, it triggers and acquires the second interactive question-and-answer information that reflects the adaptability of term information in real time. An interactive voice information construction module is used to construct the interactive voice information of an initial virtual digital human based on the first interactive question-and-answer information and the second interactive question-and-answer information.

8. An electronic device, characterized in that, include: Memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the virtual digital human intelligent generation method based on multimodal fusion as described in any one of claims 1-6.

9. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the virtual digital human intelligent generation method based on multimodal fusion as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps in the multimodal fusion-based virtual digital human intelligent generation method as described in any one of claims 1 to 6.

Images