Virtual digital human generation and interaction optimization system based on multi-modal data fusion

Abstract

The invention relates to the technical field of artificial intelligence and virtual reality, and discloses a virtual digital human generation and interaction optimization system based on multi-modal data fusion, and the system comprises a multi-modal input fusion module which is used for collecting and fusing multi-modal data from audio, video and touch; the adaptive behavior adjustment module is used for adaptively adjusting the behavior and response strategy of the virtual digital human according to the user input and the interaction history; the real-time dynamic rendering module is used for generating and rendering an image of the virtual digital human and synchronizing actions, expressions and language output of the virtual digital human in real time; the personalized customization interface module is used for providing personalized customization functions of the appearance, the sound and the behavior of the virtual digital human for the user; and the system integration and optimization module is used for integrating the modules. By fusing data (such as audio, video and touch) of multiple modes, the system can comprehensively understand behaviors and emotional states of the user, interaction delay is reduced, and the user can experience a smoother interaction process.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and virtual reality technology, and in particular to a virtual digital human generation and interaction optimization system based on multimodal data fusion. Background Art

[0002] With the rapid development of artificial intelligence (AI) and virtual reality (VR) technologies, virtual human technology has been widely used in many fields, including online customer service, educational counseling, telemedicine, interactive entertainment, etc. These virtual digital humans interact with users by simulating human appearance, behavior and language, aiming to provide a natural communication experience similar to that between people. However, the existing virtual digital human system still has many technical limitations, and it is difficult to meet user expectations in terms of realism, naturalness and diversity.

[0003] The current implementation of virtual digital human technology mainly relies on single-modal data input, such as audio or video, which limits the system's ability to understand and process diverse user input. In voice-driven virtual digital human systems, voice input is used to generate corresponding facial expressions and body movements, but this single-modal input cannot fully capture the user's emotional and behavioral details, making it difficult for virtual digital humans to achieve natural and smooth interaction. In addition, because the system cannot effectively combine multimodal data, such as facial expressions, body movements, and touch feedback, existing virtual digital humans have significant deficiencies in expression synchronization, movement coordination, and response speed, resulting in a poor sense of immersion in the user experience. Summary of the invention

[0004] In view of the shortcomings of the existing technology, the present invention provides a virtual digital human generation and interaction optimization system based on multimodal data fusion, which solves the problem that the implementation of current virtual digital human technology mainly relies on single-modal data input, limiting the system's ability to understand and process diversified user input.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: A virtual digital human generation and interaction optimization system based on multimodal data fusion, comprising: Multimodal input fusion module, used to collect and fuse data from multiple modalities such as audio, video, and touch; An adaptive behavior adjustment module, used to adaptively adjust the behavior and response strategy of the virtual digital human according to user input and interaction history; Real-time dynamic rendering module, used to generate and render the image of virtual digital human, and synchronize its movements, expressions and language output in real time; A personalized customization interface module is used to provide users with personalized customization functions for the appearance, voice and behavior of virtual digital humans; The system integration and optimization module is used to integrate various modules and optimize them through performance monitoring and parameter tuning.

[0006] Preferably, the multimodal input fusion module includes: The data acquisition and preprocessing unit is used to collect raw data including audio, video and touch through microphones, cameras and touch screens, and perform noise reduction, background removal and gesture recognition preprocessing on the collected raw data; A feature extraction unit for extracting modal features from audio, video and touch data using a convolutional neural network (CNN) and a time series analysis method; The multimodal fusion unit is used to fuse the extracted features of each modality through time synchronization and attention mechanism.

[0007] Preferably, the data acquisition and preprocessing unit includes: An audio data acquisition subunit, used to collect voice data through a microphone and pre-process the voice data through a noise reduction algorithm; The video data acquisition subunit is used to capture facial expressions and body movements through a camera, and perform background removal and contour detection processing on the video data; The touch data acquisition subunit is used to capture user gesture data through the touch screen and perform time series analysis to identify the user's interaction intention.

[0008] Preferably, the feature extraction unit comprises: An audio feature extraction subunit, for extracting spectrogram and mel-spectrogram features from speech using a convolutional neural network (CNN); A video feature extraction subunit, for extracting facial and body features from video data using a convolutional neural network (CNN); The touch feature extraction subunit is used to extract the spatial and temporal features of the touch gesture based on time series analysis.

[0009] Preferably, the multimodal fusion unit comprises: The feature alignment subunit is used to perform time synchronization alignment on data of different modalities; The multimodal fusion strategy subunit is used to map each modal information to a unified feature space through an attention mechanism or a multimodal feature fusion method based on a graph convolutional network (GCN).

[0010] Preferably, the adaptive behavior adjustment module includes: A state representation unit, used to represent the interaction history and environment between the user and the virtual digital human; A reinforcement learning strategy selection unit, used to select the response strategy of the virtual digital human according to the state-action-reward model; The strategy updating and optimization unit is used to continuously optimize the behavior strategy of virtual digital humans through parameter updating and experience replay technology.

[0011] Preferably, the reinforcement learning strategy selection unit includes: The state-action-reward modeling subunit is used to build a state-action-reward model and use the deep Q network DQN for policy optimization; The behavior library and strategy selection subunit are used to select the most suitable behavior mode according to the current state and user input.

[0012] Preferably, the real-time dynamic rendering module includes: A virtual digital human modeling unit, used to establish the image characteristics of a virtual digital human through 3D modeling and pre-training; A dynamic rendering unit, which is used to quickly generate images of virtual digital humans using a high-performance graphics processing unit (GPU) and real-time ray tracing technology; The dynamic adjustment and synchronization unit is used to dynamically adjust the posture, expression and action of the virtual digital human according to the user input, and synchronize the language output with the action.

[0013] Preferably, the personalized customization interface module includes: A user interface design unit, which is used to provide a customized editor for appearance, sound and behavior, so that users can adjust the appearance characteristics of the virtual digital human; The parameter adjustment and saving unit is used for users to adjust parameters and preview the effects in real time, as well as to save and load users' personalized settings.

[0014] Preferably, the system integration and optimization module includes: Module integration unit, which is used to integrate various modules into a unified platform and use standard communication protocols and interfaces to ensure efficient communication and data exchange between modules; The performance monitoring and optimization unit is used to monitor the operating status in real time and automatically adjust parameters based on the monitoring results.

[0015] The present invention provides a virtual digital human generation and interaction optimization system based on multimodal data fusion, which has the following beneficial effects: 1. By integrating data from multiple modalities (such as audio, video, and touch), the system can fully understand the user's behavior and emotional state. Compared with traditional systems that rely on a single data modality, multimodal fusion significantly improves the naturalness of the interaction between virtual digital humans and users. At the same time, using efficient data processing and fusion technology, the system can respond to user input in real time, reduce interaction delays, and allow users to experience a smoother interaction process.

[0016] 2. The system of the present invention continuously learns the user's interaction history through reinforcement learning algorithms, thereby optimizing the behavior strategy of virtual digital humans and making them more personalized and adaptive. By dynamically adjusting behavior patterns (such as friendliness, humor, care, etc.), virtual digital humans can effectively establish emotional connections with users, provide personalized service experiences, and enhance user satisfaction and long-term engagement.

[0017] 3. The system of the present invention uses advanced rendering technologies such as GPU and real-time ray tracing to achieve synchronous rendering of virtual digital humans and user input, ensuring that the expressions and movements of virtual digital humans are natural and smooth. Combined with precise lip sync and body posture adjustment, virtual digital humans can express complex emotions in detail, enhance the realism and consistency of the interactive process, and give users a more immersive interactive experience.

[0018] 4. The modular design of the system of the present invention enables it to adapt to different application scenarios, including customer service, educational guidance and interactive entertainment, etc., which not only meets the diverse needs of users but also improves the commercial application scope of virtual digital people. In addition, through the personalized customization interface, users can adjust the appearance, voice and behavior of virtual digital people according to their needs, making them suitable for different occasions, further enhancing the practicality and market appeal of virtual digital people. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a system architecture diagram of the virtual digital human generation and interaction optimization system based on multimodal data fusion of the present invention; Figure 2 Schematic diagram of the multimodal input fusion module architecture of the present invention; Figure 3 It is a schematic diagram of the data acquisition and preprocessing unit architecture of the present invention; Figure 4 It is a schematic diagram of the feature extraction unit architecture of the present invention; Figure 5 It is a schematic diagram of the multimodal fusion unit architecture of the present invention; Figure 6 It is a schematic diagram of the architecture of the adaptive behavior adjustment module of the present invention; Figure 7 A schematic diagram of the reinforcement learning strategy selection unit architecture of the present invention; Figure 8 This is a schematic diagram of the real-time dynamic rendering module architecture of the present invention; Fig. 9 This is a schematic diagram of the personalized customization interface module architecture of the present invention; Fig.10 It is a schematic diagram of the system integration and optimization module architecture of the present invention. DETAILED DESCRIPTION

[0020] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Please see attached Figure 1 - Attachment Fig.10 The embodiment of the present invention provides a virtual digital human generation and interaction optimization system based on multimodal data fusion, including: Multimodal input fusion module, used to collect and fuse data from multiple modalities such as audio, video, and touch; An adaptive behavior adjustment module, used to adaptively adjust the behavior and response strategy of the virtual digital human according to user input and interaction history; Real-time dynamic rendering module, used to generate and render the image of virtual digital human, and synchronize its movements, expressions and language output in real time; A personalized customization interface module is used to provide users with personalized customization functions for the appearance, voice and behavior of virtual digital humans; The system integration and optimization module is used to integrate various modules and optimize them through performance monitoring and parameter tuning.

[0022] Specifically, the system's multimodal input fusion module combines audio, video and touch data, and uses multimodal convolution and attention mechanisms to fuse multimodal data. This fusion method enables the virtual digital human to fully perceive the user's state and understand the user's emotions and intentions by combining the characteristics of different modalities. This directly improves the naturalness and diversity of the virtual digital human's interaction, allowing it to respond more naturally to user input.

[0023] The multimodal input fusion module includes: The data acquisition and preprocessing unit is used to collect raw data including audio, video and touch through microphones, cameras and touch screens, and perform noise reduction, background removal and gesture recognition preprocessing on the collected raw data; A feature extraction unit for extracting modal features from audio, video and touch data using a convolutional neural network (CNN) and a time series analysis method; The multimodal fusion unit is used to fuse the extracted features of each modality through time synchronization and attention mechanism.

[0024] The data acquisition and preprocessing unit includes: An audio data acquisition subunit, used to collect voice data through a microphone and pre-process the voice data through a noise reduction algorithm; The video data acquisition subunit is used to capture facial expressions and body movements through a camera, and perform background removal and contour detection processing on the video data; The touch data acquisition subunit is used to capture user gesture data through the touch screen and perform time series analysis to identify the user's interaction intention.

[0025] The feature extraction unit includes: An audio feature extraction subunit, for extracting spectrogram and mel-spectrogram features from speech using a convolutional neural network (CNN); A video feature extraction subunit, for extracting facial and body features from video data using a convolutional neural network (CNN); The touch feature extraction subunit is used to extract the spatial and temporal features of the touch gesture based on time series analysis.

[0026] The multimodal fusion unit includes: The feature alignment subunit is used to perform time synchronization alignment on data of different modalities; The multimodal fusion strategy subunit is used to map each modal information to a unified feature space through an attention mechanism or a multimodal feature fusion method based on a graph convolutional network (GCN).

[0027] Specifically, the role of the multimodal input fusion module is to fuse data from multiple sensors so that the virtual digital human can fully understand the user's behavior and emotional state, thereby conducting natural and efficient interactions.

[0028] Data acquisition and preprocessing unit Audio data acquisition subunit: Use microphone to collect user's voice input, pre-process the voice data through adaptive filtering and background noise suppression technology, remove environmental noise, and ensure the accuracy of subsequent voice analysis. Noise reduction processing uses multi-stage filters, combined with frequency-domain-based noise suppression methods, which is particularly suitable for real-time interaction in different background environments.

[0029] Video data acquisition subunit: collects real-time video data of the user's face and body through a high-definition camera. During the video data acquisition process, background removal is first performed, and unnecessary background information is removed through image segmentation technology to ensure that the user's expression and movement are extracted more accurately. The use of skin color segmentation and contour detection methods can effectively locate the user's facial features, facilitating subsequent expression recognition.

[0030] Touch data acquisition subunit: Use the touch screen to collect user gesture input, such as click, slide, and pinch. The data preprocessing part uses time series analysis technology to identify the starting and ending points of the gesture, and combines the position and time features to identify the specific gesture type. For complex gestures, an algorithm based on pattern recognition is also introduced to enhance the accuracy of gesture classification.

[0031] Feature Extraction Unit Audio feature extraction subunit: Use convolutional neural networks (CNNs) to extract spectrograms and mel-spectrograms from speech data to analyze the pitch, emotion, and speech content of the speech. LSTM networks are used to process time series features to capture the user's emotional changes from the temporal dynamics of the speech. Through these feature extractions, the virtual digital human can understand the user's basic intentions and emotional state.

[0032] Video feature extraction subunit: Uses convolutional neural network (CNN) to extract user facial and body features, including the location information of facial features, expression type, body movements, etc. Through the key point detection algorithm, 68 facial feature points can be captured to express facial changes of different emotions. At the same time, body movement detection technology is used to identify whether the user has positive or negative body postures.

[0033] Touch feature extraction subunit: Uses convolutional networks combined with time series analysis to perform multi-level feature extraction on the user's touch data. First, identify the temporal characteristics of the touch trajectory, such as speed and acceleration, and then use spatial features to determine the specific type of gesture. Through feature extraction, the system can fully understand the user's interaction methods and intentions on the touch screen.

[0034] Multimodal fusion unit Feature alignment subunit: To ensure the fusion effect of multimodal data, the audio, video and touch data are first synchronized in time, and the correspondence of multimodal data on the time axis is ensured by feature alignment technology. The synchronization technology based on cross-correlation is used to handle the time delay between different modalities to achieve seamless data connection.

[0035] Multimodal fusion strategy subunit: Use the attention mechanism to fuse multimodal data to effectively combine the features of each modality and improve the accuracy and expression ability of data fusion. Use the graph convolutional network (GCN) to map the information of each modality into a unified feature space to ensure that the fused features can fully describe the user's behavior and emotional state.

[0036] The adaptive behavior adjustment module includes: A state representation unit, used to represent the interaction history and environment between the user and the virtual digital human; A reinforcement learning strategy selection unit, used to select the response strategy of the virtual digital human according to the state-action-reward model; The strategy updating and optimization unit is used to continuously optimize the behavior strategy of virtual digital humans through parameter updating and experience replay technology.

[0037] The reinforcement learning strategy selection unit includes: The state-action-reward modeling subunit is used to build a state-action-reward model and use the deep Q network DQN for policy optimization; The behavior library and strategy selection subunit are used to select the most suitable behavior mode according to the current state and user input.

[0038] Specifically, the adaptive behavior adjustment module is mainly responsible for adjusting the behavior of the virtual digital human according to the user's input data to achieve more personalized and adaptive interaction.

[0039] Status display unit Interaction history state representation subunit: By recording the interaction history between the user and the virtual digital human, and using the embedding representation method to encode the interaction information into a vector form for subsequent behavioral decision-making. The interaction state library stores information such as the user's behavior patterns and emotional changes, so that the system can comprehensively consider historical data and provide more personalized responses.

[0040] Environmental modeling subunit: Models the user's physical and social environment, extracts environmental features such as background sound, light brightness, etc., and combines these environmental factors to adjust the virtual digital human's response. By introducing environmental modeling, the virtual digital human can make adaptive behavioral adjustments in different scenarios, such as adjusting the volume of speech in a quiet environment.

[0041] Reinforcement Learning Strategy Selection Unit State-Action-Reward Modeling Subunit: Establishes a State-Action-Reward model to guide the behavior selection of virtual digital humans. The system uses a deep Q network (DQN) to optimize the model so that the virtual digital human can learn the optimal strategy from past interactions to improve the user experience. For example, based on the user's feedback (reward), adjust the expression or conversation content in the interaction.

[0042] Behavior library and strategy selection subunit: Build a behavior library, including multiple interaction modes, such as friendly, formal, humorous, etc. Based on the user's input and the current situation, the system will select the most suitable behavior mode from the behavior library. For example, when it is detected that the user is in a low mood, the "caring" behavior mode can be selected to provide positive feedback.

[0043] Strategy Update and Optimization Unit Parameter update subunit: Based on user feedback, the system will continuously adjust the parameters of the behavior strategy. The model parameters are updated through the gradient descent algorithm to optimize the effect of the interaction. The system can dynamically adjust the interaction mode, for example, when the user shows discomfort with a certain behavior, the strategy will be adjusted quickly.

[0044] Experience replay subunit: Use experience replay technology to store the interaction process between users and virtual digital humans. Train the reinforcement learning model by randomly extracting historical interaction records to improve learning efficiency and model stability. This method can help the system find better behavior strategies in different user scenarios.

[0045] The real-time dynamic rendering module includes: A virtual digital human modeling unit, used to establish the image characteristics of a virtual digital human through 3D modeling and pre-training; A dynamic rendering unit, which is used to quickly generate images of virtual digital humans using a high-performance graphics processing unit (GPU) and real-time ray tracing technology; The dynamic adjustment and synchronization unit is used to dynamically adjust the posture, expression and action of the virtual digital human according to the user input, and synchronize the language output with the action.

[0046] Specifically, the real-time dynamic rendering module ensures the synchronization of the visual performance of the virtual digital human and the user interaction process to enhance the sense of reality.

[0047] Virtual Human Modeling Unit Image feature pre-training subunit: Pre-train the facial model of the virtual digital human through a large-scale image data set to learn the contours of the facial features and detailed features of the face. Use 3D modeling technology to build the skeletal structure of the virtual digital human, so that the virtual digital human can simulate human facial movements and expression changes.

[0048] Dynamic Rendering Unit Real-time rendering engine subunit: Uses high-performance graphics processing units (GPUs) and real-time ray tracing technology to render images of virtual digital humans. The real-time rendering engine can quickly update the expressions and movements of virtual digital humans based on user input to ensure synchronization with user interactions, such as generating lip shapes that match user voices in real time.

[0049] Expression and action generation subunit: Use joint-driven animation methods to generate the actions and expressions of virtual digital humans. For example, when a user smiles, the virtual digital human will respond in real time and make corresponding expression changes. By adjusting the motion trajectory of the skeletal joints, the movements of the virtual digital human are ensured to be natural and smooth.

[0050] Dynamic adjustment and synchronization unit Posture adjustment subunit: Uses the inverse kinematics (IK) algorithm to adjust the posture of the virtual digital human to ensure that its movements conform to the natural laws of the human body. The system can dynamically change the body posture of the virtual digital human according to the user's instructions (such as nodding, gestures).

[0051] Language and action synchronization subunit: Through Lip-sync technology, the lip synchronization of the virtual digital human when speaking is achieved, and combined with the synchronous adjustment of facial expressions and body language, the overall performance of the virtual digital human is more natural and realistic.

[0052] The personalized interface module includes: A user interface design unit, which is used to provide a customized editor for appearance, sound and behavior, so that users can adjust the appearance characteristics of the virtual digital human; The parameter adjustment and saving unit is used for users to adjust parameters and preview the effects in real time, as well as to save and load users' personalized settings.

[0053] Specifically, the personalized customization interface module allows users to personalize the virtual digital human according to their own preferences.

[0054] User Interface Design Unit Appearance Editor Subunit: Users can use the Appearance Editor to adjust the appearance of the virtual digital human, such as hairstyle, clothing, and skin color. A real-time preview function is provided, so users can view the adjustment effects at any time during the editing process so that they can make choices that suit their own aesthetics.

[0055] Voice and Behavior Editor Subunit: Users can select the voice of the virtual digital person, including pitch, volume, and speaking speed. In addition, users can set different behavioral styles for the virtual digital person, such as humorous, formal, or friendly, to suit the needs of different occasions.

[0056] Parameter adjustment and storage unit Real-time parameter adjustment subunit: Users can adjust the appearance, sound and behavior parameters of the virtual digital human in real time through the interface. The system will instantly apply these changes and generate the corresponding virtual image.

[0057] Parameter saving and loading subunit: Users can save their own personalized settings and reload them when needed. For example, users can save specific configurations for specific occasions or interaction types, so that they can be directly called next time.

[0058] System integration and optimization modules include: Module integration unit, which is used to integrate various modules into a unified platform and use standard communication protocols and interfaces to ensure efficient communication and data exchange between modules; The performance monitoring and optimization unit is used to monitor the operating status in real time and automatically adjust parameters based on the monitoring results.

[0059] Specifically, the system integration and optimization module ensures the efficient integration of each subsystem and the optimization of the overall system.

[0060] Module integrated unit Unified integrated platform subunit: Using modular design methods, various functional modules are integrated into a unified platform. The system uses standardized interfaces (such as RESTful API) to communicate data between modules, ensuring that modules can work together seamlessly and share data.

[0061] Communication protocol and interface standardization subunit: defines the communication protocol between modules to ensure the stability and efficiency of data transmission. Standardized interface design is adopted to ensure that the system can be expanded to integrate more functional modules or connect external devices.

[0062] Performance Monitoring and Optimization Unit Performance monitoring subunit: The system has built-in performance monitoring tools to monitor the system's operating status in real time, including response time, resource utilization, etc. The monitoring results are presented through a graphical interface, making it easier for developers and operation and maintenance personnel to understand the system's operating status in a timely manner.

[0063] Parameter tuning and performance optimization subunit: Based on the performance monitoring results, the system's operating parameters are automatically adjusted to ensure optimal performance under different hardware and network environments. Through load balancing and resource management technology, the system's stability and response speed are improved to enable it to adapt to complex operating conditions.

[0064] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A virtual digital human generation and interaction optimization system based on multimodal data fusion, characterized in that: include: Multimodal input fusion module, used to collect and fuse data from multiple modalities such as audio, video, and touch; An adaptive behavior adjustment module, used to adaptively adjust the behavior and response strategy of the virtual digital human according to user input and interaction history; Real-time dynamic rendering module, used to generate and render the image of virtual digital human, and synchronize its movements, expressions and language output in real time; A personalized customization interface module is used to provide users with personalized customization functions for the appearance, voice and behavior of virtual digital humans; The system integration and optimization module is used to integrate various modules and optimize them through performance monitoring and parameter tuning.

2. The virtual digital human generation and interaction optimization system based on multimodal data fusion according to claim 1 is characterized in that: The multimodal input fusion module includes: The data acquisition and preprocessing unit is used to collect raw data including audio, video and touch through microphones, cameras and touch screens, and perform noise reduction, background removal and gesture recognition preprocessing on the collected raw data; A feature extraction unit for extracting modal features from audio, video and touch data using a convolutional neural network (CNN) and a time series analysis method; The multimodal fusion unit is used to fuse the extracted features of each modality through time synchronization and attention mechanism.

3. The virtual digital human generation and interaction optimization system based on multimodal data fusion according to claim 2 is characterized in that: The data acquisition and preprocessing unit comprises: An audio data acquisition subunit, used to collect voice data through a microphone and pre-process the voice data through a noise reduction algorithm; The video data acquisition subunit is used to capture facial expressions and body movements through a camera, and perform background removal and contour detection processing on the video data; The touch data acquisition subunit is used to capture user gesture data through the touch screen and perform time series analysis to identify the user's interaction intention.

4. The virtual digital human generation and interaction optimization system based on multimodal data fusion according to claim 2 is characterized in that: The feature extraction unit comprises: An audio feature extraction subunit, for extracting spectrogram and mel-spectrogram features from speech using a convolutional neural network (CNN); A video feature extraction subunit, for extracting facial and body features from video data using a convolutional neural network (CNN); The touch feature extraction subunit is used to extract the spatial and temporal features of the touch gesture based on time series analysis.

5. The virtual digital human generation and interaction optimization system based on multimodal data fusion according to claim 2 is characterized in that: The multimodal fusion unit comprises: The feature alignment subunit is used to perform time synchronization alignment on data of different modalities; The multimodal fusion strategy subunit is used to map each modal information to a unified feature space through an attention mechanism or a multimodal feature fusion method based on a graph convolutional network (GCN).

6. The virtual digital human generation and interaction optimization system based on multimodal data fusion according to claim 1 is characterized in that: The adaptive behavior adjustment module comprises: A state representation unit, used to represent the interaction history and environment between the user and the virtual digital human; A reinforcement learning strategy selection unit, used to select the response strategy of the virtual digital human according to the state-action-reward model; The strategy updating and optimization unit is used to continuously optimize the behavior strategy of virtual digital humans through parameter updating and experience replay technology.

7. The virtual digital human generation and interaction optimization system based on multimodal data fusion according to claim 6 is characterized in that: The reinforcement learning strategy selection unit comprises: The state-action-reward modeling subunit is used to build a state-action-reward model and use the deep Q network DQN for policy optimization; The behavior library and strategy selection subunit are used to select the most suitable behavior mode according to the current state and user input.

8. The virtual digital human generation and interaction optimization system based on multimodal data fusion according to claim 1 is characterized in that: The real-time dynamic rendering module includes: A virtual digital human modeling unit, used to establish the image characteristics of a virtual digital human through 3D modeling and pre-training; A dynamic rendering unit, which is used to quickly generate images of virtual digital humans using a high-performance graphics processing unit (GPU) and real-time ray tracing technology; The dynamic adjustment and synchronization unit is used to dynamically adjust the posture, expression and action of the virtual digital human according to the user input, and synchronize the language output with the action.

9. The virtual digital human generation and interaction optimization system based on multimodal data fusion according to claim 1 is characterized in that: The personalized customization interface module includes: A user interface design unit, which is used to provide a customized editor for appearance, sound and behavior, so that users can adjust the appearance characteristics of the virtual digital human; The parameter adjustment and saving unit is used for users to adjust parameters and preview the effects in real time, as well as save and load users' personalized settings.

10. The virtual digital human generation and interaction optimization system based on multimodal data fusion according to claim 1 is characterized in that: The system integration and optimization module includes: Module integration unit, which is used to integrate various modules into a unified platform and use standard communication protocols and interfaces to ensure efficient communication and data exchange between modules; The performance monitoring and optimization unit is used to monitor the operating status in real time and automatically adjust parameters based on the monitoring results.

Images