Digital human interaction control method and device based on multi-modal

Abstract

The application discloses a multi-modal-based digital human interaction control method and device, and relates to the technical field of artificial intelligence digital human, and specifically comprises the following steps: collecting multi-source heterogeneous multi-modal data of user interaction with a digital human in real time; constructing a dynamic adaptive multi-modal space-time hypergraph Transformer network, deeply fusing the multi-source heterogeneous multi-modal data, and obtaining a fused interaction feature vector; generating a multi-dimensional interaction situation tensor based on the fused interaction feature vector; generating a digital human response action sequence, voice content, expression posture and interaction strategy according to the multi-dimensional interaction situation tensor, and driving the digital human to execute; acquiring feedback data of the digital human after executing the response in real time, and optimizing parameters of the dynamic adaptive multi-modal space-time hypergraph Transformer network based on the feedback data; and effectively improving the naturalness and response accuracy of digital human interaction, and reducing the interaction misunderstanding rate and user cognitive load.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence digital human technology, and more specifically to a multimodal digital human interaction control method and device. Background Technology

[0002] With the rapid development of applications such as metaverse, virtual assistants, intelligent customer service, and virtual anchors, digital humans have become an important carrier of human-computer interaction. Traditional digital human interaction mainly relies on a single modality, which suffers from problems such as partial understanding, loss of context, and lack of emotion, and cannot meet the needs of natural, immersive, and multi-turn interaction.

[0003] While existing technologies have made some progress in multimodal fusion, they still have the following drawbacks: (1) Multimodal data is difficult to achieve deep fusion under a unified spatiotemporal framework, and the modeling of intermodal correlations is insufficient; (2) Lack of dynamic global awareness of the interaction situation, unable to capture the user's immediate intent and long-term contextual evolution at the same time; (3) The response generation is static, making it difficult to adapt and adjust according to user emotions and scenarios; (4) The system lacks an effective closed-loop optimization mechanism and has poor long-term interactive adaptability.

[0004] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal digital human interaction control method and device to solve the problems of low interaction naturalness, inaccurate understanding of intent, and poor adaptability in the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal digital human interaction control method, specifically comprising: Real-time acquisition of multi-source heterogeneous and multimodal data on user interactions with digital humans; A dynamic adaptive multimodal spatiotemporal hypergraph Transformer network is constructed to deeply fuse the multi-source heterogeneous multimodal data and obtain a fused interactive feature vector. Based on the fused interactive feature vector, a multidimensional interactive situation tensor is generated; Based on the multidimensional interactive situation tensor, a digital human response action sequence, voice content, facial expressions and gestures, and interaction strategies are generated, and the digital human is driven to execute them. Real-time feedback data is acquired after the user responds to the digital human, and the parameters of the dynamic adaptive multimodal spatiotemporal hypergraph Transformer network are optimized based on the feedback data.

[0007] Preferably, as a preferred embodiment of the multimodal-based digital human interaction control method of the present invention, wherein: Based on the data type and source, establish a multi-channel parallel acquisition interface to connect to multi-source heterogeneous and multimodal data sources, specifically including: (1) Real-time acquisition of user video data and user posture data through cameras and depth cameras, including user facial expressions, eye movements, gestures, body posture and body movement videos; (2) Real-time acquisition of user voice data through microphone array, including voice content, intonation, emotional acoustic features and ambient sound; (3) Collect user text input data and operation command data in real time through text input interface, keyboard and touch screen; (4) Collect users’ physiological data in real time through wearable devices, including heart rate, skin conductance and eye movement trajectory; (5) Collect digital human current status data in real time, including current posture, expression, dialogue history, scene context and task objectives; Timestamp alignment for multi-source heterogeneous multimodal data; A unified spatiotemporal reference framework is constructed based on interaction time, interaction rounds, and user spatial coordinates, and the multi-source heterogeneous multimodal data is labeled with a unified spatiotemporal tag. The multi-source heterogeneous multimodal data undergoes preliminary cleaning, noise reduction, and format standardization, including: Keyframe extraction and preliminary detection and annotation of face and pose are performed on the user video data; The user's voice data is subjected to speech recognition, preliminary emotion classification, and smoothing processing. When the missing rate of the multi-source heterogeneous data exceeds a preset threshold of 3%, a supplementary data collection instruction is automatically triggered, and the data quality assessment is fed back to the closed-loop optimization.

[0008] Preferably, as a preferred embodiment of the multimodal-based digital human interaction control method of the present invention, wherein: Mapping the multi-source heterogeneous multimodal data to a spatiotemporal reference frame specifically includes: Spatiotemporal visual features of user video data and user posture data are extracted using the Video-Swin Transformer model; Temporal acoustic and semantic features of user speech data are extracted using the Conformer model. Extract semantic features from user text input data using the BERT model; Physiological signal features and contextual state features are extracted using graph convolutional networks; A unified spatiotemporal location code and dialogue turn code are added to each extracted data feature to construct a multi-source heterogeneous multimodal feature set; Initialize the node set based on digital human joints, user key points, and interactive objects; Based on the hierarchical characteristics of the interaction scenario, a multi-layered hypergraph containing local, meso, and global scales is constructed. The learnable adaptive edge generator dynamically generates and updates the hyperedge weights and structure in real time based on the multi-source heterogeneous multimodal feature set. The multi-layer hypergraph is input into a dynamic adaptive multimodal spatiotemporal hypergraph Transformer network for hierarchical attention calculation, including: (1) Local fine-grained attention layer: focuses on the facial expressions, gestures and voice of the current frame in the multi-source heterogeneous multimodal data to capture the user's real-time intentions; (2) Mesoscale contextual attention layer: Integrates multi-source heterogeneous multimodal data within the current dialogue round to assess the user's complex intent; (3) Global conversation attention layer: Modeling the history of multiple dialogue turns, long-term user preferences, and the evolution trend of interaction scenarios; By using gated fusion and multi-head hypergraph attention mechanisms, the interaction relationships of the multi-source heterogeneous multimodal feature set in the spatiotemporal and semantic dimensions are modeled to obtain preliminary cross-modal fusion features; By aggregating features across scales, the final multimodal spatiotemporal fusion features are obtained; The network will be injected with prior knowledge of digital human interaction based on human kinematics and the laws of emotional expression. Real-time mining of user intent causal chains based on causal attention mechanisms; The uncertainty of the multimodal spatiotemporal fusion features is quantified by Bayesian variational inference mechanism, and the fusion interaction feature vector and uncertainty confidence field are output.

[0009] Preferably, as a preferred embodiment of the multimodal-based digital human interaction control method of the present invention, wherein: Based on the dimensions of interaction time, user location, and intent type, a multi-scale aggregation is performed to construct an initial multi-dimensional interaction situation tensor. By combining a multilayer perceptron with an attention mechanism, the initial multidimensional interaction situation tensor is nonlinearly mapped to calculate the current interaction situation strength and user intent confidence. Based on local real-time signals and global context, the current interactive situation distribution field is generated; By modeling the historical multidimensional interaction situation tensor through a temporal evolution network, the evolution trend of intent and response demand within the future interaction window can be predicted. Based on the dynamic hypergraph structure, the association features of intent in the semantic and behavioral spaces are calculated through the hypergraph attention mechanism; The uncertainty confidence field is incorporated into the calculation to generate a confidence-weighted multidimensional interactive situation tensor; The multidimensional interaction situation tensor is subjected to intent type classification and quantification, and the dominant intent, potential ambiguity and highly sensitive interaction points are identified.

[0010] Preferably, as a preferred embodiment of the multimodal-based digital human interaction control method of the present invention, wherein: The comprehensive interaction index is calculated based on situation strength, evolution trend, contextual consistency, and uncertainty confidence field. The digital human multimodal response is generated based on the comprehensive interaction index, including: Speech content generation using natural language generation models; Generate facial expressions, poses, and action sequences using a diffusion model; The interaction strategy is selected based on the interaction context, and the interaction strategy includes proactively asking questions, explaining, comforting, or performing tasks. The digital human's multimodal responses are prioritized using a multi-objective optimization algorithm, taking into account naturalness, task completion, and user emotional adaptation. The response is driven by the digital human rendering engine in real time; The decision output, actual user feedback, and effect data are fed back to the closed-loop optimization in real time.

[0011] Preferably, as a preferred embodiment of the multimodal-based digital human interaction control method of the present invention, wherein: Collect user feedback data in real time, including explicit feedback, implicit feedback, task completion status, and subsequent interaction effects; The user feedback data and multi-source heterogeneous multimodal data are spatiotemporally aligned and stored in an optimized database to construct a prediction-actual comparison sample pair. The user feedback data is validated for validity, and tagged optimized samples are generated. The dynamic adaptive multimodal spatiotemporal hypergraph Transformer network is updated using a hybrid strategy that combines online incremental learning with periodic batch optimization. Continuous parameter updates across all steps are achieved through adjustments to the differentiable hypergraph structure. Dynamically adjust internal strategies, including intent classification thresholds, response generation weights, attention mechanism preferences, and collection frequency; The user preference case library and response template library are updated synchronously. When the amount of user style drift or scene change is detected to exceed a preset threshold, local retraining is triggered.

[0012] Preferably, as a preferred embodiment of the multimodal digital human interaction control device of the present invention, wherein: Specifically, it includes: data acquisition hardware components, text input interface, wearable device interface, state storage unit, processing unit, storage unit, and digital human rendering and driving components. The data acquisition hardware components specifically include: At least one camera and one depth camera are used to capture real-time video of the user's facial expressions, eye movements, gestures, posture, and body movements; At least one microphone array is used to acquire speech content, intonation, emotional acoustic features, and ambient sound in real time; The text input interface is used to collect user text input data and operation command data; Wearable device interface for collecting heart rate, skin conductance, and eye movement data; The state storage unit is used to store the digital human's current posture, expression, dialogue history, scene context, and task objectives; The processing unit is electrically connected to the data acquisition hardware component; The storage unit is electrically connected to the processing unit and is used to store a user preference case library, a response template library, and an optimization database. The optimization database stores 'prediction-actual' comparison sample pairs. The digital human rendering and driving component, electrically connected to the processing unit, specifically includes: A display used to show the expressions and gestures of a digital human; A speaker, used to output the digital human's voice content; Motion-driven interface for controlling digital human limb movement sequences.

[0013] On the other hand, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements the steps of the multimodal digital human interaction control method as described above.

[0014] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements the steps of the multimodal digital human interaction control method as described above in the present invention.

[0015] The technical effects and advantages provided by the present invention in the above technical solution are as follows: (1) By constructing a multi-layer hypergraph network that includes local, meso, and global scales, and using a learnable adaptive edge generator, heterogeneous data such as facial expressions, speech intonation, text semantics, and physiological signals can be associated within the same hyperedge. Existing techniques mostly employ cascaded splicing or simple cross-attention mechanisms, which are difficult to capture nonlinear and cross-modal deep associations.

[0016] (2) By using the hierarchical calculation of local fine-grained attention, mesoscale contextual attention and global conversational attention, the digital human can not only accurately capture the user's current instantaneous intent, but also combine long-term user preferences and multi-turn dialogue history to achieve ultra-high accuracy in composite intent recognition, effectively solving the problems of polysemy and ambiguity.

[0017] (3) By using the Bayesian variational reasoning mechanism, the uncertainty of the multimodal fusion features is quantified and the confidence field is output. When facing conflict inputs or external high noise interference, the situation tensor can be smoothed and corrected by confidence weighting, which fundamentally reduces the probability of digital humans making erroneous actions, speaking incoherently or responding illogically.

[0018] (4) By setting a precise data loss rate threshold, once hardware frame loss, sensor disconnection or high noise packet loss occurs, automatic re-collection is triggered and fed back to the optimization end.

[0019] (5) Unlike the traditional simple “input-response” mapping, it uses a causal attention mechanism to mine the cause chain of user intent in real time, enabling digital humans to choose the most logical and reasonable interaction strategy.

[0020] (6) Injecting domain prior knowledge of kinematics and emotional expression laws into the network, and combining diffusion model and multi-objective optimization algorithm, language, expression and posture are jointly generated and prioritized, taking into account the naturalness of interaction, task completion and emotional adaptability, so that the performance of digital human is closer to that of real people. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0022] Figure 1 This is a flowchart of the multimodal digital human interaction control method of the present invention. Detailed Implementation

[0023] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0024] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides a multimodal digital human interaction control method, specifically including: Real-time acquisition of multi-source heterogeneous and multimodal data on user interactions with digital humans; Based on the data type and source, establish a multi-channel parallel acquisition interface to connect to multi-source heterogeneous and multimodal data sources, specifically including: (1) Real-time acquisition of user video data and user posture data through cameras and depth cameras, including user facial expressions, eye movements, gestures, body posture and body movement videos; (2) Real-time acquisition of user voice data through microphone array, including voice content, intonation, emotional acoustic features and ambient sound; (3) Collect user text input data and operation command data in real time through text input interface, keyboard and touch screen; (4) Collect users’ physiological data in real time through wearable devices, including heart rate, skin conductance and eye movement trajectory; (5) Collect digital human current status data in real time, including current posture, expression, dialogue history, scene context and task objectives; Timestamp alignment for multi-source heterogeneous multimodal data; A unified spatiotemporal reference framework is constructed based on interaction time, interaction rounds, and user spatial coordinates, and the multi-source heterogeneous multimodal data is labeled with a unified spatiotemporal tag. The multi-source heterogeneous multimodal data undergoes preliminary cleaning, noise reduction, and format standardization, including: Keyframe extraction and preliminary detection and annotation of face and pose are performed on the user video data; The user's voice data is subjected to speech recognition, preliminary emotion classification, and smoothing processing. When the missing rate of the multi-source heterogeneous data exceeds a preset threshold of 3%, a supplementary data collection instruction is automatically triggered, and the data quality assessment is fed back to the closed-loop optimization.

[0025] A dynamic adaptive multimodal spatiotemporal hypergraph Transformer network is constructed to deeply fuse the multi-source heterogeneous multimodal data and obtain a fused interactive feature vector. Mapping the multi-source heterogeneous multimodal data to a spatiotemporal reference frame specifically includes: Spatiotemporal visual features of user video data and user posture data are extracted using the Video-Swin Transformer model; Temporal acoustic and semantic features of user speech data are extracted using the Conformer model. Extract semantic features from user text input data using the BERT model; Physiological signal features and contextual state features are extracted using graph convolutional networks; A unified spatiotemporal location code and dialogue turn code are added to each extracted data feature to construct a multi-source heterogeneous multimodal feature set; Initialize the node set based on digital human joints, user key points, and interactive objects; Based on the hierarchical characteristics of the interaction scenario, a multi-layered hypergraph containing local, meso, and global scales is constructed. The learnable adaptive edge generator dynamically generates and updates the hyperedge weights and structure in real time based on the multi-source heterogeneous multimodal feature set. The multi-layer hypergraph is input into a dynamic adaptive multimodal spatiotemporal hypergraph Transformer network for hierarchical attention calculation, including: (1) Local fine-grained attention layer: focuses on the facial expressions, gestures and voice of the current frame in the multi-source heterogeneous multimodal data to capture the user's real-time intentions; (2) Mesoscale contextual attention layer: Integrates multi-source heterogeneous multimodal data within the current dialogue round to assess the user's complex intent; (3) Global conversation attention layer: Modeling the history of multiple dialogue turns, long-term user preferences, and the evolution trend of interaction scenarios; By using gated fusion and multi-head hypergraph attention mechanisms, the interaction relationships of the multi-source heterogeneous multimodal feature set in the spatiotemporal and semantic dimensions are modeled to obtain preliminary cross-modal fusion features; By aggregating features across scales, the final multimodal spatiotemporal fusion features are obtained; The network will be injected with prior knowledge of digital human interaction based on human kinematics and the laws of emotional expression. Real-time mining of user intent causal chains based on causal attention mechanisms; The uncertainty of the multimodal spatiotemporal fusion features is quantified by Bayesian variational inference mechanism, and the fusion interaction feature vector and uncertainty confidence field are output.

[0026] Based on the fused interactive feature vector, a multidimensional interactive situation tensor is generated; Based on the dimensions of interaction time, user location, and intent type, a multi-scale aggregation is performed to construct an initial multi-dimensional interaction situation tensor. By combining a multilayer perceptron with an attention mechanism, the initial multidimensional interaction situation tensor is nonlinearly mapped to calculate the current interaction situation strength and user intent confidence. Based on local real-time signals and global context, the current interactive situation distribution field is generated; By modeling the historical multidimensional interaction situation tensor through a temporal evolution network, the evolution trend of intent and response demand within the future interaction window can be predicted. Based on the dynamic hypergraph structure, the association features of intent in the semantic and behavioral spaces are calculated through the hypergraph attention mechanism; The uncertainty confidence field is incorporated into the calculation to generate a confidence-weighted multidimensional interactive situation tensor; The multidimensional interaction situation tensor is subjected to intent type classification and quantification, and the dominant intent, potential ambiguity and highly sensitive interaction points are identified.

[0027] Based on the multidimensional interactive situation tensor, a digital human response action sequence, voice content, facial expressions and gestures, and interaction strategies are generated, and the digital human is driven to execute them. The comprehensive interaction index is calculated based on situation strength, evolution trend, contextual consistency, and uncertainty confidence field. The digital human multimodal response is generated based on the comprehensive interaction index, including: Speech content generation using natural language generation models; Generate facial expressions, poses, and action sequences using a diffusion model; The interaction strategy is selected based on the interaction context, and the interaction strategy includes proactively asking questions, explaining, comforting, or performing tasks. The digital human's multimodal responses are prioritized using a multi-objective optimization algorithm, taking into account naturalness, task completion, and user emotional adaptation. The response is driven by the digital human rendering engine in real time; The decision output, actual user feedback, and effect data are fed back to the closed-loop optimization in real time.

[0028] Real-time feedback data is acquired after the user responds to the digital human, and the parameters of the dynamic adaptive multimodal spatiotemporal hypergraph Transformer network are optimized based on the feedback data; Collect user feedback data in real time, including explicit feedback, implicit feedback, task completion status, and subsequent interaction effects; The user feedback data and multi-source heterogeneous multimodal data are spatiotemporally aligned and stored in an optimized database to construct a prediction-actual comparison sample pair. The user feedback data is validated for validity, and tagged optimized samples are generated. The dynamic adaptive multimodal spatiotemporal hypergraph Transformer network is updated using a hybrid strategy that combines online incremental learning with periodic batch optimization. Continuous parameter updates across all steps are achieved through adjustments to the differentiable hypergraph structure. Dynamically adjust internal strategies, including intent classification thresholds, response generation weights, attention mechanism preferences, and collection frequency; The user preference case library and response template library are updated synchronously. When the amount of user style drift or scene change is detected to exceed a preset threshold, local retraining is triggered.

[0029] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0030] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the multimodal digital human interaction control method and device proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Example 2

[0031] The following is another embodiment of the present invention, which provides a digital human interaction control method based on multimodality. In order to verify the beneficial effects of the present invention, a simulation experiment is conducted for scientific demonstration.

[0032] A digital human interaction dataset was constructed using publicly available multi-source heterogeneous multimodal data for fine-tuning and simulation testing; Three mainstream networks were selected as comparison objects, specifically: Baseline Method A: Extract features from each modality separately, then perform simple concatenation, and output the result through a fully connected layer.

[0033] Baseline Method B: Cross-modal alignment and fusion are performed using conventional Cross-Attention.

[0034] Baseline method C: Temporal feature capture is performed using a regular graph convolutional network, without an adaptive hypergraph structure.

[0035] The evaluation is based on three dimensions: intent recognition accuracy, real-time performance of multimodal fusion, and robustness against interference. Specifically, it includes: Intent recognition accuracy: Evaluates the accuracy of identifying dominant intent and potential ambiguity; End-to-end latency: The time from data acquisition to the response of the driving engine, verifying whether the 200ms requirement for real-time interaction is met; Robustness metrics: White noise was added to the speech by human intervention and the video was processed with a 5% frame loss. The system performance was evaluated after the confidence field and the supplementary sampling mechanism were triggered.

[0036] Test set based on the aforementioned digital human interaction dataset; The dominant intent classification and polysemy recognition of the three mainstream networks were tested separately, specifically including:

[0037] Table 1 The test results above demonstrate that, by constructing a dynamic adaptive multimodal spatiotemporal hypergraph Transformer network, this invention improves the intent recognition accuracy by 9.7% to 20.6% compared to baseline methods A and B. Furthermore, due to its hierarchical attention computation structure (local-mesoscale-global), it maintains high accuracy while achieving an end-to-end latency of only 118ms, far below the 200ms hysteresis threshold for human visual and auditory perception, thus fully meeting real-time control requirements.

[0038] The beneficial effects of the uncertainty confidence field and the automatic data acquisition mechanism were verified by conducting artificial interference tests (adding -5dB noise to the speech and randomly dropping 5% of the video frames), specifically including:

[0039] Table 2 The test results above show that in harsh interactive environments with high noise and partially missing data, the accuracy of the two baseline methods drops significantly (up to 22.3%), and the digital human is prone to making erroneous movements or mismatched expressions. In contrast, the performance of the method proposed in this invention drops by only 5.3%. By outputting an uncertainty confidence field through a Bayesian variational inference mechanism and setting a 3% data missing threshold to trigger re-sampling, noise in multi-source heterogeneous data can be effectively suppressed.

[0040] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A multimodal digital human interaction control method, characterized in that, Specifically, it includes: Real-time acquisition of multi-source heterogeneous and multimodal data on user interactions with digital humans; A dynamic adaptive multimodal spatiotemporal hypergraph Transformer network is constructed to deeply fuse the multi-source heterogeneous multimodal data and obtain a fused interactive feature vector. Mapping the multi-source heterogeneous multimodal data to a spatiotemporal reference frame specifically includes: Spatiotemporal visual features of user video data and user posture data are extracted using the Video-Swin Transformer model; Temporal acoustic and semantic features of user speech data are extracted using the Conformer model. Extract semantic features from user text input data using the BERT model; Physiological signal features and contextual state features are extracted using graph convolutional networks; A unified spatiotemporal location code and dialogue turn code are added to each extracted data feature to construct a multi-source heterogeneous multimodal feature set; Initialize the node set based on digital human joints, user key points, and interactive objects; Based on the hierarchical characteristics of the interaction scenario, a multi-layered hypergraph containing local, meso, and global scales is constructed. The learnable adaptive edge generator dynamically generates and updates the hyperedge weights and structure in real time based on the multi-source heterogeneous multimodal feature set. The multi-layer hypergraph is input into a dynamic adaptive multimodal spatiotemporal hypergraph Transformer network for hierarchical attention calculation, including: (1) Local fine-grained attention layer: focuses on the facial expressions, gestures and voice of the current frame in the multi-source heterogeneous multimodal data to capture the user's real-time intentions; (2) Mesoscale contextual attention layer: Integrates multi-source heterogeneous multimodal data within the current dialogue round to assess the user's complex intent; (3) Global conversation attention layer: Modeling the history of multiple dialogue turns, long-term user preferences, and the evolution trend of interaction scenarios; By using gated fusion and multi-head hypergraph attention mechanisms, the interaction relationships of the multi-source heterogeneous multimodal feature set in the spatiotemporal and semantic dimensions are modeled to obtain preliminary cross-modal fusion features; By aggregating features across scales, the final multimodal spatiotemporal fusion features are obtained; The network will be injected with prior knowledge of digital human interaction based on human kinematics and the laws of emotional expression. Real-time mining of user intent causal chains based on causal attention mechanisms; The uncertainty of the multimodal spatiotemporal fusion features is quantified by Bayesian variational inference mechanism, and the fusion interaction feature vector and uncertainty confidence field are output. Based on the fused interactive feature vector, a multidimensional interactive situation tensor is generated; Based on the multidimensional interactive situation tensor, a digital human response action sequence, voice content, facial expressions and gestures, and interaction strategies are generated, and the digital human is driven to execute them. Real-time feedback data is acquired after the user responds to the digital human, and the parameters of the dynamic adaptive multimodal spatiotemporal hypergraph Transformer network are optimized based on the feedback data.

2. The multimodal digital human interaction control method according to claim 1, characterized in that: Based on the data type and source, establish a multi-channel parallel acquisition interface to connect to multi-source heterogeneous and multimodal data sources, specifically including: (1) Real-time acquisition of user video data and user posture data through cameras and depth cameras, including user facial expressions, eye movements, gestures, body posture and body movement videos; (2) Real-time acquisition of user voice data through microphone array, including voice content, intonation, emotional acoustic features and ambient sound; (3) Collect user text input data and operation command data in real time through text input interface, keyboard and touch screen; (4) Collect users’ physiological data in real time through wearable devices, including heart rate, skin conductance and eye movement trajectory; (5) Collect digital human current status data in real time, including current posture, expression, dialogue history, scene context and task objectives; Timestamp alignment for multi-source heterogeneous multimodal data; A unified spatiotemporal reference framework is constructed based on interaction time, interaction rounds, and user spatial coordinates, and the multi-source heterogeneous multimodal data is labeled with a unified spatiotemporal tag. The multi-source heterogeneous multimodal data is initially cleaned, denoised, and format-standardized.

3. The multimodal digital human interaction control method according to claim 1, characterized in that: Based on the dimensions of interaction time, user location, and intent type, a multi-scale aggregation is performed to construct an initial multi-dimensional interaction situation tensor. The initial multidimensional interactive situation tensor is nonlinearly mapped by a multilayer perceptron combined with an attention mechanism to calculate the current interactive situation strength and user intent confidence. Based on local real-time signals and global context, the current interactive situation distribution field is generated; By modeling the historical multidimensional interaction situation tensor through a temporal evolution network, the evolution trend of intent and response demand within the future interaction window can be predicted. Based on the dynamic hypergraph structure, the association features of intent in the semantic and behavioral spaces are calculated through the hypergraph attention mechanism; The uncertainty confidence field is incorporated into the calculation to generate a confidence-weighted multidimensional interactive situation tensor; The multidimensional interaction situation tensor is subjected to intent type classification and quantification, and the dominant intent, potential ambiguity and highly sensitive interaction points are identified.

4. The multimodal digital human interaction control method according to claim 1, characterized in that: The comprehensive interaction index is calculated based on situation strength, evolution trend, contextual consistency, and uncertainty confidence field. The digital human multimodal response is generated based on the comprehensive interaction index; The digital human's multimodal responses are prioritized using a multi-objective optimization algorithm, taking into account naturalness, task completion, and user emotional adaptation. The response is driven by the digital human rendering engine in real time; The decision output, actual user feedback, and effect data are fed back to the closed-loop optimization in real time.

5. The multimodal digital human interaction control method according to claim 1, characterized in that: Real-time collection of user feedback data; The user feedback data and multi-source heterogeneous multimodal data are spatiotemporally aligned and stored in an optimized database to construct a prediction-actual comparison sample pair. The user feedback data is validated for validity, and tagged optimized samples are generated. The dynamic adaptive multimodal spatiotemporal hypergraph Transformer network is updated using a hybrid strategy that combines online incremental learning with periodic batch optimization. Continuous parameter updates across all steps are achieved through adjustments to the differentiable hypergraph structure. Dynamically adjust internal strategies; The user preference case library and response template library are updated synchronously.

6. A multimodal digital human interaction control device, characterized in that, The device is configured to perform the method as described in any one of claims 1-5, comprising: a data acquisition hardware component, a text input interface, a wearable device interface, a state storage unit, a processing unit, a storage unit, and a digital human rendering and driving component; The data acquisition hardware components specifically include: At least one camera and one depth camera are used to capture real-time video of the user's facial expressions, eye movements, gestures, posture, and body movements; At least one microphone array is used to acquire speech content, intonation, emotional acoustic features, and ambient sound in real time; The text input interface is used to collect user text input data and operation command data; Wearable device interface for collecting heart rate, skin conductance, and eye movement data; The state storage unit is used to store the digital human's current posture, expression, dialogue history, scene context, and task objectives; The processing unit is electrically connected to the data acquisition hardware component; The storage unit is electrically connected to the processing unit and is used to store a user preference case library, a response template library, and an optimization database. The optimization database stores 'prediction-actual' comparison sample pairs. The digital human rendering and driving component, electrically connected to the processing unit, specifically includes: A display used to show the expressions and gestures of a digital human; A speaker, used to output the digital human's voice content; Motion-driven interface for controlling digital human limb movement sequences.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the multimodal digital human interaction control method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the multimodal digital human interaction control method according to any one of claims 1 to 5.

Images