Multimodal adaptive interactive ai digital human system and method

Abstract

The application relates to the technical field of digital human systems, and provides a multi-modal adaptive interaction AI digital human system and method, which comprises a multi-modal sensing terminal, a semantic understanding terminal, a behavior generation terminal, a digital human presentation terminal and an interaction management terminal; the multi-modal sensing terminal is used for acquiring multi-modal input and forming a multi-modal input feature package; the semantic understanding terminal is used for analyzing the multi-modal input feature package and collating the multi-modal input feature package into an intention vector; the behavior generation terminal is used for generating behavior output instructions according to the intention vector; the digital human presentation terminal is used for receiving the behavior output instructions and sequentially driving voice output, lip synchronization, facial expression change and limb action presentation of a digital human model to generate visualized and audible interaction effects matched with user input; and the interaction management terminal is used for uniformly coordinating the running states, data flow directions and response rhythms of the terminals. The application has the effect of improving the real-time interaction quality of the digital human.

Description

Technical Field

[0001] This invention relates to the technical field of digital human systems, specifically to a multimodal adaptive interactive AI digital human system and method. Background Technology

[0002] Currently, with the rapid development of artificial intelligence technologies such as speech recognition, computer vision, and natural language processing, human-computer interaction based on virtual avatars or digital humans is being widely applied in scenarios such as customer service consultation, online education, cultural tourism interpretation, and live-streaming e-commerce. Compared with traditional voice assistants or text dialogue systems, digital humans typically possess multiple expressive forms, including voice output, facial expressions, and body movements, making them visually closer to real people and offering significant advantages in emotional expression and immersive interaction. Therefore, how to comprehensively utilize multimodal information such as voice, images, gestures, and text content to drive digital humans to achieve natural, coherent, and emotionally rich interactions has become a key technological focus for the industry.

[0003] In the existing technology, several patent solutions related to virtual human or digital human interaction have emerged. For example, CN107765852A discloses a multimodal interaction processing method and system based on virtual humans. This scheme collects multimodal inputs such as voice and images, identifies the user's intent, and drives the virtual human to make corresponding interactive responses. However, its multimodal processing mainly focuses on the fusion of input feature layers and lacks fine control and unified coordination of subsequent behavior generation stages. CN107944542A proposes a multimodal interaction output method and system based on virtual humans. By jointly controlling the audio and video output of the virtual human, the virtual human's facial expressions, lip movements, and voices are more matched. However, it mainly focuses on the multimodal presentation on the output side and pays insufficient attention to the modeling of multimodal intents on the input side and the intermediate semantic expression forms. CN113992929A discloses a virtual digital human interaction method and system. By pre-producing digital human video clips in the cloud and selecting the corresponding video to play according to the client's interaction request, the virtual digital human can respond to user questions. This scheme has a relatively fixed interaction form and relies more on pre-made content rather than online behavior generation, and has limited adaptability to real-time multimodal interaction in complex situations.

[0004] Overall, the aforementioned existing technologies still have several shortcomings: First, most solutions simply integrate multimodal information at the feature layer or decision layer, lacking a unified semantic expression form for the entire interaction link, making it difficult to stably maintain the continuity and consistency of user intent in complex scenarios; Second, existing technologies generally lack refined compensation mechanisms for temporal misalignment, emotional conflict, or semantic shift between different modalities such as voice, facial expressions, and actions. When situations such as "voice comes out first, facial expression lags behind" or "calm tone, intense action" occur, the system often fails to automatically identify and correct cross-modal deviations; Third, existing solutions mostly control the digital human's voice output, lip movements, facial expressions, and body movements as relatively independent channels, lacking a consistency calibration strategy at the behavior level, which easily leads to a fragmented overall performance style and inconsistent emotions in the digital human; Fourth, in the case of multi-terminal collaboration, system load fluctuations, and network latency changes during actual deployment, existing digital human interaction systems usually lack dedicated interaction management and rhythm coordination modules, making it difficult to dynamically adjust the output complexity and interaction rhythm according to changes in the scenario, resulting in unstable interaction quality in high-concurrency or complex application scenarios.

[0005] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the aforementioned digital human systems by proposing a multimodal adaptive interactive AI digital human system and method.

[0007] The present invention adopts the following technical solution:

[0008] The multimodal adaptive interactive AI digital human system includes a multimodal perception terminal, a semantic understanding terminal, a behavior generation terminal, a digital human presentation terminal, and an interactive management terminal;

[0009] The multimodal perception terminal is used to acquire voice signals, image information, action postures and text content from users and the environment as multimodal inputs, and to perform feature extraction and structured processing on the multimodal inputs to form a multimodal input feature package.

[0010] The semantic understanding terminal is used to perform cross-modal alignment, content understanding, emotion recognition, and scene intent inference on the multimodal input feature package, and organizes the parsed user intent information and context parameters into a unified intent vector;

[0011] The behavior generation terminal is used to generate the response content, voice parameters, facial expression parameters and action parameters of the digital human according to the intent vector, and to perform consistency calibration on each parameter to form behavior output instructions suitable for the digital human presentation terminal.

[0012] The digital human presentation terminal is used to receive the behavior output instructions and sequentially drive the digital human model's voice output, lip-sync, facial expression changes and body movement presentation to generate visual and audible interactive effects that match the user input.

[0013] The interactive management terminal is used to coordinate the operating status, data flow, and response rhythm of each terminal.

[0014] Optionally, the multimodal perception terminal includes a voice acquisition module, a visual perception module, and a motion and posture detection module; the voice acquisition module is used to acquire the user's voice signal and perform noise reduction, voiceprint annotation, and semantic segmentation; the visual perception module is used to acquire the user's facial image and scene image, extract facial expression feature points and key environmental area information, and provide a primary facial posture reference to the motion and posture detection module; the motion and posture detection module is used to identify the user's motion trajectory, posture skeleton, and gesture information, and calibrate timing nodes based on the primary facial posture reference to generate a multimodal input feature package.

[0015] Optionally, the semantic understanding terminal includes a cross-modal semantic alignment module, an emotion and scene inference module, and a cross-modal offset self-compensation module. The cross-modal semantic alignment module is used to perform temporal synchronization and semantic-level alignment of speech features, visual features, and action features, so that the same user's behavioral intention is uniformly expressed across different modalities. The emotion and scene inference module is used to infer the user's emotional state, interaction purpose, and environmental context based on the aligned multimodal features, forming a primary intention fragment for behavior generation. The cross-modal offset self-compensation module is used to perform self-compensation processing on semantic offsets, emotional conflicts, or action lags between cross-modal features, and automatically generates compensation vectors by monitoring abnormal difference components to reconstruct a consistent intention vector when there is a delay in user facial expression changes, reversal of speech emotion, or ambiguity in action intention.

[0016] Optionally, the behavior generation terminal includes a response content generation module, an action / expression parameter module, and a behavior consistency calibration module. The response content generation module is used to generate the digital human's response text based on the intent vector and match the corresponding tone type. The action / expression parameter module is used to generate lip-shape parameters, voice parameters, expression parameters, and action parameters based on the response text and emotional state, and convert them into executable output instructions. The behavior consistency calibration module is used to perform rhythm synchronization, emotion fusion, and behavior smoothing processing on the behavior parameter set corresponding to the output instructions, so that the voice output, lip-shape changes, and action presentation maintain a consistent style.

[0017] Optionally, the digital human presentation terminal includes a voice rendering module, an expression animation module, and a motion driving module; the voice rendering module is used to synthesize the voice parameters in the output command into a continuous audio signal and provide a lip-sync driving reference; the expression animation module is used to dynamically render the digital human's facial muscles based on the lip-sync driving reference and expression parameters, and provide a synchronization rhythm to the motion driving module; the motion driving module is used to drive the digital human's body, posture, and camera movement according to the motion parameters and the synchronization rhythm of the expression animation module, and output visual dynamic content that matches the user input.

[0018] Optionally, the interactive management terminal includes a status monitoring module, a strategy adjustment module, and an interaction rhythm coordination module; the status monitoring module is used to monitor the operating status and data throughput of the multimodal perception terminal, semantic understanding terminal, behavior generation terminal, and digital human presentation terminal in real time; the strategy adjustment module is used to dynamically adjust the output style, emotion intensity, and interaction method of the behavior generation terminal according to the scene tags output by the semantic understanding terminal; the interaction rhythm coordination module is used to synchronize the processing rhythm of each terminal according to the latency and load status provided by the status monitoring module.

[0019] A multimodal adaptive interactive AI digital human method is applied to the multimodal adaptive interactive AI digital human system described above. The multimodal adaptive interactive AI digital human method includes:

[0020] S1, acquire voice signals, image information, action postures and text content from the user and environment as multimodal input, and perform feature extraction and structured processing on the multimodal input to form a multimodal input feature package;

[0021] S2, perform cross-modal alignment, content understanding, emotion recognition and scene intent inference on the multimodal input feature package, and organize the parsed user intent information and context parameters into a unified intent vector;

[0022] S3, generate the digital human's response content, voice parameters, facial expression parameters and action parameters according to the intent vector, and perform consistency calibration on each parameter to form a behavior output instruction suitable for the digital human presentation terminal;

[0023] S4, receive the behavior output instruction, and sequentially drive the digital human model's voice output, lip-sync, facial expression changes and body movement presentation to generate a visual and audible interactive effect that matches the user input;

[0024] S5 coordinates the operating status, data flow, and response rhythm of each terminal in a unified manner.

[0025] The beneficial effects achieved by this invention are:

[0026] 1. By setting up multimodal perception terminals, semantic understanding terminals, behavior generation terminals, digital human presentation terminals, and interaction management terminals, the system can realize a complete interaction chain from multimodal input acquisition, intent parsing, behavior generation to digital human presentation under a unified architecture. This is conducive to improving the response consistency of digital humans under complex input conditions, thereby improving the real-time interaction quality of digital humans, and thus facilitating the realization of a natural, stable, and continuous human-computer interaction experience.

[0027] 2. By setting up a voice acquisition module, a visual perception module, and a motion and posture detection module in the multimodal perception terminal, the system can acquire input features with rich semantic and emotional cues from the three modalities of voice, image, and motion. This helps to improve the authenticity and expressive completeness of multimodal input, thereby enhancing the parsing accuracy of the downstream semantic understanding terminal, which in turn helps the overall interactive system to accurately grasp the user's true intentions.

[0028] 3. By setting up a cross-modal semantic alignment module, an emotion and scene inference module, and a cross-modal offset self-compensation module in the semantic understanding terminal, the system can achieve automatic semantic fusion and compensation in situations where there are differences in multimodal semantics, asynchronous emotion expression, or delayed actions. This helps to reduce the interference caused by inconsistencies between different modal inputs on intent parsing, thereby improving the stability and uniformity of intent vectors, and thus helping to achieve reliable semantic understanding capabilities of digital humans in complex situations.

[0029] 4. By setting up a response content generation module, an action and expression parameter module, and a behavior consistency calibration module in the behavior generation terminal, the system can generate text responses, voice parameters, expression parameters, and action parameters from a unified intent vector, and perform rhythm and emotion consistency calibration on multimodal behavior output. This helps to avoid problems such as inconsistency between voice and expression, and lack of coordination between action and tone in digital human output, thereby improving the naturalness and immersion of behavior performance, which in turn helps to improve the overall expression coherence and realism of digital humans in interaction.

[0030] 5. By setting up a voice rendering module, facial expression animation module, and motion-driven module in the digital human presentation terminal, the digital human can transform behavioral output commands into visual representations of continuous audio, dynamic facial expressions, and synchronized movements. This facilitates the complete presentation of the digital human's auditory, visual, and physical interaction content, thereby enhancing the user's perception and understanding of the digital human's behavior and ultimately achieving a dynamic presentation effect of the digital human that meets the user's expectations.

[0031] 6. By setting up a status monitoring module, a strategy adjustment module, and an interaction rhythm coordination module in the interactive management terminal, the system can monitor the operating status of multiple terminals in real time and dynamically adjust the output strategy based on the current latency, load, and scene labels. This helps maintain the stability of the system under high concurrency and complex scenarios, thereby ensuring that the multimodal processing link maintains consistent rhythm and continuous response.

[0032] To further understand the features and technical content of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are for reference and illustration only and are not intended to limit the present invention. Attached Figure Description

[0033] Figure 1 This is a block diagram showing the overall structure and connections of the present invention;

[0034] Figure 2 This is a schematic diagram of the overall structure of the multimodal adaptive interactive AI digital human system in this invention;

[0035] Figure 3 This is a schematic diagram of the method flow of the multimodal adaptive interactive AI digital human method in this invention;

[0036] Figure 4 This is a three-dimensional distribution diagram of cross-modal migration and combined migration intensity in another embodiment of the present invention;

[0037] Figure 5 This is a graph showing the relationship between the adjusted offset intensity and time lag in another embodiment of the present invention;

[0038] Figure 6 This is a UI (user interface) effect diagram of the digital human system displayed on the digital human presentation terminal in another embodiment of the present invention;

[0039] Explanation of reference numerals in the attached figures:

[0040] 1. Multimodal sensing terminal; 2. Semantic understanding terminal; 3. Behavior generation terminal; 4. Digital human presentation terminal; 5. Interactive management terminal; 6. Modular equipment cabinet. Detailed Implementation

[0041] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can understand the advantages and effects of the present invention from the content disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and various details in this specification can also be modified and changed based on different viewpoints and applications without departing from the spirit of the present invention. Furthermore, the accompanying drawings of the present invention are for simple illustrative purposes only and are not depictions of actual dimensions; this is stated in advance. The following embodiments will further describe the relevant technical content of the present invention in detail, but the disclosed content is not intended to limit the scope of protection of the present invention.

[0042] Example 1: Combining Figure 1 and Figure 2 As shown, this embodiment provides a multimodal adaptive interactive AI digital human system. The multimodal adaptive interactive AI digital human system includes a multimodal perception terminal 1, a semantic understanding terminal 2, a behavior generation terminal 3, a digital human presentation terminal 4, and an interaction management terminal 5. In this embodiment, the multimodal perception terminal 1, semantic understanding terminal 2, behavior generation terminal 3, digital human presentation terminal 4, and interaction management terminal 5 are mounted in a modular equipment cabinet 6.

[0043] The multimodal perception terminal 1 is used to acquire voice signals, image information, action postures and text content from the user and the environment as multimodal inputs, and to perform feature extraction and structuring processing on the multimodal inputs to form a multimodal input feature package.

[0044] The semantic understanding terminal 2 is used to perform cross-modal alignment, content understanding, emotion recognition and scene intent inference on the multimodal input feature package, and organize the parsed user intent information and context parameters into a unified intent vector;

[0045] The behavior generation terminal 3 is used to generate the response content, voice parameters, facial expression parameters and action parameters of the digital human according to the intent vector, and to perform consistency calibration on each parameter to form a behavior output instruction suitable for the digital human presentation terminal 4.

[0046] The digital human presentation terminal 4 is used to receive the behavior output instructions and sequentially drive the digital human model's voice output, lip-sync, facial expression changes and body movement presentation to generate visual and audible interactive effects that match the user input.

[0047] The interactive management terminal 5 is used to coordinate the operating status, data flow, and response rhythm of each terminal.

[0048] Optionally, the multimodal perception terminal 1 includes a voice acquisition module, a visual perception module, and a motion and posture detection module; the voice acquisition module is used to acquire the user's voice signal and perform noise reduction, voiceprint annotation, and semantic segmentation; the visual perception module is used to acquire the user's facial image and scene image, extract facial expression feature points and key environmental area information, and provide a primary facial posture reference to the motion and posture detection module; the motion and posture detection module is used to identify the user's motion trajectory, posture skeleton, and gesture information, and calibrate timing nodes based on the primary facial posture reference to generate a multimodal input feature package.

[0049] Optionally, the semantic understanding terminal 2 includes a cross-modal semantic alignment module, an emotion and scene inference module, and a cross-modal offset self-compensation module. The cross-modal semantic alignment module is used to perform temporal synchronization and semantic-level alignment of speech features, visual features, and action features, so that the same user's behavioral intention is uniformly expressed across different modalities. The emotion and scene inference module is used to infer the user's emotional state, interaction purpose, and environmental context based on the aligned multimodal features, forming a primary intention fragment for behavior generation. The cross-modal offset self-compensation module is used to perform self-compensation processing on semantic offsets, emotional conflicts, or action lags between cross-modal features, and automatically generates compensation vectors by monitoring abnormal difference components to reconstruct a consistent intention vector when there is a delay in user facial expression changes, reversal of speech emotion, or ambiguity in action intention.

[0050] Optionally, the behavior generation terminal 3 includes a response content generation module, an action and expression parameter module, and a behavior consistency calibration module. The response content generation module is used to generate the digital human's response text based on the intent vector and match the corresponding tone type. The action and expression parameter module is used to generate lip shape parameters, voice parameters, expression parameters, and action parameters based on the response text and emotional state, and convert them into executable output instructions. The behavior consistency calibration module is used to perform rhythm synchronization, emotion fusion, and behavior smoothing processing on the behavior parameter set corresponding to the output instructions, so that the voice output, lip shape changes, and action presentation maintain a consistent style.

[0051] Optionally, the digital human presentation terminal 4 includes a voice rendering module, an expression animation module, and a motion driving module; the voice rendering module is used to synthesize the voice parameters in the output command into a continuous audio signal and provide a lip-sync driving reference; the expression animation module is used to dynamically render the digital human's facial muscles based on the lip-sync driving reference and expression parameters, and provide a synchronization rhythm to the motion driving module; the motion driving module is used to drive the digital human's body, posture, and camera movement according to the motion parameters and the synchronization rhythm of the expression animation module, and output visual dynamic content that matches the user input.

[0052] Optionally, the interactive management terminal 5 includes a status monitoring module, a strategy adjustment module, and an interaction rhythm coordination module; the status monitoring module is used to monitor the operating status and data throughput of the multimodal perception terminal 1, semantic understanding terminal 2, behavior generation terminal 3, and digital human presentation terminal 4 in real time; the strategy adjustment module is used to dynamically adjust the output style, emotion intensity, and interaction method of the behavior generation terminal 3 according to the scene tags output by the semantic understanding terminal 2; the interaction rhythm coordination module is used to synchronize the processing rhythm of each terminal according to the latency and load status provided by the status monitoring module.

[0053] In an exemplary implementation, the system constructs a data flow path in the order of multimodal input, semantic parsing, behavior generation, and digital human presentation, with each terminal maintaining low-latency collaboration through internal communication links. After system startup, the multimodal perception terminal simultaneously receives raw data from the voice source, camera, motion sensor, and text input channel, and performs signal preprocessing, feature extraction, and structured encoding, enabling different modalities to be uniformly expressed in downstream modules. To avoid modal misalignment caused by acquisition jitter, the multimodal perception terminal can automatically adjust the feature generation rhythm based on the camera frame rate, voice sampling timestamps, and keyframes of action nodes.

[0054] In this embodiment, the semantic understanding terminal performs cross-modal fusion on the input feature packets from the multimodal perception terminal. Through temporal synchronization and semantic alignment, speech segments, facial expression feature points, and action trajectories can be mapped to the same semantic space. Furthermore, the emotion and scene inference module can, for example, combine voiceprint intensity, facial tension changes, action amplitude, and background area information to infer the user's interaction intent, such as distinguishing between behavioral semantics like "asking a question," "expressing surprise," and "requesting assistance." The cross-modal offset self-compensation module is used to generate compensation vectors when there is a reversal in speech emotion, lag in facial expression, or ambiguity in action, enabling the system to form a stable interpretation of the user's true intent.

[0055] After receiving the intent vector, the behavior generation terminal can generate corresponding response text, tone configuration, tone markers, facial expression parameters, and action parameters. To ensure consistent presentation, the terminal will use a behavior consistency calibration module to rhythmically align the multimodal output parameters, ensuring that the digital human follows a consistent emotional style during speech, facial expression changes, and action execution, avoiding incongruous behaviors such as a gentle tone with a serious expression or hurried movements.

[0056] In the digital human presentation stage, this embodiment uses voice rendering, facial animation, and motion-driven modules to transform upstream behavioral output commands into audible, visual, and perceptible digital human performance. The voice rendering module generates audio waveforms in real time based on voice parameters; the facial animation module renders facial muscle deformations based on lip movements and facial expression parameters; and the motion-driven module combines facial rhythm to drive body movements, enabling the digital human to exhibit a natural and continuous interactive state.

[0057] During system operation, the interactive management terminal assumes the roles of scheduling and monitoring. By collecting real-time status and data throughput data from multiple terminals, it rationally allocates resources when response latency increases or load grows. The strategy adjustment module can adjust the output style of the behavior generation terminal based on the semantic understanding of the terminal's scene tags. For example, it can reduce animation complexity in emergency scenarios and enhance facial expression rendering in emotional dialogues. The interaction rhythm coordination module is used to maintain the rhythm consistency of the multimodal processing chain under conditions of high-concurrency user input or high-frequency interaction.

[0058] This embodiment, through the aforementioned modular structure and processing link, enables the digital human system to maintain stable, smooth, and emotionally consistent interaction capabilities under complex, variable, and cross-modal input conditions.

[0059] In another preferred embodiment, this embodiment further describes the data structure, triggering relationship, and optional anomaly handling strategies among the terminals. When performing speech sampling, image acquisition, and action posture detection, the multimodal perception terminal can uniformly organize the features of different modalities into structured data packets in the form of timestamp sequences. Each data packet contains at least a modality identifier, a timestamp, a feature vector, and a confidence parameter. Through this unified temporal structure, the semantic understanding terminal can use timestamps to synchronize multimodal inputs during cross-modal semantic alignment, enabling the system to obtain stable alignment results even when the speech sampling frequency and image frame rate are inconsistent. Furthermore, in scenarios such as sudden changes in user illumination, partial occlusion, or transient speech distortion, the multimodal perception terminal can trigger degradation processing strategies based on the confidence parameter, such as automatically reducing the weight of visual features or enabling action nodes as auxiliary modalities, thereby maintaining the availability of the input feature packets.

[0060] Furthermore, to make the structure of the intent vector clearer, this embodiment can describe the intent vector as a multi-dimensional vector set composed of semantic principal components, emotion parameters, action indicator factors, and scene labels. The semantic principal components represent the core meaning of the user's behavior, the emotion parameters express emotional intensity or polarity, the action indicator factors identify whether a dynamic action response needs to be generated, and the scene labels guide the interaction management terminal in implementing strategy adjustments. In the cross-modal offset self-compensation module, when semantic conflicts occur between different modalities, such as when the voice expresses "affirmation" while the facial expression shows "confusion," the system calculates the abnormal difference component and generates a compensation vector based on the difference component to readjust the emotion parameters and semantic principal components, making the final intent vector more consistent with the user's actual interaction purpose.

[0061] This embodiment further explains the rhythm synchronization mechanism within the behavior generation terminal. After generating the response text, facial expression parameters, and action parameters, the behavior consistency calibration module sets an emotional rhythm benchmark based on the scene label in the current intent vector and propagates this benchmark to the speech parameters, lip-sync parameters, and action parameters, ensuring that each output channel follows the same rhythm sequence. For example, in a "soothing" context, the system automatically reduces speech rate, decreases the amplitude of movements, and slows down the speed of facial expression changes; while in a "stimulating" scenario, it increases facial tension and speeds up the output rhythm to maintain the consistency of the digital human's overall emotional style. When rendering resources are limited or the output pressure is high, the behavior generation terminal can also execute parameter compression strategies based on the load information of the interaction management terminal to smooth action keyframes, thereby avoiding interaction stuttering caused by over-computation.

[0062] In the optional implementation of the digital human presentation terminal, the digital human model can adopt a skeletal-driven structure with an expression-driven structure, depending on the rendering environment and interaction requirements. The speech rendering module generates continuous audio by mapping speech parameters to an acoustic model and transmits key lip-sync events in the audio packets to the expression animation module to drive real-time deformation of the mouth muscles. After receiving the emotional rhythm and action parameters, the motion-driven module performs posture interpolation based on the skeletal weights of the digital human, ensuring that gestures, body center of gravity, and head gaze are consistent with the generated behavior, thus presenting natural and coherent body language during interaction.

[0063] To ensure the real-time performance and stability of the overall system, the interactive management terminal in this embodiment can monitor system load and network status, and dynamically adjust the output complexity of the digital human through a strategy adjustment module. For example, when an increase in system latency is detected, the refresh frequency of the motion-driven module can be reduced to prioritize the real-time performance of voice output and facial expressions; when the scene label indicates "emotional communication," the weight of the facial animation module can be increased, allowing the digital human to present more subtle facial expressions. Through the above coordination mechanism, the system in this embodiment can maintain a natural, stable, and consistent interactive experience even in environments with complex multimodal inputs, parallel multi-link operations, and changing external conditions.

[0064] The multimodal adaptive interactive AI digital human method is applied to the multimodal adaptive interactive AI digital human system described above, combined with... Figure 3 The multimodal adaptive interactive AI digital human method includes:

[0065] S1, acquire voice signals, image information, action postures and text content from the user and environment as multimodal input, and perform feature extraction and structured processing on the multimodal input to form a multimodal input feature package;

[0066] S2, perform cross-modal alignment, content understanding, emotion recognition and scene intent inference on the multimodal input feature package, and organize the parsed user intent information and context parameters into a unified intent vector;

[0067] S3, generate the digital human's response content, voice parameters, facial expression parameters and action parameters according to the intent vector, and perform consistency calibration on each parameter to form a behavior output instruction suitable for the digital human presentation terminal;

[0068] S4, receive the behavior output instruction, and sequentially drive the digital human model's voice output, lip-sync, facial expression changes and body movement presentation to generate a visual and audible interactive effect that matches the user input;

[0069] S5 coordinates the operating status, data flow, and response rhythm of each terminal in a unified manner.

[0070] Example 2: This example further improves upon Example 1, providing a cross-modal offset measurement and self-compensation algorithm for reconstructing intent vectors, specifically for the cross-modal offset self-compensation module in a semantic understanding terminal. This algorithm, considering semantic differences, emotional conflicts, and time lag, introduces a user's respiratory rhythm fluctuation factor to characterize changes in the user's physiological tension during the same round of interaction, using it as a modulation term for cross-modal compensation intensity. This allows intent reconstruction to not only rely on surface behavioral features but also reflect the impact of the user's internal emotional fluctuations on the semantic consistency of the interaction. Combined with... Figure 4As shown, this illustrates the statistical relationships between various types of offsets in the cross-modal offset recognition algorithm, as well as the overall distribution trend of the comprehensive offset intensity. Combined with... Figure 5 As shown, the cross-modal migration assessment model exhibits a trend of overall migration intensity under different time lags. The statistical graph uses three broken lines to illustrate the changes in overall migration intensity with time lag when semantic and emotional migration are at different levels, reflecting the sensitivity and response characteristics of the system under different multimodal misalignment conditions.

[0071] In this embodiment, the cross-modal migration self-compensation module performs the following in each time window: Within this process, semantic feature vectors are extracted from the speech feature stream, visual feature stream, and action feature stream, respectively. , and and emotional characteristic scalar , , And combined with the time center of each mode , , Perform offset measurement. To characterize multimodal semantic consistency, within the window... Internally defined semantic offset index for:

[0072]

[0073] in, It is a 2-norm. To prevent tiny positive numbers with a denominator of zero, the larger this index is, the more dispersed the distribution of speech, facial expressions, and actions in the semantic space within the same time window, and the worse the semantic consistency.

[0074] To measure the degree of conflict at the emotional level, this embodiment defines an emotional conflict index. for:

[0075]

[0076] in, The normalization scaling constant for emotion intensity is... It is a small positive number. The hyperbolic tangent function is used to compress the intensity of emotional conflict to a specific value. The range. The closer this indicator is to 1, the greater the divergence in emotional polarity or intensity between different modalities, such as a voice expressing "excitement" while the facial expression leans towards "indifference."

[0077] Regarding time alignment, this embodiment constructs a time lag index based on the time center differences of each modality. :

[0078]

[0079] in, , , windows respectively The temporal center of features for internal speech, vision, and motion. This refers to the window duration. This metric reflects the degree of rhythm misalignment in a user's speech, facial expressions, and body movements. When a user exhibits phenomena such as "speech preceding movement" or "facial expressions preceding speech," It will increase significantly.

[0080] The key parameter introduced in this embodiment is the user's respiratory rhythm fluctuation factor. This factor, obtained through bandpass filtering and period detection of the low-frequency energy components in the voice acquisition module, is used to characterize the degree of deviation of the user's breathing rhythm within the current interaction window from its recent baseline rhythm. Specifically, in the recent... Calculate the average respiratory cycle within each time window. The estimated respiratory cycle within the current window is... Then the definition is:

[0081]

[0082] in, To prevent division by zero of tiny positive numbers. A higher value indicates a more significant deviation in the user's breathing rhythm within the current window compared to their recent state, typically corresponding to internal state changes such as tension, anxiety, or sudden mood swings. This parameter does not rely on additional physiological sensors; it extracts only the low-frequency components of existing speech signals, representing a non-obvious improvement over traditional interactive systems that rely solely on speech content, facial expressions, and gestures.

[0083] After combining the above three offset indices with the respiratory rhythm fluctuation factor, this embodiment constructs a cross-modal comprehensive offset intensity index. Used to guide self-compensation strength:

[0084]

[0085] in, , which is a non-negative adjustment coefficient, used to control the relative impact of semantic shift, emotional conflict, and time lag on the overall shift; This is the amplification factor for the respiratory rhythm fluctuation factor. When a user's internal emotional excitement causes a significant deviation in their respiratory rhythm, even if the surface semantic offset is still at a moderate level, Still due to factors This amplifies the effect, triggering a stronger self-compensation mechanism that enhances the system's sensitivity to intention reconstruction even in seemingly normal but physiologically stressful interactive states.

[0086] Based on comprehensive offset strength In this embodiment, the intent vector Self-compensation updates are performed in multiple iterations. Let... The initial intent vector is directly provided by the emotion and scene inference module. For the first The intent vector after compensation, the compensation increment vector It can be represented as:

[0087]

[0088] in, Based on the compensation step size, The nonlinear adjustment function is related to the offset intensity, preferably:

[0089]

[0090] To control the scale parameter of the response sensitivity, when When the offset is small (slight), When it is close to 0, the self-compensation amplitude is very small; when When the deviation is large (significant and possibly accompanied by abnormal respiratory rhythm), As the value rapidly approaches 1, the self-compensation module will perform a significant intention correction. (Vector) The offset direction vector in the current iteration can be obtained by superimposing the multimodal semantic bias directions, for example, by... The difference direction in the semantic space is projected onto the intention vector space to obtain a normalized direction, so that the compensation process proceeds along the direction of "reducing modal differences".

[0091] In the above formula, An intent vector is a set of real-valued vector components with fixed dimensions, where each dimension can correspond to a semantic principal component, emotion parameter, action indicator factor, or scene label encoding. , , These respectively reflect the relative intensity of semantic shift, emotional conflict, and time lag; The user's respiratory rhythm fluctuation factor characterizes the change in the user's physiological rhythm relative to the recent baseline within the current window; To comprehensively measure the offset intensity, and to uniformly characterize the degree of cross-modal inconsistency after the superposition of multiple factors; and These are the hyperparameters of the algorithm, which can be determined through system calibration or empirical configuration; The contribution ratio of various indicators can be reasonably set according to different application scenarios.

[0092] Through the algorithm design of this embodiment, the cross-modal offset self-compensation module, when performing intent vector reconstruction, not only considers three explicit offset factors—semantics, emotion, and time—but also introduces a user's respiratory rhythm fluctuation factor derived from the low-frequency components of the speech signal. This makes the system more sensitive to the user's internal tension or emotional excitement. In actual interaction, when the user maintains polite language but their breathing rhythm changes rapidly, the system in this embodiment will proactively increase the compensation intensity and reconstruct the intent vector more cautiously. This reduces misjudgments of "superficial consistency but internal inconsistency," improving the robustness and interactive experience of the multimodal adaptive interactive AI digital human in complex emotional scenarios.

[0093] To better understand the aforementioned offset metric and self-compensation algorithm, this embodiment further explains the parameters. Regarding semantic features, , , Representing speech, visual, and motion modalities in the window respectively. The semantic embedding vectors within the modality, which can be real-number vectors with dimensions between 64 and 512, are extracted by the feature encoder corresponding to the modality and are used to represent semantic content, action intent, or facial expression semantics within a specific time window. Emotion scalar. Values ​​can be taken from The range is defined as follows: values ​​closer to 1 indicate stronger positive emotions, values ​​closer to -1 indicate stronger negative emotions, and 0 indicates neutral emotions. (Emotional normalization scaling) This is used to balance the magnitude differences in emotional characteristics across different modalities, with typical values ​​ranging from 0.5 to 2.0.

[0094] Time Center This can be obtained by taking a weighted average of the keyframe timestamps of the corresponding modality within the window, and is used to reflect the central position of the user's speech rhythm, facial expression change rhythm, and movement rhythm. Breathing cycle The reference period is estimated from the envelope peak spacing of the low-frequency portion of the speech signal (typically 0.1–0.4 Hz). The coefficient is updated using a moving average over the most recent L windows to reflect the user's recent stable respiratory rhythm. The contributions of semantic shift, emotional conflict, and temporal lag to the overall shift intensity were controlled separately, with typical values ​​ranging from 0.3 to 3.0; while the amplification factor... It reflects the sensitivity of respiratory rhythm fluctuations to the intensity of deviation. The larger the value, the more the system tends to perform stronger compensation when the user is under physiological stress.

[0095] In the process of compensating the intention vector, the step size parameter The update magnitude of each iteration is controlled; a larger value results in faster compensation, but an excessively large step size may cause oscillations in the intention vector. (Scale parameter) Controlling the response sensitivity of g(Ψ), when Smaller, the system is more sensitive to offset intensity; when When the value is large, the compensation change is smoother. Offset direction vector. It can be obtained by projecting the offset direction of the modality difference vector in the semantic embedding space to the intent vector space and then standardizing it, ensuring that the compensation direction always points to the region with "higher multimodal consistency".

[0096] The nonlinear adjustment function is used to adjust the compensation increment in each iteration. The intensity, making it consistent with the current overall offset intensity. The relationship between them exhibits a non-linear growth. Specifically, when When the value of g(Ψ) is small, it is close to zero, resulting in a smaller compensation magnitude, thus avoiding over-correction of the intent vector under slight modal shifts; when As semantic shifts, emotional conflicts, or respiratory rhythm fluctuations increase, g(Ψ) will rapidly approach 1, bringing the compensation increment close to the maximum step size, thereby quickly correcting the intent vector and making it more stable in high-risk shift scenarios.

[0097] The compensation process is executed iteratively until any of the following stopping conditions are met: (1) the norm of the current compensation increment. Less than the preset threshold (2) The number of iterations has reached the upper limit. (3) Cross-modal migration strength The rate of increase decreases with iteration and stabilizes within a certain range. To avoid overcompensation, this embodiment can also introduce a compensation suppression mechanism. If the growth rate of g(Ψ) decreases or... If the direction of change no longer points in the opposite direction of the offset direction, the compensation process is paused to ensure smooth updating of the intent vector.

[0098] Through the above compensation process, the algorithm in this embodiment can respond quickly when there are significant inconsistencies in voice, action and facial expression modalities, and adjust them in combination with respiratory rhythm fluctuation factors, so that the digital human system can more accurately capture the user's true interactive intentions and significantly improve the robustness and naturalness in complex emotional interaction scenarios.

[0099] Example 3: In another embodiment, the multimodal adaptive interactive AI digital human system can be deployed in a comprehensive hardware environment consisting of a computing platform, sensing and acquisition devices, graphics rendering devices, and network communication modules to support the entire processing chain of multimodal perception, semantic understanding, behavior generation, and digital human presentation. This embodiment further illustrates the feasibility of this system by describing the hardware structure and program execution method. Combined with... Figure 6 As shown, Figure 6 The image shows the UI (User Interface) of the digital human system, which is presented on the digital human presentation terminal. It demonstrates the UI interface through which users can interact with the system.

[0100] In an exemplary hardware deployment, the system includes a processor, system memory, a graphics rendering unit, an audio processing unit, a camera acquisition device, a motion detection device, and a network communication module. The processor can be a multi-core CPU, an AI acceleration chip, or a dedicated processing unit with vector computing capabilities, used to execute software instructions for the multimodal perception terminal, semantic understanding terminal, behavior generation terminal, and interaction management terminal. The system memory stores program code to be executed, multimodal feature cache, intent vectors, compensation vectors, and behavior output instructions. The graphics rendering unit executes facial expression animation and motion-driven rendering of the digital human model, and the audio processing unit performs speech parameter conversion and audio synthesis. The camera acquisition device captures facial and scene images, the motion detection device acquires posture skeletons or acceleration information, and the network communication module communicates with cloud resources or other interactive devices.

[0101] In a preferred embodiment, the voice acquisition, image acquisition, and motion detection processing required by the multimodal perception terminal can be provided by a microphone array, an RGB camera, and a posture sensor, respectively. The acquired data is transmitted to the processor via the system bus. Under the scheduling of the software framework, the processor first performs processes such as speech denoising, feature segmentation, facial expression feature point extraction, and motion skeleton sequence generation, and stores the results in system memory to form a structured multimodal input feature package. Subsequently, the software module of the semantic understanding terminal is loaded, and the processor sequentially executes the algorithm logic of cross-modal semantic alignment, emotion and scene inference, and cross-modal offset self-compensation, thereby obtaining a unified intent vector for behavior generation.

[0102] During the behavior generation phase, the processor invokes the response content generation module, action and expression parameter module, and behavior consistency calibration module stored in memory to generate the digital human's text response, tone markers, lip-sync parameters, speech parameters, expression parameters, and action parameters based on the intent vector. The generated multimodal parameter set is transmitted to the graphics rendering unit and audio processing unit. The graphics rendering unit executes facial animation and motion-driven actions, while the audio processing unit performs speech synthesis. The rendered image frames and audio signals can be presented to the user through display and audio output devices, thus creating a digital human interaction effect that matches the user's input.

[0103] In this embodiment, the software module of the interactive management terminal runs resident on the processor to monitor the operating load, latency, data throughput, and module status of various parts of the system. When the system detects that the multimodal perception terminal experiences a drop in frame rate, the semantic understanding terminal experiences excessive resource consumption, the behavior generation terminal experiences output blockage, or the rendering terminal experiences delays and frame skipping, the interactive management terminal triggers the strategy adjustment module to adjust the output style, animation complexity, and rhythm synchronization parameters of the behavior generation terminal. At the same time, the interactive rhythm coordination module reorganizes the overall processing rhythm of the system in real time, so that the system can maintain smooth and consistent interaction even under high load or unstable network conditions.

[0104] In another optional implementation, the system can also be deployed using a collaborative approach between edge devices and the cloud. The multimodal perception terminal and digital human presentation terminal can be deployed on local devices for data acquisition and output, while the computationally intensive semantic understanding terminal and behavior generation terminal can be deployed on a cloud server, transmitting instructions and sending back results via a network communication module. To reduce the impact of network latency on the interactive experience, the interaction management terminal can perform some policy adjustment functions locally. When network latency exceeds a threshold, it automatically reduces the refresh rate of the action-driven module or uses a simplified facial animation template to ensure real-time synchronization between voice output and lip movements.

[0105] In this embodiment, the terminal functions of the multimodal adaptive interactive AI digital human system can be written to a readable storage medium, such as a flash memory card, solid-state drive, ROM, RAM, or cloud storage. The readable storage medium stores instructions that can be executed by a processor. When the processor executes these instructions, the multimodal perception terminal can perform multimodal feature extraction and structured encoding; the semantic understanding terminal can perform cross-modal semantic alignment, emotion inference, and cross-modal offset self-compensation; the behavior generation terminal can generate multimodal behavior parameters and complete consistency calibration; the digital human presentation terminal can perform speech synthesis, facial animation, and motion driving; and the interaction management terminal can execute strategy adjustment and rhythm coordination logic according to the system state, thereby fully realizing the multimodal adaptive digital human interaction process of this invention.

[0106] Through the above description of the hardware platform, data processing architecture, and program execution method, this embodiment further illustrates the operation process of the digital human system described in this invention in a real device, enabling those skilled in the art to effectively implement all the functions of this invention based on existing hardware and software frameworks.

[0107] The content disclosed above is only a preferred and feasible embodiment of the present invention, and is not intended to limit the scope of protection of the present invention. Therefore, all equivalent technical changes made based on the content of the present invention specification and drawings are included within the scope of protection of the present invention. Furthermore, the elements therein can be updated as technology develops.

Claims

1. A multimodal adaptive interactive AI digital human system, characterized in that, This includes multimodal sensing terminals, semantic understanding terminals, behavior generation terminals, digital human presentation terminals, and interactive management terminals; The multimodal perception terminal is used to acquire voice signals, image information, action postures and text content from users and the environment as multimodal inputs, and to perform feature extraction and structured processing on the multimodal inputs to form a multimodal input feature package. The semantic understanding terminal is used to perform cross-modal alignment, content understanding, emotion recognition, and scene intent inference on the multimodal input feature package, and organizes the parsed user intent information and context parameters into a unified intent vector; The behavior generation terminal is used to generate the response content, voice parameters, facial expression parameters and action parameters of the digital human according to the intent vector, and to perform consistency calibration on each parameter to form a behavior output instruction suitable for the digital human presentation terminal. The digital human presentation terminal is used to receive the behavior output instructions and sequentially drive the digital human model's voice output, lip-sync, facial expression changes and body movement presentation to generate visual and audible interactive effects that match the user input. The interactive management terminal is used to coordinate the operating status, data flow, and response rhythm of each terminal.

2. The multimodal adaptive interactive AI digital human system as described in claim 1, characterized in that, The multimodal perception terminal includes a voice acquisition module, a visual perception module, and a motion and posture detection module. The voice acquisition module is used to acquire the user's voice signal and perform noise reduction, voiceprint annotation, and semantic segmentation. The visual perception module is used to acquire the user's facial image and scene image, extract facial expression feature points and key environmental area information, and provide a primary facial posture reference to the motion and posture detection module. The motion and posture detection module is used to identify the user's motion trajectory, posture skeleton, and gesture information, and calibrate temporal nodes based on the primary facial posture reference to generate a multimodal input feature package.

3. The multimodal adaptive interactive AI digital human system as described in claim 2, characterized in that, The semantic understanding terminal includes a cross-modal semantic alignment module, an emotion and scene inference module, and a cross-modal offset self-compensation module. The cross-modal semantic alignment module is used to perform temporal synchronization and semantic-level alignment of speech features, visual features, and action features, so that the same user's behavioral intention is uniformly expressed across different modalities. The emotion and scene inference module is used to infer the user's emotional state, interaction purpose, and environmental context based on the aligned multimodal features, forming a primary intention fragment for behavior generation. The cross-modal offset self-compensation module is used to perform self-compensation processing on semantic offsets, emotional conflicts, or action lags between cross-modal features. It automatically generates compensation vectors by monitoring abnormal difference components to reconstruct a consistent intention vector when there is a delay in user facial expression changes, reversal of speech emotion, or ambiguity in action intention.

4. The multimodal adaptive interactive AI digital human system as described in claim 3, characterized in that, The behavior generation terminal includes a response content generation module, an action and expression parameter module, and a behavior consistency calibration module. The response content generation module generates response text for the digital human based on the intent vector and matches it with the corresponding tone type. The action and expression parameter module generates lip-shape parameters, voice parameters, expression parameters, and action parameters based on the response text and emotional state, and converts them into executable output instructions. The behavior consistency calibration module performs rhythm synchronization, emotion fusion, and behavior smoothing on the behavior parameter set corresponding to the output instructions, so that the voice output, lip-shape changes, and action presentation maintain a consistent style.

5. The multimodal adaptive interactive AI digital human system as described in claim 4, characterized in that, The digital human presentation terminal includes a voice rendering module, an expression animation module, and a motion driving module. The voice rendering module synthesizes the voice parameters in the output command into a continuous audio signal and provides a lip-sync reference. The expression animation module dynamically renders the digital human's facial muscles based on the lip-sync reference and expression parameters, and provides a synchronization rhythm to the motion driving module. The motion driving module drives the digital human's body, posture, and camera movement according to the motion parameters and the synchronization rhythm of the expression animation module, and outputs visual dynamic content that matches the user input.

6. The multimodal adaptive interactive AI digital human system as described in claim 5, characterized in that, The interactive management terminal includes a status monitoring module, a strategy adjustment module, and an interaction rhythm coordination module. The status monitoring module is used to monitor the operating status and data throughput of the multimodal perception terminal, semantic understanding terminal, behavior generation terminal, and digital human presentation terminal in real time. The strategy adjustment module is used to dynamically adjust the output style, emotional intensity, and interaction method of the behavior generation terminal based on the scene tags output by the semantic understanding terminal. The interaction rhythm coordination module is used to synchronize the processing rhythm of each terminal based on the latency and load status provided by the status monitoring module.

7. A multimodal adaptive interactive AI digital human method, applied to the multimodal adaptive interactive AI digital human system as described in claim 6, characterized in that, The multimodal adaptive interactive AI digital human method includes: S1, acquire voice signals, image information, action postures and text content from the user and environment as multimodal input, and perform feature extraction and structured processing on the multimodal input to form a multimodal input feature package; S2, perform cross-modal alignment, content understanding, emotion recognition and scene intent inference on the multimodal input feature package, and organize the parsed user intent information and context parameters into a unified intent vector; S3, generate the digital human's response content, voice parameters, facial expression parameters and action parameters according to the intent vector, and perform consistency calibration on each parameter to form a behavior output instruction suitable for the digital human presentation terminal; S4, receive the behavior output instruction, and sequentially drive the digital human model's voice output, lip-sync, facial expression changes and body movement presentation to generate a visual and audible interactive effect that matches the user input; S5 coordinates the operating status, data flow, and response rhythm of each terminal in a unified manner.

Images