An agent digital image interaction generation method based on multi-modal perception

Abstract

The application discloses an agent digital image interaction generation method based on multi-modal perception, which comprises collecting multi-modal input data of a user, pre-processing and feature extraction of the multi-modal input data; inputting into an improved high-efficiency modal cross-learning network for multi-modal feature fusion processing; constructing a semantic intention graph, introducing time index edge weight and emotion-driven edge weight, and using a structure perception graph neural network to encode the graph; through a cross-modal style contrast learning mechanism, a modal style vector is extracted, and a personal style code vector is generated through a hierarchical nested structure; a personal regulation gate mechanism is inputted, a feature channel linear modulation method is adopted to regulate the intermediate layer representation in the interaction strategy generation process; the representation vector is inputted into a behavior strategy generation module to generate a multi-modal behavior output sequence; the output sequence drives the digital image to respond synchronously, and the natural response generation in the user interaction process is completed.

Description

Technical Field

[0001] This invention belongs to the field of multimodal perception and graph neural network modeling technology, and particularly relates to a method for generating interactive digital images of intelligent agents based on multimodal perception. Background Technology

[0002] Against the backdrop of rapid development in human-computer interaction technology, digital humans, as an important carrier for multimodal interaction between artificial intelligence and humans, are being widely applied in various fields such as intelligent customer service, virtual companionship, education and training, and emotional interaction. Traditional human-computer interaction methods are mostly based on text or voice, which, limited by factors such as single modality and limited depth of understanding, make it difficult to meet users' needs for natural, emotional, and personalized responses. Especially in complex contexts, single-modal information often fails to accurately convey users' intentions and emotional states, resulting in problems such as stiff digital human responses and disjointed interactions. Therefore, how to integrate multiple perceptual modalities and build a high-quality, multi-dimensional intelligent interaction mechanism has become one of the key challenges in current technological development.

[0003] Some existing multimodal interaction systems have attempted to achieve information fusion across multiple perceptual channels, such as speech, image, and text, to enhance the understanding and responsiveness of digital humans. For example, some systems use convolutional neural networks to extract image features, combine them with recurrent neural networks to process text semantics, and then supplement them with speech recognition modules to extract speech features, thereby driving the virtual human to perform actions or output speech. However, these systems often employ parallel processing for feature extraction and fusion, failing to fully exploit the contextual relationships and semantic complementarity between modalities. This results in a lack of deep interactive information in the fusion results, making them prone to ambiguity or misjudgment in semantic expression. Furthermore, some research has adopted attention-based fusion structures, which, while enhancing the alignment ability between modalities to some extent, still face problems such as inconsistent feature dimensions of heterogeneous modalities, a single fusion method, and poor robustness, making it difficult to support the high demands of digital human behavior response in complex interaction scenarios.

[0004] In terms of interaction strategy generation, existing technologies generally adopt static rule-driven or single-strategy modeling approaches, ignoring the significant differences in individual user behavior, tone of voice, and emotional responses. This makes it impossible for digital humans to generate personalized strategies when facing different users, resulting in highly homogenized output behaviors and a lack of necessary contextual adaptability and user-customized experiences. For example, some current digital human systems that generate behavioral instructions based on templates or reinforcement learning only select behaviors based on the current semantics or task state, ignoring deeper factors such as long-term user interaction preferences, language styles, and emotional characteristics. This leads to a lack of individuality in the output of speech tone, vivid facial expressions, and gestures, making it impossible to establish long-term and effective emotional connections.

[0005] In semantic modeling, existing research has begun to explore the introduction of graph structure representations, such as abstracting dialogue structures into graphs and modeling semantic relationships between nodes, to improve the ability to identify user intent. However, these methods are usually limited to the construction of static semantic structures and do not consider the dynamic changes during the dialogue process, especially the profound impact of temporal evolution and emotional driving factors on interaction intent. For example, during long-term continuous user interaction, semantic intent may deviate due to changes in context, and emotional state may also fluctuate. Static graph structures often cannot reflect such dynamic evolution processes, resulting in lag or distortion in intent modeling.

[0006] Furthermore, existing digital human systems generally lack a mechanism to coordinate the consistency of multimodal outputs such as speech, facial expressions, and body movements. During multimodal behavioral output, different modalities often have different temporal granularities, response delays, and control parameters. Without a unified scheduling and alignment mechanism, problems such as output conflicts and rhythm misalignment may arise, thus affecting the user's overall perception of the digital human's "natural interaction." For example, in some systems, while the speech output possesses semantic integrity, facial expressions and movements fail to be generated synchronously or do not match the semantic rhythm, leading to a decline in the user's evaluation of the smoothness of the interaction. Summary of the Invention

[0007] To address the problems existing in the prior art, this invention proposes a method for generating interactive digital avatars of intelligent agents based on multimodal perception.

[0008] The technical solution of the present invention is as follows:

[0009] A method for generating interactive digital avatars of intelligent agents based on multimodal perception, comprising:

[0010] Collect multimodal input data from users, perform preprocessing and feature extraction on the multimodal input data in sequence, and generate a feature vector set;

[0011] The feature vector set is input into an improved high-efficiency modal cross-learning network for multimodal feature fusion processing to obtain semantic embedding vectors.

[0012] Based on the interaction context information, a semantic intent graph is constructed, and time index edge weights and emotion-driven edge weights are introduced into the semantic intent graph. The graph is encoded using a structure-aware graph neural network to output a semantic intent vector.

[0013] Collect user behavior data during the interaction process, extract modal style vectors through a cross-modal style contrast learning mechanism, and generate personalized style encoding vectors through a hierarchical nested structure;

[0014] The semantic embedding vector, semantic intent vector, and personalized style encoding vector are input into the personalized control gating mechanism. The intermediate layer representation in the interaction strategy generation process is controlled by the feature channel linear modulation method, and the representation vector is output.

[0015] Input the representation vector into the behavior strategy generation module to generate a multimodal behavior output sequence;

[0016] The digital avatar is driven to respond synchronously based on the structured multimodal behavior output sequence, thus generating natural responses during user interaction.

[0017] Furthermore, the process of collecting the user's multimodal input data, and then preprocessing and extracting features from the multimodal input data to generate a feature vector set, specifically involves:

[0018] Collect user's multimodal input data, which includes voice data, facial images, and text information;

[0019] The speech data is subjected to noise reduction, endpoint detection and speech segmentation. The short-time energy, spectral envelope and Mel frequency cepstral coefficients of the speech are extracted to generate speech feature vectors.

[0020] Face detection, key point localization, and expression segmentation are performed on facial images. The motion amplitude features and muscle movement amplitude change features of facial regions are extracted to generate image feature vectors.

[0021] The text information is segmented, part-of-speech tagging and dependency parsing are performed to extract semantic relationship features and keyword vector representations, and generate text semantic vectors.

[0022] The speech feature vectors, image feature vectors, and text semantic vectors are aligned according to timestamps, and a unified vector dimension normalization method is used to generate a time-synchronized multimodal input feature vector set.

[0023] Furthermore, the specific method for inputting the feature vector set into the improved efficient modal cross-learning network for multimodal feature fusion processing to obtain the semantic embedding vector includes:

[0024] The speech feature vectors, image feature vectors, and text semantic vectors in the feature vector set are respectively input into the speech channel, image channel, and text channel of the improved efficient modality cross-learning network to construct a multi-channel fusion input structure;

[0025] Bidirectional cross-attention paths are constructed between the speech channel and the image channel, the image channel and the text channel, and the speech channel and the text channel, and the cross-attention weights between each modal channel are calculated.

[0026] Based on the cross-attention weights, a context enhancement operation is performed on the feature vector within each modal channel to generate a cross-modal enhancement vector;

[0027] Residual connection and normalization are performed on the cross-modal enhancement vector and feature vector set to preserve modal independence and improve fusion robustness;

[0028] The residual connections and normalized modal feature vectors are concatenated and nonlinearly transformed to output semantic embedding vectors, which serve as the fusion result of the improved efficient modal cross-learning network.

[0029] Furthermore, the specific method for constructing a semantic intent graph based on interaction context information, introducing time-indexed edge weights and emotion-driven edge weights into the semantic intent graph, encoding the graph using a structure-aware graph neural network, and outputting semantic intent vectors includes:

[0030] Collect voice input, text input, system task response, and explicit feedback data during user interaction with the agent within a preset time window to generate interaction context information;

[0031] The interaction context information is organized according to the interaction round number and timestamp, and the corresponding text semantic content, interaction intent identifier, context state parameters and voice emotion vector in each round of interaction are extracted to construct a set of interaction round sequences;

[0032] Based on the set of interaction round sequences, key semantic units are identified and abstracted into intent nodes, task nodes, and emotion nodes. A node set is constructed, and each node is given a timestamp, emotion intensity value, and interaction round identifier.

[0033] A set of directed edges is established based on the semantic dependency relationship and temporal causal relationship between nodes. This set includes intention-dependent edges that reflect semantic triggering, time-indexed edges that reflect temporal evolution, and emotion-driven edges that reflect user emotional changes. Each edge is assigned a decay factor, an emotion coupling factor, and a priority factor as edge weight parameters.

[0034] The set of nodes and the set of directed edges are combined to form a semantic intent graph. A structure-aware graph neural network based on edge type grouping is introduced to perform neighbor aggregation operation on different edge types and fuse information of multiple edge types through a gating mechanism.

[0035] In the aggregation stage of the structure-aware graph neural network, a time decay function and an emotion modulation function are introduced to dynamically weight the feature propagation process of time-indexed edges and emotion-driven edges, generating a semantic intent vector containing information from multiple semantic pathways.

[0036] Furthermore, the specific method for collecting user's personalized behavior data during the interaction process, extracting modal style vectors through a cross-modal style contrastive learning mechanism, and generating personalized style encoding vectors through a hierarchical nested structure includes:

[0037] Collect personalized behavioral data of users during interaction with the intelligent agent, including voice tone change parameters, facial expression image frame sequences and natural language text input, and synchronize and align them according to the interaction rounds and timestamps;

[0038] Style feature normalization is performed on the individual behavior data, and a modal style comparison sample set of positive sample pairs and negative sample pairs is constructed. The positive sample pairs are derived from multiple rounds of behavior expression of the same user, and the negative sample pairs are derived from samples generated by other users or the system.

[0039] Speech style vectors, image style vectors, and text style vectors are extracted through a modal style coding network, and contrastive learning training is performed based on the modal style contrastive loss function to enhance intramodal consistency and intermodal discriminability.

[0040] Speech style vectors, image style vectors and text style vectors are input into a hierarchical nested structure to generate low-level style representation, mid-level aggregation representation and high-level fusion representation in sequence. The mid-level aggregation representation fuses modal style features through an aggregation network. The high-level fusion representation further fuses user behavior history summary vectors and long-term semantic preference vectors.

[0041] The low-level style representation, the mid-level aggregation representation, and the high-level fusion representation are fused into a personalized style encoding vector through a jump connection mechanism.

[0042] Furthermore, the specific method for inputting the semantic embedding vector, semantic intent vector, and personalized style encoding vector into the personalized control gating mechanism, and using the feature channel linear modulation method to regulate the intermediate layer representation in the interaction strategy generation process, and outputting the representation vector includes:

[0043] It receives semantic embedding vectors, semantic intent vectors, and personalized style encoding vectors, adjusts them to feature representations of the same dimension, and generates policy control channel representations through the linear mapping module.

[0044] The adjusted semantic embedding vector, semantic intent vector, and personalized style encoding vector are subjected to feature channel-level weight calculation to construct a three-way control weight matrix, which is used to represent semantic dominance, intent urgency, and personalized priority, respectively. The feature channel-level weight calculation is jointly calculated by learnable parameters and context state.

[0045] The three-way control weight matrix is ​​input into the feature channel linear modulation module to recalibrate the channel weights of the intermediate feature layer in the behavior strategy generation module, thereby strengthening the personality-related dimensions and suppressing irrelevant or redundant feature dimensions.

[0046] A gating selection mechanism is added after the linear modulation module of the feature channel. Based on the current environmental state, task type and user's historical policy response, the optimal path is selected from multiple policy sub-paths for feature propagation, and the final regulated representation vector is output.

[0047] Furthermore, the specific method for generating a multimodal behavior output sequence by inputting the representation vector into the behavior strategy generation module includes:

[0048] The representation vector is input into the multi-channel behavior strategy generation module and mapped to the speech response channel, facial expression channel and body movement channel respectively.

[0049] In the speech response channel, the language generation network is used to generate speech command vectors with specific rhythm, emotional intensity and prosodic features based on the current semantic embedding and intonation control parameters, and outputs a text and audio dual-mode structure to drive the speech synthesizer.

[0050] In the facial expression channel, based on the emotion state encoding and personality style vector, the corresponding expression parameter set is selected and fine-tuned, including eyebrow and eye state, mouth dynamics and head posture, to generate and output a standardized facial movement control instruction sequence.

[0051] In the limb movement channel, the semantic intent vector and the action type library are combined to generate limb movement control parameters through the action prediction network, including trunk posture changes, hand posture key points and movement trajectory control vectors, and output a continuous and driveable command stream.

[0052] The instructions output from the voice response channel, facial expression channel, and body movement channel are synchronized in time, conflict detected, and prioritized to generate a unified structured multimodal behavior output sequence.

[0053] Furthermore, the specific method for driving the digital avatar to synchronously respond based on the structured multimodal behavior output sequence to complete the generation of natural responses during user interaction includes:

[0054] Receive multimodal behavior output sequences, extract voice response commands, facial expression commands, and body movement control commands by channel, and perform frame-level synchronization according to a unified timeline;

[0055] The voice response command is input into the speech synthesis module, which generates a speech audio stream with personalized prosodic features based on the target intonation parameters, semantic content and speech style vector, and synchronizes with the lip animation generation module.

[0056] The facial expression action commands are mapped to the high-precision expression driving engine of the digital image, and the facial muscle control parameters are adjusted step by step according to the key action frames to achieve continuous and natural dynamic evolution of expressions, and the expression intensity is automatically adjusted according to the context and emotional state.

[0057] Inputting limb movement control commands into the skeletal drive system of the digital avatar drives the multi-joint movements of the torso, hands, and head to execute posture changes, gestures, and interactive actions, achieving humanoid responses that match semantic commands;

[0058] A response consistency scheduler is introduced into the overall driving process to monitor the synchronization indicators of the three channels of voice, expression and action in real time, and to fine-tune and repair based on frame rate offset, response delay and action conflict signals.

[0059] The collaboratively driven digital avatar behavior results are rendered and output in real time, enabling the generation of natural responses with semantic consistency, emotional coherence, and personalized expression characteristics, thus completing a full round of interaction with the user.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] This invention proposes a method for generating interactive digital avatars based on multimodal perception. This method comprehensively enhances the agent's semantic understanding, individual perception, and multimodal behavior generation capabilities by designing an improved, efficient modal cross-learning network, constructing a semantic intent graph and encoding a structure-aware graph neural network, employing a cross-modal style contrast learning mechanism, and using a feature channel linear modulation method. This achieves a natural, coherent, and personalized digital human interaction process. The method introduces a time-indexed and emotion-driven semantic graphing mechanism, effectively enhancing the dynamic adaptability of semantic intent recognition. Simultaneously, through individual behavior modeling and policy channel regulation, it achieves highly personalized behavior output generation, coordinating and scheduling at the multimodal output command level to ensure the naturalness and fluency of interactive responses, thereby significantly improving the agent's interactive performance and user experience in real-world environments.

[0062] First, this invention introduces an improved, efficient modal cross-learning network. Independent feature channels are constructed for user-input speech, image, and text modal data. Through a cross-channel bidirectional cross-attention mechanism, the alignment and fusion of deep semantic relationships between modalities are achieved, ultimately generating semantic embedding vectors. This structure differs from traditional parallel fusion methods, not only improving the collaborative modeling capability between modalities but also significantly enhancing robust recognition performance against semantic ambiguity, polysemous expressions, and contextual default information. It effectively avoids interaction biases caused by missing or abnormal modalities, providing a structurally stable and semantically rich foundational vector representation for subsequent intent construction.

[0063] Secondly, in terms of semantic intent modeling, this invention proposes a semantic intent graph construction method that integrates time-indexed edge weights and emotion-driven edge weights. It also uses a structure-aware graph neural network to aggregate multi-type edge relationships and jointly modulate temporal emotions within the graph structure. By constructing a heterogeneous graph structure composed of intent nodes, task nodes, and emotion nodes, and introducing a time decay function and an emotion modulation function, the system can accurately capture the semantic causal evolution path and dynamic changes in emotional state when processing multi-turn dialogues in continuous interaction processes. This enables more context-aware and emotionally sensitive interactive intent recognition. This mechanism significantly overcomes the limitations of existing graph modeling methods that rely solely on static semantic relationships, effectively improving the accuracy and timeliness of intent recognition.

[0064] Furthermore, this invention introduces a cross-modal style contrast learning mechanism and a hierarchical nested style coding structure in user personalized response modeling. For the first time, it unifies the modeling of a user's speech intonation features, facial expression image sequences, and text style elements, and extracts distinctive personalized style features by constructing positive and negative sample pairs for style contrast training. Subsequently, it generates personalized style coding vectors through low-level representation aggregation, mid-level semantic alignment, and high-level behavioral preference fusion, combined with a jump connection mechanism. This enables the digital human to not only possess semantic adaptability during policy generation but also to showcase the user's unique personality in dimensions such as tone of voice, facial expression intensity, and action style, achieving highly personalized and consistent interactive behavior output.

[0065] In terms of behavior response generation, this invention further designs a method based on feature channel linear modulation, which uniformly maps semantic embedding vectors, semantic intent vectors, and personalized style encoding vectors to a policy control space, constructs a three-channel modulation weight matrix, and performs channel-level reweighting on the intermediate layer of the behavior policy generation module. A gating selection mechanism is introduced by combining environmental states and user response history to dynamically switch the optimal policy path, effectively improving the accuracy of behavior policy selection and context adaptability. This modulation mechanism enables the digital human to possess full-dimensional perception and policy adjustment capabilities oriented towards task objectives, emotional states, and individual characteristics, significantly improving the decision-making flexibility of interactive behavior and the matching degree of personalized responses.

[0066] Ultimately, this invention constructs a unified behavior output structure and timing synchronization mechanism in the multimodal behavior output stage. It performs unified timeline alignment and conflict detection on the control commands generated by the voice response channel, facial expression channel, and body movement channel. A response consistency scheduler is used to achieve fine-tuning, repair, and synchronized scheduling of the three-channel output. The system supports frame-level monitoring and dynamic correction mechanisms, effectively solving the problems of asynchronous multimodal output delays, rhythmic inconsistencies, and response conflicts in traditional systems. This ensures that the digital avatar exhibits natural, coherent, and rhythmically unified interactive performance during the linkage of voice, facial expressions, and movements. Attached Figure Description

[0067] Figure 1 This is a flowchart illustrating the intelligent agent digital image interaction generation method based on multimodal perception according to the present invention. Detailed Implementation

[0068] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0069] Example:

[0070] This invention provides a method for generating interactive digital avatars of intelligent agents based on multimodal perception, such as... Figure 1 As shown, it includes:

[0071] Step 1) Collect the user's multimodal input data, and perform preprocessing and feature extraction on the multimodal input data in sequence to generate a feature vector set;

[0072] Step 2) Input the feature vector set into the improved high-efficiency modal cross-learning network to perform multimodal feature fusion processing and obtain semantic embedding vectors;

[0073] Step 3) Based on the interaction context information, construct a semantic intent graph, and introduce time index edge weights and emotion-driven edge weights into the semantic intent graph. Use a structure-aware graph neural network to encode the graph and output a semantic intent vector.

[0074] Step 4) Collect user's personalized behavior data during the interaction process, extract modal style vectors through cross-modal style contrast learning mechanism, and generate personalized style encoding vectors through hierarchical nesting structure;

[0075] Step 5) Input the semantic embedding vector, semantic intent vector, and personalized style encoding vector into the personalized control gating mechanism, and use the feature channel linear modulation method to control the intermediate layer representation in the interaction strategy generation process, and output the representation vector.

[0076] Step 6) Input the representation vector into the behavior strategy generation module to generate a multimodal behavior output sequence;

[0077] Step 7) Drive the digital avatar to respond synchronously based on the structured multimodal behavior output sequence, and complete the generation of natural responses during the user interaction process.

[0078] In this invention, the entire process consists of four stages: multimodal feature processing, semantic modeling, personalized encoding, and behavior generation. The steps work together to enable the intelligent agent to efficiently understand and respond naturally to user input. First, preprocessing and feature extraction of multimodal input data such as speech, images, and text form a unified and standardized feature vector set, providing a structural foundation for subsequent fusion. Next, an improved efficient modal cross-learning network is used to achieve deep fusion of features from different modalities, outputting semantically consistent embedding vectors. Then, a semantic intent graph containing time and emotion weights is constructed by combining interaction context information, and structure-aware graph neural network encoding is used to extract intent vectors reflecting semantic evolution trends. Based on this, user-specific behavior data is further collected, and personalized style encoding vectors are generated through a cross-modal style contrast learning mechanism and a hierarchical nested structure. Semantic, intent, and personality vectors are input into a linear modulation mechanism in the feature channel to dynamically adjust the representation information during strategy generation, enhancing the targeting and personalization of behavioral output. Subsequently, the adjusted representation vectors are input into the behavior strategy generation module, outputting a multimodal behavior control sequence. Finally, this sequence drives the digital avatar to synchronously respond with speech, facial expressions, and actions, achieving natural, fluent, and personalized digital interaction. The process as a whole achieves closed-loop control from understanding multimodal input to generating multimodal response, and has significant advantages such as accurate semantic perception, full expression of individuality and natural interactive response.

[0079] In one embodiment, multimodal input data from users is collected, and the multimodal input data is preprocessed and feature extracted to generate a feature vector set, specifically as follows:

[0080] Collect multimodal input data from users, including voice data, facial images, and text information;

[0081] The speech data is subjected to noise reduction, endpoint detection and speech segmentation. The short-time energy, spectral envelope and Mel frequency cepstral coefficients of the speech are extracted to generate speech feature vectors.

[0082] Face detection, key point localization, and expression segmentation are performed on facial images. The motion amplitude features and muscle movement amplitude change features of facial regions are extracted to generate image feature vectors.

[0083] The text information is segmented, part-of-speech tagging, and dependency parsing are performed to extract semantic relationship features and keyword vector representations, generating a text semantic vector; the text information is obtained by direct user input or by speech recognition conversion;

[0084] The speech feature vectors, image feature vectors, and text semantic vectors are aligned according to timestamps, and a unified vector dimension normalization method is used to generate a time-synchronized multimodal input feature vector set.

[0085] This invention first performs modal acquisition and feature extraction on the user's multimodal input data to construct a unified and standardized feature vector set. Specifically, for speech data, the system acquires the user's speech signal in real time through a microphone array and uses a multi-channel time-frequency domain adaptive filter for noise suppression. Then, an endpoint detection method based on energy thresholds is used to segment the continuous speech stream and extract effective speech segments. For each speech segment, the system further extracts short-time energy, spectral envelope, and Mel-frequency cepstral coefficients to form a multidimensional speech feature vector, comprehensively representing the timbre, intonation, and rhythm information of the user's speech. For image data, the system acquires a sequence of user facial image frames through a camera device and uses a face detection algorithm based on a deep convolutional network to locate facial regions. A facial keypoint localization network is used to obtain the positional information of action-sensitive areas such as the eyes, eyebrows, and mouth, and a facial expression partitioning strategy is combined to divide the regions. Based on this, feature information such as changes in the amplitude of facial movements, muscle movement trends, and keypoint movement trajectories are extracted to generate facial expression feature vectors in the image modality, reflecting the user's visual emotional expression in the current interaction state. For text data, the system collects both user-inputted text and text transcribed by the speech recognition module, and performs a unified language preprocessing workflow on both. This workflow includes Chinese word segmentation, part-of-speech tagging, named entity recognition, and dependency parsing, extracting semantic relationship structures and keywords from the text, constructing word vector representations based on context windows, and finally generating text semantic vectors to represent user intent and linguistic expression. After feature extraction from the three modalities, the speech feature vectors, image feature vectors, and text semantic vectors are aligned at the frame level based on the acquisition timestamps of each modality. Vector interpolation and normalization based on the least common dimension are then used to unify the dimensions and normalize the amplitude of each modality vector, constructing a structurally consistent and time-synchronized multimodal input feature vector set, providing a consistent input structure for subsequent fusion network modeling. This step effectively ensures the comparability and fusionability of multimodal data in the time and feature domains, and is a fundamental step in achieving high-quality multimodal semantic modeling.

[0086] In one embodiment, a specific method for inputting a feature vector set into an improved high-efficiency modal cross-learning network and performing multimodal feature fusion processing to obtain semantic embedding vectors includes:

[0087] The speech feature vectors, image feature vectors, and text semantic vectors in the feature vector set are respectively input into the speech channel, image channel, and text channel of the improved efficient modality cross-learning network to construct a multi-channel fusion input structure;

[0088] Bidirectional cross-attention paths are constructed between the speech channel and the image channel, the image channel and the text channel, and the speech channel and the text channel, and the cross-attention weights between each modal channel are calculated.

[0089] Based on the cross-attention weights, a context enhancement operation is performed on the feature vector within each modal channel to generate a cross-modal enhancement vector;

[0090] The cross-modal enhancement vectors and feature vector sets are subjected to residual connection and normalization to preserve modal independence and improve fusion robustness. The feature vector set includes speech feature vectors from the speech channel input, visual feature vectors from the image channel input, and text semantic vectors from the text channel input. It is the original input layer feature basis for the improved efficient cross-modal learning network.

[0091] The residual connections and normalized modal feature vectors are concatenated and nonlinearly transformed to output semantic embedding vectors, which serve as the fusion result of the improved efficient modal cross-learning network.

[0092] This invention constructs an improved, efficient modal cross-learning network to fuse feature vectors from speech, image, and text modalities. Specifically, it includes: inputting feature vectors from the three modalities into the speech, image, and text channels respectively, constructing a multi-channel fusion input structure; establishing bidirectional cross-attention paths between the three channels and calculating cross-attention weights to uncover alignment and complementarity relationships between modalities; performing context enhancement operations on the features within each modal channel based on the cross-attention weights to obtain cross-modal enhancement vectors. Subsequently, the method performs residual connection and normalization processing on the enhanced features and the original input to preserve modal independence and improve fusion stability; finally, the processed vectors are concatenated and input into a nonlinear transformation module to achieve consistent cross-modal semantic mapping. The final output semantic embedding vector possesses multimodal semantic collaborative expression capabilities, serving as a unified representation basis for downstream intent recognition and policy generation processes.

[0093] In one embodiment, a semantic intent graph is constructed based on interaction context information, and time-indexed edge weights and emotion-driven edge weights are introduced into the semantic intent graph. A structure-aware graph neural network is then used to encode the graph, and a specific method for outputting semantic intent vectors includes:

[0094] Collect voice input, text input, system task response, and explicit feedback data during user interaction with the agent within a preset time window to generate interaction context information;

[0095] The interaction context information is organized according to the interaction round number and timestamp information. The corresponding text semantic content, interaction intent identifier, context state parameters and voice emotion vector in each round of interaction are extracted to construct a set of interaction round sequences.

[0096] Based on the set of interaction round sequences, key semantic units are identified and abstracted into intent nodes, task nodes, and emotion nodes. A node set is constructed, and each node is given a timestamp, emotion intensity value, and interaction round identifier.

[0097] A set of directed edges is established based on the semantic dependency relationship and temporal causal relationship between nodes. This set includes intention-dependent edges that reflect semantic triggering, time-indexed edges that reflect temporal evolution, and emotion-driven edges that reflect user emotional changes. Each edge is assigned a decay factor, an emotion coupling factor, and a priority factor as edge weight parameters.

[0098] The set of nodes and the set of directed edges are combined to form a semantic intent graph. A structure-aware graph neural network based on edge type grouping is introduced to perform neighbor aggregation operation on different edge types and fuse information of multiple edge types through a gating mechanism.

[0099] In the aggregation stage of the structure-aware graph neural network, a time decay function and an emotion modulation function are introduced to dynamically weight the feature propagation process of time-indexed edges and emotion-driven edges, generating a semantic intent vector containing information from multiple semantic pathways.

[0100] This invention constructs a semantic intent graph based on interactive context information and encodes it using a structure-aware graph neural network. Specifically, the method extracts information such as voice input, text input, system response, and user feedback from multi-turn interaction history and organizes them in a structured manner according to time sequence and turn labels. The method constructs a semantic graph using semantic units, task intent, and emotional state as nodes, and builds a set of directed edges based on semantic dependencies, temporal evolution, and emotional changes, introducing time-indexed edge weights and emotion-driven edge weights as dynamic attributes of the graph. The graph is then subjected to edge type grouping and gating fusion using a structure-aware graph neural network. During the aggregation process, a time decay function and an emotion modulation function are introduced to enhance the model's ability to model semantic evolution paths and emotional state changes. The final output semantic intent vector comprehensively reflects the user's current intent state and interaction evolution trend, providing a semantic decision-making basis for personalized behavior generation.

[0101] In one embodiment, the specific method for collecting user behavior data during interaction, extracting modal style vectors through a cross-modal style contrastive learning mechanism, and generating personalized style encoding vectors through a hierarchical nested structure includes:

[0102] Collect personalized behavioral data of users during interaction with the intelligent agent, including voice tone change parameters, facial expression image frame sequences and natural language text input, and synchronize and align them according to the interaction rounds and timestamps;

[0103] Style feature normalization is performed on the individual behavior data, and modal style comparison sample sets of positive sample pairs and negative sample pairs are constructed. Positive sample pairs are derived from multiple rounds of behavioral expressions of the same user, while negative sample pairs are derived from samples generated by other users or the system.

[0104] Speech style vectors, image style vectors, and text style vectors are extracted through a modal style coding network, and contrastive learning training is performed based on the modal style contrastive loss function to enhance intramodal consistency and intermodal discriminability.

[0105] The speech style vector, image style vector and text style vector are input into a hierarchical nested structure to generate a low-level style representation, a middle-level aggregation representation and a high-level fusion representation in sequence. The middle-level aggregation representation fuses modal style features through an aggregation network, and the high-level fusion representation further fuses user behavior history summary vector and long-term semantic preference vector.

[0106] The low-level style representation, the mid-level aggregation representation, and the high-level fusion representation are fused into a personalized style encoding vector through a jump connection mechanism.

[0107] This invention first collects personalized behavioral data of users during interactions with an intelligent agent. This data includes speech intonation variation parameters, facial expression image frame sequences, and natural language text input. To ensure temporal consistency between modalities, the method performs frame-level synchronization alignment of the data based on the interaction round number and timestamp information, constructing a structured sequence of personalized behavioral samples. Subsequently, the method performs style feature normalization processing on the collected multimodal behavioral data and constructs positive and negative sample pairs for contrastive learning. Positive sample pairs originate from the same user's behavioral expressions in different interaction rounds, while negative sample pairs consist of behaviors generated by other users or the system. The method uses a modal style coding network to encode speech, image, and text modalities respectively, extracting speech style vectors, image style vectors, and text style vectors. Contrastive learning training is completed by introducing a modal style contrastive loss function, improving intramodal consistency and intermodal discriminability. To further integrate multimodal style features, this method employs a hierarchical nested structure to sequentially generate a low-level style representation, a mid-level aggregation representation, and a high-level fusion representation. The mid-level aggregation representation integrates style information across the three modalities, while the high-level fusion representation incorporates user behavior history summary vectors and semantic preference vectors. Finally, the low-level, mid-level, and high-level representations are fused into a unified personalized style encoding vector via skip connections, serving as one of the key inputs in the behavior decision-making process.

[0108] In one embodiment, the semantic embedding vector, semantic intent vector, and personalized style encoding vector are input into the personalized control gating mechanism. A feature channel linear modulation method is used to regulate the intermediate layer representation in the interaction strategy generation process, and the specific method for outputting the representation vector includes:

[0109] It receives semantic embedding vectors, semantic intent vectors, and personalized style encoding vectors, adjusts them to feature representations of the same dimension, and generates policy control channel representations through the linear mapping module.

[0110] The adjusted semantic embedding vector, semantic intent vector, and personalized style encoding vector are subjected to feature channel-level weight calculation to construct a three-way control weight matrix, which is used to represent semantic dominance, intent urgency, and personalized priority, respectively. The feature channel-level weight calculation is jointly calculated by learnable parameters and context state.

[0111] The three-way control weight matrix is ​​input into the feature channel linear modulation module to recalibrate the channel weights of the intermediate feature layer in the behavior strategy generation module, thereby strengthening the personality-related dimensions and suppressing irrelevant or redundant feature dimensions.

[0112] A gating selection mechanism is added after the linear modulation module of the feature channel. Based on the current environmental state, task type and user's historical policy response, the optimal path is selected from multiple policy sub-paths for feature propagation, and the final regulated representation vector is output.

[0113] This invention introduces a feature channel linear modulation mechanism in the fusion representation stage, combined with a gating path selection strategy, to enhance the semantic adaptability and personalized expression of the final behavior output. The method first unifies the dimensions of the semantic embedding vector, semantic intent vector, and personalized style encoding vector, and constructs corresponding policy control channel representations through linear mapping operations. Based on this, channel-level weight coefficients reflecting semantic dominance, intent urgency, and personality priority are calculated, considering the current interaction context and task features. By introducing a feature channel linear modulation method, the representation of the intermediate layer in the policy generation process is weighted and controlled, strengthening feature channels highly correlated with user personality and current intent, and weakening redundant or irrelevant feature dimensions, thereby improving the accuracy and efficiency of the policy generation representation. Furthermore, after modulation, the method introduces a gating selection mechanism, dynamically selecting the optimal feature transmission path from multiple preset policy paths based on task type, environmental state, and historical behavior response information, achieving targeted output of behavior representation and providing a structurally optimized policy foundation for subsequent behavior generation.

[0114] In one embodiment, the specific method for generating a multimodal behavior output sequence from the characterization vector input behavior strategy generation module includes:

[0115] The representation vector is input into the multi-channel behavior strategy generation module and mapped to the speech response channel, facial expression channel and body movement channel respectively.

[0116] In the speech response channel, the language generation network is used to generate speech command vectors with specific rhythm, emotional intensity and prosodic features based on the current semantic embedding and intonation control parameters, and outputs a text and audio dual-mode structure to drive the speech synthesizer.

[0117] In the facial expression channel, based on the emotion state encoding and personality style vector, the corresponding expression parameter set is selected and fine-tuned, including eyebrow and eye state, mouth dynamics and head posture, to generate and output a standardized facial movement control instruction sequence.

[0118] In the limb movement channel, the semantic intent vector and the action type library are combined to generate limb movement control parameters through the action prediction network, including trunk posture changes, hand posture key points and movement trajectory control vectors, and output a continuous and driveable command stream.

[0119] The instructions output from the voice response channel, facial expression channel, and body movement channel are synchronized in time, conflict detected, and prioritized to generate a unified structured multimodal behavior output sequence.

[0120] This invention first inputs a modulated representation vector into a multi-channel behavior strategy generation module, mapping it to a speech response channel, a facial expression channel, and a body movement channel to generate control commands for different expression modalities. In the speech response channel, the method, based on a language generation network, combines semantic embedding vectors and intonation control parameters to generate speech command vectors with rhythmic features, emotional intensity, and personalized prosody, simultaneously outputting dual-modal data of text and audio to drive the speech synthesizer. In the facial expression channel, the method selects and fine-tunes a set of expression parameters based on emotion state encoding and personality style vectors, generating a standardized sequence of facial movement control commands, including eyebrow and eye states, mouth dynamics, and head posture. In the body movement channel, the method combines semantic intent vectors and a preset action type library, utilizing a motion prediction network to generate body movement control parameters, including torso posture changes, hand keypoint distribution, and movement trajectory, outputting a continuous and executable stream of action commands. To ensure the coordination and naturalness of the final output behavior, this method performs time-series synchronization processing on the control commands of the three-channel output. Combined with a conflict detection mechanism and a priority sorting strategy, it generates a unified and structured multimodal behavior output sequence, providing a complete and controllable basis for the synchronous response of the digital image.

[0121] In one embodiment, a specific method for generating natural responses during user interaction by driving a digital avatar to respond synchronously based on a structured multimodal behavior output sequence includes:

[0122] Receive multimodal behavior output sequences, extract voice response commands, facial expression commands, and body movement control commands by channel, and perform frame-level synchronization according to a unified timeline;

[0123] The voice response command is input into the speech synthesis module, which generates a speech audio stream with personalized prosodic features based on the target intonation parameters, semantic content and speech style vector, and synchronizes with the lip animation generation module.

[0124] The facial expression action commands are mapped to the high-precision expression driving engine of the digital image, and the facial muscle control parameters are adjusted step by step according to the key action frames to achieve continuous and natural dynamic evolution of expressions, and the expression intensity is automatically adjusted according to the context and emotional state.

[0125] Inputting limb movement control commands into the skeletal drive system of the digital avatar drives the multi-joint movements of the torso, hands, and head to execute posture changes, gestures, and interactive actions, achieving humanoid responses that match semantic commands;

[0126] A response consistency scheduler is introduced into the overall driving process to monitor the synchronization indicators of the three channels of voice, expression and action in real time, and to fine-tune and repair based on frame rate offset, response delay and action conflict signals.

[0127] The collaboratively driven digital avatar behavior results are rendered and output in real time, enabling the generation of natural responses with semantic consistency, emotional coherence, and personalized expression characteristics, thus completing a full round of interaction with the user.

[0128] This invention provides synchronous response control for digital avatars based on structured multimodal behavior output sequences, enabling natural response generation during user interaction. Specifically, it first receives the multimodal behavior output sequence and extracts corresponding control commands for three channels: speech, facial expressions, and body movements. Simultaneously, all channel commands are processed for frame-level synchronization along a unified timeline. In the speech channel, speech response commands are input to a speech synthesis module, which combines target intonation parameters, semantic content, and speech style vectors to generate a personalized prosodic audio stream. This stream is then linked with a lip-syncing animation module to ensure a high degree of consistency between speech output and lip movement dynamics. In the facial expression channel, facial expression movement commands are mapped to the digital avatar's expression-driven engine. Keyframes adjust facial muscle parameters to achieve natural and continuous expression transitions, and the intensity of expressions is dynamically adjusted based on the context and emotional state. In the body movement channel, motion control commands are input to a skeletal drive system, driving the digital avatar's torso, hands, head, and other multi-joint structures to complete motion outputs, including posture changes and gesture expressions, achieving semantically consistent humanoid behavioral responses. To ensure the temporal and logical consistency of multimodal output, the system introduces a response consistency scheduler to monitor the synchronization indicators of the three channels—voice, facial expression, and motion—in real time, and dynamically correct them based on frame rate offset, response latency, and conflict signals. Finally, the coordinated digital avatar behavior is rendered and output in real time, generating a highly integrated, natural, and fluid interactive response in terms of semantics, emotion, and personality, completing a full interaction with the user.

[0129] To verify the feasibility and effectiveness of this invention in real-world interactive scenarios, this embodiment deploys a multimodal perception-based intelligent agent digital avatar interaction generation method of this invention in a company's digital employee reception system. The scenario is an intelligent guide and interactive platform in the company's smart exhibition hall. This platform is used to provide visiting guests with comprehensive services such as company introduction, business consultation, interactive Q&A, and avatar demonstration. It supports various human-computer interaction forms such as voice dialogue, facial expression feedback, and action demonstration, and needs to have a high degree of interaction naturalness, emotional expression ability, and user adaptability.

[0130] Traditional intelligent tour guide systems typically rely on voice broadcasts for responses, often accompanied by fixed facial expressions or simple action templates. This approach lacks understanding and adaptation to the visitor's semantic intent, emotional state, and personality traits, resulting in rigid and inflexible responses and a poor user experience. For example, when a visitor is upset or asks a sensitive question, the system often responds with a mechanical voice, sometimes even leading to semantic misunderstandings or inconsistencies between facial expressions and tone of voice, failing to create a genuine and warm interactive experience.

[0131] In this scenario, the present invention deploys a digital avatar intelligent agent with multimodal feature processing capabilities to collect visitors' voice tone, facial expressions and language text content, extract the current conversation goal using semantic embedding and intent modeling mechanisms, and then dynamically generate multimodal responses with semantic, emotional and personality adaptation capabilities by combining personalized behavior data.

[0132] To verify system performance, the testing team deployed and ran the system continuously for 7 days in an exhibition hall, collecting key indicators including interaction response latency, emotion consistency score, and user satisfaction, and conducting comparative tests with traditional voice guide systems. The results are as follows:

[0133] Table 1 lists the average performance data (in standardized proportions, milliseconds, or percentages) of the method of this invention under multiple indicators, covering dimensions such as response timeliness, emotion matching rate, personal expression suitability, and user satisfaction. Data collection was based on a total of 156 sets of visitor interaction samples to ensure the statistical reliability of the results.

[0134] Table 1

[0135] Serial Number Test metrics Average value of the method of the present invention Traditional system average value range of change 1 Average response latency (ms) 480 960 ↓50.0% 2 Consistency score of emotional expression (out of 5 points) 4.63 2.71 ↑70.8% 3 Action and speech synchronization deviation (frames) 2.1 6.9 ↓69.6% 4 Personalized intonation accuracy rate (%) 91.2% 58.6% ↑55.6% 5 Facial expression adjustment response accuracy (%) 87.5% 46.3% ↑89.0% 6 Multimodal behavioral output coordination score 4.71 3.05 ↑54.4% 7 User interaction satisfaction rating (out of 10) 9.1 6.3 ↑44.4% 8 Interaction interruption rate (%) 2.3% 8.5% ↓72.9% 9 Model real-time load rate (CPU%) 67.4% 83.9% ↓19.7% 10 Number of manual intervention requests per day 1.1 times 5.7 times ↓80.7%

[0136] As can be seen from the above data, the method of the present invention is significantly superior to traditional systems in terms of response timeliness, accuracy of emotion expression, coordination of voice and action, and user experience satisfaction. In particular, it has obvious advantages in personalized tone recognition and consistency of behavior output. It effectively solves the problems of lack of naturalness in interaction and rigid and monotonous feedback in the prior art, and fully demonstrates the advanced nature and practical value of the present invention in practical application scenarios.

[0137] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating interactive digital avatars of intelligent agents based on multimodal perception, characterized in that, include: Collect multimodal input data from users, perform preprocessing and feature extraction on the multimodal input data in sequence, and generate a feature vector set; The feature vector set is input into an improved high-efficiency modal cross-learning network for multimodal feature fusion processing to obtain semantic embedding vectors. Based on the interaction context information, a semantic intent graph is constructed, and time index edge weights and emotion-driven edge weights are introduced into the semantic intent graph. The graph is encoded using a structure-aware graph neural network to output a semantic intent vector. Collect user behavior data during the interaction process, extract modal style vectors through a cross-modal style contrast learning mechanism, and generate personalized style encoding vectors through a hierarchical nested structure; The semantic embedding vector, semantic intent vector, and personalized style encoding vector are input into the personalized control gating mechanism. The intermediate layer representation in the interaction strategy generation process is controlled by the feature channel linear modulation method, and the representation vector is output. Input the representation vector into the behavior strategy generation module to generate a multimodal behavior output sequence; The digital avatar is driven to respond synchronously based on the structured multimodal behavior output sequence, thus generating natural responses during user interaction. The specific method for inputting the feature vector set into the improved high-efficiency modal cross-learning network and performing multimodal feature fusion processing to obtain semantic embedding vectors includes: The speech feature vectors, image feature vectors, and text semantic vectors in the feature vector set are respectively input into the speech channel, image channel, and text channel of the improved efficient modality cross-learning network to construct a multi-channel fusion input structure; Bidirectional cross-attention paths are constructed between the speech channel and the image channel, the image channel and the text channel, and the speech channel and the text channel, and the cross-attention weights between each modal channel are calculated. Based on the cross-attention weights, a context enhancement operation is performed on the feature vector within each modal channel to generate a cross-modal enhancement vector; Residual connection and normalization are performed on the cross-modal enhancement vector and feature vector set to preserve modal independence and improve fusion robustness; The residual connections and normalized modal feature vectors are concatenated and nonlinearly transformed to output semantic embedding vectors, which serve as the fusion result of the improved efficient modal cross-learning network.

2. The method for generating interactive digital avatars of intelligent agents based on multimodal perception according to claim 1, characterized in that, The process involves collecting multimodal input data from users, preprocessing the multimodal input data, and extracting features to generate a feature vector set. Specifically: Collect user's multimodal input data, which includes voice data, facial images, and text information; The speech data is subjected to noise reduction, endpoint detection and speech segmentation. The short-time energy, spectral envelope and Mel frequency cepstral coefficients of the speech are extracted to generate speech feature vectors. Face detection, key point localization, and expression segmentation are performed on facial images. The motion amplitude features and muscle movement amplitude change features of facial regions are extracted to generate image feature vectors. The text information is segmented, part-of-speech tagging and dependency parsing are performed to extract semantic relationship features and keyword vector representations, and generate text semantic vectors. The speech feature vectors, image feature vectors, and text semantic vectors are aligned according to timestamps, and a unified vector dimension normalization method is used to generate a time-synchronized multimodal input feature vector set.

3. The method for generating interactive digital avatars of intelligent agents based on multimodal perception according to claim 2, characterized in that, The specific method for constructing a semantic intent graph based on interaction context information, introducing time-indexed edge weights and emotion-driven edge weights into the semantic intent graph, encoding the graph using a structure-aware graph neural network, and outputting semantic intent vectors includes: Collect voice input, text input, system task response, and explicit feedback data during user interaction with the agent within a preset time window to generate interaction context information; The interaction context information is organized according to the interaction round number and timestamp, and the corresponding text semantic content, interaction intent identifier, context state parameters and voice emotion vector in each round of interaction are extracted to construct a set of interaction round sequences; Based on the set of interaction round sequences, key semantic units are identified and abstracted into intent nodes, task nodes, and emotion nodes. A node set is constructed, and each node is given a timestamp, emotion intensity value, and interaction round identifier. A set of directed edges is established based on the semantic dependency relationship and temporal causal relationship between nodes. This set includes intention-dependent edges that reflect semantic triggering, time-indexed edges that reflect temporal evolution, and emotion-driven edges that reflect user emotional changes. Each edge is assigned a decay factor, an emotion coupling factor, and a priority factor as edge weight parameters. The set of nodes and the set of directed edges are combined to form a semantic intent graph. A structure-aware graph neural network based on edge type grouping is introduced to perform neighbor aggregation operation on different edge types and fuse information of multiple edge types through a gating mechanism. In the aggregation stage of the structure-aware graph neural network, a time decay function and an emotion modulation function are introduced to dynamically weight the feature propagation process of time-indexed edges and emotion-driven edges, generating a semantic intent vector containing information from multiple semantic pathways.

4. The method for generating interactive digital avatars of intelligent agents based on multimodal perception according to claim 3, characterized in that, The specific method for collecting user's personalized behavior data during the interaction process, extracting modal style vectors through a cross-modal style contrast learning mechanism, and generating personalized style encoding vectors through a hierarchical nested structure includes: Collect personalized behavioral data of users during interaction with the intelligent agent, including voice tone change parameters, facial expression image frame sequences and natural language text input, and synchronize and align them according to the interaction rounds and timestamps; Style feature normalization is performed on the individual behavior data, and a modal style comparison sample set of positive sample pairs and negative sample pairs is constructed. The positive sample pairs are derived from multiple rounds of behavior expression of the same user, and the negative sample pairs are derived from samples generated by other users or the system. Speech style vectors, image style vectors, and text style vectors are extracted through a modal style coding network, and contrastive learning training is performed based on the modal style contrastive loss function to enhance intramodal consistency and intermodal discriminability. Speech style vectors, image style vectors and text style vectors are input into a hierarchical nested structure to generate low-level style representation, mid-level aggregation representation and high-level fusion representation in sequence. The mid-level aggregation representation fuses modal style features through an aggregation network. The high-level fusion representation further fuses user behavior history summary vectors and long-term semantic preference vectors. The low-level style representation, the mid-level aggregation representation, and the high-level fusion representation are fused into a personalized style encoding vector through a jump connection mechanism.

5. The method for generating interactive digital avatars of intelligent agents based on multimodal perception according to claim 4, characterized in that, The specific method for inputting semantic embedding vectors, semantic intent vectors, and personalized style encoding vectors into the personalized control gating mechanism, and using a feature channel linear modulation method to regulate the intermediate layer representation in the interaction strategy generation process, and outputting the representation vector includes: It receives semantic embedding vectors, semantic intent vectors, and personalized style encoding vectors, adjusts them to feature representations of the same dimension, and generates policy control channel representations through the linear mapping module. The adjusted semantic embedding vector, semantic intent vector, and personalized style encoding vector are subjected to feature channel-level weight calculation to construct a three-way control weight matrix, which is used to represent semantic dominance, intent urgency, and personalized priority, respectively. The feature channel-level weight calculation is jointly calculated by learnable parameters and context state. The three-way control weight matrix is ​​input into the feature channel linear modulation module to recalibrate the channel weights of the intermediate feature layer in the behavior strategy generation module, thereby strengthening the personality-related dimensions and suppressing irrelevant or redundant feature dimensions. A gating selection mechanism is added after the linear modulation module of the feature channel. Based on the current environmental state, task type and user's historical policy response, the optimal path is selected from multiple policy sub-paths for feature propagation, and the final regulated representation vector is output.

6. The method for generating interactive digital avatars of intelligent agents based on multimodal perception according to claim 5, characterized in that, The specific method for generating a multimodal behavior output sequence by inputting the representation vector into the behavior strategy generation module includes: The representation vector is input into the multi-channel behavior strategy generation module and mapped to the speech response channel, facial expression channel and body movement channel respectively. In the speech response channel, the language generation network is used to generate speech command vectors with specific rhythm, emotional intensity and prosodic features based on the current semantic embedding and intonation control parameters, and outputs a text and audio dual-mode structure to drive the speech synthesizer. In the facial expression channel, based on the emotion state encoding and personality style vector, the corresponding expression parameter set is selected and fine-tuned, including eyebrow and eye state, mouth dynamics and head posture, to generate and output a standardized facial movement control instruction sequence. In the limb movement channel, the semantic intent vector and the action type library are combined to generate limb movement control parameters through the action prediction network, including trunk posture changes, hand posture key points and movement trajectory control vectors, and output a continuous and driveable command stream. The instructions output from the voice response channel, facial expression channel, and body movement channel are synchronized in time, conflict detected, and prioritized to generate a unified structured multimodal behavior output sequence.

7. The method for generating interactive digital avatars of intelligent agents based on multimodal perception according to claim 6, characterized in that, The specific method for generating natural responses during user interaction by driving digital avatars to synchronously respond based on structured multimodal behavior output sequences includes: Receive multimodal behavior output sequences, extract voice response commands, facial expression commands, and body movement control commands by channel, and perform frame-level synchronization according to a unified timeline; The voice response command is input into the speech synthesis module, which generates a speech audio stream with personalized prosodic features based on the target intonation parameters, semantic content and speech style vector, and synchronizes with the lip animation generation module. The facial expression action commands are mapped to the high-precision expression driving engine of the digital image. The facial muscle control parameters are adjusted step by step according to the key action frames to achieve continuous and natural dynamic evolution of expressions. The expression intensity is automatically adjusted according to the context and emotional state. Inputting limb movement control commands into the skeletal drive system of the digital avatar drives the multi-joint movements of the torso, hands, and head to execute posture changes, gestures, and interactive actions, achieving humanoid responses that match semantic commands; A response consistency scheduler is introduced into the overall driving process to monitor the synchronization indicators of the three channels of voice, expression and action in real time, and to fine-tune and repair based on frame rate offset, response delay and action conflict signals. The collaboratively driven digital avatar behavior results are rendered and output in real time, enabling the generation of natural responses with semantic consistency, emotional coherence, and personalized expression characteristics, thus completing a full round of interaction with the user.

Images