Virtual human real-time generation method and system based on expression control embedding space

Abstract

The invention relates to a multi-modal virtual human real-time generation method based on an expression control embedding space, and belongs to the field of artificial intelligence. According to the method, an expression control embedding space is constructed and used for fusing voice semantics, a rhythm structure and multi-dimensional emotion information, and continuous and controllable multi-modal driving vectors are generated. The whole system has an end-to-end linkage mechanism from audio input to expression and action output. Semantic features, rhythm structures and emotional states jointly act on generation paths of lip and upper body postures and expression modalities, and all modal features are fused and expressed in a unified control space through a collaborative coding and time sequence alignment mechanism. And finally, a high-consistency and high-fidelity virtual human video is generated in real time through an output scheduling mechanism. The method has remarkable advantages in the aspects of modal fusion consistency, generation expression naturalness and emotion control flexibility, and can be widely applied to key scenes such as virtual human broadcasting, voice interaction agency and meta-universe digital identity construction.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and relates to a method and system for real-time generation of multimodal virtual humans based on expression control embedding space. Background Technology

[0002] In recent years, with the rapid development of deep neural networks, speech synthesis, and computer graphics, virtual humans, as intelligent agents that integrate multimodal generation, real-time interaction, and personalized expression, are being widely applied in various scenarios such as virtual anchors, intelligent customer service, metaverse social networking, and immersive education. In particular, visual virtual human generation systems that achieve realistic lip movements, natural facial expressions, and upper body motion linkage under voice-driven conditions are gradually becoming a research hotspot in human-computer interaction technology. However, existing methods still face many technical challenges in areas such as multimodal feature modeling, modal coordination and fusion, emotion regulation and expression, and real-time response performance.

[0003] On the one hand, traditional voice-driven face synthesis systems mostly focus only on the correspondence between phonemes and lip movements, lacking the ability to adjust facial expressions and body postures through emotional semantics. This results in relatively rigid generated content at the expression level, unable to adapt to complex contextual changes and diverse user styles. On the other hand, current multimodal generation methods mostly adopt a phased, loosely coupled serial modeling pattern. Voice, emotion, expression, and posture control are often processed independently in different sub-modules, making it difficult to build a coordinated generation mechanism under a unified expressive semantic framework. Furthermore, although some methods can generate high-quality frame images, their complex structures or lengthy inference processes often fail to meet latency requirements when facing high-frequency tasks such as real-time dialogue or continuous interaction, limiting their practical application.

[0004] More critically, existing systems generally lack mechanisms for maintaining the physical structure and identity characteristics of virtual humans over the long term, making it difficult to achieve facial geometric consistency, style continuity, and behavioral regularity. This is especially true when generating long, emotionally rich, and coherent video sequences, which are prone to issues such as expression drift, style misalignment, and skipped frames, severely impacting the credibility and immersive experience of the virtual human's output. Therefore, a virtual human generation technology is still needed that is highly coupled and integrated across multiple modalities, possesses expressive control capabilities and adjustable emotion mechanisms, and supports low-latency real-time generation to achieve a more natural, credible, and controllable human-computer interaction process. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a method and system for real-time generation of multimodal virtual humans based on an expression control embedding space.

[0006] To achieve the above objectives, the present invention provides the following technical solution: On the one hand, a real-time generation method for multimodal virtual humans based on expression control embedding space is proposed, which includes: S1. Using speech modality and visual modality as input, construct a multimodal expression feature that includes speech semantic features, emotional state features, and static appearance features; S2. Alignment of speech semantic features, emotional state features and static appearance features based on a multimodal dynamic alignment mechanism using structural field embedding; S3. Establish an expression control embedding space based on a modality-dominated fusion mechanism. Dynamically adjust the dominant priority of speech, emotion, and posture in the generation-driven process according to the current context state, modality confidence, and historical expression state, and establish an expression-dominated control factor vector. S4. Through behavior-layered decoding, temporal flow smoothing and cross-modal collaborative correction, the fused expression vector is transformed into an executable sequence of behavioral instructions with coordinated linkage capabilities, ultimately driving the natural generation of the three modalities of lip movement, facial expression and posture. S5. Design modal mapping and image-driven signal generation mechanism, image rendering structure and enhancement mechanism, and real-time output control and rhythm scheduling mechanism, and perform end-to-end image synthesis based on the constructed fusion expression control vector.

[0007] Furthermore, in step S1, the process of establishing speech semantic features is as follows: the original speech is preprocessed by the speech analysis module, a short-time stable spectral structure is constructed using an overlapping frame segmentation algorithm with a Hanning window, and the Mel spectrum and fundamental frequency are extracted. The low-level acoustic features of speech energy form a time-series feature matrix. This involves introducing a speech-to-text converter to map audio into text representation, and using BERT for context embedding extraction to obtain a word-granular semantic vector sequence. Then, time synchronization and context joint encoding are completed through a bidirectional gated recurrent unit to obtain a unified semantic driving vector:

[0008] in, This is a gating matrix used to control the flow intensity of each dimension of the fused feature; These are the gating parameters that can be learned during training. These are the acoustic and semantic projection matrices, respectively. This represents the final fused speech features.

[0009] Furthermore, in step S1, the process of establishing emotional state features is as follows: a continuous emotion estimation mechanism based on latent variable modeling is adopted to construct vector representations on the emotional dimensions of arousal, pleasure, and dominance. Speech acoustic features The input consists of three convolutional channels, with each channel extracting the emotional response signal for a different time scale. The outputs of each branch are fused by the context-aware attention module to form an expression vector. Then, a control vector is generated by combining learnable channel weights:

[0010] in The channel attention weights are dynamically adjusted by contextual features and the generation state. Used to maintain the numerical stability of the emotion fusion vector; Then, an inter-channel residual coupling term is introduced. This is used to simulate the nonlinear interactions between different emotional timescales.

[0011] in The coupling strength coefficient between each channel pair is adaptively learned by the model during training; This indicates element-wise multiplication, used to extract co-activation dimensions; This indicates a channel coupling term, used to enhance the dynamic adjustment capability of the final output; The final control vector update is as follows:

[0012] in This is a residual adjustment factor used to control the degree of influence of the coupling term on the final emotion control signal.

[0013] Furthermore, in step S1, the process of constructing static appearance features is as follows: the input image is first standardized and then sent to the visual feature extraction module to generate an identity semantic representation. Simultaneously, a topological structure is constructed based on facial landmark maps, and structure vectors are extracted through graph convolutional networks. Introducing a structural attention weight matrix By strengthening the adjustment of important regional features, the system ultimately generates a static image embedding representation through a unified fusion expression mapping function; the process is represented as follows:

[0014] in, Represents a learnable transformation matrix. Represents the structural attention matrix, dynamically adjusting the region response; Represents a multi-layer nonlinear activation mapping. For bias; This is the final static embedding vector for the image.

[0015] Furthermore, in step S2, based on the multimodal dynamic alignment mechanism of structural field embedding, combined with knowledge graph structural representation and temporal decomposition strategy, the intermodal alignment relationship is constructed in the expression control embedding space. The process is as follows: First, a set of modal structures is constructed, corresponding to speech, action, and visual modalities respectively; within each modality, a dynamic semantic flow representation is defined as a time-series embedding:

[0016] in, , These represent the speech modality, action modality, and visual modality, respectively. Secondly, a unified modality graph is constructed based on a large-scale semantic mapping library. Node set Represents a multimodal unit, edge set Representing semantic or cooperative relationships; decomposing the original modal temporal features into coarse-grained and fine-grained paths through a two-layer temporal alignment mechanism:

[0017]

[0018] in, and This indicates shallow and deep multimodal alignment. Representing modes The Segment-level semantic embedding, based on graph mapping function Build; Indicates the first 3D representation features of frames; Indicates cross-modal attention to its matrix; It serves as a regularization factor, used to balance semantic synchronization and cross-attention consistency; Representing the Frobenius norm, pay attention to matrix differences during quantization; The L2 norm measures the projection error of a mode onto a semantic field.

[0019] Furthermore, in step S3, the process of constructing the dominant control factor vector is as follows: A combined priority modulation model is performed on the aligned modality embedding vectors to construct a three-channel attention modulation path, including a context guidance channel, a confidence evaluation channel, and a personalized adjustment channel. The context-guided channel integrates the fusion vector from the previous time step with the current semantic state to construct a modality-dominated contextual semantic reference. It introduces attention mechanisms and gating strategies to make the fusion process temporally continuous and semantically coherent. When the expression trend changes, the modality-dominant weights are adjusted to achieve dynamic responsive control. The confidence assessment channel performs multi-dimensional evaluation on the semantic stability, structural integrity, and temporal consistency of the current modality input, generates a confidence score as a reference for fusion weights, automatically detects noise, missing frames, or modality quality anomalies, and reduces the dominance of that modality in fusion. The personalized adjustment channel adjusts the fusion method based on the user-defined expression style tags or external adjustment factors, ultimately constructing an expression-dominant control factor vector:

[0020] in Represents the current modality confidence vector; Represents the modal adjustment projection matrix; Indicates a splicing operation; Ensure that weights are uniform and that a competitive mechanism is in place; Then, the modal embeddings are weighted and fused, and a cross-modal residual modulation term is introduced to capture the potential behavioral coupling relationships between the multimodalities.

[0021] Furthermore, in step S4, the process of behavior-layered decoding, temporal flow smoothing, and cross-modal collaborative correction is as follows: First, a unified driving vector parsing and modal deconstruction are performed: the fused expression control vector is input into the behavior-driven decoder and deconstructed into a set of modal components.

[0022] in These are the decoded lip movement parameter sequence, facial expression action unit sequence, and posture joint action sequence, respectively. For the number of frames, there are three learnable transformation matrices. With bias Used to model the driving bias of each modality; Then, temporal consistency optimization and dynamic constraints are established: a dynamic optimization strategy based on differential consistency constraints is introduced to normalize the differences in behavioral change rates across modalities, so that the output frame sequence maintains continuity and consistency in the dimensions of emotional intensity, facial tension, and movement amplitude.

[0023] in Indicates the modal output at time step The range of change, and the constraints ensure that the behavioral rhythms of the three modalities remain coupled with the dynamic trends; Finally, a mechanism for correcting behavioral conflicts and maintaining style continuity is designed. A style-guided behavior inhibition gate is introduced, using the parsed style intent as a guide. In the modal decoding output, abnormal actions that deviate from the style trajectory are subject to amplitude pruning and rhythm readjustment, ensuring that the expression maintains semantic style consistency at the visual perception level. The correction process is defined as follows:

[0024] in Represented as a style gating vector, it controls whether each modal channel is suppressed or relaxed. This represents the time-domain moving average function.

[0025] Furthermore, in step S5, the modality mapping and image driving signal generation mechanism is as follows: The modal control vectors output by the behavior consistency-driven module are decoded into frame-level image control tensors, and structural attention enhancement and temporal caching mechanisms are introduced to ensure that image generation has sufficient response accuracy and temporal continuity. It then fuses the three behavioral components after decoding the control vectors: lip movement parameter vectors, facial expression vectors, and upper body pose vectors. Three modality-specific mapping sub-networks are introduced to guide image generation: Lip2UV, Face2AU, and Pose2HM. Lip2UV maps lip movement parameters to lip UV offset fields to control lip shape region mesh deformation; Face2AU maps facial expression vectors to AU action unit activation maps to drive the facial expression generation module; and Pose2HM converts pose vectors into keypoint heatmaps to represent head and shoulder movement trajectories. The above process is uniformly modeled as follows:

[0026] in For modal mapping networks, This represents a cross-modal splicing operation, which ultimately generates a control tensor. It possesses spatial features and structures exhibiting multimodal behavior; A structure-aware attention mechanism is introduced into the expression path. Utilizing the static portrait features output by the image encoding module and combined with the expression vector of the current frame, a structure-weighted map is generated through an additive attention function to refine the expression responses of key facial regions.

[0027] in, This is a learnable facial expression response mapping matrix.

[0028] Furthermore, in step S5, the image rendering structure and enhancement mechanism adopt a two-level generator architecture, including a coarse-level configuration predictor and a detail enhancement decoder, wherein... The coarse-level configuration predictor is composed of a multi-scale convolutional network. It uses structure-preserving attention to perform preliminary modeling of the lip, eyebrow, and eyelid regions. In the upsampling stage, the feature map is combined with the cross-modal feature residual map to achieve the preliminary mapping of the driving signal to the visual structure. The frame control tensor is input into the coarse-level configuration predictor, which generates an initial medium-resolution image frame based on the encoder-decoder network as a spatial layout sketch of pose and expression. The detail enhancement decoder takes the coarse image frame and the original static reference image features as input. It adopts a branched residual fusion structure, which processes local features through expression branches, skin color branches, and head structure branches, and uses a style reprojection module to regulate the consistency of image style. The final output image frame retains the static portrait identity features and is consistent with the control signal in terms of lip movement, expression, and posture. The style reprojection module introduces a style distribution projection matrix at the end of the decoding to project and compress potential activation features from different modalities. We design an expression-aware reconstruction loss function, and introduce expression semantic consistency supervision and structural adversarial loss on the basis of L1 reconstruction:

[0029] in This represents the function for extracting facial expression action units; Cross-entropy; This indicates the adversarial loss guided by the discriminator.

[0030] Furthermore, in step S5, the output control and rhythm scheduling mechanism includes a frame buffer scheduling module, a speech-modal rhythm synchronization mechanism, and an output frame rate adaptive control mechanism. The frame buffer scheduling module introduces a multi-level frame buffer, including an action buffer queue, an expression buffer queue, and a video output frame buffer queue. Each queue maintains past frame buffers. The frame modal output uses a time-sliding window mechanism for frame interpolation and resampling of missing frames. When a frame of data in a certain modality is lost or arrives late, dynamic frame replacement is performed using the residual from the previous frame. The replacement formula is as follows:

[0031] in, Feature tensors representing posture or facial expression modalities Indicates the time smoothing factor; The speech-modal rhythm synchronization mechanism uses audio prosodic features as the driving rhythm benchmark, constructs a beat time axis, and performs aligned resampling on the modal control signal: extracting energy peaks, pause intervals, and beat periods from the speech signal; and outputting the control tensor for each frame based on the time axis. Perform constraint matching; if the current modal response time difference This triggers modal delay compensation; the rhythm alignment function is expressed as:

[0032] Among them, when Exceeding the semantic rhythm error threshold At the same time, the temporal output timing is dynamically adjusted according to the rhythm fitting function to ensure temporal consistency between lip movements, actions and syllable semantic nodes; The output frame rate adaptive control mechanism adjusts the rendering frame rate according to the strategy. The output frame rate scheduling function is as follows:

[0033] in, Indicates the current output frame rate; , These represent the minimum output frame rate and the high output frame rate, respectively. Indicates the current frame generation delay; Indicates the maximum generation delay threshold; This indicates the current occupancy rate of the output buffer queue; This indicates the output frame rate at the previous moment.

[0034] On the other hand, a real-time multimodal virtual human generation system based on an expression control embedding space is also proposed. This system is used to execute the aforementioned real-time multimodal virtual human generation method based on an expression control embedding space. The system includes: a speech analysis module, an emotion recognition and control module, an image encoder, an expression control embedding space, a multimodal generation architecture module, a low-latency rendering scheduling module, a modality mapping and image-driven signal generation mechanism module, an image rendering structure and enhancement mechanism module, and a real-time output control and rhythm scheduling mechanism module. The speech analysis module performs multi-scale spectral analysis and rhythm deconstruction on the input speech signal, extracts low-level acoustic features including Mel spectrum and fundamental frequency, enhances the semantic representation of language, obtains speech features, and transmits them to the emotion recognition and control module. The emotion recognition and control module identifies emotional information in speech and generates adjustable emotion embedding vectors through channel attention mechanism and language context alignment strategy. The image encoder processes a static reference image, extracts identity-preserving and shape information, generates texture embeddings of the static image, and extracts static appearance features. The expression control embedding space serves as a unified representation and control domain for all modal features, mapping features from all different modalities to a shared coordinate system and performing alignment and dynamic adjustment between modalities. The multimodal generation architecture module receives control vectors generated in the representational control embedding space and uses the fused representational control vectors to perform end-to-end image synthesis. The low-latency rendering scheduling module dynamically monitors inter-frame changes in the control vector and determines whether to trigger frame-level rendering based on the expression mutation threshold. The modality mapping and image-driven signal generation mechanism module decodes the modality control vector output by the behavior consistency driving module into a frame-level image control signal and generates a control tensor for use in image generation. During the rendering stage, the image rendering structure and enhancement mechanism module combines the conditional control generation framework and the expression-aware enhancement module to improve the detail and style consistency of the image through a two-level generator architecture. The real-time output control and rhythm scheduling mechanism module ensures time consistency and efficient real-time output during the image generation process, and optimizes the generation effect through rhythm synchronization and frame buffer scheduling.

[0035] The beneficial effects of this invention are as follows: Compared with existing virtual human generation technologies, the multimodal virtual human generation method and system based on expression control embedding space proposed in this invention have significant improvements in multimodal alignment and fusion, semantic-driven expression, continuous emotion control, and real-time generation performance, specifically in the following aspects: First, this invention breaks through the structural limitations of traditional "modal isolation modeling" by constructing a unified Expression Control Embedding Space (ECES), enabling cross-modal alignment and synchronous generation of lip movements, facial expressions, and upper body actions driven by speech. This control space possesses dynamic and dominant adjustment capabilities, adaptively adjusting modal weights based on context, speech rate, and emotional intensity. This results in stronger semantic consistency and behavioral coordination at the expression level, significantly improving the naturalness and contextual awareness of the generated content.

[0036] Secondly, this invention introduces an adjustable emotion vector modeling mechanism, which differs from traditional discrete emotion classification. It uses a continuous latent variable expression method to express emotional parameters such as arousal, pleasure, and sense of control in the dimensional space, possessing micro-adjustment capabilities and personalized mapping capabilities. The emotion control module establishes a linkage mapping relationship with speech rhythm, facial expression intensity, and posture response amplitude, enabling the system to achieve a more delicate and natural emotional expression capability in the emotional dimension.

[0037] Furthermore, in terms of generating structures, this invention proposes an identity-preserving image coding and structure enhancement module. By combining face topology graph modeling and graph convolution structure feature extraction, it maintains the facial consistency and style stability of virtual humans in consecutive video frames, overcoming phenomena such as facial drift and style jumps in existing systems, and effectively improving the visual credibility and consistency of virtual human images in long-term generation tasks.

[0038] In terms of generation efficiency, this invention designs a frame buffer scheduling mechanism and a rhythm-aware driving strategy. It utilizes the synchronous modal control of speech prosody signals to control the rhythm, combined with frame-level dynamic sampling and adaptive latency adjustment, to achieve millisecond-level low-latency response and stable video frame rate output. Compared to traditional serial multi-stage inference models, this mechanism significantly improves the system's real-time performance and adaptability, making it suitable for high-frequency real-time interactive needs in scenarios such as virtual human broadcasting, interactive dialogue, and video generation.

[0039] In addition, the system has a good modular design and scalability, supports multi-language input, multi-style expression templates and external expression modifier access, and can be widely used in complex application environments such as metaverse virtual character generation, virtual anchor system, intelligent customer service interface, human-computer voice interaction platform, and education companion system. It has broad engineering promotion value and commercialization prospects.

[0040] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0041] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the overall process of the real-time generation method of multimodal virtual human based on expression control embedding space according to an embodiment of the present invention; Figure 2 This is a diagram of the semantic-driven emotion embedding network structure according to an embodiment of the present invention; Figure 3 This is a simplified schematic diagram of the multimodal fusion architecture according to an embodiment of the present invention. Detailed Implementation

[0042] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0043] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0044] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0045] Please see Figures 1-3 This is a method and system for real-time generation of multimodal virtual humans based on expression control embedding space.

[0046] Example 1 This embodiment first describes a method for real-time generation of multimodal virtual humans based on expression control embedding space, such as... Figure 1 As shown, it specifically includes the following steps: S1: Multimodal Expression Feature Construction. First, the system introduces a semantic-emotion co-modeling mechanism to achieve dual modeling of language structure and emotional intent in speech information. Speech modal input first undergoes multi-scale spectral analysis and rhythm deconstruction by the Speech Analysis Module (SAM), while pre-trained semantic embedding (BERT) is introduced to enhance the language semantic representation. Subsequently, the speech semantic features are fed into the Emotion-Controlled Understanding Engine (E-CUE), where an adjustable emotion embedding vector is constructed through channel attention mechanisms and language context alignment strategies. This vector possesses cross-linguistic adaptability and personalized adjustment capabilities. Visual modal input is a static reference image, which is used by an image encoder to extract identity preservation and facial features to construct a human image texture embedding. The overall input process completes the construction of three modalities: speech semantics, emotional state, and static appearance features, and guides them uniformly to subsequent driving processes.

[0047] More specifically, in step S1, based on a user-inputted speech and a static image, and focusing on the three core dimensions of "expression drivers"—semantic rhythm, emotional intent, and identity structure—the process involves extracting linguistic and prosodic information from the speech, constructing an adjustable emotion vector, and establishing a visual identity feature representation from the static image. Specific steps include: S11: Speech Semantic-Driven Modeling. In this invention, speech modality serves as the core driving information, responsible for providing three types of driving elements: content semantics, prosodic rhythm, and emotional tone. First, the system preprocesses the original speech through a speech analysis module, employs an overlapping framing algorithm with a Hanning window to construct a short-time stable spectral structure, and extracts the Mel spectrum and fundamental frequency. Low-level acoustic features such as speech energy form a time-series feature matrix. To enhance language comprehension, the system introduces a speech-to-text converter to map audio into text representation, and uses BERT for context embedding extraction to obtain word-level semantic vector sequences. Then, time synchronization and context joint encoding are completed through a bidirectional gated recurrent unit to obtain a unified semantic driving vector. Unlike traditional systems, this invention introduces a gated semantic fusion mechanism to adjust the weights of acoustic and language modalities in real time, dynamically adjusting the main sources of lip movements, actions, and rhythm within the generated context. The final fused speech features are represented as follows: As shown in equation (1):

[0048] in This is a gating matrix used to control the flow intensity of each dimension of the fused feature; These are the gating parameters that can be learned during training. These are the acoustic and semantic projection matrices, respectively. The fused vector ultimately enters the control embedding space. Maintain content-rhythm synergy and serve as the semantic master entry point for subsequent generation processes.

[0049] S12: Adjustable Emotion Vector Modeling. To ensure that the generated video frames exhibit consistent emotional expression and a natural style, this invention introduces an emotion recognition and control module to construct a continuous, controllable, and semantically driven emotion modulation path. Unlike traditional systems based on discrete classification (such as anger and happiness), the emotion recognition and control module employs a continuous emotion estimation mechanism based on latent variable modeling, constructing vector representations across three emotion dimensions: arousal, pleasure, and dominance. Speech acoustic features The signals are fed into a three-branch convolutional channel, with each channel extracting emotional response signals for different time scales (short-term intonation changes, medium-term rhythm fluctuations, and long-term style tone). The outputs of each branch are fused by the context-aware attention module to form an expression vector. .

[0050] To support the generation of multiple styles, the emotion vector needs to be further modulated to generate a usable control vector. The system uses a learnable channel weight combination, as shown in equation (2):

[0051] in The channel attention weights are dynamically adjusted by contextual features and the generation state. The significance lies in maintaining the numerical stability of the emotion fusion vector, which helps the training of the generation module converge. In the formula, this vector is not only used for adjusting facial expression output, but also for fine-tuning lip-width and movement speed, achieving resonant control between speech content and emotional expression during the modality fusion stage.

[0052] To enhance the completeness of cross-channel emotional feature expression, the system introduces an inter-channel residual coupling term. This term is used to simulate the nonlinear interaction between different emotional time scales, such as the strengthening or suppression of style tone by rhythmic emotions, and is defined as shown in equation (3).

[0053]

[0054] in The coupling strength coefficient between each channel pair is adaptively learned by the model during training; This represents element-wise multiplication (Hadamard product), used to extract co-activation dimensions; This represents the channel coupling term, used to enhance the dynamic adjustment capability of the final output. The final control vector is updated as follows (4):

[0055] in The residual adjustment factor is used to control the degree of influence of the coupling term on the final emotion control signal. This mechanism not only improves the expressive subtlety of emotion modeling, but also enhances the system's adaptability to non-standard emotional states.

[0056] S13: To ensure individual consistency, visual stability, and cross-frame coherence during the virtual human generation process, this invention proposes a static image coding and identity preservation modeling mechanism (IPVE). This mechanism constructs a stable identity constraint vector by performing deep-level structural representation modeling of the input image to drive fusion control. For example... Figure 2 As shown, the input image is first standardized and then sent to the visual feature extraction module to generate an identity semantic representation. Simultaneously, a topological structure is constructed based on the facial landmark map, and a structure vector is extracted through a graph convolutional network (GCN). To further improve the response accuracy in key areas, a structural attention weight matrix is ​​introduced. Strengthen the adjustment of features in important regions (such as eyebrows, eyes, and mouth). Finally, the system generates a static image embedding representation by unifying and fusing the expression mapping function, as defined in equation (5):

[0057] in Represents a learnable transformation matrix. Represents the structural attention matrix, dynamically adjusting the region response; Represents a multi-layer nonlinear activation mapping. For bias; This is the final static embedding vector for the image. This embedding vector is used throughout the entire virtual human generation process, serving as a core control signal for identity preservation and visual style consistency. It participates in the expression, lip-sync, and motion synthesis modules, significantly improving the consistency of appearance and expression stability across time frames.

[0058] S2: Expressive Control Embedding Space Modeling and Multimodal Fusion. To achieve highly consistent, multimodal interactive human image generation control, this invention constructs an Expressive Control Embedding Space (ECES) as a unified representation and control domain for all modal features. ECES possesses heterogeneous modality alignment capabilities, temporal preservation properties, and an expressive-dominant adjustment mechanism. Specifically, as... Figure 3 As shown, the system projects lip movement parameters, upper body posture vectors, and emotion embedding vectors onto a shared coordinate system using a low-dimensional mapper. It also introduces a set of modality control factors, including lip movement weights, posture weights, and expression weights, to dynamically adjust the dominance of each modality during generation. The fusion strategy employs a multimodal hierarchical decoding structure, performing semantic enhancement and behavioral consistency alignment between modalities at the deep expression layer to ensure high expressive coordination and continuity in the output vectors. The final fused expression control vector is input as a unified driving signal to the generation system, providing the expressive semantic foundation for video frame construction.

[0059] More specifically, the expression control embedding space designed in step S2 includes the following steps: S21: A multimodal dynamic alignment mechanism based on structural field embedding is proposed. To achieve dynamic semantic alignment and structural synchronization control of heterogeneous modalities in a unified representation domain, this invention proposes a multimodal dynamic alignment mechanism based on structural field embedding. This mechanism combines knowledge graph structural representation and temporal decomposition strategy to construct intermodal alignment relationships in the Expression Control Embedding Space (ECES). First, a set of modal structures is constructed, corresponding to speech, action, and visual modalities respectively; the dynamic semantic flow representation within each modality is defined as a time-series embedding, as shown in Equation (6):

[0060] in, , These represent the speech modality, action modality, and visual modality, respectively. Secondly, the system constructs a unified modality graph based on a large-scale semantic mapping library (such as an emotion ontology dictionary, action corpus, and facial coding map). Node set Representing multimodal units (such as "joy", "turns up", "open arms"), edge sets It indicates semantic or cooperative relationships.

[0061] To achieve modal synchronization in the time dimension, the system decomposes the original modal temporal features into coarse-grained (segment-level) and fine-grained (frame-level) paths through a two-layer temporal alignment mechanism, as defined in equations (7) and (8):

[0062]

[0063] in and This indicates shallow and deep multimodal alignment. Representing modes The Segment-level semantic embedding, based on graph mapping function Build; Indicates the first 3D representation features of frames; Indicates cross-modal attention to its matrix; It serves as a regularization factor, used to balance semantic synchronization and cross-attention consistency; Representing the Frobenius norm, pay attention to matrix differences during quantization; The L2 norm measures the projection error of a mode onto a semantic field.

[0064] This mechanism supports semantic anchoring, behavior synchronization, and cross-channel interaction alignment between dynamic modalities, and provides a unified temporal representation after alignment. By introducing driving pathways into the expression control embedding space (ECES), the stability and naturalness of multimodal co-expression are ensured.

[0065] S22: To achieve expressive coordination and stability of generation behavior in the multimodal driving process, this invention constructs a modality-dominated fusion mechanism as the core fusion method in the Expression Control Embedding Space (ECES). This mechanism aims to dynamically adjust the dominant priority of speech, emotion, and posture modalities in the generation driving process based on the current context state, modal confidence, and historical expression dynamics, ensuring that the generated virtual human exhibits semantic-emotion-action consistent expressive behavior. The system first performs combined priority control modeling on the aligned modal embedding vectors to construct a three-channel attention modulation path: (1) Context-guided channel: By integrating the fusion vector from the previous time step with the current semantic state, a contextual semantic reference for modality-dominated regulation is constructed. The system introduces attention mechanisms and gating strategies to ensure temporal continuity and semantic coherence in the fusion process. When the expression trend changes (such as the tone changing from slow to fast), the modality-dominant weights can be adjusted in a timely manner to achieve dynamic responsive control.

[0066] (2) Confidence assessment channel: Multidimensional assessment of the semantic stability, structural integrity, and temporal consistency of the current modality input is performed to generate a confidence score as a reference for fusion weights. The system automatically detects noise, missing frames, or modality quality anomalies and reduces the dominance of that modality in the fusion process, thereby enhancing the stability and robustness of the multimodal fusion process.

[0067] (3) Personalized adjustment channel: The fusion method is adjusted according to the expression style label or external adjustment factor set by the user. For example, the emotion weight is higher for the "positive" style. Finally, the expression dominance control factor vector is constructed, as shown in Equation (9):

[0068] in This represents the current modality confidence vector (output by the entropy attention extraction module); Represents the modal adjustment projection matrix; Indicates a splicing operation; Ensure that the weights are uniform and that there is a competition mechanism. Then, the inputs of each modality are embedded and weighted and fused, and a cross-modal residual modulation term is introduced to capture the potential behavioral coupling relationship between multimodalities, such as the head lift caused by the rise in speech tone.

[0069] S23: Proposes a behavior consistency-driven mechanism for modeling. After constructing the Expression Control Embedding Space (ECES) and scheduling modal fusion weights, to ensure the final generated sequence maintains consistent expressiveness in terms of semantic fluency, modal coherence, and expressive naturalness, this invention designs a behavior consistency-driven mechanism oriented towards temporal stability and modal consistency. This mechanism transforms the fused expression vector into an executable sequence of behavioral instructions with coordinated linkage capabilities through three main paths: hierarchical behavior decoding, temporal flow smoothing, and cross-modal collaborative correction, ultimately driving the natural generation of lip movements, facial expressions, and poses. The main contents include: (I) Unified Driving Vector Analysis and Modal Deconstruction The system inputs the fused expression control vector into the behavior-driven decoder, which deconstructs it into a set of modal components, as shown in Equation (10):

[0070] in These are the decoded lip movement parameter sequence, facial expression action unit sequence, and posture joint action sequence, respectively. For the number of frames. Three learnable transformation matrices. With bias Used to model the driving bias of each modality.

[0071] (II) Timing Consistency Optimization and Dynamic Constraints To address the differences in temporal granularity and expression rhythm between modalities, the system introduces a dynamic optimization strategy based on differential consistency constraints. This mechanism normalizes the differences in behavioral change rates across modalities, ensuring that the output frame sequence maintains continuity and consistency in dimensions such as emotional intensity, facial tension, and movement amplitude, as shown in Equation (11):

[0072] in Indicates the modal output at time step The range of variation and the constraints ensure that the behavioral rhythms and dynamic trends of the three modalities remain coupled, and that modal shift problems such as "premature lip movement" and "delayed facial expression" do not occur.

[0073] (iii) Behavioral conflict correction and style continuity maintenance Furthermore, to prevent inconsistencies such as "facial over-motion + smooth speech" in the generated sequence, the system further introduces a style-guided behavior suppression gate. This module, guided by the parsed style intent, performs amplitude pruning and rhythm readjustment on abnormal actions that deviate from the style trajectory in the modal decoding output, ensuring that the expression maintains semantic style consistency at the visual perception level. The correction process is defined as shown in equation (12):

[0074] in Represented as a style gating vector, it controls whether each modal channel is suppressed or relaxed. This represents the time-domain moving average function.

[0075] This behavior consistency-driven mechanism, serving as the output execution layer of the expressive control embedded spatial framework, ensures that multimodal behaviors, dominated by fused control vectors, possess high rhythmic matching, emotional coordination, and visual coherence. Through the synergistic effect of modal residual modulation and style gating mechanisms, the system can effectively avoid abnormal output problems such as modal drift and emotional discontinuity, providing a stable, natural, and stylistically consistent flow of control instructions for the subsequent virtual human image rendering stage, further enhancing the subjective controllability and user-perceived realism of the digital human generation process.

[0076] S3: Image Generation and Output Scheduling Control. In the output stage, the system performs end-to-end image synthesis based on the constructed fused expression control vector through the Multimodal Expressive Generation Architecture (MEGA) module. The generation system integrates a semantic guidance mechanism, an image consistency maintainer, and a step-by-step renderer to ensure that video frames maintain a high degree of visual consistency with the current speech expression. To meet real-time interaction requirements, the system introduces a Low-Overhead Rendering Scheduler (LORS) module, constructing a Modality Shift-Aware Activation (MSA-A) strategy. This dynamically monitors the degree of inter-frame changes in the control vector and determines whether to trigger frame-level rendering based on the expression mutation threshold. This strategy supports mechanisms such as prioritizing the synthesis of key expressions and automatically delaying the output of redundant frames, achieving an optimized balance between maintaining expression peaks and dynamically saving resources. The final generated video frame sequence features voice-driven, emotion-modulated, posture-synchronized, and natural facial expressions, and can be applied to digital human generation, virtual interaction, intelligent broadcasting, and metaverse scenarios.

[0077] More specifically, step S3 focuses on efficiently transforming the expression control vector into a visualized frame-level portrait image. Through mechanisms such as rhythm synchronization, image enhancement, and frame rate scheduling, it ensures an optimal balance between the generated content's naturalness of expression, structural continuity, and real-time performance. This constructs an integrated generation and control architecture oriented towards driving decoding and output optimization, specifically including the following steps: S31: A modal mapping and image-driven signal generation mechanism is proposed. To achieve image-level synthesis control of the three-modal output of a virtual human, this invention designs a modal mapping and image-driven signal generation mechanism as the core interface between the multimodal fusion system (ECES) and the image renderer. This module decodes the modal control vector output by the behavior consistency driving module into a frame-level image control tensor and introduces structural attention enhancement and temporal buffering mechanisms to ensure that image generation has sufficient response accuracy and temporal continuity. The system input consists of three behavioral components after decoding the fused control vector: lip movement parameter vector, facial expression vector, and upper body posture vector. To ensure that these high-semantic-level control signals can accurately guide image generation, the system introduces three modality-specific mapping sub-networks: (1) Lip2UV: Maps the lip movement parameters to the lip UV offset field, which is used to control the mesh deformation of the mouth region; (2) Face2AU: Maps the facial expression vectors to AU action unit activation maps to drive the facial expression generation module; (3) Pose2HM: Converts the pose vector into a key point heatmap to represent the movement trajectory of the head and shoulders.

[0078] The above process is uniformly modeled as shown in formula (13):

[0079] in For modal mapping networks, This represents a cross-modal splicing operation, which ultimately generates a control tensor. It possesses spatial feature structures with multimodal behavior. To further improve the structural consistency and emotional expression stability of the generated face region, this module introduces a structural attention refining (SAR) mechanism in the expression path. SAR utilizes the static portrait features output by the image encoding module, combined with the current frame expression vector, to generate a structure-weighted map through an additive attention function, which is used to refine the expression response of key facial regions, as shown in Equation (14):

[0080] in This is a learnable facial expression response mapping matrix. This mechanism can effectively alleviate phenomena such as facial muscle faking and blurred movement boundaries, and improve the textural coherence and dynamic naturalness of overall expression generation.

[0081] S32: Image Rendering Structure and Enhancement Mechanism. To achieve high-quality generation of portrait images driven by multimodal expression, this invention designs a fusion-based image rendering structure. This structure combines a conditional control generation framework with an expression-aware enhancement module to establish an image mapping process from control vectors to visual output. This module receives the frame-level control tensor output by the modality mapping network, constructs a conditional generation path, and introduces an expression-adaptive enhancement mechanism at the detail layer to improve the stability and realism of the output frame in dimensions such as facial regions, boundary dynamics, and pose switching.

[0082] The overall system architecture adopts a two-stage generator architecture, consisting of a coarse structure estimator (CSE) and a fine-grained refinement decoder (FRD).

[0083] (1) Coarse-level configuration predictor The CSE module consists of a multi-scale convolutional network that utilizes structure-aware attention to perform preliminary salient modeling of regions such as the lips, eyebrows, and eyelids. During the upsampling stage, the feature maps are combined with cross-modal feature residual maps to achieve an initial mapping from driving signals to visual structures. The frame control tensor is input into the CSE module, which generates initial medium-resolution image frames based on an encoder-decoder network, serving as a spatial layout sketch of pose and expression.

[0084] (2) Detail Enhancement Decoder The coarse image frame and the features of the original static reference image are fed into the FRD module for detailed reconstruction. The FRD employs a branched residual fusion structure, processing local features through expression, skin color, and head structure branches respectively, and utilizing a Style Reprojection Module (SRM) to regulate image style consistency. The final output image frame retains the static portrait identity features and is highly consistent with the control signals in terms of lip movement, facial expression, and pose. The SRM module introduces a style distribution projection matrix at the end of the decoding process to project and compress potential activation features from different modalities, reducing the interference of modal conflicts on style. To enhance the emotional expressiveness and fluency of the generation process, the system also designed an expression perception reconstruction loss function, which introduces expression semantic consistency supervision and structural adversarial loss on the basis of L1 reconstruction, as shown in formula (15):

[0085] in This represents the function for extracting facial expression action units; Cross-entropy; This represents the discriminator-guided adversarial loss. During the training phase, this loss guides the model to learn higher-fidelity facial micro-motion responses, ensuring consistent and stable visual output under different emotional expressions and complex action conditions.

[0086] S33: Design of Real-Time Output Control and Rhythm Scheduling Mechanism. After the multimodal driven image generation architecture completes the reconstruction of each modality's behavior and frame-level image rendering, in order to achieve low-latency output, time consistency, and user interaction adaptability of the system, this invention designs a real-time output control and rhythm scheduling mechanism. This mechanism is used to perform frame-level buffer management, modal consistency detection, and dynamic rhythm adjustment on the generated results, realizing the coordinated output of voice, facial expression, and action on the time axis, and ensuring the system's response performance and generation smoothness.

[0087] (1) Frame buffer scheduling module To support stable system operation under high-frequency generation conditions and avoid "frame bursts" and "output jitter" issues, the system introduces a multi-level frame buffer module, establishing three buffer sub-queues: motion buffer queue, expression buffer queue, and video output frame buffer queue; each queue maintains past... The frame modal output uses a time sliding window mechanism for frame interpolation and resampling of missing frames. When a frame of a certain modal data is lost or arrives late, the system will use the behavioral residual of the previous frame to dynamically fill in the missing frame as shown in formula (16):

[0088] in Feature tensors representing posture or facial expression modalities This represents the time smoothing factor (usually set to 0.3~0.5). This frame interpolation strategy can effectively smooth modal fluctuations and prevent sudden changes in video output.

[0089] (2) Speech-modal rhythm synchronization mechanism To ensure the synchronization of speech content, lip movements, and facial expressions, this invention proposes a rhythm-driven detection mechanism. This mechanism uses audio prosodic features as the driving rhythm benchmark, constructs a beat timeline, and performs aligned resampling of the modal control signal. 1. Extract energy peaks, pause intervals, and beat cycles from speech signals; 2. Output time of control tensor for each frame based on the time axis Perform constraint matching; 3. If the current modal response time difference If this occurs, the modal delay compensation module will be triggered.

[0090] The rhythm alignment function is shown in formula (17):

[0091] in, ;when Exceeding the semantic rhythm error threshold At the same time, the system dynamically adjusts the timing of modal outputs according to the rhythm fitting function, so that the timing of lip movements, actions and syllable semantic nodes remains consistent.

[0092] (3) Output frame rate adaptive control To ensure consistent image quality and semantics, and to adapt to different hardware platforms and bandwidth conditions, this system introduces an output frame rate self-scheduling module. The module adjusts the rendering frame rate using a strategy, and the specific output frame rate scheduling function is shown in equation (18).

[0093]

[0094] in Indicates the current output frame rate; , These represent the minimum output frame rate and the high output frame rate, respectively. Indicates the current frame generation delay (the total time elapsed from receiving the output image); This indicates the maximum acceptable generation delay threshold; This indicates the current occupancy rate of the output buffer queue; This indicates the output frame rate at the previous moment.

[0095] Example 2 This embodiment describes a real-time multimodal virtual human generation system based on an expression control embedding space. This system is used to execute the real-time multimodal virtual human generation method based on an expression control embedding space described in Embodiment 1 above. Specifically, the system includes a speech analysis module, an emotion recognition and control module, an image encoder, an expression control embedding space, a multimodal generation architecture module, a low-latency rendering scheduling module, a modality mapping and image-driven signal generation mechanism module, an image rendering structure and enhancement mechanism module, and a real-time output control and rhythm scheduling mechanism module. The speech analysis module performs multi-scale spectral analysis and rhythm deconstruction on the input speech signal, extracting low-level acoustic features such as Mel spectrum and fundamental frequency, while enhancing the semantic representation of language; the speech analysis module provides speech feature input for the subsequent emotion recognition and control module.

[0096] The emotion recognition and control module identifies emotional information in speech and generates adjustable emotion embedding vectors through channel attention mechanisms and language context alignment strategies. These emotion vectors can be expressed across languages ​​and have personalized adjustment capabilities. The emotion recognition and control module receives speech features from the speech analysis module and outputs emotion embedding vectors for subsequent multimodal fusion.

[0097] The image encoder processes the static reference image, extracts identity-preserving and shape information, and generates texture embeddings for the static image. This provides visual features for subsequent multimodal fusion control, ensuring that the virtual human's identity and appearance remain consistent.

[0098] The expression control embedding space serves as a unified representation and control domain for all modal features. It is responsible for mapping features from different modalities (such as lip movements, upper body posture, and emotion embedding vectors) to a shared coordinate system and performing alignment and dynamic adjustment between modalities. All input modal features (speech, emotion, and image) ultimately enter this space for fusion and control, providing a unified driving signal for the generation system.

[0099] The multimodal generation architecture module utilizes fused representational control vectors for end-to-end image synthesis, ensuring that video frames are visually highly consistent with modalities such as speech and facial expressions. The multimodal generation architecture module receives control vectors generated from the representational control embedding space and is responsible for the final generation of video frames.

[0100] To enable real-time interaction, the low-latency rendering scheduling module dynamically monitors inter-frame changes in the control vector and determines whether to trigger frame-level rendering based on an expression mutation threshold, thereby achieving efficient and low-latency image generation. The low-latency rendering scheduling module works closely with the multimodal generation architecture module to ensure that the generated video frames respond promptly to user needs.

[0101] The modality mapping and image-driven signal generation mechanism module decodes the modality control vector output by the behavior consistency driving module into frame-level image control signals and generates control tensors for use in image generation. It provides the necessary control signals for image generation, ensuring that each modality (such as lip movements, facial expressions, and poses) is accurately mapped to the generated image.

[0102] In the rendering stage, the image rendering structure and enhancement mechanism module, combined with a conditional control generation framework and an expression-aware enhancement module, improves the detail and style consistency of the image through a two-level generator architecture. Its processing utilizes the control signals output by the modality mapping network to ultimately generate an image that meets the control requirements.

[0103] The real-time output control and rhythm scheduling mechanism module ensures time consistency and efficient real-time output during image generation, optimizing the generation effect through rhythm synchronization and frame buffer scheduling. It works closely with all modules to ensure that the final output video frames have efficient and stable real-time performance.

[0104] In a practical process, the procedure is as follows: Input end: Voice input: Voice features are extracted by the voice analysis module and input into the emotion recognition module.

[0105] Image input: Static image features are extracted by an image encoder and provided to the expression control system.

[0106] Intermediate process: Expression control embedding space: Features obtained from speech analysis, emotion control and image encoder are uniformly mapped to generate control vectors to drive the generation module.

[0107] Generation process: Multimodal generation architecture: Transforms expressive control vectors into visual image outputs, handling the generation of actions including lip movements, facial expressions, and poses.

[0108] Rendering and Output: Image rendering and enhancement mechanism: responsible for ensuring the consistency of image details and style during the generation stage.

[0109] Real-time output control and rhythm scheduling mechanism: ensures stable output frame rate and adjusts output time.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-modal virtual human real-time generation method based on expression control embedding space, characterized in that: The method comprises: S1, taking voice modalities and visual modalities as inputs, and constructing multi-modal expression features comprising voice semantic features, emotional state features and static appearance features; S2, aligning the voice semantic features, emotional state features and static appearance features based on a multi-modal dynamic alignment mechanism of structure field embedding; S3, establishing an expression control embedding space based on a modal dominant fusion mechanism, dynamically adjusting the dominant priority of voice, emotion and posture in the generation driving process according to the current context state, modal confidence and historical expression state, and establishing an expression dominance control factor vector; S4, converting the fusion expression vector into an executable behavior instruction sequence with coordinated linkage capability through behavior hierarchical decoding, time domain flow smoothing and cross-modal collaborative correction, and finally driving the three modalities of lip movement, expression and posture to generate naturally; S5, designing a modal mapping and image driving signal generation mechanism, an image rendering structure and enhancement mechanism, and a real-time output control and rhythm scheduling mechanism, and performing end-to-end image synthesis based on the constructed fusion expression control vector.

2. The method of claim 1, wherein the method is based on expression control embedded space. In step S1, the process of establishing voice semantic features is as follows: the original voice is preprocessed by a voice analysis module, a short-time stable spectrum structure is constructed by using an overlapping frame algorithm with a Hann window, low-level acoustic features including a mel spectrum, a fundamental frequency and voice energy are extracted, and a time sequence feature matrix is formed wherein a voice transcriber is introduced to map the audio into a text representation, and a BERT is used for context embedding extraction to obtain a word granularity semantic vector sequence and then a bidirectional gate recurrent unit is used to complete time synchronization and context joint coding to obtain a unified semantic driving vector​ wherein, is a gating matrix for controlling the flow strength of each dimension of fused features; is a learnable gating parameter in training; are acoustic and semantic projection matrices, respectively, represents the final fused speech features; In step S1, the process of establishing the emotional state feature is: using a continuous emotion estimation mechanism based on latent variable modeling, a vector representation is constructed on the emotional dimensions of arousal, valence and dominance ; voice acoustic features Input three-branch convolutional channels, each channel extracts emotional response signals respectively for different time scales , after fusion by the context-aware attention module, the output of each branch forms an expression vector ; then a learnable channel weight combination is used to generate a control vector: wherein denotes the channel attention weight, dynamically adjusted by the context feature and the generation state; for maintaining numerical stability of the emotion fusion vector; Then inter-channel residual coupling terms are introduced to model the non-linear interaction between different emotional time scales: wherein is the coupling strength coefficient between each pair of channels, which is learned adaptively by the model during training; denotes element-wise multiplication, which is used to extract the co-activation dimension; denotes the channel coupling term, which is used to enhance the dynamic adjustment ability of the final output; The final control vector is updated as: wherein is a residual adjustment factor for controlling the degree of influence of the coupling term on the final emotion control signal; In step S1, the process of constructing static appearance features is: the input image is first normalized and then sent to the visual feature extraction module to generate an identity semantic representation , and a topology structure is constructed based on the face key point map, and a structure vector is extracted through a graph convolution network ; a structure attention weight matrix is introduced to strengthen the adjustment of important area features, and finally, the system generates a static image embedding representation through a unified fusion expression mapping function; the process is represented as: wherein, denotes a learnable transformation matrix, denotes a structural attention matrix, dynamically adjusting the respective regions; denotes a multi-layer nonlinear activation mapping, is a bias; is the final image static embedding vector.

3. The method of claim 1, wherein the method is based on expression control embedded space. In step S2, the multi-modal dynamic alignment mechanism based on structure field embedding combines knowledge graph structure representation and time sequence decomposition strategy to construct the alignment relationship between modalities in the expression control embedding space, and the process is as follows: First, a modal structure set is constructed, corresponding to voice, action and visual modalities respectively; the dynamic semantic flow of each modality is defined as a time sequence embedding: wherein, , respectively represent the speech modality, the action modality and the visual modality; Second, based on the large-scale semantic mapping library to build a unified modal atlas , node set representing multimodal units, edge set representing semantic or synergistic relationship; the original modal time sequence characteristics are decomposed into coarse-grained and fine-grained paths through a double-layer time sequence alignment mechanism: wherein, and denote shallow and deep multimodal alignment, denote modal segment-level semantic embeddings, constructed based on a graph mapping function denote the dimension table representation of the frame; denote cross-modal attention attention matrices; is a regularization factor for balancing semantic synchronization and cross-attention consistency; denote the Frobenius norm, quantifying the attention matrix difference; is an L2 norm measuring the projection error of the modal pair to the semantic field.​​ 4. The method of claim 1, wherein the method is characterized in that: In step S3, the construction process of the expression dominance control factor vector is as follows: The combined priority control modeling is performed on the aligned modal embedding vectors to construct a three-channel attention modulation path comprising a context guide channel, a confidence evaluation channel and a personalized adjustment channel, wherein, The context guide channel integrates the fusion vector of the previous time step and the current semantic state to construct the context semantic reference for modality dominant adjustment, wherein the attention mechanism and the gating strategy are introduced to make the fusion process have time continuity and semantic coherence, and when the expression trend changes, the modality dominant weight is adjusted to realize dynamic response control; The confidence evaluation channel performs multi-dimensional evaluation on the semantic stability, structural integrity and time sequence consistency of the current modality input, generates a confidence score as a fusion weight reference, automatically detects noise, missing frames or modality quality abnormalities, and reduces the dominance of the modality in fusion; The personalized adjustment channel adjusts the fusion mode according to the user-set expression style label or external adjustment factor to finally construct the expression dominance control factor vector: wherein denotes the current modality confidence vector; denotes the modality adjusted projection matrix; denotes the stitching operation; ensuring weight normalization and competition mechanism; Then, the modal embedding inputs are weighted and fused, and a cross-modal residual modulation term is introduced to capture the potential behavior coupling relationship between multi-modalities.

5. The method of claim 1, wherein: In step S4, the process of behavior hierarchical decoding, time domain flow smoothing and cross-modal collaborative correction is as follows: First, the unified driving vector analysis and modality decomposition are performed: the fusion expression control vector is input into the behavior driving decoder to be decomposed into a set of modal components: wherein are decoded lip motion parameter sequences, expression motion unit sequences and pose joint motion sequences, respectively, is a frame number, and three learnable transformation matrices and a bias are used to model the driving bias of each modality; Then the temporal consistency optimization and dynamic constraints are established: a dynamic optimization strategy based on differential consistency constraints is introduced to regularize the difference in behavior change rate between cross-modal, so that the output frame sequence maintains continuity and consistency in the dimensions of emotional intensity, expression tension and action amplitude: wherein represents the change magnitude of the modal output at time step , the constraint term ensures that the behavior pace of the three modalities remains coupled with the dynamic trend; Finally, the behavior conflict correction and style continuity maintenance mechanism is designed, and the behavior suppression gate under style guidance is introduced to guide the obtained style intention vector and to perform amplitude clipping and rhythm retiming on the abnormal actions that deviate from the style trajectory in the modal decoding output, so as to ensure that the expression maintains semantic style consistency in the visual perception level; The correction process is defined as: wherein is denoted as a style-gating vector, controlling whether each modality channel accepts suppression or relaxation, is denoted as a time-domain sliding average function.

6. The method of claim 1, wherein the method is based on a multi-modal virtual human real-time generation method of expression control embedded space. In step S5, the modal mapping and image driving signal generation mechanism is: The modal control vector output by the behavior consistency driving module is decoded into a frame-level image control tensor, and a structure attention enhancement and temporal cache mechanism is introduced to ensure that the image generation has sufficient response accuracy and temporal continuity; it decodes the three behavior components after fusing the control vector: the lip movement parameter vector, the expression vector and the upper body posture vector; three modal-specific mapping sub-networks are introduced to guide image generation, including Lip2UV, Face2AU and Pose2HM, wherein Lip2UV maps the lip movement parameters to the lip UV offset field to control the grid deformation of the mouth shape area; Face2AU maps the expression vector to the AU action unit activation map to drive the expression generation module; and Pose2HM converts the posture vector into a key point heat map to represent the head and shoulder motion trajectory; The above process is modeled as: wherein is a modal mapping network, denotes a cross-modal stitching operation, finally generating a control tensor with a spatial feature structure of multi-modal behavior; The structure perception attention mechanism is introduced in the expression path, which uses the static portrait features output by the image encoding module to jointly generate a structure weighted graph with the current frame expression vector through an additive attention function, which is used to refine the expression response of the key facial area: wherein, is a learnable expression response mapping matrix.

7. The method of claim 1, wherein: In step S5, the image rendering structure and enhancement mechanism adopts a two-level generator architecture, including a coarse-level contour predictor and a detail enhancement decoder, wherein The coarse-level contour predictor is composed of a multi-scale convolutional network, which uses structure-preserving attention to preliminarily model the lip, eyebrow and eyelid regions, wherein the feature map combines the cross-modal feature residual graph in the upsampling stage to realize the preliminary mapping of the driving signal to the visual structure, and the frame control tensor is input into the coarse-level contour predictor to generate an initial medium-resolution image frame based on the encoding-decoding network, which is used as a spatial layout sketch of the posture and expression; The detail enhancement decoder takes the coarse image frame and the original static reference image features as input, and uses a branch residual fusion structure to process local features through the expression branch, skin color branch and head structure branch, respectively, and uses a style re-projection module to regulate the image style consistency; finally, the output image frame retains the static portrait identity features and is consistent with the control signal in terms of lip movement, expression and posture performance; the style re-projection module introduces a style distribution projection matrix at the end of the decoding to project and compress the latent activation features from different modalities; The expression perception reconstruction loss function is designed, which introduces expression semantic consistency supervision and structure adversarial loss on the basis of L1 reconstruction: wherein represents an expression motion unit extraction function; is the cross-entropy; represents a discriminator-guided adversarial loss.

8. The method of claim 1, wherein the method is based on a multi-modal virtual human real-time generation method of expression control embedded space. In step S5, the output control and rhythm scheduling mechanism designs a frame buffer scheduling module, a voice-modal rhythm synchronization mechanism, and an output frame rate adaptive control mechanism, wherein, The frame buffer scheduling module introduces multi-level frame buffers, including an action buffer queue, an expression buffer queue and a video output frame buffer queue, each of which maintains past Frame modal output, through a time sliding window mechanism, frame interpolation and frame missing resampling operation is carried out, when a certain modal data is lost frame or delayed, the last frame behavior residual is used for dynamic frame filling, and the frame filling formula is: wherein, a feature tensor representing a pose or expression modality, denotes a temporal smoothing factor; The voice-modal rhythm synchronization mechanism takes audio prosody features as the driving rhythm reference, constructs a beat time axis, and aligns and resamples the modal control signal: energy peaks, pause intervals, and beat periods are extracted from the voice signal; the output time of each frame control tensor is aligned based on the time axis Constrained matching is performed; if the current modal response time difference is greater than the threshold, modal delay compensation is triggered; the rhythm alignment function is represented as: wherein, when the semantic rhythm error threshold is exceeded the modal output timing is dynamically adjusted according to the rhythm fitting function, so that the time consistency between the lip movement, action and semantic nodes of syllables is maintained; The output frame rate adaptive control mechanism adjusts the rendering frame rate according to the strategy, and the output frame rate scheduling function is: wherein, represents the current time output frame rate; , respectively represent the minimum output frame rate and the high output frame rate; represents the current frame generation delay; represents the maximum generation delay threshold value; represents the current output buffer queue occupancy rate; represents the output frame rate at the previous time.

9. A multi-modal virtual human real-time generation system based on expression control embedded space, used for executing the multi-modal virtual human real-time generation method based on expression control embedded space in any one of the preceding claims 1-8. The system comprises a voice analysis module, an emotion recognition and control module, an image encoder, an expression control embedding space, a multi-modal generation architecture module, a low-latency rendering scheduling module, a modal mapping and image driving signal generation mechanism module, an image rendering structure and enhancement mechanism module, and a real-time output control and rhythm scheduling mechanism module, wherein The voice analysis module performs multi-scale spectral analysis and rhythm deconstruction on the input voice signal, extracts low-level acoustic features including mel spectrum and fundamental frequency, and enhances language semantic representation to obtain voice features, and transmits them to the emotion recognition and control module; The emotion recognition and control module identifies the emotion information in the voice, and generates adjustable emotion embedding vectors through channel attention mechanism and language context alignment strategy; The image encoder processes the static reference image, extracts the identity preservation and topography information, generates the texture embedding of the static image, and extracts the static appearance features; The expression control embedding space is a unified representation and control domain of all modal features, which maps all different modal features to a shared coordinate system and aligns and dynamically adjusts between modalities; The multi-modal generation architecture module receives the control vectors generated by the expression control embedding space, and uses the fused expression control vectors for end-to-end image synthesis; The low-latency rendering scheduling module dynamically monitors the inter-frame changes of the control vectors, and determines whether to trigger frame-level rendering according to the expression mutation threshold; The modal mapping and image driving signal generation mechanism module decodes the modal control vectors output by the behavior consistency driving module into frame-level image control signals, and generates control tensors for image generation; The image rendering structure and enhancement mechanism module combines the conditional control generation framework and expression perception enhancement module in the rendering stage, and improves the detail and style consistency of the image through a two-level generator architecture; The real-time output control and rhythm scheduling mechanism module ensures the time consistency and efficient real-time output in the image generation process, and optimizes the generation effect through rhythm synchronization and frame buffer scheduling.

Images