A virtual image model construction method and system based on image cloning

Abstract

The application discloses a kind of virtual image model construction method and system based on image clone, it is related to computer graphics field, including: from real-time audio and video stream of real person, the multimodal data of fusion vision, voice and action are obtained;Respectively extract facial expression, voice and action emotional data;Adopt dynamic time warping algorithm to calculate the similarity between each modal emotional data, generate the modal correlation mapping matrix of quantitative modal synchronization relationship;Adopt machine learning model to construct emotional label to limb posture expression action mapping model;According to the deviation of expression and action, the feature is adjusted back;Finally, based on the feature after adjustment, mapping matrix and mapping model, generate the virtual image that expression, voice and action are accurately aligned on time axis.The application establishes the quantitative correlation and feedback adjustment mechanism between modal, significantly improves the real sense and coordination of virtual image when cloning real person in multidimensional emotional expression.

Description

Technical Field

[0001] This invention relates to the field of computer graphics, specifically to a method and system for constructing virtual character models based on image cloning. Background Technology

[0002] In emerging application areas such as live interactive streaming, virtual reality (VR), and the metaverse, high-fidelity virtual avatars play a central role. Among these, image cloning technology—which captures and analyzes the voice, appearance, and body language of real people in real time and accurately transfers their expressiveness to virtual characters—has become crucial for enhancing user immersion and the realism of interaction. The ultimate goal of this technology is to create a "digital twin" that can synchronously reflect all the subtle emotional expressions of a real person, especially when processing unstructured and constantly changing real-time video streams originating from live broadcasts, which places high demands on the technology.

[0003] However, achieving the ideal "digital twin" level cloning effect presents significant technical challenges when integrating multimodal data from real people, including visual, vocal, and motor data. The core difficulty lies in the highly personalized and complexly asynchronous interaction between facial expressions, vocal tone, and body language when real humans express emotions. For example, in a live stream, a host's impromptu smile might precede their actual laughter by a fraction of a second, while a relaxed posture might accompany the entire process. This cross-modal temporal difference stemming from real physiological responses leads to a mismatch between the virtual avatar's facial expressions and the emotions conveyed by its voice, or its body language failing to accurately and naturally reflect its emotional state. This inconsistency in multimodal performance not only severely impacts the realism of the "cloning" but also becomes a key bottleneck restricting the development of this technology in high-fidelity applications.

[0004] Therefore, the core technical problem currently facing the field is: how to design an effective cross-validation and fusion mechanism to achieve precise and dynamic coordination between facial expressions, voice emotions and body movements, and to use a reliable algorithm framework to detect and correct inconsistencies that may occur between modalities, so as to ensure that the final generated virtual image achieves a high degree of coordination and unity in multi-dimensional emotional expression.

[0005] Especially in specific application scenarios where the realism and subtlety of emotional interaction are extremely important, such as virtual psychological counseling, intelligent emotional companionship, or highly realistic customer service, existing technologies fall short. In these scenarios, virtual avatars not only need to convey information but also need to establish emotional connections and trust. A virtual counselor whose facial expressions and voices are disconnected, and whose actions contradict their emotional state, will provide "empathic" feedback that appears fake and mechanical. This not only fails to provide effective psychological guidance but may even negatively impact the user experience. Therefore, this inconsistency in multimodal expressiveness has become a key technological bottleneck restricting the development of virtual avatar technology towards more advanced and sensitive human-computer interaction fields. Summary of the Invention

[0006] To address the technical problems of inconsistent emotional expression and insufficient realism in virtual avatars due to insufficient multimodal data fusion in existing technologies, this invention provides a method and system for constructing virtual avatar models based on avatar cloning.

[0007] To address the technical problems mentioned in the background art, such as insufficient multimodal data fusion, lack of dynamic correlation mapping, and lack of inconsistency correction mechanism, this invention proposes the following: deep feature fusion using a Transformer-based attention mechanism; dynamic time warping (DTW) algorithm to quantify the synchronization relationship between modes and generate a modal correlation mapping matrix; and constructing a closed-loop feedback adjustment system based on deviation detection and linear interpolation.

[0008] One aspect of the present invention provides a method for constructing a virtual avatar model based on image cloning, the method comprising the following steps:

[0009] First, visual, audio, and motion data are acquired from real-time video streams featuring real individuals as multimodal data. A Transformer-based attention mechanism is then used to fuse this multimodal data. The Transformer model is a deep learning model based on self-attention, whose multi-head attention structure effectively captures long-distance dependencies between different data sequences. Feature concatenation is performed using this model to generate a first multimodal feature representation, which includes visual feature vectors, audio emotion feature vectors, and motion pattern feature vectors.

[0010] Subsequently, visual feature vectors are extracted from the first multimodal feature representation, and facial keypoint coordinate sequences and facial texture information are separated from them. A ResNet-50 convolutional neural network is used to process the facial texture information; ResNet-50 is a deep residual network suitable for image feature extraction tasks. This processing generates first facial expression emotion data, which includes emotion labels based on the FER-2013 standard (a facial expression recognition benchmark) and expression intensity values ​​output by the Softmax function (a function that transforms vectors into probability distributions).

[0011] Next, a speech emotion feature vector, including pitch frequency features, speech rate change rate, and volume intensity value, is obtained from the first multimodal feature representation. Combined with the emotion tags in the first facial expression emotion data obtained in the previous step, a statistical model is used to analyze the fundamental frequency (F0) range of the pitch frequency features, thereby generating first speech emotion data corresponding to facial expression emotions.

[0012] Simultaneously, action pattern feature vectors, including posture angle sequences and motion velocity vectors, are obtained from the first multimodal feature representation. Similarly, by combining the emotion tags from the first facial expression emotion data, a motion amplitude parameter is calculated. This parameter is normalized based on the range of joint angle changes, thereby generating the first action emotion data and aligning its timestamp with the timestamp of the first speech emotion data.

[0013] The contribution of this invention lies in transforming the abstract synchronization relationships between different modalities into computable and usable quantitative features. Specifically, the method obtains the respective emotion tags and timestamps from the previously generated first facial expression emotion data, first speech emotion data, and first action emotion data, and maps the emotion tags into numerical vectors. Then, the Dynamic Time Warping (DTW) algorithm is employed, which is used to measure the similarity between two time series, and is particularly suitable for sequences of different lengths or speeds. The Euclidean distance between the emotional numerical vectors of different modalities is calculated using the DTW algorithm, with the formula D = sqrt(sum((x_i - y_i)^2)), where x_i and y_i are the numerical vectors obtained after the emotion tags are transformed by one-hot encoding or word embedding techniques. A similarity matrix is ​​generated based on the calculated distance, and inconsistency intervals are marked according to a preset threshold, ultimately generating a first modal association mapping matrix containing time axis synchronization weights. This weight is obtained by normalizing the DTW path. It quantifies the synchronization relationship between modes into a weight matrix, providing a quantitative basis for subsequent feature mapping and data generation.

[0014] The contribution of this invention lies in its automatic learning and construction of a direct, quantified mapping relationship from high-level emotions to underlying physical parameters through a machine learning model. Specifically, the method obtains the time axis synchronization weights from the first modal association mapping matrix generated in the previous step and combines them with the posture angle sequence and motion velocity vector in the first multimodal feature representation. A linear regression model is used to fit these input data, thereby generating an expression and movement mapping model. This mapping model explicitly defines the correspondence between emotion labels and specific limb posture angles (such as the angle range of the shoulder and elbow), achieving data-driven, high-precision cross-modal mapping.

[0015] Furthermore, this invention establishes a feedback adjustment closed-loop system to ensure high coordination of multimodal data during the generation process. The method obtains the joint angle change rate from the facial expression mapping model and calculates the deviation between the two by combining the expression intensity value of the first facial expression emotion data. If the deviation exceeds a preset threshold, a linear interpolation method is used to smoothly adjust the motion amplitude parameter and rhythm parameter (i.e., the normalized motion speed) based on the timestamp. This continuous verification and calibration process effectively solves the problem of inconsistent performance that may occur in multimodal data during generation, thereby generating a second multimodal feature representation containing the adjusted features.

[0016] Subsequently, a comprehensive representation of visual feature vectors, speech emotion feature vectors, and action pattern feature vectors is obtained from the adjusted multimodal feature representation. A weighted summation method is used for fusion, where the weights are derived from the time-axis synchronization weights in the aforementioned first modality association mapping matrix, thereby generating unified first emotion sequence data, which includes emotion tags, intensity values, and timestamps.

[0017] Preferably, to make the emotional transitions of the virtual avatar more natural, this invention introduces a dynamic weight adjustment mechanism based on emotional inertia. When generating the first emotional sequence data using weighted summation, an emotional inertia model, such as a recurrent neural network (RNN), can be introduced. This model receives historical first emotional sequence data as input and outputs an emotional inertia coefficient. This emotional inertia coefficient serves as a dynamic adjustment factor, performing secondary modulation on the time axis synchronization weights from the first modality association mapping matrix. When a drastic change in emotional label is detected, this coefficient actively smooths the weight changes, resulting in smoother transitions in generated expressions, actions, and speech.

[0018] Finally, emotion tags, intensity values, and timestamps are extracted from the first emotion sequence data and combined with the facial expression-motion mapping model, the pitch frequency features in the first speech emotion data, and the posture angle sequence and motion amplitude parameters in the second multimodal feature representation. A keypoint-driven algorithm is used to generate facial expression animation by interpolating existing facial keypoint coordinate sequences; a fundamental frequency adjustment algorithm is used to generate speech intonation by smoothing the pitch frequency features; and joint angle interpolation is used to generate motion sequences. Ultimately, time-aligned virtual avatar clone data containing facial expression animation, speech intonation, and motion sequences is generated.

[0019] Another aspect of the present invention provides a virtual avatar model construction system based on image cloning, the system being configured to perform the above-described method, comprising:

[0020] The multimodal feature generation module is used to acquire and fuse visual, speech, and action data to generate the first multimodal feature representation;

[0021] The facial expression emotion recognition module is used to generate first facial expression emotion data from visual features;

[0022] The speech sentiment analysis module is used to generate first speech sentiment data from speech features;

[0023] The action emotion generation module is used to generate first action emotion data from action features;

[0024] The modal association mapping module is used to generate a first modal association mapping matrix containing time axis synchronization weights using a dynamic time warping algorithm;

[0025] The facial expression and motion mapping generation module is used to generate a facial expression and motion mapping model that defines the correspondence between emotions and body postures using a linear regression model;

[0026] The multimodal feature adjustment module is used to adjust feature parameters based on the deviation between facial expressions and actions, and generate a second multimodal feature representation;

[0027] The sentiment sequence generation module is used to generate a unified first sentiment sequence data through weighted summation.

[0028] And a virtual avatar clone generation module, which is used to generate virtual avatar clone data containing facial animation, voice tone and action sequence based on emotional sequence data and mapping model. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating a method for constructing a virtual image model based on image cloning, provided in an embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram illustrating the principle of the modal association mapping process in an embodiment of the present invention.

[0032] Figure 3 This is a schematic diagram illustrating the principle of the multimodal feature feedback adjustment process in an embodiment of the present invention.

[0033] Figure 4 This is a schematic diagram of modal emotion sequence alignment based on the Dynamic Time Warping (DTW) algorithm in an embodiment of the present invention, demonstrating how facial expressions and speech emotion sequences with temporal differences can find the optimal alignment path through the DTW algorithm.

[0034] Figure 5 This is a schematic diagram of the multimodal feature feedback adjustment effect in an embodiment of the present invention, showing how the system smoothly corrects the action parameters through a feedback adjustment mechanism to synchronize them with the expression intensity when the action amplitude and expression intensity are detected to be inconsistent.

[0035] Figure 6 This is a schematic diagram comparing the transition effects of emotional sequences with and without an emotional inertia model in this embodiment of the invention, showing the difference in the smoothness of abrupt changes in emotional state when there is emotional inertia.

[0036] Figure 7 This is a schematic diagram of multimodal input data in an embodiment of the present invention. The facial key points in the diagram only schematically show some of the backbone nodes. Those skilled in the art should understand that in the specific implementation, a standard facial coordinate sequence containing 68 key points is used for refined expression analysis.

[0037] Figure 8 This is a schematic diagram comparing the virtual image generation effect of the embodiments of the present invention with that of the prior art. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0039] Example 1

[0040] This embodiment provides a method for constructing a virtual avatar model based on image cloning. (Refer to...) Figure 1 This method is a method executed in a computer, and may specifically include the following steps:

[0041] Step S1: Taking the real-time image cloning of a streamer in a live broadcast scene as an example, obtain the streamer's visual data, voice data, and motion data, and use a Transformer-based attention mechanism to fuse the multimodal data to generate a first multimodal feature representation.

[0042] In this embodiment, the acquisition of multimodal data can rely on standard live streaming capture equipment. Specifically, visual data can be captured by one or more cameras facing the broadcaster, with a resolution of at least 1920x1080 pixels and a frame rate of at least 30fps; audio data can be captured by a high-fidelity microphone worn by the broadcaster at a sampling rate of 48kHz and quantization precision of 16 bits; motion data can be acquired by a full-body motion capture system based on an inertial measurement unit (IMU) or by computer vision-based pose estimation techniques (such as OpenPose), recording the position and pose of the broadcaster's main joints in the form of three-dimensional coordinates or joint rotation quaternions. After acquiring the raw data, preprocessing is required, such as face detection and alignment of image data, extraction of facial coordinate sequences containing 68 key points, noise reduction and frame segmentation of audio data, and noise filtering and coordinate system normalization of motion data.

[0043] After preprocessing, various data types are converted into feature vectors. For example, visual data is encoded as a 2048-dimensional feature vector, speech data is converted into a 128-dimensional MFCC (Mel-frequency cepstral coefficient) feature vector, and motion data is converted into a 100-dimensional feature vector containing rotational information of 17 major upper body joints, including the head, shoulders, elbows, and wrists. These feature vectors from different modalities are then aligned and concatenated along the temporal dimension to form a combined input sequence. To enable the model to distinguish the origins of different modalities, a learnable modality embedding can be added to the feature vectors of each modality. The concatenated sequence is then input into a Transformer-based encoder model. This model preferably contains six encoder layers, each containing a multi-head self-attention module preferably configured with eight attention heads and a feedforward neural network. The multi-head self-attention mechanism can compute the association weights between features from different modalities and between different time steps within the same modality in parallel, thereby capturing deep semantic dependencies across modalities. For example, when a user utters a word expressing surprise, the strong correlation between the visual feature of raised eyebrows and the vocal feature of a sudden increase in pitch is captured by the model. After processing by the Transformer encoder, the output sequence is the first multimodal feature representation, which is a high-dimensional vector sequence resulting from the deep fusion of information from the three modalities in a shared semantic space. This step leverages the sequence relationship modeling capability of the Transformer model to solve the problem of insufficient intermodal information correlation caused by simple feature concatenation in traditional methods, providing a high-quality feature foundation for subsequent collaborative analysis. As an alternative, a Gated Recurrent Unit (GRU) network with an attention mechanism can also be used to fuse multimodal sequences. Although its ability to capture long-distance dependencies may be weaker than that of the Transformer, it is still an effective implementation method in scenarios with limited computational resources.

[0044] Step S2: Extract visual features from the first multimodal feature representation and generate first facial expression emotion data based on the visual features.

[0045] Specifically, feature vectors corresponding to the visual data are extracted from the time series of the first multimodal feature representation. These visual feature vectors already contain the facial keypoint coordinate sequence and facial texture information. To generate the first facial expression emotion data, the facial texture information is input into a pre-trained convolutional neural network. In this embodiment, the ResNet-50 network model is preferably used. This model is first pre-trained on a large general-purpose image dataset (such as ImageNet), and then fine-tuned on specialized facial expression datasets (such as FER-2013 and AffectNet), where the facial images input to the model are normalized to a resolution of 256x256 pixels. The FER-2013 dataset contains seven basic emotion labels: happy, sad, angry, surprised, fearful, disgusted, and neutral. After inputting the facial texture information, the ResNet-50 network extracts features from low-level edges to high-level semantics layer by layer through its deep residual structure, and finally outputs a 7-dimensional vector through a global average pooling layer and a fully connected layer. The vector is processed by the Softmax activation function, transforming it into a probability distribution where each dimension corresponds to the confidence level of an emotion label. For example, the output vector [0.1, 0.8, 0.05, 0.02, 0.01, 0.01, 0.01] indicates that the model's confidence level in classifying the current expression as 'sad' is 0.8. Therefore, the generated first facial expression emotion data consists of two parts: the emotion label with the highest probability (e.g., "sad") and the corresponding probability value (e.g., 0.8), the latter being defined as the expression intensity value. This step leverages the image representation capabilities of deep learning models to achieve accurate quantitative recognition of subtle facial expressions, providing a reliable benchmark for multimodal emotion alignment.

[0046] Step S3: Separate speech features from the first multimodal feature representation and combine them with the first facial expression emotion data to generate the first speech emotion data.

[0047] In this step, a feature vector corresponding to the speech data is extracted from the first multimodal feature representation. This vector includes acoustic features such as pitch frequency (fundamental frequency F0), speech rate change rate, and volume intensity (energy). To ensure consistency between speech emotion and facial expression emotion, this method uses the emotion label in the first facial expression emotion data obtained in step S2 as strong prior knowledge. Specifically, to ensure the uniqueness and accuracy of speech emotion judgment, this method uses the emotion label in the first facial expression emotion data obtained in step S2 as strong prior knowledge. For each emotion label, a multidimensional acoustic feature statistical model based on a Gaussian mixture model (GMM) is pre-established. This model not only considers the typical distribution range of the fundamental frequency (F0) (for example, the fundamental frequency probability distribution of a happy state is more concentrated in the range of 150Hz to 350Hz), but also comprehensively analyzes multidimensional acoustic feature vectors including energy, speech rate, and Mel-frequency cepstral coefficients (MFCC). When the facial expression identified in step S2 is 'happy', this step calculates the posterior probability of the multidimensional acoustic feature vector of the current speech frame under the GMM model corresponding to the 'happy' emotion. If this probability value is the highest, the speech emotion is considered to be consistent with the facial expression emotion. Therefore, even if an F0 value (e.g., 280Hz) falls within the F0 value range of both 'happy' and 'angry', this invention can make an accurate and unique judgment by comprehensively probabilistically modeling the multidimensional features, solving the ambiguity problem of single feature judgment, and marking the speech segment as 'happy' to generate the first speech emotion data. This data includes emotion labels consistent with facial expressions, as well as real-time pitch frequency features, speech rate change rate, and volume intensity values. This analysis method, which combines prior knowledge, avoids misjudgments that may be caused by relying solely on acoustic features (e.g., misjudging the rising intonation of a question as 'surprise'), significantly improving the accuracy and robustness of speech emotion analysis. As an alternative, a separate speech emotion recognition model (such as a CNN-LSTM hybrid model) can be trained to output a speech emotion probability distribution. This distribution can then be weighted and fused with the probability distribution of facial expressions or subjected to a consistency test to determine the final speech emotion.

[0048] Step S4: Separate the action features from the first multimodal feature representation and combine them with the first facial expression emotion data to generate the first action emotion data.

[0049] Similar to step S3, this step first extracts the feature vector portion corresponding to the action data from the first multimodal feature representation. This vector contains the posture angle sequence and motion velocity vector of key joints throughout the body. Similarly, to ensure the emotion expressed by the limb movements is coordinated with facial expressions, the emotion label in the first facial expression emotion data obtained in step S2 is used as a guide. Specifically, a quantified motion amplitude parameter is calculated. This parameter can be defined based on the maximum angle change range of a set of key joints (e.g., shoulder, elbow, wrist) within a certain time window (e.g., 1 second). To eliminate individual differences, this angle change range needs to be normalized, for example, by dividing it by the maximum range of motion of the joint at its physiological limit. For example, for the emotion label "anger," the system maps this state to a larger amplitude and faster speed movement; while for "sadness," it maps it to a smaller amplitude and slower speed movement. The first action emotion data is generated by comparing the calculated normalized motion amplitude parameter with a preset expected range under the emotion label. This data contains an emotion label consistent with the facial expression emotion, as well as the quantified motion amplitude parameter. Simultaneously, for subsequent synchronous processing, the timestamp of the first action emotion data is aligned with the timestamp of the first speech emotion data generated in step S3. This step associates abstract body language with explicit emotion labels, laying the foundation for subsequent cross-modal consistency quantification analysis.

[0050] Step S5: Refer to Figure 2 and Figure 4 Consistency analysis is performed on the first facial expression emotion data, the first voice emotion data, and the first action emotion data to generate a first modality association mapping matrix.

[0051] After obtaining the sentiment data for each individual modality, this step, which addresses the core issue of missing intermodal correlations mentioned in the background art, is one of the core contributions of this invention. It aims to transform the synchronization relationships between modalities into computable quantitative features. First, sentiment tags (such as "happy" and "sad") in each modality's data are mapped to numerical vectors using word embedding techniques (such as Word2Vec) or simple one-hot encoding. This results in three parallel sentiment time series: facial expression sentiment series, speech sentiment series, and action sentiment series. Then, Dynamic Time Warping (DTW) is used to measure the similarity between these time series. Taking the alignment of facial expression and speech sentiment as an example, the DTW algorithm constructs a cost matrix where each element (i, j) represents the Euclidean distance between the facial expression sequence at time i and the speech sentiment sequence at time j, calculated as D = sqrt(sum((x_i - y_i)^2)), where x_i and y_i are two sentiment vectors. The goal of the DTW algorithm is to find a path from the bottom left to the top right corner of this cost matrix that minimizes the total cost along the path. This optimal path represents the best alignment between two time series and can effectively handle the small time differences in emotional expression, such as a smile preceding laughter.

[0052] A similarity matrix is ​​generated. The lower the cost at each point on the path, the higher the consistency between the two modalities at that time point. Normalizing and inverting these costs yields a similarity score between 0 and 1. Further, a preset threshold (e.g., a distance greater than 0.2) can be set based on the calculated Euclidean distance to mark time intervals of modal inconsistency. Finally, the similarity scores and time alignment information of all modal pairs (face-speech, face-action, speech-action) are integrated to form a first modal association mapping matrix containing temporal synchronization weights. This matrix quantifies the degree of synchronization between different modalities at each time point. For example, at a certain moment, the synchronization weight between face and speech is 0.9, while the synchronization weight between face and action is only 0.4, indicating that the character's facial expressions and voice are highly coordinated, but the correlation between body movements is weak. Thus, through the generation of the first modal association mapping matrix, this invention successfully transforms the abstract synchronization relationship between different modalities into concrete quantified weights that can be used in subsequent steps, providing an important basis for subsequent feature fusion and model generation. As an alternative, mutual information can be used to measure the nonlinear correlation between two time series, but DTW is more advantageous in handling time axis stretching and compression.

[0053] Step S6: Generate an expression and action mapping model based on the first modal association mapping matrix and the first multimodal feature representation.

[0054] The purpose of this step is to establish a direct, quantitative mapping relationship from high-level emotions to low-level physical parameters. Specifically, a machine learning model is used to learn the correspondence between emotions and body movements. The model's input includes: a sequence of posture angles and a motion velocity vector from the first multimodal feature representation, and time axis synchronization weights obtained from the first modal association mapping matrix. To reduce the dimensionality of the motion velocity vector and extract key dynamic features, principal component analysis (PCA) is preferably used to preprocess the vector. The model's output is specific body posture parameters, such as the target angle range for the shoulder and elbow. In this embodiment, a linear regression model is preferably used. This model learns a mapping function by fitting a large amount of labeled data, for example: target shoulder angle = c1 * emotion label vector + c2 * expression intensity value + c3 * synchronization weight + intercept. Where c1, c2, c3 and the intercept are coefficients learned by the model, and the "synchronization weight" is extracted from the first modal association mapping matrix for a specific mapping relationship. Specifically, in the facial expression-motion mapping model used to calculate the "shoulder target angle," the "synchronization weight" is the synchronization weight value of the "face-action" modality pair corresponding to the current time point. This weight quantifies the degree of coordination between facial expression and action, serving as the basis for adjusting the final posture output. By learning such a set of coefficients for each emotion and each key joint, an "facial expression-motion mapping model" is constructed. This model is not a static lookup table, but rather a set of linear regression models themselves and their learned weight parameters. It explicitly defines what kind of body posture the virtual avatar should present under different emotional states and synchronization conditions. For example, when the input is "happy," the intensity is 0.9, and the synchronization weight is 0.8, the mapping model will output a set of posture parameters representing slightly outstretched arms and a slightly forward-leaning body. This step achieves data-driven cross-modal mapping, making the generated actions more accurate and natural. As an alternative, more complex nonlinear models, such as gradient boosting decision trees or small neural networks, can be used to capture the more complex nonlinear relationship between emotion and action.

[0055] Step S7: Refer to Figure 3 and Figure 5 Feedback adjustments are made to the multimodal features to generate a second multimodal feature representation.

[0056] To address the shortcomings of existing technologies in dynamically generating virtual characters that cannot correct inconsistencies and ensure high consistency across all dimensions, this invention establishes a closed-loop feedback adjustment system. This system continuously verifies and calibrates the generation process. Specifically, from the facial expression mapping model generated in step S6, a desired joint angle change rate can be predicted based on the current emotional state. Simultaneously, the first facial expression emotion data obtained in step S2 contains real-time expression intensity values. The system calculates the deviation between these two quantitative indicators, for example, by calculating the absolute value of the difference after normalization. If a character's facial expression intensity is very high (e.g., "anger" intensity is 0.95), but the amplitude or speed of their body movements (reflected by the joint angle change rate) is small, this constitutes an inconsistency in performance. When the calculated deviation exceeds a preset threshold set based on empirical data (e.g., greater than 0.3 after normalization), the system triggers an adjustment mechanism.

[0057] The adjustment mechanism employs linear interpolation to smoothly adjust motion-related parameters. Specifically, it adjusts motion amplitude and rhythm parameters (i.e., normalized motion speed) based on timestamps. For example, if insufficient motion amplitude is detected, the system sets a target amplitude that better matches the current facial expression intensity. Then, over the next N frames (e.g., 15 frames, or 0.5 seconds), it smoothly transitions the current motion amplitude to the target amplitude through linear interpolation. This verification and calibration process, by quantitatively comparing facial expression intensity and motion amplitude, achieves dynamic alignment of cross-modal performance intensity, effectively resolving potential dynamic inconsistencies during generation and ensuring the consistency of the final output. As an alternative, a proportional-integral-derivative (PID) controller can be designed to replace linear interpolation, enabling more precise and rapid dynamic adjustment of motion parameters.

[0058] Step S8: Integrate the adjusted features to generate unified first emotional sequence data, and finally generate virtual avatar clone data.

[0059] The emotional information from the three modalities is finally fused using a weighted summation method. The emotional intensity I(t) for each modality originates from the intensity value in the emotional data of each modality, while the weight w(t) for each modality originates from the corresponding time-axis synchronization weights in the first modality association mapping matrix generated in step S5. Specifically, the weight of each single modality is calculated from its pairwise synchronization weights with all other modalities. For example, at time point t, the face weight w_face(t) can be defined as the average of the face-voice synchronization weight and the face-motion synchronization weight at that time point; the voice weight w_voice(t) can be defined as the average of the face-voice synchronization weight and the voice-motion synchronization weight at that time point; and the motion weight w_motion(t) can be defined as the average of the face-motion synchronization weight and the voice-motion synchronization weight at that time point. In this way, the pairwise association strengths are transformed into a measure of the contribution of each modality to the current unified emotional expression. For example, at a certain time point t, the uniform emotion intensity can be calculated by the following formula: E_unified(t) = w_face(t) * I_face(t) + w_voice(t) * I_voice(t) + w_motion(t) * I_motion(t), where I_face(t) is the facial expression intensity value, I_voice(t) is the voice emotion intensity value, and I_motion(t) is the action emotion intensity value. This generates uniform first emotion sequence data, which provides a single, clear, and highly consistent emotion-driven signal for subsequent generation steps, containing emotion labels, intensity values, and timestamps.

[0060] Preferably, to make the emotional transitions of the virtual avatar more natural, an emotional inertia model, such as a simple recurrent neural network (RNN), can be introduced in this fusion step. This RNN receives historical first emotional sequence data as input, and its hidden state represents the character's current emotional state. When a new emotional label is inconsistent with the historical emotional state, the model outputs an emotional inertia coefficient to smooth the changes in weights, thereby making the transitions of generated expressions, actions, and speech smoother and more continuous. The beneficial effects can be seen in [reference needed]. Figure 6 As shown.

[0061] Example 2

[0062] This embodiment will use a specific application scenario—real-time interaction between a virtual psychological counselor and a user—to illustrate in detail the specific execution process of the method described in this invention, aiming to demonstrate the synergistic effect and practical results of each technical link. In this scenario, the system aims to generate a virtual psychological counselor avatar that is able to provide real-time, coherent, and empathetic feedback based on the user's multimodal emotional expressions.

[0063] In a specific interactive session, the system first collects the user's multimodal data through hardware devices. The specific configuration is as follows: a high-definition webcam captures the user's facial and upper body visual data at 1920x1080 resolution and 30fps; a high-fidelity condenser microphone collects the user's voice data at a 48kHz sampling rate and 16-bit precision; simultaneously, the system's built-in OpenPose pose estimation algorithm analyzes the video stream in real time, outputting two-dimensional coordinates containing 17 key upper body joints at a frequency of 30 frames per second, serving as the user's motion data. These raw data streams undergo preprocessing, such as face detection and keypoint localization for video data, Wiener filtering for audio data, and coordinate normalization for motion data, before being fed into a 6-layer, 8-head Transformer encoder model. The model concatenates feature vectors from different modalities (such as 2048-dimensional visual features, 128-dimensional MFCC speech features, and 100-dimensional gesture features) and adds learnable modal embedding vectors. Through deep fusion via a multi-head self-attention mechanism, it captures the intrinsic correlation between the user's head-down posture, slowed speech rate, and specific words, ultimately generating a first multimodal feature representation that includes the user's comprehensive emotional state.

[0064] The visual features extracted from the first multimodal feature representation were fed into a ResNet-50 model fine-tuned on the AffectNet dataset. This model can recognize more subtle emotions than FER-2013; for example, it detects micro-expressions such as raised inner eyebrows and downturned corners of the mouth. Combining this with texture information, the model outputs the emotion label "sadness," and calculates its expression intensity value of 0.85 through a Softmax layer. This constitutes the first facial expression emotion data. Meanwhile, the speech feature vectors extracted from the feature representation show that the user's fundamental frequency (F0) means a drop to approximately 110Hz, and the speech rate is significantly lower than its baseline level. The system uses the "sadness" label identified by facial recognition as prior knowledge to perform confirmatory analysis on these acoustic features, rather than judging them in isolation as "calm" or "fatigue," thereby generating first speech emotion data highly consistent with facial expressions. Similarly, the analysis of motion features showed that the user's posture was characterized by shoulders turned inward and head slightly drooping. The range of joint angle changes in the past 3 seconds was less than 20% of the average value, and the calculated normalized motion amplitude parameters were extremely low. This was marked as the first motion emotion data corresponding to the emotion of "sadness" and aligned with the timestamp of the voice data.

[0065] To achieve empathic feedback from the virtual tutor, the system must first accurately quantify the consistency of the user's emotional expressions across different modalities. The system performs one-hot encoding on the labels ("sadness") of the three emotional sequences (facial, vocal, and motor) obtained in the preceding steps, forming three parallel numerical vector time series. The Dynamic Time Warping (DTW) algorithm is used to calculate the pairwise similarity between these three sequences. For example, when calculating the alignment of the facial and vocal emotional sequences, the DTW algorithm constructs a cost matrix where the (i, j) element represents the Euclidean distance between the facial emotional vector of frame i and the vocal emotional vector of frame j. The optimal path found by the algorithm indicates that the user's vocal choking (drastic fluctuations in volume and pitch) lags slightly by about 200 milliseconds behind their tearful expression. By normalizing the cost values ​​on the DTW path, the system generates a first modality association mapping matrix. In this interaction segment, due to the high consistency of the user's emotional expressions, the synchronization weights for corresponding time periods in the matrix are generally above 0.9, providing a strong basis for generating coherent feedback subsequently.

[0066] The system begins constructing the virtual counselor's response actions. A pre-trained linear regression model, namely the facial expression mapping table described in this invention, is invoked. The model's input consists of the target emotion (e.g., "empathy," which can be defined as a mixture of "sadness" and "concern") set by the system, the target emotion intensity (e.g., 0.8), and the high synchronization weight (0.9) obtained from the first modality association mapping matrix. Based on this, the model outputs a set of target posture parameters, for example, recommending that the counselor's shoulder joint abduction be 5 degrees and body lean forward 3 degrees to form a listening and accepting posture.

[0067] However, to avoid generating mechanical and stiff movements, a closed-loop feedback adjustment mechanism is used to adjust the multimodal features. The system initially generates the image of the mentor: a gentle, empathetic facial expression with an intensity value of 0.8. However, the initial rate of change in body movements is low, deviating from the strong empathy expressed by the face. The system calculates in real time that the normalized deviation between the facial expression intensity value and the rate of change in joint angles is 0.4, exceeding the preset threshold of 0.3. This triggers the multimodal feature adjustment module. This module uses a linear interpolation method to smoothly adjust the mentor's motion amplitude parameters from the current value to a target value that better matches the expression intensity over the next 30 frames (1 second). Visually, this adjustment process is manifested as the mentor, from a relatively static posture, very naturally and slowly leaning forward further, accompanied by a slight nod indicating understanding. After this adjustment, a second multimodal feature representation containing optimized motion features is generated.

[0068] Finally, the system enters the final generation stage. Features from the adjusted facial, voice, and motion modalities are weighted and summed according to the high synchronization weights in the first modality association mapping matrix, fusing them into a unified first emotion sequence data. This sequence serves as the final driving signal, guiding the generation of the virtual avatar. The system calls a keypoint-driven algorithm to interpolate key points of the tutor's facial model, generating coherent and natural empathic facial expression animations. Simultaneously, it calls a fundamental frequency adjustment algorithm to generate a soft tone and slow speech rate, and based on the facial expression and motion mapping table and the adjusted parameters, it generates a forward-leaning and nodding motion sequence through joint angle interpolation. Finally, the system outputs a fully time-aligned virtual avatar clone data containing facial expression animations, voice tone, and motion sequences. The entire process ensures a high degree of coordination and unity between the virtual tutor's appearance, voice, and body language, providing users with an extremely realistic and trustworthy interactive experience, fully demonstrating the technical advantages of this invention in constructing high-fidelity, emotionally consistent virtual avatars.

[0069] In contrast, virtual avatars using existing technology, when faced with the same user input, may generate a fixed template of sadness in terms of expression and action due to the lack of dynamic feedback adjustment mechanisms. This fails to reflect the subtle changes in the user's emotions from venting to a slight easing, and cannot make such natural and synchronized empathetic actions, resulting in a stiff interactive experience and an inability to establish an effective trust relationship.

[0070] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing a virtual avatar model based on image cloning, characterized in that, include: Visual data, speech data, and motion data are acquired and fused to generate the first multimodal feature representation; Visual features, speech features, and action features are separated from the first multimodal feature representation, and first facial expression emotion data, first speech emotion data, and first action emotion data are generated respectively. The emotional tags and timestamps in the first facial expression emotional data, the first voice emotional data, and the first action emotional data are obtained. The emotional tags are mapped into numerical vectors. The distance between the emotional numerical vectors of different modalities is calculated by using a dynamic time warping algorithm, thereby generating a first modality association mapping matrix containing time axis synchronization weights. Feedback adjustment is performed on the first multimodal feature representation to generate a second multimodal feature representation. The feedback adjustment includes: calculating the deviation between the expression intensity value in the first facial expression emotion data and the motion feature parameters included in the first multimodal feature representation, and adjusting the first multimodal feature representation based on whether the deviation exceeds a preset threshold. Specifically, calculating the deviation between the expression intensity value in the first facial expression emotion data and the motion feature parameters included in the first multimodal feature representation includes: obtaining the time axis synchronization weight from the first modality association mapping matrix, combining it with the posture angle sequence and motion velocity vector in the first multimodal feature representation, and fitting it using a linear regression model to generate an expression-motion mapping model that defines the correspondence between emotion labels and limb posture angles; obtaining the joint angle change rate from the expression-motion mapping model, and calculating the deviation between the two in conjunction with the expression intensity value of the first facial expression emotion data. Based on the first modality association mapping matrix and the second multimodal feature representation, a unified first sentiment sequence data is generated; Based on the first emotion sequence data, virtual avatar clone data containing facial expression animation, voice tone and action sequence is generated; The step of acquiring visual data, voice data, and motion data, and fusing them to generate a first multimodal feature representation includes: acquiring the visual data, voice data, and motion data from a live video stream formed by real-time capture of real people; The visual data, speech data, and motion data are fused using a Transformer-based attention mechanism, and a first multimodal feature representation containing visual feature vectors, speech emotion feature vectors, and motion pattern feature vectors is generated by feature concatenation.

2. The method according to claim 1, characterized in that, The Transformer attention mechanism is configured with eight attention heads and extracts a sequence of facial coordinates containing 68 key points from the visual data.

3. The method according to claim 1, characterized in that, In the step of generating virtual avatar clone data, the expression and motion mapping model is combined with joint angle interpolation to generate motion sequences.

4. The method according to claim 1, characterized in that, The step of adjusting the first multimodal feature representation in the feedback adjustment includes: A linear interpolation method is used to smoothly adjust the motion amplitude parameter and the rhythm parameter, which is the normalized motion speed, based on the timestamp; wherein the preset threshold used to trigger the adjustment is 0.

3.

5. The method according to claim 1, characterized in that, The step of generating the first sentiment sequence data further includes: An emotional inertia model is introduced, which receives historical first emotional sequence data as input and outputs an emotional inertia coefficient. The emotional inertia coefficient is used to dynamically adjust the time axis synchronization weight in the first modality association mapping matrix.

6. The method according to claim 1, characterized in that, The step of calculating the distance between sentiment value vectors of different modalities using the dynamic time warping algorithm includes: The distance between the sentiment value vectors is calculated using Euclidean distance, and a similarity matrix is ​​generated based on the calculated distance to mark inconsistency intervals; wherein, when the distance exceeds a preset threshold of 0.2, the interval is marked as inconsistent.

7. The method according to claim 1, characterized in that, The step of generating the first facial expression emotion data includes: Facial texture information is extracted from the visual features, and the facial texture information is processed using a ResNet-50 convolutional neural network to generate first facial expression emotion data containing emotion labels based on the FER-2013 standard and expression intensity values ​​output by the Softmax function; wherein the facial texture information input to the ResNet-50 convolutional neural network is processed to a resolution of 256x256 pixels.

8. The method according to claim 1, characterized in that, When generating the first voice emotion data, a Gaussian mixture model was used to analyze the fundamental frequency range under different emotion labels. The fundamental frequency range of the happy state is 150Hz to 350Hz, the fundamental frequency range of the sad state is 80Hz to 200Hz, and the fundamental frequency range of the angry state is 150Hz to 400Hz.

9. A virtual avatar model construction system based on image cloning, characterized in that, include: The multimodal feature generation module is used to acquire and fuse visual, speech, and action data to generate the first multimodal feature representation; The step of acquiring visual data, voice data, and motion data, and fusing them to generate a first multimodal feature representation includes: acquiring the visual data, voice data, and motion data from a live video stream formed by real-time capture of real people; The visual data, speech data and action data are fused using a Transformer-based attention mechanism, and a first multimodal feature representation containing visual feature vectors, speech emotion feature vectors and action pattern feature vectors is generated by feature concatenation. The facial expression emotion recognition module, the voice emotion analysis module, and the action emotion generation module are used to generate first facial expression emotion data, first voice emotion data, and first action emotion data respectively from the first multimodal feature representation; The modality association mapping module is used to obtain sentiment tags and timestamps, map the sentiment tags into numerical vectors, and use a dynamic time warping algorithm to calculate the distance between sentiment numerical vectors of different modalities, thereby generating a first modality association mapping matrix containing time axis synchronization weights. A multimodal feature adjustment module is used to provide feedback adjustment to the first multimodal feature representation to generate a second multimodal feature representation. The feedback adjustment includes: calculating the deviation between the expression intensity value in the first facial expression emotion data and the motion feature parameters included in the first multimodal feature representation, and adjusting the first multimodal feature representation based on whether the deviation exceeds a preset threshold. Specifically, calculating the deviation between the expression intensity value in the first facial expression emotion data and the motion feature parameters included in the first multimodal feature representation includes: obtaining the time axis synchronization weight from the first modality association mapping matrix, combining it with the posture angle sequence and motion velocity vector in the first multimodal feature representation, and fitting it using a linear regression model to generate an expression-motion mapping model that defines the correspondence between emotion labels and limb posture angles; obtaining the joint angle change rate from the expression-motion mapping model, and calculating the deviation between the two in conjunction with the expression intensity value of the first facial expression emotion data. The sentiment sequence generation module is used to generate unified first sentiment sequence data based on the first modality association mapping matrix and the second multimodal feature representation; The virtual avatar clone generation module is used to generate virtual avatar clone data containing facial expression animation, voice tone and action sequence based on the first emotion sequence data.

Images