Portrait animation generation method and system based on reversible encoder and dynamic optimization

Abstract

The invention provides a portrait animation generation method and system based on a reversible encoder and dynamic optimization, and relates to the technical field of video portrait reconstruction, high-precision portrait animation generation is realized through multi-module collaborative innovation, multi-scale appearance features are extracted by using a VAE encoder, identity vectors extracted by ArcFace and global appearance embedding of CLIP are combined, and the portrait animation generation method and system based on the reversible encoder and dynamic optimization are realized. Constructing a multi-modal characteristic system; a high-precision attitude estimation algorithm is adopted to extract fine joint points, and an error between a generated attitude and an original attitude is calibrated in real time through a dynamic optimization strategy, so that the attitude driving accuracy is remarkably improved. A reversible residual block is designed to realize lossless two-way transmission of features, and the problem of information compression of a traditional encoder is avoided; a high-frequency detail injection mechanism is introduced, high-frequency features such as facial textures are extracted through a pre-training model and fused with shallow features, an attention mechanism guided by identity features is introduced based on the generation process of a diffusion model, and the consistency of character identities in the generation process is ensured.

Description

Technical Field

[0001] This invention belongs to the field of video portrait reconstruction, specifically relating to a method and system for generating portrait animation based on a reversible encoder and dynamic optimization. Background Technology

[0002] With the rapid development of multimedia technology and artificial intelligence, human animation generation is increasingly widely used in fields such as virtual live streaming, film and television production, and human-computer interaction. Diffusion models have demonstrated powerful capabilities in image and video synthesis, bringing new opportunities for the development of image animation technology. Especially in human image animation, this technology can drive reference images based on a series of preset poses, making them dynamic and thus generating highly controllable human animation videos. However, traditional methods have significant shortcomings in feature processing, pose driving, and dynamic optimization, making it difficult to meet the needs of high-precision human animation generation. Existing human animation generation technologies (such as the Animate Anyone model) mainly extract features from reference images and pose sequences, using diffusion models to generate animated videos.

[0003] The key to human portrait animation generation lies in synthesizing dynamic videos based on reference images and input action sequences, while ensuring the consistency of the person's identity features (especially facial information). However, existing methods still face many challenges when dealing with complex action changes: (1) Identity consistency is compromised, as facial areas are prone to deformation and distortion under intense movements, making it difficult to maintain stable individual features. (2) Video quality deteriorates, as the most advanced human portrait animation generation models (such as MimicMotion and ControlneXt) rely on external face-swapping tools (FaceFusion) for post-processing. While this method can improve local details, it often reduces the overall visual quality of the video. (3) The contradiction between spatial and temporal modeling: even though there are many image generation models that focus on identity consistency, directly embedding them into video diffusion models often leads to modeling conflicts. The fundamental reason is that after introducing a temporal modeling layer into the video diffusion model, the originally stable spatial feature distribution is disturbed, while image-based ID protection methods usually rely on static spatial feature distribution. This imbalance leads to a decrease in identity preservation capabilities and creates an irreconcilable contradiction between ID restoration and video temporal smoothness.

[0004] Currently, widely used human portrait animation generation technology mainly includes the following steps: First, a unidirectional encoder such as ResNet is used to extract the appearance features of the reference human portrait, and a 2D pose sequence is obtained by using a model such as OpenPose; the appearance features and pose features are input into a U-Net diffusion generator, and the generation process is guided by a cross-attention mechanism; finally, the latent features are converted into video frames using a VAE decoder.

[0005] However, the aforementioned portrait animation generation technology still has the following shortcomings:

[0006] 1. Feature Compression and Information Loss: Traditional encoders compress facial details (such as texture and expression) and identity features (such as facial structure) in the reference image, leading to problems such as facial blurring and identity drift in the generated portrait. For example, when generating a profile, structural distortions such as a crooked nose bridge and disproportionate eyes may occur, or there may be appearance deviations such as skin color and hairstyle that are inconsistent with the reference image.

[0007] 2. Insufficient accuracy of pose-driven methods: Existing pose estimation models (such as OpenPose) can only extract 18 key points of the body, lacking fine pose detection for the hands (such as finger bending) and the face (such as micro-expressions), and do not dynamically optimize the pose sequence. The generated portrait is prone to reverse joint movements or disjointed movements when performing complex actions (such as gestures and head turns), such as finger joint misalignment when waving or asymmetry of the corners of the mouth when smiling.

[0008] 3. Lack of Temporal Continuity and Physical Constraints: Existing diffusion models do not explicitly model the temporal dependencies between video frames and lack dynamic optimization mechanisms such as optical flow constraints, resulting in texture flickering and ghosting issues during fast movements (such as running and waving). At the same time, facial expression changes (such as the transition from smiling to serious) lack geometric consistency constraints, which can easily lead to abrupt changes in expression or breaks in identity features.

[0009] 4. Insufficient Identity Consistency and Detail Preservation: Existing methods rely on semantic features such as CLIP for identity constraints, lacking strong discriminative features for faces (such as ArcFace features), and cannot accurately control the facial identity consistency of generated portraits. In long video generation, differences in facial features of the same person in different frames (such as changes in eye color and face distortion) are prone to occur, seriously affecting the generation quality. Summary of the Invention

[0010] To address the shortcomings of existing technologies, this invention provides a method and system for generating human portrait animation based on a reversible encoder and dynamic optimization. It solves problems such as insufficient feature compression, pose-driven accuracy, poor temporal coherence, and identity consistency in existing technologies. It also addresses issues like feature loss of high-frequency information such as facial texture and expression during portrait reconstruction, which can lead to blurred skin and broken hair in the generated portrait. This invention extracts key points from the body, hands, and face, supporting millimeter-level reconstruction of fine movements such as "making a heart shape" and "holding a pen." A pose feedback loss is introduced into the diffusion model, and joint positions are calibrated in real time during generation, improving the cosine similarity between the generated pose and the input sequence. Spatiotemporal dimension calculations are separated in UNet, prioritizing time-axis dependencies and supporting motion coherence modeling and smooth motion transitions in videos. The aim is to achieve high-precision, high-fidelity human portrait animation generation, meeting the needs of virtual live streaming, film and television production, and other fields for high-quality human portrait animation.

[0011] The technical solution of this invention is as follows:

[0012] A method for generating portrait animation based on a reversible encoder and dynamic optimization includes the following steps:

[0013] Step S1: Acquire a reference portrait image and perform preprocessing to obtain a preprocessed reference portrait image;

[0014] Specifically, this involves acquiring still images of the target person, i.e., reference portrait images, using high-resolution cameras or multi-camera systems. N is the frame number of the reference portrait image, and N is the number of frames of the reference portrait image.

[0015] For each reference portrait image, face detection and alignment are performed, the image is cropped to 256×256 pixels, and color normalization is applied.

[0016] ;

[0017] in, and These are the mean and standard deviation of the dataset, respectively. Indicates a reference portrait image. This represents the normalized image;

[0018] Step S2: Obtain the motion-driven sequence based on the preprocessed reference portrait image;

[0019] The DWPose-based action sequence estimation algorithm is used to extract action-driven sequences. Where T is the number of frames and K is the number of coordinates of two-dimensional joints;

[0020] Then, the coordinates of the two-dimensional joints are normalized to obtain the normalized coordinate points (x', y'):

[0021] ;

[0022] Where (x, y) are the original pixel coordinates of the pixels in the image. This represents the coordinates of the center point c of the image, where d is half the length of the image diagonal, ensuring that the coordinate range is normalized to [-1, 1].

[0023] Step S3: Extract features from the preprocessed reference portrait image to obtain multi-scale feature maps, normalized identity vectors, and global feature vectors;

[0024] The process of obtaining the multi-scale feature map is as follows: the preprocessed reference portrait image is converted into latent variable features by the frozen VAE encoder, the latent variable features are copied to match the video frame number, and spliced ​​with the action-driven sequence to ensure temporal consistency.

[0025] The VAE encoder is a variational autoencoder, and its hierarchical structure includes a bottom feature layer, a middle semantic layer, and a high abstraction layer.

[0026] Bottom feature layer: 2 layers of 3×3 convolution, stride 2, channels 64→128, extracting color distribution and edge information;

[0027] Middle semantic layer: 2 layers of 3×3 convolutions, stride 2, channels 128→256, capturing texture patterns;

[0028] High-level abstraction layer: 1 layer of global average pooling, outputting a 256-dimensional vector. It represents the overall semantics of the image;

[0029] ;

[0030] in This represents the low-level feature vector of the i-th layer. This represents the semantic feature vector of the middle layer of the i-th layer. Let H represent the high-level feature vector of the i-th layer, H represent the original height of the input image, W represent the original width of the input image, C1 represent the number of channels of the low-level features, and C2 represent the number of channels of the middle-level features. The vector has dimension D, is a one-dimensional real vector, and H=W=32, C1=C2=64, D=512;

[0031] The final output multi-scale feature map F app ={f low f mid f glb The corresponding resolutions are respectively , ;

[0032] The process of obtaining the normalized identity vector specifically involves: taking the preprocessed reference portrait image... Facial features are extracted using ArcFace and then fed into a denoising U-Net network to guide the generation process.

[0033] The denoising U-Net network comprises a backbone network, a feature normalization layer, and a classification head;

[0034] Backbone network: 4-stage residual blocks, outputting 2048-dimensional features;

[0035] Feature normalization layer: L2 normalization;

[0036] Classification head: Combine ArcFace loss to enhance inter-class distance:

[0037] ;

[0038] in, This represents the ArcFace loss function, where s represents the scaling factor. The cosine similarity between the feature vector and the weight vector representing the real identity is denoted by m, where m represents the angular margin, r represents the real identity label, and j represents the non-target category. The angle represents the cosine similarity between the feature vector and the weight vector of the non-target category j.

[0039] The final output is a normalized identity vector. ;

[0040] The process of obtaining the global feature vector is as follows: the preprocessed reference portrait image is simultaneously passed through the CLIP image encoder to extract the global feature vector, and the global feature vector is input into each cross-attention layer of the denoising U-Net network to enhance the appearance consistency of the generated image; specifically as follows:

[0041] Patch embedding: The input image is segmented into 24×24=576 patches, and each patch is mapped to a 768-dimensional vector;

[0042] Location coding: Add learnable location coding to preserve spatial information;

[0043] Transformer layer: Outputs a sequence through 14 layers of self-attention processing. ;

[0044] Global features: Extracting global embeddings using CLS tokens Local features are extracted by average pooling. global feature vector ;

[0045] Step S4: Preprocess and optimize the action-driven sequence to obtain a latent variable sequence with identity consistency, posture coherence and detail fidelity;

[0046] Step S4.1: Initialize the sequence of random latent variables It follows a Gaussian distribution. Let t represent the t-th latent variable in the sequence, and T represent the length of the sequence;

[0047] Step S4.2: During each inverse diffusion iteration, based on the currently generated key points... With the original key points Calculate feedback loss L feedback : ;

[0048] Where K represents the original keypoint coordinates in the action-driven sequence. Keypoint weights;

[0049] Step S4.3: Model the diffusion process as a continuous-time dynamic system and construct the HJB equation to solve for the optimal noise strategy;

[0050] ;

[0051] Where V(z,t) is the value function, which represents the optimal cumulative loss at state z and time t. The value function includes the difference between the generated facial features and ArcFace reference features and the continuity constraint of pose change. Indicates noise; The dot product of the gradient of the value function V with respect to the latent variable z and the noise is used to represent the influence of noise on the value function.

[0052] By solving the HJB equation, the noise of each step of reverse diffusion is obtained. Combined with feedback loss The latent variables are iteratively updated until all back diffusion is completed. Finally, all the updated latent variables are collected to form a sequence of latent variables.

[0053] Step S5: Transfer the low-level feature vectors Mid-level semantic feature vector With normalized identity vector The input is fed into the reversible encoding and feature fusion module to obtain a reversible feature representation of multimodal fusion;

[0054] The reversible encoding and feature fusion module achieves lossless feature compression and multi-source information fusion through a bidirectional mapping mechanism of reversible residual blocks; specifically, it includes:

[0055] Reversible coding layer: The reversible residual block performs dimensional alignment and reversible transformation on the low-level feature vector and the middle-level semantic feature vector, while preserving spatial details and semantic information, and compressing the features to a low-dimensional space.

[0056] Feature fusion layer: The normalized identity vector is concatenated with the compressed multi-scale features, and the association between "identity and appearance" is strengthened through an attention mechanism;

[0057] The reversible encoding and feature fusion module ultimately generates a reversible feature representation fused from multiple modalities.

[0058] Step S5.1: Construct a reversible residual block;

[0059] The input to the reversible residual block is: low-level feature vector. Mid-level semantic feature vector First, the resolution is adjusted to 64×64 and 32×32 through interpolation, and then the number of channels is unified to 64 through 1×1 convolution; the normalized identity vector is then... Compression is achieved through a fully connected layer;

[0060] Step S5.2: Multimodal fusion of reversible residual blocks;

[0061] The Glow architecture is used to achieve bidirectional mapping. The adapted low-level feature vector and the normalized identity feature vector are first fused by the conditional convolution function, and then concatenated with the middle-level semantic feature vector. The input is two sets of 3×3 convolutional layers, and the output is the preliminary fused feature. The reverse reconstruction capability is retained through the inverse operation. ;

[0062] in, , Representing two branches of the input features, It is the input feature branch direct mapping, It is the input feature branch The result after exponential scaling and offset transformation Inject a normalized identity vector into the conditional convolution function. With high-level feature vectors ;

[0063] Each invertible residual block contains two sets of 3×3 convolutional layers, with a mapping function f(·):

[0064] ;

[0065] Reverse operation passed ; The input feature tensor representing the residual block; This represents the intermediate output features of the first 3×3 convolution, which are derived from the input features. The results after preliminary feature extraction; The output of the mapping function representing the residual block is a feature of the intermediate output. Residual characteristics after further processing; The output feature tensor of the residual block represents the fusion result of the input features and residual features. ReLU is the activation function, and Conv... 3×3 Refers to a 3×3 convolution operation;

[0066] Step S5.3: High-frequency detail feature injection mechanism:

[0067] High-frequency detail features are obtained by extracting pre-processed reference portrait images using a pre-trained ESRGAN network. Specifically, the pre-trained ESRGAN network extracts four levels of high-frequency features, ranging from 16×16 to 256×256, and the second-level feature of 64×64×64 is taken as the high-frequency detail feature. ;

[0068] Low-frequency fundamental features are fused with high-frequency detailed features through reversible convolutional layers:

[0069] ;

[0070] in, This represents the final feature after fusing high-frequency details; it is the output resulting from combining low-frequency fundamental features with high-frequency detail features. This is the output of the encoder's first layer, i.e., the low-level basic features. Indicates high-frequency detail features Perform a 3×3 convolution operation;

[0071] Step S6: Input the reversible feature representation, latent variable sequence, normalized identity vector, and entropy-based feature into the conditional diffusion generator to obtain the latent feature sequence. ;

[0072] Step S6.1: Forward noise process;

[0073] According to the preset noise scheduling sequence sequentially for the initial latent variables Add noise:

[0074] Positive noise addition: ;

[0075] in, This represents the latent variable after adding noise at step t; This represents the noise scheduling coefficient, which increases with time step t. This represents standard Gaussian noise, which is an injected random disturbance;

[0076] Step S6.2: Reverse noise process;

[0077] Entropy features As an additional condition, it is input into the denoising U-Net network.

[0078] Inverse noise reduction: ;

[0079] in, This represents the mean of the latent variables after denoising. This represents the cumulative noise figure, which is the total noise impact from the initial step to the t-th step. This represents the noise predicted by the denoised U-Net;

[0080] Step S6.3: Cross-attention guidance;

[0081] In each Transformer block, a normalized identity vector is used. For the query, the features extracted by the denoising U-Net network are key values, and the following calculations are performed:

[0082] ;

[0083] In this context, the normalized identity vector serves as the key K and value V, which are the features extracted by U-Net, containing spatial structure and detailed information; d represents the scaling factor, and Pos represents the positional encoding.

[0084] Step S6.4: Noise prediction and update;

[0085] based on The denoising network predicts noise, and the update formula is:

[0086] ;

[0087] in ;

[0088] Step S7: Perform spatiotemporal optimization and video decoding on the temporally consistent latent feature sequences to obtain a human portrait animation video with consistent identity, coherent movements, and high detail.

[0089] Step S7.1: Specifically, the time axis and spatial axis are processed separately through the axial time attention mechanism;

[0090] Time axis processing: The dimension of the latent feature sequence is adjusted to RT×(H′W′C), and cross-frame dependencies are captured through MultiHeadAttn multi-head attention.

[0091] ;

[0092] ;

[0093] Where R is the number of time step repetitions, T is the number of video frames, H′ and W′ are the feature map height and width, and C is the number of channels. Represents the original temporal potential features, It is a time-optimized feature sequence;

[0094] Spatial axis processing: Apply standard spatial attention to the single-frame feature RH′×W′×C to enhance local details.

[0095] Step S7.2: Implementation of optical flow constraint;

[0096] Specifically, perform RAFT optical flow estimation: Input z t z t+1 Output optical flow field ;

[0097] Step S7.3: Latent feature update;

[0098] ;

[0099] Where γ is the learning rate. To optimize the gradient for HJB, z t v represents the latent feature at the current time t. t The optical flow graph represents the current moment. Indicates characteristic distortion based on optical flow. This represents the potential features of the next frame after the update;

[0100] Loss function L HJB for:

[0101] ;

[0102] Where L face For facial identity feature matching loss, L pose For the smoothness loss of the pose sequence, L flow For optical flow consistency loss, To balance the weights;

[0103] Facial identity feature matching loss L face as follows:

[0104] ;in For facial feature extractors, f ref ArcFace features of the reference image, This means taking the expectation of the latent features over all times t, which is equivalent to calculating the average facial feature error;

[0105] attitude sequence smoothness loss L pose as follows:

[0106] ;

[0107] in, For the generated pose keypoints in frame t, Let be the attitude velocity in frame t. This represents the average of the attitude change smoothing error over time step t from 1 to T−1.

[0108] Optical flow uniformity loss as follows: ;

[0109] in represents the optical flow field estimated by RAFT; Warp is a feature warping operation based on optical flow.

[0110] The diffusion process is considered as a continuous-time dynamic system with facial and posture constraints, with the state variable z(t) and the control variable noise. The Hamiltonian H is constructed as follows:

[0111] ;

[0112] Discretization: The Euler method is used to map the continuous time step t∈[0,1] to the diffusion step T, and the value function is calculated iteratively. ;

[0113] Optimal control: Solving for optimal noise through backpropagation ,make Minimize, the formula is as follows: ;

[0114] Step S8: Balance the overall parameters of the model;

[0115] Denoising network weight adjustment: Facial features have the highest priority, i.e. The optical flow constraint weight is lower than the attitude weight, i.e. ;

[0116] Input to the denoising network: Pose features are extracted using DWPose and encoded into a latent vector, along with z... t Concatenated input denoising UNet network; optical flow field v t After downsampling to z t After resolution matching, the feature space is embedded through convolutional layers;

[0117] During the training phase of the denoising network: the HJB optimization objective is added as a regularization term to the loss function of the diffusion model and jointly optimized with the original noise prediction loss;

[0118] Joint optimization function: ;

[0119] in, It is the original loss of the diffusion model. It is a coefficient that controls the weight of the regularization term.

[0120] Inference phase: During the reverse diffusion process, z is iteratively adjusted using the HJB equation. t Ensure that the generated facial latent features at each step are close to the ArcFace reference features, and that the generated pose and action are consistent with the input sequence.

[0121] Step S9: Decode the VAE encoder by upsampling layer by layer through multi-scale transposed convolution to a 256×256 output frame; select key frames every m frames and use the DAIN algorithm to interpolate between key frames to 30fps;

[0122] Transposed convolution layers: Layer 1: Transposed convolution (4×4, stride 2), channels 512→256, ReLU activation; Layer 2: Transposed convolution (4×4, stride 2), channels 256→128, ReLU activation; Layer 3: Convolution (3×3, stride 1), channels 128→3, Sigmoid activation, output pixel values ​​∈ [0,1].

[0123] Step S10: Color correction and optimization;

[0124] Specifically, the timing post-processing is adopted: the key frames generated by inter-frame interpolation are used to generate intermediate frames using bidirectional interpolation based on optical flow, thereby increasing the frame rate to 30fps; then color consistency correction is performed, and histogram matching is used to ensure that the skin color difference ΔE between adjacent frames is less than 2.0.

[0125] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0126] First, a reversible encoder structure was constructed, and reversible residual blocks were designed to achieve lossless transmission and bidirectional reconstruction of feature information. This module effectively integrates multi-scale latent image features extracted by the VAE encoder with ArcFace identity features, and accurately recovers the original input through reversible operations during the decoding stage. This significantly reduces feature redundancy and image degradation, improving the clarity and identity consistency of the generated animation images.

[0127] Secondly, a dynamic feedback optimization strategy combining optimal control theory is proposed to address the noise interference problem in diffusion-based generation. The optimal generation path for facial and action poses in portrait generation is solved using the Hamilton–Jacobi–Bellman (HJB) equation. In each iteration of diffusion, the distribution of injected noise is dynamically adjusted based on the difference between the currently generated keypoints and the target pose, thereby accurately fitting the driving skeletal motion. Simultaneously, the ArcFace identity vector is introduced as a strong supervisory signal, and cross-attention and conditional normalization mechanisms constrain the generated facial features, ensuring that facial expression remains consistent throughout continuous movements.

[0128] Meanwhile, in terms of feature modeling, multimodal appearance information is integrated, including low / medium / global features generated by the VAE encoder, identity features extracted by ArcFace, and appearance vectors embedded by CLIP. In addition, high-frequency detail textures extracted by ESRGAN are injected into shallow features through a dedicated fusion module, effectively enhancing the texture clarity of key areas such as the face and hair.

[0129] Furthermore, this invention designs a conditional diffusion generation structure, introducing a cross-attention mechanism into the traditional U-Net architecture, so that the identity feature f id The noise prediction path of the diffusion process can be precisely controlled in each layer to achieve stable preservation of individual identities. A spatiotemporal joint optimization framework is constructed, which captures the continuity of action between time series frames and spatial structural details through a two-dimensional attention mechanism. Combined with the optical flow field estimated by RAFT, the latent variables between frames are deformed and aligned to improve the smoothness and stability of the generated video in dynamic scenes.

[0130] In summary, the present invention significantly outperforms existing solutions in terms of identity preservation, pose fitting, detail enhancement, and inter-frame consistency, and is applicable to various human image generation application scenarios such as virtual humans, digital avatars, and film and television compositing. Attached Figure Description

[0131] Figure 1 This is a flowchart of a portrait animation generation method based on a reversible encoder and dynamic optimization in an embodiment of the present invention;

[0132] Figure 2 This is a network module diagram that integrates a multi-attention mechanism in an embodiment of the present invention. Detailed Implementation

[0133] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0134] Example 1:

[0135] On the one hand, portrait animation generation methods based on reversible encoders and dynamic optimization, such as Figure 1 As shown, it includes the following steps:

[0136] Step S1: Acquire a reference portrait image and perform preprocessing to obtain a preprocessed reference portrait image;

[0137] Specifically, this involves acquiring still images of the target person, i.e., reference portrait images, using a high-resolution camera (at least 4K resolution) or a multi-camera system. The frame number of the reference portrait image is N, and the number of frames of the reference portrait image is N; at least N≥10 frames are collected, covering different expressions such as front view, left and right side view, smiling, and raised eyebrows;

[0138] For each reference portrait image, face detection and alignment are performed, the image is cropped to 256×256 pixels, and color normalization is applied.

[0139] ;

[0140] Where μ and σ are the mean and standard deviation of the dataset, respectively. Indicates a reference portrait image. This represents the normalized image;

[0141] Step S2: Obtain the motion-driven sequence based on the preprocessed reference portrait image;

[0142] The DWPose-based action sequence estimation algorithm is used to extract action-driven sequences. , where T is the number of frames (e.g., T=60) and K=128 is the number of coordinates of two-dimensional joints;

[0143] Then, the coordinates of the two-dimensional joints are normalized to obtain the normalized coordinate points (x', y'):

[0144] ;

[0145] Where (x, y) are the original pixel coordinates of the pixels in the image. This represents the coordinates of the center point c of the image, and d is half the length of the image diagonal, ensuring that the coordinate range is normalized to [-1, 1].

[0146] In this embodiment, the action sequence estimation algorithm of DWPose adopts a two-stage distillation architecture:

[0147] Teacher model: HRNet-W48, outputting 128 coordinates (body 18 + hands 42 + face 68), resolution 1 / 4 of the original image;

[0148] Student model: Lightweight MobileNetV3, which improves inference speed by 85 times through knowledge distillation compression.

[0149] Step S3: Extract features from the preprocessed reference portrait image to obtain multi-scale feature maps, normalized identity vectors, and global feature vectors;

[0150] The process of obtaining the multi-scale feature map is as follows: the preprocessed reference portrait image is converted into latent variable features by the frozen VAE encoder, that is, the feature vector after compression and high-dimensional semanticization. The latent variable features are copied to match the video frame number and spliced ​​with the action-driven sequence to ensure temporal consistency.

[0151] The VAE encoder is a variational autoencoder, and its hierarchical structure includes a bottom feature layer, a middle semantic layer, and a high abstraction layer.

[0152] Bottom feature layer: 2 layers of 3×3 convolution, stride 2, channels 64→128, extracting color distribution and edge information;

[0153] Middle semantic layer: 2 layers of 3×3 convolution, stride 2, channels 128→256, capturing texture patterns (such as skin pores, facial textures).

[0154] High-level abstraction layer: 1 layer of global average pooling, outputting a 256-dimensional vector. It represents the overall semantics of the image;

[0155] ;

[0156] in This represents the low-level feature vector of the i-th layer. This represents the semantic feature vector of the middle layer of the i-th layer. Let H represent the high-level feature vector of the i-th layer, H represent the original height of the input image, W represent the original width of the input image, C1 represent the number of channels of the low-level features, and C2 represent the number of channels of the middle-level features. The vector has dimension D, is a one-dimensional real vector, and H=W=32, C1=C2=64, D=512;

[0157] The final output multi-scale feature map F app ={f low f mid f glb The corresponding resolutions are respectively , ;

[0158] The process of obtaining the normalized identity vector specifically involves: taking the preprocessed reference portrait image... Facial features are extracted using ArcFace to improve facial alignment and identity consistency. The extracted facial features are then input into a denoising U-Net network to guide the generation process.

[0159] The denoising U-Net network comprises a backbone network, a feature normalization layer, and a classification head;

[0160] Backbone network: 4-stage residual blocks, outputting 2048-dimensional features;

[0161] Feature normalization layer: L2 normalization;

[0162] Classification head: Combines ArcFace loss to enhance inter-class distance:

[0163] ;

[0164] in, represents the ArcFace loss function, which improves the accuracy of identity recognition by minimizing this loss; s represents the scaling factor, used to amplify the difference in the magnitude of feature vectors; The cosine similarity between the feature vector and weight vector representing the real identity is denoted by m, which represents the angular margin, typically set to around 0.5; r represents the real identity label; and j represents the non-target category. The angle represents the cosine similarity between the feature vector and the weight vector of the non-target category j.

[0165] The final output is a normalized identity vector. ;

[0166] The process of obtaining the global feature vector is as follows: the preprocessed reference portrait image is simultaneously passed through the CLIP image encoder to extract the global feature vector, and the global feature vector is input into each cross-attention layer of the denoising U-Net network to enhance the appearance consistency of the generated image; specifically as follows:

[0167] Patch embedding: The input image is segmented into 24×24=576 patches, and each patch is mapped to a 768-dimensional vector;

[0168] Location coding: Add learnable location coding to preserve spatial information;

[0169] Transformer layer: Output sequence through 14 layers of self-attention processing. ;

[0170] Global features: Extracting global embeddings using CLS tokens Local features are extracted by average pooling. global feature vector .

[0171] The CLIP image encoder described in this embodiment adopts a CLIP-L / 14@336px architecture, with 14 Transformer layers, a patch size of 14×14, and an input resolution of 336×336, thereby capturing finer visual details.

[0172] Step S4: Preprocess and optimize the action-driven sequence to obtain a latent variable sequence with identity consistency, posture coherence and detail fidelity;

[0173] Step S4.1: Initialize the sequence of random latent variables It follows a Gaussian distribution. Let t represent the t-th latent variable in the sequence, and T represent the length of the sequence;

[0174] Step S4.2: During each inverse diffusion iteration, based on the currently generated key points... With the original key points Calculate feedback loss L feedback : ;

[0175] Where K represents the original keypoint coordinates in the action-driven sequence. As a keypoint weight, higher weights can be assigned to facial keypoints (such as eyes and lips). );

[0176] Step S4.3: Model the diffusion process as a continuous-time dynamic system and construct the HJB equation to solve for the optimal noise strategy;

[0177] ;

[0178] Where V(z,t) is the value function, which represents the optimal cumulative loss at state z and time t. The value function includes the difference between the generated facial features and ArcFace reference features and the continuity constraint of pose change. Indicates noise; The dot product of the gradient of the value function V with respect to the latent variable z and the noise is used to represent the influence of noise on the value function.

[0179] By solving the HJB equation, the noise of each step of reverse diffusion is obtained. Combined with feedback loss The latent variables are iteratively updated until all back diffusion is completed. Finally, all updated latent variables are collected to form a sequence of latent variables with consistent identity, coherent posture, and accurate details.

[0180] Step S5: Transfer the low-level feature vectors Mid-level semantic feature vector With normalized identity vector The input is fed into the reversible encoding and feature fusion module to obtain a reversible feature representation of multimodal fusion, such as... Figure 2 As shown;

[0181] The reversible encoding and feature fusion module achieves lossless feature compression and multi-source information fusion through a bidirectional mapping mechanism of reversible residual blocks; specifically, it includes:

[0182] Reversible coding layer: Encodes lower-level feature vectors using reversible residual blocks. and mid-level semantic feature vectors By performing dimensional alignment and reversible transformation, spatial details and semantic information are preserved while features are compressed into a low-dimensional space;

[0183] Feature fusion layer: The normalized identity vector is concatenated with the compressed multi-scale features, and the association between "identity and appearance" is strengthened through an attention mechanism;

[0184] The reversible encoding and feature fusion module ultimately generates a reversible feature representation that combines multi-scale spatial details, identity semantics, and reversible recovery capabilities, providing a unified input for the subsequent cross-modal generation of the denoising U-Net and ensuring the coordinated optimization of the generated image in terms of detail fidelity, identity consistency, and multi-scale feature integrity.

[0185] Step S5.1: Construct a reversible residual block;

[0186] The input to the reversible residual block is: low-level feature vector. Mid-level semantic feature vector (From VAE encoder), first adjust the resolution to 64×64 and 32×32 through interpolation, then unify the number of channels to 64 through 1×1 convolution; then normalize the identity vector. Compression is performed through fully connected layers to ensure matching with the feature channel dimensions, forming a "multimodal input set that can be directly fused";

[0187] Step S5.2: Multimodal fusion of reversible residual blocks;

[0188] The Glow architecture is used to achieve bidirectional mapping. The adapted low-level feature vector (64×64×64) and the normalized identity feature vector (64-dimensional) are first fused by a conditional convolution function (using the normalized identity feature vector as a constraint on the convolution kernel parameters), and then concatenated with the mid-level semantic feature vector (32×32×64, upsampled to 64×64). The result is then input into two sets of 3×3 convolutional layers and the output is the preliminary fused feature. The reverse reconstruction capability is preserved through the inverse operation to ensure no information loss. ;

[0189] in, , This represents two branches of the input features (such as spatial features and identity features). It is the input feature branch direct mapping, It is the input feature branch The result after exponential scaling and offset transformation Inject a normalized identity vector into the conditional convolution function. With high-level feature vectors ;

[0190] Each invertible residual block contains two sets of 3×3 convolutional layers, with a mapping function f(·):

[0191] ;

[0192] Reverse operation passed ; The input feature tensor representing the residual block; This represents the intermediate output features of the first 3×3 convolution, which are derived from the input features. The results after preliminary feature extraction; The output of the mapping function representing the residual block is a feature of the intermediate output. Residual characteristics after further processing; The output feature tensor representing the residual block is the fusion result of the input features and the residual features. It is an activation function, short for "Rectified Linear Unit". 3×3 This refers to the 3×3 convolution operation, which is an abbreviation for "Convolution".

[0193] Traditional ReferenceNet may employ a unidirectional fixed mapping, which can easily lead to information loss and blurred details. The residual-driven mechanism, by explicitly introducing temporal constraints (which can be viewed as optical flow or feature differences), significantly improves the inter-frame consistency and smoothness of the generated video. By using reversible residual blocks to fuse multimodal features (extracted from the reference image by VAE Enc, mainly containing information such as color and texture; a 128-dimensional vector extracted from the reference image by ArcFace, which can uniquely identify a person; and global feature embedding extracted by the CLIP image encoder to ensure the proportional coordination and correct positioning of the reconstructed portrait), and inserting reversible convolutions in shallow layers to fuse high-frequency detail features, it is possible to better preserve the information of different modal features and improve the quality and consistency of the generated portrait. In ReferenceNet, appearance features, identity features, and facial landmark features are respectively input into reversible residual blocks for fusion. Each feature, when passing through the reversible residual block, not only interacts with other features but also retains its own information. This fusion approach avoids the information loss problems that may occur in traditional feature fusion methods, enabling features from different modalities to work together better and providing richer and more accurate information for the subsequent generation process. To preserve and fuse high-frequency detail features, reversible convolutional layers are inserted into the shallow layers of ReferenceNet. The role of the reversible convolutional layers is to ensure bidirectional lossless information transmission while fusing high-frequency detail features. In this way, high-frequency detail features can be effectively integrated into the feature representation of ReferenceNet, allowing the generated portrait to retain overall structure and identity information while also possessing rich detail and realism.

[0194] Step S5.3: High-frequency detail feature injection mechanism:

[0195] High-frequency detail features are obtained by extracting pre-processed reference portrait images using a pre-trained ESRGAN network. Specifically, the pre-trained ESRGAN network extracts four levels of high-frequency features, ranging from 16×16 to 256×256, and the second-level feature of 64×64×64 is taken as the high-frequency detail feature. ;

[0196] Low-frequency fundamental features are fused with high-frequency detailed features through reversible convolutional layers:

[0197] ;

[0198] in, The final feature after fusing high-frequency details is the output after combining low-frequency basic features with high-frequency detail features, which is used to enhance the texture details of the generated image. This is the output of the encoder's first layer, i.e., the basic low-level features (64×64 resolution, 64 channels), and gradient vanishing is avoided through residual connections. Indicates high-frequency detail features The purpose of performing a 3×3 convolution operation is to adjust the dimension of high-frequency features;

[0199] Step S6: Input the reversible feature representation, latent variable sequence, normalized identity vector, and entropy-based feature from the multimodal fusion into the conditional diffusion generator to obtain a temporally consistent, identity-matched, and detailed latent feature sequence. ;

[0200] Step S6.1: Forward noise process;

[0201] According to the preset noise scheduling sequence sequentially for the initial latent variables Add noise:

[0202] Positive noise addition: ;

[0203] in, This represents the latent variable after adding noise at step t; This represents the noise scheduling coefficient, which increases with time step t. This represents standard Gaussian noise, which is an injected random disturbance;

[0204] S6.2: Reverse noise process;

[0205] Entropy features As an additional condition, it is input into the denoising U-Net network.

[0206] Inverse noise reduction: ;

[0207] in, This represents the mean of the latent variables after denoising. This represents the cumulative noise figure, which is the total noise impact from the initial step to the t-th step. This represents the noise predicted by the denoised U-Net;

[0208] Step S6.3: Cross-attention guidance;

[0209] In each Transformer block, a normalized identity vector is used. For the query, the features extracted by the denoising U-Net network are key values, and the following calculations are performed:

[0210] ;

[0211] In this context, the normalized identity vector serves as the key K and value V, representing the features extracted by U-Net, which contain spatial structure and detailed information; d represents the scaling factor, which is the dimension of the U-Net feature; and Pos represents the positional encoding, used to preserve the spatial positional information of the U-Net feature, ensuring the spatial correspondence between details and structure.

[0212] Step S6.4: Noise prediction and update;

[0213] based on The denoising network predicts noise, and the update formula is:

[0214] ;

[0215] in ;

[0216] Step S7: Perform spatiotemporal optimization and video decoding on the temporally consistent latent feature sequences to obtain a human portrait animation video with consistent identity, coherent movements, and high detail.

[0217] Function: Solves the problems of temporal dependence and the continuity of fast-moving actions in long videos.

[0218] Step S7.1: Specifically, the time axis and spatial axis are processed separately through the axial time attention mechanism;

[0219] Time axis processing: The dimension of the latent feature sequence is adjusted to RT×(H′W′C), and cross-frame dependencies are captured through MultiHeadAttn multi-head attention.

[0220] ;

[0221] ;

[0222] Where R is the number of time step repetitions, T is the number of video frames, H′ and W′ are the feature map height and width, and C is the number of channels. Represents the original temporal potential features, It is a time-optimized feature sequence that already possesses consistency and coherence in the time dimension;

[0223] Spatial axis processing: Apply standard spatial attention to the single-frame feature RH′×W′×C to enhance local details.

[0224] Step S7.2: Implementation of optical flow constraint;

[0225] Specifically, perform RAFT optical flow estimation: Input z t z t+1 Output optical flow field ;

[0226] Step S7.3: Latent feature update;

[0227] ;

[0228] Where γ is the learning rate. To optimize the gradient for HJB, z t v represents the latent feature at the current time t. t The optical flow graph represents the current moment. Indicates characteristic distortion based on optical flow. This represents the potential features of the next frame after the update;

[0229] Loss function L HJB for:

[0230] ;

[0231] Where L face For facial identity feature matching loss, L pose For the smoothness loss of the pose sequence, L flow For optical flow consistency loss, To balance the weights (default) );

[0232] Facial identity feature matching loss L face Ensure the generated facial features match the reference ArcFace features, as follows:

[0233] ;in For facial feature extractors (such as the ResNet-50 head, which outputs a 128-dimensional vector), f ref ArcFace features (128-dimensional, L2 normalized) of the reference image. This means taking the expectation of the latent features over all times t, which is equivalent to calculating the average facial feature error;

[0234] attitude sequence smoothness loss L pose Constrain the temporal smoothness of the generated pose sequence to avoid abrupt joint movements:

[0235] ;

[0236] in, The generated pose keypoints for frame t (128×2 dimensional). The attitude velocity in frame t (calculated by the difference between adjacent frames): ), This represents the average of the attitude change smoothing error over time step t from 1 to T-1.

[0237] Optical flow uniformity loss Force the latent features of adjacent frames to satisfy optical flow constraints to ensure that pixel motion is physically reasonable: ;

[0238] in For the optical flow field estimated by RAFT ( ); Warp is a feature warping operation based on optical flow (implemented by bilinear interpolation).

[0239] The diffusion process is considered as a continuous-time dynamic system with facial and posture constraints, with the state variable z(t) and the control variable noise. The Hamiltonian H is constructed as follows:

[0240] ;

[0241] Discretization: The Euler method is used to map the continuous time step t∈[0,1] to the diffusion step T, and the value function is calculated iteratively. ;

[0242] Optimal control: Solving for optimal noise through backpropagation ,make Minimize, the formula is as follows: ;

[0243] Step S8: Balance the overall parameters of the model;

[0244] Denoising network weight adjustment: Facial features have the highest priority, i.e. =1, ensuring identity does not drift; optical flow constraint weight is lower than attitude, i.e. =0.5, to avoid overly smooth movements that could lead to stiffness;

[0245] Input to the denoising network: Pose features are extracted using DWPose and encoded into latent vectors (e.g., compressed to 512 dimensions using LSTM), and then compared with z. t Concatenated input denoising UNet network; optical flow field v t After downsampling to z t After resolution matching, the feature space is embedded through convolutional layers;

[0246] During the training phase of the denoising network: The HJB optimization objective is added as a regularization term to the loss function of the diffusion model and jointly optimized with the original noise prediction loss (such as L2 loss).

[0247] Joint optimization function: ;

[0248] in, It is the original loss of the diffusion model. It is a coefficient that controls the weight of the regularization term.

[0249] Inference phase: During the reverse diffusion process, adjustments are made iteratively using the HJB equation. (Similar to the optimization steps in DDIM sampling), ensuring that the facial latent features generated in each step are close to the ArcFace reference features, and that the generated poses and actions are consistent with the input sequence.

[0250] Step S9: Decode the VAE encoder, specifically by upsampling layer by layer through multi-scale transposed convolution to output a 256×256 frame;

[0251] Frame rate improvement: Select key frames every m frames (m=5) and use the DAIN algorithm to interpolate between key frames to 30fps;

[0252] Transposed convolution layers: Layer 1: Transposed convolution (4×4, stride 2), channels 512→256, ReLU activation; Layer 2: Transposed convolution (4×4, stride 2), channels 256→128, ReLU activation; Layer 3: Convolution (3×3, stride 1), channels 128→3, Sigmoid activation, output pixel values ​​∈ [0,1].

[0253] S10: Color correction and optimization;

[0254] Specifically, the timing post-processing is adopted: the key frames generated by inter-frame interpolation (1 frame is selected every 5 frames) are used to generate intermediate frames using optical flow-based bidirectional interpolation (such as the DAIN algorithm) to increase the frame rate to 30fps; then color consistency correction is performed, and histogram matching is used to ensure that the skin color difference ΔE between adjacent frames is less than 2.0.

[0255] This invention achieves efficient fusion of lossless feature transmission and multimodal information by combining a reversible encoder structure with a dynamic optimization strategy, completely breaking through the bottlenecks of traditional methods in identity consistency, attitude accuracy, and time sequence coherence. Regarding identity consistency, ArcFace features are combined with HJB optimization, along with a reversible encoder, to achieve a cosine similarity of ≥0.98 for facial identities in the generated sequence, effectively avoiding identity drift and ensuring stable and consistent facial features across different frames. In terms of pose accuracy and naturalness, DWPose and HJB-based pose feedback optimization reduce the joint position keypoint error from 8px to 3px, improving hand movement accuracy by 40%, making the generation of complex movements (such as gestures and micro-expressions) more natural and accurate. Regarding temporal coherence and visual quality, spatiotemporal joint modeling and optical flow constraints reduce inter-frame pixel displacement error to <2px, lowering the FVD index by 25%, significantly reducing texture flicker and ghosting issues during fast movements, resulting in video smoothness close to real-world scenes. In terms of information preservation and detail fidelity, relying on the bidirectional mapping of the reversible encoder and ESRGAN high-frequency detail injection, the feature preservation rate is increased from 75% to 92%, facial texture PSNR is improved by 2.5dB, and the clarity of details such as hair strands and pupils is significantly enhanced, greatly improving the visual quality of generated portraits. In multi-dataset validation, the generated human portrait maintains stable performance even when faced with complex actions and long animation sequences. Simultaneously, through multimodal feature fusion and lightweight design, the network reduces bitrate requirements by 50% compared to traditional solutions while ensuring high-fidelity generation, meeting the needs of real-time generation and edge deployment. In summary, this invention achieves an organic unity of high-precision identity preservation, high-naturalness action generation, highly coherent temporal output, and efficient computation. It can be widely applied to virtual human animation production, film and television special effects generation, human-computer interaction, and other scenarios. It provides a highly efficient solution for high-precision human portrait animation generation that combines technological advancement and engineering practicality while maintaining the same subjective quality, demonstrating significant economic and social value.

Claims

1. A method and system for generating human portrait animation based on a reversible encoder and dynamic optimization, characterized in that, Includes the following steps: Step S1: Acquire a reference portrait image and perform preprocessing to obtain a preprocessed reference portrait image; Step S2: Obtain the motion-driven sequence based on the preprocessed reference portrait image; Step S3: Extract features from the preprocessed reference portrait image to obtain multi-scale feature maps, normalized identity vectors, and global feature vectors; Step S4: Preprocess and optimize the action-driven sequence to obtain a latent variable sequence with identity consistency, posture coherence and detail fidelity; Step S5: Transfer the low-level feature vectors Mid-level semantic feature vector With normalized identity vector The input is fed into the reversible encoding and feature fusion module to obtain a reversible feature representation of multimodal fusion; Step S6: Input the reversible feature representation, latent variable sequence, normalized identity vector, and entropy-based feature into the conditional diffusion generator to obtain the latent feature sequence. ; Step S7: Perform spatiotemporal optimization and video decoding on the temporally consistent latent feature sequences to obtain a human portrait animation video with consistent identity, coherent movements, and high detail. Step S8: Balance the overall parameters of the model; Step S9: Decode the VAE encoder by upsampling layer by layer through multi-scale transposed convolution to a 256×256 output frame; select key frames every m frames and use the DAIN algorithm to interpolate between key frames to 30fps; Transposed convolutional layers: Layer 1: Transposed convolution 4×4, stride 2, channels 512→256, ReLU activation; Layer 2: Transposed convolution 4×4, stride 2, channels 256→128, ReLU activation; Layer 3: Convolution 3×3, stride 1, channels 128→3, Sigmoid activation, output pixel values ​​∈ [0,1]; Step S10: Color correction and optimization; specifically, time-series post-processing is used: inter-frame interpolation pairs the key frames generated, and intermediate frames are generated using optical flow-based bidirectional interpolation to increase the frame rate to 30fps; then color consistency correction is performed, and histogram matching is used to ensure that the skin color difference ΔE between adjacent frames is less than 2.

0.

2. The method and system for generating human portrait animation based on a reversible encoder and dynamic optimization according to claim 1, characterized in that, Step S1 specifically involves: acquiring a static image of the target person, i.e., a reference portrait image, using a high-resolution camera or a multi-camera system. N is the frame number of the reference portrait image, and N is the number of frames of the reference portrait image. For each reference portrait image, face detection and alignment are performed, the image is cropped to 256×256 pixels, and color normalization is applied. ； in, and These are the mean and standard deviation of the dataset, respectively. Indicates a reference portrait image. This represents the normalized image.

3. The method and system for generating human portrait animation based on a reversible encoder and dynamic optimization according to claim 1, characterized in that, Step S2 specifically uses the DWPose-based action sequence estimation algorithm to extract the action-driven sequence. Where T is the number of frames and K is the number of coordinates of two-dimensional joints; Then, the coordinates of the two-dimensional joints are normalized to obtain the normalized coordinate points (x', y'): ； Where (x, y) are the original pixel coordinates of the pixels in the image. This represents the coordinates of the center point c of the image, and d is half the length of the image diagonal, ensuring that the coordinate range is normalized to [-1, 1].

4. The method and system for generating human portrait animation based on a reversible encoder and dynamic optimization according to claim 1, characterized in that, The process of obtaining the multi-scale feature map in step S3 is as follows: the preprocessed reference portrait image is converted into latent variable features by the frozen VAE encoder, the latent variable features are copied to match the video frame number, and spliced ​​with the action-driven sequence to ensure temporal consistency. The VAE encoder is a variational autoencoder, and its hierarchical structure includes a bottom feature layer, a middle semantic layer, and a high abstraction layer. Bottom feature layer: 2 layers of 3×3 convolution, stride 2, channels 64→128, extracting color distribution and edge information; Middle semantic layer: 2 layers of 3×3 convolutions, stride 2, channels 128→256, capturing texture patterns; High-level abstraction layer: 1 layer of global average pooling, outputting a 256-dimensional vector. It represents the overall semantics of the image; ； in This represents the low-level feature vector of the i-th layer. This represents the semantic feature vector of the middle layer of the i-th layer. Let H represent the high-level feature vector of the i-th layer, H represent the original height of the input image, W represent the original width of the input image, C1 represent the number of channels of the low-level features, and C2 represent the number of channels of the middle-level features. The vector has dimension D, is a one-dimensional real vector, and H=W=32, C1=C2=64, D=512; The final output multi-scale feature map F app ={f low f mid f glb The corresponding resolutions are respectively , ; The process of obtaining the normalized identity vector specifically involves: taking the preprocessed reference portrait image... Facial features are extracted using ArcFace and then fed into a denoising U-Net network to guide the generation process. The denoising U-Net network comprises a backbone network, a feature normalization layer, and a classification head; Backbone network: 4-stage residual blocks, outputting 2048-dimensional features; Feature normalization layer: L2 normalization; Classification head: Combines ArcFace loss to enhance inter-class distance: ； in, This represents the ArcFace loss function, where s represents the scaling factor. The cosine similarity between the feature vector and the weight vector representing the real identity is denoted by m, where m represents the angular margin, r represents the real identity label, and j represents the non-target category. The angle representing the cosine similarity between the feature vector and the weight vector of the non-target class j; The final output is a normalized identity vector. ; The process of obtaining the global feature vector is as follows: the preprocessed reference portrait image is simultaneously passed through the CLIP image encoder to extract the global feature vector, and the global feature vector is input into each cross-attention layer of the denoising U-Net network to enhance the appearance consistency of the generated image; specifically as follows: Patch embedding: The input image is segmented into 24×24=576 patches, and each patch is mapped to a 768-dimensional vector; Location coding: Add learnable location coding to preserve spatial information; Transformer layer: Outputs a sequence through 14 layers of self-attention processing. ; Global features: Extracting global embeddings using CLS tokens Local features are extracted by average pooling. global feature vector .

5. The method and system for generating human portrait animation based on a reversible encoder and dynamic optimization according to claim 1, characterized in that, Step S4 specifically includes the following steps: Step S4.1: Initialize the sequence of random latent variables It follows a Gaussian distribution. Let t represent the t-th latent variable in the sequence, and T represent the length of the sequence; Step S4.2: During each inverse diffusion iteration, based on the currently generated key points... With the original key points Calculate feedback loss L feedback : ; Where K represents the original keypoint coordinates in the action-driven sequence. Keypoint weights; Step S4.3: Model the diffusion process as a continuous-time dynamic system and construct the HJB equation to solve for the optimal noise strategy; ； Where V(z,t) is the value function, which represents the optimal cumulative loss at state z and time t. The value function includes the difference between the generated facial features and ArcFace reference features and the continuity constraint of pose change. Indicates noise; The dot product of the gradient of the value function V with respect to the latent variable z and the noise is used to represent the influence of noise on the value function. By solving the HJB equation, the noise of each step of reverse diffusion is obtained. Combined with feedback loss The latent variables are iteratively updated until all back diffusion is completed. Finally, all the updated latent variables are collected to form a sequence of latent variables.

6. The method and system for generating human portrait animation based on a reversible encoder and dynamic optimization according to claim 1, characterized in that, The reversible encoding and feature fusion module described in step S5 achieves lossless feature compression and multi-source information fusion through a bidirectional mapping mechanism of reversible residual blocks; specifically, it includes: Reversible coding layer: The reversible residual block performs dimensional alignment and reversible transformation on the low-level feature vector and the middle-level semantic feature vector, while preserving spatial details and semantic information, and compressing the features to a low-dimensional space. Feature fusion layer: The normalized identity vector is concatenated with the compressed multi-scale features, and the "identity-appearance" association is strengthened through an attention mechanism; The reversible encoding and feature fusion module ultimately generates a reversible feature representation fused from multiple modalities.

7. The method and system for generating human portrait animation based on a reversible encoder and dynamic optimization according to claim 6, characterized in that, Step S5 includes the following steps: Step S5.1: Construct a reversible residual block; The input to the reversible residual block is: low-level feature vector. Mid-level semantic feature vector First, the resolution is adjusted to 64×64 and 32×32 through interpolation, and then the number of channels is unified to 64 through 1×1 convolution; the normalized identity vector is then... Compression is achieved through a fully connected layer; Step S5.2: Multimodal fusion of reversible residual blocks; The Glow architecture is used to achieve bidirectional mapping. The adapted low-level feature vector and the normalized identity feature vector are first fused by the conditional convolution function, and then concatenated with the middle-level semantic feature vector. The input is two sets of 3×3 convolutional layers, and the output is the preliminary fused feature. The reverse reconstruction capability is retained through the inverse operation. ; in, , Representing two branches of the input features, It is the input feature branch direct mapping, It is the input feature branch The result after exponential scaling and offset transformation Inject a normalized identity vector into the conditional convolution function. With high-level feature vectors ; Each invertible residual block contains two sets of 3×3 convolutional layers, with a mapping function f(·): ； Reverse operation passed ; The input feature tensor representing the residual block; This represents the intermediate output features of the first 3×3 convolution, which are derived from the input features. The results after preliminary feature extraction; The output of the mapping function representing the residual block is a feature of the intermediate output. Residual characteristics after further processing; The output feature tensor representing the residual block is the fusion result of the input features and the residual features. It is an activation function, Conv 3×3 Refers to a 3×3 convolution operation; Step S5.3: High-frequency detail feature injection mechanism: High-frequency detail features are obtained by extracting pre-processed reference portrait images using a pre-trained ESRGAN network. Specifically, the pre-trained ESRGAN network extracts four levels of high-frequency features, ranging from 16×16 to 256×256, and the second-level feature of 64×64×64 is taken as the high-frequency detail feature. ; Low-frequency fundamental features are fused with high-frequency detailed features through reversible convolutional layers: ； in, This represents the final feature after fusing high-frequency details; it is the output resulting from combining low-frequency fundamental features with high-frequency detail features. This is the output of the encoder's first layer, i.e., the low-level basic features. Indicates high-frequency detail features Perform a 3×3 convolution operation.

8. The method and system for generating human portrait animation based on a reversible encoder and dynamic optimization according to claim 1, characterized in that, Step S6 specifically includes the following steps: Step S6.1: Forward noise process; According to the preset noise scheduling sequence sequentially for the initial latent variables Add noise: Positive noise addition: ; in, This represents the latent variable after adding noise at step t; This represents the noise scheduling coefficient, which increases with time step t. This represents standard Gaussian noise, which is an injected random disturbance; Step S6.2: Reverse noise process; Entropy features As an additional condition input to the denoising U-Net network; Inverse noise reduction: ; in, This represents the mean of the latent variables after denoising. This represents the cumulative noise figure, which is the total noise impact from the initial step to the t-th step. This represents the noise predicted by the denoised U-Net; Step S6.3: Cross-attention guidance; In each Transformer block, a normalized identity vector is used. For the query, the features extracted by the denoising U-Net network are key values, and the following calculations are performed: ； In this context, the normalized identity vector serves as the key K and value V, which are the features extracted by U-Net, containing spatial structure and detailed information; d represents the scaling factor, and Pos represents the positional encoding. Step S6.4: Noise prediction and update; based on The denoising network predicts noise, and the update formula is: ； in .

9. The method and system for generating human portrait animation based on a reversible encoder and dynamic optimization according to claim 1, characterized in that, Step S7 specifically includes the following steps: Step S7.1: Specifically, the time axis and spatial axis are processed separately through the axial time attention mechanism; Time axis processing: The dimension of the latent feature sequence is adjusted to RT×(H′W′C), and cross-frame dependencies are captured through MultiHeadAttn multi-head attention. ； ； Where R is the number of time step repetitions, T is the number of video frames, H′ and W′ are the feature map height and width, and C is the number of channels. Represents the original temporal potential features, It is a time-optimized feature sequence; Spatial axis processing: Apply standard spatial attention to the single-frame feature RH′×W′×C to enhance local details; Step S7.2: Implementation of optical flow constraint; Specifically, perform RAFT optical flow estimation: Input z t z t+1 Output optical flow field ; Step S7.3: Latent feature update; ； Where γ is the learning rate. To optimize the gradient for HJB, z t v represents the latent feature at the current time t. t The optical flow graph represents the current moment. Indicates characteristic distortion based on optical flow. Indicates the potential features of the next frame after the update; Loss function L HJB for: ； Among them, L face For facial identity feature matching loss, L pose L is the smoothness loss for the pose sequence. flow For optical flow consistency loss, To balance the weights; Facial identity feature matching loss L face as follows: ;in For facial feature extractors, f ref ArcFace features of the reference image, This means taking the expectation of the latent features over all times t, which is equivalent to calculating the average facial feature error; attitude sequence smoothness loss L pose as follows: ； in, For the generated pose keypoints in frame t, Let be the attitude velocity in frame t. This represents the average of the attitude change smoothing error over time step t from 1 to T−1; Optical flow uniformity loss as follows: ; in The optical flow field is estimated using RAFT; Warp is a characteristic warping operation based on optical flow. The diffusion process is considered as a continuous-time dynamic system with facial and posture constraints, with the state variable z(t) and the control variable noise. The Hamiltonian H is constructed as follows: ； Discretization: The Euler method is used to map the continuous time step t∈[0,1] to the diffusion step T, and the value function is calculated iteratively. ; Optimal control: Solving for optimal noise through backpropagation ,make Minimize, the formula is as follows: .

10. The method and system for generating human portrait animation based on a reversible encoder and dynamic optimization according to claim 1, characterized in that, Step S8 specifically involves: adjusting the weights of the denoising network: facial features have the highest priority, i.e. =1, the optical flow constraint weight is lower than the attitude, that is =0.5; Input to the denoising network: Pose features are extracted using DWPose and encoded into a latent vector, along with z... t Concatenated input denoising UNet network; optical flow field v t After downsampling to z t After resolution matching, the feature space is embedded through convolutional layers; During the training phase of the denoising network: the HJB optimization objective is added as a regularization term to the loss function of the diffusion model and jointly optimized with the original noise prediction loss; Joint optimization function: ; in, It is the original loss of the diffusion model. It is a coefficient that controls the weight of the regularization term; Inference phase: During the reverse diffusion process, z is iteratively adjusted using the HJB equation. t Ensure that the generated facial latent features at each step are close to the ArcFace reference features, and that the generated pose and action are consistent with the input sequence.

Images