Identity preserving video generation method and system based on slot timing identity encoding

Abstract

The application discloses an identity maintaining video generation method and system based on slot time sequence identity coding. A target person reference video is coded into a plurality of space-time reference feature markers, and a smaller number of slots are used for Sinkhorn iteration distribution and aggregation. After multiple iteration updates, time sequence identity condition feature markers are obtained. Image anchoring identity feature markers are obtained from the same person reference portrait, and two types of identity markers are weighted by using gate weights changing with the denoising stage. After splicing with text markers, the pre-trained text video model is input, and two-way global identity vectors are simultaneously fused to perform linear modulation on features, and the iteration denoising decoding output video is output. The application can reduce the identity drift and posture locking problems under large postures and motion changes, and improve the identity maintaining effect.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and generative artificial intelligence, and in particular to a method and system for generating identity-preserving videos based on slot temporal identity coding. Background Technology

[0002] Text-to-video technology can synthesize videos with corresponding content and motion based on natural language descriptions. In custom character videos, in addition to requiring the generated result to match the actions, scenes, clothing, or camera movements described in the text, it is also required that the characters in the generated video maintain the identity characteristics of the specified individuals across different frames, perspectives, and expressions.

[0003] Existing solutions that do not require retraining a complete generative model for each object typically extract facial recognition features, image-text features, or image-coded features from a reference portrait image and inject these extracted features as visual feature tokens or identity-enhanced text feature tokens into the video generation model. A single static image can provide relatively stable information about a frontal appearance, but it cannot fully reflect the identity-related features of a person under side view, expression changes, lighting changes, and continuous movement.

[0004] When the video to be generated contains significant head rotation, full-body movement, facial expression changes, object interactions, or partial occlusion, single-image identity conditions may lead to facial geometric distortion, identity feature drift over time, smoothing of fine-grained facial features, or the generated pose being excessively close to the pose in the reference image. The latter can cause the generation model to maintain the orientation or pose of the reference image while performing the actions required by the text, resulting in reference pose locking.

[0005] Short reference videos contain observations of the same person at consecutive moments and from multiple perspectives, supplementing the temporal and multi-view identity information missing in a single reference image. However, the identity information in reference videos is distributed across a large number of spatiotemporal features. Given the limited number of conditional feature labels that a video generation model can receive, directly inputting all spatiotemporal features increases computational cost; performing global average pooling on all spatiotemporal features may eliminate local details relevant to the person's identity. Adjacent video frames have high similarity, and using ordinary attention normalization may cause a small number of high-response regions to occupy most of the weights, resulting in multiple output feature labels repeatedly reading the same evidence, reducing the stability and complementarity of identity representations.

[0006] Therefore, there is a need for a technical solution that can stably extract complementary temporal identity information from short reference videos with a limited number of conditional feature labels and combine this information with stable reference portrait information to improve the preservation of human identity under changes in perspective, expression, lighting and motion, while maintaining the responsiveness of the generated results to text prompts. Summary of the Invention

[0007] The purpose of this invention is to solve the problems that the identity information contained in the reference video is difficult to represent compactly and stably under a limited feature labeling budget, and that identity drift and reference pose locking are prone to occur under large pose and motion changes in a single reference image. The invention provides an identity-preserving video generation method and system based on slot temporal identity coding, and provides corresponding training methods, electronic devices, storage media and computer program products.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A method for generating identity-preserving video based on slot-based temporal identity encoding includes the following steps:

[0010] S1. Obtain a reference video clip containing the target person, a reference portrait image containing the same target person, and a text prompt describing the content of the video to be generated.

[0011] S2. Perform latent space coding and spatiotemporal feature coding on the reference video segment to obtain a reference feature tag sequence composed of multiple spatiotemporal reference feature tags; perform at least two iterative readings on the reference feature tag sequence using a preset slot state with fewer than the number of reference feature tags, calculate the similarity matrix between the slot and the feature tag each time, obtain a coupling matrix that satisfies the preset two-sided edge distribution through Sinkhorn iteration, aggregate the reference feature tags and update the slot state according to the coupling matrix, and project the final slot state to obtain the temporal identity condition feature tag;

[0012] S3. Encode and project the identity features of the reference portrait image to obtain image anchor identity feature markers; encode the text prompt to obtain text feature markers; determine the gating weights according to the video denoising stage, and use the gating weights to weight the temporal identity conditional feature markers and the image anchor identity feature markers respectively, and concatenate the weighted temporal identity conditional feature markers, the weighted image anchor identity feature markers, and the text feature markers to form a conditional feature marker sequence;

[0013] S4. Obtain the corresponding global identity vectors from the identity branches of the reference video and the reference portrait image respectively, and use the gate weights to weighted sum to obtain the fused global identity vector. Perform feature-wise linear modulation on the intermediate features of the pre-trained text-to-video model.

[0014] S5. Using the conditional feature marker sequence and the fused global identity vector as generation conditions, perform iterative denoising on the video latent variables to decode and obtain an output video that matches the text prompt and retains the target person's identity and appearance features.

[0015] Preferably, in step S2, a variational autoencoder consistent with the pre-trained text-to-video model is used to encode the reference video segment into a reference video latent variable; the temporal difference latent variable at the first time step is set to zero, and the temporal difference latent variables at the remaining time steps are set to the difference between the reference video latent variable at the current time step and the reference video latent variable at the previous time step; the reference video latent variable and the temporal difference latent variable are concatenated along the channel dimension to obtain a combined latent variable; three-dimensional block embedding is performed on the combined latent variable to obtain an initial spatiotemporal feature label, and the feature is interacted through at least one spatiotemporal self-attention layer with three-dimensional rotation position encoding to obtain the reference feature label sequence.

[0016] Preferably, in step S2, the reference feature label sequence is used as both the key matrix and the value matrix, and the initial slot state is set as a learnable parameter shared across samples. During each iteration, the scaled dot product similarity matrix between the current slot state and the reference feature label is calculated, scaled by the temperature parameter, and used as the logarithmic kernel matrix. Under the constraint of uniform edge distribution, Sinkhorn iteration is performed to obtain a slot-feature label coupling matrix that satisfies the preset two-sided edge distribution. The reference feature labels are weighted and aggregated according to the coupling matrix, and the slot state is updated using a gated recurrent unit. After the last iteration, the temporal identity conditional feature label is obtained through layer normalization and linear mapping projection.

[0017] Preferably, the reference portrait image is selected from the reference video clip: multiple candidate frames are sampled, the target person's face region is detected and cropped, and the background of the candidate frames is removed; the candidate frame with the highest score is selected as the reference portrait image based on the joint quality index score of clarity, frontal pose degree and expression neutrality degree; the gating weight ranges from zero to one, and gradually decreases from the beginning to the end along the denoising execution order.

[0018] Preferably, in step S4, the video global identity vector and the image global identity vector are fused according to the gating weights; the fused global identity vector is used to perform feature-wise linear modulation on the network blocks of the second half of the pre-trained text-to-video model; the spatiotemporal feature encoding adopts two spatiotemporal self-attention layers, each layer is set with sixteen attention heads, six slots, three iterations, and two image anchor identity feature markers.

[0019] An identity-preserving video generation system based on slot-based temporal identity encoding includes an input acquisition module, a temporal encoding and slot reading module, an image identity encoding module, a conditional fusion module, a global identity modulation module, and a video generation module. The input acquisition module acquires reference video clips, reference human images, and text prompts. The temporal encoding and slot reading module encodes the reference video to obtain a reference feature tag sequence, and iteratively reads and projects slots with fewer slots than the reference feature tags to generate temporal identity conditional feature tags. The image identity encoding module encodes and projects the reference human image to obtain image-anchored identity feature tags. The conditional fusion module determines gating weights according to the denoising stage, weights the two types of identity feature tags, and concatenates them with text feature tags to form a conditional feature tag sequence. The global identity modulation module fuses two global identity vectors and modulates the intermediate features of the generation model. The video generation module iteratively denoises the latent variables based on conditions and decodes and outputs the video. The system also includes a processor and a memory, with the memory storing computer programs, and each module is implemented by the processor executing the programs.

[0020] Preferably, the temporal coding and slot reading module includes a latent space coding unit, a temporal difference calculation unit, a three-dimensional block embedding unit, and a spatiotemporal self-attention unit; the latent space coding unit uses a variational autoencoder consistent with the generation model to encode the reference video, the temporal difference calculation unit generates latent differential variables between adjacent frames, the three-dimensional block embedding unit divides the spliced ​​combined latent variables into blocks to obtain initial feature labels, and the spatiotemporal self-attention unit uses three-dimensional rotational position coding to complete spatiotemporal feature interaction.

[0021] Preferably, the timing encoding and slot reading module further includes a similarity calculation unit, a temperature scaling unit, a Sinkhorn iteration unit, a weighted aggregation unit, and a slot update unit; the similarity calculation unit calculates the scaled dot product similarity between the slot and the feature marker, and after temperature scaling, it is input into the Sinkhorn iteration unit to generate a coupling matrix; the weighted aggregation unit aggregates the feature markers according to the coupling matrix; and the slot update unit iteratively updates the slot state through a gated loop unit.

[0022] Preferably, the input acquisition module includes a portrait filtering unit, which samples candidate frames from the reference video, crops the face area and removes the background, and selects a reference portrait image based on a joint score of clarity, frontal pose degree and expression neutrality degree; in the conditional fusion module, the gating weight gradually decreases from the beginning to the end of the denoising stage.

[0023] Preferably, the global identity modulation module extracts global identity vectors from the video branch and the image branch respectively, and after weighted fusion according to the gate weight, performs feature-wise linear modulation on the second half of the network block of the pre-trained text-to-video model; the system is configured with six slots, three iterations of reading, two spatiotemporal self-attention layers, sixteen attention heads and two image anchor identity feature markers.

[0024] Compared with the prior art, the beneficial effects of the present invention are:

[0025] This invention uses a fixed number of slots to compress multiple spatiotemporal reference feature markers corresponding to a reference video into a small number of temporal identity feature markers. When the number of slots is less than the number of reference feature markers, it controls the number of conditional feature markers and the computational cost of subsequent cross-attention, while retaining evidence from multiple frames and multiple perspectives.

[0026] This invention uses Sinkhorn normalization with row and column edge constraints to allocate reference feature markers, reducing the situation where a small number of high-response reference feature markers occupy all slots, thus enabling better coverage of identity information read from multiple slots; and improves the stability of the correspondence between slots and feature markers in adjacent highly similar frames through recursive iterative updates of slot states.

[0027] This invention supplements single-frame identity features by jointly encoding latent variables of reference video and differential latent variables of adjacent frames, and explicitly utilizing short-term motion and appearance change information.

[0028] This invention utilizes dual-source gating conditions that vary with the denoising stage to provide stable appearance anchors using reference portrait images in the pre-denoising stage and provides temporal and multi-view identity evidence using reference videos in the post-denoising stage, thereby balancing identity preservation with the ability to change actions and postures.

[0029] This invention reduces the number of parameters that need to be updated and stored by keeping the basic parameters of the video generation backbone unchanged and training the identity condition module and low-rank adaptation parameters, while maintaining the compatibility between the identity condition module and the pre-trained generated representation. Attached Figure Description

[0030] Figure 1 This is an overall flowchart of the identity-preserving video generation method of the present invention;

[0031] Figure 2 This is a schematic diagram of the identity-preserving video generation system of the present invention;

[0032] Figure 3 This is a flowchart of the slot iterative reading process based on Sinckhorn normalization in this invention;

[0033] Figure 4 This is a flowchart of the model training method of the present invention. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments. The following embodiments are used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Unless otherwise specified, the first dimension of the tensor is the batch dimension. As used herein, "feature token" refers to a feature vector mapped to a preset feature dimension; "slot" refers to a learnable query state used to read a type of identity evidence from a reference feature token sequence; "identity preservation" means that the identity features of the person in each frame of the generated video correspond to the target person, and does not require the generated video to copy the actions, clothing, background or shooting angle of the reference video.

[0035] Example 1

[0036] This embodiment describes a method for generating identity-preserving video based on slot-based temporal identity encoding, executed by an electronic device with graphics computing capabilities. The overall process is as follows: Figure 1 As shown, it specifically includes five steps from S1 to S5, and each step is detailed below.

[0037] S1. Obtain input data:

[0038] Obtain reference video clips containing the target person, reference portrait images containing the same target person, and text prompts describing the content of the video to be generated.

[0039] Reference video clip acquisition and preprocessing:

[0040] Reference video clips are used to provide identification information of the target person at consecutive moments, with multiple head orientations or facial expressions. For the original video, the dominant person is detected in each frame, and the detection results of the dominant person in consecutive frames are merged into a temporally stable square cropping region. The frame is then cropped according to this region and the size of each frame is uniformly adjusted to obtain the reference video clip, while preserving the original background context.

[0041] Reference portrait image selection:

[0042] The reference portrait image is preferably a clear, near-frontal image with a relatively neutral expression and background removed, used to provide a stable identity anchor. The reference portrait image is selected from the reference video clip in the following manner:

[0043] Multiple candidate frames are sampled from a reference video clip; the target person is detected in each candidate frame and the face region of the target person is cropped, and the background is removed from the cropped candidate images; each candidate image is scored according to a joint quality index including at least sharpness, frontal pose degree, and expression neutrality, and the candidate image with the highest joint quality score is used as the reference portrait image. In one specific implementation, the size of the reference portrait image is 512×512 pixels.

[0044] Text prompt retrieval:

[0045] Text prompts are used to describe actions, scenes, clothing, camera movements, or other content in the video to be generated, and are used to constrain the semantic content and motion performance of the generated video.

[0046] S2. Reference video timing encoding and slot iterative reading:

[0047] The reference video segment is latent space encoded and spatiotemporal feature encoded to obtain a reference feature tag sequence composed of multiple spatiotemporal reference feature tags. The reference feature tag sequence is read at least twice using a preset slot state with fewer than the number of reference feature tags. Each time, the similarity matrix between the slot and the feature tag is calculated. The coupling matrix that satisfies the preset two-sided edge distribution is obtained through Sinkhorn iteration. The reference feature tags are aggregated according to the coupling matrix and the slot state is updated. The final slot state is projected to obtain the temporal identity condition feature tag.

[0048] Latent space coding:

[0049] Employing a variational autoencoder consistent with pre-trained text-to-video generation models ncoder, VA The reference video segment is encoded as a reference video latent variable.

[0050] Let the input reference video be:

[0051]

[0052] in, For batch size, For reference video frame rate, and These are the frame height and frame width, respectively.

[0053] After encoding by a variational autoencoder, the latent variables of the reference video are obtained:

[0054]

[0055] in, For the number of latent variable channels, and The spatial dimensions of the latent space. Indicates the latent variable of the reference video in the th... A slice of latent variables at each time point.

[0056] Construction of latent variables for time series differences:

[0057] To explicitly represent short-term motion and appearance change information, a system is constructed using... The set of time-series difference latent variables at each moment Set the temporal difference latent variable corresponding to the first time step to zero, and set the temporal difference latent variable corresponding to each of the remaining time steps to the difference between the reference video latent variable at the current time step and the reference video latent variable at the previous time step, i.e., the first... Temporal difference latent variables at each time point satisfy:

[0058]

[0059] By concatenating the latent variables of the reference video and the temporal difference latent variables along the channel dimension, we obtain the combined latent variables:

[0060]

[0061] Among the symbols This indicates a splicing operation along the channel dimension.

[0062] 3D Block Embedding and Spatiotemporal Feature Encoding:

[0063] The kernel size for the combined latent variables is [value missing]. The core dimensions are respectively and Step-wise 3D block embedding yields initial spatiotemporal feature labels. The total number of reference feature labels... The calculation formula is:

[0064]

[0065] In the formula, , and These represent the time dimension, height, and width of the combined latent variables, respectively.

[0066] The initial spatiotemporal feature labels are then processed through at least one spatiotemporal self-attention layer employing 3D rotational position encoding. This spatiotemporal self-attention layer uses 3D rotational position encoding to distinguish the position of the feature labels in the time, height, and width directions, and can utilize a memory-efficient attention kernel. After spatiotemporal feature mixing, the reference feature label sequence is obtained:

[0067]

[0068] in For batch size, For reference feature label number, For feature dimensions.

[0069] Iterative reading of slots:

[0070] use Each slot state performs at least two iterative reads on the reference feature label sequence, and the number of slots... The overall process for slot reading is as follows: Figure 3 As shown, the reference feature label sequence is simultaneously used as the key matrix. Sum matrix The initial slot state is set to a learnable parameter shared across samples and replicated in the batch dimension. The specific process of each iteration of reading is as follows:

[0071] Similarity calculation:

[0072] Calculate the scaled dot product similarity matrix between the current slot state and the reference feature label:

[0073]

[0074] in, For the first Slot status after the next iteration. This is the transpose of the key matrix. For feature dimension, This is a scaling factor used to stabilize the range of inner product values.

[0075] Temperature scaling:

[0076] Using positive temperature parameters Scaling the similarity matrix yields a temperature-scaled similarity matrix:

[0077]

[0078] in This is a positive temperature parameter. During the training phase, the temperature parameter can dynamically decrease with the number of global training steps, while during the inference phase, it takes a fixed value after training convergence.

[0079] Sinkhorn Iterative Normalization:

[0080] Based on the similarity matrix after temperature scaling, the slot-feature label coupling matrix that satisfies the preset two-sided edge distribution is obtained by performing entropy regularization optimal transmission normalization through Sinkhorn iteration using preset slot side edge distribution and preset feature label side edge distribution.

[0081] Specifically, the temperature-scaled similarity matrix is ​​used as the logarithmic kernel. Target edge on the slot side Feature marker side target edge ,in and They are lengths and The vector is all 1s, meaning that the target edge distribution is uniform on both sides.

[0082] Initialize dual variables , The following updates are performed alternately in the logarithmic field:

[0083]

[0084]

[0085] in The logsumexp operation is used to ensure numerical stability.

[0086] After updating the dual variable a predetermined number of times, calculate the coupling matrix in the logarithmic field:

[0087]

[0088] After taking the exponent, we obtain the slot-feature tag coupling matrix:

[0089]

[0090] The coupling matrix is ​​further normalized by rows and columns to correct the numerical deviations in edge distribution caused by a finite number of iterations. By using bilateral edge constraints, it is ensured that the total weight assigned to each slot is consistent and the total weight read by each reference feature tag is consistent, avoiding the repeated reading of a small number of high-response features and improving the complementarity of identity information in multiple slots.

[0091] Slot-level weighted aggregation:

[0092] Based on the coupling matrix, the value matrix of the reference feature markers is weighted and aggregated to obtain the slot-level aggregated features:

[0093]

[0094] in Indicates batch size. Indicates the number of slots. Indicates the feature dimension.

[0095] Slot status recursive update: The slot status is recursively updated using a gated recurrent unit (GRU).

[0096]

[0097] in This represents the single-step update function of the gated loop unit. Through recursive updates, the slot can adjust the direction of the next query based on the content already read, extracting complementary identity information from adjacent highly similar frames.

[0098] After repeated iterations, a readout head consisting of a layer normalization layer and a linear mapping layer is used to project the final slot state to obtain a temporal identity conditional feature label sequence:

[0099]

[0100] in This indicates a projection operation. For the first Slot status after the next iteration. This represents the total number of iterations.

[0101] In a specific parameter configuration, the number of slots The number of iterations is 6, the number of readings is 3, and the spatiotemporal feature encoding uses 2 spatiotemporal self-attention layers with 16 attention heads in each layer.

[0102] S3. Fusion of dual-source identity features and textual features:

[0103] The reference portrait image is encoded and projected to obtain image-anchored identity feature markers; the text prompt is encoded to obtain text feature markers; a gating weight is determined according to the video denoising stage, and the temporal identity conditional feature markers and the image-anchored identity feature markers are weighted using the gating weights respectively, and the weighted temporal identity conditional feature markers, the weighted image-anchored identity feature markers, and the text feature markers are concatenated to form a conditional feature marker sequence.

[0104] Image-anchored identity feature tag generation:

[0105] A near-frontal reference portrait image with the background removed is input into a variational autoencoder and a projection layer. The resulting image features are then projected onto the video generation model. In the 3D conditional feature space, image anchor identity feature labels are obtained:

[0106]

[0107] in The number of identity feature tags anchored to the image. In a specific configuration, the number of identity feature tags anchored to the image is 2.

[0108] The text prompt is processed by a text encoder to obtain text feature tags. .

[0109] Gating weighted:

[0110] Let the gate weight be... ,and The gating weights gradually decrease from the beginning to the end of the actual denoising execution sequence: a larger weight is set in the beginning of the denoising process. This allows the generation process to prioritize the use of clear reference portrait identity anchors that are less affected by the scene; as denoising progresses, the number of anchors decreases. This allows the generation process to gradually increase the use of multi-view and temporal identity information in the reference video.

[0111] Calculate the weighted identity feature labels for the two categories separately:

[0112]

[0113]

[0114] Conditional sequence concatenation:

[0115] The weighted image anchored identity feature tags, temporal identity conditional feature tags, and text feature tags are concatenated along the sequence dimension to form a conditional prefix feature tag sequence:

[0116]

[0117] Among the symbols This represents a concatenation operation along the sequence dimension. The conditional feature-labeled sequence is used as the conditional sequence for cross-attention, enabling the video generation backbone to read image identity, temporal identity, and textual semantic information at each denoising step.

[0118] S4. Global Identity Vector Fusion and Feature Modulation:

[0119] The corresponding global identity vectors are obtained from the identity branches of the reference video and the reference portrait image respectively. The fused global identity vector is obtained by weighted summation using the gate weights. Feature-wise linear modulation (FiLM) is then performed on the intermediate features of the pre-trained text-to-video model.

[0120] Global identity vector fusion:

[0121] Calculate the global identity vector of the video from the reference video identity branch and the reference human image identity branch respectively. and image global identity vector The fusion process employs gating weights consistent with the feature label level.

[0122]

[0123] in To fuse global identity vectors.

[0124] Characteristic-wise linear modulation:

[0125] The fused global identity vector is converted into channel-wise scaling and offset parameters, and then used to perform feature-wise linear modulation on the intermediate features of the pre-trained text-to-video model. In one specific configuration, feature-wise linear modulation is performed on the last 50% of the network blocks of the pre-trained text-to-video model using the fused global identity vector.

[0126] Feature label prefixes provide local identity information that can be selected by cross-attention, while global vectors provide continuous global identity priors. Together, they strengthen identity constraints.

[0127] S5. Conditional Denoising and Video Decoding:

[0128] Using the conditional feature tag sequence and the fused global identity vector as generation conditions, iterative denoising is performed on the video latent variables to decode and obtain an output video that matches the text prompt and retains the target person's identity and appearance features.

[0129] The pre-trained text generation video model uses a latent space diffusion transformer or a multimodal diffusion transformer with a conditional cross-attention interface as its backbone. At the start of inference, the video noise latent variables conforming to a standard Gaussian distribution are initialized according to the target number of frames and spatial size.

[0130] In each denoising step, the gating weights are calculated based on the current denoising stage. The system constructs a conditional feature label sequence and fuses a global identity vector, then generates a backbone prediction velocity or stream from the video to update the video latent variables. After a preset number of denoising operations, the decoder of the variational autoencoder decodes the resulting latent variables into an output video in pixel space.

[0131] The scene, actions, clothing, and camera angles of the output video are constrained by text cues, while the identities of the characters are constrained by both image-anchored identity feature markers and temporal identity feature markers. The reference video is only used to extract identity information; the output video is not required to replicate the background and actions of the reference video.

[0132] Model training methods:

[0133] The model corresponding to the identity-preserving video generation method described in this embodiment is trained in the following manner, the training process being as follows: Figure 4 As shown.

[0134] Training data construction:

[0135] The process involves acquiring training videos and corresponding text descriptions, and constructing target video clips, reference video clips, and reference portrait images containing the same target person from the training videos. In one specific embodiment, each target video clip contains 65 frames.

[0136] The target video segment is selected from consecutive frames of a specified duration in the source video. Person detection is performed on these frames, and a temporally stable square cropping region is generated. The reference video segment is preferentially selected from another time interval of the same source video, featuring the same person as the target video segment but without temporal overlap, to guide the model to learn identity transfer capabilities rather than reconstructing from the reference segment. Additionally, clear, near-frontal candidate frames with neutral expressions are selected from videos featuring the same person, and after face cropping and background removal, a reference portrait image is obtained. The target video segment, reference video segment, and reference portrait image all correspond to the same person.

[0137] Training data and target reference data used in the inference phase should be used only with appropriate authorization and in compliance with applicable data and privacy protection requirements.

[0138] Parameter freezing strategy:

[0139] During training, the basic parameters of the variational autoencoder, text encoder, and pre-trained text-to-video generation model are kept unchanged. The parameters of the identity conditional modules used to perform reference video spatiotemporal feature encoding, slot iterative reading, temporal identity conditional feature label projection, image anchor identity feature label projection, global identity vector readout, and feature-wise linear modulation are updated. The low-rank adaptation parameters set in the key and value projections of the cross-attention layers of the pre-trained text-to-video generation model are also updated. In one specific embodiment, the video generation backbone adopts the Wan-2.1 T2V-14B latent space text-to-video generation model.

[0140] Noise sampling and interpolation latent variable construction:

[0141] The target video segment is encoded into a target video latent variable, and noise is added to the target video latent variable to obtain a noisy video latent variable. Specifically, the target video latent variable is denoted as... The sampling follows a standard Gaussian distribution. noise and obey interval Uniformly distributed time parameters ,make , The interpolation latent variable and the target velocity are calculated using the following formula:

[0142]

[0143]

[0144] in Represents the identity matrix. For noisy interpolation latent variables, This corresponds to the actual speed.

[0145] Condition building and model prediction:

[0146] The gating weights are determined based on the denoising stage corresponding to the added noise, with the gating weights for the preceding denoising stage being greater than those for the following denoising stage. Because... A value close to 1 corresponds to higher noise levels and the initial stage of the actual denoising execution order. A value close to 0 corresponds to lower noise and the later stage of denoising; during training, this can be adjusted according to... The gating weights are determined for the corresponding denoising stage.

[0147] Temporal identity conditional feature labels and image-anchored identity feature labels are weighted separately using gating weights. The weighted two types of identity feature labels are then concatenated with text feature labels obtained from text descriptions to obtain a conditional feature label sequence. The video global identity vector and the image global identity vector are fused using gating weights to obtain a fused global identity vector.

[0148] The conditional feature label sequence, the fused global identity vector, the temporal parameters, and the latent variables of the noisy video are input into a pre-trained text-to-video model to obtain the predicted speed. ,in This represents the parameters to be trained.

[0149] Loss function and parameter updates:

[0150] Training uses flow matching loss, and the loss function is:

[0151]

[0152] in Represents the mathematical expectation. Represents the L2 norm, Label the sequence for conditional features. To fuse global identity vectors.

[0153] The identity condition module and low-rank adaptation parameters are updated through backpropagation. During training, feature labels in the conditional feature label sequence are randomly dropped at a per-feature label drop rate of 0.05 to reduce the risk of over-conditioning and improve the model's generalization ability.

[0154] Temperature parameter decay strategy:

[0155] During training, the temperature parameters read from the slot change linearly from the initial temperature to the final temperature according to the following formula:

[0156]

[0157] in, This represents the total number of training steps. This indicates the number of training steps used to represent temperature changes. Indicates the initial temperature. Indicates the termination temperature, and .

[0158] This temperature change method allows for the acquisition of relatively dispersed slot-feature label correspondences in the early stages of training, and relatively concentrated correspondences in the later stages of training.

[0159] Example 2

[0160] This embodiment provides an identity-preserving video generation system based on slot-based temporal identity encoding, the overall structure of which is as follows: Figure 2 As shown, this system is used to implement the identity-preserving video generation method described in Embodiment 1. The system includes an input acquisition module, a timing encoding and slot reading module, an image identity encoding module, a conditional fusion module, a global identity modulation module, and a video generation module; the system also includes a processor and a memory, the memory storing a computer program, and each module is implemented by the processor executing the computer program.

[0161] Input Acquisition Module:

[0162] The input acquisition module is used to read or receive reference video clips, reference portrait images and text prompts, and to perform data preprocessing and quality screening to provide standardized input for subsequent modules.

[0163] The module has a built-in video preprocessing unit that can perform person detection, tracking and spatiotemporal alignment on the original video: it detects the dominant person in each frame, merges the detection results of consecutive frames into a time-stable square cropping region, crops by region and unifies the frame size, and generates a reference video segment with the main person in the center, while preserving the original background context.

[0164] The module includes a portrait filtering unit that automatically selects high-quality reference portraits from reference videos: it uniformly samples multiple candidate frames from the reference video, detects and crops the face region, performs portrait segmentation to remove the background, and constructs a joint quality scoring function based on sharpness, frontal pose degree, and expression neutrality, selecting the candidate frame with the highest score as the reference portrait image. In a typical configuration, the output reference portrait image size is 512×512 pixels.

[0165] The module also supports the preprocessing and standardization of text prompts, converting the natural language descriptions input by the user into a format that the text encoder can accept, and outputting it to the subsequent text encoding unit.

[0166] Timing encoding and slot reading module:

[0167] The temporal coding and slot reading module is the core feature extraction unit of the system. It is responsible for converting reference videos into compact temporal identity conditional representations, enabling the extraction of complementary and stable identity information from a large number of spatiotemporal features. The module is internally divided into a latent space coding unit, a temporal difference calculation unit, a 3D block embedding unit, a spatiotemporal self-attention unit, a similarity calculation unit, a temperature scaling unit, a Sinkhorn iteration unit, a weighted aggregation unit, a slot update unit, and a projection readout unit.

[0168] The latent space coding unit uses a variational autoencoder consistent with the generation backbone to map the reference video from the pixel space to a unified latent space, ensuring the compatibility of feature distribution with the generation model.

[0169] The temporal difference calculation unit calculates the difference of latent variables between adjacent frames, explicitly encodes short-term motion and appearance change information, supplements the dynamic dimension of static appearance features, and improves the ability to represent identity under large pose changes.

[0170] The three-dimensional block embedding unit performs block and linear mapping on the spliced ​​combined latent variables along the time and space dimensions to generate an initial spatiotemporal feature label sequence, and converts the three-dimensional latent features into a sequential feature representation.

[0171] The spatiotemporal self-attention unit consists of multiple spatiotemporal self-attention layers with 3D rotational position encoding, enabling global feature interaction across temporal and spatial dimensions and outputting a reference feature label sequence with spatiotemporal context awareness. In a typical configuration, this unit contains two spatiotemporal self-attention layers, each with 16 attention heads.

[0172] The similarity calculation unit uses the reference feature label as the key and the previous round slot status as the query to calculate the scaled dot product similarity matrix, quantifying the degree of association between each slot and each reference feature label.

[0173] The temperature scaling unit adjusts the sharpness of the similarity distribution through configurable temperature parameters. During the training phase, it supports linear decay of temperature with the number of training steps, realizing the slot allocation learning process from dispersed to concentrated.

[0174] The Sinkhorn iterative unit performs entropy regularization optimal transmission calculation under the constraint of uniform edge on both sides, and outputs a balanced slot-feature tag coupling matrix. This mechanism avoids a few high-response features from dominating all slots and ensures that the identity information read from multiple slots has coverage and complementarity.

[0175] The weighted aggregation unit performs a weighted summation on the reference feature markers based on the coupling matrix to obtain the aggregated identity features corresponding to each slot, thereby compressing a large number of reference markers into a small number of slot features.

[0176] The slot update unit adopts a gated loop unit structure, recursively fusing historical slot states with current aggregated features. Through multiple iterations, it optimizes the slot query direction, improving the stability of identity representation. In a typical configuration, the module has 6 slots and performs 3 iterations of reading.

[0177] The projection readout unit performs layer normalization and linear mapping on the final slot state, and outputs temporal identity condition feature labels whose dimensions match the condition space of the generated model.

[0178] Image identity encoding module:

[0179] The image identity encoding module is responsible for extracting stable identity representations from reference portrait images, and includes an image latent feature extraction unit, a feature label projection unit, and a global identity readout unit.

[0180] The image latent feature extraction unit uses a variational autoencoder consistent with the temporal coding branch to extract latent space features of portrait images, ensuring the uniformity of the feature space.

[0181] The feature label projection unit maps latent image features to a fixed number of image anchor identity feature labels through a dedicated linear projection layer. These labels serve as identity anchor constraint generation processes, providing a reliable identity benchmark in the early stages of denoising. In a typical configuration, two image anchor identity feature labels are output.

[0182] The global identity readout unit outputs the global identity vector of the image through global pooling and mapping layers, which is used for subsequent global feature modulation and provides global identity prior.

[0183] Conditional fusion module:

[0184] The conditional fusion module enables the weighted fusion and concatenation of dual-source identity features and text features, providing a unified conditional feature label sequence for the generative model.

[0185] The module has a built-in gating weight generation unit that outputs monotonically decreasing gating weights based on the proportion of the current denoising steps to the total steps: the weights of the image branches in the front stage of denoising are higher to prioritize the accuracy of identity and correct the instability of temporal features; the weights of the temporal branches in the back stage of denoising are increased to supplement multi-view dynamic identity information and alleviate the reference pose locking problem.

[0186] The module performs weighted operations on the image anchor identity feature tags and the temporal identity conditional feature tags respectively, and then concatenates them with the text feature tags along the sequence dimension to form a unified conditional feature tag sequence. This sequence is input into the cross-attention layer of the video generation module as the conditional input for each denoising step. During the training phase, the module supports random discarding of identity feature tags in the conditional feature tag sequence tag by tag, with a discard rate of 0.05, to reduce the risk of over-conditioning and improve the model's generalization ability.

[0187] Global Identity Modulation Module:

[0188] The global identity modulation module is responsible for fusing two global identity vectors and injecting the global identity prior into the generated backbone through feature-wise linear modulation, complementing the label-level local conditions. The module receives the video global identity vector and the image global identity vector respectively, and performs weighted fusion using the same gating weights as the conditional fusion module to obtain a fused global identity vector, ensuring the consistency of the weights of the two conditions. Subsequently, the module converts the fused global identity vector into channel-wise scaling and offset parameters, and performs linear modulation on intermediate features at a specified level in the generated backbone. In a typical configuration, the modulation only applies to the last 50% of the backbone network blocks, strengthening identity constraints while minimizing interference with basic generation capabilities, thus balancing identity preservation and video generation quality.

[0189] Video generation module:

[0190] The video generation module uses a pre-trained text-to-video model as its backbone. It receives a sequence of conditional feature tags and a fused global identity vector, performs iterative latent variable denoising and video decoding, and outputs the final identity-preserving video. The module includes a latent variable initialization unit, an iterative denoising unit, and a video decoding unit.

[0191] The latent variable initialization unit generates initial noise latent variables that conform to a standard Gaussian distribution based on the frame number and spatial resolution of the target video, which serve as the starting point for the denoising process.

[0192] The iterative denoising unit updates the latent variables step by step according to the preset denoising schedule. At each step, the backbone network is generated, and the denoising direction is predicted by combining the current condition information, and the latent variables are iteratively optimized.

[0193] The video decoding unit employs the same variational autoencoder decoder as the encoding end, mapping the final latent variables back to pixel space to output the final identity-preserving video. The semantic content, motion, and scene of the output video are constrained by text cues, while the character's identity is jointly constrained by dual-source identity conditions.

[0194] This system achieves end-to-end extraction, fusion, and injection of dual-source identity conditions through modular design. It can realize high-quality character video customization without retraining the basic generative model, effectively reducing identity drift and pose locking problems in large poses, large expressions, and complex motion scenes, while maintaining a high degree of compliance with text prompts.

[0195] Experimental verification:

[0196] On a set of identity-preserving text-generated video test data separated from the training data, each test sample is provided with a short reference video, a clear frontal reference frame obtained from the video, and a text prompt requiring the character to make a significant movement or change in perspective. Under a unified generation and evaluation process, the following metrics are used: Face Similarity is the frame-by-frame average of the identity embedding similarity between each detectable face in the generated video and the reference face; Naturalness is the average score given by the multimodal evaluation model based on uniformly sampled frames from the video and according to a fixed subscale of 1 to 5; Text Compliance is the cross-modal feature similarity between the text and the generated video. Higher values ​​for each metric indicate better evaluation.

[0197] When using the complete technical solution, the face similarity is 0.729, the naturalness is 3.917, and the text compliance is 0.634. Under the same evaluation process, the face similarity of the seven comparison schemes ranges from 0.298 to 0.699, the naturalness ranges from 2.897 to 3.912, and the text compliance ranges from 0.560 to 0.643. These results indicate that, in the test set and evaluation process used in this embodiment, the method improves face similarity and naturalness while maintaining text compliance within the comparison range.

[0198] To verify the effectiveness of each technical component, the ablation model was retrained using the same training data, optimized settings, low-rank adaptation strategy, and conditional injection interface. The results are as follows:

[0199] The full solution has facial similarity, naturalness, and text compliance of 0.729, 3.917, and 0.634, respectively; after removing the temporal difference branch, the corresponding values ​​are 0.711, 3.910, and 0.638, respectively.

[0200] After replacing the Sinkhorn normalized slot read with the normalized slot read, the face similarity, naturalness, and text compliance were 0.517, 3.225, and 0.616, respectively.

[0201] After retaining the Sinkhorn assignment but replacing multiple iterative reads with a single read, the face similarity, naturalness, and text compliance were 0.405, 2.934, and 0.596, respectively.

[0202] The ablation results above indicate that, under this experimental setup, the temporal difference branch provides additional temporal variation information, and slot routing with row and column edge constraints, along with multiple slot state updates, significantly improves identity preservation and generation naturalness. Qualitative observations also show that, under conditions of large viewpoint changes, rapid movement, exaggerated expressions, object interactions, and partial occlusion, the complete scheme reduces identity drift, facial shape changes, and reference pose locking.

[0203] In another peer review experiment, 15 identities were selected from the test data, and videos were generated for each method with matching text prompts. Over 50 participants rated facial similarity, visual quality, and text consistency on anonymized and randomly ordered results using a scale of 1 to 5. The complete scheme achieved average opinion scores of 3.837, 3.890, and 3.639 on the three metrics, respectively. These results only apply to the stated sample and peer review process and do not represent the same values ​​across all data distributions.

[0204] The present invention also provides an electronic device, including one or more central processing units, graphics processing units, neural network processors or other computing processors, as well as a memory, a storage device and a communication interface. When a computer program stored in the memory is executed by the processor, it implements the identity-preserving video generation method or model training method described in this specification.

[0205] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned identity-preserving video generation method or model training method. The computer-readable storage medium may be a magnetic medium, an optical medium, a semiconductor memory, or other non-transitory storage medium.

[0206] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the aforementioned identity-preserving video generation method or model training method.

[0207] This embodiment is more likely to obtain stable identity conditions when the subject in the reference video is clear, the person's identity is identifiable, and the video length is within the model's training range. The identity preservation effect may decrease when there is severe occlusion, low-quality reference video, or the generation duration significantly exceeds the training distribution.

[0208] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A method for generating identity-preserving video based on slot-based temporal identity coding, characterized in that, Includes the following steps: S1. Obtain a reference video clip containing the target person, a reference portrait image containing the same target person, and a text prompt describing the content of the video to be generated. S2. Perform latent space coding and spatiotemporal feature coding on the reference video segment to obtain a reference feature tag sequence composed of multiple spatiotemporal reference feature tags; The reference feature mark sequence is read at least twice using a preset slot state with fewer slots than the reference feature mark. Each time, the similarity matrix between the slot and the feature mark is calculated. The coupling matrix that satisfies the preset two-sided edge distribution is obtained through Sinkhorn iteration. The reference feature marks are aggregated and the slot state is updated according to the coupling matrix. The final slot state is projected to obtain the temporal identity condition feature mark. S3. Encode and project the identity features of the reference portrait image to obtain the image anchored identity feature marker; The text prompt is encoded to obtain text feature tags; The gating weights are determined based on the video denoising stage. The gating weights are used to weight the temporal identity conditional feature markers and the image anchor identity feature markers respectively. The weighted temporal identity conditional feature markers, the weighted image anchor identity feature markers, and the text feature markers are concatenated to form a conditional feature marker sequence. S4. Obtain the corresponding global identity vectors from the identity branches of the reference video and the reference portrait image respectively, and use the gate weights to weighted sum to obtain the fused global identity vector. Perform feature-wise linear modulation on the intermediate features of the pre-trained text-to-video model. S5. Using the conditional feature marker sequence and the fused global identity vector as generation conditions, perform iterative denoising on the video latent variables to decode and obtain an output video that matches the text prompt and retains the target person's identity and appearance features.

2. The identity-preserving video generation method according to claim 1, characterized in that, In step S2, a variational autoencoder consistent with the pre-trained text-to-video model is used to encode the reference video segment into a reference video latent variable; the temporal difference latent variable at the first time step is set to zero, and the temporal difference latent variables at the remaining time steps are set to the difference between the reference video latent variable at the current time step and the reference video latent variable at the previous time step; the reference video latent variable and the temporal difference latent variable are concatenated along the channel dimension to obtain a combined latent variable; three-dimensional block embedding is performed on the combined latent variable to obtain an initial spatiotemporal feature label, and the feature is interacted through at least one spatiotemporal self-attention layer with three-dimensional rotation position encoding to obtain the reference feature label sequence.

3. The identity-preserving video generation method according to claim 1, characterized in that, In step S2, the reference feature label sequence is used as both the key matrix and the value matrix, and the initial slot state is set as a learnable parameter shared across samples. During each iteration, the scaled dot product similarity matrix between the current slot state and the reference feature label is calculated, scaled by the temperature parameter, and used as the logarithmic kernel matrix. Under the constraint of uniform edge distribution, Sinkhorn iteration is performed to obtain the slot-feature label coupling matrix that satisfies the preset two-sided edge distribution. The reference feature labels are weighted and aggregated according to the coupling matrix, and the slot state is updated using a gated recurrent unit. After the final iteration, the temporal identity conditional feature labels are obtained through layer normalization and linear mapping projection.

4. The identity-preserving video generation method according to claim 1, characterized in that, The reference portrait image is selected from the reference video clip: multiple candidate frames are sampled, the target person's face region is detected and cropped, and the background of the candidate frames is removed; the candidate frame with the highest score is selected as the reference portrait image based on the joint quality index score of sharpness, frontal pose degree and expression neutrality; the gating weight ranges from zero to one and gradually decreases from the beginning to the end along the denoising execution order.

5. The identity-preserving video generation method according to claim 4, characterized in that, In step S4, the video global identity vector and the image global identity vector are fused according to the gating weights; the fused global identity vector is used to perform feature-wise linear modulation on the network blocks of the second half of the pre-trained text-to-video model. The spatiotemporal feature encoding adopts a two-layer spatiotemporal self-attention layer, with sixteen attention heads in each layer, six slots, three iterations, and two image anchor identity feature tags.

6. A video generation system for maintaining identity based on slot-based temporal identity coding, characterized in that, It includes an input acquisition module, a temporal coding and slot reading module, an image identity coding module, a conditional fusion module, a global identity modulation module, and a video generation module; the input acquisition module acquires reference video clips, reference portrait images, and text prompts; the temporal coding and slot reading module encodes the reference video to obtain a reference feature label sequence, and uses slots with fewer than the number of reference feature labels to iteratively read and project to generate temporal identity conditional feature labels; The image identity encoding module projects the encoding onto the reference human image to obtain the image anchor identity feature marker; The conditional fusion module determines the gating weights according to the denoising stage, and after weighting the two types of identity feature labels, it concatenates them with the text feature labels to form a conditional feature label sequence; the global identity modulation module fuses the two global identity vectors and modulates them to generate intermediate features of the model. The video generation module denoises and decodes the output video based on conditional iteration of latent variables; the system also includes a processor and a memory, with the memory storing computer programs and each module being implemented by the processor executing the programs.

7. The identity-preserving video generation system according to claim 6, characterized in that, The temporal coding and slot reading module includes a latent space coding unit, a temporal difference calculation unit, a three-dimensional block embedding unit, and a spatiotemporal self-attention unit. The latent space coding unit uses a variational autoencoder consistent with the generation model to encode the reference video. The temporal difference calculation unit generates latent differential variables between adjacent frames. The three-dimensional block embedding unit divides the spliced ​​combined latent variables into blocks to obtain initial feature labels. The spatiotemporal self-attention unit uses three-dimensional rotational position coding to complete spatiotemporal feature interaction.

8. The identity-preserving video generation system according to claim 6, characterized in that, The timing encoding and slot reading module also includes a similarity calculation unit, a temperature scaling unit, a Sinkhorn iteration unit, a weighted aggregation unit, and a slot update unit. The similarity calculation unit calculates the scaled dot product similarity between the slot and the feature marker, and after temperature scaling, it is input into the Sinkhorn iteration unit to generate a coupling matrix. The weighted aggregation unit aggregates the feature markers according to the coupling matrix, and the slot update unit iteratively updates the slot state through a gated loop unit.

9. The identity-preserving video generation system according to claim 6, characterized in that, The input acquisition module includes a portrait filtering unit, which samples candidate frames from the reference video, crops the face area and removes the background, and selects a reference portrait image based on a joint score of clarity, frontal pose degree and expression neutrality; in the conditional fusion module, the gating weight gradually decreases from the beginning to the end of the denoising stage.

10. The identity-preserving video generation system according to claim 6, characterized in that, The global identity modulation module extracts global identity vectors from the video branch and the image branch respectively, and after weighted fusion according to the gate weight, performs feature-wise linear modulation on the second half of the network block of the pre-trained text generation video model. The system is configured with six slots, three iterations of reading, two spatiotemporal self-attention layers, sixteen attention heads, and two image anchoring identity feature markers.

Images