Personalized video generation method and system for identity preservation

Abstract

The invention provides a personalized video generation method and system for identity preservation, and relates to the technical field of personalized video generation, and the method comprises the steps: obtaining a reference image and a text prompt; generating a joint latent marker by using the reference image and the text prompt, and segmenting and recombining the reference image to obtain a face marker sequence; a preset local router is adopted to fuse the joint latent marker and the face marker sequence, and an enhanced latent marker is generated; de-noising the enhanced latent marker to obtain a preliminary de-noised latent marker; inputting the preliminary denoising latent marker into a time autoregression module to obtain a time enhanced latent marker; and inputting the time-enhanced latent markers into a decoder to generate a personalized video. According to the method, the local router and the time autoregression module are introduced, so that the identity retention capability and the time-space consistency in the video generation process are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of personalized video generation, and in particular to a personalized video generation method and system for identity preservation. BACKGROUND

[0002] In the text-to-video generation (T2V) task, personalized generation as an important research direction in recent years aims to accurately preserve the fine-grained facial features and behavior style of the target identity in the video synthesis process. Among them, identity-preserving text-to-video generation (IPT2V) is widely used in virtual digital human generation, AI anchor driving, immersive interaction and other scenarios, and its generation quality and identity consistency directly determine the usability and user acceptance of the synthesized video.

[0003] At present, the mainstream research methods of the IPT2V task mainly include two categories. The first category is personalized video generation methods based on diffusion transformers (DiTs). This kind of method models the spatio-temporal features of the video through multi-layer global attention mechanism, has good generation quality and semantic consistency, and representative works such as ConsisID, Magic Mirror and Concat-ID. This kind of method usually uses frequency domain decomposition, feature adaptation or cross-video pairing training strategies to improve the identity preservation ability, but due to its architecture essentially relying on global attention module, the fine-grained features of local facial structure (such as eyes, nose, mouth, etc.) are not modeled well, and the problems of facial detail loss, identity ambiguity and inter-frame inconsistency are prone to occur in the generation process; the second category is hybrid models that fuse autoregressive and diffusion mechanisms, such as ACDiT and CausVid. This kind of method tries to combine the time continuity of autoregressive modeling and the high-quality generation advantage of diffusion model to improve the spatio-temporal consistency of the generated sequence. However, hybrid models mostly do not explicitly construct long-range dependency mechanism in time modeling, and it is difficult to fully capture the identity feature evolution across frames, and the generated results still have the phenomena of flicker and distortion.

[0004] In summary, the existing technology still faces the bottleneck of identity information loss and spatio-temporal inconsistency in the identity-preserving personalized video generation task. SUMMARY

[0005] The present application proposes a personalized video generation method and system for identity preservation, aiming to realize high-fidelity personalized video generation with stable cross-frame identity information and spatio-temporal consistency.

[0006] In order to achieve the above technical effects, the technical scheme of the present application is as follows: A personalized video generation method for identity preservation, comprising the following steps: S1: obtaining a reference image and a text prompt; S2: generating a joint latent mark using the reference image and the text prompt, and segmenting and recombining the reference image to obtain a face mark sequence; S3: fusing the joint latent mark and the face mark sequence using a preset local router to obtain an enhanced latent mark; S4: denoising the enhanced latent mark to obtain a preliminary denoised latent mark; S5: inputting the preliminary denoised latent mark into a time self-recurrent module to obtain a time-enhanced latent mark; S6: inputting the time-enhanced latent mark into a decoder to generate a personalized video.

[0007] Preferably, the joint latent mark is generated using the reference image and the text prompt, comprising the following steps: S11: inputting the reference image into a preset visual encoder to obtain image encoding features; S12: inputting the text prompt into a preset text encoder to obtain text encoding features; S13: fusing the image encoding features, the text encoding features, and random noise to obtain a joint latent mark.

[0008] Preferably, the face mark sequence is obtained by segmenting and recombining the reference image, comprising the following steps: S21: inputting the reference image into a preset segmentation network to obtain a face component; S22: inputting the face component into a preset local encoder for encoding to obtain a face mark sequence.

[0009] Preferably, the joint latent mark and the face mark sequence are fused using a local router to obtain an enhanced latent mark, comprising the following steps: S31: performing layer normalization on the joint latent mark and the face mark sequence respectively to obtain a layer-normalized joint latent mark and a layer-normalized face mark sequence; S32: calculating the correlation weight of the joint latent mark and the face mark sequence from the layer-normalized joint latent mark and the layer-normalized face mark sequence; S33: weighting and fusing the joint latent mark and the face mark sequence using the correlation weight to obtain an enhanced latent mark.

[0010] Preferably, the correlation weight is calculated according to the following formula:

[0011] wherein, is a layer-normalized facial landmark sequence, is a layer-normalized joint latent landmark, is a joint latent landmark correlation summary matrix, is a facial landmark sequence linear transformation matrix, is a Softmax function.

[0012] Preferably, the enhanced latent landmark is calculated as follows:

[0013] wherein, is a joint latent landmark, is a facial landmark sequence, is a hyperparameter controlling the scale of spatial enhancement, is a correlation weight, is a cross-enhancement function, is the number of elements in the facial landmark sequence, is an element-wise multiplication.

[0014] Preferably, the temporal autoregressive module divides the preliminary denoised latent landmark into multiple blocks along the frame axis, and predicts a potential bias in a preset order, and embeds the potential bias into the blocks to obtain a temporally enhanced latent landmark.

[0015] Preferably, the temporal autoregressive module comprises a Transformer block and a cross-attention module connected in sequence, the Transformer block comprises a rotary position embedding, a self-attention module, a causal mask embedding and a feedforward layer, the rotary position embedding inputs the self-attention module in combination with the preliminary denoised latent landmark to obtain a self-attention latent landmark, the causal mask embedding inputs the feedforward layer in combination with the self-attention latent landmark to obtain a self-attention mask latent landmark, the cross-attention module receives the input of the self-attention mask latent landmark and outputs the potential bias.

[0016] Preferably, the local router is trained using a cross-entropy loss function, and the cross-entropy loss function is calculated as follows:

[0017] wherein, is a ground truth, is a number of landmarks; is a correlation weight; is an element-wise multiplication, and M is a number of facial components.

[0018] The application also provides a personalized video generation system for identity preservation for implementing the above method, and the system comprises: An acquisition module is configured to acquire a reference image and a text prompt; A joint latent mark generation module is configured to generate a joint latent mark by using the reference image and the text prompt, and segment and reorganize the reference image to obtain a face mark sequence; A fusion module is configured to fuse the joint latent mark and the face mark sequence by using a preset local router to obtain an enhanced latent mark; A denoising module is configured to denoise the enhanced latent mark to obtain a preliminary denoised latent mark; A time enhancement module is configured to input the preliminary denoised latent mark into a time self-recurrent module to obtain a time-enhanced latent mark; A decoding module is configured to input the time-enhanced latent mark into a decoder to generate a personalized video.

[0019] Compared with the prior art, the application has the following beneficial effects: The application provides a personalized video generation method and system for identity preservation. First, a joint latent mark is generated by acquiring a reference image and a text prompt, and a face mark sequence is extracted, so that identity structure information is introduced while the semantic consistency is maintained. Then, the joint latent mark and the face mark sequence are fused by using a preset local router to obtain an enhanced latent mark, so that the identity key features are strengthened in the latent space, and the spatial consistency is effectively improved. Next, the enhanced latent mark is diffused and denoised to generate a preliminary denoised latent mark, and a time self-recurrent module is introduced to construct a time-enhanced latent mark, so that the cross-frame consistency and the time sequence stability are further enhanced. Finally, the time-enhanced latent mark is input into a decoder to generate a personalized video, so that the identity features are effectively maintained, and the spatial-temporal consistency is greatly improved without changing the original diffusion model structure. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A structure block diagram of the personalized video generation method for identity preservation proposed in the embodiment of the application is shown in the figure. Figure 2 An architecture diagram of the personalized video generation method for identity preservation proposed in the embodiment of the application is shown in the figure. Figure 3 A comparison diagram of the time self-recurrent module proposed in the embodiment of the application and the time self-recurrent module in the prior art is shown in the figure. Figure 4 An intuitive comparison diagram of the application and the prior art in the embodiment of the application is shown in the figure. Figure 5 A model component visual ablation research diagram in the embodiment of the application is shown in the figure. Figure 6A case diagram for an embodiment of the present application is shown. Figure 7 A structural diagram of a personalized video generation system for identity preservation proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] The accompanying drawings are only for illustrative purposes and should not be construed as limiting the present patent; It is understandable to some known content descriptions in the drawings can be omitted for those skilled in the art The technical solutions of the present application will be further described below in combination with the drawings and embodiments.

[0022] Embodiment 1 The present application proposes a personalized video generation method for identity preservation, as shown in the figure, the method comprises: Figure 1 As shown in the figure, the method comprises: S1: obtaining a reference image and a text prompt; S2: generating a joint latent mark using the reference image and the text prompt, and segmenting and recombining the reference image to obtain a face mark sequence; S3: fusing the joint latent mark and the face mark sequence using a preset local router to obtain an enhanced latent mark; S4: denoising the enhanced latent mark to obtain a preliminary denoised latent mark; S5: inputting the preliminary denoised latent mark into a time self-recurrence module to obtain a time-enhanced latent mark; S6: inputting the time-enhanced latent mark into a decoder to generate a personalized video.

[0023] In S2, specifically, the joint latent mark is generated by a preset text and visual encoder of the baseline model combined with the reference image and the text prompt; at the same time, the local face components (such as eyes, nose and lips) are extracted from the target image through a preset segmentation network. Then, the local encoder embeds these face components into the local face components to obtain the face mark sequence. As for the local encoder, we use the architecture proposed in ConsisID, which is originally designed to extract contour and high-frequency patterns. However, since the goal of our local encoder is to embed local facial structures, we fine-tune the local encoder together with the local router and the time self-recurrence module during the training of the model, rather than fixing it.

[0024] In S3, the face mark sequence is injected into the first DiT block of the preset baseline model through the preset local router. The router aims to model the correlation between the joint latent mark and the local face mark sequence, so as to adaptively utilize the face structure to perfect the joint latent mark and better preserve the identity information; Specifically, let denote a facial landmark sequence, where L is the number of landmarks, and D is the feature dimension, For simplicity of notation, we omit the time step index of the denoised time steps and represent the joint latent landmarks as The local router computes the weight for each component as follows:

[0025] where, is the layer-normalized facial landmark sequence, is the layer-normalized joint latent landmark, and apply linear transformations to and respectively, which project and into the same latent space; denotes the matrix transpose operator; is actually modeling the correlation between all landmarks in and and is the joint latent landmark correlation summary matrix, which summarizes the correlation of each landmark in ; is the Softmax function, which normalizes the correlation in all facial components.

[0026] Subsequently, the component weights are used to refine and improve spatial consistency as follows:

[0027] where, is a function that reconstructs using the landmarks in as a basis, similar to the cross-attention mechanism of feature interaction in pulid; is both key and value, while is treated as a query, is element-wise multiplication, is a hyperparameter that controls the scale of spatial augmentation; according to the formula, we can expect that a properly optimized router will assign higher weights to the identity-related and distinctive landmarks, thus facilitating effective identity protection.

[0028] Specifically, the local router is trained using a cross-entropy loss function that is calculated as follows:

[0029] where, is the ground truth, is the number of labels; is the relevance weight; is the element-wise multiplication.

[0030] Specifically, the total objective function of training is The calculation formula is as follows:

[0031] where, and is the hyperparameter balancing overall video quality and identity protection, and different proportions can be selected to achieve the best balance effect, is the standard diffusion loss.

[0032] The standard diffusion loss The calculation formula is as follows:

[0033] where, denotes the LDM with trainable parameters 𝜃. It can be interpreted as an iterative process consisting of a denoising step that gradually recovers the joint latent representation of the reference image and the text prompt from random noise . denotes an arbitrary denoising step size, is a hyperparameter that controls the noise scale at the 𝑡th step.

[0034] In S4, specifically, the enhanced latent label is processed via the DiT block in the preset baseline model, i.e., to obtain the preliminary denoised latent label; our module models the long-range temporal dependency between frames at the block level. Please note that this module is applied to the denoised label, so it does not change the latent space of the baseline DiT, while effectively utilizing the advantages of diffusion models and autoregressive models.

[0035] In S5, specifically, the temporal autoregressive module takes the preliminary denoised latent label before decoding as input, divides it into multiple blocks along the frame axis, and sequentially predicts the corresponding bias to perform temporal refinement. In addition, at the beginning of each block, the last enhanced frame of the previous block is inserted, forming a kind of "teacher forcing" paradigm to prevent the enhanced label from deviating too much from the original content. Assuming that the shape of the preliminary denoised latent label remains , the number of blocks is K, and we denote an arbitrary block as , . The process of obtaining is as follows:

[0036]

[0037] where denotes the predicted label bias. is a cross-attention module with the same structure as the local router. is an efficient transformer module that can be alternately stacked with multi-head self-attention modules and feed-forward layers. We employ two techniques in to facilitate temporal context modeling. First, we incorporate the input tokens of with their frame-indexed rotational position embeddings (RoPE) to incorporate temporal information. Second, to impose temporal constraints on the frames, we employ causal masking on the output of the self-attention module. Thus, can capture the temporal context in and interact with and similar to the spatial enhancement paradigm proposed in the formula. is a hyperparameter for weighting the bias.

[0038] Before decoding the output videos, we further improve their temporal consistency by a temporal autoregressive module, whose core idea is to incorporate explicit temporal dependencies into the latent tokens to alleviate abrupt changes in appearance and motion. The temporal autoregressive module divides the denoised tokens into multiple temporal blocks (i.e., along the frame axis) and predicts their bias according to the tokens in the previous temporal block, thereby refining the tokens in a temporal block step by step. This ensures explicit long-range temporal dependencies between denoised tokens, thereby overcoming the limitations of temporal independence and unstructured denoising processes of the baseline model.

[0039] The following briefly introduces the temporal autoregressive module proposed in this embodiment compared with other autoregressive-based mechanisms. As Figure 2As shown, existing autoregressive methods rely on predicting (a) discrete tokens, (b) multi-scale tokens, or (c) inter-frame tokens, which still cannot match the visual quality of diffusion models. (d) Hybrid models leverage the advantages of diffusion models in visual quality and autoregressive models in temporal modeling. However, their solutions are complex and require a large amount of resources. (e) Our module models long-range temporal dependencies between frames at the patch level. Note that this module is applied to denoised tokens, so it does not change the latent space of the baseline DiT while effectively leveraging the advantages of diffusion models and autoregressive models. Compared with existing methods, our module has three advantages: (i) Compared with methods that only rely on autoregression, our module can fully leverage the superior generation capability of mature diffusion models. (ii) Instead of directly predicting latent tokens, our method predicts the deviation. This is more stable than compared to because the latent tokens initially generated by the baseline LDM are a favorable starting point, helping to solve the problem of deviation accumulation. (iii) Unlike hybrid models, our module is used as a plug-in at the end of the denoising process. In this way, we can efficiently train the baseline model without changing its original feature space (using only one GPU).

[0040] In this embodiment, we implement a state-of-the-art text-to-video model based on where multiple DiT blocks with three-dimensional hybrid full attention are used to simulate the global spatio-temporal context. Existing models ignore the facial structure in the attention module and calculate the correlation of all latent tokens, which can easily cause feature interference and loss of target recognition. The steps of the method of the embodiment can solve these limitations. The specific structure block diagram of the embodiment is shown in Figure 3 .

[0041] Embodiment 2 This embodiment compares the method in the above embodiment with the prior art: The personalized video generation method and system for identity preservation provided by the present application, based on the local progressive diffusion architecture, introduces innovative modules in terms of spatial structure modeling and temporal consistency, has the following specific and significant beneficial effects, we use a plurality of widely used indicators, including the subject / background consistency indicators from vbench to measure the time quality, the FID indicators from Gans to evaluate the realism of the synthesized video, and the FaceSim-Curricular / ArcFace indicators from Arcface to calculate the ID similarity between the reference image and the generated video.

[0042] Table 1: Quantitative Comparison of Our System Framework with State-of-the-Art Methods

[0043] 1. Significantly improves identity preservation capabilities in video generation. By introducing a local router module, key facial regions (such as eyes, nose, and mouth) are extracted from the reference image and weighted fusion is performed, effectively enhancing the expression of potential features related to the target identity and significantly suppressing interference from non-target regions. Compared with existing methods such as ConsisID and ID-Animator, this invention shows significant improvements in identity consistency metrics (FaceSim-Curricular and FaceSim-ArcFace) (increasing to 0.425 and 0.401 respectively, leading by approximately 10-30 percentage points).

[0044] 2. Enhance inter-frame temporal consistency to improve video smoothness and coherence. By employing a temporal autoregressive module, the video representation is divided into multiple temporal blocks, and an inter-block bias prediction mechanism is established to explicitly capture long-range temporal dependencies. This solves the problems of frame flickering and abrupt changes caused by the independent modeling of traditional diffusion models in the temporal dimension. Experiments show that the proposed method achieves scores of 0.747 and 0.773 in "subject consistency" and "background consistency" metrics, respectively, surpassing the performance of the existing state-of-the-art methods ConsisID and CogVideoX+IPA.

[0045] 3. It balances model performance and inference efficiency, and has good practicality. This invention employs a modular plug-in design, allowing the local router and temporal autoregressive module to remain unchanged from the original diffusion model's structure, facilitating integration and deployment. Verification has shown that this method can perform complete training and efficient inference on a single A100 GPU (total training time approximately 60 hours, inference time approximately 10 seconds per video segment), significantly reducing resource requirements and enhancing its practical application in engineering. 4. Superior video quality and text alignment, adaptable to diverse and complex scenarios. This method not only maintains identity consistency but also accurately understands and reproduces the poses, actions, and background elements specified in the text description. For example, in scenarios involving multi-person interaction, dynamic actions, or changes in lighting, the generated video's dynamic range (DD) and text alignment (TA) are superior to the comparison methods. Comprehensive user surveys show that LaVieID received the highest voting rate (50.4%) in the identity similarity (IDS) dimension, fully validating its practical effectiveness. 5. Comprehensive advantages in supporting experimental and user research Through extensive comparative experiments and subjective evaluations by 30 users, the proposed method outperforms existing mainstream methods in multiple dimensions, including visual quality, identity preservation, and temporal consistency, demonstrating the comprehensive competitiveness of this invention in both academic research and practical applications. In addition, such as Figure 4 As shown, Figure 4 A visual comparison of video examples generated by different methods was presented. Compared to the baseline, LaVieID generated satisfactory quality identity-faithful content, even under complex conditions involving different poses, actions, expressions, and scenes. In contrast, ID-animator failed to maintain the coherence of facial representations and occasionally failed to focus on the subject (as in the first example). While ConsisID showed relatively stable identities, it suffered from poor text alignment in some cases, such as the loss of scene descriptions (e.g., "autumn trees" and "leaves") in the second example. Furthermore, we found that it unintentionally altered some facial attributes. For example, in... Figure 4 (Second example) and Figure 4 In the third example, it incorrectly turned two fairly "young" objects into old ones by adding a large number of wrinkles. The qualitative example of CogVideoX+IPA supports our view that it performs poorly in out-of-distribution generalization and preserving target identity. In conclusion, the quantitative results of LaVieID are in excellent agreement with its qualitative results, suggesting that it is an important solution for IPT2V generation. In summary, this invention not only excels in generation accuracy, identity restoration, and spatiotemporal consistency, but also offers lower costs, greater adaptability, and a better user experience in practical applications. It solves the core problem that existing personalized video generation technologies struggle to simultaneously achieve identity consistency and text matching, and thus possesses promising prospects for promotion and industrialization. To validate the effectiveness and contribution of each proposed component in the model, we conducted a series of ablation studies. Due to computational resource constraints, a subset of the test dataset was selected for evaluation.

[0046] Table 2: Ablation Study of Components in LaVieID

[0047] To evaluate the contribution of each key component in this system, we constructed two variants for evaluation: one combining the baseline DiT with the local router (denoted as w / LR), and the other combining it with the temporal autoregressive module (denoted as w / TAM). Quantitative results for these two variants are reported in Table 2, showing that both components offer significant performance improvements across all metrics. It is noteworthy that these two components have different objectives: the local router aims for spatial identity consistency, while the temporal autoregressive module, as its name suggests, is designed to improve temporal identity consistency. Therefore, it is reasonable that the baseline with LR outperforms the baseline with TAM on identity-centric metrics (FaceSim-Curricular / ArcFace). A more intuitive comparison based on temporal quality metrics further supports this. Specifically, the baseline using LR exhibits higher subject consistency, while the baseline using TAM performs better in terms of context consistency. These results indicate that the temporal autoregressive module tends to improve overall temporal consistency, while the local router focuses more on subject-centricity. Therefore, we can conclude that both components effectively achieve their intended purpose, and their integration within this system framework yields optimal overall performance.

[0048] like Figure 5 As shown, Figure 5 The visual comparison between the aforementioned variants of our method is shown. Compared to the visual examples generated by the baseline and LR methods, the visual examples generated by the baseline and TAM methods exhibit some identity distortion. For example, the latter variant alters the hair of two subjects. This aligns with our discussion above, that identity consistency benefits more from the local router, while temporal consistency is improved by the proposed autoregressive module. Another interesting phenomenon is that the videos generated by the baseline with TAM tend to have smaller but more dynamic subjects, which may explain its lower subject consistency performance compared to the baseline with LR. On the other hand, this also implies that the temporal autoregressive module helps generate more diverse and smoother motion. For example, in Figure 5 In the results, neither the baseline nor the variant with LR exhibited the "soft downward gaze" motion, while the baseline with TAM successfully achieved this motion. This provides a potential solution to alleviate the limitations of the IPT2V model in motion control. That is, if we can establish a long-distance and appropriate temporal correlation between the textual description of the motion and the latent markers, then it is possible to accurately generate the subject's motion from text alone. Finally, our system, employing the complete architecture, overcomes the limitations of both variants and achieves optimal visual results.

[0049] Example 3 This embodiment utilizes the method described in the above embodiments to generate a personalized video that preserves identity.

[0050] like Figure 6 As shown, this embodiment selects four common application scenarios for stylistic customization as examples, including text-driven secondary creation of anime characters, transformation of women into ink painting style, and application scenarios of transforming infants into adult men to change their age and customize their hairstyles. As can be seen from the figures, using the model proposed in this paper, Ao Bing can hold a coffee cup while reading and drinking coffee; Ms. Taylor can be presented in an ink painting style, which is different from a realistic style; a baby driven by the word "man" can generate a mature man's appearance, retaining the consistency of irrelevant details while adding adult features such as a beard; Scarlett can change from an updo to long hair, achieving a natural hairstyle change with a single click.

[0051] Example 4 like Figure 6 As shown, the present invention also provides a personalized video generation system for identity preservation, used to implement the above-described method, the system comprising: Acquisition module: Acquires reference images and text prompts; Joint latent marker generation module: generates joint latent markers using the reference image and text prompts, and segments and reassembles the reference image to obtain a facial marker sequence; Fusion module: uses a preset local router to fuse the joint latent marker and the facial marker sequence to obtain an enhanced latent marker; Denoising module: Denoises the enhanced latent markers to obtain preliminary denoised latent markers; Temporal enhancement module: The initial denoising latent markers are input into the temporal autoregressive module to obtain temporally enhanced latent markers; Decoding module: Inputs the time-enhanced latent markers into the decoder to generate personalized video.

[0052] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for generating personalized videos for identity preservation, characterized in that, Includes the following steps: S1: Obtain reference images and text prompts; S2: Generate joint latent markers using the reference image and text prompts, and segment and reassemble the reference image to obtain a facial marker sequence; S3: The joint latent marker and the facial marker sequence are fused using a preset local router to obtain an enhanced latent marker; S4: Denoise the enhanced latent markers to obtain preliminary denoised latent markers; S5: Input the preliminary denoised latent markers into the time autoregression module to obtain time-enhanced latent markers; S6: Input the time-enhanced latent marker into the decoder to generate a personalized video.

2. The personalized video generation method for identity preservation according to claim 1, characterized in that, The method of generating joint latent markers using the reference image and text prompts includes the following steps: S11: Input the reference image into a preset visual encoder to obtain image coding features; S12: Input the text prompt into a preset text encoder to obtain text encoding features; S13: The image coding features, the text coding features, and random noise are fused to obtain a joint latent label.

3. The personalized video generation method for identity preservation according to claim 1, characterized in that, The step of segmenting and recombining the reference image to obtain a facial marker sequence includes the following steps: S21: Input the reference image into a preset segmentation network to obtain facial components; S22: Input the facial components into a preset local encoder for encoding to obtain a facial marker sequence.

4. The personalized video generation method for identity preservation according to claim 1, characterized in that, The method of fusing the joint latent marker and facial marker sequence using a preset local router to obtain enhanced latent markers includes the following steps: S31: Perform layer normalization on the joint latent marker and the facial marker sequence to obtain layer normalized joint latent marker and layer normalized facial marker sequence, respectively; S32: Calculate the correlation weights of the joint latent markers and the face marker sequences using the layer-normalized joint latent markers and the layer-normalized face marker sequences; S33: The joint latent marker and the facial marker sequence are weighted and fused using the correlation weight to obtain the enhanced latent marker.

5. The personalized video generation method for identity preservation according to claim 4, characterized in that, The correlation weight The calculation formula is as follows: in, For layer-normalized facial marker sequences, For layer-normalized joint latent labels, This is a summary matrix of the correlations between the joint latent markers. The linear transformation matrix of the facial marker sequence, This refers to the Softmax function.

6. The personalized video generation method for identity preservation according to claim 1, characterized in that, The enhanced latent marker The calculation formula is as follows: in, For joint underwater markers, For facial marker sequences, To control the hyperparameters of spatial augmentation scale, For relevance weights, For cross-enhancement functions, The number of elements in the facial marker sequence. This is element-wise multiplication.

7. The personalized video generation method for identity preservation according to claim 1, characterized in that, The time autoregressive module divides the initial denoising latent marker into multiple blocks along the frame axis, predicts potential deviations in a preset order, and embeds the potential deviations into the blocks to obtain the time-enhanced latent marker.

8. The personalized video generation method for identity preservation according to claim 1, characterized in that, The time autoregression module includes a Transformer block and a cross-attention module connected in sequence. The Transformer block includes a rotation position embedding, a self-attention module, a causal mask embedding, and a feedforward layer. The rotation position embedding is combined with the initial denoising latent label input to the self-attention module to obtain a self-attention latent label. The causal mask embedding is combined with the self-attention latent label input to the feedforward layer to obtain a self-attention mask latent label. The cross-attention module receives the input of the self-attention mask latent label and outputs the latent bias.

9. The personalized video generation method for identity preservation according to claim 1, characterized in that, The local router is trained using a cross-entropy loss function. The calculation formula is as follows: in, For the basic truth, For the number of tags; Relevance weights; For element-wise multiplication, M is the number of facial components.

10. A personalized video generation system for identity preservation, characterized in that, include: Acquisition module: Acquires reference images and text prompts; Joint latent marker generation module: generates joint latent markers using the reference image and text prompts, and segments and reassembles the reference image to obtain a facial marker sequence; Fusion module: uses a preset local router to fuse the joint latent marker and the facial marker sequence to obtain an enhanced latent marker; Denoising module: Denoises the enhanced latent markers to obtain preliminary denoised latent markers; Temporal enhancement module: The initial denoising latent markers are input into the temporal autoregressive module to obtain temporally enhanced latent markers; Decoding module: Inputs the time-enhanced latent markers into the decoder to generate personalized video.

Images