Consistent video generation method and device based on wavelet transform and frequency decomposition

Abstract

The present invention provides a method and device for consistent video generation based on wavelet transform and frequency decomposition, which relates to the field of image processing technology. The present invention utilizes a wavelet multi-level frequency band decoupling-fusion framework and a dynamic cross-modal attention mechanism, combined with a dynamic alignment mechanism of a distributed perception adapter and a hierarchical diffusion HJB optimization strategy, to achieve identity representation fusion across spatiotemporal dimensions while ensuring parameter efficiency. This solves the problems of high-frequency detail drift, identity distortion in dynamic scenes, and poor stability in long-sequence generation in traditional video generation methods, significantly improving the quality and stability of generated videos and providing more advanced technical support for video processing and related applications.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for generating consistent video based on wavelet transform and frequency decomposition. Background Art

[0002] In recent years, generative AI-based video synthesis technology has made significant progress. Identity-Preserving Text-to-Video (IPT2V) has garnered significant attention due to its potential applications in virtual character-driven and personalized content creation. This technology must simultaneously meet the two core requirements of text-driven dynamic scene generation and maintaining the identity of the reference image. However, existing methods still face significant bottlenecks in cross-modal feature fusion, dynamic identity preservation, and long-sequence stability.

[0003] In existing technologies, models based on the U-Net architecture inject identity information through a text-image cross-attention mechanism. However, in actual applications, these models suffer from problems such as high-frequency detail drift and temporal motion interference. For example, high-frequency identity markers (such as facial texture and pupil details) are susceptible to motion deformation, resulting in insufficient consistency between generated frames. Low-frequency contour features (such as facial proportions and posture) are difficult to decouple dynamic scenes from static identity information due to the high coupling between the model's shallow features. Although recent studies have attempted to introduce frequency decomposition strategies to optimize feature expression, they do not fully utilize the multi-scale frequency band analysis capabilities of the wavelet transform, resulting in inefficient cross-band information fusion. In addition, frequency domain features are susceptible to noise interference in complex motion scenes.

[0004] To address the temporal modeling problem, diffusion-based video generation methods (such as Stable Video Diffusion) achieve motion control by adding a spatiotemporal coupling layer. However, their distribution alignment mechanisms fail to effectively reconcile the statistical differences between identity features and diffusion latent variables. Furthermore, existing methods generally rely on post-processing tools to fix generation flaws, resulting in video domain mismatch, increased computational overhead, and the inability to achieve end-to-end optimization.

[0005] Summarizing the bottlenecks of existing technologies, the core issues can be summarized as follows: 1) Insufficient multi-scale frequency domain decoupling, resulting in a failure to adaptively fuse cross-level frequency band features through wavelet transforms; 2) Lack of dynamic distribution alignment, where the mean-variance statistics of identity embeddings and diffusion latent variables are not explicitly aligned, leading to identity distortion under temporal layer interference; 3) Poor stability in long sequence generation, lacking optical flow constraints and sliding window fusion mechanisms, making it difficult to balance motion coherence and identity consistency. To address these issues, a new technical framework is urgently needed that integrates frequency domain decomposition, dynamic distribution alignment, and end-to-end optimization. Summary of the Invention

[0006] In response to the above technical problems in the related art, the present invention proposes a consistent video generation method and device based on wavelet transform and frequency decomposition.

[0007] In a first aspect, the present invention provides a method for generating consistent video based on wavelet transform and frequency decomposition, comprising the following steps:

[0008] S1. Based on the frequency band decoupling of multi-scale Haar wavelet decomposition, the low-frequency contour features and high-frequency texture features of the reference image are extracted, and the text embedding of the text prompt is obtained through the CLIP text encoder. The low-frequency contour features, high-frequency texture features, and text embedding are subjected to cross-modal attention to obtain the multi-scale wavelet feature F. wavelet ;

[0009] S2, through the frequency band adaptive convolution module and inverse wavelet reconstruction fusion multi-scale wavelet features F wavelet The multi-level features of the image are outputted by retaining cross-scale details. fused ;

[0010] S3, embed the face of the reference image obtained by the ArcFace network into E arc The global context feature E of the reference image obtained by CLIP-ViT encoder clip Generating refined facial embeddings E by Q-Former fusion final ;

[0011] S4, diffuse latent variable Z t , fusion image features F fused and identity embedding E final Input to the dual-path cross attention module to align the identity distribution of the latent variable space and the temporal smoothing constraint to obtain the aligned latent variable Z aligned ;

[0012] S5. Align the latent variable Z aligned With text embedded E text Injecting the spatiotemporal DiT architecture to generate preliminary temporal latent variables Z time , and the diffusion latent variable Z t Generate the optimized latent variable Z through Hamilton-Jacobi optimization t-1 , and then the preliminary time series latent variable Z time and optimize the latent variable Z t-1 Generate high-quality video V via a multi-frame autoregressive generator output .

[0013] Specifically, step S1 includes the following steps:

[0014] S11, the reference image Input multi-level Haar wavelet transform, and recursively decompose L levels to obtain low-frequency components and high frequency components H is the image height, W is the image width;

[0015] S12, frequency band feature reorganization, align and splice the low-frequency components of each layer through bicubic interpolation, and fuse them to generate the global low-frequency feature F low , sum the high-frequency components of each layer by weight according to the level to generate the local high-frequency feature F high ;

[0016] S13, input the text prompt T into the CLIP text encoder, extract the text embedding E text , through the interaction of cross-modal attention and wavelet features, the multi-scale wavelet feature F is finally output wavelet =(F low ,F high ); where F low is the low-frequency feature; F high It is a high frequency feature.

[0017] Specifically, step S2 includes the following steps:

[0018] S21, multi-scale wavelet feature F wavelet =(F low ,F high ) The frequency band adaptive convolution module is injected to perform hierarchical inverse reconstruction and inverse wavelet transform reconstruction to obtain the reconstructed feature R (i) ;

[0019] S22, cross-level feature aggregation, reconstructing feature R through skip connection and hierarchical weighting (i) Perform aggregation and output fusion image feature F fused :

[0020] Specifically, step S3 includes the following steps:

[0021] S31, reference image I ref Input to the ArcFace network, generate facial embedding E through multi-scale feature fusion and global average pooling arc ;

[0022] S32, the reference image I ref Input to CLIP-ViT encoder, after block embedding, position encoding and multi-layer Transformer, output global context feature E clip ;

[0023] S33, embed the face into E arc and global context feature E clipInput to the Q-Former module, through the Q-Former module's learnable query matrix projection and cross attention calculation, a refined facial embedding E is generated. final .

[0024] Specifically, step S4 includes the following steps:

[0025] S41, diffuse latent variable Z t Inject the two-way cross attention module, respectively with the fusion image feature F fused Interactively obtain the image feature diffusion variable Z img , and the identity embedding E final Interaction to obtain identity embedding diffusion variable Z face The dual-path cross attention module includes image path cross attention and identity path cross attention;

[0026] S42, diffuse the image feature variable Z img and identity embedding diffusion variable Z face Input to the dynamic distribution alignment module, first calculate the image feature mean μ through the statistical calculation formula img and identity embedding mean μ face and the image feature standard deviation σ img and identity embedding standard deviation σ face , the projection distribution is obtained through the distribution projection formula Fuse the residuals and add the gate weight γ to output the aligned latent variable Z aligned ;

[0027] S43, to align the latent variable Z aligned Add a time domain smoothing constraint: The time domain smoothing constraint formula is as follows:

[0028]

[0029] Among them, L align is a loss function with time domain smoothness constraint, which is used to measure the alignment of latent variables. The difference between its average pooling results at different time steps; To align the latent variable Z aligned Representation at the tth time step; AvgPool() is the average pooling operation function; T is the total number of denoising steps of the diffusion model.

[0030] Specifically, step S5 includes the following steps:

[0031] S51, align the latent variable Z aligned With text embedded E text Injecting the spatiotemporal DiT architecture, the preliminary temporal latent variable Z is generated by shallow low-frequency feature fusion and deep high-frequency cross attentiontime ; The shallow low-frequency feature fusion is performed by convolution after bilinear upsampling;

[0032] S52, diffuse latent variable Z t Input Hamilton-Jacobi optimization module, reverse optimization based on identity similarity loss, and adjust the diffusion latent variable Z t The distribution of the output optimization latent variable Z t-1 ;

[0033] S53, optimize the latent variable Z t-1 Inject multi-frame autoregressive generator, through the optical flow constraint loss L flow And sliding window weighted fusion, finally output identity consistent high-quality video

[0034] Specifically, step S11 includes the following steps:

[0035] S111, wavelet transform initialization, input reference image With the text prompt T, a predefined Haar wavelet filter bank {f LL ,f LH ,f HL ,f HH},in:

[0036]

[0037] to I ref Perform the i-th level two-dimensional discrete wavelet transform (2D-DWT):

[0038]

[0039] in The output resolution of each level is reduced to 1 / 2 of the previous level; the value range of i is 1 to L; stride = 2 means that the output resolution of each level is reduced to 1 / stride of the previous level; is the previous level image of the i-th level two-dimensional discrete wavelet transform;

[0040] S112, multi-level recursive decomposition, low-frequency components Recursively perform L-level wavelet decomposition:

[0041]

[0042] Output multi-scale feature group in is the low-frequency component, is the high frequency component, and L is a positive integer greater than or equal to 3.

[0043] Specifically, step S12 includes the following steps:

[0044] S121, global low-frequency feature F low Align and concatenate the low-frequency components of each layer through bicubic interpolation get:

[0045]

[0046] Among them, Concat() is the fusion function; Upsample() is the upsampling function;

[0047] S122, local high-frequency feature F high Through the weighted fusion of high-frequency components of each layer:

[0048]

[0049] in is the decay weight, For channel splicing.

[0050] Specifically, step S13 includes the following steps:

[0051] S131, text conditional injection, input the text prompt T into the CLIP text encoder, generate text embedding after Tokenize and position encoding

[0052] E text =CLIP text (T)=Transformer 12L (Embed(T))

[0053] CLIP text (T) indicates that the text prompt T is input into the text branch of the CLIP model for processing to obtain preliminary text features; Transformer 12L (Embed(T)) means that the text prompt T is converted into an embedding vector through Embed(T), and then the embedding vector is input into a 12-layer Transformer model for Tokenize and position encoding, and finally the text embedding of the text prompt T is obtained. Where N represents the number of video frames generated in a single shot in the multi-frame autoregressive generator, which is used to control the step size of the sliding window; d is the feature dimension of the diffusion latent variable, which is used to represent the spatial information of the compressed video frame;

[0054] S132, E through cross-modal attention text Inject wavelet features and output multi-scale wavelet features F wavelet =(F low ,Fhigh ):

[0055] F wavelet =CrossAttn(Q=F low ‖F high ,K=V=E text )

[0056] Among them, CrossAttn is the cross-modal attention function, || represents splicing along the channel dimension, the number of attention heads is 8, and the scaling factor is Q is the query vector, which is composed of features of different scales of the image; K is the key vector; V is the value vector; F low is the low-frequency feature; F high It is a high frequency feature.

[0057] In a second aspect, the present invention provides a consistent video generation device based on wavelet transform and frequency decomposition, based on the consistent video generation method based on wavelet transform and frequency decomposition described in the first aspect, comprising the following units:

[0058] The wavelet feature extraction unit extracts the low-frequency contour features and high-frequency texture features of the reference image based on the frequency band decoupling of the multi-scale Haar wavelet decomposition, obtains the text embedding of the text prompt through the CLIP text encoder, and performs cross-modal attention on the low-frequency contour features, high-frequency texture features, and text embedding to obtain the multi-scale wavelet feature F wavelet ;

[0059] Cross-scale fusion unit, used to fuse multi-scale wavelet features F through frequency band adaptive convolution and inverse wavelet reconstruction wavelet The multi-level features of the image are outputted by retaining cross-scale details. fused ;

[0060] The face embedding generation unit is used to embed the face of the reference image obtained by the ArcFace network into E arc The global context feature E of the reference image obtained by CLIP-ViT encoder clip Generating refined facial embeddings E by Q-Former fusion final ;

[0061] Alignment latent variable generation unit, used to diffuse latent variable Z t , fusion image features F fused and identity embedding E final Input to the dual-path cross attention module to align the identity distribution of the latent variable space and the temporal smoothing constraint to obtain the aligned latent variable Z aligned ;

[0062] The consistent video generation unit is used to align the latent variables Z alignedWith text embedded E text Injecting the spatiotemporal DiT architecture to generate preliminary temporal latent variables Z time , and the diffusion latent variable Z t Generate the optimized latent variable Z through Hamilton-Jacobi optimization t-1 , and then the preliminary time series latent variable Z time and optimize the latent variable Z t-1 Generate high-quality video V via a multi-frame autoregressive generator output .

[0063] The present invention adopts a wavelet multi-level frequency band decoupling-fusion framework and a dynamic cross-modal attention mechanism, combined with the dynamic alignment mechanism of the distributed perception adapter and the HJB optimization strategy of hierarchical diffusion, to achieve identity representation fusion across time and space dimensions while ensuring parameter efficiency. It solves the problems of high-frequency detail drift, identity distortion in dynamic scenes, and poor stability of long sequence generation in traditional video generation methods, significantly improves the quality and stability of generated videos, and provides more advanced technical support for video processing and related applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0065] Figure 1 A schematic diagram of a consistent video generation method based on wavelet transform and frequency decomposition provided by an embodiment of the present invention;

[0066] Figure 2 A schematic diagram of a consistent video generation device based on wavelet transform and frequency decomposition provided by an embodiment of the present invention;

[0067] Figure 3 A schematic diagram of a consistent video generation device based on wavelet transform and frequency decomposition provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The present invention can be explained in detail through the following examples. The purpose of providing the present invention is to protect all technical improvements within the scope of the present invention. In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" can explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0069] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0070] Example 1

[0071] refer to Figure 1 This embodiment provides a consistent video generation method based on wavelet transform and frequency decomposition, comprising the following steps:

[0072] S1. Based on the frequency band decoupling of multi-scale Haar wavelet decomposition, the low-frequency contour features and high-frequency texture features of the reference image are extracted, and the text embedding of the text prompt is obtained through the CLIP text encoder. The low-frequency contour features, high-frequency texture features, and text embedding are subjected to cross-modal attention to obtain the multi-scale wavelet feature F. wavelet ;

[0073] Step S1 specifically includes the following steps:

[0074] S11, the reference image Input multi-level Haar wavelet transform, and recursively decompose L levels to obtain low-frequency components and high frequency components H is the image height, W is the image width;

[0075] The two-dimensional discrete wavelet transform (2D-DWT) is a method that extends the one-dimensional discrete wavelet transform to two dimensions and is used to process two-dimensional signals such as images. Its core concept is to decompose an image into subbands (low-frequency and high-frequency components) of varying scales through filtering and downsampling, thereby enabling multiresolution analysis. The 2D-DWT is a universal image decomposition method that can utilize different wavelet basis functions (such as Haar, Daubechies, and Symlets), each producing different results.

[0076] The multi-scale Haar wavelet decomposition in this embodiment is a two-dimensional discrete wavelet transform using Haar wavelet basis functions.

[0077] The core of the Haar wavelet transform is the binary hierarchical decomposition of the signal. For one-dimensional signals, the low-frequency and high-frequency components are separated by averaging and differencing adjacent data. For two-dimensional signals (such as images), four subbands are generated by two one-dimensional transformations of rows and columns:

[0078] Low-frequency component (LL): retains the overall outline of the signal or the average information of the image;

[0079] Horizontal high-frequency component (HL): captures changes in the vertical direction (such as horizontal edges);

[0080] Vertical high frequency component (LH): captures changes in the horizontal direction (such as vertical edges);

[0081] Diagonal high frequency component (HH): captures oblique edges or texture details;

[0082] In the Haar wavelet transform, a signal or image can be subjected to multiple wavelet transforms, each transform decomposing the signal or image into low-frequency and high-frequency components. Similarly, in the 2D-DWT, an image can be subjected to multiple decompositions, each decomposition decomposing the image into four subbands.

[0083] The specific steps include:

[0084] S111, wavelet transform initialization, input reference image With the text prompt T, a predefined Haar wavelet filter bank {f LL ,f LH ,f HL ,f HH},in:

[0085]

[0086] to I ref Perform the i-th level two-dimensional discrete wavelet transform (2D-DWT):

[0087]

[0088] in The output resolution of each level is reduced to 1 / 2 of the previous level; the value range of i is 1 to L; stride = 2 means that the output resolution of each level is reduced to 1 / stride of the previous level; is the previous level image of the i-th level two-dimensional discrete wavelet transform;

[0089] S112, multi-level recursive decomposition, low-frequency components Recursively perform L-level wavelet decomposition:

[0090]

[0091] Output multi-scale feature group in is the low-frequency component, is the high frequency component, L is the wavelet decomposition series, and L is a positive integer greater than or equal to 3;

[0092] S12, frequency band feature reorganization, align and splice the low-frequency components of each layer through bicubic interpolation, and fuse them to generate the global low-frequency feature F low , that is, low-frequency contour features (contour information); the high-frequency components of each layer are weighted and summed according to the level to generate local high-frequency features F high , that is, high-frequency texture features (texture details);

[0093] S121, global low-frequency feature F low Align and concatenate the low-frequency components of each layer through bicubic interpolation get:

[0094]

[0095] Among them, Concat() is the fusion function; Upsample() is the upsampling function;

[0096] S122, local high-frequency feature F high Through the weighted fusion of high-frequency components of each layer:

[0097]

[0098] in is the decay weight, For channel splicing.

[0099] S13, input the text prompt T into the CLIP text encoder, extract the text embedding E text , through the interaction of cross-modal attention and wavelet features, the multi-scale wavelet feature F is finally output wavelet =(F low ,F high ); where F low is the low-frequency feature; F high It is a high frequency feature.

[0100] S131, text conditional injection, input the text prompt T into the CLIP text encoder, generate text embedding after Tokenize and position encoding Its expression formula is:

[0101] E text =CLIP text (T)=Transformer 12L (Embed(T))

[0102] CLIPtext (T) indicates that the text prompt T is input into the text branch of the CLIP model for processing to obtain preliminary text features; Transformer 12L (Embed(T)) means that the text prompt T is converted into an embedding vector through Embed(T), and then the embedding vector is input into a 12-layer Transformer model for Tokenize and position encoding, and finally the text embedding of the text prompt T is obtained. Where N is the number of video frames generated in a single shot in the multi-frame autoregressive generator in the subsequent step, which is used to control the step size of the sliding window; d is the feature dimension of the diffusion latent variable (LatentVariable) in the subsequent step, which is used to represent the spatial information of the compressed video frame;

[0103] Tokenize is a key step in natural language processing (NLP) to segment text into meaningful units (such as words, punctuation, etc.). Its core role is to provide structured data support for subsequent tasks.

[0104] S132, E through cross-modal attention text Inject wavelet features and output multi-scale wavelet features F wavelet =(F low ,F high ):

[0105] F wavelet =CrossAttn(Q=F low ‖F high ,K=V=E text )

[0106] Among them, CrossAttn is the cross-modal attention function, || represents splicing along the channel dimension, the number of attention heads is 8, and the scaling factor is Q is the query vector, which is composed of features of different scales of the image; K is the key vector; V is the value vector; F low is the low-frequency feature; F high It is a high frequency feature.

[0107] S2, through the frequency band adaptive convolution module and inverse wavelet reconstruction fusion multi-scale wavelet features F wavelet The multi-level features of the image are outputted by retaining cross-scale details. fused ;

[0108] The specific steps include:

[0109] S21, multi-scale wavelet feature F wavelet =(F low ,F high) The frequency band adaptive convolution module is injected to perform hierarchical inverse reconstruction and inverse wavelet transform reconstruction to obtain the reconstructed feature R (i) ;

[0110] S211, constructing a frequency band adaptive convolution kernel of a frequency band adaptive convolution module; the frequency band adaptive convolution kernel includes a low-frequency channel convolution kernel and a high-frequency channel convolution kernel; the total number of levels of the hierarchical structure of the frequency band adaptive convolution module is 3 layers;

[0111] The low-frequency channel convolution kernel adopts a decomposition sparse convolution structure:

[0112]

[0113] The high-frequency channel convolution kernel adopts a dense small kernel structure:

[0114]

[0115] Among them, the low-frequency channel uses 51×5 and 5×51 cores; the high-frequency channel uses 3×3 dense cores; C low is the number of low-frequency channels, C high is the number of high-frequency channels;

[0116] S212, hierarchical inverse reconstruction: multi-scale wavelet features are reconstructed by the frequency band adaptive convolution kernel Recursively perform inverse wavelet transform and convolution operations to generate reconstructed features R (i) , the i-th level reconstruction formula is:

[0117]

[0118] in, Reconstruct low-frequency features for level i; Reconstruct high-frequency features for the i-th level; IWT(·) is the inverse wavelet transform, and IWT(·) uses the Haar wavelet inverse transform kernel Conv(·) is the convolution function;

[0119] It can be understood that the multi-scale wavelet feature F wavelet =(F low ,F high )and are equivalent, F low With F high They all contain L-level features, where L is the wavelet decomposition level; and are equivalent; the value of i ranges from 1 to L;

[0120] In the hierarchical inverse reconstruction of step S212, the number of levels is consistent with the decomposition level L in step S11 (usually L = 3), that is, the inverse reconstruction is recursively performed L times before it ends. For example, if the original image undergoes three levels of wavelet decomposition (L = 3), the reconstruction requires the same three levels of inverse operations, with each level gradually restoring the resolution through the band-adaptive convolution module, and finally outputting the fused image features.

[0121] S22, cross-level feature aggregation, reconstructing feature R through skip connection and hierarchical weighting (i) Perform aggregation and output fusion image feature F fused :

[0122]

[0123] in is the level attenuation weight; Upsample(·,k) represents the bicubic interpolation upsampling k times.

[0124] S3, embed the face of the reference image obtained by the ArcFace network into E arc The global context feature E of the reference image obtained by CLIP-ViT encoder clip Generating refined facial embeddings E by Q-Former fusion final ;

[0125] This step combines ArcFace biometrics with CLIP global context, and uses Q-Former to implement identity-scene dynamic interaction to build a global content-aware facial encoder. The specific steps include:

[0126] S31, reference image I ref Input to the ArcFace network, generate facial embedding E through multi-scale feature fusion and global average pooling arc ;

[0127] Specifically, the ArcFace network is a ResNet-50 backbone network;

[0128] Specifically, ArcFace network is used to extract facial features: Input reference image Extract multi-scale features through the ResNet-50 backbone network

[0129]

[0130] Multi-scale features Equivalent to stride 2 k , and then fuse multi-scale features And generate the facial embedding E arc :

[0131]

[0132] in, is the backbone network parameter; Concat() is the fusion function; GAP(·) is the global average pooling; FC(·) is the fully connected layer with a dimension of 512;

[0133] The classification loss function L used in the multi-scale feature fusion and global average pooling stages of the ArcFace network arc Using additive angular spacing loss:

[0134]

[0135] Where, e is the base of natural logarithm; log is the natural logarithm function; θ y is the angle of the real sample; m is the additive angle interval; y represents the real category label; j represents the possible category index of the traversal index; θ j is the angle of the jth sample; s is the scaling factor, which is used to adjust the effect of the angle so that the distance between samples of similar categories is smaller and the distance between different categories is larger;

[0136] Classification loss function L arc (Additive Angular Margin Loss) is used in step S31 to optimize the facial embedding generation of the ArcFace network by expanding the inter-class angular margins of different identities and reducing the intra-class differences, thereby enhancing the discriminative power of identity features. In step S31, this loss function acts on the multi-scale feature fusion and global average pooling stages of the ArcFace network, directly driving the network to generate a highly discriminative facial embedding E arc ;

[0137] S32, the reference image I ref Input to CLIP-ViT encoder, after block embedding, position encoding and multi-layer Transformer, output global context feature E clip ;

[0138] The CLIP text encoder and CLIP-ViT encoder are both part of the CLIP model. The former processes text to generate semantic embeddings, while the latter processes images to generate visual embeddings.

[0139] The CLIP-ViT encoder is an image encoder that combines the CLIP (Contrastive Language-Image Pre-training) encoder and the Vision Transformer (ViT). It is a key component of the CLIP model, used to encode image content into high-dimensional feature vectors. These feature vectors can be compared with text feature vectors in various tasks to achieve image-to-text matching. In the CLIP model, the image encoder is typically based on the ViT architecture. The CLIP-ViT encoder uses ViT as its backbone network to extract image features.

[0140] The CLIP-ViT encoder first divides the input image into image patches, then linearly embeds and positionally encodes these patches to preserve spatial information. These embedded image patches are then fed into a multi-layer Transformer network, which learns the relationships between image patches through a self-attention mechanism and outputs global contextual features of the image.

[0141] Specifically including: input reference image I ref The CLIP encoder extracts global features and generates global embedding through the ViT structure, namely the global context feature E clip , with a dimension of 768:

[0142]

[0143] VIT specific operation process:

[0144] Image segmentation:

[0145]

[0146] Transformer encoding:

[0147] E clip =MultiHeadAttn(LN(P pos ))

[0148] Among them, P is the image block; PatchEmbed() is the image block function, MultiHeadAttn() is the multi-head attention function, LN() is the layer normalization function, P pos PatchPosition Encoding: Assign position information to each image patch in ViT to preserve spatial relationships.

[0149] S33, embed the face into E arc and global context feature Eclip Input to the Q-Former module, through the Q-Former module's learnable query matrix projection and cross attention calculation, a refined facial embedding E is generated. final ;

[0150] The Q-Former module is a cross-modal interaction component used to fuse CLIP's text / image features with external identity features (such as ArcFace's facial embedding) to achieve multimodal alignment through learnable queries.

[0151] CLIP's dual encoder provides basic semantic-visual representation, and Q-Former introduces a dynamic attention mechanism on this basis to enhance the fine-grained fusion of identity-related features.

[0152] Specifically including: cross-modal interaction through the Q-Former module, input facial embedding With global features Through the learnable Query matrix W Q Projection embeds the face into E arc Projected to the face embedding projection Q, the global context feature E is projected into the face embedding projection Q through the Key-Value projection clip Project to the key feature projection K and the value feature projection V, then calculate the facial embedding projection Q, the key feature projection K and the value feature projection V through cross attention to obtain the attention facial embedding E face Finally, the refined facial embedding E is obtained through feature fusion final ;

[0153] Learnable Query matrix W Q The projection formula is as follows:

[0154]

[0155] The Key-Value projection formula is as follows:

[0156]

[0157] The cross attention calculation formula is as follows:

[0158]

[0159] The feature fusion formula is as follows:

[0160] E final =LayerNorm(E arc +FC(Flatten(E face )))

[0161] Among them, M is the number of heads in Multi-Head Attention, and its value is 32; D q is the dimension of the query vector (Query), and its value is 256; W K is the weight matrix dimension of the key; W V is the weight matrix dimension of the value; D k is the final dimension of the key vector (Key), whose value is 256; D v is the final dimension of the value vector (Value), which is 256; H' is the spatial height of the image; W' is the spatial width of the image; E face is the attention face embedding; LayerNorm() is the layer normalization function; Flatten() is the flattening operation used to expand the multidimensional tensor into a one-dimensional sequence; E final It is Enhanced Face Embedding with a dimension of 512.

[0162] S4, diffuse latent variable Z t , fusion image features F fused and identity embedding E final Input to the dual-path cross attention module to align the identity distribution of the latent variable space and the temporal smoothing constraint to obtain the aligned latent variable Z aligned ;

[0163] Z aligned The input is sent to the time domain constraint module, optimized through the sliding average optical flow loss, and finally outputs a time-consistent latent variable sequence;

[0164] The specific steps include:

[0165] S41, diffuse latent variable Z t Inject the two-way cross attention module, respectively with the fusion image feature F fused Interactively obtain the image feature diffusion variable Z img , and the identity embedding E final Interaction to obtain identity embedding diffusion variable Z face The dual-path cross attention module includes image path cross attention and identity path cross attention;

[0166] The two-way cross-attention module is an independent cross-modal fusion component in this method, which is used to align the diffusion latent variable with the identity feature (as described in step S41), while the CLIP model is used to provide the basic features of text / image encoding (text encoder and CLIP-ViT encoder).

[0167] The diffusion latent variable Z tInitialized to random Gaussian noise; the diffusion latent variable Z t The feature dimension is d.

[0168] Specifically, the input diffusion latent variable Image features Identity Embedding To the two-way cross attention module, the image feature diffusion variable Z is obtained through the image path cross attention formula img , the image path cross attention formula is as follows:

[0169]

[0170] Among them, CrossAttn is the cross attention function; is the query projection weight matrix in the image path; is the key projection weight matrix in the image path; D h is the feature dimension of a single attention head (HeadDimension); is the value projection weight matrix in the image path; T is the total number of denoising steps of the diffusion model;

[0171] The identity embedding diffusion variable Z is obtained through the identity path cross attention formula face , the identity path cross attention formula is as follows:

[0172]

[0173] Among them, Expand() is a tensor dimension expansion operation; is the projection weight matrix of the query in the identity path; is the projection weight matrix of the key in the identity path; is the projection weight matrix of the identity path median value (Value); T is the total denoising steps of the diffusion model;

[0174] Image feature diffusion variable Z img is the output of the image path cross attention module, representing the diffusion latent variable guided by the fusion image features;

[0175] Identity embedding diffusion variable Z face The latent variable generated by cross-attention of identity paths represents the distribution correction result guided by identity embedding. Purpose: Inject facial identity features (such as pupils and outlines) into the latent variable space to ensure the consistency of character identity in the generated video.

[0176] S42, diffuse the image feature variable Z img and identity embedding diffusion variable Z faceInput to the dynamic distribution alignment module, first calculate the image feature mean μ through the statistical calculation formula img and identity embedding mean μ face and the image feature standard deviation σ img and identity embedding standard deviation σ face , the projection distribution is obtained through the distribution projection formula Fuse the residuals and add the gate weight γ to output the aligned latent variable Z aligned ;

[0177] The dynamic distribution alignment module includes statistics calculation, distribution projection calculation and residual fusion calculation;

[0178] The statistical calculation formula is as follows:

[0179]

[0180] Among them, Mean() is the mean function; Std() is the standard deviation function; Dim is the dimension; and the four formulas are statistical formulas.

[0181] The distribution projection formula is as follows:

[0182]

[0183] The residual fusion formula is as follows:

[0184]

[0185] Among them, LayerNorm is the layer normalization function, which is used to reduce the feature distribution offset through normalization, making the alignment of latent variables of identity and image paths more efficient and reliable; γ is the gating weight.

[0186] S43, time domain robustness enhancement, to align the latent variable Z aligned Add a time domain smoothing constraint: The time domain smoothing constraint formula is as follows:

[0187]

[0188] Among them, L align is a loss function with time domain smoothness constraint, which is used to measure the alignment of latent variables. The difference between its average pooling results at different time steps; To align the latent variable Z aligned Representation at the tth time step; AvgPool() is the average pooling operation function; T is the total number of denoising steps of the diffusion model.

[0189] S5. Align the latent variable Z aligned With text embedded E textInjecting the spatiotemporal DiT architecture to generate preliminary temporal latent variables Z time , and the diffusion latent variable Z t Generate the optimized latent variable Z through Hamilton-Jacobi optimization t-1 , and then the preliminary time series latent variable Z time and optimize the latent variable Z t-1 Generate high-quality video V via a multi-frame autoregressive generator output .

[0190] Diffusion latent variable Z t It is the intermediate state of the diffusion model, representing the latent space characteristics during the denoising process. The diffusion model achieves video generation by adjusting the distribution of latent variables. The two have a "framework-core variable" relationship.

[0191] The diffusion model is a generative model that generates data by gradually adding noise (forward diffusion) and then back-denoising. In the embodiment, the diffusion model refers to a video generation framework based on latent variable optimization, that is, step S5, converting random Gaussian noise into video latent variables through multi-step denoising.

[0192] The latent variable Z will be aligned aligned With text embedded E text Inject the spatiotemporal DiT architecture and generate preliminary temporal latent variables through shallow low-frequency feature fusion (bilinear upsampling + convolution) and deep high-frequency cross attention; t Input Hamilton-Jacobi optimization module, based on identity similarity loss L id Reverse optimization, adjust the diffusion latent variable Z t Distribution, output optimized Z t-1 ; Inject the optimized latent variable sequence into the multi-frame autoregressive generator, and use the optical flow constraint loss L flow And sliding window weighted fusion, finally output identity consistent high-quality video

[0193] S51, align the latent variable Z aligned With text embedded E text Injecting the spatiotemporal DiT architecture, the preliminary temporal latent variable Z is generated by shallow low-frequency feature fusion and deep high-frequency cross attention time ; The shallow low-frequency feature fusion is performed by convolution after bilinear upsampling;

[0194] The spatiotemporal DiT architecture is an improved structure combining the diffusion model and the Transformer. Its core is to introduce a spatiotemporal separation attention mechanism (spatial 3D convolution + temporal self-attention) based on the standard DiT to simultaneously model the spatial details within the video frame and the continuity of the inter-frame motion. This architecture is a custom design of the method in this embodiment. The basic configuration consists of 6 layers of stacked Transformer blocks, each layer integrating spatial local perception and global temporal interaction modules, supporting multi-scale latent variable optimization.

[0195] In step S51 , the spatiotemporal DiT architecture is used to iteratively denoise and generate a video.

[0196] Specifically: align the latent variables Text Embedding Input to the spatiotemporal DiT framework;

[0197] Shallow low-frequency injection, the first three layers of DiT blocks inject multi-scale wavelet low-frequency features The fusion formula is:

[0198]

[0199] Among them, Upsample() is the upsampling function; Conv() is the convolution function; l is the layer index of the spatiotemporal DiT architecture, indicating the lth layer Transformer block currently being processed; DiTBlock() is a spatiotemporal separation Transformer module, which consists of spatial 3D convolution and temporal self-attention, supporting local spatial perception and global motion modeling; is the input latent variable of the lth layer, which is the intermediate feature after being processed by the previous layer (l-1); is the output projection weight matrix of the lth layer, which is used to map the output features of DiTBlock to the input dimension of the next layer;

[0200] Deep high-frequency injection, the 4th to 6th layer DiT blocks inject high-frequency features Adopting cross attention mechanism CrossAttn:

[0201]

[0202] In this embodiment and The meaning is consistent; and The meaning is consistent.

[0203] The CrossAttn mechanism adopts a spatiotemporal attention separation mechanism, where each DiT block contains two types of attention executed sequentially:

[0204] Spatial Attention:

[0205]

[0206] Time Attention:

[0207]

[0208] Among them, Q space is the spatial query matrix; is the spatial bond matrix; V space is a spatial value matrix; Softmax() is a normalization function; Q time is the time query matrix; is the time key matrix; V time is the time value matrix; H' is the height of the feature map after bilinear upsampling; W' is the width of the feature map after bilinear upsampling, which is scaled synchronously with H'; D h is the diffusion latent variable Z t The feature dimension represents the length of each latent variable vector; T is the total denoising steps of the diffusion model, which controls the number of iterations from noise to clear video;

[0209] S52, diffuse latent variable Z t Input Hamilton-Jacobi optimization module, reverse optimization based on identity similarity loss, and adjust the diffusion latent variable Z t The distribution of the output optimization latent variable Z t-1 .

[0210] The Hamilton-Jacobi (HJB) optimization module is a diffusion model training framework based on optimal control theory. Its function is to adjust the diffusion latent variable Z through dynamic programming strategy. t The gradient update path ensures the stability and distribution alignment of the video generation process. The Hamilton-Jacobi (HJB) optimization module is the core technology for diffusion model training in this embodiment (not part of CLIP). It is independent of CLIP's encoding function and only acts on the latent variable optimization stage (step S52), achieving high-quality video generation by minimizing the loss function.

[0211] Hamilton-Jacobi optimization, in the denoising step t∈{T,...,1}, for the latent variable Z t Apply facial optimization constraints to obtain the optimized latent variable Z t-1 , the expression is as follows:

[0212]

[0213] Among them, EDM_Solver is an optimization solver based on the Hamilton-Jacobi equation, which uses back propagation gradient Dynamically adjust the latent variable distribution and gradually adjust Z t Optimize towards identity consistency and finally output Z t-1 ;L id () is the identity similarity loss function; Represents the identity similarity loss function L id For the diffusion latent variable Z t The partial derivative (i.e. gradient); η is the learning rate;

[0214] S53, optimize the latent variable Z t-1 Inject multi-frame autoregressive generator, through the optical flow constraint loss L flow And sliding window weighted fusion, finally output identity consistent high-quality video

[0215] Diffusion latent variable Z t Initialized to random Gaussian noise, multi-step iterative denoising is performed through the spatiotemporal DiT architecture, and Z is gradually updated t To approximate the real video distribution.

[0216] Z time It is the latent variable after the time dimension is expanded in the space-time DiT. t is the optimization subject of the diffusion model, Z time It is its spatiotemporal extension form, and the two work together to achieve temporal and spatial consistency in video generation.

[0217] The multi-frame autoregressive generator is the final module for video time series generation in the generation method of this embodiment, and its function is to generate the video time series based on the optimized diffusion latent variable Z. t-1 , generating video sequences frame by frame in an autoregressive manner, specifically including:

[0218] The multi-frame autoregressive generator gradually generates the optimized latent variable Z according to the sliding window t-1 The latent variable sequence of the note is generated. Each time a new window is generated, the weighted average fusion is performed with the overlapping area of ​​the previous window to generate the final latent variable Final latent variable Through the decoder module (VAE decoder) in the diffusion model framework, it is mapped to the pixel space and converted into a visual video frame; through the optical flow constraint loss L flow Enforce motion consistency between adjacent frames to reduce jitter; ultimately, stitch the frames generated by all windows together in chronological order, and output a complete video by smoothing the video frames in time, i.e., a high-quality video with consistent identity. The time series smoothing process is time domain average pooling;

[0219] The optical flow constraint loss L flow The calculation formula is as follows:

[0220]

[0221] The Flow() function is used to calculate the optical flow between adjacent frames and output a pixel-level motion vector field to represent the direction and amplitude of object motion between frames (such as changes in facial micro-expressions between adjacent frames).

[0222] The Warp() function performs spatiotemporal warping on the latent variables based on the optical flow field. It maps the latent variables of the current frame to the coordinate system of the next frame according to the optical flow, forces the feature spaces of adjacent frames to be aligned, and reduces timing jitter.

[0223] It is understandable that Z in the formula i , Z i+1 To represent the generated optimized latent variable Z t-1 Process latent variables;

[0224] The sliding window gradually generates the final latent variable Specifically, it includes: step generation with 4 frames as the window, and weighted average of overlapping areas to generate the final latent variable

[0225]

[0226] where Z new The new latent variable sequence generated for the current sliding window represents the latest generated 4 frames of latent features;

[0227] Z prev The latent variable sequence generated for the previous window is used for weighted fusion of the overlapping area with the current window;

[0228] is the final latent variable from time window t to t+3, the optimization result after optical flow constraint and weighted averaging;

[0229] α is a fusion weight parameter, and its value is (0, 1). In this embodiment, the preferred value is 0.7;

[0230] Finally, the frames generated by all windows are spliced ​​in chronological order, and the complete video is output through temporal smoothing processing (such as temporal average pooling);

[0231] The final latent variable The latent space feature sequence used to represent the video frame needs to be mapped to the pixel space through the decoder module (VAE decoder) in the diffusion model framework and converted into a visual video frame.

[0232] Specifically, all windows generate After the sequences are spliced ​​in chronological order, they are input into the decoder for feature reconstruction to generate original pixel-level video frames; then, inter-frame jitter is further eliminated through temporal smoothing processing (such as temporal average pooling), and finally high-quality video is output.

[0233] The number of video frames generated in a single time by the multi-frame autoregressive generator is N;

[0234] Temporal Average Pooling (TAP) is a common operation in time series data processing, particularly in fields like video and audio processing. Its basic idea is to reduce the dimensionality of data along the time dimension, averaging the features within a specific time window to obtain a representative feature vector. This is a prior art technique and will not be further described here.

[0235] This embodiment uses a wavelet multi-level frequency band decoupling-fusion framework and a dynamic cross-modal attention mechanism, combined with the dynamic alignment mechanism of the distributed perception adapter and the HJB optimization strategy of hierarchical diffusion, to achieve identity representation fusion across time and space dimensions while ensuring parameter efficiency. It solves the problems of high-frequency detail drift, identity distortion in dynamic scenes, and poor stability of long sequence generation in traditional video generation methods, significantly improves the quality and stability of generated videos, and provides more advanced technical support for video processing and related applications.

[0236] Example 2

[0237] refer to Figure 2 This embodiment provides a consistent video generation device based on wavelet transform and frequency decomposition, including the following units:

[0238] The wavelet feature extraction unit extracts the low-frequency contour features and high-frequency texture features of the reference image based on the frequency band decoupling of the multi-scale Haar wavelet decomposition, obtains the text embedding of the text prompt through the CLIP text encoder, and performs cross-modal attention on the low-frequency contour features, high-frequency texture features, and text embedding to obtain the multi-scale wavelet feature F wavelet ;

[0239] Cross-scale fusion unit, used to fuse multi-scale wavelet features F through frequency band adaptive convolution and inverse wavelet reconstruction wavelet The multi-level features of the image are outputted by retaining cross-scale details. fused ;

[0240] The face embedding generation unit is used to embed the face of the reference image obtained by the ArcFace network into E arc The global context feature E of the reference image obtained by CLIP-ViT encoder clip Generating refined facial embeddings E by Q-Former fusion final ;

[0241] Alignment latent variable generation unit, used to diffuse latent variable Z t , fusion image features F fused and identity embedding E final Input to the dual-path cross attention module to align the identity distribution of the latent variable space and the temporal smoothing constraint to obtain the aligned latent variable Z aligned ;

[0242] The consistent video generation unit is used to align the latent variables Z aligned With text embedded E text Injecting the spatiotemporal DiT architecture to generate preliminary temporal latent variables Z time , and the diffusion latent variable Z t Generate the optimized latent variable Z through Hamilton-Jacobi optimization t-1 , and then the preliminary time series latent variable Z time and optimize the latent variable Z t-1 Generate high-quality video V via a multi-frame autoregressive generator output .

[0243] Example 3

[0244] refer to Figure 3 , Figure 3 2 is a schematic diagram of the structure of a consistent video generation device based on wavelet transform and frequency decomposition according to this embodiment. This consistent video generation device 20 based on wavelet transform and frequency decomposition includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps of the aforementioned method embodiments. Alternatively, when the processor 21 executes the computer program, it implements the functions of the modules / units in the aforementioned apparatus embodiments.

[0245] Exemplarily, the computer program can be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to implement the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program in the consistent video generation device 20 based on wavelet transform and frequency decomposition. For example, the computer program can be divided into the modules described in Example 2. The specific functions of each module can be referred to the operating process of the device described in the above embodiment, and will not be repeated here.

[0246] The consistent video generation device 20 based on wavelet transform and frequency decomposition may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the consistent video generation device 20 based on wavelet transform and frequency decomposition, and does not constitute a limitation of the consistent video generation device 20 based on wavelet transform and frequency decomposition. The consistent video generation device 20 based on wavelet transform and frequency decomposition may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the consistent video generation device 20 based on wavelet transform and frequency decomposition may also include input and output devices, network access devices, buses, etc.

[0247] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 21 is the control center of the consistent video generation device 20 based on wavelet transform and frequency decomposition, and utilizes various interfaces and lines to connect the various parts of the consistent video generation device 20 based on wavelet transform and frequency decomposition.

[0248] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements the various functions of the consistent video generation device 20 based on wavelet transform and frequency decomposition by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory 22 can include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0249] If the modules / units integrated in the wavelet transform and frequency decomposition-based consistent video generation device 20 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the above-mentioned embodiment methods by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor 21, it can implement the steps of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0250] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0251] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 Multiple processes and / or boxes Figure 1 A device that performs the functions specified in one or more boxes.

[0252] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0253] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0254] The parts of the present invention that are not described in detail are prior art. It is obvious to those skilled in the art that the present invention is not limited to the details of the above-mentioned exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and all changes that fall within the meaning and scope of equivalent elements are intended to be included in the present invention.

Claims

1. A consistent video generation method based on wavelet transform and frequency decomposition, characterized in that: The following steps are involved: S1. Based on the frequency band decoupling of multi-scale Haar wavelet decomposition, the low-frequency contour features and high-frequency texture features of the reference image are extracted, and the text embedding of the text prompt is obtained through the CLIP text encoder. The low-frequency contour features, high-frequency texture features, and text embedding are subjected to cross-modal attention to obtain the multi-scale wavelet feature F. wavelet ; S2, through the frequency band adaptive convolution module and inverse wavelet reconstruction fusion multi-scale wavelet features F wavelet The multi-level features of the image are outputted by retaining cross-scale details. fused ; S3, embed the face of the reference image obtained by the ArcFace network into E arc The global context feature E of the reference image obtained by CLIP-ViT encoder clip Generating refined facial embedding E by Q-Former fusion final ; S4, diffuse latent variable Z t , fusion image features F fused and identity embedding E final Input to the dual-path cross attention module to align the identity distribution of the latent variable space and the temporal smoothing constraint to obtain the aligned latent variable Z aligned ; Step S4 specifically includes the following steps: S41, diffuse latent variable Z t Inject the two-way cross attention module, respectively with the fusion image feature F fused Interactively obtain the image feature diffusion variable Z img , and the identity embedding E final Interaction to obtain identity embedding diffusion variable Z face The dual-path cross attention module includes image path cross attention and identity path cross attention; S42, diffuse the image feature variable Z img and identity embedding diffusion variable Z face Input to the dynamic distribution alignment module, first calculate the image feature mean μ through the statistical calculation formula img and identity embedding mean μ face and the image feature standard deviation σ img and identity embedding standard deviation σ face , the projection distribution is obtained through the distribution projection formula Fuse the residuals and add the gate weight γ to output the aligned latent variable Z aligned ; S43, to align the latent variable Z aligned Add a time domain smoothing constraint: The time domain smoothing constraint formula is as follows: Among them, L align is a loss function with time domain smoothness constraint, which is used to measure the alignment of latent variables. The difference between its average pooling results at different time steps; To align the latent variable Z aligned Representation at the tth time step; AvgPool() is the average pooling operation function; T is the total denoising steps of the diffusion model; S5. Align the latent variable Z aligned With text embedded E text Injecting the spatiotemporal DiT architecture to generate preliminary temporal latent variables Z time , and the diffusion latent variable Z t Generate the optimized latent variable Z through Hamilton-Jacobi optimization t-1 , and then the preliminary time series latent variable Z time and optimize the latent variable Z t-1 Generate high-quality video V via a multi-frame autoregressive generator output .

2. The consistent video generation method based on wavelet transform and frequency decomposition according to claim 1 is characterized in that: Step S1 specifically includes the following steps: S11, the reference image Input multi-level Haar wavelet transform, and recursively decompose L levels to obtain low-frequency components and high-frequency components H is the image height, W is the image width; S12, frequency band feature reorganization, align and splice the low-frequency components of each layer through bicubic interpolation, and fuse them to generate the global low-frequency feature F low , sum the high-frequency components of each layer by weight according to the level to generate the local high-frequency feature F high ; S13, input the text prompt T into the CLIP text encoder, extract the text embedding E text , through the interaction of cross-modal attention and wavelet features, the multi-scale wavelet feature F is finally output wavelet =(F low ,F high ); where F low is the low-frequency feature; F high It is a high frequency feature.

3. The consistent video generation method based on wavelet transform and frequency decomposition according to claim 2, characterized in that: Step S2 specifically includes the following steps: S21, multi-scale wavelet feature F wavelet =(F low ,F high ) The frequency band adaptive convolution module is injected to perform hierarchical inverse reconstruction and inverse wavelet transform reconstruction to obtain the reconstructed feature R (i) ; S22, cross-level feature aggregation, reconstructing feature R through skip connection and hierarchical weighting (i) Perform aggregation and output fusion image feature F fused .

4. The consistent video generation method based on wavelet transform and frequency decomposition according to claim 2, characterized in that: Step S3 specifically includes the following steps: S31, reference image I ref Input to the ArcFace network, generate facial embedding E through multi-scale feature fusion and global average pooling arc ; S32, reference image I ref Input to CLIP-ViT encoder, after block embedding, position encoding and multi-layer Transformer, output global context feature E clip ; S33, embed the face into E arc and global context feature E clip Input to the Q-Former module, through the Q-Former module's learnable query matrix projection and cross attention calculation, a refined facial embedding E is generated. final .

5. The consistent video generation method based on wavelet transform and frequency decomposition according to claim 1 is characterized in that: Step S5 specifically includes the following steps: S51, align the latent variable Z aligned With text embedded E text Injecting the spatiotemporal DiT architecture, the preliminary temporal latent variable Z is generated by shallow low-frequency feature fusion and deep high-frequency cross attention time ; The shallow low-frequency feature fusion is performed by convolution after bilinear upsampling; S52, diffuse latent variable Z t Input Hamilton-Jacobi optimization module, reverse optimization based on identity similarity loss, and adjust the diffusion latent variable Z t The distribution of the output optimization latent variable Z t-1 ; S53, optimize the latent variable Z t-1 Inject multi-frame autoregressive generator, through the optical flow constraint loss L flow And sliding window weighted fusion, finally output identity consistent high-quality video 6. The consistent video generation method based on wavelet transform and frequency decomposition according to claim 2, characterized in that: Step S11 specifically includes the following steps: S111, wavelet transform initialization, input reference image With the text prompt T, a predefined Haar wavelet filter bank {f LL ,f LH ,f HL ,f HH },in: to I ref Perform the i-th level two-dimensional discrete wavelet transform: in The output resolution of each level is reduced to 1 / 2 of the previous level; the value range of i is 1 to L; stride = 2 means that the output resolution of each level is reduced to 1 / stride of the previous level; is the previous level image of the i-th level two-dimensional discrete wavelet transform; S112, multi-level recursive decomposition, low-frequency components Recursively perform L-level wavelet decomposition: Output multi-scale feature group in is the low-frequency component, is the high frequency component, and L is a positive integer greater than or equal to 3.

7. The consistent video generation method based on wavelet transform and frequency decomposition according to claim 2, characterized in that: Step S12 specifically includes the following steps: S121, global low-frequency feature F low Align and concatenate the low-frequency components of each layer through bicubic interpolation get: Among them, Concat() is the fusion function; Upsample() is the upsampling function; S122, local high-frequency feature F high Through the weighted fusion of high-frequency components of each layer: in is the decay weight, For channel splicing.

8. The consistent video generation method based on wavelet transform and frequency decomposition according to claim 1 is characterized in that: Step S13 specifically includes the following steps: S131, text conditional injection, input the text prompt T into the CLIP text encoder, generate text embedding after Tokenize and position encoding E text =CLIP text (T)=Transformer 12L (Embed(T)) CLIP text (T) indicates that the text prompt T is input into the text branch of the CLIP model for processing to obtain preliminary text features; Transformer 12L (Embed(T)) means that the text prompt T is converted into an embedding vector through Embed(T), and then the embedding vector is input into a 12-layer Transformer model for Tokenize and position encoding, and finally the text embedding of the text prompt T is obtained. Where N represents the number of video frames generated in a single shot in the multi-frame autoregressive generator, which is used to control the step size of the sliding window; d is the feature dimension of the diffusion latent variable, which is used to represent the spatial information of the compressed video frame; S132, E through cross-modal attention text Inject wavelet features and output multi-scale wavelet features F wavelet =(F low ,F high ): F wavelet =CrossAttn(Q=F low ‖F high ,K=V=E text ) Among them, CrossAttn is the cross-modal attention function, || represents splicing along the channel dimension, the number of attention heads is 8, and the scaling factor is Q is the query vector, which is composed of features of different scales of the image; K is the key vector; V is the value vector; F low is the low-frequency feature; F high It is a high frequency feature.

9. A consistent video generation device based on wavelet transform and frequency decomposition, characterized in that: Includes the following units: The wavelet feature extraction unit extracts the low-frequency contour features and high-frequency texture features of the reference image based on the frequency band decoupling of the multi-scale Haar wavelet decomposition, obtains the text embedding of the text prompt through the CLIP text encoder, and performs cross-modal attention on the low-frequency contour features, high-frequency texture features, and text embedding to obtain the multi-scale wavelet feature F wavelet ; Cross-scale fusion unit, used to fuse multi-scale wavelet features F through frequency band adaptive convolution and inverse wavelet reconstruction wavelet The multi-level features of the image are outputted by retaining cross-scale details. fused ; The face embedding generation unit is used to embed the face of the reference image obtained by the ArcFace network into E arc The global context feature E of the reference image obtained by CLIP-ViT encoder clip Generating refined facial embedding E by Q-Former fusion final ; Alignment latent variable generation unit, used to diffuse latent variable Z t , fusion image features F fused and identity embedding E final Input to the dual-path cross attention module to align the identity distribution of the latent variable space and the temporal smoothing constraint to obtain the aligned latent variable Z aligned ; The alignment latent variable generation unit specifically includes the following steps: S41, diffuse latent variable Z t Inject the two-way cross attention module, respectively with the fusion image feature F fused Interactively obtain the image feature diffusion variable Z img , and the identity embedding E final Interaction to obtain identity embedding diffusion variable Z face The dual-path cross attention module includes image path cross attention and identity path cross attention; S42, diffuse the image feature variable Z img and identity embedding diffusion variable Z face Input to the dynamic distribution alignment module, first calculate the image feature mean μ through the statistical calculation formula img and identity embedding mean μ face and the image feature standard deviation σ img and identity embedding standard deviation σ face , the projection distribution is obtained through the distribution projection formula Fuse the residuals and add the gate weight γ to output the aligned latent variable Z aligned ; S43, to align the latent variable Z aligned Add a time domain smoothing constraint: The time domain smoothing constraint formula is as follows: Among them, L align is a loss function with time domain smoothness constraint, which is used to measure the alignment of latent variables. The difference between its average pooling results at different time steps; To align the latent variable Z aligned Representation at the tth time step; AvgPool() is the average pooling operation function; T is the total denoising steps of the diffusion model; The consistent video generation unit is used to align the latent variables Z aligned With text embedded E text Injecting the spatiotemporal DiT architecture to generate preliminary temporal latent variables Z time , and the diffusion latent variable Z t Generate the optimized latent variable Z through Hamilton-Jacobi optimization t-1 , and then the preliminary time series latent variable Z time and optimize the latent variable Z t-1 Generate high-quality video V via a multi-frame autoregressive generator output .

Images