Personalized customized video object replacement method capable of maintaining layout and motion modes

Abstract

The invention discloses a personalized customized video object replacement method capable of maintaining layout and motion modes, which comprises the following steps of: 1) acquiring a reference image of a target object and preprocessing the reference image to obtain text description and a target object mask so as to construct an image data set; 2) finely adjusting the video diffusion model, and optimizing text embedding and attention layer weight of the model, so that the model learns to generate a video containing a target object; 3) inputting a to-be-edited original video into the model to obtain space attention features and motion features required in an object replacement process; and 4) under the guidance of the spatial attention features and the motion features, sampling by the fine-tuned video diffusion model to generate a video after object replacement. According to the method, different parameters of the model are optimized in a combined manner, the visual features of the target object are effectively reserved, and meanwhile, motion feature guidance and space attention feature injection are combined in the object replacement process, so that the video after object replacement better retains the original video space layout and motion mode.

Description

Technical Field

[0001] The present invention belongs to the field of video generation, and in particular relates to a personalized customized video object replacement method that maintains layout and motion mode. Background Art

[0002] Due to the rapid development of diffusion models, video editing methods based on diffusion models have become an active research field. Most existing editing methods are text-driven, so users can only specify the modified content through text and cannot fine-tune the generated content, which limits the user's ability to generate content that meets their needs. Subject-driven video editing has been proposed to allow users to specify the modified content more flexibly by providing a target subject image. This method is more intuitive and user-friendly. The present invention focuses on a subtask of subject-driven video editing - personalized video subject replacement, which aims to generate personalized video content by replacing the subject in the input video with the target subject specified by the user.

[0003] Personalized video subject replacement can be viewed as a combination of subject learning and video editing. For subject learning, a range of fine-tuning methods exist that optimize different parameter spaces, including text embeddings and base model weights, to associate a specific concept or subject with a particular tag. For video editing, the current major challenge is to capture and preserve the dominant motion patterns in the source video. Most existing methods rely on text-to-image models and introduce extended self-attention modules to maintain temporal dependencies between frames. While this approach is effective in enforcing temporal coherence, it does not incorporate general motion knowledge and may fail to generate realistic motion patterns in some cases. Some work has also explored the use of additional motion representations to explicitly model motion patterns, including optical flow, edge / depth maps, and sparse semantic point correspondences. This requires additional processes to extract and inject these motion representations, which incurs additional overhead and has its own limitations.

[0004] Therefore, how to generate high-quality personalized edited videos while preserving the spatial layout and motion information of the original video as much as possible is still an underexplored problem. Summary of the Invention

[0005] The purpose of the present invention is to solve the problems existing in the prior art and provide a personalized customized video object replacement method that maintains layout and motion mode.

[0006] In order to achieve the above-mentioned object of the invention, the present invention specifically adopts the following technical solutions:

[0007] A personalized video object replacement method that maintains layout and motion mode includes the following steps:

[0008] S1: Obtain a reference image of the target object and preprocess the reference image to obtain a text description of the reference image and a target object mask to construct an image dataset for fine-tuning the video diffusion model;

[0009] S2: Use the image dataset in S1 to fine-tune the video diffusion model, and optimize the text embedding and attention layer weights of the video diffusion model so that the video diffusion model learns to generate videos containing the target object;

[0010] S3: Input the original video to be edited into the fine-tuned video diffusion model to obtain the spatial attention features and motion features required for the object replacement process;

[0011] S4: Guided by the spatial attention features and motion features extracted in S3, the fine-tuned video diffusion model is used to sample and generate videos after object replacement.

[0012] Based on the above solution, each step can be implemented in the following preferred specific manner.

[0013] Preferably, in step S1, multiple reference images are obtained for each target object, a text description of each reference image is generated using an image description model, a target object mask of each reference image is generated using a segmentation model, and the image dataset is constructed by each reference image and its corresponding text description and target object mask.

[0014] Furthermore, in step S1, the BLIP-2 model is used as the image description model, and the Grounded SAM model is used as the segmentation model.

[0015] Preferably, in step S2, during the process of fine-tuning the video diffusion model, the text description of the reference image is used as a text condition, and the reference image is input into the video diffusion model as a video with a frame number of 1. The optimization parameters are the learnable text embedding and the spatial attention layer weights in the video diffusion model, and other weights remain fixed; wherein, the text embedding is two learnable embedding vectors, and the spatial attention layer performs parameter updates through low-rank adaptation.

[0016] Preferably, in step S2, when fine-tuning the video diffusion model, a Gaussian noise vector is first sampled from a normal distribution with a mean of 0 and a variance of 1, and then the latent variable at the noise level t and the text conditional representation after the fusion of the learnable text embedding are input into the video diffusion model to obtain a denoised feature vector, and the difference between the Gaussian noise vector and the denoised feature vector is calculated, and the difference calculation result is multiplied point by point with the target object mask and then the expectation is taken as the loss function; the latent variable is obtained by a pre-set forward diffusion process.

[0017] The functional form of the above loss function L(Δθ,e) is:

[0018]

[0019] z t =α t z+σ t ∈

[0020]

[0021] in, represents the expectation; z is a single reference image sampled from the image dataset; represents a normal distribution with mean 0 and variance 1; represents a Gaussian noise vector sampled from the normal distribution; is the number of denoising steps; In the set {1,2,…,T max Uniform distribution on T max represents the maximum number of denoising steps; m is the target object mask; ∈ θ represents the video diffusion model with weight θ; ∈ θ+Δθ represents the video diffusion model with weight θ+Δθ; z t is the latent variable when the noise level is t; e = [e1, e2] is a learnable text embedding, which consists of two learnable embedding vectors e1 and e2; y(e) represents the conditional representation of the text after the fusion text embedding e; represents the square of the L2 norm; α t and σ t is a predefined noise parameter.

[0022] Preferably, in step S3, the spatial attention features include the attention map in the spatial cross-attention layer and the output features in the spatial self-attention layer.

[0023] The calculation formulas for the attention map M in the spatial cross-attention layer and the output feature ψ in the spatial self-attention layer are:

[0024]

[0025] Where Q1 and K1 are the query and key in the spatial cross-attention layer, respectively; Q2, K2, and V2 are the query, key, and value in the spatial self-attention layer, respectively; Softmax(·) represents the Softmax activation function; d represents the feature dimension size of the spatial attention layer; and the superscript T represents transposition.

[0026] Preferably, in step S3, the motion features are extracted from the temporal attention layer of the video diffusion model, which is the principal component of the temporal attention map in the temporal attention layer; wherein the temporal attention map when the denoising step number is t With principal component mask Perform point-by-point multiplication to obtain the principal component H of the temporal attention map t , the calculation formula is as follows:

[0027]

[0028] Preferably, the principal component mask is used to extract the principal components in the temporal attention map, where each value is defined as: a row vector is formed by the values ​​of the p-th row and i-th column of all channels of the temporal attention map, and an index set is formed by the indices corresponding to the largest k values ​​in the row vector. When j belongs to the index set, the value of the j-th channel of the p-th row, i-th column of the principal component mask is 1, otherwise it is 0.

[0029] Preferably, the specific process of step S4 is as follows:

[0030] S41: When using the video diffusion model to sample and generate the video after object replacement, for the first T0 denoising steps, the spatial attention features calculated from the video after object replacement are replaced with the spatial attention features extracted in S3 to inject the layout information of the original video. By minimizing the difference between the motion features of the video after object replacement and the motion features of the original video extracted in S3, the sampling direction of the video after object replacement is changed to inject the motion information of the original video.

[0031] S42: For the remaining (T max -T0) denoising steps, the mask of the foreground object is obtained by the spatial cross attention layer in the video diffusion model, and the original video feature map output by the video diffusion model and the video feature map after the object is replaced are fused with the foreground object mask, so that the video after the object is replaced retains the background information of the original video. After completing all denoising steps, the video after the object is replaced is output; where T max Indicates the maximum number of denoising steps.

[0032] Preferably, in step S41, the principal component mask is multiplied point by point by the temporal attention map of the original video to obtain the motion features of the original video, and the principal component mask is multiplied point by point by the temporal attention map of the video after the object is replaced to obtain the motion features of the video after the object is replaced. The square of the L2 norm of the difference between the two motion features is calculated as the motion feature difference value, which is used to guide the video generation after the object is replaced in combination with the classifier-free guidance:

[0033]

[0034] in, represents the latent vector after denoising; s is the guidance strength without classifier guidance; v is the guidance strength of the motion feature difference value; Indicates that no text is entered. Represents the motion feature difference value g for the latent variable z t gradient.

[0035] The motion feature difference value g is calculated as follows:

[0036]

[0037] in, represents the temporal attention map from the original video; O t Represents the temporal attention map from the video after object replacement; Represents the square of the L2 norm.

[0038] Preferably, in step S42, the mask of the foreground object is used as a weight, and the original video feature map and the video feature map after object replacement are fused by weighted averaging, which is specifically expressed as follows:

[0039]

[0040] Among them, z final Represents the fused feature map.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] Compared with the existing technology, the present invention makes full use of the motion prior knowledge in the video diffusion model, so that the edited video (i.e., the video after object replacement) can accurately capture the visual characteristics of the target subject, faithfully preserve the spatial layout of the original video, the posture and main motion pattern of the original subject, while maintaining the integrity of the unmodified background area, significantly improving the fidelity and temporal coherence of the personalized edited video. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Schematic diagram of the steps of the method of the present invention;

[0044] Figure 2 is a flow chart of the method of the present invention;

[0045] Figure 3 This figure shows some results of the embodiments of the present invention. DETAILED DESCRIPTION

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in the various embodiments of the present invention can be combined accordingly without conflicting with each other.

[0047] Since video diffusion models are typically trained on large-scale video datasets and have a better understanding of motion priors, we chose to use them as the base model in this paper. Furthermore, to better preserve the spatial layout and motion information of the original video, we combined spatial feature injection and motion feature guidance during the editing process.

[0048] like Figure 1 As shown, in a preferred implementation of the present invention, the above-mentioned personalized customized video object replacement method maintaining layout and motion mode includes the following steps S1 to S4. The specific implementation process is described in detail below.

[0049] S1: Obtain a reference image of the target object and preprocess the reference image to obtain a text description of the reference image and a target object mask to construct an image dataset for fine-tuning the video diffusion model.

[0050] It should be noted that in step S1 of the present invention, multiple reference images are acquired for each target object, a text description for each reference image is generated using an image description model, and a target object mask for each reference image is generated using a segmentation model. The image dataset is constructed from each reference image and its corresponding text description and target object mask. Both the image description model and the segmentation model can be selected by those skilled in the art based on actual needs.

[0051] As a preferred implementation method of an embodiment of the present invention, 4 to 6 reference images are obtained for each target object, and then the BLIP-2 model is used as the image description model to generate a text description for each reference image. The Grounded SAM model is used as the segmentation model to generate a target object mask for each reference image. An image dataset is constructed from each reference image and its corresponding text description and target object mask.

[0052] It should be noted that in the present invention, the implementation of the video diffusion model belongs to the prior art, and those skilled in the art can select an appropriate video diffusion model as needed. As a preferred implementation of the embodiment of the present invention, the video diffusion model uses the VideoCrafter2 model.

[0053] S2: Use the image dataset in S1 to fine-tune the video diffusion model, and simultaneously optimize the text embedding and attention layer weights of the video diffusion model so that the video diffusion model learns to generate videos containing the target object.

[0054] As a preferred implementation of the embodiment of the present invention, in the process of fine-tuning the video diffusion model of the present invention, the text description of the reference image can be used as a text condition, and the reference image can be regarded as a video with a frame number of 1 and input into the video diffusion model. The optimization parameters are the learnable text embedding and the spatial attention layer in the video diffusion model ( Figure 2 The spatial attention module in , and other weights remain unchanged, such as Figure 2 The temporal attention module and convolutional layer in

[15] have fixed weights. The text embedding is two learnable embedding vectors, and the spatial attention layer updates its parameters via low-rank adaptation (LoRA).

[0055] It should be noted that in step S2 of the present invention, when fine-tuning the video diffusion model, a Gaussian noise vector is first sampled from a normal distribution with a mean of 0 and a variance of 1. The latent variable at the noise level t and the text conditional representation after integrating the learnable text embedding are then input into the video diffusion model to obtain a denoised feature vector. The difference between the Gaussian noise vector and the denoised feature vector is calculated, and the difference calculation result is multiplied point by point with the target object mask and then the expectation is taken as the loss function L(Δθ,e). The corresponding function form is:

[0056]

[0057] in, represents the expectation; z is a single reference image sampled from the image dataset; represents a normal distribution with mean 0 and variance 1; represents a Gaussian noise vector sampled from the normal distribution; is the number of denoising steps; In the set {1,2,…,T max Uniform distribution on T max represents the maximum number of denoising steps; m is the target object mask; ∈ θ represents the video diffusion model with weight θ; ∈ θ+Δθ represents the video diffusion model with weight θ+Δθ; z t is the latent variable when the noise level is t; e = [e1, e2] is a learnable text embedding, which consists of two learnable embedding vectors e1 and e2; y(e) represents the conditional representation of the text after the fusion text embedding e; represents the square of L2 norm;

[0058] Latent variable z t Obtained by the pre-set forward diffusion process:

[0059] z t =α t z+σ t ∈

[0060]

[0061] Among them, α t and σ t is a predefined noise parameter.

[0062] It should be noted that if Figure 2 As shown, in this embodiment of the present invention, step S2 corresponds to the first stage, and the reference image used for training is added with noise to obtain the latent variable z with noise. t , by fine-tuning the video diffusion model to make it learn according to the latent variable z t The added noise is predicted and the denoised image is obtained. After fine-tuning, the video diffusion model can generate videos containing the target object based on the learned text embeddings.

[0063] S3: Input the original video to be edited into the fine-tuned video diffusion model to obtain the spatial attention features and motion features required for the object replacement process.

[0064] It should be noted that, for step S3 of the present invention, the spatial attention feature includes the attention map M in the spatial cross attention layer and the output feature ψ in the spatial self-attention layer, and the calculation formula is as follows:

[0065]

[0066] Where Q1 and K1 are the query and key in the spatial cross-attention layer, respectively; Q2, K2, and V2 are the query, key, and value in the spatial self-attention layer, respectively; Softmax(·) represents the Softmax activation function; d represents the feature dimension size of the spatial attention layer; and the superscript T represents transposition.

[0067] The motion feature is extracted from the temporal attention layer of the video diffusion model. Specifically, the motion feature H is the principal component of the temporal attention map in the temporal attention layer. For the temporal attention map with denoising steps t Combine it with the principal component mask Perform point-by-point multiplication to obtain a temporal attention map with denoising steps of t The principal component H t , the calculation formula is as follows:

[0068]

[0069] in, represents a set of real numbers; h and w represent the height and width of the temporal attention map respectively; f represents the number of original video frames.

[0070] The principal component mask Used to extract the principal components in the temporal attention map, where each value is defined as: a row vector consisting of the values ​​of the pth row and the ith column of all channels in the temporal attention map The index set is composed of the indices corresponding to the largest k values ​​in the row vector when When the value of the jth channel in the pth row, ith column of the principal component mask is is 1, otherwise it is 0, that is:

[0071]

[0072] In the present invention, the indexes corresponding to one or more values ​​in the row vector can be selected to form an index set. In other words, the number of indexes included in the index set can be specified by those skilled in the art based on actual needs. In this embodiment, k=1 is selected, that is, the index set is composed only of the index of the maximum value in the row vector.

[0073] S4: Guided by the spatial attention features and motion features extracted in S3, the fine-tuned video diffusion model is used to sample and generate videos after object replacement.

[0074] As a preferred implementation of the embodiment of the present invention, Figure 2 As shown, the specific process of step S4 is as S41-S42:

[0075] S41: When using the video diffusion model to sample and generate the video after object replacement, for the first T0 denoising steps, the spatial attention features extracted in S3 are used to replace the spatial attention features calculated from the video after object replacement to inject the layout information of the original video. By minimizing the difference between the motion features of the video after object replacement and the motion features of the original video extracted in S3, the sampling direction of the video after object replacement is changed to inject the motion information of the original video.

[0076] It should be noted that both spatial attention feature replacement and motion feature guidance are performed only in the early denoising steps. In this embodiment, the maximum number of denoising steps is set to 50, which means that the above two processes can be performed in the first 10 or 20 denoising steps.

[0077] It should be noted that in step S41 of the present invention, the principal component mask is multiplied point by point with the temporal attention map of the original video to obtain the motion features of the original video, and the principal component mask is multiplied point by point with the temporal attention map of the video after the object is replaced to obtain the motion features of the video after the object is replaced. The square of the L2 norm of the difference between the two motion features is calculated as the motion feature difference value g, which is calculated as follows:

[0078]

[0079] in, represents the temporal attention map from the original video; O t Represents the temporal attention map from the video after object replacement; Represents the square of the L2 norm.

[0080] Motion feature difference values ​​combined with classifier-free guidance to guide object replacement video generation:

[0081]

[0082] in, represents the latent vector after denoising; s is the guidance strength without classifier guidance; v is the guidance strength of the motion feature difference value; Indicates that no text is entered. Represents the motion feature difference value for the latent variable z t gradient.

[0083] S42: For the remaining (T max -T0) denoising steps, the spatial cross-attention layer in the video diffusion model obtains the mask of the foreground object, and the feature map of the original video and the feature map of the video after the object replacement are fused with the mask of the foreground object. This allows the video after the object replacement to retain the background information of the original video. After completing all denoising steps, the video after the object replacement is output.

[0084] It should be noted that in step S42 of the present invention, the mask of the foreground object is As the weight, the original video feature map z is used src and the video feature map z after object replacement tgt The fusion is performed in a weighted average manner, which is specifically expressed as follows:

[0085]

[0086] Among them, z final Represents the fused feature map.

[0087] The present invention will now use a specific example to demonstrate the application effect of the personalized video object replacement method that maintains layout and motion pattern described in S1 to S4 of the above embodiments on a specific data set, so as to facilitate understanding of the essence of the present invention.

[0088] Example

[0089] The specific implementation process of the personalized customized video object replacement method for maintaining layout and motion mode adopted in this embodiment is as described above and will not be repeated here.

[0090] This example collected images of 15 different target subjects, including 8 animals and 7 objects. Each subject contained 4 to 6 reference images for training. In addition, 20 original videos to be edited were collected, each containing 16 frames and a resolution of 512×320.

[0091] like Figure 1 As shown, this example follows the process described in S1 to S4 above, fine-tunes the video diffusion model, and then performs object replacement editing on the original video. Some of the editing results are shown in Figure 3 As shown. The experimental results show that the present invention achieves two main goals: (1) accurately capturing the visual features of the target subject, and (2) faithfully preserving the spatial layout of the original video, the posture and main motion pattern of the original subject, while maintaining the integrity of the unmodified background area. This proves the effectiveness of the present invention in generating personalized videos with high fidelity and temporal coherence. In addition, this embodiment also quantitatively evaluates the performance of the model, and the results are shown in Table 1. Among them, DreamSwap represents the method of the present invention. It can be seen from Table 1 that the method of the present invention achieves better video-text matching and temporal consistency, and reaches a comparable level with the VideoSwap method in terms of visual fidelity of the target object.

[0092] Table 1 Quantitative experimental results

[0093]

[0094] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A personalized video object replacement method that maintains layout and motion pattern, characterized in that: The following steps are involved: S1: Obtain a reference image of the target object and preprocess the reference image to obtain a text description of the reference image and a target object mask to construct an image dataset for fine-tuning the video diffusion model; S2: Use the image dataset in S1 to fine-tune the video diffusion model, and optimize the text embedding and attention layer weights of the video diffusion model so that the video diffusion model learns to generate videos containing the target object; S3: Input the original video to be edited into the fine-tuned video diffusion model to obtain the spatial attention features and motion features required for the object replacement process; S4: Guided by the spatial attention features and motion features extracted in S3, the fine-tuned video diffusion model is used to sample and generate videos after object replacement.

2. The method for replacing a personalized video object while maintaining layout and motion pattern according to claim 1, wherein: In step S1, multiple reference images are obtained for each target object, a text description of each reference image is generated using an image description model, a target object mask of each reference image is generated using a segmentation model, and the image dataset is constructed by each reference image and its corresponding text description and target object mask.

3. The method for replacing a personalized video object while maintaining layout and motion pattern according to claim 1, wherein: In step S2, during the fine-tuning of the video diffusion model, the text description of the reference image is used as a text condition, and the reference image is input into the video diffusion model as a video with a frame number of 1. The optimization parameters are the learnable text embedding and the spatial attention layer weights in the video diffusion model, and other weights remain fixed; among them, the text embedding is two learnable embedding vectors, and the spatial attention layer updates its parameters through low-rank adaptation.

4. The method for replacing a personalized video object while maintaining layout and motion pattern according to claim 3, wherein: In step S2, when fine-tuning the video diffusion model, a Gaussian noise vector is first sampled from a normal distribution with a mean of 0 and a variance of 1, and then the latent variable at the noise level t and the text conditional representation after integrating the learnable text embedding are input into the video diffusion model to obtain a denoised feature vector, and the difference between the Gaussian noise vector and the denoised feature vector is calculated. The difference calculation result is multiplied point by point with the target object mask and then the expectation is taken as the loss function; the latent variable is obtained by a pre-set forward diffusion process.

5. The personalized video object replacement method for maintaining layout and motion pattern according to claim 1, characterized in that: In step S3, the spatial attention features include the attention map in the spatial cross-attention layer and the output features in the spatial self-attention layer.

6. The personalized video object replacement method for maintaining layout and motion pattern according to claim 5, characterized in that: In step S3, the motion features are extracted from the temporal attention layer of the video diffusion model and are the principal components of the temporal attention map in the temporal attention layer; wherein, the temporal attention map when the denoising step number is t is multiplied point by point with the principal component mask to obtain the principal components of the temporal attention map.

7. The method for replacing a personalized video object while maintaining layout and motion pattern according to claim 6, wherein: The principal component mask is used to extract the principal components in the temporal attention map, where each value is defined as: a row vector is formed by the values ​​of the p-th row and i-th column of all channels of the temporal attention map, and an index set is formed by the indices corresponding to the largest k values ​​in the row vector. When j belongs to the index set, the value of the j-th channel of the p-th row, i-th column of the principal component mask is 1, otherwise it is 0.

8. The method for replacing a personalized video object while maintaining layout and motion pattern according to claim 6, wherein: The specific process of step S4 is as follows: S41: When using the video diffusion model to sample and generate the video after object replacement, for the first T0 denoising steps, the spatial attention features calculated from the video after object replacement are replaced with the spatial attention features extracted in S3 to inject the layout information of the original video. By minimizing the difference between the motion features of the video after object replacement and the motion features of the original video extracted in S3, the sampling direction of the video after object replacement is changed to inject the motion information of the original video. S42: For the remaining (T max -T0) denoising steps, the mask of the foreground object is obtained by the spatial cross attention layer in the video diffusion model, and the original video feature map output by the video diffusion model and the video feature map after the object is replaced are fused with the foreground object mask, so that the video after the object is replaced retains the background information of the original video. After completing all denoising steps, the video after the object is replaced is output; where T max Indicates the maximum number of denoising steps.

9. The personalized video object replacement method for maintaining layout and motion pattern according to claim 8, characterized in that: In step S41, the principal component mask is multiplied point by point with the temporal attention map of the original video to obtain the motion features of the original video, and the principal component mask is multiplied point by point with the temporal attention map of the video after object replacement to obtain the motion features of the video after object replacement. The square of the L2 norm of the difference between the two motion features is calculated as the motion feature difference value, which is used to generate the video after object replacement in combination with the guidance of no classifier.

10. The personalized video object replacement method with layout and motion pattern preservation according to claim 8, characterized in that: In step S42, the mask of the foreground object is used as a weight, and the original video feature map and the video feature map after object replacement are fused by taking a weighted average.

Images