A method for synthesizing multiple video objects

Abstract

The present application discloses a method for synthesizing multiple video objects, which relates to the field of image generation technology. The original object video is tracked by an image segmentation algorithm and a video object tracking algorithm, and the target object video existing in the original object video is extracted. Then, the target object video and the required background video are inverted by a DDIM inversion method to obtain an inversion result. The target object video and the required background video are imported into a visual editing interface for editing and exporting a layout video. Finally, the inversion result is used to reconstruct the video, and a video generation model is started to process the layout video to obtain a video containing the object and the background. The video generation is guided by the object video and the background video to generate a high-quality synthesized video that retains the appearance and motion characteristics of the original video object and has a physical interaction effect, and has strong controllability.

Description

Technical Field

[0001] The present application relates to the technical field of image generation, and in particular to a method for synthesizing multiple video objects. Background Art

[0002] Video Object Composition is an advanced video processing technology that aims to synthesize objects or elements from different video sources into a new video stream. This technology is widely used in virtual reality (VR), augmented reality (AR), advertising production, movie special effects, and video editing. Through video object synthesis, functions such as virtual scene reconstruction, object replacement, and dynamic background replacement can be achieved.

[0003] Video generation models are developing rapidly. Video generation technology based on denoising models has become relatively mature. It can generate high-quality videos through text or pictures at a low cost and has been widely used in video content production.

[0004] With the development and improvement of technologies such as target detection, semantic segmentation and video tracking, the technology for extracting target objects in different videos has become very mature. However, synthesizing multiple video objects on the same background to obtain a new video often requires dealing with problems such as color matching and light and shadow effects. The commonly used method is to manually adjust the color and add shadows, which is costly and ineffective. Summary of the invention

[0005] The purpose of this application is to provide a multi-video object synthesis method, device, equipment and storage medium in order to overcome the existing technical defects, and adopt a video generation model to solve the problems of high cost and poor effect of the current video fusion method in multi-video object synthesis.

[0006] The purpose of this application is achieved through the following technical solutions:

[0007] In a first aspect, the present application proposes a method for synthesizing multiple video objects, the method comprising:

[0008] Using image segmentation algorithm and video object tracking algorithm to track the original object video and extract the target object video existing in the original object video;

[0009] The DDIM inversion method is used to invert the target object video and the required background video to obtain the inversion result;

[0010] Import the target object video and the required background video into the visual editing interface for editing, and export the layout video;

[0011] The inversion result is used to reconstruct the video, and the video generation model is started to process the layout video to obtain a video containing the object and the background.

[0012] In a possible implementation, the steps of tracking the original object video and extracting the target object video in the original object video by using an image segmentation algorithm and a video object tracking algorithm include:

[0013] An image segmentation algorithm is used to perform image segmentation on the original object video to obtain a target object mask of a key frame of the original object video;

[0014] A video object tracking algorithm is used to track the target object mask to obtain a temporal mask of the target object;

[0015] A target object video is extracted from an original object video based on a temporal mask of the target object.

[0016] In a possible implementation, the step of using the DDIM inversion method to invert the target object video and the required background video to obtain an inversion result includes:

[0017] Use a text feature extractor to process the text prompt to obtain text features;

[0018] Use an image feature extractor to extract the first frame of the target object video and the required background video to obtain image features;

[0019] Using a video encoder to process the target object video and the required background video to obtain latent space features;

[0020] Input text features, image features and latent space features into the video denoising model to predict noise and obtain noise;

[0021] The inverse sampler is used to add noise to the latent space feature to obtain the second latent space feature. The second latent space feature, text feature and image feature are input into the video denoising model again for noise prediction, and the inversion result is obtained by iterating until the preset iteration threshold is reached.

[0022] In a possible implementation, the step of importing the target object video and the required background video into the visual editing interface and exporting the layout video includes:

[0023] Import the target object video and the required background video into the visual editing interface for editing to obtain the layer position relationship;

[0024] Export layout video based on layer position relationship.

[0025] In a possible implementation, the step of using the inversion result to reconstruct the video and starting the video generation model to process the layout video to obtain a video containing the object and the background includes:

[0026] The inversion result is used to reconstruct the video and obtain the intermediate features in the video reconstruction;

[0027] Generate an image of the first frame of the layout video to obtain a first frame image including the object and the background;

[0028] Guided by the first frame image containing the object and the background, the intermediate features in the video reconstruction are combined and the video generation model is started to obtain the video containing the object and the background.

[0029] In a possible implementation, the step of performing video reconstruction using the inversion result to obtain intermediate features in video reconstruction includes:

[0030] Use a text feature extractor to process the text prompt to obtain text features;

[0031] Use an image feature extractor to extract the first frame of the target object video to obtain image features;

[0032] The inversion result is used as the third latent space feature, text feature, and image feature and inputted into the video denoising model for noise prediction to obtain the second noise;

[0033] The forward sampler is used to subtract the second noise from the third latent space feature to obtain the fourth latent space feature. The fourth latent space feature, text feature and image feature are input into the video denoising model again for noise prediction, and the process is iterated until the preset iteration threshold is reached to obtain the intermediate feature.

[0034] In a possible implementation, the step of performing image generation on the first frame of the layout video to obtain a first frame image including the object and the background includes:

[0035] Process the first frame mask of the layout video to obtain the minimum bounding rectangle of each object;

[0036] Expand the minimum bounding rectangle to determine the binary mask corresponding to the redraw area;

[0037] Using a text feature extractor to process the input text prompt word to extract a second text feature;

[0038] Using an image encoder to process the first frame of the layout video to obtain the fifth latent space feature;

[0039] Downsampling the binary mask according to the size of the fifth latent space feature to obtain a downsampled binary mask;

[0040] Inputting the second text feature, the fifth latent space feature, and the downsampled binary mask into the image denoising model to predict the third noise;

[0041] The forward sampler is used to subtract the third noise from the fifth latent space feature to obtain the sixth latent space feature, and the sixth latent space feature, the second text feature and the down-sampled binary mask are input into the image denoising model again for noise prediction, and the final denoising result is obtained by iterating until a preset iteration threshold is reached;

[0042] The final denoising result is processed using an image decoder to obtain the first frame image containing the object and the background.

[0043] In a possible implementation, the step of taking the first frame image containing the object and the background as a guide, combining the intermediate features in the video reconstruction and starting the video generation model to obtain the video containing the object and the background includes:

[0044] Using a text feature extractor to process the input text prompt word to extract a third text feature;

[0045] Using an image feature extractor to extract features from a first frame image containing an object and a background to obtain features of a second image;

[0046] Use the inversion result to initialize the latent space feature to obtain the seventh latent space feature;

[0047] Inputting the third text feature, the second image feature, the seventh latent space feature, and the intermediate feature into the video denoising model to predict a fourth noise;

[0048] Use a forward sampler to subtract the fourth noise from the seventh latent space feature to obtain an eighth latent space feature, input the eighth latent space feature, the third text feature, and the second image feature into the video denoising model, input the intermediate feature through the three-dimensional convolution module, the spatial self-attention module, and the temporal self-attention module in the video denoising model to perform noise prediction, and iterate in sequence until a preset iteration threshold is reached to obtain a second denoising result;

[0049] The second denoising result is processed using a video decoder to obtain a video containing the object and the background.

[0050] In a possible implementation, the calculation formula for initializing the latent space feature is: ,in Initialize the result of latent space features, is the background video inversion result, and is the inversion result of the two object videos, and is the timing mask of the two object videos relative to the re-layout video, is the randomly initialized latent space feature, is the latent space feature mixing coefficient.

[0051] In a possible implementation, the formula for the intermediate feature input is: ,in is the final output of the t-th denoising feature, is the intermediate feature of the t-th denoising of the background video, and is the intermediate feature of the t-th denoising of the two object videos, and is the timing mask of the two object videos relative to the re-layout video, is the original output of the t-th denoising feature, Inject coefficients into intermediate features.

[0052] The above main scheme of the present application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection in the present application; and in the present application, (non-conflicting options) options and other options can also be freely combined. After understanding the scheme of the present application, those skilled in the art can understand that there are many combinations based on the existing technology and common knowledge, all of which are technical schemes to be protected by the present application, and they are not exhaustively listed here.

[0053] The present application discloses a method for synthesizing multiple video objects, which relates to the field of image generation technology. The original object video is tracked by an image segmentation algorithm and a video object tracking algorithm, and the target object video existing in the original object video is extracted. Then, the target object video and the required background video are inverted by a DDIM inversion method to obtain an inversion result. The target object video and the required background video are imported into a visual editing interface for editing, and a layout video is exported. Finally, the inversion result is used for video reconstruction, and a video generation model is started to process the layout video to obtain a video containing the object and the background. Video generation is guided by object video and background video to generate a high-quality synthetic video that retains the appearance and motion characteristics of the original video object and has a physical interaction effect, and has strong controllability. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1A schematic flow chart of a multi-video object synthesis method proposed in an embodiment of the present application is shown.

[0056] Figure 2 A schematic diagram showing an example result of object and background video reorganization proposed in an embodiment of the present application is shown.

[0057] Figure 3 A schematic diagram of an example result of video fusion generation proposed in an embodiment of the present application is shown.

[0058] Figure 4 A schematic diagram showing an example result of first frame image generation proposed in an embodiment of the present application is shown.

[0059] Figure 5 A schematic diagram of the process of video fusion generation proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0060] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0061] Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of this application.

[0062] In the existing technology, with the development and improvement of technologies such as target detection, semantic segmentation and video tracking, the technology for extracting target objects in different videos has become very mature. However, synthesizing multiple video objects on the same background to obtain a new video often requires dealing with problems such as color matching and light and shadow effects. The commonly used method is to manually adjust the color and add shadows. This method is costly and ineffective.

[0063] In order to solve the above technical problems, the embodiment of the present application proposes a method for synthesizing multiple video objects, which adopts a video generation model to solve the problems of high cost and poor effect of the current video fusion method in the synthesis of multiple video objects. The object video and the background video are decomposed to guide the video generation, and a high-quality synthesized video is generated that retains the appearance and motion characteristics of the original video object and has a physical interaction effect. The creativity of the video generation model is fully utilized and has strong controllability, and can be widely used in video editing and video generation, which will be explained below.

[0064] Please refer to Figure 1 , Figure 1 A schematic diagram of a process flow of a multi-video object synthesis method proposed in an embodiment of the present application is shown, and the method includes:

[0065] Step S100: Track the original object video using an image segmentation algorithm and a video object tracking algorithm and extract the target object video in the original object video.

[0066] Multi-video object extraction: Use image segmentation techniques to identify and separate different objects in the original video. By analyzing the video frames, the foreground (i.e. the object of interest) and the background are distinguished. Subsequently, video object tracking methods are used to track the movement of these objects in the video sequence, and they can continue to be accurately identified even if the objects move, deform, or are occluded by other objects. The target objects are extracted separately from multiple videos, laying the foundation for subsequent processing.

[0067] Step S100 includes:

[0068] Step S110 uses an image segmentation algorithm to perform image segmentation on the original object video to obtain a target object mask of a key frame of the original object video;

[0069] Step S120: using a video object tracking algorithm to track the target object mask to obtain a temporal mask of the target object;

[0070] Step S130 extracts the target object video from the original object video based on the temporal mask of the target object.

[0071] The key frames of the original video are analyzed using the advanced SAM (Segement Anything Model) image segmentation algorithm, which can identify and mark the boundaries of different objects in the video frame to obtain an accurate mask of the target object.

[0072] Video object tracking: Using the Cutie video tracking algorithm, the obtained key frame target object mask is matched and tracked between consecutive frames. This ensures that even if the target object moves, deforms or is blocked throughout the video sequence, its temporal mask can be continuously identified and obtained to maintain the consistency of the target object.

[0073] Extract object video: Crop the video clip containing only the target object from the original video based on the obtained temporal mask. Each target object will be extracted independently to form a separate object video. This process may also include background removal, transparency processing, etc., to facilitate subsequent synthesis work.

[0074] Step S200: Invert the target object video and the required background video using the DDIM inversion method to obtain an inversion result.

[0075] DDIM (Denoising Diffusion Implicit Models) inversion is a diffusion model-based method that converts clear video frames into Gaussian noise distribution, and then generates video content through an inverse process (inversion). The object video and the selected background video extracted in step S200 are both inverted and converted into a noise form.

[0076] Step S200 includes:

[0077] S210, using a text feature extractor to process the text prompt to obtain text features;

[0078] S220, using an image feature extractor to extract the first frames of the target object video and the required background video to obtain image features;

[0079] S230, using a video encoder to process the target object video and the required background video to obtain latent space features;

[0080] S240, inputting text features, image features, and latent space features into a video denoising model to perform noise prediction to obtain noise;

[0081] S250, using an inverse sampler to add noise to the latent space feature to obtain a second latent space feature, and input the second latent space feature, text feature, and image feature into the video denoising model again for noise prediction, and iterate in sequence until a preset iteration threshold is reached to obtain an inversion result.

[0082] The preparation phase involves explicitly specifying that the object video and background video will be used as input for inversion, setting the preset text prompt to empty and inputting the target object video and the desired background video. Using text-guided inversion, a text feature extractor will be used to process the text and extract feature vectors that can guide the generation process. The image feature extractor processes the first frame of the target object video and the desired background video to capture preliminary information about the visual style or content, which will be used in subsequent steps together with the deep representation of the video.

[0083] Video encoder processing: Use a video encoder to convert the entire video to be inverted into a series of latent space features, which serve as a mapping from video to latent code, which carries the main content information of the video. The obtained text features (even if empty), image features, and latent space features are combined and input into the video denoising model to predict the corresponding Gaussian noise. Through the inverse sampler (such as the DDIM sampling process), the predicted noise is superimposed back to the latent space features, and this prediction and denoising process is iterated multiple times. Each iteration optimizes and refines the final inversion result until satisfactory output quality is achieved.

[0084] Step S300: import the target object video and the required background video into the visual editing interface for editing, and export the layout video.

[0085] Object and background video reorganization: Determine the spatial relationship by adjusting the previously extracted object videos, scaling, rotating, and translating them to fit the selected background video. Design the position and size of the object in the background to ensure visual consistency. Figure 2 A schematic diagram showing an example result of object and background video reorganization proposed in an embodiment of the present application is shown.

[0086] Step S300 includes:

[0087] Step S310: import the target object video and the required background video into the visual editing interface for editing to obtain the layer position relationship;

[0088] Step S320: export the layout video according to the layer position relationship.

[0089] After importing the object video and background video into the visual editing interface, you can freely adjust the video layers in the interface, that is, decide which objects appear in the front, which are in the back, and how they interact with each other. The object video is translated, rotated, and scaled to complete the rearrangement of the object and background videos. By adjusting the size, position, and transparency of each video, you can precisely control the layout of the object in the background, and record this layer position relationship after editing. After the layout is completed, two main products are exported according to the layer relationship set previously: one is the final layout video with all objects and background, and the other is the time-series mask sequence of the target object video relative to the layout video.

[0090] Step S400: reconstruct the video using the inversion result, and start the video generation model to process the layout video to obtain a video containing the object and the background.

[0091] Using advanced video generation models, the processed object video is fused with the background video. This process does not rely solely on simple superposition, but rather uses a deep learning model to understand and learn the complex relationship between the object and the background based on the information accumulated above, generating a new video containing multiple objects and a rich background. The generated video should have high resolution, natural motion smoothness, and seamless integration between the object and the background, creating a visual effect that is both realistic and creative. Figure 3 A schematic diagram of an example result of video fusion generation proposed in an embodiment of the present application is shown.

[0092] Step S400 includes:

[0093] Step S410 uses the inversion result to perform video reconstruction to obtain intermediate features in video reconstruction.

[0094] Using the Gaussian noise obtained by the previously mentioned DDIM or other inversion methods as a starting point, the inverse diffusion process is applied to the noise to gradually restore clear video frames. During this process, the system will continue to iterate and optimize, gradually resolving the video content from the noise to obtain the intermediate features generated in this reconstruction process.

[0095] Step S410 includes:

[0096] Use a text feature extractor to process the text prompt to obtain text features;

[0097] Use an image feature extractor to extract the first frame of the target object video to obtain image features;

[0098] The inversion result is used as the third latent space feature, text feature, and image feature and inputted into the video denoising model for noise prediction to obtain the second noise;

[0099] The forward sampler is used to subtract the second noise from the third latent space feature to obtain the fourth latent space feature. The fourth latent space feature, text feature and image feature are input into the video denoising model again for noise prediction, and the process is iterated until the preset iteration threshold is reached to obtain the intermediate feature.

[0100] First, the text prompt is preset to be empty, and the first frame of the object video is input. The text prompt is processed by the text feature extractor to extract text features, and the first frame of the object video is processed by the image feature extractor to extract image features. The inversion result of the corresponding video is used as the latent space feature input, and the text features, image features, and latent space features are input into the video denoising model to predict the noise. Finally, the noise is obtained by subtracting the video denoising model prediction from the input latent space features through the forward sampler, and the new latent space features, text features, and image features are input into the video denoising model for noise prediction and denoising again. Iterate multiple times and obtain the intermediate features obtained in each denoising process.

[0101] Taking two object videos as an example, the formula for the intermediate feature input is: ,in is the final output of the t-th denoising feature, is the intermediate feature of the t-th denoising of the background video, and is the intermediate feature of the t-th denoising of the two object videos, and is the timing mask of the two object videos relative to the re-layout video, is the original output of the t-th denoising feature, Inject coefficients into intermediate features.

[0102] Step S420 generates an image of the first frame of the layout video to obtain a first frame image including the object and the background.

[0103] Next, the first frame of the layout video is selected for processing, and the previously extracted and processed objects are superimposed on it using image generation technology. This step involves an image synthesis algorithm to ensure that the object blends naturally into the background while maintaining visual consistency. The generated first frame image is not only the starting point for the entire video synthesis, but also a reference for verifying the consistency between the input and output of the video generation model.

[0104] Step S420 includes:

[0105] Process the first frame mask of the layout video to obtain the minimum bounding rectangle of each object;

[0106] Expand the minimum bounding rectangle to determine the binary mask corresponding to the redraw area;

[0107] Using a text feature extractor to process the input text prompt word to extract a second text feature;

[0108] Using an image encoder to process the first frame of the layout video to obtain the fifth latent space feature;

[0109] Downsampling the binary mask according to the size of the fifth latent space feature to obtain a downsampled binary mask;

[0110] Inputting the second text feature, the fifth latent space feature, and the downsampled binary mask into the image denoising model to predict the third noise;

[0111] A forward sampler is used to subtract the third noise from the fifth latent space feature to obtain a sixth latent space feature, and the sixth latent space feature, the second text feature, and the downsampled binary mask are input into the image denoising model again.

[0112] Perform noise prediction and iterate in sequence until the preset iteration threshold is reached to obtain the final denoising result;

[0113] The final denoising result is processed using an image decoder to obtain the first frame image containing the object and the background.

[0114] Figure 4The schematic diagram of the example result of the first frame image generation proposed in the embodiment of the present application is shown. The mask of the first frame of the re-layout video is processed to obtain the minimum bounding rectangle of each object, and the minimum bounding rectangle is expanded to determine the area to be redrawn and obtain a binary mask with a value of 1 for the redrawn area and a value of 0 for the remaining area. Input the text prompt word, the first frame of the re-layout video and the binary mask. Use the text feature extractor to process the text prompt to extract the text feature. Use the image encoder to process the first frame of the re-layout video to obtain the latent space feature. The binary mask is directly downsampled to the same size as the latent space feature. The text feature, the latent space feature and the binary mask are input into the image denoising model to predict the noise. The noise is obtained by subtracting the image denoising model prediction from the input latent space feature through the forward sampler, and the new latent space feature, text feature and binary mask are input into the image denoising model for noise prediction and denoising again, and the final denoising result is obtained by iterating multiple times. The final denoising result is processed by the image decoder to obtain the first frame image containing the object and the background.

[0115] Step S430 uses the first frame image containing the object and the background as a guide, combines the intermediate features in the video reconstruction, and starts the video generation model to obtain a video containing the object and the background.

[0116] The video generation model starts working by using the first frame image containing the object and background as a guide and combining the intermediate features obtained during video reconstruction. The model will dynamically generate subsequent frames based on the scene layout set in the first frame and combined with time series information. Each frame ensures the coordination and consistency of the object and the background, as well as the overall smoothness and coherence of the video. At this stage, the model learns and simulates dynamic factors such as object movement and light and shadow changes, and finally outputs a complete video with both clear objects and rich backgrounds.

[0117] Step S430 includes:

[0118] Using a text feature extractor to process the input text prompt word to extract a third text feature;

[0119] Using an image feature extractor to extract features from a first frame image containing an object and a background to obtain features of a second image;

[0120] Use the inversion result to initialize the latent space feature to obtain the seventh latent space feature;

[0121] Inputting the third text feature, the second image feature, the seventh latent space feature, and the intermediate feature into the video denoising model to predict a fourth noise;

[0122] Use a forward sampler to subtract the fourth noise from the seventh latent space feature to obtain an eighth latent space feature, input the eighth latent space feature, the third text feature, and the second image feature into the video denoising model, input the intermediate feature through the three-dimensional convolution module, the spatial self-attention module, and the temporal self-attention module in the video denoising model to perform noise prediction, and iterate in sequence until a preset iteration threshold is reached to obtain a second denoising result;

[0123] The second denoising result is processed using a video decoder to obtain a video containing the object and the background.

[0124] Figure 5 The flowchart of the video fusion generation proposed in the embodiment of the present application is shown. First, input the text prompt word and the first frame image containing the object and the background. Use the text feature extractor to process the text prompt to extract the text feature. Use the image feature extractor to process the first frame image containing the object and the background to extract the image feature. Use the inversion results of the object video and the background video to initialize the latent space feature. The text features, image features and latent space features are directly input into the video denoising model, and the intermediate features are respectively injected from the three-dimensional convolution module, the spatial self-attention module and the temporal self-attention module to predict the noise. Finally, the noise is obtained by subtracting the prediction of the video denoising model from the input latent space features through the forward sampler, and the new latent space features, text features and image features are directly input into the video denoising model, and the intermediate features are injected from the three-dimensional convolution module, the spatial self-attention module and the temporal self-attention module. Noise prediction and denoising are performed again, and the final denoising result is obtained by multiple iterations.

[0125] The calculation formula for initializing the latent space feature is: ,in Initialize the result of latent space features. is the background video inversion result, and is the inversion result of the two object videos, and is the timing mask of the two object videos relative to the re-layout video, is the randomly initialized latent space feature, is the latent space feature mixing coefficient.

[0126] In one possible implementation, the video denoising model adopts a U-shaped network structure, which is divided into two parts: encoding and decoding, and consists of a three-dimensional convolution module, a cross-attention module, a spatial self-attention module, and a temporal self-attention module.

[0127] The image denoising model can also adopt a U-shaped network structure, which is divided into two parts: encoding and decoding, and consists of a two-dimensional convolution module, a cross-attention module, and a spatial self-attention module.

[0128] The intermediate features obtained in the denoising process also include the three-dimensional convolution module features, spatial self-attention module features, and temporal self-attention module features of the decoding part in the U-shaped network structure.

[0129] A possible implementation of a multi-video object synthesis device is provided below, which is used to execute the various execution steps and corresponding technical effects of the multi-video object synthesis method shown in the above embodiment and possible implementation. The multi-video object synthesis device includes:

[0130] A multi-video object extraction module is used to track the original object video and extract the target object video existing in the original object video by using an image segmentation algorithm and a video object tracking algorithm;

[0131] The object and background video inversion module is used to invert the target object video and the required background video using the DDIM inversion method to obtain an inversion result;

[0132] The object and background video reorganization module is used to import the target object video and the required background video into the visual editing interface for editing and export the layout video;

[0133] The video fusion generation module is used to reconstruct the video using the inversion result, and start the video generation model to process the layout video to obtain a video containing the object and the background.

[0134] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for synthesizing multiple video objects, characterized in that: The method comprises: Using image segmentation algorithm and video object tracking algorithm to track the original object video and extract the target object video existing in the original object video; The DDIM inversion method is used to invert the target object video and the required background video to obtain the inversion result; Import the target object video and the required background video into the visual editing interface for editing, and export the layout video; The inversion result is used to reconstruct the video and obtain the intermediate features in the video reconstruction; Process the first frame mask of the layout video to obtain the minimum bounding rectangle of each object; Expand the minimum bounding rectangle to determine the binary mask corresponding to the redraw area; Using a text feature extractor to process the input text prompt word to extract a second text feature; Using an image encoder to process the first frame of the layout video to obtain the fifth latent space feature; Downsampling the binary mask according to the size of the fifth latent space feature to obtain a downsampled binary mask; Inputting the second text feature, the fifth latent space feature, and the downsampled binary mask into the image denoising model to predict the third noise; The forward sampler is used to subtract the third noise from the fifth latent space feature to obtain the sixth latent space feature, and the sixth latent space feature, the second text feature and the down-sampled binary mask are input into the image denoising model again for noise prediction, and the final denoising result is obtained by iterating until a preset iteration threshold is reached; The final denoising result is processed using an image decoder to obtain a first frame image containing the object and the background; Using a text feature extractor to process the input text prompt word to extract a third text feature; Using an image feature extractor to extract features from a first frame image containing an object and a background to obtain features of a second image; The inversion result is used to initialize the latent space feature to obtain the seventh latent space feature. The calculation formula for initializing the latent space feature is: ,in Initialize the result of latent space features. is the background video inversion result, and is the inversion result of the two object videos, and is the timing mask of the two object videos relative to the re-layout video, is the randomly initialized latent space feature, is the latent space feature mixing coefficient; The third text feature, the second image feature, the seventh latent space feature and the intermediate feature are input into the video denoising model to predict the fourth noise. The formula for the intermediate feature input is: ,in is the final output of the t-th denoising feature, is the intermediate feature of the t-th denoising of the background video, and is the intermediate feature of the t-th denoising of the two object videos, and is the timing mask of the two object videos relative to the re-layout video, is the original output of the t-th denoising feature, Inject coefficients into intermediate features; Use a forward sampler to subtract the fourth noise from the seventh latent space feature to obtain an eighth latent space feature, input the eighth latent space feature, the third text feature, and the second image feature into the video denoising model, input the intermediate feature through the three-dimensional convolution module, the spatial self-attention module, and the temporal self-attention module in the video denoising model to perform noise prediction, and iterate in sequence until a preset iteration threshold is reached to obtain a second denoising result; The second denoising result is processed using a video decoder to obtain a video containing the object and the background.

2. The method for synthesizing multiple video objects as claimed in claim 1, characterized in that: The steps of tracking the original object video and extracting the target object video existing in the original object video by using the image segmentation algorithm and the video object tracking algorithm include: An image segmentation algorithm is used to perform image segmentation on the original object video to obtain a target object mask of a key frame of the original object video; A video object tracking algorithm is used to track the target object mask to obtain a temporal mask of the target object; A target object video is extracted from an original object video based on a temporal mask of the target object.

3. The method for synthesizing multiple video objects as claimed in claim 1, characterized in that: The steps of inverting the target object video and the required background video using the DDIM inversion method to obtain the inversion result include: Use a text feature extractor to process the text prompt to obtain text features; An image feature extractor is used to extract the first frame of the target object video and the required background video to obtain image features; Using a video encoder to process the target object video and the required background video to obtain latent space features; Input text features, image features and latent space features into the video denoising model to predict noise and obtain noise; The inverse sampler is used to add noise to the latent space feature to obtain the second latent space feature. The second latent space feature, text feature and image feature are input into the video denoising model again for noise prediction, and the inversion result is obtained by iterating until the preset iteration threshold is reached.

4. The method for synthesizing multiple video objects as claimed in claim 1, wherein: The steps of importing the target object video and the required background video into the visual editing interface for editing and exporting the layout video include: Import the target object video and the required background video into the visual editing interface for editing to obtain the layer position relationship; Export layout video based on layer position relationship.

5. The method for synthesizing multiple video objects as claimed in claim 1, characterized in that: The steps of using the inversion result to reconstruct the video and obtain the intermediate features in the video reconstruction include: Use a text feature extractor to process the text prompt to obtain text features; Use an image feature extractor to extract the first frame of the target object video to obtain image features; The inversion result is used as the third latent space feature, text feature, and image feature and inputted into the video denoising model for noise prediction to obtain the second noise; The forward sampler is used to subtract the second noise from the third latent space feature to obtain the fourth latent space feature. The fourth latent space feature, text feature and image feature are input into the video denoising model again for noise prediction, and the process is iterated until the preset iteration threshold is reached to obtain the intermediate feature.

Images